An accurate late UWF-FA image generation method based on LPUWF-LDM model

Abstract

The present invention discloses a method for accurately generating late-stage UWF-FA images based on the LPUWF-LDM model, comprising: S1: collecting and preprocessing image data; S2: building a network model for accurately generating late-stage UWF-FA images based on the LPUWF-LDM model. The overall framework of the network model for accurately generating late-stage UWF-FA images based on the LPUWF-LDM model mainly consists of three parts: a VAE module with a gated convolutional encoder, a noise addition module using low-frequency enhancement noise, and a noise prediction backbone trained using CTRD Loss. S3: repeatedly training the network model, optimizing network parameters, and continuously performing iterative optimization; S4: inputting the preprocessed UWF-SLO image into the trained network model for accurately generating late-stage UWF-FA images based on the LPUWF-LDM model, ultimately generating high-quality late-stage UWF-FA images.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal image conversion and generation, and in particular to an accurate late UWF-FA image generation method based on an LPUWF-LDM model. Background Art

[0002] Ultra-Wide-Field Fluorescein Angiography (UWF-FA) is a key dynamic imaging technology used for the diagnosis and treatment of fundus-related diseases. The process involves injecting fluorescein dye into the patient's vein, which then flows to the fundus, which may cause nausea or vomiting. There are also risks for patients with serious heart diseases. In contrast, Ultra-Wide-Field Scanning Laser Ophthalmoscopy (UWF-SLO) uses laser imaging to quickly scan the retina without adverse effects on the patient. However, the vascular images produced by UWF-SLO have less detail. UWF-FA is mainly divided into two stages: early and late, and doctors will interpret the images at specific times depending on the condition of the disease. Early images are particularly important for retinal vascular problems, such as central retinal artery occlusion and retinal vein occlusion; late images are more critical when evaluating retinal structure or barrier problems, such as central serous chorioretinopathy. Furthermore, some features of the disease become more apparent in later stages, such as the 'nut sign' that appears in late stages of central serous choroidal disease and tiny hemorrhages in diabetic retinopathy.

[0003] Previous studies, such as VTGAN and Reg-GAN, have attempted to address this challenge by using cross-modal generation techniques to convert UWF-SLO images into UWF-FA images. However, these methods rely solely on supervised learning of a single UWF-SLO and early-stage UWF-FA images, thereby ignoring the dynamic characteristics of UWF-FA images over time. For the assessment of diseases involving retinal structure, such as central serous chorioretinopathy, key information is often reflected in late-stage UWF-FA images. However, due to the scarcity of high-quality, paired UWF-SLO, early-stage, and late-stage UWF-FA datasets, generating high-quality late-stage UWF-FA images from UWF-SLO images has become a daunting challenge.

[0004] In general, the current technical problems are as follows:

[0005] (1) Generate high-quality late-stage UWF-FA with detailed lesion information to assist diagnosis; (2) Maintain the quality of the generated late-stage UWF-FA in the case of a limited paired dataset; (3) UWF-SLO and UWF-FA images have significant differences in imaging principles, information density, and lesion manifestations. It is necessary to effectively fuse the information of these two modalities to preserve key lesion features and improve image quality during the conversion process. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention proposes a method for generating accurate late UWF-FA images based on the LPUWF-LDM model. This method uses the input UWF-SLO image as the data source and automatically synthesizes high-resolution, highly clear, and detailed late UWF-FA images through advanced algorithms, thereby significantly improving image quality and diagnostic accuracy. In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] A method for generating an accurate late UWF-FA image based on the LPUWF-LDM model includes the following steps:

[0008] S1. Collect image data and preprocess them, wherein one set of image data includes paired UWF-SL0 images and early UWF-FA images, and the other set of image data includes paired UWF-SLO images, early UWF-FA images, and late UWF-FA images;

[0009] Regarding data preprocessing, to ensure data consistency, the image resolution of both datasets was adjusted to 853x682 pixels, and various enhancement measures were applied, including random rotation within [-5°, 5°], cropping to 512x512 pixels, and random flipping.

[0010] S2. Build an accurate late UWF-FA image generation network model based on the LPUWF-LDM model. The overall framework of the accurate late UWF-FA image generation network model based on the LPUWF-LDM model mainly consists of three parts: a VAE module with a gated convolutional encoder, a noise addition module using low-frequency enhancement noise, and a noise prediction backbone trained by a cross-temporal regional difference loss (CTRD loss) function.

[0011] The VAE module with gated convolutional encoder consists of a VAE backbone and a gated convolutional encoder. The VAE backbone contains an encoder and a decoder, as well as a discriminator to improve the quality of the generated image. First, the late UWF-FA image (y i) is fed into the encoder. It first passes through a convolutional layer and then through three downsampling layers, each of which consists of two residual blocks and one downsampling block. The output of the encoder provides the mean and variance in the latent space, from which the latent vector is sampled The latent vector is then fed into the decoder, which consists of three upsampling blocks. Since the decoder also receives additional spatial information, each upsampling block checks this additional information and merges it into the vector of the current layer if available. The vector then passes through two residual blocks and a self-attention block, and finally it is reconstructed through a group normalization layer and a convolutional layer. and convert it into the final output. It can be expressed as:

[0012]

[0013] Where σ represents a nonlinear computation, Conv represents a convolution operation, and · represents an element-wise multiplication. The entire VAE training consists of two steps. First, transfer learning is performed from the stable diffusion AutoencoderKL, and then it is fine-tuned on the dataset. The loss function used is:

[0014]

[0015] Where D is the discriminator, Perc is the perceptual loss, and KL is the Kullback-Leibler loss. The loss function of the D discriminator is:

[0016]

[0017] When training the gated convolutional encoder, only set its parameters to be trainable and remove the KL loss from the loss function while keeping other components unchanged. Continue training until the model is well fitted.

[0018] Regarding the noise addition module that utilizes low-frequency enhancement noise, this is achieved by adding a low-frequency noise component to the original Gaussian noise, which follows a standard normal distribution. To generate this low-frequency noise, a value a is first sampled from N(0,1) and a scaling factor β is sampled from N(0,0.5) to control the degree of low-frequency noise. Then, a and β are multiplied and the resulting product is replicated in the height and width dimensions of the image to create a low-frequency noise term ∈ L By adding this low-frequency noise term to the original noise, the new noise vector z t The identical distribution assumption is maintained. Therefore, the new z t It can be expressed as:

[0019]

[0020] Where ∈ and ∈ L represent pure noise and low-frequency noise respectively.

[0021] In the above technical solution, under the common noise adding strategy, the low frequency still maintains a relatively clear texture, while the strategy of the present invention makes the damage of high-frequency and low-frequency noise more balanced.

[0022] Regarding the design of the cross-time regional difference loss function. The overall learning goal based on the diffusion model is:

[0023]

[0024] Here, α is a hyperparameter representing the weight of this loss term. In the experiments, α is gradually increased from 0.25 to 1 in the first half of the training cycle and maintained at 1 in the second half of the training cycle. w represents the heat map mentioned above, which is related to the early and late stages of UWF-FA. Represents a noise image, where t represents the current step size of adding noise, x i Enter the condition.

[0025] S3, using the pre-processed UWF-SLO images and early and late UWF-FA as input to repeatedly train the network model, optimize the network parameters, and continuously perform iterative optimization;

[0026] S4. The preprocessed UWF-SLO image is input into the trained accurate late UWF-FA image generation network model based on the LPUWF-LDM model, and finally a high-quality late UWF-FA image is generated.

[0027] The present invention has the following characteristics and beneficial effects:

[0028] The innovative technical solution of this invention truly meets the medical need for automatically generating high-quality late-stage ultra-wide-angle fluorescein angiography images from ultra-wide-angle color fundus photographs. Its key features include: 1) A cross-temporal regional difference loss module is introduced to capture the dynamic changes in UWF-FA images from early to late stages. This design enables the model to more accurately reflect the true temporal progression of the disease when generating late-stage UWF-FA images, particularly with respect to lesion areas and fine details. 2) During the diffusion forward modeling process, this invention employs a low-frequency noise enhancement strategy. This strategy helps enhance the realism of the generated images, making the generated late-stage UWF-FA images visually closer to the actual images captured, thus increasing the reliability of medical diagnoses. 3) This invention utilizes a gated convolutional encoder to extract additional information from conditional images (such as UWF-SLO images). The gating mechanism adaptively controls the flow of information, improving the encoder's mapping capabilities on a limited dataset, thereby generating higher-quality late-stage UWF-FA images. The beneficial effects of the present invention are as follows: 1) By generating high-quality late-stage UWF-FA images, the present invention helps doctors to more accurately evaluate retinal structure and vascular lesions, thereby improving the diagnostic accuracy of fundus-related diseases. 2) Traditional UWF-FA requires injection of dye, which may cause adverse reactions to patients. The present invention reduces the need for dye injection and reduces the risk to patients by generating UWF-FA images from UWF-SLO images. 3) Due to the scarcity of existing high-quality and paired UWF-SLO and UWF-FA datasets, the successful application of the present invention provides new ideas and methods for similar medical image processing problems, and is expected to be promoted and applied in other fields. 4) The present invention has made important progress in the fields of cross-modal generation technology, latent diffusion models and medical image processing, providing a solid foundation and reference for future research and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 It is a schematic diagram of the process of the present invention;

[0031] Figure 2 The overall architecture of the accurate late UWF-FA image generation network model based on the LPUWF-LDM model of the present invention;

[0032] Figure 3The architecture diagram of the VAE module with a gated convolutional encoder.

[0033] Figure 4 (a), (b), and (c) are the comparison results of the real UWF-SLO image, the real late UWF-FA image, and the late UWF-FA image generated by the model. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0035] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0036] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0037] The present invention provides an accurate late UWF-FA image generation method based on the LPUWF-LDM model, such as Figure 1 The specific operations are as follows:

[0038] Step 1: Data Acquisition and Preprocessing. This example utilizes datasets from two partner hospitals. The dataset from the first hospital primarily consists of paired UWF-SL0 images and early UWF-FA images. In this example, these data were first rigorously screened to exclude images that could affect clinical diagnosis. Specific screening requirements included: 1) capture intervals no longer than three months; 2) visible fresh bleeding; 3) severe eyelash occlusion; and 4) poorly focused images. Ultimately, 304 pairs of high-quality images were selected from this hospital. The dataset from the second hospital provided paired UWF-SLO images, early UWF-FA images, and late UWF-FA images. After the same screening process, 387 sets of images were selected from this hospital, each containing all three image types. These images were then divided into 309 pairs as a training set and 78 pairs as a test set.

[0039] It should be noted that in this example, early UWF-FA images and late UWF-FA images were categorized according to physician definitions. Early UWF-FA images were taken 0-15 seconds after fluorescein dye injection, while late UWF-FA images were taken 10 minutes after fluorescein dye injection. Furthermore, regarding data preprocessing, to ensure data consistency, all images were resized to 853x682 pixels, and various enhancements were applied. These included random rotation within [-5°, 5°], cropping to 512x512 pixels, and random flipping. Specifically, in this example, these enhancements effectively expanded the original training dataset from 304 pairs to 19,456 pairs, and from 309 groups to 6,192 groups, totaling 25,648 UWF-SL0 images, 25,648 early UWF-FA images, and 6,192 late UWF-FA images. Finally, all early-stage UWF-FA and matching UWF-SLO images from the datasets provided by the two hospitals were used as the first dataset (a total of 25,648 UWF-SL0 images and 25,648 early-stage UWF-FA images). This dataset will be used for extensive model training, allowing the model to first learn the structural features of the images. All late-stage UWF-FA and matching UWF-SLO images from the datasets provided by the two hospitals were used as the second dataset (a total of 6,192 UWF-SL0 images and 6,192 early-stage UWF-FA images). The second dataset is used for training the model to generate late-stage UWF-FA images.

[0040] Step 2: If Figure 2 As shown in Figure 2, we build an accurate late UWF-FA image generation network model based on the LPUWF-LDM model. Specifically, the overall framework of the accurate late UWF-FA image generation network model based on the LPUWF-LDM model mainly consists of three parts: a VAE module with a gated convolutional encoder (see Figure 3 ), a noise addition module that utilizes low-frequency enhancement noise, and a noise prediction backbone trained with a cross-temporal region difference loss function.

[0041] Specifically, a VAE module with a gated convolutional encoder, such as Figure 3 As shown in Figure 2, it consists of a VAE backbone and a gated convolutional encoder. The VAE backbone contains an encoder and a decoder, as well as a discriminator to improve the quality of the generated image. First, the late UWF-FA image (y i ) is fed into the encoder. It first passes through a convolutional layer and then through three downsampling layers, each of which consists of two residual blocks and one downsampling block. The output of the encoder provides the mean and variance in the latent space, from which the latent vector is sampled The latent vector is then fed into the decoder, which consists of three upsampling blocks. Since the decoder also receives additional spatial information, each upsampling block checks this additional information and merges it into the vector of the current layer if available. The vector then passes through two residual blocks and a self-attention block, and finally it is reconstructed through a group normalization layer and a convolutional layer. and convert it into the final output. It can be expressed as:

[0042]

[0043] Where σ represents a nonlinear computation, Conv represents a convolution operation, and · represents an element-wise multiplication. The entire VAE training consists of two steps. First, transfer learning is performed from the stable diffusion AutoencoderKL, and then it is fine-tuned on the dataset. The loss function used is:

[0044]

[0045] Where D is the discriminator, Perc is the perceptual loss, and KL is the Kullback-Leibler loss. The loss function of the D discriminator is:

[0046]

[0047] When training the gated convolutional encoder, only its parameters are set as trainable and the KL loss is removed from the loss function, while keeping the other components unchanged. Training is continued until the model is well fitted. Regarding the noise addition module that utilizes low-frequency enhancement noise, the traditional approach is to add the same distribution to the image. Gaussian noise affects high-frequency and low-frequency information differently, and when the noisy UWF-FA image is decomposed into these components, a clear difference in noise injection between high-frequency and low-frequency regions is observed. Specifically, the low-frequency region is less affected by noise than the high-frequency region. This difference impairs the model's ability to recover rich low-frequency details in medical images. To address this issue, the present embodiment adds noise interference to the low-frequency information. This is achieved by adding a low-frequency noise component to the original Gaussian noise, which follows a standard normal distribution.

[0048] Specifically, in order to generate this low-frequency noise, first sample a value a from N(0,1) and a scaling factor β from N(0,0.5) to control the degree of low-frequency noise. Then, multiply a and β and replicate the resulting product in the height and width dimensions of the image to create a low-frequency noise term ∈ L Under the common noise adding strategy, the low frequency still maintains a relatively clear texture, while the strategy of this embodiment makes the damage of high frequency and low frequency noise more balanced. By adding this low frequency noise term to the original noise, the new noise vector z t The identical distribution assumption is maintained. Therefore, the new z t It can be expressed as:

[0049]

[0050] Where ∈ and ∈ L represent pure noise and low-frequency noise respectively.

[0051] Regarding the cross-temporal regional difference loss function. It should be noted that the early UWF-FA is very clear in terms of structural information, such as the locations of arteries, veins, and optic discs. Therefore, the model can better learn the mapping from UWF-SLO to early UWF-FA. However, although structural information can guide the realism of image generation, it is not conducive to generating detailed information related to lesions, because more lesion information is actually reflected in the later UWF-FA. Therefore, the designed function mainly solves two problems: 1) how to locate the lesion area without manual annotation; 2) how to make the diffusion model pay more attention to these areas.

[0052] Therefore, in this embodiment, a cross-time regional difference loss function is proposed. Specifically, the early and late UWF-FA images are first aligned, and then the absolute difference is calculated pixel by pixel, and the result is normalized to the range of (0,1) to obtain a heat map with the same shape and size as the UWF-FA image. Areas with high values ​​indicate that the difference between the early and late stages is large, and areas with low values ​​indicate that the difference is small. After statistics, the average difference induced by fluorescence leakage is 0.87, and the average difference induced by neovascularization is 0.71. Positions with higher values ​​indicate that the model should pay more attention to generating these areas, forming an unsupervised attention to the lesion area. Since diffusion is trained in the latent space, the heat map is resized to match the same size as the latent space. Therefore, the overall learning goal based on the diffusion model is:

[0053]

[0054] Here, α is a hyperparameter representing the weight of this loss term. In this example, α is gradually increased from 0.25 to 1 in the first half of the training cycle and remains at 1 in the second half of the training cycle. w represents the aforementioned heat map, which is related to the early and late stages of UWF-FA. Represents a noise image, where t represents the current step size of adding noise, x i Enter the condition.

[0055] Step 3: In this example, a precise late UWF-FA image generation network model based on the LPUWF-LDM model was constructed using the PyTorch 2.0 and CUDA 11.8 frameworks. It was trained on two NVIDIA A100 GPUs, encompassing both the VAE component and the noise prediction backbone. The noise prediction backbone was first extensively trained on the first dataset, with 1000 epochs and a batch size of 32 for 56 hours. Training was then performed on the second dataset, with both the VAE component and the noise prediction backbone trained for 1000 epochs. The VAE component used a batch size of 8 for 52 hours, while the noise prediction backbone used a batch size of 32 for 33 hours. The Adam algorithm was used for optimization during training, with a learning rate of 0.0002 for the VAE component and 0.0001 for the noise prediction backbone.

[0056] Step 4: In the model inference phase, the test set is applied and the UWF-SLO image is input. After the UWF-SLO has undergone the preprocessing phase, the trained accurate late UWF-FA image generation network model based on the LPUWF-LDM model is used to finally generate a high-quality late UWF-FA image. Figure 4From left to right, the comparison results of the real UWF-SLO image, the real late UWF-FA image and the late UWF-FA image generated by the model are shown respectively; Figure 4 From left to right, we show a comparison of ultra-wide-angle fundus color photographs, real late ultra-wide-angle fluorescein angiography images, and model-generated late ultra-wide-angle fluorescein angiography images. The first two rows show the normal conversion from UWF-SLO images to late UWF-FA images, while the last row shows the conversion in the presence of a significant lesion. The method in this example performs well in generating vascular and optic disc structures, with more continuous vessels and a thickness closer to reality. Furthermore, in the third row, our method is more successful than other methods in generating laser scars, and our method is able to correctly identify and convert lesions even with a small dataset. This demonstrates the particular advantages of our model for generation tasks with limited data.

[0057] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating accurate late UWF-FA images based on the LPUWF-LDM model, characterized in that: The steps include: S1. Collect and preprocess two sets of image data, where one set of image data includes paired UWF-SL0 images and early UWF-FA images, and the other set of image data includes paired UWF-SLO images, early UWF-FA images, and late UWF-FA images; S2. Building an accurate late UWF-FA image generation network model based on the LPUWF-LDM model, wherein the accurate late UWF-FA image generation network model based on the LPUWF-LDM model includes a VAE module with a gated convolutional encoder, a noise addition module using low-frequency enhancement noise, and a noise prediction backbone trained by a cross-temporal region difference loss function; The VAE module with a gated convolutional encoder includes a VAE backbone and a gated convolutional encoder, wherein the VAE backbone includes an encoder and a decoder, as well as a discriminator to improve the quality of the generated image; The implementation method of the VAE module with gated convolutional encoder is as follows: The late UWF-FA image y i The output of the encoder provides the mean and variance in the latent space, from which the latent vector is sampled. The latent vector is then fed into the decoder, which consists of three upsampling blocks. Since the decoder also receives additional spatial information, each upsampling block checks this additional information and merges it into the vector of the current layer if available. The vector then passes through two residual blocks and a self-attention block, and finally it is reconstructed through a group normalization layer and a convolutional layer. and convert it into the final output, It can be expressed as: Where σ represents nonlinear computation, Conv represents convolution operation, and · represents element-wise multiplication. S3, using the preprocessed UWF-SLO image and early and late UWF-FA as input to repeatedly train the accurate late UWF-FA image generation network model based on the LPUWF-LDM model, optimize the network parameters, and continuously perform iterative optimization; S4. The preprocessed UWF-SLO image is input into the trained accurate late UWF-FA image generation network model based on the LPUWF-LDM model, and finally a high-quality late UWF-FA image is generated.

2. The method for generating an accurate late UWF-FA image based on the LPUWF-LDM model according to claim 1, characterized in that: The preprocessing method is as follows: the image resolution of the two datasets is adjusted to 853x682 pixels, and data augmentation is performed by randomly rotating within [-5°, 5°], cropping to 512x512 pixels, and randomly flipping.

3. The method for generating an accurate late UWF-FA image based on the LPUWF-LDM model according to claim 1, characterized in that: The training method of the VAE module with the gated convolutional encoder includes: First, transfer learning is performed from the Stable Diffusion AutoencoderKL, which is then fine-tuned on the dataset using the following loss function: Where D is the discriminator, Perc is the perceptual loss, and KL is the Kullback-Leibler loss; When training the gated convolutional encoder, only its parameters are set to trainable and the KL loss is removed from the loss function while keeping other components unchanged. Continue training until the model is well fitted.

4. The method for generating an accurate late UWF-FA image based on the LPUWF-LDM model according to claim 3, characterized in that: The loss function of the D discriminator is:

5. The method for generating an accurate late UWF-FA image based on the LPUWF-LDM model according to claim 1, characterized in that: In step S2, the method of adding noise using the noise adding module of low-frequency enhanced noise is: First, a value a is sampled from N(0,1), and a scaling factor β is sampled from N(0,0.5) to control the degree of low-frequency noise; Then, a and β are multiplied and the resulting product is replicated across the height and width dimensions of the image, creating a low-frequency noise term ∈ L ; By adding this low-frequency noise term to the original noise, the new noise vector z t The same distribution assumption is maintained, so the new z t It can be expressed as: Where ∈ and ∈ L represent pure noise and low-frequency noise respectively.

6. The method for generating an accurate late UWF-FA image based on the LPUWF-LDM model according to claim 1, characterized in that: In step S3, the cross-time region difference loss function is: Where α is a hyperparameter representing the weight of this loss term, and w represents the heat map, which is related to the early and late UWF-FA. Represents a noise image, where t represents the current step size of adding noise, x i Enter the condition.

Images