Generation method of fluorescein angiography image and training method of generation model

Abstract

The embodiment of the invention relates to the technical field of cross-modal medical image generation, provides a generation method of a fluorescein angiography image and a training method of a generation model, and designs a potential diffusion model-based deep learning framework, namely the generation model of the fluorescein angiography image, the generation model is composed of two parts, one is an automatic encoder capable of encoding an image into a hidden space, the automatic encoder comprises an encoder and a decoder, the other is a diffusion network used for learning image features, and the image encoder is combined to perform staged training on the generation model by utilizing an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image, so that the generation model is obtained. According to the method, the accuracy of model generation information is effectively improved, the generation effect of the fundus fluorescein angiography is improved, an ophthalmologist can be better assisted in completing fundus examination, and assistance is provided for diagnosis of eye diseases.

Description

Technical Field

[0001] The present disclosure relates to the technical field of cross-modal medical image generation, and in particular to a method for generating fluorescein angiography images and a training method for generating a model. Background Art

[0002] Fundus photography is an important means of clinically diagnosing fundus diseases. In recent years, the clinical applications of ultra-wide-field (UWF) fundus photography, ultra-wide-field fluoresceinangiography (UWF-FA), and ultra-wide-field scanning laser ophthalmoscopy (UWF-SLO) have gradually expanded due to their large imaging range. For example, UWF-FA can be used to comprehensively detect early signs of diabetic retinopathy and can also be used to identify intraocular tumors such as choroidoma and retinal hemangioma. At the same time, UWF-SLO can also help doctors comprehensively determine the type and degree of retinal detachment, so as to formulate appropriate treatment plans. However, UWF-FA examination requires the injection of sodium fluorescein into the human body, which may cause adverse reactions such as nausea, vomiting, anaphylactic shock, and even life-threatening.

[0003] In recent years, Generative Adversarial Network (GAN) and its variants have been widely used in this field.

[0004] For example, Kamran et al. proposed a semi-supervised conditional GAN, namely VTGAN, which can capture the features of fundus color photographs (CFP) images across scales and generate realistic fluorescein fundus angiography (FFA) images through the combination of multi-scale coarse and fine generators, thereby improving the capture of small blood vessel information and realizing the synthesis of fluorescein angiography images from fundus color photographs. At the same time, VTGAN introduced Transformer in the discriminator to distinguish between healthy and diseased FFAs and diagnose vascular diseases, which helps to synthesize vivid images. However, VTGAN can only synthesize fluorescein angiography images from fundus color photographs, which has poor performance for ultra-wide-angle fundus imaging images. In addition, the resolution of VTGAN synthesized images is low. VTGAN can only synthesize fluorescein angiography images of size 512×512, which cannot meet the actual needs of high resolution. In addition, due to the limitations of the generative adversarial network itself, VTGAN is also prone to training instability and mode collapse.

[0005] Fang et al. proposed a UWAT-GAN method, and for the first time proposed a study on synthesizing UWF-FA from UWF-SLO. By using a multi-scale discriminator and a multi-scale generator, the limitations of UWF-FA imaging were overcome. By using a combination of a global generator and a local generator, the global and local features of ultra-wide-angle fundus imaging images can also be extracted. However, the UWAT-GAN method is not accurate enough in generating vascular information and lesion information, and cannot provide accurate and complete vascular information for the detection and diagnosis of various ophthalmic diseases. In addition, the UWF-FA generated by the UWAT-GAN method as a whole has many artifacts and little usable information.

[0006] Although the above GAN-based method is very powerful for generation tasks, it also faces challenges such as training instability and mode collapse. At the same time, the high resolution of fundus images leads to more image artifacts in GAN-generated images, which cannot provide effective assistance for clinical diagnosis.

[0007] Recently, the diffusion model has become one of the most promising generative models. It consists of two basic processes, namely the forward process and the backward process. The forward process adds noise and the backward process gradually removes the noise in the image. In the field of medical imaging, Wu et al. first proposed a diffusion-based medical image segmentation model MedSegDiff, which introduced the diffusion model into the medical field. Jiang et al. proposed Cola-Diff, which first introduced diffusion into magnetic resonance imaging (MRI) synthesis and realized the generation of multimodal nuclear magnetic resonance images. However, the randomness of the diffusion model generation process is in sharp contrast to the determinism of cross-modal image generation. In addition, the performance of the diffusion model will decrease as the amount of data decreases, which makes it difficult to apply the diffusion-based model to FFA image generation.

[0008] In summary, the above FFA image generation method has the following two main shortcomings:

[0009] 1. The above method is not accurate enough in generating vascular information and lesion information, and is of limited help in the detection and diagnosis of various ophthalmic diseases.

[0010] 2. Due to the limited medical data, the performance of the above methods is seriously affected. Summary of the invention

[0011] The present disclosure aims to solve at least one of the problems existing in the prior art and provides a method for generating a fluorescein angiography image and a training method for generating a model.

[0012] In one aspect of the present disclosure, a method for training a generative model of fluorescein angiography images is provided, wherein the generative model comprises an autoencoder and a diffusion network; the training method comprises:

[0013] Obtain ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images;

[0014] Using the autoencoder, encoding the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image into the same latent space to obtain a latent representation of the ultra-wide-angle fundus imaging image and a latent representation of the corresponding fluorescein angiography image;

[0015] Using a pre-trained image encoder, encoding the ultra-wide-angle fundus imaging image into an image embedding to obtain an ultra-wide-angle fundus imaging image embedding;

[0016] Training the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image;

[0017] Inputting the fluorescent angiography image into the generation model to obtain a corresponding fluorescent angiography generation image;

[0018] The autoencoder is trained based on the fluorescein angiography image and its corresponding fluorescein angiography generated image.

[0019] Optionally, the training of the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image comprises:

[0020] Performing noise processing on the latent representation of the fluorescein angiography image by using Gaussian noise to obtain a noisy latent representation of the fluorescein angiography image;

[0021] splicing the ultra-wide-angle fundus imaging image latent representation and the corresponding noisy fluorescein angiography image latent representation together and inputting them into a denoising diffusion network, and embedding the ultra-wide-angle fundus imaging image into a cross-attention layer input into the denoising diffusion network, so that the denoising diffusion network outputs predicted noise;

[0022] Correcting the Gaussian noise and the predicted noise using a first loss function to obtain updated Gaussian noise;

[0023] The updated Gaussian noise is used to re-noise the latent representation of the fluorescein angiography image.

[0024] Optionally, the denoising diffusion network includes a denoising U-net network; the cross-attention layer includes a Transformer layer; and the first loss function includes a mean square error loss function.

[0025] Optionally, the training of the autoencoder based on the fluorescent angiography image and the corresponding fluorescent angiography generated image includes:

[0026] Freeze the encoder parameters in the autoencoder and the parameters of the denoising diffusion network, use a second loss function to constrain the fluorescent angiography image and its corresponding fluorescent angiography generated image, and adjust the decoder parameters in the autoencoder to make the fluorescent angiography generated image closer to the fluorescent angiography image.

[0027] Optionally, the second loss function includes a generation loss function and an identification loss function; wherein the identification loss function includes a binary cross entropy loss function.

[0028] Optionally, the step of obtaining an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image includes:

[0029] Based on the stable diffusion V1.5 model, image augmentation was performed on ultra-wide-angle fundus imaging and its corresponding fluorescein angiography images.

[0030] Another aspect of the present disclosure provides a method for generating a fluorescein angiography image, the method comprising:

[0031] Obtain target ultra-wide-angle fundus imaging;

[0032] The target ultra-wide-angle fundus imaging image is input into a trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image; wherein the generation model of the fluorescein angiography image is trained using the training method of the generation model of the fluorescein angiography image described above.

[0033] Another aspect of the present disclosure provides a training device for a generation model of fluorescein angiography images, wherein the generation model of fluorescein angiography images includes an autoencoder and a diffusion network; the training device includes:

[0034] An acquisition module, used for acquiring an ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image;

[0035] An encoding module, used for encoding the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image into the same latent space by using the autoencoder, to obtain a latent representation of the ultra-wide-angle fundus imaging image and a latent representation of the corresponding fluorescein angiography image;

[0036] An embedding module, used to encode the ultra-wide-angle fundus imaging image into an image embedding using a pre-trained image encoder to obtain an image embedding of the ultra-wide-angle fundus imaging image;

[0037] A first training module, configured to train the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image;

[0038] A generation module, used for inputting the fluorescent angiography image into the generation model to obtain a corresponding fluorescent angiography generation image;

[0039] The second training module is used to train the automatic encoder based on the fluorescent angiography image and the corresponding fluorescent angiography generated image.

[0040] Another aspect of the present disclosure provides a device for generating a fluorescein angiography image, the device comprising:

[0041] An image acquisition module, used to acquire a target ultra-wide-angle fundus imaging image;

[0042] An image generation module is used to input the target ultra-wide-angle fundus imaging image into a trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image; wherein the generation model of the fluorescein angiography image is trained using the training method of the generation model of the fluorescein angiography image described above.

[0043] Another aspect of the present disclosure provides an electronic device, including:

[0044] at least one processor; and,

[0045] a memory communicatively connected to at least one processor; wherein,

[0046] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the training method of the generation model of the fluorescein angiography image or the generation method of the fluorescein angiography image described above.

[0047] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method of the generation model of fluorescein angiography images or the generation method of fluorescein angiography images described above.

[0048] Compared with the prior art, the present invention designs a deep learning framework based on a latent diffusion model, namely, a generative model of fluorescein angiography images. The generative model consists of two parts: one is an autoencoder for encoding images into a latent space, and the autoencoder includes an encoder and a decoder; the other is a diffusion network for learning image features. The generative model is trained in stages using ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images in combination with the image encoder, which effectively improves the accuracy of the model generation information and the generation effect of the fundus fluorescein angiography images, and can better assist ophthalmologists in completing fundus examinations and provide assistance for the diagnosis of eye diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0050] Figure 1 A flowchart of a method for training a generation model of fluorescein angiography images provided in one embodiment of the present disclosure;

[0051] Figure 2 A schematic diagram of the overall framework of a generation model of fluorescein angiography images provided by another embodiment of the present disclosure;

[0052] Figure 3 A schematic diagram of a denoising diffusion network provided in accordance with another embodiment of the present disclosure;

[0053] Figure 4 A flowchart of a method for generating a fluorescein angiography image provided by another embodiment of the present disclosure;

[0054] Figure 5 A schematic diagram of the structure of a training device for a generation model of fluorescein angiography images provided by another embodiment of the present disclosure;

[0055] Figure 6 A schematic structural diagram of a device for generating fluorescein angiography images provided by another embodiment of the present disclosure;

[0056] Figure 7 A schematic structural diagram of an electronic device provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that in each embodiment of the present disclosure, many technical details are proposed in order to enable readers to better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed for protection in the present disclosure can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present disclosure. The various embodiments can be combined and referenced with each other without contradiction.

[0058] One embodiment of the present disclosure relates to a method for training a generation model of fluorescein angiography images, the process of which is as follows: Figure 1 As shown in Figure 1, the generative model of fluorescein angiography images consists of two parts, namely, an autoencoder and a diffusion network. The autoencoder includes an encoder and a decoder, which are located at the input and output of the diffusion network respectively. The autoencoder can encode the image into a latent space, while the diffusion network can learn the features of the image. The overall framework of the generative model is shown in Figure 1. Figure 2 shown.

[0059] like Figure 1 As shown, the training method of the generation model of fluorescein angiography images provided in this embodiment includes steps S110 to S160. Figure 2 , steps S110 to S160 are described in detail.

[0060] Step S110, obtaining an ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image.

[0061] Specifically, step S110 can obtain a preset number of ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images to construct a data set required for model training. The fluorescein fundus angiography image can be a real image taken with an ophthalmoscope or a fundus camera, or an image generated with other tools, and this embodiment does not limit this. In order to avoid adverse reactions to the human body caused by taking real fluorescein fundus angiography images with an ophthalmoscope or a fundus camera, this embodiment chooses to use a scanning laser ophthalmoscope (Scanning Laser Ophthalmoscopy, SLO) to generate a fluorescein angiography image.

[0062] In particular, in order to solve the problem of small data volume of ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images, step S110 may also include: based on the stable diffusion V1.5 model, performing image expansion on the ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images, thereby effectively increasing the number of images in the data set and solving the problem of limited medical data sets affecting model performance.

[0063] Step S120, using an automatic encoder, encodes the ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image into the same latent space to obtain a latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image.

[0064] Specifically, step S120 mainly performs data preprocessing on the ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image. In order to realize the generation of fluorescein angiography image from the ultra-wide-angle fundus imaging image, step S120 encodes the ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image into the same latent space, so as to operate the ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image in the same latent space, thereby enhancing data relevance.

[0065] Step S130, using a pre-trained image encoder, encodes the ultra-wide-angle fundus imaging image into an image embedding to obtain an ultra-wide-angle fundus imaging image embedding.

[0066] Specifically, the present embodiment does not limit the specific type of the image encoder, as long as the pre-trained image encoder can encode the ultra-wide-angle fundus imaging image to obtain the corresponding ultra-wide-angle fundus imaging image embedding.

[0067] Step S140 , training a diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image.

[0068] Specifically, step S120 to step S140 belong to the first stage of model training, and are used to train the diffusion network in the generation model of fluorescein angiography images.

[0069] Exemplary, combined Figure 3Step S140 specifically includes: using Gaussian noise to perform noise processing on the potential representation of the fluorescein angiography image to obtain the noisy potential representation of the fluorescein angiography image; splicing the potential representation of the ultra-wide-angle fundus imaging image and its corresponding noisy potential representation of the fluorescein angiography image together and inputting them into the denoising diffusion network, and embedding the ultra-wide-angle fundus imaging image into the cross-attention layer of the input denoising diffusion network, so that the denoising diffusion network outputs prediction noise; using the first loss function to correct the Gaussian noise and the prediction noise to obtain updated Gaussian noise; and using the updated Gaussian noise to perform noise processing on the potential representation of the fluorescein angiography image again.

[0070] Specifically, this embodiment sets the diffusion network as a denoising diffusion network, and trains the denoising diffusion network through a forward denoising process and a reverse denoising process, so that the denoising diffusion network can learn semantic features. Among them, the denoising diffusion network includes a denoising U-net network. The cross attention layer includes a Transformer layer. The first loss function includes a mean squared error loss (MSE) function.

[0071] In other words, the first stage of model training mainly includes the following contents: (1) The denoising diffusion network takes the processed ultra-wide-angle fundus imaging image and its paired fluorescein angiography image, i.e., the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image as input. (2) An ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image are encoded into the same latent space using an autoencoder so as to operate in the same latent space in subsequent steps. At the same time, a pre-trained image encoder is used to encode the ultra-wide-angle fundus imaging image into an image embedding to obtain an ultra-wide-angle fundus imaging image embedding. (3) A forward denoising and a reverse denoising process are performed in the aforementioned latent space. Specifically, the latent representation of the ultra-wide-angle fundus imaging image and the latent representation of the fluorescein angiography image after denoising are concatenated together and input into a denoising U-net network. At the same time, the image embedding of the ultra-wide-angle fundus imaging image is input into the denoising U-net network so that the image embedding of the ultra-wide-angle fundus imaging image performs a cross-attention operation with the Transformer layer in the denoising U-net network, thereby realizing feature injection of the ultra-wide-angle fundus imaging image. Among them, this embodiment selects the mean square error loss function as the loss function used for training the denoising diffusion network, and uses the mean square error loss function to correct the pixel-level difference between the Gaussian noise added to the potential representation of the fluorescein angiography image and the predicted noise output by the denoising diffusion network at each step, thereby realizing forward denoising. For example, the forward denoising can be iterated for 1000 steps to better improve the training effect of the denoising diffusion network.

[0072] Step S150, inputting the fluorescent angiography image into the generation model to obtain a corresponding fluorescent angiography generation image.

[0073] Specifically, the fluorescence angiography image here is a real fluorescence angiography image, and the corresponding fluorescence angiography generated image is a generated fluorescence angiography image, which can be generated based on the real fluorescence angiography image using the generation model of fluorescein angiography images trained in the first stage.

[0074] Step S160 , training the autoencoder based on the fluorescein angiography image and its corresponding fluorescein angiography generated image.

[0075] Specifically, step S160 belongs to the second stage of model training, which focuses on enhancing the model's ability to acquire high-frequency information of images.

[0076] Exemplarily, step S160 includes: freezing the encoder parameters in the autoencoder and the parameters of the denoising diffusion network, using a second loss function to constrain the fluorescent angiography image and its corresponding fluorescent angiography generated image, and adjusting the decoder parameters in the autoencoder to make the fluorescent angiography generated image closer to the fluorescent angiography image.

[0077] Specifically, in the second stage of model training, this embodiment freezes the parameters corresponding to the encoder in the autoencoder, i.e., the encoder parameters, and at the same time freezes the parameters of the denoising diffusion network that has completed the first stage of training, and only fine-tunes the parameters corresponding to the decoder in the autoencoder, i.e., the decoder parameters, to improve the model's ability to obtain high-frequency information. Among them, the second loss function includes a generation loss function and an identification loss function. Among them, the identification loss function includes a binary cross entropy loss function (Binary Cross Entropy Loss).

[0078] In other words, the second stage of model training mainly includes the following: (1) Freeze the parameters of the denoising U-Net trained in the first stage, and freeze the encoder parameters of the autoencoder, and only fine-tune the decoder parameters in the autoencoder. (2) The generated fluorescein angiography image, that is, the generated fluorescein angiography image, is constrained by the second loss function with the real fluorescein angiography image, so that the generated fluorescein angiography image is closer to its corresponding real fluorescein angiography image. (3) The second loss function in this stage includes two parts, namely the generation loss function and the identification loss function. Among them, the generation loss function consists of perceptual loss, reconstruction loss and generation loss. The perceptual loss is mainly used to evaluate the difference between the generated image and the real image in high-level features, the reconstruction loss is used to evaluate the difference between the generated image and the original input image, and the generation loss is used to evaluate the difference between the image generated by the generator and the real image. The identification loss function uses the binary cross entropy loss function to measure the classification performance of the discriminator for real images and generated images.

[0079] The training method of the generative model of fluorescein angiography images provided in the embodiment of the present disclosure, compared with the prior art, designs a deep learning framework based on the latent diffusion model, namely, the generative model of fluorescein angiography images. The generative model consists of two parts: one is an automatic encoder for encoding images into a latent space, and the automatic encoder includes an encoder and a decoder; the other is a diffusion network for learning image features. The generative model is trained in stages using ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images in combination with the image encoder, which effectively improves the accuracy of the model generation information and the generation effect of the fundus fluorescein angiography images, can better assist ophthalmologists in completing fundus examinations, and provide assistance for the diagnosis of eye diseases.

[0080] Another embodiment of the present disclosure relates to a method for generating a fluorescein angiography image, the process of which is as follows: Figure 4 As shown, it includes steps S410 to S420.

[0081] Step S410, obtaining a target ultra-wide-angle fundus imaging image.

[0082] Specifically, the target ultra-wide-angle fundus imaging image refers to an ultra-wide-angle fundus imaging image for which a corresponding fluorescein angiography image needs to be generated.

[0083] Step S420, input the target ultra-wide-angle fundus imaging image into the trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image. The generation model of fluorescein angiography images is trained using the training method of the generation model of fluorescein angiography images provided in the above embodiment.

[0084] Compared with the prior art, the method for generating fluorescein angiography images provided in the embodiments of the present disclosure can effectively improve the generation effect of fundus fluorescein angiography images by using a trained generation model of fluorescein angiography images, thereby better assisting ophthalmologists in completing fundus examinations and providing assistance in the diagnosis of eye diseases.

[0085] Another embodiment of the present disclosure relates to a training device for a generation model of fluorescein angiography images, wherein the generation model of fluorescein angiography images includes an autoencoder and a diffusion network. Figure 5 As shown, the training device for the generation model of the light element angiography image includes an acquisition module 510, an encoding module 520, an embedding module 530, a first training module 540, a generation module 550, and a second training module 560.

[0086] The acquisition module 510 is used to acquire an ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image.

[0087] The encoding module 520 is used to encode the ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image into the same latent space using an automatic encoder to obtain a latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image.

[0088] The embedding module 530 is used to encode the ultra-wide-angle fundus imaging image into an image embedding using a pre-trained image encoder to obtain an image embedding of the ultra-wide-angle fundus imaging image.

[0089] The first training module 540 is used to train the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image.

[0090] The generation module 550 is used to input the fluorescent angiography image into the generation model to obtain the corresponding fluorescent angiography generation image.

[0091] The second training module 560 is used to train the autoencoder based on the fluorescein angiography image and its corresponding fluorescein angiography generated image.

[0092] The specific implementation method of the training device for the generation model of fluorescein angiography images provided in the embodiments of the present disclosure can be found in the training method for the generation model of fluorescein angiography images provided in the embodiments of the present disclosure, and will not be repeated here.

[0093] The training device for the generative model of fluorescein angiography images provided in the embodiment of the present disclosure, compared with the prior art, designs a deep learning framework based on the latent diffusion model, namely, a generative model of fluorescein angiography images. The generative model consists of two parts: one is an automatic encoder for encoding images into a latent space, and the automatic encoder includes an encoder and a decoder; the other is a diffusion network for learning image features. The generative model is trained in stages using ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images in combination with the image encoder, which effectively improves the accuracy of the model generation information and the generation effect of the fundus fluorescein angiography images, can better assist ophthalmologists in completing fundus examinations, and provide assistance for the diagnosis of eye diseases.

[0094] Another embodiment of the present disclosure relates to a device for generating a fluorescein angiography image, such as Figure 6 As shown, it includes an image acquisition module 610 and an image generation module 620.

[0095] The image acquisition module 610 is used to acquire a target ultra-wide-angle fundus imaging image.

[0096] The image generation module 620 is used to input the target ultra-wide-angle fundus imaging image into the trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image; wherein the generation model of the fluorescein angiography image is trained using the training method of the generation model of the fluorescein angiography image provided in the above-mentioned embodiment.

[0097] The specific implementation method of the device for generating fluorescein angiography images provided in the embodiments of the present disclosure can be found in the method for generating fluorescein angiography images provided in the embodiments of the present disclosure, and will not be described in detail here.

[0098] Compared with the prior art, the device for generating fluorescein angiography images provided in the embodiments of the present disclosure can effectively improve the generation effect of fundus fluorescein angiography images by using a trained generation model of fluorescein angiography images, thereby better assisting ophthalmologists in completing fundus examinations and providing assistance for the diagnosis of eye diseases.

[0099] Another embodiment of the present disclosure relates to an electronic device, such as Figure 7 As shown, including:

[0100] at least one processor 701; and,

[0101] A memory 702 is communicatively connected to at least one processor 701; wherein,

[0102] The memory 702 stores instructions that can be executed by at least one processor 701, and the instructions are executed by at least one processor 701 so that the at least one processor 701 can execute the training method of the generation model of the fluorescein angiography image or the generation method of the fluorescein angiography image described in the above embodiment.

[0103] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0104] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0105] Another embodiment of the present disclosure relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for training a generation model of fluorescein angiography images or the method for generating fluorescein angiography images described in the above embodiments.

[0106] That is, those skilled in the art can understand that all or part of the steps in the method described in the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0107] Those skilled in the art will appreciate that the above-mentioned embodiments are specific embodiments for implementing the present disclosure, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present disclosure.

Claims

1. A method for training a generative model of fluorescein angiography images, characterized in that: The generation model includes an autoencoder and a diffusion network; the training method includes: Obtain ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images; Using the autoencoder, encoding the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image into the same latent space to obtain a latent representation of the ultra-wide-angle fundus imaging image and a latent representation of the corresponding fluorescein angiography image; Using a pre-trained image encoder, encoding the ultra-wide-angle fundus imaging image into an image embedding to obtain an ultra-wide-angle fundus imaging image embedding; Training the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image; Inputting the fluorescent angiography image into the generation model to obtain a corresponding fluorescent angiography generation image; The autoencoder is trained based on the fluorescein angiography image and its corresponding fluorescein angiography generated image.

2. The training method according to claim 1, characterized in that: The step of training the diffusion network based on the potential representation of the ultra-wide-angle fundus imaging image and its corresponding potential representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image comprises: Performing noise processing on the latent representation of the fluorescein angiography image by using Gaussian noise to obtain a noisy latent representation of the fluorescein angiography image; splicing the ultra-wide-angle fundus imaging image latent representation and the corresponding noisy fluorescein angiography image latent representation together and inputting them into a denoising diffusion network, and embedding the ultra-wide-angle fundus imaging image into a cross-attention layer input into the denoising diffusion network, so that the denoising diffusion network outputs predicted noise; Correcting the Gaussian noise and the predicted noise using a first loss function to obtain updated Gaussian noise; The updated Gaussian noise is used to re-noise the latent representation of the fluorescein angiography image.

3. The training method according to claim 2, characterized in that: The denoising diffusion network includes a denoising U-net network; the cross attention layer includes a Transformer layer; and the first loss function includes a mean square error loss function.

4. The training method according to claim 2, characterized in that: The step of training the autoencoder based on the fluorescent angiography image and the corresponding fluorescent angiography generated image comprises: Freeze the encoder parameters in the autoencoder and the parameters of the denoising diffusion network, use a second loss function to constrain the fluorescent angiography image and its corresponding fluorescent angiography generated image, and adjust the decoder parameters in the autoencoder to make the fluorescent angiography generated image closer to the fluorescent angiography image.

5. The training method according to claim 4, characterized in that: The second loss function includes a generation loss function and an identification loss function; wherein the identification loss function includes a binary cross entropy loss function.

6. The training method according to any one of claims 1 to 5, characterized in that: The method of obtaining an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image comprises: Based on the stable diffusion V1.5 model, image augmentation was performed on ultra-wide-angle fundus imaging and its corresponding fluorescein angiography images.

7. A method for generating a fluorescein angiography image, characterized in that: The generation method comprises: Obtain target ultra-wide-angle fundus imaging; The target ultra-wide-angle fundus imaging image is input into a trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image; wherein the generation model of fluorescein angiography images is trained using the training method described in any one of claims 1 to 5.

8. A training device for a generation model of fluorescein angiography images, characterized in that: The generation model of the fluorescein angiography image includes an autoencoder and a diffusion network; the training device includes: An acquisition module, used for acquiring an ultra-wide-angle fundus imaging image and its corresponding fluorescein angiography image; An encoding module, used for encoding the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image into the same latent space by using the autoencoder, to obtain a latent representation of the ultra-wide-angle fundus imaging image and a latent representation of the corresponding fluorescein angiography image; An embedding module, used to encode the ultra-wide-angle fundus imaging image into an image embedding using a pre-trained image encoder, to obtain an image embedding of the ultra-wide-angle fundus imaging image; A first training module, configured to train the diffusion network based on the latent representation of the ultra-wide-angle fundus imaging image and its corresponding latent representation of the fluorescein angiography image and the image embedding of the ultra-wide-angle fundus imaging image; A generation module, used for inputting the fluorescent angiography image into the generation model to obtain a corresponding fluorescent angiography generation image; The second training module is used to train the automatic encoder based on the fluorescent angiography image and the corresponding fluorescent angiography generated image.

9. A device for generating a fluorescein angiography image, characterized in that: The generating device comprises: An image acquisition module, used to acquire a target ultra-wide-angle fundus imaging image; An image generation module is used to input the target ultra-wide-angle fundus imaging image into a trained generation model of fluorescein angiography images to obtain an ultra-wide-angle fluorescein angiography image; wherein the generation model of the fluorescein angiography image is trained using the training method described in any one of claims 1 to 5.

10. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method described in any one of claims 1 to 5 or the generation method described in claim 6.

Images