Fluorescein angiography image generation model training method, generation method and device

Abstract

The embodiments of the present disclosure disclose a training method, a generation method and a device for a fluorescein angiography image generation model. The generation model includes a global generation module, a local generation module and a multi-scale discriminator module; the training method includes: obtaining an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image; performing vascular segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding vascular information image; based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, respectively training the global generation module and the local generation module, and using the multi-scale discriminator module to judge the generated image to obtain a trained generation model. This embodiment inputs vascular information into the model as a condition, emphasizes the vascular structure, improves the accuracy of vascular information, and makes the vascular information of the synthesized fluorescein angiography image more accurate, thereby better assisting the analysis of the structure and the diagnosis of the disease.

Description

Technical Field

[0001] The embodiments of the present disclosure belong to the field of computer vision technology, and specifically relate to a fluorescein angiography image generation model training method, generation method and device. Background Art

[0002] Fundus imaging is an important examination for clinical diagnosis of fundus diseases. In recent years, ultra-wide-angle fundus imaging (UWF), ultra-wide-angle fluorescein angiography (UW-FFA) and ultra-wide-angle scanning laser ophthalmoscope (UWF-SLO) technologies have been widely used. For example, UW-FFA is used to detect early signs of retinal vascular diseases such as diabetic retinopathy; UW-FFA is used to detect intraocular tumors, including choroidoma and retinal hemangioma; UWF-SLO is used to help doctors determine the type and degree of retinal detachment and help develop a treatment plan. Among them, UWF-FFA examination requires the injection of sodium fluorescein, which can cause adverse reactions such as nausea, vomiting, anaphylactic shock and even death.

[0003] In recent years, Generative Adversarial Networks (GANs) and their variants have made breakthroughs in this field. For example, in order to synthesize clearer images, the idea of ​​PatchGAN is proposed. An end-to-end multi-input and multi-output deep adversarial learning network for magnetic resonance imaging (MR) image synthesis. A Transfer-GAN model that combines transfer learning with GAN to improve the resolution of multimodal CT images.

[0004] By incorporating multi-scale generators, these networks can explore both fine and coarse features from images. For example, a semi-supervised model, called VTGAN, introduces a Transfomer module in the discriminator, which helps in vivid synthesis of images. However, previous methods produce lower resolution situations, and most discriminators can only accept square inputs (width equals height). In addition, data inconsistency, low attention to disease-related areas, and poor correctness of synthesized lesions remain significant problems. In addition, the highly nonlinear relationship between different modalities makes it difficult to learn the mapping from one modality to another.

[0005] At present, there is very little research on the generation of FFA images. It was not until 2023 that someone first proposed a UWAT-GAN method, which overcomes the limitations of UW-FFA imaging by using multi-scale discriminators and multi-scale generators. Although this method has constraints on the details of the generated image, there are the following problems: First, for the optic disc part, the generated optic disc area is relatively blurred and cannot highlight its details, and thus cannot provide good assistance to ophthalmologists in the diagnosis of eye diseases; secondly, the vascular information of the generated FFA image is not accurate enough, and it cannot accurately assist in the analysis of the structure and the diagnosis of the disease. In summary, the above-mentioned FFA image generation method has the following two main shortcomings:

[0006] 1. This method does not generate a clear image of the optic disc area and is of limited help in fundus examination.

[0007] 2. This method is not accurate enough in generating vascular information and cannot provide accurate and complete important vascular information for the detection and diagnosis of various ophthalmic diseases. Summary of the invention

[0008] The embodiments of the present disclosure aim to solve at least one of the technical problems existing in the prior art, and disclose a fluorescein angiography image generation model training method, generation method and device.

[0009] In a first aspect, an embodiment of the present disclosure provides a training method for a fluorescein angiography image generation model, wherein the generation model includes a global generation module, a local generation module, and a multi-scale discriminator module; the training method includes:

[0010] Acquire ultra-wide-angle fundus imaging images and corresponding fluorescein angiography images;

[0011] Performing blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image;

[0012] Based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, the global generation module and the local generation module are trained respectively, and the generated image is judged using the multi-scale discriminator module to obtain the trained generation model.

[0013] In some embodiments, the global generation module is trained based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and the generated image is judged by the multi-scale discriminator, including:

[0014] Input the ultra-wide-angle fundus imaging image and the corresponding vascular information image as a whole into the global generation module to generate a corresponding fluorescein angiography image and extract the global information features of the ultra-wide-angle fundus imaging image; splice the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and input them into the multi-scale discriminator module to obtain the first feature representation and the first loss function;

[0015] The ultra-wide-angle fundus imaging image and the generated fluorescein angiography image are spliced ​​and input into the multi-scale discriminator module to obtain the second feature representation and the second loss function thereof.

[0016] In some embodiments, the training of the local generation module based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and the use of the multi-scale discriminator to judge the generated image include:

[0017] Randomly cropping the ultra-wide-angle fundus imaging image, the blood vessel information image, the fluorescein angiography image, and the extracted global information features to generate one-to-one corresponding ultra-wide-angle fundus imaging image blocks, blood vessel information image blocks, fluorescein angiography image blocks, and global information feature blocks;

[0018] splicing the ultra-wide-angle fundus imaging image block with the blood vessel information image block, and inputting the corresponding global information feature block into the local generator module together to generate a corresponding fluorescein angiography image block;

[0019] splicing the ultra-wide-angle fundus imaging image block and the corresponding fluorescein angiography image block and inputting them into the multi-scale discriminator module to obtain a third feature representation and a third loss function thereof;

[0020] The ultra-wide-angle fundus imaging image block and the generated fluorescein angiography image block are spliced ​​and input into the multi-scale discriminator module to obtain the fourth feature representation and the fourth loss function thereof.

[0021] In some embodiments, after obtaining the trained generative model, the method further includes the step of optimizing the trained generative model, specifically including:

[0022] Performing target optic disc detection on the ultra-wide-angle fundus imaging image to obtain the center coordinates of the optic disc, and cropping to obtain an ultra-wide-angle optic disc image block, a vascular information optic disc image block, and a fluorescein optic disc image block, and at the same time cropping the global information features according to the center coordinates to obtain a corresponding optic disc information feature block;

[0023] splicing the ultra-wide-angle optic disc image block with the vascular information optic disc image block, and inputting them together with the optic disc information feature block into the local generator module to generate a corresponding fluorescein optic disc image block;

[0024] splicing the ultra-wide-angle optic disc image block and the corresponding fluorescein optic disc image block and inputting them into the multi-scale discriminator module to obtain the fifth feature representation and the fifth loss function thereof;

[0025] The ultra-wide-angle optic disc image block and the generated fluorescein optic disc image block are spliced ​​and input into the multi-scale discriminator module to obtain the sixth feature representation and the sixth loss function thereof.

[0026] In some embodiments, the loss function of the multi-scale discriminator module adopts a binary cross entropy loss function, the loss functions of the global generation module and the local generation module adopt a mean square error loss function, and GAN loss functions loss_G_F_GAN and loss_G_C_GAN are introduced, corresponding to the fine generator and the rough generator, respectively.

[0027] In a second aspect, an embodiment of the present disclosure provides a method for generating a fluorescein angiography image, the method comprising:

[0028] Acquire actual ultra-wide-angle fundus imaging images;

[0029] The actual ultra-wide-angle fundus imaging image is input into a trained generation model to generate an ultra-wide-angle fluorescein angiography image; wherein the generation model is trained using the training method described above.

[0030] In a third aspect, an embodiment of the present disclosure provides a training device for a fluorescein angiography image generation model, wherein the generation model includes a global generation module, a local generation module, and a multi-scale discriminator module; the training device includes:

[0031] A first acquisition module is used to acquire an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image;

[0032] A segmentation module, used to perform blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image;

[0033] A training module is used to train the global generation module and the local generation module based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and use the multi-scale discriminator module to judge the generated image to obtain the trained generation model.

[0034] In a fourth aspect, an embodiment of the present disclosure provides a fluorescein angiography image generating device, the generating device comprising:

[0035] The second acquisition module is used to acquire the actual ultra-wide-angle fundus imaging image;

[0036] A generation module is used to input the actual ultra-wide-angle fundus imaging image into a trained generation model to generate an ultra-wide-angle fluorescein angiography image; wherein the generation model is trained using the training method described above.

[0037] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0038] one or more processors;

[0039] A storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method described above.

[0040] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described above can be implemented.

[0041] The training method, generation method and device of the fluorescein angiography image generation model of the disclosed embodiment designs a deep learning framework for multi-scale generation of fluorescein angiography guided by fundus structure. The framework consists of three parts: a local generation module responsible for processing local information, a global generation module responsible for processing global information, and a multi-scale discriminator module responsible for judging the generated image. The generation model is trained by using ultra-wide-angle fundus imaging images and corresponding fluorescein angiography images and vascular information images to obtain a trained generation model, so that the generation model can be used to accurately generate fluorescein angiography images corresponding to any ultra-wide-angle fundus imaging image. This embodiment takes into account the importance of vascular information in fundus fluorescein angiography images. In the model design, vascular information is input into the model as a condition, emphasizing the vascular structure, which will prompt the accuracy of vascular information, making the vascular information of the synthesized fluorescein angiography image more accurate, thereby better assisting structural analysis and disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0043] Figure 1 A flowchart of a method for training a fluorescein angiography image generation model according to an embodiment of the present disclosure;

[0044] Figure 2 It is a schematic diagram of the overall framework of the fluorescein angiography image generation model according to an embodiment of the present disclosure;

[0045] Figure 3 is a schematic diagram of a global generator module according to an embodiment of the present disclosure;

[0046] Figure 4 is a schematic diagram of a local generator module according to an embodiment of the present disclosure;

[0047] Figure 5 A schematic diagram of the training process of the fluorescein angiography image generation model according to an embodiment of the present disclosure;

[0048] Figure 6 A flowchart of a method for generating a fluorescein angiography image according to another embodiment of the present disclosure;

[0049] Figure 7 A schematic diagram of the structure of a training device for a fluorescein angiography image generation model according to an embodiment of the present disclosure;

[0050] Figure 8 It is a schematic structural diagram of a fluorescein angiography image generating device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0053] Figure 1 The present invention is a flowchart of a method for training a fluorescein angiography image generation model according to an embodiment of the present invention. Figure 2 Schematic diagram of the overall framework of the fluorescein angiography image generation model according to an embodiment of the present disclosure.

[0054] like Figure 1 and Figure 2As shown, a training method S100 for a fluorescein angiography image generation model includes a global generation module, a local generation module and a multi-scale discriminator module. The training method S100 includes the following steps S110 to S130.

[0055] Step S110 , obtaining an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image.

[0056] Specifically, in this step, a data set is obtained by acquiring a large number of ultra-wide-angle fundus imaging images and corresponding fluorescein angiography images.

[0057] Step S120 , performing blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image.

[0058] Specifically, in this step, in order to introduce vascular information, this embodiment performs vascular segmentation on the ultra-wide-angle fundus imaging image to obtain its corresponding vascular information image. In order to solve the problem of small data volume of ultra-wide-angle fundus imaging images and their corresponding fluorescein angiography images, this embodiment also introduces a random cropping data enhancement method, which randomly crops 40 images of size 832×1088 from each pair of images in the data set, thereby increasing the data set and improving the data volume.

[0059] Step S130, based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, respectively train the global generation module and the local generation module, and use the multi-scale discriminator module to judge the generated image to obtain the trained generation model.

[0060] Specifically, in this step, Figures 3 to 5 . Figure 3 Schematic diagram of a global generator module according to an embodiment of the present disclosure. Figure 4 It is a schematic diagram of a local generator module according to an embodiment of the present disclosure. Figure 5 Schematic diagram of the training process of the fluorescein angiography image generation model according to an embodiment of the present disclosure.

[0061] like Figures 3 to 5As shown in the figure, during the training process of the model, (1) the network takes the processed ultra-wide-angle fundus imaging image and its paired vascular information image and fluorescein angiography image as input. (2) First, the global generator Gen_Coarse is trained, and the ultra-wide-angle fundus imaging image (real) and the vascular information image (real) are input into the global generator as a whole, thereby generating the corresponding fluorescein angiography image (generated) and extracting the global information features of the ultra-wide-angle fundus imaging image. (3) The ultra-wide-angle fundus imaging image (real) and the corresponding fluorescein angiography image (real) are then spliced ​​and input into the multi-scale discriminator to obtain its feature representation d_coarse_real and loss function d_loss1. (4) The ultra-wide-angle fundus imaging image (real) and the generated fluorescein angiography image (generated) are spliced ​​and input into the multi-scale discriminator to obtain its feature representation d_coarse_fake and loss function d_loss2. (5) Then train the local generator Gen_Fine, by randomly cutting an image block of size 256 from the ultra-wide-angle fundus imaging image, fluorescein angiography image, and global information features extracted by Gen_Coarse, and splicing the ultra-wide-angle fundus imaging image block (real) with the above-mentioned global information feature block and inputting it into the local generator, thereby generating the corresponding fluorescein angiography image block (generated). (6) Splice the ultra-wide-angle fundus imaging image block (real) with the corresponding fluorescein angiography image block (real) and input it into the multi-scale discriminator to obtain its feature representation d_fine_real and loss function d_loss3. (7) Splice the ultra-wide-angle fundus imaging image block (real) with the generated fluorescein angiography image block (generated) and input it into the multi-scale discriminator to obtain its feature representation d_fine_fake and loss function d_loss4.

[0062] like Figure 2 As shown, the overall framework of the fluorescein angiography image generation model of the disclosed embodiment: the fluorescein angiography image has both global features such as eye structure and long blood vessels, as well as local features such as small lesions and capillaries. However, it is challenging to use a single generator to simultaneously generate images with global and local features. In order to solve this problem, two different levels of generators are designed in the disclosed embodiment. GenC is used to extract global information and generate global results. GenF extracts local information and generates local results. The global and local complement each other and serve as references to each other. For the discriminator, a multi-scale discriminator is used to capture the features of the image at different scales to improve the effect of adversarial training, thereby improving the quality of the generated image.

[0063] like Figure 3As shown, the global generative adversarial network of the embodiment of the present disclosure is: C-GAN is used for the global generative adversarial network. By introducing the global vascular information of vascular segmentation as a condition, combined with the complete image, it is input into the global generator to improve the acquisition of global information.

[0064] like Figure 4 As shown, the local generative adversarial network of the embodiment of the present disclosure: For the local adversarial network, Patch-GAN is used. At the same time, for the construction method of its data stream, the local generative adversarial network has two streams. The first data stream is used to process random image blocks, and the image blocks with a size of 256×256 are randomly cut from the complete image and input into the local generator GenC. The other data stream is used to process the image blocks around the optic disc, and the image block size is adjusted to 256 before being input into the local generator GenC.

[0065] In some embodiments, after obtaining the trained generative model, the method further includes the step of optimizing the trained generative model, specifically including:

[0066] The target optic disc is detected on the ultra-wide-angle fundus imaging image to obtain the center coordinates of the optic disc, and the ultra-wide-angle optic disc image block, the vascular information optic disc image block and the fluorescein optic disc image block are cut and the global information features are cut according to the center coordinates to obtain the corresponding optic disc information feature block. The ultra-wide-angle optic disc image block is spliced ​​with the vascular information optic disc image block, and input together with the optic disc information feature block into the local generator module to generate the corresponding fluorescein optic disc image block. The ultra-wide-angle optic disc image block and the corresponding fluorescein optic disc image block are spliced ​​and input into the multi-scale discriminator module to obtain the fifth feature representation and the fifth loss function. The ultra-wide-angle optic disc image block and the generated fluorescein optic disc image block are spliced ​​and input into the multi-scale discriminator module to obtain the sixth feature representation and the sixth loss function.

[0067] Specifically, in this step, Figures 3 to 5 , the target detection algorithm YOLOv8 is used to detect the optic disc of the input ultra-wide-angle fundus imaging image, and obtain the BoundingBox of the optic disc. The center coordinates of the optic disc are calculated through the upper left corner coordinates and the lower right corner icon of the BoundingBox, and the corresponding image block is cropped. The training process is the same as above. The feature representations are d_optic_real and d_optic_fake, and the loss functions are d_loss5 and d_loss6 respectively.

[0068] In some embodiments, for the loss function in the training process, the loss function d_loss of the discriminator uses a binary cross entropy loss function (Binary Cross Entropy Loss) to measure the classification performance of the discriminator for real images and generated images. For the loss function g_loss of the generator, the mean squared error loss function (MSE) is used to measure the pixel-level difference between the generated image and the real image. At the same time, the GAN losses loss_G_F_GAN and loss_G_C_GAN of the generator are introduced, corresponding to the fine generator and the rough generator, respectively. They measure the extent to which the output of the generator is misclassified on the discriminator, that is, the extent to which the generator attempts to deceive the discriminator. These losses usually use a binary cross entropy loss function. Feature matching losses loss_G_F_GAN_Feat and loss_G_C_GAN_Feat are introduced, which encourage the generator to generate more realistic images by comparing the feature representations of the discriminator for real and generated images. Introduce VGG loss loss_G_F_VGG and loss_G_C_VGG. Use the pre-trained VGG19 network to extract features, initialize 5 modules, namely slice1, slice2, slice3, slice4, slice5 to store the extracted five layers of features, and define a weight list [1.0 / 32, 1.0 / 16, 1.0 / 8, 1.0 / 4, 1.0] to represent the weight coefficient of each layer. Traverse different feature layers, calculate the mean squared error (MSE) loss of the current feature layer, and then multiply it by the predefined weight coefficient to get the final VGG loss. It is used to measure the content similarity between generated images and real images. They encourage generated images to match real images at specific levels, which helps to generate more content images. These losses help improve the quality of generated images.

[0069] The training method of the fluorescein angiography image generation model of the disclosed embodiment adopts the Patch-GAN architecture for the local generative adversarial network, and proposes two data streams, which are used to process the random part patch and the designated part patch respectively. For the former, the random patch is obtained; for the latter, the designated part is located by using the target detection algorithm, and then the patch is input. In addition to the optic disc area highlighted above, the designated part can also be the lesion area. For the global generative adversarial network, the C-GAN architecture is adopted to process the segmented global vascular information in combination with the original image. The supplemented vascular structure information can well constrain the generation of vascular texture and topology in the image.

[0070] The training method of the fluorescein angiography image generation model of the disclosed embodiment designs a deep learning framework for multi-scale generation of fluorescein angiography guided by fundus structure. The framework consists of three parts: a local generation module responsible for processing local information, a global generation module responsible for processing global information, and a multi-scale discriminator module responsible for judging the generated image. The trained generation model is obtained by training the generation model using ultra-wide-angle fundus imaging images and corresponding fluorescein angiography images and vascular information images, so that the generation model can be used to accurately generate fluorescein angiography images corresponding to any ultra-wide-angle fundus imaging image. This embodiment takes into account the importance of vascular information in fundus fluorescein angiography images. In the model design, vascular information is input into the model as a condition, emphasizing the vascular structure, which will prompt the accuracy of vascular information, making the vascular information of the synthesized fluorescein angiography image more accurate, thereby better assisting structural analysis and disease diagnosis.

[0071] In addition, the training method of the embodiment of the present disclosure includes optimizing the above-mentioned trained generation model. Starting from the local, this embodiment introduces a separate data stream to process the image blocks of the optic disc, so that the optic disc area information in the generated FFA image is clearer. This embodiment will improve the generation information of the optic disc area, making the optic disc area clearer, better assisting ophthalmologists to complete fundus examinations, and providing assistance for the diagnosis of eye diseases.

[0072] Based on the same inventive concept, another embodiment of the present disclosure provides a method for generating a fluorescein angiography image. Figure 6 FIG. 4 is a flow chart of a method for generating a fluorescein angiography image according to another embodiment of the present disclosure. Figure 6 As shown, a method S600 for generating a fluorescein angiography image is provided. The generating method S600 includes step S610 and step S620.

[0073] Step S610: Acquire an actual ultra-wide-angle fundus imaging image.

[0074] Step S620: input the actual ultra-wide-angle fundus imaging image into the trained generation model to generate an ultra-wide-angle fluorescein angiography image; wherein the generation model is trained using the training method described above.

[0075] The fluorescein angiography image generation method of this embodiment uses the generation model trained in the previous text to generate the corresponding actual ultra-wide-angle fluorescein angiography image, so that the vascular information of the synthesized fluorescein angiography image is more accurate, thereby better assisting the analysis of structures and the diagnosis of diseases.

[0076] Based on the same inventive concept, another embodiment of the present disclosure provides a training device for a fluorescein angiography image generation model. Figure 7 FIG. 1 is a schematic diagram of the structure of a training device for a fluorescein angiography image generation model according to an embodiment of the present disclosure. Figure 7 As shown, a training device 700 for a fluorescein angiography image generation model, the generation model includes a global generation module, a local generation module and a multi-scale discriminator module; the training device 700 includes: a first acquisition module 710, a segmentation module 720 and a training module 730. The first acquisition module 710 is used to acquire an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image. The segmentation module 720 is used to perform blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image. The training module 730 is used to train the global generation module and the local generation module based on the ultra-wide-angle fundus imaging image, the corresponding fluorescein angiography image and the blood vessel information image, respectively, and use the multi-scale discriminator module to judge the generated image to obtain the trained generation model.

[0077] The training device of the fluorescein angiography image generation model of the disclosed embodiment takes into account the importance of vascular information in fundus fluorescein angiography images. In the model design, the vascular information is input into the model as a condition, which emphasizes the vascular structure, prompts the accuracy of the vascular information, and makes the vascular information of the synthesized fluorescein angiography image more accurate, thereby better assisting the analysis of the structure and the diagnosis of the disease.

[0078] Based on the same inventive concept, another embodiment of the present disclosure provides a fluorescein angiography image generating device. Figure 8 FIG. 8 is a schematic diagram of the structure of a fluorescein angiography image generating device 800 according to another embodiment of the present disclosure. Figure 8 As shown, the generating device 800 includes: a second acquisition module 810 and a generating module 820. The second acquisition module 810 is used to acquire an actual ultra-wide-angle fundus imaging image. The generating module 820 is used to input the actual ultra-wide-angle fundus imaging image into a trained generating model to generate an actual ultra-wide-angle fluorescein angiography image; wherein the generating model is trained using the training method described above.

[0079] The fluorescein angiography image generation device of this embodiment uses the generation model trained in the previous text to generate the corresponding actual ultra-wide-angle fluorescein angiography image, so that the vascular information of the synthesized fluorescein angiography image is more accurate, thereby better assisting the analysis of structures and the diagnosis of diseases.

[0080] Based on the same inventive concept, another embodiment of the present disclosure provides an electronic device, including:

[0081] one or more processors;

[0082] A storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method described above.

[0083] Based on the same inventive concept, another embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described above can be implemented.

[0084] The computer-readable medium may be included in the apparatus, device, or system of the present invention, or may exist independently.

[0085] Among them, computer-readable storage media can be any tangible media that contains or stores a program, which can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment. More specific examples include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, an optical fiber, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0086] The computer-readable storage medium may also include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code, specific examples of which include but are not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0087] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A training method for a fluorescein angiography image generation model, characterized in that: The generation model includes a global generation module, a local generation module and a multi-scale discriminator module; the training method includes: Acquire ultra-wide-angle fundus imaging images and corresponding fluorescein angiography images; Performing blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image; Based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, the global generation module and the local generation module are trained respectively, and the generated image is judged by the multi-scale discriminator module to obtain the trained generation model; The step of training the global generation module based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and using the multi-scale discriminator to judge the generated image includes: Input the ultra-wide-angle fundus imaging image and the corresponding vascular information image as a whole into the global generation module, generate a corresponding fluorescein angiography image and extract the global information features of the ultra-wide-angle fundus imaging image at the same time; splice the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and input them into the multi-scale discriminator module to obtain the first feature representation and the first loss function; splicing the ultra-wide-angle fundus imaging image and the generated fluorescein angiography image and inputting them into the multi-scale discriminator module to obtain a second feature representation and a second loss function; The training of the local generation module based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and judging the generated image using the multi-scale discriminator, includes: Randomly cropping the ultra-wide-angle fundus imaging image, the blood vessel information image, the fluorescein angiography image, and the extracted global information features to generate one-to-one corresponding ultra-wide-angle fundus imaging image blocks, blood vessel information image blocks, fluorescein angiography image blocks, and global information feature blocks; splicing the ultra-wide-angle fundus imaging image block with the blood vessel information image block, and inputting the corresponding global information feature block into the local generation module to generate a corresponding fluorescein angiography image block; splicing the ultra-wide-angle fundus imaging image block and the corresponding fluorescein angiography image block and inputting them into the multi-scale discriminator module to obtain a third feature representation and a third loss function thereof; The ultra-wide-angle fundus imaging image block and the generated fluorescein angiography image block are spliced ​​and input into the multi-scale discriminator module to obtain the fourth feature representation and the fourth loss function thereof.

2. The training method according to claim 1, characterized in that: After obtaining the trained generative model, the method further includes a step of optimizing the trained generative model, specifically including: Performing target optic disc detection on the ultra-wide-angle fundus imaging image to obtain the center coordinates of the optic disc, and cropping to obtain an ultra-wide-angle optic disc image block, a vascular information optic disc image block, and a fluorescein optic disc image block, and at the same time cropping the global information features according to the center coordinates to obtain a corresponding optic disc information feature block; splicing the ultra-wide-angle optic disc image block and the vascular information optic disc image block, and inputting them together with the optic disc information feature block into the local generation module to generate a corresponding fluorescein optic disc image block; splicing the ultra-wide-angle optic disc image block and the corresponding fluorescein optic disc image block and inputting them into the multi-scale discriminator module to obtain the fifth feature representation and the fifth loss function thereof; The ultra-wide-angle optic disc image block and the generated fluorescein optic disc image block are spliced ​​and input into the multi-scale discriminator module to obtain its sixth feature representation and sixth loss function.

3. The training method according to claim 2, characterized in that: The loss function of the multi-scale discriminator module adopts a binary cross entropy loss function, and the loss functions of the global generation module and the local generation module adopt a mean square error loss function. At the same time, GAN loss functions corresponding to the fine generator and the rough generator are introduced respectively.

4. A method for generating a fluorescein angiography image, characterized in that: The generation method comprises: Acquire actual ultra-wide-angle fundus imaging images; The actual ultra-wide-angle fundus imaging image is input into a trained generation model to generate an ultra-wide-angle fluorescein angiography image; wherein the generation model is trained using the training method described in any one of claims 1 to 3.

5. A training device for a fluorescein angiography image generation model, characterized in that: The generation model includes a global generation module, a local generation module and a multi-scale discriminator module; the training device includes: A first acquisition module is used to acquire an ultra-wide-angle fundus imaging image and a corresponding fluorescein angiography image; A segmentation module, used to perform blood vessel segmentation on the ultra-wide-angle fundus imaging image to obtain a corresponding blood vessel information image; A training module, used to train the global generation module and the local generation module respectively based on the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and the vascular information image, and use the multi-scale discriminator module to judge the generated image to obtain the trained generation model; The training module is further specifically used for: Input the ultra-wide-angle fundus imaging image and the corresponding vascular information image as a whole into the global generation module, generate a corresponding fluorescein angiography image and extract the global information features of the ultra-wide-angle fundus imaging image at the same time; splice the ultra-wide-angle fundus imaging image and the corresponding fluorescein angiography image and input them into the multi-scale discriminator module to obtain the first feature representation and the first loss function; splicing the ultra-wide-angle fundus imaging image and the generated fluorescein angiography image and inputting them into the multi-scale discriminator module to obtain a second feature representation and a second loss function; Randomly cropping the ultra-wide-angle fundus imaging image, the blood vessel information image, the fluorescein angiography image, and the extracted global information features to generate one-to-one corresponding ultra-wide-angle fundus imaging image blocks, blood vessel information image blocks, fluorescein angiography image blocks, and global information feature blocks; splicing the ultra-wide-angle fundus imaging image block with the blood vessel information image block, and inputting the corresponding global information feature block into the local generation module to generate a corresponding fluorescein angiography image block; splicing the ultra-wide-angle fundus imaging image block and the corresponding fluorescein angiography image block and inputting them into the multi-scale discriminator module to obtain a third feature representation and a third loss function thereof; The ultra-wide-angle fundus imaging image block and the generated fluorescein angiography image block are spliced ​​and input into the multi-scale discriminator module to obtain the fourth feature representation and the fourth loss function thereof.

6. A fluorescein angiography image generating device, characterized in that: The generating device comprises: The second acquisition module is used to acquire the actual ultra-wide-angle fundus imaging image; A generation module is used to input the actual ultra-wide-angle fundus imaging image into a trained generation model to generate an ultra-wide-angle fluorescein angiography image; wherein the generation model is trained using the training method described in any one of claims 1 to 3.

7. An electronic device, characterized in that: include: one or more processors; A storage unit, used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 can be implemented.

Images