A method for generating fluorescence angiography images based on multi-scale conditional generative adversarial networks

Abstract

This invention discloses a method for generating fluorescence angiography images based on a multi-scale conditional generative adversarial network (GAN), comprising the following steps: S1, acquiring ultra-wide-angle fundus color images and matching ultra-wide-angle fluorescence angiography images of the same patient; including and excluding the acquired paired images, and performing sharpening, automatic registration, and random cropping operations; S2, building a fully supervised conditional generative adversarial network (GAN) model; S3, training the model; S4, inputting the preprocessed ultra-wide-angle fundus color images into the trained fully supervised GAN model, ultimately generating ultra-wide-angle fluorescence angiography images with a distribution highly similar to the real images. This method automatically generates high-resolution, high-definition ultra-wide-angle fluorescence angiography images that clearly show the areas of microvascular lesions based on the input ultra-wide-angle fundus color images.

Description

Technical Field

[0001] This invention relates to the field of cross-modal image conversion technology, specifically to a method for generating fluorescence angiography images based on multi-scale conditional generative adversarial networks. Background Technology

[0002] Fluorescein angiography is a common imaging method used to detect and diagnose fundus diseases. It is widely used for vascular structure imaging and dynamic observation of contrast agent circulation and leakage within blood vessels. However, fluorescein angiography requires injecting a fluorescent dye into the patient's anterior vein, which then circulates through the bloodstream to the fundus vessels. Some patients may experience adverse reactions such as vomiting or nausea during or after the procedure. Furthermore, this invasive procedure is not suitable for patients with severe cardiovascular or other systemic diseases.

[0003] Color fundus photography is the most commonly used method for retinal examination. It utilizes a confocal laser scanning ophthalmoscope and has advantages such as being non-invasive, convenient, fast, having no adverse reactions, and having low requirements for the patient's overall health. However, many dynamic structural lesions of the fundus, such as the appearance of neovascularization, the formation of macular edema, and non-perfusion areas of microvessels, cannot be shown in color fundus photography, thus affecting the accuracy of diagnosis and prediction of fundus diseases.

[0004] In recent years, the emergence of ultra-wide-angle technology has greatly improved the width and depth of image field of view. Ultra-wide-angle fluorescein angiography images and ultra-wide-angle fundus color images have also been gradually applied to the field of fundus imaging. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for generating fluorescence angiography images based on multi-scale conditional generative adversarial networks. This method automatically generates high-resolution, high-definition, and clearly displaying ultra-wide-angle fluorescence angiography images of microvascular lesions based on input ultra-wide-angle fundus color images.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A method for generating fluorescence angiography images based on multi-scale conditional generative adversarial networks includes the following steps:

[0008] S1. Acquire image data and preprocess it.

[0009] S1-1. Acquire ultra-wide-angle fundus color images and matching ultra-wide-angle fluorescence angiography images of the same patient;

[0010] S1-2. The acquired paired images are included or excluded, and sharpened, automatically registered, and randomly cropped.

[0011] S2. Construct a fully supervised conditional generative adversarial network (GAN) model. This model comprises two parallel GANs: GAN 1 and GAN 2. GAN 1 is composed of a fine-grained generator, a fine-grained discriminator, and an attention concatenation module. GAN 2 is composed of a coarse-grained generator, a coarse-grained discriminator, and an attention concatenation module. The fine-grained generator primarily captures global image features, while the coarse-grained generator primarily captures local image features. The fine-grained discriminator distinguishes between the image generated by the fine-grained generator and the real image, while the coarse-grained discriminator distinguishes between the image generated by the coarse-grained generator and the real image. Intuitively, the fine-grained generator processes images with more pixels and requires more computation, while the coarse-grained generator does the opposite.

[0012] S3, Model Training

[0013] S3-1. The resolution of the preprocessed image in step S1 is reduced to 1 / 3 of the original resolution, thereby obtaining a paired ultra-wide-angle fundus color image and an ultra-wide-angle fluorescence angiography image with lower resolution.

[0014] S3-2. Input the paired and low-resolution ultra-wide-angle fundus color images and ultra-wide-angle fluorescence angiography images into the coarse generator and coarse discriminator to obtain coarse-grained images;

[0015] S3-3. Input the image randomly cropped in step S1 into the fine generator and fine discriminator to obtain the true value image;

[0016] S3-4. Using the input ultra-wide-angle fluorescence angiography image as the discrimination standard, the fully supervised conditional generative adversarial network model is repeatedly trained to optimize the network parameters. Dynamic game theory is used to continuously iterate and optimize until the two reach a balance, so as to minimize the difference between the real value image and the coarse-grained image.

[0017] S4. Input the preprocessed ultra-wide-angle fundus color image into the fully supervised conditional generative adversarial network model that has been trained, and finally generate an ultra-wide-angle fluorescence angiography image that is extremely similar to the distribution of the real image.

[0018] Preferably, the specific method of step S1-1 is as follows:

[0019] S1-1-1, ultra-wide-angle fundus color photography and ultra-wide-angle fluorescein angiography images with an interval of more than 3 months and obvious fresh bleeding;

[0020] S1-1-2, Retinal laser photocoagulation is performed between the two image capture intervals;

[0021] S1-1-3, Screening out images of moderate to severe cataracts and moderate to severe vitreous opacities;

[0022] S1-1-4. Remove images where eyelashes obscure more than a quarter of the image or where the lens is out of focus.

[0023] S1-1-5. Based on historical experience, filter out other images that may interfere with the test results.

[0024] Preferably, in step S1-2, the automatically registered images are uniformly adjusted to 3432×3702.

[0025] Preferably, in step S1-2, the image with uniformly adjusted size is randomly cropped into different small blocks of 608×768.

[0026] Preferably, the loss function of the fully supervised conditional generative adversarial network model is defined as follows:

[0027]

[0028]

[0029] in

[0030] L cGAN =E (c,x) [log(D(c,x))]+E c [log(1-D(G(c),c)] (2)

[0031]

[0032] G in formula (1) C and G F D represents the coarse generator and the fine generator, respectively. C1 and D C2 D represents a coarse discriminator. F Describing a fine discriminator, λ FMC ,λ VGGC ,λ FMF ,λ VGGF Both represent adjustable weight parameters. In formula (2), c in the variable pair (c,x) represents the original image distribution as a condition, while x represents the distribution of the actual ultra-wide-angle fluorescence angiography vascular image. In formula (3), D (i) This represents the i-th layer of the network for feature extraction, where T is the total number of layers in the network, and N is the number of layers in the network. i This represents the number of elements in each layer. In formula (4), N is the total number of layers in the network, and M represents the number of elements in each layer. i V represents the number of elements in each level. i This is the i-th layer of the VGG19 network architecture.

[0033] This invention has the following characteristics and beneficial effects:

[0034] By employing the above technical solution, this invention truly fulfills the medical need for automatically generating ultra-wide-angle fluorescence angiography images based on ultra-wide-angle fundus color photographs. Specifically, it includes: the ability to perform cross-modal conversion and generation of ultra-wide-angle fundus images; the generated ultra-wide-angle fluorescence angiography images exhibit a high degree of similarity to real images, to the point that some clinicians cannot distinguish between them, indicating that the model has a good image generation effect; the generated ultra-wide-angle fluorescence angiography images can clearly display microvascular lesions and some lesion areas, such as neovascularization areas and microvascular non-perfusion areas in the retina; the ultra-wide-angle fluorescence angiography images generated from ultra-wide-angle fundus color photographs can reduce the adverse reactions and invasive procedures of fluorescence angiography, and can reveal non-perfusion areas and neovascularization areas, thereby guiding laser, drug injection, and other treatments, effectively providing assistance and intervention for the diagnosis and treatment of fundus diseases. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the process of the present invention;

[0037] Figure 2 This is the overall architecture of the fully supervised multi-scale conditional generative adversarial network model in this embodiment of the invention;

[0038] Figure 3 (a), (b), and (c) are respectively ultra-wide-angle fundus color photographs under normal retina, and schematic diagrams comparing real ultra-wide-angle fluorescence angiography images with fluorescence angiography images generated by the model;

[0039] Figure 4 (a), (b), and (c) are ultra-wide-angle fundus color images containing microvascular non-perfusion areas, respectively, and schematic diagrams comparing real ultra-wide-angle fluorescence angiography images with fluorescence angiography images generated by the model.

[0040] Figure 5 (a), (b), and (c) are ultra-wide-angle fundus color photographs containing cotton wool spots, and schematic diagrams comparing real ultra-wide-angle fluorescence angiography images with fluorescence angiography images generated by the model.

[0041] Figure 6 (a), (b), and (c) are respectively ultra-wide-angle fundus color images containing laser spots, and schematic diagrams comparing real ultra-wide-angle fluorescence angiography images with fluorescence angiography images generated by the model. Detailed Implementation

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0043] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0045] This invention provides a method for generating fluorescence angiography images based on multi-scale conditional generative adversarial networks, such as... Figure 1 As shown, the specific operation is as follows:

[0046] Step 1: Acquire ultra-wide-angle fundus color images and matching ultra-wide-angle fluorescein angiography images of the same patient, both 3900×3072 pixels. Exclusion criteria for ultra-wide-angle fundus images are as follows:

[0047] (1) The interval between ultra-wide-angle fundus color photography and ultra-wide-angle fluorescein angiography images is >3 months or there is obvious fresh bleeding, etc.

[0048] (2) Retinal laser photocoagulation, etc., between the two image captures;

[0049] (3) There are many factors that interfere with image quality, such as moderate or higher cataracts and moderate or higher vitreous opacity, which affect clinical diagnosis.

[0050] (4) Images that affect clinical diagnosis, such as those with eyelashes covering more than a quarter of the image or those with inaccurate lens focus.

[0051] (5) Other images that researchers judged may interfere with the test results.

[0052] The acquired paired images were included or excluded, and preprocessed with sharpening, automatic registration, and random cropping. Local adaptive histogram equalization, a common sharpening technique, was used to enhance image edges and areas of abrupt grayscale changes. Automatic registration software was used to pair the images to ensure spatial consistency. The registered images were then uniformly adjusted to 3432×3702 pixels and randomly cropped into smaller blocks of 608×768 pixels.

[0053] Step 2: As Figure 2 As shown, a fully supervised conditional generative adversarial network model is constructed that integrates a multi-scale discriminator, a generator, and an attention splicing module.

[0054] Specifically, the fully supervised conditional generative adversarial network (GAN) model comprises two parallel GANs: GAN 1 and GAN 2. GAN 1 is composed of a fine-grained generator, a fine-grained discriminator, and an attention concatenation module, while GAN 2 is composed of a coarse-grained generator, a coarse-grained discriminator, and an attention concatenation module. The fine-grained generator primarily captures global image features, while the coarse-grained generator primarily captures local image features. The fine-grained discriminator distinguishes between the image generated by the fine-grained generator and the real image, while the coarse-grained discriminator distinguishes between the image generated by the coarse-grained generator and the real image. Intuitively, the fine-grained generator takes in and processes images with more pixels, requiring more computation, while the coarse-grained generator does the opposite.

[0055] Step 3: During model training, to obtain coarse-grained images, the image resolution was reduced to one-third of its original value. Therefore, the inputs to the coarse generator and coarse discriminator in the model are filtered and paired ultra-wide-angle fundus color images and ultra-wide-angle fluorescence angiography images with lower resolution. The inputs to the fine generator and fine discriminator are different small patches of 608×768 pixels randomly cropped in Step 1. Using the input ultra-wide-angle fluorescence angiography image as the discrimination criterion, the following model loss function is defined:

[0056]

[0057] in

[0058] L cGAN =E (c,x) [log(D(c,x))]+E c [log(1-D(G(c),c)] (2)

[0059]

[0060] G in formula (1) C and G F D represents the coarse generator and the fine generator, respectively. C1 and D C2 D represents a coarse discriminator. F Describing a fine discriminator, λ FMC ,λ VGGC ,λ FMF ,λ VGGF Both represent adjustable weight parameters. In formula (2), c in the variable pair (c,x) represents the original image distribution as a condition, while x represents the distribution of the actual ultra-wide-angle fluorescence angiography vascular image. In formula (3), D (i) This represents the i-th layer of the network for feature extraction, where T is the total number of layers in the network, and N is the number of layers in the network. i This represents the number of elements in each layer. In formula (4), N is the total number of layers in the network, and M represents the number of elements in each layer. i V represents the number of elements in each level. i Let D be the i-th layer of the VGG19 network architecture. Given a conditional distribution c, our goal is to maximize D. C1 and D C2 and D F The loss, while minimizing G C and G F The loss is minimized by repeatedly training the model and optimizing the network parameters. This improves both the image generation capability of the generator and the discrimination capability of the discriminator. Dynamic game theory is used to iteratively optimize the model until the two reach a balance, thereby minimizing the difference between real and model-generated ultra-wide-angle fluorescence angiography images.

[0061] Step 4: In the model inference stage, an ultra-wide-angle fundus color image is input. After preprocessing methods such as image sharpening and random cropping, the image is then processed by a pre-trained fully supervised conditional generative adversarial network model to generate an ultra-wide-angle fluorescence angiography image that is extremely similar to the distribution of the real image, achieving the goal of cross-modal image conversion and generation. The accompanying diagram in the manual... Figures 3 to 6 The invention presents ultra-wide-angle fundus color images containing different fundus diseases, and a comparative schematic diagram of real ultra-wide-angle fluorescence angiography images and model-generated fluorescence angiography images. We can see that the ultra-wide-angle fluorescence angiography images generated by the model are extremely similar to the real images, to the point that it is impossible to determine whether the images are generated by the model or taken in real photographs. At the same time, the ultra-wide-angle fluorescence angiography images generated by the model clearly and accurately synthesize the relevant lesion areas and also capture tiny vascular lesions. This demonstrates that the present invention can effectively provide assistance and intervention for the diagnosis and treatment of fundus diseases.

[0062] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for generating a fluorescent angiography image based on a multi-scale conditional generative adversarial network, characterized in that, It comprises the following steps: S1, collecting image data and preprocessing S1-1, collecting ultra-wide-angle fundus color photographs and matched ultra-wide-angle fluorescence angiography images of the same patient; S1-2, inclusion and exclusion of the collected paired images, and sharpening, automatic registration and random cropping operations; S2, building a fully supervised conditional generative adversarial network model, which comprises two parallel generative adversarial networks, namely generative adversarial network one and generative adversarial network two, the generative adversarial network one is fused by a fine generator, a fine discriminator and an attention splicing module, the generative adversarial network two is fused by a coarse generator, a coarse discriminator and an attention splicing module, wherein the fine generator is mainly used to capture the global features of the image, and the coarse generator is mainly used to capture the local features of the image; the fine discriminator is used to distinguish the image generated by the fine generator from the real image, and the coarse discriminator is used to distinguish the image generated by the coarse generator from the real image; S3, model training S3-1, the resolution of the preprocessed image in step S1 is changed to 1 / 3 of the original, thereby obtaining paired and low-resolution ultra-wide-angle fundus color photographs and ultra-wide-angle fluorescence angiography images; S3-2, inputting the paired and low-resolution ultra-wide-angle fundus color photographs and ultra-wide-angle fluorescence angiography images into the coarse generator and coarse discriminator to obtain coarse-grained images; S3-3, inputting the randomly cropped images in step S1 into the fine generator and fine discriminator to obtain real value images; S3-4, repeatedly training the fully supervised conditional generative adversarial network model with the input ultra-wide-angle fluorescence angiography image as the discrimination standard, optimizing the network parameters, and using dynamic game means to continuously iterate and optimize until the two reach balance, so as to minimize the difference between the real value image and the coarse-grained image; S4, inputting the preprocessed ultra-wide-angle fundus color photographs into the fully supervised conditional generative adversarial network model completed the training, and finally generating ultra-wide-angle fluorescence angiography images similar to the real image distribution.

2. The multi-scale condition generative adversarial network-based fluorescent angiography image generation method of claim 1, wherein, The specific method of step S1-1 is as follows: S1-1-1, the interval between the ultra-wide-angle fundus color photograph and the ultra-wide-angle fluorescence angiography image is more than 3 months, and there is obvious fresh hemorrhage; S1-1-2, the interval between the two images is row retinal laser photocoagulation; S1-1-3, excluding images with moderate and above cataract and vitreous turbidity above moderate; S1-1-4, excluding images with more than one fourth of the eyelashes blocked and the lens focus not accurate; S1-1-5, excluding images that interfere with the test results according to historical experience. 3.The method of claim 1, wherein, In step S1-2, the images after automatic registration are uniformly adjusted to 3432x3702.

4. The multi-scale condition generative adversarial network-based fluorescent angiography image generation method according to claim 3, characterized in that, In step S1-2, the images with uniform size are randomly cropped into different small blocks with a size of 608x768.

5. The multi-scale conditional generative adversarial network-based fluorescent angiography image generation method according to any one of claims 1-4, characterized in that, The loss function of the fully supervised conditional generative adversarial network model is defined as follows: wherein L cGAN = E (c,x) [log(D(c,x))] + E c [log(1-D(G(c),c))] (2) G C and G F represent the coarse generator and the fine generator, respectively, and D C1 and D C2 represent the coarse discriminator, D F represent the fine discriminator, λ FMC , λ VGGC , λ FMF , λ VGGF are all adjustable weight parameters, the variable c of the pair (c, x) in equation (2) represents the original image distribution as a condition, and x represents the distribution of the real ultra-widefield fluorescence angiography image, and represents the i-th layer network for extracting features, T is the total number of layers of the network, N i represents the number of elements of each layer, N in equation (4) is the total number of layers of the network, M i is the number of elements of each layer, and V i is the i-th layer of the VGG19 network architecture.

Images