A method and system for generating fluorescein angiography images from fundus color images

Abstract

The present invention discloses a method and system for generating a fluorescein angiography image from a fundus color image, belonging to the field of medical image processing, and the method comprises the following steps: obtaining a plurality of pairs of fundus images, wherein a pair of fundus images comprises a fundus color image and a fluorescein angiography image; preprocessing and registering the plurality of pairs of fundus images to obtain a registered image; constructing a generative adversarial network model, training the generative adversarial network model based on the registered image to obtain a trained generative adversarial network model; inputting the fundus color image into the trained generative adversarial network model to obtain a fluorescein angiography composite image. The present invention has the advantages of being non-contact and non-destructive, and can improve the safety of patients by reducing the demand for fluorescein angiography.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method and system for generating fluorescein angiography images from fundus color images. Background Art

[0002] Fundus color imaging and fluorescein angiography are both common and very valuable diagnostic tools in ophthalmology. Fundus color imaging has the advantages of being easily available and non-invasive, but its disadvantage is that it cannot observe microstructural changes in the retina. Fluorescein angiography allows direct imaging of the blood vessels and blood flow in the retina, and provides very useful information about the state of retinal blood vessels, the presence of new blood vessels, delayed venous filling, retinal leakage, and macular edema. However, its adverse reaction rate is as high as 10%, and 1 in every 200,000 cases may lead to death.

[0003] The adoption of non-invasive and feasible imaging technology with high quality and diagnostic capabilities will help improve the diagnosis and treatment of retinal diseases without fluorescein angiography, especially in remote areas with poor medical conditions. With the development of artificial intelligence technology in the field of imaging, it is technically feasible to directly synthesize the corresponding fluorescein angiography image from the fundus color image. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method and system for generating fluorescein angiography images from fundus color images, so as to solve the above problems existing in the prior art.

[0005] To achieve the above object, the present invention provides a method for generating a fluorescein angiography image from a fundus color image, comprising:

[0006] Acquire a plurality of pairs of fundus images, wherein a pair of fundus images includes a fundus color image and a fluorescein angiography image;

[0007] Preprocessing and registering several pairs of fundus images to obtain registered images;

[0008] Constructing a generative adversarial network model, and training the generative adversarial network model based on the registered image to obtain a trained generative adversarial network model;

[0009] The fundus color image is input into the trained generative adversarial network model to obtain a fluorescein angiography composite image.

[0010] Optionally, the process of acquiring a plurality of pairs of fundus images includes:

[0011] With the posterior pole as the center, the fundus color image is acquired at a shooting angle of 45 degrees;

[0012] Fluorescein angiography images at different stages were selected based on disease characteristics.

[0013] Optionally, the process of preprocessing and registering a plurality of pairs of fundus images to obtain registered images includes:

[0014] Each pair of fundus color images and fluorescein angiography images was processed in sequence by histogram equalization, removal of red channel, normalization, and averaging to obtain a preprocessed image pair;

[0015] The pre-processed image pair is registered using the SIFT algorithm to obtain a registered image.

[0016] Optionally, the process of registering the preprocessed image pair includes:

[0017] Constructing a Gaussian pyramid, and using a Harris corner detection algorithm based on the Gaussian pyramid to identify key corner points of the preprocessed image pair;

[0018] generating a multi-scale intensity invariant descriptor based on the key corner points;

[0019] The correspondence between feature points is established based on the multi-scale intensity-invariant descriptors of the preprocessed fundus color image and fluorescein angiography image;

[0020] Based on the feature point correspondence relationship, the matching points are screened using a main direction consistency algorithm and a spatial distribution consistency algorithm to obtain screened matching points;

[0021] The image is converted based on the geometric transformation relationship of the screened matching points, and the fluorescein angiography image is registered with the fundus color image as a reference image to generate a registered image.

[0022] Optionally, the feature point correspondence is established based on a BBF kd tree nearest neighbor query algorithm.

[0023] Optionally, a generative adversarial network model is constructed based on pix2pix, wherein the convolutional structure of the generative adversarial network model is a residual network Resnet.

[0024] The present invention also discloses a system for generating fluorescein angiography images from fundus color images, including an image acquisition module, an image preprocessing module, an image registration module and a deep learning model development module;

[0025] The image acquisition module is used to acquire a plurality of pairs of fundus images to construct an image set; wherein a pair of fundus images includes a fundus color image and a fluorescein angiography image;

[0026] The image preprocessing module is used to preprocess the image set;

[0027] The image registration module is used to register the fluorescein angiography image to the fundus color image;

[0028] The deep learning model development module is used to construct a generative adversarial network model and generate a fluorescein angiography synthetic image based on the generative adversarial network model.

[0029] Optionally, the image registration module includes: a feature extraction unit, a corner point detection unit, a multi-scale description unit, a feature point matching unit and a registration map generation unit;

[0030] Wherein, the feature extraction unit is based on constructing a Gaussian pyramid for multi-scale analysis and feature extraction;

[0031] The corner point detection unit is used to identify key corner points in the image using the Harris corner point detection algorithm and construct a feature point candidate set based on the key corner points;

[0032] The multi-scale description unit is used to represent the local features of the feature points;

[0033] The feature point matching unit is used to establish a corresponding relationship between feature points of the fundus color image and the fluorescein angiography image, and to screen matching points;

[0034] The registration image generating unit is used to register the fluorescein angiography image to the fundus color image based on the screened matching points to generate a registered image.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] The present invention is based on fundus color images and fluorescein angiography images, and adopts SIFT image registration algorithm combined with pix2pix generative adversarial network deep learning algorithm to realize the method technology of automatically synthesizing the corresponding fluorescein angiography image from the fundus color image, which plays an important auxiliary role in helping doctors clinically evaluate retinal diseases. In addition, the present invention also has the advantages of non-contact and non-destructive, and can further improve the safety of patients by reducing the demand for fluorescein angiography. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 Flowchart of a method for generating a fluorescein angiography image from a fundus color image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Embodiment 1

[0042] The present invention obtains the fundus color image and the corresponding fluorescein angiography image of the same eye through an image acquisition module, then performs histogram equalization, red channel removal, normalization, and averaging preprocessing on the image, and then uses the SIFT image registration algorithm combined with the pix2pix generative adversarial network deep learning algorithm to realize the automatic synthesis of the fundus color image into the corresponding fluorescein angiography image.

[0043] like Figure 1 As shown, in this embodiment, a method for generating a fluorescein angiography image from a fundus color image is provided, comprising: obtaining a plurality of pairs of fundus images, wherein a pair of fundus images comprises a fundus color image and a fluorescein angiography image; preprocessing and registering the plurality of pairs of fundus images to obtain registered images; constructing a generative adversarial network model, training the generative adversarial network model based on the registered images to obtain a trained generative adversarial network model; inputting the fundus color image into the trained generative adversarial network model to obtain a fluorescein angiography composite image. The specific implementation steps are as follows:

[0044] S1: Acquire fundus color images and corresponding fluorescein angiography images; acquire fundus color images with the posterior pole as the center and a shooting angle of 45 degrees; select fluorescein angiography images at different stages based on the characteristics of the disease.

[0045] S2: Each pair of fundus color images and fluorescein angiography images is subjected to image preprocessing by histogram equalization, removal of the red channel, normalization, and averaging;

[0046] S3: Use SIFT algorithm to register the images; the process of registering the preprocessed image pair includes: constructing a Gaussian pyramid, and using Harris corner detection algorithm based on the Gaussian pyramid to identify key corner points of the preprocessed image pair; generating multi-scale intensity invariant descriptors based on the key corner points; establishing feature point correspondence based on the multi-scale intensity invariant descriptors of the preprocessed fundus color image and fluorescein angiography image; based on the feature point correspondence, using the main direction consistency algorithm and the spatial distribution consistency algorithm to filter the matching points to obtain the filtered matching points; based on the geometric transformation relationship of the filtered matching points, converting the image to register the fluorescein angiography image to the fundus color image to generate a registered image.

[0047] Considering that a pair of (CF, FA) images are taken at the same time, they may be deformed due to movements such as eye movement, frowning, muscle contraction, etc., and different imaging mechanisms may also cause changes in the appearance of blood vessels or retinal microstructures. Therefore, the present invention uses the SIFT algorithm to perform cross-modal image registration on each pair of (CF, FA) images after preprocessing. The specific steps are as follows:

[0048] 1) Constructing Gaussian pyramid: Using Gaussian pyramid technology on CF and FA images, image pyramids are constructed at different scales for multi-scale analysis and feature extraction. The present invention sets the downsampling unit G_resize of the Gaussian pyramid to 3, the blurring unit G_sigma of the Gaussian pyramid to 1.6, the number of layers per group of the Gaussian pyramid to numLayers to 4, and the number of Gaussian pyramid groups to numOctaves to 3.

[0049] 2) Harris corner detection: Harris corner detection algorithm is used on the Gaussian pyramid constructed in 1) to identify key corners in the image as candidate sets of feature points. The present invention sets the upper limit of the standard deviation of the Harris local weighted Gaussian kernel sigma = 20, the Harris corner response discrimination threshold thresh = 50, the Harris local non-maximum suppression window radius radius = 15, and the feature point number selection threshold N = 1000.

[0050] 3) Generate a multi-scale intensity invariant descriptor (PIIFD): For each Harris corner point detected in step 2), a multi-scale partial intensity invariant descriptor (PIIFD) is generated to represent the local features of the feature point.

[0051] 4) Feature point matching: The BBF (Best Bin First) kd-tree nearest neighbor query algorithm is used to match the feature points of the PIIFD descriptors generated by CF and FA, and establish the correspondence between the feature points of the two images.

[0052] 5) Delete matching points: After the feature points are matched, the main direction consistency algorithm and the spatial distribution consistency algorithm are used to screen the matching points and delete unstable or erroneous matching points to improve the accuracy of the registration results.

[0053] 6) Conversion and registration: According to the geometric transformation relationship between the matching points, the image is transformed, the target image FA to be registered is mapped to the coordinate system of the reference image CF, and the registration map rFA of FA is generated.

[0054] S4: Develop the pix2pix generative adversarial network model; adjust the image resolution to 256*256, based on the Pytorch framework, use the pix2pix generative adversarial network model, and use the residual network Resnet as the convolutional structure. The target loss function of the network is the sum of the mean absolute error (L1) loss and the generative adversarial loss, where the weight factor of the L1 loss is set to 100. The network was trained on the NVIDIA HGX A100 4-GPU for 200 rounds.

[0055] S5: Input the fundus color image, and the model outputs the corresponding fluorescein angiography composite image.

[0056] Embodiment 2

[0057] This embodiment provides a system for generating fluorescein angiography images from fundus color images, the system comprising an image acquisition module, an image preprocessing module, an image registration module, and a deep learning model development module, wherein:

[0058] Image acquisition module: CF and FA images are acquired through fundus cameras such as Zeiss or wide-area fundus cameras such as Oubao, and the CF and corresponding FA images taken by the same person and the same eye at the same time are paired one by one, recorded as (CF, FA). The present invention includes a total of 1,000 pairs of (CF, FA) atlases from different human eyes.

[0059] Image preprocessing module: In order to improve the contrast of the image, all CF and FA images are subjected to histogram equalization, red channel removal, normalization, and averaging.

[0060] Image registration module: For each specific pair of (CF, FA) images, CF is used as the reference image and FA as the image to be registered. The SIFT algorithm is used to register FA to CF. The registered FA image pair is recorded as rFA.

[0061] Deep learning model development module: Based on the pix2pix generative adversarial network, 1000 pairs of (CF, rFA) atlases are input into the network for training, and a deep learning model is constructed that can automatically generate fluorescein angiography images from fundus color images. The synthesized fluorescein angiography images are denoted as sFA.

[0062] The above-mentioned image acquisition module uses a Zeiss non-mydriatic fundus camera (VISUCAM, Carl Zeiss Meditec, Jena, Germany) to capture CF and FA images. All CF images are centered on the posterior pole and the shooting angle is 45 degrees. For FA images, images of different imaging periods can be selected according to the characteristics of the disease, which can better show the scope and severity of retinal diseases.

[0063] The above image preprocessing module, for each CF and FA image, has the following preprocessing process: first, the image contrast is improved by histogram equalization; then the red channel of the image is removed to eliminate the interference of choroidal blood vessels; then the image pixel data is normalized, that is, the pixel value is normalized to the range of [0, 1]; then the color image is converted into a grayscale image, and then the pixel data of the grayscale image is averaged, that is, the pixel value is divided by twice the average pixel value of the entire image to enhance the image contrast.

[0064] The above-mentioned image registration module includes: a feature extraction unit, a corner point detection unit, a multi-scale description unit, a feature point matching unit and a registration map generation unit; wherein the feature extraction unit is based on constructing a Gaussian pyramid for multi-scale analysis and feature extraction; the corner point detection unit is used to identify key corner points in the image using the Harris corner point detection algorithm and construct a feature point candidate set based on the key corner points; the multi-scale description unit is used to represent the local features of the feature points; the feature point matching unit is used to establish a corresponding relationship between the feature points of the fundus color image and the fluorescein angiography image, and to screen the matching points; the registration map generation unit is used to register the fluorescein angiography image to the fundus color image based on the screened matching points to generate a registered image.

[0065] The above model development module builds a deep learning model based on the pix2pix generative adversarial network that can generate FA from CF. In order to adapt to the model input, the resolution of the image is adjusted to 256*256. Based on the Pytorch framework, the pix2pix generative adversarial network model is used, and the residual network Resnet is used as the convolution structure. The target loss function of the network is the sum of the mean absolute error (L1) loss and the generative adversarial loss, where the L1 loss is set to a weight factor of 100. The network was trained on an NVIDIA HGX A100 4-GPU for 200 iterations. When predicting, a fundus color image is input, and the system calls the trained model to generate a fluorescein angiography composite image sFA.

[0066] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating a fluorescein angiography image from a fundus color image, characterized in that: The following steps are involved: Acquire a plurality of pairs of fundus images, wherein a pair of fundus images includes a fundus color image and a fluorescein angiography image; Preprocessing and registering several pairs of fundus images to obtain registered images; Constructing a generative adversarial network model, inputting the registered image into the generative adversarial network model for training to obtain a trained generative adversarial network model; constructing a generative adversarial network model based on pix2pix, wherein the convolutional structure of the generative adversarial network model is a residual network Resnet; the target loss function of the residual network Resnet is the sum of the mean absolute error loss and the generative adversarial loss, wherein the weight factor of the mean absolute error loss is set to 100; Inputting the fundus color image into the trained generative adversarial network model to obtain a synthetic fluorescein angiography image; The process of preprocessing and registering several pairs of fundus images to obtain registered images includes: Each pair of fundus color images and fluorescein angiography images was processed in sequence by histogram equalization, removal of red channel, normalization, and averaging to obtain a preprocessed image pair; Using SIFT algorithm to register the preprocessed image pair to obtain a registered image; The process of registering the preprocessed image pair includes: Construct a Gaussian pyramid, and use the Harris corner detection algorithm based on the Gaussian pyramid to identify the key corner points of the preprocessed image pair; wherein the downsampling unit of the Gaussian pyramid is 3, the blur unit of the Gaussian pyramid is 1.6, the number of layers in each group of the Gaussian pyramid is 4, and the number of Gaussian pyramid groups is 3; the upper limit of the Harris local weighted Gaussian kernel standard deviation is 20, the Harris corner response discrimination threshold is 50, the Harris local non-maximum suppression window radius is 15, and the threshold for selecting the number of feature points is 1000; generating a multi-scale intensity invariant descriptor based on the key corner points; The correspondence between feature points is established based on the multi-scale intensity-invariant descriptors of the preprocessed fundus color image and fluorescein angiography image; Based on the feature point correspondence relationship, the matching points are screened using a main direction consistency algorithm and a spatial distribution consistency algorithm to obtain screened matching points; Convert the image based on the geometric transformation relationship of the screened matching points, and register the fluorescein angiography image with the fundus color image as a reference image to generate a registered image; The corresponding relationship of feature points is established based on the BBF kd tree nearest neighbor query algorithm.

2. The method for generating a fluorescein angiography image from a fundus color image according to claim 1, characterized in that: The process of acquiring several pairs of fundus images includes: With the posterior pole as the center, the fundus color image is acquired at a shooting angle of 45 degrees; Fluorescein angiography images at different stages were selected based on disease characteristics.

3. A system for generating fluorescein angiography images from fundus color images, characterized in that: It includes image acquisition module, image preprocessing module, image registration module and deep learning model development module; The image acquisition module is used to acquire a plurality of pairs of fundus images to construct an image set; wherein a pair of fundus images includes a fundus color image and a fluorescein angiography image; The image preprocessing module is used to preprocess the image set, and the specific process includes: performing histogram equalization, removing the red channel, normalizing, and averaging on each pair of fundus color images and fluorescein angiography images in sequence to obtain a preprocessed image pair; The image registration module is used to register the fluorescein angiography image to the fundus color image, wherein the pre-processed image pair is registered using a SIFT algorithm to obtain a registered image; The deep learning model development module is used to construct a generative adversarial network model, and generate a synthetic image of fluorescein angiography based on the generative adversarial network model; wherein the generative adversarial network model is constructed based on pix2pix, wherein the convolutional structure of the generative adversarial network model is a residual network Resnet; the target loss function of the residual network Resnet is the sum of the mean absolute error loss and the generative adversarial loss, wherein the loss of the mean absolute error loss sets the weight factor to 100; The image registration module includes: a feature extraction unit, a corner point detection unit, a multi-scale description unit, a feature point matching unit and a registration map generation unit; Wherein, the feature extraction unit is based on constructing a Gaussian pyramid for multi-scale analysis and feature extraction; The corner point detection unit is used to identify key corner points in the image by using the Harris corner point detection algorithm and construct a feature point candidate set based on the key corner points; wherein the downsampling unit of the Gaussian pyramid is 3, the blur unit of the Gaussian pyramid is 1.6, the number of layers in each group of the Gaussian pyramid is 4, and the number of Gaussian pyramid groups is 3; the upper limit of the Harris local weighted Gaussian kernel standard deviation is 20, the Harris corner point response discrimination threshold is 50, the Harris local non-maximum suppression window radius is 15, and the threshold for selecting the number of feature points is 1000; The multi-scale description unit is used to represent the local features of the feature points; The feature point matching unit is used to establish a corresponding relationship between feature points of the fundus color image and the fluorescein angiography image, and to screen the matching points using a principal direction consistency algorithm and a spatial distribution consistency algorithm to obtain screened matching points; The registration image generating unit is used to register the fluorescein angiography image to the fundus color image based on the screened matching points to generate a registered image.

Images