Label transfer classification method for fundus images and angiography images based on deep learning

Abstract

The present invention discloses a label migration classification method for fundus images and angiography images based on deep learning, including data preprocessing, label migration, classification network training and feature visualization. Specifically, the obtained fundus image data and fluorescein fundus angiography image data are preprocessed to highlight the lesion area; the label features of the preprocessed fluorescein fundus angiography image are transferred to the label features of the fundus image; the fundus image after label migration is input into the neural network BathNet for classification training; the classified image data is feature visualized to display the lesion area of ​​the fundus image. Using the method of the present invention, only one fundus image is needed to directly locate the lesion area and determine the possible disease characteristics, thereby improving the diagnostic efficiency and effectively reducing the cost of fundus disease diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for label migration and classification of fundus images and angiography images based on deep learning. Background Art

[0002] The fundus refers to the tissues at the back of the eyeball, namely the inner membrane of the eyeball—the retina, optic disc, macula, and central retinal artery and vein. Many diseases can be detected from the fundus. The retinal vessels in the fundus are the most visible blood vessels in the human body, and doctors use them as a window into the vascular conditions of other organs. Therefore, changes in them, to a certain extent, reflect the extent of changes in certain organs. For example, retinal arteriosclerosis can be seen in the fundus of patients with hypertension, while capillary hemangiomas, small hemorrhages, and exudates can be seen in the fundus of patients with diabetes, reflecting, to a certain extent, systemic vascular changes. Doctors can usually analyze and assess the severity of a disease based on fundus images, also known as CFP.

[0003] Fundus fluorescein angiography (FFA) is an ocular vascular examination method that uses a contrast agent injected into the elbow vein to image the fundus vessels and their perfusion status using a specific filter and fundus camera. Fluorescein angiography can be used to examine patients and observe changes in fundus microvasculature and blood flow dynamics.

[0004] Diabetic retinopathy (DR) is a vascular disease that affects the normal blood vessels in the eye and is the leading cause of preventable blindness worldwide. Patients with eye disease often do not realize that the condition is worsening without symptoms, making early screening and treatment of eye disease particularly important.

[0005] However, the diagnosis of eye diseases is often challenging due to the complex structure of the eye and the small size of the lesion area. This places high demands on medical personnel's diagnostic skills and medical equipment, resulting in a shortage of medical resources. Therefore, the use of artificial intelligence (AI) deep learning technology to assist in disease diagnosis is necessary. However, most eye image recognition methods only determine the presence or absence of disease and are unable to grade the severity of the disease.

[0006] However, deep learning neural networks differ from traditional disease diagnosis and classification in their ability to diagnose eye diseases. Fundus images offer lower diagnostic accuracy than fluorescein angiography (FAA) images due to their small size and unclear feature areas. Furthermore, some pathological features cannot be identified using FAA images, necessitating FAA. However, FAA images are significantly more expensive to capture than FAA images, making pathological diagnosis based on FAA images more complex. Summary of the Invention

[0007] To address these issues, the present invention provides a method that maps disease features diagnosed by fluorescein fundus angiography images to fundus image features, thereby training a highly efficient bilinear model classification network, BathNet. This method allows fundus diseases that typically require angiography images to be diagnosed by machine-generated lesions directly from a single fundus image, thereby identifying potential disease features and grading their severity. Because fundus images are significantly cheaper and more convenient than angiography images, they can effectively reduce the cost of fundus disease diagnosis.

[0008] To achieve the above objectives, the present invention adopts a label migration classification method for fundus images and angiographic images based on deep learning. The specific technical solution of the present invention is as follows:

[0009] 1. Data preprocessing stage: The obtained fundus image CFP data and fluorescein fundus angiography image FFA data are processed for redundancy removal, size normalization and feature enhancement to highlight the lesion area.

[0010] 2. Label transfer stage: transfer the label features of the fluorescein fundus angiography image FFA obtained in step 1 to the label features of the fundus image CFP.

[0011] 3. Classification network training stage: The fundus image CFP after label transfer obtained in step 2 is input into the neural network BathNet for classification training.

[0012] 4. Feature visualization stage: The classified image data obtained in step 3 is subjected to feature visualization processing to display the lesion area of ​​the fundus image CFP.

[0013] The beneficial effects of the present invention compared to the prior art are:

[0014] The present invention adopts label transfer technology, which maps the disease characteristics of fluorescein fundus angiography (FFA) images to the characteristics of fundus images (CFP) and inputs them into a training network model. The training network model adopts a bilinear model efficient classification network BathNet, thereby effectively improving the classification accuracy.

[0015] The present invention visualizes the lesion area, allowing medical personnel to locate the lesion area and judge the severity of the disease only through the fundus image CFP. This is more intuitive, reduces diagnostic costs, and has significant clinical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the present invention;

[0017] Figure 2 This is the flowchart of image preprocessing;

[0018] Figure 3 This is the BathNet network structure diagram;

[0019] Figure 4 This is the network structure diagram for feature extraction in the BathNet network structure;

[0020] Figure 5 This is the structural diagram of the convolutional layer in the feature extraction network. DETAILED DESCRIPTION

[0021] In order to illustrate the purpose of the present invention, make its technical solution more intuitive, and highlight its advantages, the present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1 As shown, the present invention includes the following steps:

[0023] Step 1: Data preprocessing stage.

[0024] The fundus image CFP data and fluorescein fundus angiography image FFA data are divided into training samples, verification samples and test samples. When the fundus image CFP and fluorescein fundus angiography image FFA are input into the image preprocessing program, the redundant black edge background in the image is first removed, leaving the target detection area. The image size is then unified to 512*512 pixels. After that, the image is sharpened by smoothing the noise area and eliminating the interference of noise in the image, and the brightness is corrected by HSV. Finally, the contrast of the lesion area is enhanced according to the local contrast enhancement algorithm based on standard deviation. The process is summarized in Figure 2 .

[0025] Step 2: Label migration phase.

[0026] Each fundus image (CFP) and fluorescein angiography (FFA) image carries a label representing the corresponding diabetic retinopathy characteristics. These labels are classified into two types: classification labels and graded labels. Classification labels are characteristic labels for diabetic retinopathy, such as microaneurysms, infiltrations, nonperfused areas, and macular edema. Each classification label is accompanied by a graded label indicating severity, such as 0, 1, or 2. The graded labels for each classification label in the several fluorescein angiography (FFA) images corresponding to each fundus image (CFP) preprocessed in step 1 are then unioned. The resulting fluorescein angiography (FFA) labels are then transferred to the fundus images for training. This allows the CFP images to be trained using the same labels as the fluorescein angiography (FFA) images. Experimental results show that using the fundus images with these transferred labels for training a neural network model yields better model parameters, significantly improving network training and enabling better classification of fundus images (CFP).

[0027] Step 3: Classification network training phase.

[0028] The neural network BathNet combines the new Backbone network LGXNet and the spatial feature attention mechanism, and adopts a bilinear model mechanism. Its specific structure is shown in Figure 3 .

[0029] 1. Feature extraction: First, LGXNet is used as the Backbone network to extract the feature information from the CFP fundus image after the label migration input in step 2. LGXNet can solve the Representational Bottleneck problem that may exist in traditional network design due to the extreme compression of channel dimensions (it means that the middle layer of the network will greatly compress the dimension of the feature, and the size of the feature from input to output is significantly reduced, causing feature loss), so as to better extract features. LGXNet is improved based on MobileNetV2 and the middle convolution layer is redesigned. It is mainly based on the input channel size of the extended convolution layer and the penultimate layer, selecting the appropriate activation function SiLU, and adding multiple extension layers to weaken the influence of Representational Bottleneck. Its specific structure can be seen Figure 4 and Figure 5 After the fundus image CFP is input into LGXNet, a feature image Feature Map is generated, and part of the output is further sent to A-Net to generate Attention Map.

[0030] 2. Spatial feature attention: The spatial feature attention mechanism is introduced into the A-Net network structure, which combines spatial attention and channel attention in parallel and serial. In order to better achieve the effect of letting the neural network focus on important features, it is necessary to evaluate the importance of each neuron. In neuroscience, information-rich neurons usually show different discharge patterns from surrounding neurons. Moreover, activated neurons usually inhibit surrounding neurons, that is, spatial inhibition. In other words, neurons with spatial inhibition effects should be given higher importance. The simplest way to find important neurons is to measure the linear separability between neurons. Therefore, the following energy function is defined:

[0031]

[0032] in, t and x i are the target neuron and other neurons in a single channel of the input feature X∈C×H×W. i is the index in the spatial dimension, and M=H×W is the number of neurons in the channel.t with b t are the weight parameters and bias of neuron t respectively. Equal, y0 and When they are equal, the formula gets the minimum value, where y t and y0 are two different values. By minimizing the above formula, the formula is equivalent to training the linear separability between neuron t and other neurons in the same channel. However, in order to simplify the energy function formula, we t and y0 use binary labels “-1” and “1”, and add regularization terms to the above formula. The final energy function is defined as follows:

[0033]

[0034] Theoretically, each channel has M = H × W energy functions. The above formula (2) has the following analytical solution:

[0035]

[0036]

[0037] in, is the mean and variance of all neurons in the channel except neuron t. t and b t The existing solution shown in the formula for is obtained on a single channel, so it is reasonable to assume that all pixels in a single channel follow the same distribution. Based on this assumption, the mean and variance of all neurons can be calculated and reused for all neurons on that channel. This can significantly reduce the computational cost by avoiding the iterative calculation of the mean and variance for each position. Therefore, the minimum energy can be calculated as follows:

[0038]

[0039] in,

[0040] The above formula means that the lower the energy, the greater the difference between neuron t and surrounding neurons, and the higher its importance. Therefore, the importance of a neuron can be measured by get.

[0041] According to the definition of the Attention mechanism, we need to enhance the features:

[0042]

[0043] Where E groups across channels and spatial dimensions. The sigmoid symbol is added to constrain the minimum energy If the value of is too large, it will not affect the relative importance of each neuron because the sigmoid symbol is a unit function.

[0044] After obtaining the Attention Map, we multiply the Feature Map obtained previously with the Attention Map to obtain the image mask Q. We then perform global average pooling (GAP) on the mask Q and the Attention Map to reduce parameters and avoid overfitting. Finally, in order to obtain the weight of the image and filter out irrelevant information, a division is used after the global average pooling (GAP). In summary, the output of the spatial feature attention module is:

[0045]

[0046] Among them A l With F l They are Attention Map and Feature Map, and Represent element-wise multiplication and element-wise division, respectively.

[0047] 3. Bilinear model mechanism: The bilinear network structure used in the neural network BathNet is as follows Figure 3 As shown on the right, the output of the spatial feature attention module and the output of LGXNet are used as input. The output of LGXNet is first placed in a B-Net (consisting of a convolutional layer and a ReLu activation layer) to extract features and make them the same size as the output of the spatial feature attention module. Then, the K operator (element-wise average) is used to calculate:

[0048]

[0049] Where N and Y l They are the output of the spatial feature attention module and the output of B-Net, Z l is the output of the K operator, represents element-wise addition, Represents element-wise division.

[0050] Next, we will use the K operator to output Z l The bilinear vector is obtained by operating B through the bilinear function:

[0051] B=ZZ T (9)

[0052] By symbolic square root operation And L2 normalization (L←Y / ||Y||2) to get the output of BathNet.

[0053] After performing global average pooling (GAP) on the output of the BathNet neural network and then applying a softmax layer, the resulting output is a 1×N vector, where N is the number of diabetic retinopathy-related disease features. Each column in the vector contains a label for the disease feature (e.g., microaneurysm, nonperfused area, permeability, macular edema), and each column contains information about the severity level of the disease feature. The resulting classification and graded labels are the final classification results.

[0054] Step 4: Feature visualization stage.

[0055] After the model converges, the weight parameters stored in the neural network model obtained in step 3 are used to multiply the output of each convolutional layer by the weight of the corresponding classification. The weighted result is then weighted to generate a thermal image of each disease characteristic. Finally, the Class Activation Map is obtained, which allows the focus area to be located and a thermal map of the lesion area to be generated. Based on the resulting classification results, the severity of each disease characteristic can be determined.

[0056] The embodiments of the present invention described above do not limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the claims of the present invention.

Claims

1. A label transfer classification method for fundus images and angiography images based on deep learning, characterized by The method comprises the following steps: Step 1) Data preprocessing: The acquired fundus image CFP data and fluorescein fundus angiography (FFA) data are subjected to redundancy removal, size normalization, and feature enhancement processing to highlight the lesion area; Step 2) label transfer stage: transferring the label features of the fluorescein fundus angiography image FFA obtained in step 1) to the label features of the fundus image CFP; Step 3) Classification network training phase: The fundus image CFP obtained in step 2) after label transfer is input into the neural network BathNet for classification training; Step 4) Feature visualization stage: performing feature visualization processing on the classified image data obtained in step 3) to display the lesion area of ​​the fundus image CFP; In step 2), each fundus image CFP and fluorescein fundus angiography image FFA has a corresponding label of disease characteristics related to diabetic retinopathy, and the labels are divided into classification labels and graded labels; the classification labels are characteristic type labels of diabetic retinopathy, including microaneurysms, infiltrations, non-perfused areas, and macular edema; each classification label has a graded label indicating the severity: grade 0, grade 1, and grade 2; the graded labels under each classification label of several fluorescein fundus angiography images FFA corresponding to each fundus image CFP are unioned, and the obtained fluorescein fundus angiography image FFA labels are transferred to the fundus image CFP for training, so that the fundus image CFP is trained using the labels of the fluorescein fundus angiography images FFA; Step 3) uses the bilinear model efficient classification network BathNet, which combines the new Backbone network LGXNet and the spatial feature attention mechanism; The specific steps include: extracting the feature information of the fundus image CFP after label migration in step 2), inputting it into LGXNet to generate a feature image Feature Map, inputting the Feature Map into A-Net to generate an Attention Map, multiplying the obtained Feature Map with the Attention Map to obtain an image mask, performing global average pooling and element division operations on the mask and the Attention Map respectively, and using them as the output of the spatial feature attention module; putting the output of LGXNet into B-Net and the output of the spatial feature attention module as input to perform element average operation, performing a bilinear function operation B on the output of the element average operation to obtain a bilinear vector, and finally performing a symbolic square root operation and normalization on the bilinear vector to obtain the output of BathNet; performing global average pooling on the output of the neural network BathNet, and then performing a Softmax layer operation to output a 1×N dimensional vector, where N is the number of features of diabetic retinopathy-related diseases, and obtaining the classification result after training; In order to better achieve the effect of letting the neural network focus on important features, the importance of each neuron is evaluated, and neurons with spatial suppression effects are given higher importance. The minimum energy is calculated by the following formula. The lower the energy, the greater the difference between the neuron and the surrounding neurons, and the higher the importance. The importance of the neuron is expressed by get: in, M is the number of neurons on the channel, t and x i are the target neuron and other neurons in a single channel of the input feature X, respectively.

2. The label transfer classification method for fundus images and angiographic images based on deep learning according to claim 1, characterized in that: In step 1), after the fundus image CFP data and the fluorescein fundus angiography image FFA data are input into the image preprocessing program, the redundant black edge background in the image is first removed to leave the target detection area, and then the image size is unified to 512*512 pixels. Then, the contrast of the fundus image CFP data and the fluorescein fundus angiography image FFA lesion area is enhanced according to a local contrast enhancement algorithm based on standard deviation.

3. The label transfer classification method for fundus images and angiographic images based on deep learning according to claim 1, characterized in that: In step 4), the weight parameter information saved by the neural network model obtained in step 3) is weighted by multiplying the output of each convolutional layer by the weight of the corresponding classification of this layer to obtain a thermal imaging map of each disease feature.

Images