A method for diabetic retinopathy lesion image recognition based on attention model

By constructing an attention model-based diabetic retinopathy lesions image recognition method, the problems of difficulty in detecting micro lesions and data imbalance are solved, and high-accuracy automatic screening and lesion prediction of diabetic retinopathy are achieved, which improves the model Generalization ability and interpretability.

CN114287878BActive Publication Date: 2024-11-22JIANGXI UNIVERSITY OF FINANCE AND ECONOMICS
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202111210328.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-18
Publication Date
2024-11-22
Estimated Expiration
2041-10-18

AI Technical Summary

Technical Problem

The prior art faces the problems of difficulty in detecting micro lesions and imbalance in the screening results and data distribution in automatic screening of diabetic retinopathy, resulting in insufficient generalization capabilities of the model and lack of effective automated identification methods.

Method used

Using the diabetic retinopathy lesions image recognition method based on attention model, the lesions-related information is extracted and generated by constructing a semantic segmentation attention network model, including the lesion perception module, feature maintenance module, feature fusion module and head attention module. Lesion detection probability map improves the model's ability to detect small lesions and resists interference factors, guiding the model to learn lesion characteristics from unbalanced data.

Benefits of technology

Effectively detecting diabetic retinopathy lesions in fundus color photos improves the generalization ability of the model, provides explainable evidence for automatic screening, and significantly improves the accuracy and reliability of automatic screening of diabetic retinopathy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114287878B_ABST
    Figure CN114287878B_ABST
Patent Text Reader

Abstract

The present invention provides a method for recognizing diabetic retinopathy lesion images based on an attention model, characterized in that: first, the original fundus images in the data set are preprocessed in a manner suitable for medical images; second, an attention network model based on semantic segmentation is constructed for feature extraction; then, a lesion perception module, a feature retention module, a feature fusion module, and a head attention module are constructed to extract lesion-related information from the proposed features and generate a lesion detection probability map; finally, the lesion detection probability map obtained by the attention model is used to guide eye disease screening based on the input fundus color photo, and a fundus photo recognition result with diabetic retinopathy lesion information is obtained. The present invention can effectively obtain relevant lesion information in fundus photos, generate an interpretable lesion detection probability map while automatically screening diabetic retinopathy, and can better assist ophthalmologists in diagnosis, with broad clinical application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the multidisciplinary cross-technical field of computer vision, digital image processing and ophthalmic clinical medicine, and in particular to a method for recognizing diabetic retinopathy lesion images based on an attention model. Background Art

[0002] Diabetic retinopathy is a retinal complication caused by diabetes. If not treated in time, there is a risk of irreversible vision loss or blindness. Generally speaking, the early stage of the disease does not cause visual impairment, so it is difficult for patients to detect it themselves, and they may miss the best time to see a doctor, which in turn leads to worsening of the disease. At present, fundus photography can effectively screen for retinopathy. Therefore, regular large-scale screening is very necessary. However, there is still a lack of experienced doctors who can read images, which hinders large-scale screening. In recent years, deep learning has achieved rapid development in computer vision fields such as image classification, object detection, and semantic segmentation, and has achieved certain results in the medical field. It is expected to ease the burden of doctors reading images and solve the current shortage of medical resources in some areas. Therefore, it is very important to develop a deep learning-based image recognition method for diabetic retinopathy, which is of great practical value and academic research value.

[0003] In order to meet the needs of large-scale screening tasks, accurate classification and segmentation algorithms are key parts to achieve automatic disease screening and lesion segmentation. However, these tasks face three challenges. First, microaneurysms are the earliest lesions of diabetic retinopathy, but they occupy very little space in the retina. Second, interference unrelated to diabetic retinopathy is easily amplified by convolution and nonlinear operations, which ultimately affects the screening results. Third, the distribution of lesion mask annotation data is extremely unbalanced, and unbalanced data distribution often greatly inhibits the generalization ability of the model. Therefore, it is meaningful and challenging to study deep learning-based diabetic retinopathy lesion image recognition methods.

[0004] Therefore, it is necessary to propose a diabetic retinopathy lesion image recognition method based on the attention model. The purpose of this method is to:

[0005] 1. Design a specific feature extraction module to obtain features that are sensitive to tiny lesion areas and resistant to irrelevant interference, so as to detect lesions in fundus color photos and provide an interpretable basis for automatic diabetic retinopathy screening;

[0006] 2. Guide the model to gradually learn lesion characteristics from extremely unbalanced data, improve the model's generalization ability, effectively complete the diabetic retinopathy screening task, and promote the development of artificial intelligence in the clinical field of ophthalmology.

[0007] Therefore, methods to effectively and accurately detect diabetic retinopathy lesions in fundus color photographs and predict the probability of disease will greatly promote the development of computer vision and clinical ophthalmology. Summary of the invention

[0008] In view of the above challenges and objectives, the present invention provides a method for recognizing diabetic retinopathy lesions images based on an attention model.

[0009] The present invention is achieved through the following technical solutions:

[0010] A method for recognizing diabetic retinopathy lesion images based on an attention model comprises the following steps:

[0011] S1. Obtain a dataset of diabetic retinopathy fundus color photos and perform medical image preprocessing on the original images;

[0012] S2. Construct an attention network model based on semantic segmentation for feature extraction;

[0013] S3, constructing a lesion perception module, a feature preservation module, a feature fusion module and a head attention module to extract lesion related information from the proposed features and generate a lesion detection probability map;

[0014] S4. Based on the input fundus color photograph, the lesion detection probability map obtained by this attention model is used to guide eye disease screening and obtain fundus photograph recognition results with diabetic retinopathy lesion information.

[0015] Preferably, in step S1, the diabetic retinopathy fundus color photograph dataset includes: normal fundus color photographs and non-proliferative diabetic retinopathy fundus color photographs, and contains image-level disease labels and four related lesion mask annotations at the pixel level, including microaneurysm, hemorrhage, hard exudation and soft exudation; among which non-proliferative diabetic retinopathy includes three levels: mild, moderate and severe.

[0016] Preferably, in step S1, the preprocessing of the original image includes:

[0017] A. Crop the extra black background around the image to get a square image;

[0018] B. Convert the sample image from RGB color space to Lab color space, perform adaptive histogram enhancement with limited contrast on the L channel of the brightness layer, and then convert it back to RGB color space to enhance image details;

[0019] C. Scale the image size to a uniform size for easy input into the model;

[0020] D. Use maximum and minimum standardization to normalize the pixel values ​​so that they are distributed in the range [0,1] to speed up model training;

[0021] E. Random angle center rotation, random vertical flipping and random horizontal flipping are used to augment the images in the training set to prevent overfitting.

[0022] Preferably, in step S2, the attention network model based on semantic segmentation adopts an encoder-decoder structure. ResNet-50 is used as an encoder to extract low-level features of the image, and the lesion perception module, feature preservation module, feature fusion module and head attention module described in step S3 are used to form a decoder.

[0023] Preferably, the four layers of the encoder output four low-level features of different scales and depths respectively Correspondingly, the decoder outputs four high-level features of lesions at different scales: The corresponding relationship is as follows:

[0024]

[0025] in, It is the deepest feature output by the last layer of the encoder, that is, the deep semantic feature. and Represent the operations of the ith lesion perception module, feature fusion module, and feature retention module, respectively. No weight is shared between each module. HAM (·) represents the head attention module operation; the four lesion high-level features are processed by a convolution with a kernel size of 3×3 and a Sigmoid activation function layer operation f out (·) Generate four lesion detection probability maps at four different scales

[0026] Preferably, the lesion perception module obtains lesion information through direction perception feature extraction and self-attention mechanism to obtain lesion perception high-level features x dec , the specific formula is as follows:

[0027]

[0028] Among them, x1 is the input x of the lesion perception module after the convolutional layer operation f with a kernel size of 3×3 conv (·) to obtain, f h (·) and f v (·) represent horizontal convolution with kernel size 3×1 and vertical convolution with kernel size 1×3 to obtain direction information, f ort (·) represents a convolution operation with a kernel size of 1×1, which obtains the direction-aware feature x ort;x att is the global attention feature map of dimension 1×1×C obtained by global attention extraction of x; i, j, c represent the indexes of different dimensions of the feature map, respectively, x ort With x att Perform Hadamard product to obtain the high-level features x of the lesion dec .

[0029] Preferably, the global attention extraction calculation formula is as follows:

[0030]

[0031] Among them, x2 is the input x of the lesion perception module after the convolution layer operation f with a kernel size of 1×1 conv (·) where H and W represent the height and width of x2, i, j, c represent the indexes of different dimensions of the feature map, and e is a natural constant; f g (·) represents two convolutional layer operations with a kernel size of 1×1; x g and x att They are the global features of the lesion perception module and the corresponding global attention feature map, respectively.

[0032] Preferably, the feature preservation module operates using global attention to extract and preserve deep semantic features from the last layer output of the encoder, as expressed by:

[0033]

[0034] Among them, x t It is the deep semantic feature output by the last layer of the encoder After the convolutional layer operation f with a kernel size of 1×1 conv (·) we get, H and W represent x respectively. t The height and width of , i, j, c represent the indexes of different dimensions of the feature map, e is a natural constant; f g (·) represents two convolutional layer operations with a kernel size of 1×1; x g and x FPB They are the global features of the feature preservation module and the corresponding global attention feature map, respectively.

[0035] Preferably, the feature fusion module operates by fusing the shallow low-level shape perception features of the decoder part, the deep semantic features and the lesion perception high-level features output by the lesion perception module of the previous layer of the decoder part, which is expressed as:

[0036]

[0037] x fuse (i,j,c)=f conv3(f conc (z1(i,j,c)×x FPB (c),

[0038] z2(i,j,c)×x FPB (c),

[0039] z3(i,j,c)×x FPB (c)))

[0040] Among them, f up1 (·) and f up2 (·) represents the bilinear interpolation upsampling layer operation, f conc (·) represents channel dimension concatenation, f conv1 (·),f conv2 (·) and f conv3 (·) represent convolutional layer operations with a kernel size of 3×3, x FPB is the global attention feature map output by the feature preservation module, x fuse It is the fusion feature output by the feature fusion module.

[0041] Preferably, the head attention module performs spatial attention and channel attention operations on the deep semantic features output by the last layer of the encoder.

[0042] Preferably, in step S2, the attention network model based on semantic segmentation uses stochastic gradient descent as an optimizer during training, uses a dynamic learning rate that first increases and then decreases, wherein the minimum value of the dynamic learning rate is set to 0.001 and the maximum value is set to 0.1, and the lesion detection probability maps of four different scales are bilinearly interpolated and upsampled to the original input image size, and the binary cross entropy function with positive sample weighting is used as the loss function, and the definition formula is as follows:

[0043]

[0044] Among them, H and W are the height and width of the image respectively, g(i,j) represents the lesion mask annotation, and x out (i,j) represents the probability map of lesion detection, α is the weight of positive samples, which is set to 10 here;

[0045] The losses obtained from the four different scale lesion detection probability maps are accumulated to get the total loss value, and then the weights are tuned through back propagation.

[0046] During the verification process, the average intersection-over-union ratio is used as the evaluation index to select the best model. The calculation formula of the average intersection-over-union ratio is:

[0047]

[0048] Among them, N is the number of samples, K is the number of lesion types, and They respectively represent the detection probability map of the kth lesion in the i-th fundus color photo sample and the corresponding mask annotation.

[0049] Preferably, the diabetic retinopathy screening described in step S4 is based on a semantic segmentation attention network model, and the lesion perception high-level features of four different scales of the decoder part are downsampled to the same size, and then the channel dimension is spliced ​​and a convolution layer operation is performed to obtain the screening features, and the global pooling and dense layer are used as output modules to obtain the probability of diabetic retinopathy of the input fundus color photo sample; during the fine-tuning training process, Adamw is used as the optimizer, and the cosine annealing dynamic learning rate is used, wherein the initial learning rate is set to 3e-4, and the cross entropy with label smoothing is used as the loss function, and the calculation process is as follows:

[0050]

[0051] Where N is the number of samples, y i and Represent the predicted disease probability score and smoothed disease label of the i-th fundus color photo sample, respectively. It is calculated by the following formula:

[0052]

[0053] Among them, ε is the smoothing level parameter, which is set to 0.2 here. represents the original hard label, and C is the number of categories for screening classification, which is 2 here.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] Based on the mainstream backbone network as an encoder, the proposed lesion perception module and feature retention module can effectively capture and represent the relevant information of diabetic retinopathy; the proposed module is applied to the lesion perception and screening deep neural network model, which automatically screens the possibility of diabetic retinopathy and gives a lesion prediction map as an effective interpretable evidence; verified by comparative experiments, the present invention has good performance in automatic diabetic retinopathy lesion detection and screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0057] Figure 2 The lesion perception deep neural network model structure of the present invention;

[0058] Figure 3 Screening deep neural network model structures for the present invention;

[0059] Figure 4 Comparison of the detection results of diabetic retinopathy lesions by various methods in the embodiments of the present invention;

[0060] Figure 5 1 is the ROC curve of the screening of diabetic retinopathy by various methods according to the embodiments of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The present invention is further described in detail below with reference to the accompanying drawings.

[0063] The present invention is a diabetic retinopathy lesion image recognition method based on an attention model, see Figure 1 , the present invention is implemented by the following steps:

[0064] S1. Obtain a dataset of diabetic retinopathy fundus color photos and perform preprocessing suitable for medical images on the original images;

[0065] S2. Construct an attention network model based on semantic segmentation for feature extraction;

[0066] S3, constructing a lesion perception module, a feature preservation module, a feature fusion module and a head attention module to extract lesion related information from the proposed features and generate a lesion detection probability map;

[0067] S4. Based on the input fundus color photograph, the lesion detection probability map obtained by this attention model is used to guide eye disease screening and obtain the fundus photograph diagnosis results with diabetic retinopathy lesion information.

[0068] Specifically, for step S1, first, obtain the relevant data sets IDRiD and DDR. IDRiD contains 81 fundus color photograph samples with segmentation masks of four lesions: microaneurysm, hemorrhage, hard exudation and soft exudation, named IDRiD-Seg; DDR contains 757 fundus color photograph samples with the same lesion segmentation masks as IDRiD, named DDR-Seg; in addition, DDR has 13673 fundus color photograph samples with diabetic retinopathy grading labels, and here we select a total of 11609 samples of normal and non-proliferative grades, named DDR-Scr, where the non-proliferative grade includes three specific grades: mild, moderate and severe. Then, the two data sets are divided into training set and test set respectively. Next, for the preprocessing process, it is divided into the following steps:

[0069] A. Use image processing technology to cut out the extra black background around the fundus color photo to obtain a square normalized image;

[0070] B. Convert the sample image from RGB color space to Lab color space, extract the L channel of the brightness layer to perform adaptive histogram equalization with limited contrast, then concatenate it with the original two color layers a and b channels, and convert it to RGB color space to enhance image details;

[0071] C. Scale the image to H×W for easy input to the model, where H=W=512;

[0072] D. Use maximum and minimum standardization to normalize the image pixel values ​​so that they are distributed in the range [0,1] to speed up model training;

[0073] E. Random angle center rotation, random vertical flipping and random horizontal flipping are used to augment the images in the training set to prevent overfitting.

[0074] Specifically, for step S2, Figure 2 As shown in Figure 1, the attention network model based on semantic segmentation consists of an encoder and a decoder. The encoder uses ResNet-50 as a feature extraction module.

[0075] Specifically, for step S3, the lesion perception module, the feature preservation module, the feature fusion module and the head attention module constitute the decoder part of the attention network model based on semantic segmentation.

[0076] First, the preprocessed fundus color photo sample is input, and the encoder extracts low-level features. In particular, the intermediate results of the three residual layers are output. And the last residual layer result As low-level features of different scales and depths, respectively, H i ′=W i ′=H / 2 i+1 , C i =128×2 i ,i=1,2,3,4,.

[0077] Then, for After the head attention module and the lesion perception module, the 0th output of the decoder is obtained, which is expressed as in Then, after a convolution with a kernel size of 3×3 and a channel number equal to the number of lesion types 4 and a Sigmoid activation function layer operation, a lesion detection probability map with a scale of 16×16 is obtained, which is expressed as in

[0078] Next, for After a feature preservation module, and with and They are sent together as input into a feature fusion module to obtain fused features, and then the fused features are sent to a lesion perception module to obtain the first output of the decoder, which is expressed as in Then, after a convolution with a kernel size of 3×3 and a channel number equal to the number of lesion types 4 and a Sigmoid activation function layer operation, a lesion detection probability map with a scale of 32×32 is obtained, which is expressed as in

[0079] By analogy, we can obtain four high-level features of lesion perception at different scales: and lesion detection probability map Among them, H i ″=W i ″=16×2 i ,C=256,i=0,1,2,3. The overall process is defined by the formula as follows:

[0080]

[0081] in, is the deep semantic feature output by the last layer of the encoder, and Represent the operations of the ith lesion perception module, feature fusion module, and feature retention module, respectively. No weight is shared between modules. HAM (·) represents the head attention module operation; The i-th convolution with kernel size of 3×3 and Sigmoid activation function layer operation; is the probability map of lesion detection in group i.

[0082] Then, They are all upsampled to the original size of 512×512 by bilinear interpolation method, and then the binary cross entropy function with positive sample weight is used as the loss function to calculate the loss. The loss function formula is defined as follows:

[0083]

[0084] Among them, H and W are the height and width of the image respectively, g(i,j) represents the lesion mask annotation, and x out (i,j) represents the probability map of lesion detection, α is the weight of positive samples, which is set to 10 here;

[0085] Then, the losses obtained from the four different scales of lesion detection probability maps are accumulated to get the total loss value, and then the weights are tuned through back propagation.

[0086] Specifically, for step S4, Figure 3 As shown in the figure, NPDR is the probability of non-proliferative diabetic retinopathy. The screening deep neural network model keeps the lesion perception deep neural network model unchanged. First, the high-level features of lesion perception at four different scales in the decoder are Use maximum pooling to downsample to the same size, then splice the channel dimension and pass a convolution layer to obtain the screening feature x scr Specifically, the convolution layer contains a convolution operation with a kernel size of 3×3 and an output channel number of 512, batch normalization, and Relu activation function; then the global pooling and dense layer are used as output modules. Specifically, the two results of global average pooling and global maximum pooling are added and straightened into a one-dimensional vector and sent to the dense layer to obtain the probability of diabetic retinopathy of the input fundus color photo sample. Then, the loss is calculated using the cross entropy function with label smoothing as the loss function, and the loss function is defined as follows:

[0087]

[0088] Where N is the number of samples, y i and Represent the predicted disease probability score and smoothed disease label of the i-th fundus color photo sample, respectively. It is calculated by the following formula:

[0089]

[0090] Among them, ε is the smoothing level parameter, which is set to 0.2 here. represents the original hard label, and C is the number of categories for screening classification, which is 2 here.

[0091] Then the weights are tuned through back propagation.

[0092] Specifically, after completing the model training, the fundus color photograph samples of the test set were input to obtain the output prediction maps of four types of diabetic retinopathy lesions and screening prediction probabilities to evaluate the model performance.

[0093] The test results are as follows:

[0094] Table 1 and Table 2 respectively show the objective performance evaluation of the method of the present invention (LANet) on the lesion detection task and various mainstream segmentation models on two datasets.

[0095] Table 1 Evaluation of lesion detection performance of various methods on the DDR-Seg dataset

[0096]

[0097] Table 2 Evaluation of lesion detection performance of various methods on the IDRiD-Seg dataset

[0098]

[0099] Among them, LANet* represents the cross-dataset test results of the model trained using the DDR-Seg dataset on the IDRiD-Seg dataset.

[0100] Figure 4 This is an intuitive comparison chart of the detection results of diabetic retinopathy lesions by various methods. By comparing the objective evaluation scores and subjective results of various methods, it can be seen that the method of the present invention can show good performance on both data sets, and has good performance in small sample data sets or cross-dataset generalization ability.

[0101] Table 3 shows the objective performance evaluation of the method (LASNet) of the present invention in comparison with other methods in the diabetic retinopathy screening task.

[0102] Table 3 Screening performance evaluation of various methods on the DDR-Scr dataset

[0103]

[0104]

[0105] Figure 5 The ROC curves of various methods for screening diabetic retinopathy are shown in Figure 2. By comparison, it can be seen that the method of the present invention has better performance than other methods.

[0106] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0107] Although the above-mentioned embodiments have shown and described the implementation process of the present invention, it is understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for recognizing diabetic retinopathy lesion images based on an attention model, characterized in that: The steps include: S1. Obtain a dataset of diabetic retinopathy fundus color photos and perform medical image preprocessing on the original images; S2. Construct an attention network model based on semantic segmentation for feature extraction; S3, constructing a lesion perception module, a feature preservation module, a feature fusion module and a head attention module to extract lesion related information from the proposed features and generate a lesion detection probability map; in: The diabetic retinopathy fundus color photograph dataset in step S1 includes: Color photographs of normal fundus and non-proliferative diabetic retinopathy, including image-level disease labels and pixel-level microaneurysm, hemorrhage, hard exudate and soft exudate related lesion mask annotations; non-proliferative diabetic retinopathy includes three levels: mild, moderate and severe; The medical image preprocessing of the original image in step S1 includes: A. Crop the extra black background around the image to get a square image; B. Convert the sample image from RGB color space to Lab color space, perform adaptive histogram enhancement with limited contrast on the L channel of the brightness layer, and then convert it back to RGB color space to enhance image details; C. Scale the image size to a uniform size for easy input into the model; D. Use maximum and minimum standardization to normalize the pixel values ​​so that they are distributed in the range [0,1] to speed up model training; E. Random angle center rotation, random vertical flipping and random horizontal flipping are used to augment the images in the training set to prevent overfitting; The attention network model based on semantic segmentation described in step S2 adopts an encoder-decoder structure; ResNet-50 is used as an encoder to extract low-level features of the image; the lesion perception module, feature preservation module, feature fusion module and head attention module described in step S3 are used to form a decoder; The four layers of the encoder output four low-level features of different scales and depths respectively The corresponding decoder outputs four high-level features of lesions at different scales The corresponding relationship is as follows: in, It is the deepest feature output by the last layer of the encoder, that is, the deep semantic feature. and Represent the operations of the ith lesion perception module, feature fusion module, and feature retention module, respectively. No weight is shared between modules. HAM (·) represents the head attention module operation; it consists of four lesion high-level features, which are processed by a convolution with a kernel size of 3×3 and a Sigmoid activation function layer operation f out (·) Generate four lesion detection probability maps at four different scales The lesion perception module obtains lesion information through direction perception feature extraction and self-attention mechanism to obtain lesion perception high-level features x dec , the specific formula is as follows: Among them, i, j, c represent the indexes of different dimensions of the feature map respectively, and x1 is the input x of the lesion perception module after the convolution operation f with a kernel size of 3×3 conv (·) to obtain, f h (·) and f v (·) represent horizontal convolution with kernel size 3×1 and vertical convolution with kernel size 1×3 to obtain direction information, f ort (·) represents a convolution operation with a kernel size of 1×1, which obtains the direction-aware feature x ort , x att is the global attention feature map with a dimension of 1×1×C obtained by global attention extraction of x, ort With x att Perform Hadamard product to obtain the high-level features x of the lesion dec , The global attention extraction calculation formula is as follows: Among them, i, j, c represent the indexes of different dimensions of the feature map, e is a natural constant, and x2 is the input x of the lesion perception module after the convolution operation f with a kernel size of 1×1 conv (·) We get that H and W in the global attention extraction calculation formula represent the height and width of x2 respectively; x in the global attention extraction calculation formula in the lesion perception module g is the global feature map of the lesion perception module, f g (·) represents two convolutional layer operations with a kernel size of 1×1; The feature preservation module operation uses global attention to extract and retain deep semantic features from the output of the last layer of the encoder, and the formula is expressed as: Among them, i, j, c represent the indexes of different dimensions of the feature map, e is a natural constant, and x t It is the deep semantic feature output by the last layer of the encoder After the convolutional layer operation f with a kernel size of 1×1 conv (·) We get that H and W of the feature preservation module represent x t The height and width of the feature holding module x g is the global feature of the feature preservation module, x FPB is the global attention feature map output by the feature preservation module. The feature fusion module operates by fusing the shallow low-level shape perception features of the decoder part, the deep semantic features and the lesion perception high-level features output by the lesion perception module of the previous layer of the decoder part. The formula is expressed as: x fuse (i,j,c)=f conv3 (f conc (z1(i,j,c)×x FPB (c), z2(i,j,c)×x FPB (c), z3(i,j,c)×x FPB (c))) Among them, f up1 (·) and f up2 (·) represents the bilinear interpolation upsampling layer operation, f conc (·) represents channel dimension concatenation, f conv1 (·),f conv2 (·) and f conv3 (·) represent convolutional layer operations with a kernel size of 3×3, x FPB is the global attention feature map output by the feature preservation module, x fuse The fusion feature output by the feature fusion module, the head attention module performs spatial attention operation and channel attention operation on the deep semantic features output by the last layer of the encoder, the attention network model based on semantic segmentation in step S2, during the training process, uses stochastic gradient descent as the optimizer, uses a dynamic learning rate that first increases and then decreases, wherein the minimum value of the dynamic learning rate is set to 0.001 and the maximum value is set to 0.1, and the four different scales of lesion detection probability maps are bilinearly interpolated and upsampled to the original input image size, and the binary cross entropy function with positive sample weighting is used as the loss function, and the definition formula is as follows: Among them, H and W of the loss function are the height and width of the lesion detection probability map, g(i,j) represents the lesion mask annotation, and x out (i, j) represents the probability map of lesion detection, α is the weight of positive samples, and α is 10; The losses obtained from the four different scales of lesion detection probability maps are accumulated to get the total loss value, and then the weights are tuned through back propagation. During the verification process, the average intersection-over-union ratio is used as the evaluation index to select the best model. The calculation formula of the average intersection-over-union ratio is: Among them, N is the number of samples, K is the number of lesion types, and They respectively represent the detection probability map of the kth lesion in the i-th fundus color photo sample and the corresponding mask annotation.

Citation Information

Patent Citations

  • Deep learning-based diabetic retina image classification method and system

    CN108615051A

  • Medical image labeling method and device for deep learning

    CN110993064A

  • Retinal fundus vessel segmentation method based on deep multi-scale attention convolutional neural network

    CN112102283A

  • Diabetic retinopathy automatic evaluation system based on focus perception modeling

    CN112967227A

  • Medical image segmentation and severity grading using neural network architectures with semi-supervised learning techniques

    US10430946B1