Combined grading method for diabetic retinopathy and diabetic macular edema

Abstract

The application belongs to the technical field of medical image processing, and provides a combined grading method for diabetic retinopathy and diabetic macular edema, which comprises the following steps: 1) constructing a combined grading network, wherein MobileNet V2 is used as the backbone network of the model, and an adaptive attention module is contained, the model is implemented on a PyTorch architecture, and is trained in an end-to-end manner; 2) constructing network training based on semi-supervised learning, wherein the network training based on semi-supervised learning adopts a student model and a teacher model, and the two models are trained simultaneously, and both use the proposed combined grading network; and 3) performing network grading test. The application realizes the combined grading task for diabetic retinopathy and diabetic macular edema, and the adaptive adjustment is performed in the training process, so that the flexibility is high; the semi-supervised learning strategy can make full use of unmarked data, and to a certain extent, the problem of insufficient labeled data is relieved.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a combined grading method for diabetic retinopathy and diabetic macular edema. Background Technology

[0002] Diabetic retinopathy (DR) is a serious complication of diabetes and a leading cause of blindness and vision impairment worldwide. Pathological features of DR include microaneurysms, hemorrhages, soft exudates, and hard exudates. These symptoms help ophthalmologists determine the grade of DR. Diabetic macular edema (DME) is a complication of DR caused by the accumulation of extracellular fluid in the macula, leading to retinal thickening or hard exudates. The distance between the hard exudate and the macula is used as a clue for diagnosing DME; the closer the distance, the higher the risk. If DR and DME are detected and treated early, most patients can avoid the risk of blindness. Therefore, efficient and accurate screening for DR and DME is crucial in clinical practice.

[0003] Supervised CNNs require a large amount of labeled data for training. However, due to high costs and the relative scarcity of labeled medical data, the clinical application of supervised CNNs is severely limited. Summary of the Invention

[0004] The present invention is implemented as follows: a combined grading method for diabetic retinopathy and diabetic macular edema includes the following steps:

[0005] 1) Construct a joint hierarchical network, in which MobileNet V2 is the backbone network of the model and contains an adaptive attention module. The model is implemented on the PyTorch architecture and trained in an end-to-end manner.

[0006] 2) Construct a network training based on semi-supervised learning. The semi-supervised learning network training adopts a student model and a teacher model, which are trained simultaneously and both use the proposed joint hierarchical network.

[0007] 3) Conduct network classification testing, which is divided into the following cases according to the data type:

[0008] Labeled data on DR lesion grading

[0009] b has labeled data on DME lesion grading.

[0010] c has tagged data for combined DR and DME lesion grading.

[0011] d. Unlabeled data lesion grading.

[0012] Further technical solutions, including the joint hierarchical network training configuration steps, also include:

[0013] 1. Load the pre-trained parameters from ImageNet;

[0014] 2. Crop the images in the training dataset to include only a portion of the field of view (FOV) and resize them to 250×250;

[0015] 3. Randomly scale and crop the image to 224×224, then randomly flip, rotate and affine it in the horizontal and vertical directions;

[0016] IV. Optimize the network using the Adam optimizer;

[0017] 5. The maximum learning rate is set to 0.001. The warm-up strategy is used to increase the learning rate in the first 20 periods and to decrease the learning rate in the last 12 periods.

[0018] 6. All models are trained for 100 periods, with a batch size of 80.

[0019] A further technical solution is that the steps for training the network based on semi-supervised learning include:

[0020] a) At the start of training, both learnable parameters β and γ are set to 1. During training, β and γ gradually change. The network selects whether to enable the adaptive attention module according to different tasks and adaptively adjusts the adaptive attention module.

[0021] b) Random data augmentation, using the same amount of labeled and unlabeled data for training;

[0022] c) Send labeled data to the student model to generate predictions y s And use cross-entropy loss to calculate y s The classification loss between the true label y and the actual label y;

[0023] d) Send the enhanced unlabeled data to the student and teacher models to generate predictions for y. s and pseudo-labels y t And use the mean squared error loss to calculate y s and y t Consistency loss between them;

[0024] e) The teacher model parameters are updated using the exponentially moving average (EMA) of the student model weights. In a further technical solution, the adaptive attention module includes two sub-modules: a channel attention module and a spatial attention module. By using shortcut connections to improve the convolutional block attention module, the network can adaptively adjust the attention module according to different tasks and decide whether to use the attention module and which sub-module to focus on.

[0025] A further technical solution involves training both the student model and the teacher model using the same amount of labeled and unlabeled data to avoid overfitting, and only calculating the consistency loss of the unlabeled data; for unlabeled sample X... U Two data augmentation versions are randomly generated. and in Send to the student model to generate predictions Send to the teacher model to generate predictions Predict As a pseudo-label, calculate y s and The consistency loss is used to force the student model to produce predictions consistent with the teacher model.

[0026] A further technical solution, the definition of consistency loss is as follows:

[0027]

[0028] Where N u c is the total number of unlabeled samples. i The target class of unlabeled samples y s and These are the predictions from the student model and the teacher model, respectively.

[0029] For labeled sample X L The training strategy is the same as traditional supervised learning, using randomly augmented data X′. L Send to the network to generate prediction y s =f(X′) L ,θ), and calculate y s The cross-entropy loss between the true value y and the ground truth y is used as a supervision signal for classification. The classification loss is defined as follows:

[0030]

[0031] Where N l K is the total number of labeled samples, and K is the number of disease levels; the final loss is defined as follows:

[0032] L = L cls +ω×δ×L consist (7)

[0033] A hyperparameter ω is introduced into the total loss function to control L. consist The weight, δ, is an adaptive weighting coefficient, calculated as follows:

[0034]

[0035] Where S is the current training step, L is the length, and the use of δ makes the model focus on improving scoring accuracy in the early stages of training, while paying more attention to the consistency of predictions in the later stages.

[0036] In a further technical solution, the student model and the teacher model update their model parameters at each step. The former updates the parameters using backpropagation, while the latter performs an exponential moving average (EMA) to update the generated target, as shown below.

[0037] θ′ t =αθ′ t-1 +(1-α)θ t

[0038] Where t represents the training step, θ and θ′ are the parameters of the student model and teacher model, respectively, and α is the EMA decay. In the first L-step, α = min(1 - 1 / (global_step+1), 0.6) is used, and α = 0.6 is used for the rest of the training. The adjustment function can control the change of α more smoothly and allow the teacher model to retain more previous information.

[0039] The combined grading method for diabetic retinopathy and diabetic macular edema provided in this invention has the following beneficial effects:

[0040] 1. A joint grading network for fundus lesions was proposed to achieve joint grading of diabetic retinopathy and diabetic macular edema;

[0041] 2. A general adaptive attention module is proposed, which can be embedded into any network and adaptively adjusted during training to enhance the flexibility of the attention module;

[0042] 3. Additional unlabeled data is used to perform semi-supervised learning on the network. Semi-supervised learning strategies can make full use of unlabeled data, alleviating the problem of insufficient labeled data to some extent. Attached Figure Description

[0043] Figure 1 A flowchart illustrating the combined grading method for diabetic retinopathy and diabetic macular edema provided in this embodiment of the invention;

[0044] Figure 2 A flowchart illustrating the joint grading network training configuration in the joint grading method for diabetic retinopathy and diabetic macular edema provided in this embodiment of the invention.

[0045] Figure 3 A flowchart illustrating the semi-supervised learning network training process in the combined grading method for diabetic retinopathy and diabetic macular edema provided in this embodiment of the invention.

[0046] Figure 4 This is a diagram showing the overall structure of the joint grading network in the joint grading method for diabetic retinopathy and diabetic macular edema provided in an embodiment of the present invention.

[0047] Figure 5 The architecture diagram of the adaptive attention module in the combined grading method for diabetic retinopathy and diabetic macular edema provided in the embodiments of the present invention;

[0048] Figure 6 This is a schematic diagram of semi-supervised learning in the combined grading method for diabetic retinopathy and diabetic macular edema provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0051] Methodology:

[0052] The basic structure of a joint hierarchical network is as follows: Figure 1 and 4 As shown, it consists of three parts: a backbone network, an adaptive attention module, and a classifier. The joint hierarchical network can be trained in an end-to-end manner. To make full use of unlabeled data, semi-supervised learning is used to train the network.

[0053] I. Overview of Joint Hierarchical Networks

[0054] like Figure 1 and 4 As shown, the joint hierarchical network uses a backbone network as a feature extractor to extract disease-related features from the input color image. The backbone network can be any pre-trained network. Considering the model's practicality, MobileNet V2 was chosen as the backend network, retaining only the portion before the fully connected layers. At the start of training, the model was pre-trained on the ImageNet dataset. The backbone network can generate feature maps F∈R from the last feature extraction layer. H×W×C To reduce computational cost, a 1×1 convolutional layer is added after the backbone network to perform a 1×1 convolution operation on F, resulting in Fi. reduce ∈R H ×W×C′ Where C′=C / 2. Then, according to the hierarchical task, F... reduce Send to different branches. In DR or DME hierarchical tasks, F reduceOnly sent to the DR or DME grading branch, while in a joint grading task, F reduce Simultaneously send to both the DR and DME hierarchical branches. Taking the DR hierarchical branch as an example, send F... reduce As input, it is sent to the adaptive attention module, which can adaptively receive the input from F. reduce Learn features that are beneficial for DR classification and integrate them into F reduce In the process, an attention feature map F is generated. DR ∈R H×W×C′ Then F DR The data is fed into a classifier consisting of a Global Average Pooling (GAP) layer and a Multilayer Perceptron (MLP) for grading. Finally, to fully utilize the unlabeled data, consistency regularization is introduced into the final loss, and the network is trained using semi-supervised learning.

[0055] The DR branch and DME branch are two independent branches that can be used for DR or DME hierarchies separately, or for joint DR and DME hierarchies simultaneously. GAP and MLP are abbreviations for Global Average Pooling and Multilayer Perception, respectively.

[0056] II. Adaptive Attention Module

[0057] The proposed adaptive attention module consists of two sub-modules: a channel attention module and a spatial attention module. It improves the convolutional block attention module by using shortcut connections, enabling the network to adaptively adjust the attention module according to different tasks and decide whether to use the attention module and which sub-module to focus on.

[0058] The structure of the adaptive attention module is as follows: Figure 5 As shown, the channel attention module uses F i ∈R H×W×C As input. and respectively through F i Average pooling and max pooling are performed on each feature map in the mid-channel dimension, and then sent to a shared MLP to generate attention scales in the channel directions. As described below,

[0059]

[0060] Where σ is a sigmoid function, W0∈R c×c′ and W1∈R c′×c These are the parameters of the MLP, C′=C / 2. It can enhance useful feature maps and suppress F in the channel dimension. i The feature maps are useless. Therefore, the attention F of the attention direction can be obtained as follows: c_att ,

[0061]

[0062] in This represents element-wise multiplication, where β is a learnable parameter.

[0063] To further highlight disease-specific features across spatial domains, the adaptive attention module also includes a spatial attention module, which aggregates spatial information annotations and generates a feature map by applying max pooling and average pooling operations along the spatial domain. and Then, and The data are concatenated and fed into the convolutional layer to obtain the attention scale in spatial orientation. As shown below,

[0064]

[0065] Here, Conv represents the convolutional layer operation. Therefore, the attention F in the spatial direction can be obtained as follows. c_att ,

[0066]

[0067] Here, γ is a learnable parameter.

[0068] At the start of training, both learnable parameters β and γ are set to 1. During training, β and γ gradually change, and the network can choose whether to enable the adaptive attention module based on different tasks, and adaptively adjust the adaptive attention module accordingly.

[0069] It consists of a channel attention component and a spatial attention component. Channel and spatial attention can utilize the relationships between features in channels and spatially. β and γ are two learnable hyperparameters. β = 0 and γ = 0 indicate that the channel and spatial attention components are inactive, respectively. During training, β and γ will adaptively learn to adjust the attention module.

[0070] III. Semi-supervised learning

[0071] To effectively utilize unlabeled data, semi-supervised learning is employed to train the model (e.g., Figure 3 and 6 (As shown). It does not destroy image details and does not strictly require the quality of unlabeled images, thus making it more suitable for retinal disease classification tasks. The core of training is consistency regularization. Figure 6 As shown, it is added to the loss to provide additional supervision signals for training to obtain a more accurate model.

[0072] After random data augmentation, labeled data is sent to the student model to generate predictions for y. s And use cross-entropy loss to calculate y s The classification loss is calculated between the actual label y and the true label y. Augmented unlabeled data is fed into the student and teacher models to generate predictions for y. s and pseudo-labels y t And use the mean squared error loss to calculate y s and y t Consistency loss between them. Teacher model parameters are updated using the exponential moving average (EMA) of student model weights.

[0073] Two CNN networks, a student model and a teacher model, are trained simultaneously, both using the proposed joint hierarchical network. They are trained using the same amount of labeled and unlabeled data. To avoid overfitting, the consistency loss is calculated only for the unlabeled data. For unlabeled sample X... U Two data augmentation versions are randomly generated. and in Send to the student model to generate predictions Send to the teacher model to generate predictions Predict As a pseudo-label. Calculate y. s and The consistency loss is used to force the student model to produce predictions consistent with the teacher model. The consistency loss is defined as follows:

[0074]

[0075] Where N u c is the total number of unlabeled samples. i The target class of unlabeled samples y s and These are the predictions from the student model and the teacher model, respectively.

[0076] For labeled sample X L The training strategy is the same as traditional supervised learning, using randomly augmented data X′. L Send to the network to generate prediction y s =f(X′) L ,θ), and calculate y s The cross-entropy loss between the true value y and the ground truth y is used as a supervision signal for classification. The classification loss is defined as follows:

[0077]

[0078] Where N lK is the total number of labeled samples, and K is the number of disease levels. The final loss is defined as follows:

[0079] L = L cls +ω×δ×L consist (7)

[0080] A hyperparameter ω is introduced into the total loss function to control L. consist The weight, δ, is an adaptive weighting coefficient, calculated as follows:

[0081]

[0082] Where S is the current training step and L is the length. The use of δ makes the model focus on improving scoring accuracy in the early stages of training, while emphasizing prediction consistency in the later stages.

[0083] The student model and the teacher model update their model parameters at each step. The former uses backpropagation to update the parameters, while the latter performs an exponential moving average (EMA) to update the generated target, as shown below.

[0084] θ′ t =αθ′ t-1 +(1-α)θ t (9)

[0085] Where t represents the training step, θ and θ′ are the parameters of the student model and teacher model, respectively, and α is the EMA decay. This application designs a piecewise adjustment function, where α = min(1 - 1 / (global_step+1), 0.6) is used in the first L steps, and α = 0.6 is used in the remaining training steps. This adjustment function allows for smoother changes in α and enables the teacher model to retain more previous information.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A combined grading method for diabetic retinopathy and diabetic macular edema, characterized in that, Includes the following steps: 1) Construct a joint hierarchical network, in which MobileNet V2 is the backbone network of the model and contains an adaptive attention module. The model is implemented on the PyTorch architecture and trained in an end-to-end manner. 2) Construct a network training based on semi-supervised learning. The semi-supervised learning network training adopts a student model and a teacher model, which are trained simultaneously and both use the proposed joint hierarchical network. 3) Conduct network classification testing; The joint hierarchical network includes a backbone network, an adaptive attention module, and a classifier. MobileNet V2 is the backbone network of the model. The adaptive attention module is connected after the backbone network and is used to adaptively enhance the feature maps extracted by the backbone network. The adaptive attention module includes a channel attention submodule and a spatial attention submodule.

2. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 1, characterized in that, The joint hierarchical network training configuration steps also include: a) Load the pre-trained parameters from ImageNet; b) Crop the images in the training dataset to include only a portion of the field of view and resize them to 250×250; c) Randomly scale and crop the image to 224×224, then randomly flip, rotate, and affine it in the horizontal and vertical directions; d) Optimize the network using the Adam optimizer; e) The maximum learning rate is set to 0.

001. The warm-up strategy is used to increase the learning rate in the first 20 periods and to decrease the learning rate in the last 12 periods. f) All models are trained for 100 periods, with a batch size of 80.

3. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 2, characterized in that, The steps for training the network based on semi-supervised learning include: I. At the start of training, the learnable parameters in the adaptive attention module used to control the channel attention submodule. and learnable parameters for controlling the spatial attention submodule All are set to 1; during training, and As the training process changes, the network selects whether to enable the channel attention submodule and / or spatial attention submodule of the adaptive attention module according to the current training task, and adaptively adjusts the output of the adaptive attention module.

2. Perform random data augmentation on both labeled and unlabeled data, and use the same amount of labeled and unlabeled data for training; 3. Send the labeled data, which has undergone random data augmentation, to the student model to generate predictions. And use cross-entropy loss to calculate and real labels Classification loss between; 4. Send the unlabeled data, which has undergone random data augmentation, to the student model and the teacher model respectively to generate predictions for the student model. Teacher model predicts pseudo-labels And calculate using mean squared error loss and Consistency loss between them; 5. The teacher model parameters are updated by the exponential moving average of the student model weights.

4. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 2, characterized in that, The network hierarchy test is divided into the following categories based on the data type: (1) Labeled data on the classification of diabetic retinopathy lesions, (2) Labeled data on the grading of diabetic macular edema. (3) Joint grading of diabetic retinopathy and diabetic macular edema with tagged data. (4) Unlabeled data lesion grading.

5. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 1, characterized in that, The adaptive attention module includes two sub-modules: a channel attention module and a spatial attention module. The adaptive attention module improves the convolutional block attention module by establishing shortcut connections between the input and output of the convolutional block attention module, enabling the network to adaptively adjust the attention module according to different training tasks and decide whether to use the attention module and which sub-module to focus on.

6. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 1, characterized in that, The student model and the teacher model are trained using the same amount of labeled and unlabeled data to avoid overfitting; only the consistency loss of the unlabeled data is calculated. For unlabeled samples... Two data augmentation versions are randomly generated. and ,in Send to the student model to generate predictions , Send to the teacher model to generate predictions ,in, and These are the parameters of the student model and the teacher model, respectively; the prediction... As a pseudo-label, calculation and The consistency loss is used to force the student model to produce predictions consistent with the teacher model.

7. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 6, characterized in that, The definition of consistency loss is as follows: in It is the total number of unlabeled samples. It is a data augmentation version of unlabeled samples. , and These are the predictions from the student model and the teacher model, respectively. For labeled samples The training strategy is the same as traditional supervised learning, using randomly augmented data. Send to the network to generate predictions and calculate truth value The cross-entropy loss between them is used as a supervision signal for classification.

8. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 7, characterized in that, The graded loss is defined as follows: in It is the total number of labeled samples. It refers to the number of disease severity levels; the final loss is defined as follows: Introducing a hyperparameter into the total loss function To control The weight, It is an adaptive weighting coefficient, calculated as follows: in, This is the current training step. It is length. The use of this method allows the model to focus on improving scoring accuracy in the early stages of training, while emphasizing the consistency of predictions in the later stages.

9. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 2, characterized in that, The student model and teacher model update their model parameters at each step. The former updates the parameters using backpropagation, while the latter performs an exponential moving average to update the generated target, as shown below: in, Indicates the training steps, It is an exponential moving average decay.

10. The combined grading method for diabetic retinopathy and diabetic macular edema according to claim 9, characterized in that, Before the start of training Used in each training step The remaining training uses =0.6, the adjustment function can control The transitions are smoother, and the teacher model retains more of the previous information.

Images