Fine-grained lesion characteristic diabetic retinopathy grading method and system

Abstract

The invention provides a fine-grained lesion feature diabetic retinopathy grading method and system, and belongs to the field of image processing, and the method comprises the steps: S1, inputting a sugar net fundus color image into a high and low frequency feature extraction network, and achieving the simultaneous extraction of high and low frequency features through a block embedding module and the repeated stacking of high and low frequency blocks, and obtaining a processed feature map # imgabs0 #; s2, inputting the feature map # imgabs1 # into a sample relation interaction module, performing information interaction among different samples through a self-attention mechanism of Transform to obtain a new feature map # imgabs2 #, and obtaining a grading result # imgabs3 # through a final classifier; and S3, using a momentum distillation learning module to optimize the student models proposed in S1 and S2, continuously updating the momentum model through index moving average, and then training the student models by using pseudo labels generated by the momentum model. According to the method provided by the invention, the sugar net grading effect can be improved, and a basis is provided for disease grading.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method and system for grading diabetic retinopathy based on fine-grained lesion features. Background Art

[0002] Diabetic retinopathy (DR) is one of the most common complications of diabetes and one of the leading causes of blindness. With the increase in human life expectancy, irregular lifestyles, and the influence of various other factors, the number of diabetic patients is expected to continue to rise. In this context, regular retinal screening for diabetic patients has been proven to be a cost-effective and vital means of care. Clinicians can diagnose diabetic retinopathy by observing lesions in fundus images and grade the patient's condition according to the type and number of lesions. Doctors can formulate corresponding treatment plans based on the grading results. However, ophthalmologists need to invest a lot of time and energy in the diagnosis of diabetic retinopathy, which puts a huge pressure on medical resources. Therefore, designing an automatic diabetic retinopathy diagnosis algorithm can not only reduce the workload of doctors, but also improve the efficiency and accuracy of diagnosis.

[0003] Although deep learning methods have achieved great success in disease grading of diabetic retinopathy, there are still three challenges that need to be addressed.

[0004] First, in clinical practice, ophthalmologists usually identify lesions by observing fine-grained features such as texture and color of lesions, and then grade the disease according to the type and number of lesions. At present, most diabetic retinopathy grading methods are based on convolutional neural networks. Although convolutional neural networks can effectively capture fine-grained features through local convolution, they can produce considerable results even when data is limited. However, convolutional neural networks have limitations in long-term modeling and cannot effectively capture the relationship between different lesions, which limits the further improvement of diabetic retinopathy grading accuracy. The visual Transformer model captures the dependencies between positions in the input sequence through the self-attention mechanism, thereby more effectively modeling long-distance dependencies. However, due to the lack of inductive bias, the visual Transformer requires a large amount of data to effectively extract fine-grained features and achieve satisfactory results. Although some researchers have tried to use visual transformers for diabetic retinopathy disease grading, its performance is difficult to fully meet the needs. Therefore, how to design a network architecture that has the advantages of convolutional neural networks and visual transformers to effectively extract fine-grained lesion features in diabetic retinopathy fundus images has become a challenge in diabetic retinopathy grading.

[0005] In addition, under clinical conditions, there are differences in the number of patients with different levels of diabetic retinopathy, which also leads to serious class imbalance problems in diabetic retinopathy datasets, and the overall distribution is long-tailed. This unbalanced class distribution may cause the model to pay too much attention to the diabetic retinopathy level with more samples, thus affecting the final grading performance. Although many researchers have proposed a variety of long-tail visual recognition methods, these long-tail visual recognition methods often improve the performance of the tail category at the expense of the performance of the head category. Therefore, how to optimize the performance of the tail category without compromising the performance of the head category has become a challenge in the grading of diabetic retinopathy.

[0006] Ophthalmologists usually grade lesions based on the type and number of lesions. However, some lesions are sometimes difficult to detect and label accurately because of their variable morphology or subtlety. Therefore, in the dataset, some images may be incorrectly labeled, which may cause the model to overfit to noisy labels, thereby reducing its generalization performance. In this case, designing a reasonable training method to ensure that the model is not penalized for producing reasonable predictions that are different from the labels becomes a challenge in the grading of diabetic retinopathy. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a fine-grained lesion feature diabetic retinopathy grading method, comprising the following steps:

[0008] Step S1: Input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain the processed feature map ;

[0009] Step S2: The feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier ;

[0010] Step S3: Use the high- and low-frequency feature extraction network and the sample relationship interaction module as student models, optimize the student model using a representation learning method based on momentum distillation, continuously update the momentum model through exponential moving average, and then use the pseudo labels generated by the momentum model to train the student model.

[0011] Beneficial effects:

[0012] 1. The method disclosed in the present invention proposes a feature extraction method based on the combination of high- and low-frequency features. The attention mechanism is combined with the convolution operation by using high- and low-frequency blocks. It can not only extract fine-grained lesion features, but also extract the relationship between different lesions, which can effectively improve the disease grading ability of the model.

[0013] 2. The method disclosed in the present invention proposes a long-tail visual recognition method based on sample relationships, which considers the relationship between samples from a learning perspective, obtains additional information by performing feature interaction between fine-grained lesion features of samples of different categories, so that the tail category can obtain gradients from different virtual samples, thereby optimizing the performance of the tail category without compromising the performance of the head category.

[0014] 3. The method disclosed in the present invention proposes a representation learning method based on momentum distillation, which obtains the momentum model by adopting the moving average of the model and uses the pseudo-label generated by the momentum model as an additional supervisory signal. This method can ensure that the model will not be penalized for producing outputs that are different from the true label but reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a method for grading diabetic retinopathy based on fine-grained lesion features according to an embodiment of the present invention;

[0016] Figure 2A It is a structural diagram of a high and low frequency feature extraction network in an embodiment of the present invention;

[0017] Figure 2B It is a structural diagram of high and low frequency blocks;

[0018] Figure 2C The schematic diagram of the structure of the high and low frequency mixer is shown;

[0019] Figure 3 Schematic diagram of the calculation process of the sample interaction module in an embodiment of the present invention;

[0020] Figure 4 Schematic diagram of a momentum distillation framework according to an embodiment of the present invention;

[0021] Figure 5 This is a module block diagram of a fine-grained lesion feature diabetic retinopathy grading system in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Embodiment 1

[0024] like Figure 1 As shown, a fine-grained lesion feature diabetic retinopathy grading method provided by an embodiment of the present invention includes the following steps:

[0025] Step S1: Input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain the processed feature map ;

[0026] Step S2: The feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier ;

[0027] Step S3: Use the high- and low-frequency feature extraction network and the sample relationship interaction module as the student model, optimize the student model using the representation learning method based on momentum distillation, continuously update the momentum model through exponential moving average, and then use the pseudo labels generated by the momentum model to train the student model.

[0028] In one embodiment, the above step S1: input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain a processed feature map , including:

[0029] Step S11: construct a high- and low-frequency feature extraction network, which is composed of multiple block embedding modules and high- and low-frequency block stacks; the retinal color image or feature map first passes through the block embedding module, and performs feature processing and downsampling through convolution operations to obtain the downsampled feature map ,in, , C represent feature maps The number of pixels and channels;

[0030] Step S12: feature map Apply layer normalization to obtain the normalized feature map , then Divide the channels into and , respectively input into the high and low frequency blocks consisting of high frequency mixer and low frequency mixer, where, ;

[0031] This step is crucial to speed up the convergence of model training, stabilize the training process, and effectively improve the generalization ability of the model;

[0032] Step S13: Input high frequency mixer, which consists of maximum pooling operation and depth convolution operation. Divided into and , After a maximum pooling layer and a linear layer, After a linear layer and a deep convolutional layer, formulas (1) to (2) are as follows:

[0033] (1)

[0034] (2)

[0035] Among them, DwConv represents the deep convolution operation, and represents the output of the high frequency mixer;

[0036] The high-frequency mixer consists of a maximum pooling operation and a deep convolution operation to extract and combine high-frequency features. The maximum pooling operation reduces the size of the feature map by selecting the maximum value in each local area to retain significant high-frequency features. The deep convolution operation further extracts and integrates high-frequency features through a series of convolutional layers to obtain richer expressive capabilities.

[0037] Step S14: Input the low-frequency mixer, which first uses a pooling strategy to reduce the size of the low-frequency feature map, then uses a multi-head self-attention mechanism for feature processing, and finally performs upsampling to restore the feature map to its original size, as shown in formula (3):

[0038] (3)

[0039] in, represents the average pooling operation, represents the multi-head self-attention mechanism, represents upsampling;

[0040] The low-frequency mixer is implemented using a self-attention operation to capture global low-frequency features. The self-attention mechanism associates and integrates input features by calculating the similarity and weights between features, thereby extracting the global structure and contextual information of the image.

[0041] Step S15: stack the outputs of the high-frequency mixer and the low-frequency mixer in the channel dimension, and use depthwise convolution to implement information exchange between image blocks. Formulas (4) to (5) are as follows:

[0042] (4)

[0043] (5)

[0044] The high- and low-frequency feature extraction network designed by the present invention can effectively capture the specific frequency information on the corresponding channel, thereby learning more comprehensive and rich features in a wider frequency range. The structure of this high- and low-frequency mixer provides strong support for the performance and robustness of the network.

[0045] Step S16: The feature map After global average pooling in one spatial dimension, we get the attention feature map , After the fully connected layer and Sigmoid operation, the value is mapped to [0, 1] and finally combined with the original feature map Multiply to get a new feature map ;

[0046] The high-frequency mixer and the low-frequency mixer perform feature extraction tasks in the spatial dimension, focusing on capturing local details and overall structural information of the image. On this basis, the SE attention (Squeeze-and-ExcitationAttention) mechanism further improves the model's feature expression ability by applying attention operations in the channel dimension. This attention mechanism can enhance the model's emphasis on key channels, that is, those channels with larger weights in a specific context, so that the model can focus more on the features that are most important for classification and recognition tasks.

[0047] Step S17: The feature map With feature map Add to get the feature map , used to promote rapid convergence of the model;

[0048] Step S18: Feature Map After layer normalization, it is sent to the forward feedback layer to combine the newly obtained feature map with the feature map Add together to get the feature map ;

[0049] Step S19: Repeat steps S11 to S18 to form a structure of multiple block embedding modules and high and low frequency block stacking to ensure that the high and low frequency feature extraction network is fully learned and its performance is optimized.

[0050] The number of repetitions to be performed should be determined according to the complexity of the task and the training requirements. In the embodiment of the present invention, 4 iterations are set.

[0051] Figure 2A The schematic diagram of the high- and low-frequency feature extraction network is shown, which uses the pyramid structure design mode commonly used in computer vision to extract lesion features from fundus images. Figure 2A As shown in the figure, during the feature extraction process, as the network deepens, the resolution of the feature map gradually decreases, while the number of channels gradually increases. Through the pyramid structure, the network can capture image information of different scales. As the network deepens, the resolution of the feature map gradually decreases, which helps to extract higher-level abstract features. At the same time, the increase in the number of channels enables the network to better represent rich feature information. Figure 2B It is a structural diagram of high and low frequency blocks. Figure 2C The schematic diagram of the structure of the high and low frequency mixer is shown.

[0052] In one embodiment, the above step S2: the feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier , including:

[0053] Step S21: Enter the sample interaction module, first The self-attention mechanism in the Transformer layer is applied to the sample interaction module to achieve cross-attention between different samples, thereby obtaining the feature map ;

[0054] Step S22: Use the auxiliary classifier and the final classifier to jointly classify Perform detailed classification processing, conduct in-depth analysis of features through the fully connected layer, then apply the Softmax function to calculate the probability distribution, and finally determine the category with the highest probability through the argmax operation, so as to obtain the final grading result of diabetic retinopathy .

[0055] Since the batch statistics of the test data cannot be assumed, there may be a gap in the features before and after the sample interaction module. Therefore, it is not possible to simply infer new samples by removing the sample interaction module. Therefore, in addition to the final classifier, the present invention designs a new auxiliary classifier, which not only enables the model to learn from the final classifier, but also ensures that the features before and after the sample interaction module remain consistent. To achieve this, it is only necessary to share weights between the auxiliary classifier and the final classifier. This simple and effective strategy is called a shared classifier. By using the shared classifier strategy, the sample interaction module can be removed during testing while still enjoying the advantages of the sample interaction module in sample relationship learning.

[0056] The relationships between different samples are diverse and complex. Traditional methods usually use a direct way to explore the relationships between these samples, such as joint operations on different input images, or knowledge transfer using meta-embedding or loss functions. These methods require a clear definition of the relationship between samples before they can be used for learning, and cannot achieve dynamic representation learning of sample relationships. The sample interaction module proposed in the present invention establishes a sample relationship modeling mechanism, so that the network can dynamically learn and adjust the relationship between samples at each iteration, thereby improving the model's ability to handle long-tail distribution data. Therefore, the present invention introduces a sample interaction module based on S1 to solve the long-tail visual recognition problem of DR. By combining the sample interaction module, the model can not only better understand and utilize the relationship between different samples, but also effectively deal with the problem of class imbalance in the data set, thereby significantly improving the recognition performance of minority categories.

[0057] like Figure 3 As shown, a process diagram of a sample interaction module is shown.

[0058] In one embodiment, the above step S3: using the high- and low-frequency feature extraction network and the sample relationship interaction module as a student model, optimizing the student model using a representation learning method based on momentum distillation, continuously updating the momentum model through exponential moving average, and then using the pseudo labels generated by the momentum model to train the student model, specifically includes:

[0059] Step S31: Use the exponential moving average method to obtain the momentum model, which is used as the teacher model to provide supervision for the student model. Formula (6) is as follows:

[0060] (6)

[0061] in, express Momentum model parameters at time , represents the student model weight at time t, Used to control the smoothing degree of the exponential moving average;

[0062] Step S32: By minimizing the cross entropy loss function To make the grading results predicted by the student model closely correspond to the actual grading labels; by minimizing the momentum matching loss function To reduce the impact of inaccurate labels on the classification results, formulas (7) to (9) are as follows:

[0063] (7)

[0064] (8)

[0065] (9)

[0066] in, represents the total number of samples; M represents the total number of categories; i represents the i-th sample; is the label of category c in the i-th sample, whose value is 0 or 1; is the predicted probability of the i-th sample category c; Pseudo labels generated for the momentum model; is the momentum distillation loss weight;

[0067] Step S33: Use a linear dynamic weight adjustment strategy to control the weight of the momentum distillation loss. The weight of the momentum distillation starts from a preset minimum value and gradually increases linearly in each training round until it reaches a preset maximum weight value. Formula (10) is as follows:

[0068] (10)

[0069] in, and They are the lower and upper limits of the momentum distillation loss weight, represents the current training round, and E represents the total number of training rounds.

[0070] Figure 4 A schematic diagram showing the momentum distillation learning framework.

[0071] Embodiment 2

[0072] like Figure 5 As shown, an embodiment of the present invention provides a fine-grained lesion feature diabetic retinopathy grading system, including the following modules:

[0073] The high- and low-frequency feature extraction network module 41 is used to input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain a processed feature map. ;

[0074] The sample relationship interaction module 42 is used to transform the feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier ;

[0075] The momentum distillation optimization module 43 is used to use the high- and low-frequency feature extraction network and the sample relationship interaction module as the student model, optimize the student model using a representation learning method based on momentum distillation, continuously update the momentum model through exponential moving average, and then use the pseudo labels generated by the momentum model to train the student model.

Claims

1. A fine-grained lesion feature diabetic retinopathy grading method, characterized in that: include: Step S1: Input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain the processed feature map ; Step S2: The feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier ; Step S3: Use the high- and low-frequency feature extraction network and the sample relationship interaction module as student models, optimize the student model using a representation learning method based on momentum distillation, continuously update the momentum model through exponential moving average, and then use the pseudo labels generated by the momentum model to train the student model.

2. The fine-grained lesion feature diabetic retinopathy grading method according to claim 1, characterized in that: Step S1: Input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and extract the high- and low-frequency features simultaneously through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain a processed feature map. , including: Step S11: construct a high- and low-frequency feature extraction network, which is composed of multiple block embedding modules and high- and low-frequency block stacks; the retinal fundus color image or feature map first passes through the block embedding module, and performs feature processing and downsampling through convolution operations to obtain a downsampled feature map ,in, , C represent feature maps The number of pixels and channels; Step S12: feature map Apply layer normalization to obtain the normalized feature map , then Divide the channels into and , respectively input into the high and low frequency blocks consisting of high frequency mixer and low frequency mixer, where, ; Step S13: Input high frequency mixer, which consists of maximum pooling operation and depth convolution operation. Divided into and , After a maximum pooling layer and a linear layer, After a linear layer and a deep convolutional layer, formulas (1) to (2) are as follows: (1) (2) Among them, DwConv represents the deep convolution operation, and represents the output of the high frequency mixer; Step S14: The low-frequency mixer is input. The low-frequency mixer first uses a pooling strategy to reduce the size of the low-frequency feature map, then uses a multi-head self-attention mechanism for feature processing, and finally performs upsampling to restore the feature map to its original size, as shown in formula (3): (3) in, represents the average pooling operation, represents the multi-head self-attention mechanism, represents upsampling; Step S15: stack the outputs of the high-frequency mixer and the low-frequency mixer in the channel dimension, and use depthwise convolution to implement information exchange between image blocks. Formulas (4) to (5) are as follows: (4) (5) Step S16: The feature map After global average pooling in one spatial dimension, we get the attention feature map , After the fully connected layer and Sigmoid operation, the value is mapped to [0, 1] and finally compared with the original feature map Multiply to get a new feature map ; Step S17: The feature map With feature map Add to get the feature map , used to promote rapid convergence of the model; Step S18: Feature Map After layer normalization, it is sent to the forward feedback layer to combine the newly obtained feature map with the feature map Add together to get the feature map ; Step S19: Repeat steps S11 to S18 to form a structure in which a plurality of the block embedding modules and the high and low frequency blocks are stacked to ensure that the high and low frequency feature extraction network is fully learned and its performance is optimized.

3. The fine-grained lesion feature diabetic retinopathy grading method according to claim 2, characterized in that: Step S2: transform the feature map Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier , including: Step S21: Enter the sample interaction module, first is transposed, and the self-attention mechanism in the Transformer layer is applied to the sample interaction module to achieve cross-attention between different samples, thereby obtaining the feature map ; Step S22: Use the auxiliary classifier and the final classifier to jointly classify Perform detailed classification processing, conduct in-depth analysis of features through the fully connected layer, then apply the Softmax function to calculate the probability distribution, and finally determine the category with the highest probability through the argmax operation, so as to obtain the final grading result of diabetic retinopathy .

4. The fine-grained lesion feature diabetic retinopathy grading method according to claim 3, characterized in that: The step S3: using the high- and low-frequency feature extraction network and the sample relationship interaction module as a student model, optimizing the student model using a representation learning method based on momentum distillation, continuously updating the momentum model through exponential moving average, and then using pseudo labels generated by the momentum model to train the student model, specifically includes: Step S31: Use the exponential moving average method to obtain the momentum model, which is used as the teacher model to provide supervision for the student model. Formula (6) is as follows: (6) in, express Momentum model parameters at time , represents the student model weight at time t, Used to control the smoothing degree of the exponential moving average; Step S32: By minimizing the cross entropy loss function To make the grading results predicted by the student model closely correspond to the actual grading labels; by minimizing the momentum matching loss function To reduce the impact of inaccurate labels on the classification results, formulas (7) to (9) are as follows: (7) (8) (9) in, represents the total number of samples; M represents the total number of categories; i represents the i-th sample; is the label of category c in the i-th sample, whose value is 0 or 1; is the predicted probability of the i-th sample category c; Pseudo labels generated for the momentum model; is the momentum distillation loss weight; Step S33: Use a linear dynamic weight adjustment strategy to control the weight of the momentum distillation loss. The weight of the momentum distillation starts from a preset minimum value and gradually increases linearly in each training round until it reaches a preset maximum weight value. Formula (10) is as follows: (10) in, and They are the lower and upper limits of the momentum distillation loss weight, represents the current training round, and E represents the total number of training rounds.

5. A fine-grained lesion feature diabetic retinopathy grading system, characterized in that: Includes the following modules: The high- and low-frequency feature extraction network module is used to input the diabetic retinopathy fundus color image into the high- and low-frequency feature extraction network, and realize the simultaneous extraction of high- and low-frequency features through the block embedding module and multiple stacking of high- and low-frequency blocks to obtain the processed feature map ; Sample relationship interaction module, used to transform feature maps Input sample relationship interaction module, use Transformer's self-attention mechanism to interact information between different samples, obtain gradients from different virtual samples, so that the tail category can be more fully trained and a new feature map can be obtained , and obtain the classification results through the final classifier ; The momentum distillation optimization module is used to use the high- and low-frequency feature extraction network and the sample relationship interaction module as student models, optimize the student model using a representation learning method based on momentum distillation, continuously update the momentum model through exponential moving average, and then use the pseudo labels generated by the momentum model to train the student model.

Images