Retinopathy detection method fusing multi-focus segmentation and contrast learning

Abstract

The invention provides a retinopathy detection method fusing multi-focus segmentation and comparative learning, and designs a two-stage network by closely fusing focus segmentation and grading tasks. In the lesion segmentation task of the first stage, an improved MCA-UNet network is provided, the feature extraction capability is optimized by introducing a semi-wavelet collaborative attention module, and high-precision lesion spatial feature support is provided for the subsequent grading task. In the DR grading task of the second stage, a double-branch prototype comparison learning model is constructed, and a feature learning branch strengthens intra-class compactness and inter-class separability by constructing a class prototype, so that the problem of class imbalance ubiquitous in a DR data set is effectively relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical detection, in particular to a retinopathy detection method fusing multi-lesion segmentation and contrast learning. BACKGROUND

[0002] Diabetic retinopathy (DR) is the leading cause of visual impairment and blindness worldwide, especially in long-term diabetic patients. With the rising prevalence of diabetes, the frequency of DR cases is also rising, which is a major public health problem. Timely detection and accurate classification of DR are crucial for early intervention and treatment, which can greatly reduce the risk of severe visual impairment. High blood sugar caused by diabetes can cause damage to various tissues in the eye, including microvascular lesions, retinal ischemia, retinal neuron damage, and abnormal vascular structure. Among them, microvascular lesions are the most iconic pathological features of DR from early to medium, and are one of the core bases for clinical diagnosis. Microvascular lesions mainly include microaneurysms (MA), hemorrhages (HE), hard exudates (EX) and soft exudates (SE), and the number, size, distribution range and morphological characteristics of lesions are the key visual clues for ophthalmologists to judge the severity of DR.

[0003] However, in the current DR diagnosis technology practice, the grading based on fundus images still faces many real problems. On the one hand, the clinical diagnosis is highly dependent on the detection results of retinal lesions such as microaneurysms, but the concealment of small lesions often hinders early detection of the disease. If only relying on traditional image analysis methods, it is difficult to accurately capture subtle lesion characteristics to support grading judgments; on the other hand, there is a common class imbalance problem in DR diagnosis related data sets, especially in a few classes such as severe DR and proliferative diabetic retinopathy (PDR), the sample size is significantly insufficient, which not only affects the training effect of the diagnosis model, but also may lead to a decrease in recognition accuracy for a few serious conditions, thereby affecting the overall diagnosis reliability. These problems together restrict the accuracy and stability of DR grading, and become a bottleneck that needs to be broken through in the optimization of clinical diagnosis technology. Therefore, the DR grading task still faces double challenges: one is how to provide reliable basis for grading through accurate lesion segmentation, and the other is how to improve the recognition ability of the model for a few classes in the class imbalance scenario. SUMMARY

[0004] In view of the above problems, the present application provides a retinopathy detection method fusing multi-lesion segmentation and contrast learning, which realizes accurate grading by fusing pixel-level segmentation and image-level classification. The technical scheme provided by the present application is as follows: A retinopathy detection method fusing multi-lesion segmentation and contrast learning, comprising the following steps: Step 1, input fundus image where H is high, W is wide, and 3 represents the RGB channel; Step 2, using an improved multi-scale cross-attention U-shaped network to segment the fundus image, the improved multi-scale cross-attention U-shaped network is obtained based on the expansion of the U-Net architecture, and is composed of an encoder, a multi-scale cross-jump connection and a bidirectional decoder, the improved multi-scale cross-attention U-shaped network introduces a semi-wavelet collaborative attention module to optimize the feature extraction capability; Step 3, using a double-branch prototype contrast model to perform contrast learning on the segmentation map, the double-branch prototype contrast model includes a backbone network and a double-branch network, the backbone network introduces a multi-scale attention module to enhance the recognition of key image regions, and the double-branch network includes a feature learning branch and a classifier learning branch; the training process is realized by minimizing the overall loss function; Step 4, outputting the detection result, including 4 kinds of lesion types and 5 levels of diabetic retinopathy (DR) severity levels, the 4 kinds of lesion types are microaneurysm (MA), hard exudate (EX), soft exudate (SE) and hemorrhage (HE); and 0-4 in the 5 levels of DR severity levels represent negative, mild, moderate, severe and proliferative, respectively.

[0005] Preferably, the semi-wavelet collaborative attention module includes two parallel branches of channel attention and spatial attention, two feature maps of the decoder, up-sampling features and deconvolution features are spliced as query features , features from different scales of the encoder are denoted as , and the encoder feature map and the decoder feature map are regarded as the combination of channel and spatial attention; For the channel attention branch, a global average pooling layer is used for spatial compression to obtain global spatial information, and then the global spatial information is embedded into the vector by transforming the vector as follows: (1) wherein , and are the weights of three linear layers, is a ReLU activation function; the vector is generated for recalibration or excitation , and the dependence relationship at the channel level needs to be encoded, as shown in formula (2): (2) wherein the activation function represents the i-th The importance of each channel; For the spatial attention branch, the input feature map is processed through semi-wavelet attention. , Process and and The number of feature channels is reduced to the same amount. Semi-wavelet attention for input feature map Input features Divided into two parts along the channel dimension: Preserve normal domain features and pass them directly to retain basic context information; Used for wavelet domain feature processing; Wavelet domain feature transformation and processing Perform discrete wavelet transform to obtain wavelet domain features. ;Will Inputting a dual attention unit, the wavelet domain features are weighted through channel attention and spatial attention to obtain weighted wavelet domain features. Inverse transformation of wavelet domain features for weighted wavelet domain features Perform an inverse wavelet transform to reshape it into the same shape as... For the same spatial dimensions, weighted normal domain features are obtained. ;Will and The data is concatenated, and residual features are extracted through convolutional and ReLU layers. After adjusting the channels of the input features through a 1×1 convolution, they are combined with the residual features. Add them together to get the final output features. .

[0006] Preferably, the dual-branch prototype comparison model adopts a fully convolutional network architecture and integrates RepVGG as the backbone network.

[0007] Preferably, the multi-scale attention module processing flow includes three steps: feature grouping, parallel sub-network feature encoding, and cross-space attention aggregation. During the feature grouping stage, for the feature maps output by the backbone network The multi-scale attention module divides it along the channel dimension into Sub-characteristics Each sub-feature is ; In the parallel sub-network feature encoding stage, the multi-scale attention module sets up two parallel branches: a 1x1 branch and a 3x3 branch. The 1x1 branch uses 1x1 convolutional components to model local cross-channel interactions; the 3x3 branch expands the receptive field through 3x3 convolution to enhance the feature capture of small and slender structures. In the cross-spatial attention aggregation stage, the multi-scale attention module performs 2D global average pooling on the output features of the two parallel branches to encode global spatial information. Then, it generates two spatial attention maps through matrix dot product operation and Softmax activation function. The first attention map aggregates spatial information at different scales, while the second attention map retains precise spatial location information. The output of each sub-feature is obtained by aggregating the two attention weight values ​​and calculating the Sigmoid function.

[0008] Preferably, the feature learning branch integrates a projection head layer. This layer maps image features to vector representations in the metric space. Initialize the prototype for each category. prototype The mean value of all sample features within this category is calculated, as shown in equation (4): (4) in It is a category The number of samples in It is a category The Middle Feature vectors of each sample; The feature learning process is supervised by a prototype-supervised contrastive loss function. Dominant, as shown in equation (5): (5) in Indicate category The prototype representation, Represents the current image in the metric space eigenvectors, Indicates the temperature scaling factor. This represents the total number of categories in the dataset; For each mini-batch, the prototype supervision contrast loss is calculated as shown in Equation (6): (6) in, This is the number of samples in each batch.

[0009] Preferably, the classifier learning branch uses the cross-entropy loss method, and uses the encoder weights optimized by the feature learning branch to process the image feature representation, as shown in Equation (7): (7) in, It is the input image. This represents the encoder trained and optimized through the feature learning branch, and the features output by the encoder. After GAP layer processing, it is passed to the fully connected classification layer The classification layer is responsible for mapping the optimized features to a new feature space to generate the logit for classification The specific formula is as follows: (8)

[0010] wherein, represents the log probability, i.e. the confidence prediction of the model for each class; after obtaining the log probability, the softmax function is applied to normalize it to a probability distribution: (9)

[0011] wherein, represents the classification result, and the softmax function ensures that the sum of the probabilities of all classes is 1; finally, the cross-entropy loss is calculated using the probabilities obtained from the softmax function : (10)

[0012] wherein, is the true label, is the predicted probability.

[0013] Preferably, the overall loss function combines the losses from the feature learning and classifier learning branches: (3)

[0014] wherein the weighting coefficient gradually decreases as the number of epochs increases.

[0015] Compared with the prior art, the beneficial effects achieved by the present application are: the segmentation branch innovatively designs a fundus lesion segmentation module, embeds a semi-wavelet collaborative attention mechanism, deeply excavates high-frequency features in the wavelet domain to enhance the precise capture ability of microaneurysms and other microlesions, and relies on a semi-channel selective calculation strategy to effectively reduce redundant parameters of the model and improve inference efficiency; the hierarchical branch designs a prototype contrast learning double-branch model, fuses prototype supervised contrast loss and cross-entropy loss to alleviate class imbalance, and integrates a multi-scale attention module in the backbone network to enhance the discriminative features of lesions of different scales. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation of the present application. In the drawings: Figure 1 is a collation architecture diagram of the model used in the present application; Figure 2 ​​is a half wavelet collaborative attention module structure provided by the present application; Figure 3 is a multi-scale attention module structure provided by the present application; Figure 4 is, wherein a is an RGB image, b is a label image, c is a lesion segmentation, d is a fused RGB image and lesion segmentation, the upper left corner is the result of EX segmentation, the upper right corner is the result of HE segmentation, the lower left corner is the segmentation of MA, and the lower right corner is the segmentation of SE. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] In order to make the above-mentioned purposes, features and effects of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] Embodiment 1: A retinal lesion detection method fusing multi-lesion segmentation and contrast learning, using a new multi-task learning model, simultaneously processing the lesion segmentation auxiliary task while completing the diabetic retinopathy grading main task, the overall architecture of the model is as shown in Figure 1 The method comprises the following steps: Step 1, input fundus image (high H, wide W, 3 channels of RGB).

[0020] Step 2, implement high-precision segmentation of DR lesions, especially small-size and low-contrast lesions such as early microaneurysms, by using an improved multi-scale cross-attention U-shaped network (MCA-UNet). The key optimization is to introduce a half wavelet collaborative attention (HWCA) module, which not only enhances the model's ability to capture high-frequency detailed features, but also reduces computational redundancy, effectively solving the limitations of the original MCA-UNet in DR lesion segmentation.

[0021] MCA-UNet is based on the extension of U-Net architecture, which is composed of three parts: encoder, multi-scale cross-attention and bidirectional decoder, wherein the bidirectional decoder contains three parts: double up-sampling, cross-cooperative attention and deep supervision. However, the cross-cooperative attention mechanism of the existing MCA-UNet mainly focuses on the interaction of normal domain features, and does not fully exploit the high-frequency detail information contained in the wavelet domain, such as the edge texture of small lesions and the structural features of low-contrast regions, resulting in limited ability to capture blurred or small lesions such as early microaneurysms in fundus images; at the same time, the participation of full-channel features in attention calculation easily introduces redundant information, increasing the computational complexity of the network, and the inference efficiency needs to be improved in high-resolution fundus image processing.

[0022] In view of the above problems, the present application designs a HWCA module to optimize the feature extraction capability of MCA-UNet. The core improvement of this module is reflected in two aspects: first, by introducing a wavelet domain feature processing mechanism, the high-frequency detail features of the lesion area are captured, and the ability to distinguish small lesions such as microaneurysms and punctate hemorrhage is strengthened; second, a half-channel selective operation strategy is adopted, and only half of the key channel features are subjected to wavelet transform and attention calculation, which effectively reduces the computational redundancy while enriching the feature diversity, and improves the network operation efficiency.

[0023] The half-wavelet cooperative attention module contains two parallel branches of channel attention and spatial attention, which operate simultaneously to efficiently extract complementary information, such as Figure 2 As shown in the figure, the two feature maps of the decoder, the up-sampled feature and the deconvolution feature are spliced as query features . The feature maps from different scales of the encoder represent the encoder levels, and for the first layer encoder, each pair of feature maps is input into the half-wavelet cooperative attention module.

[0024] The encoder feature map and the decoder feature map are regarded as the combination of channel and spatial attention, and for the channel level, the global average pooling layer is used for spatial compression. This operation embeds global spatial information into the vector . The transformation relationship of this vector is shown in equation (1): (1)

[0025] where , and are the weights of the three linear layers, is the ReLU activation function.

[0026] The vector for recalibration or excitation needs to encode the dependencies at the channel level, as shown in equation (2): (2)

[0027] where the activation function represents the importance of the th channel, and these importances are rescaled.

[0028] For the spatial level, the input feature map is processed by the HWAB and the number of feature channels of and are brought to the same number . The HWAB takes the input feature map and splits the input feature into two parts along the channel dimension: The normal domain features are retained and directly passed to retain the basic context information. The wavelet domain features are processed. The purpose of splitting is to reduce the computational complexity, and only half of the features are involved in the complex wavelet transform and attention operation. Wavelet domain feature conversion and processing

[0029] perform a discrete wavelet transform (DWT) on to obtain the wavelet domain feature . DWT decomposes the feature into low-frequency and high-frequency components, enhancing the ability to capture image details. The is input into the dual attention unit (DAU), which weights the wavelet domain feature through channel attention and spatial attention to obtain the weighted wavelet domain feature , highlighting important features. The inverse wavelet transform (IWT) is performed on the weighted wavelet domain feature to reshape it to the same spatial size as , obtaining the weighted normal domain feature . Feature fusion and output concatenate and , and extract residual features through convolution and ReLU layers. A shortcut connection is introduced: the input feature is adjusted in channel through 1x1 convolution, and then added to the residual feature to obtain the final output feature , which integrates the attention features of the normal domain and the wavelet domain. Finally, the features from the two attention modules are aggregated, and the final output feature is represented by concatenating all the features.

[0030] Step 3, contrastive learning is performed on the segmentation map, the present application improves the traditional prototype contrastive learning model to further improve the learning ability of unbalanced samples in the diabetic retinopathy classification task. As shown in Figure 1 The framework contains two core structures: one is the backbone network, which focuses on identifying key lesion areas in fundus images; the other is the dual-branch network, in which the feature learning branch is constructed based on prototype supervised contrastive learning, and the classifier learning branch is optimized using the cross-entropy loss function. In the encoder of the backbone network, a multi-scale attention (EMA) module is introduced to enhance the identification of key image regions, and attention weights across spatial and channel dimensions are derived for fine learning of lesion region features. The feature learning branch uses supervised prototype contrastive learning in the feature space to achieve intra-class compactness and inter-class separability, making the representation of DR images more accurate at different stages. The classifier learning branch uses high-quality, multi-stage lesion features provided by the feature learning branch to improve classification accuracy and reduce classification bias caused by sample distribution imbalance. During training, feature learning plays a dominant role in the early stages of training, and then gradually gives way to classifier learning during the training process. In the entire framework, feature learning and classifier learning branches work closely together to minimize the overall loss function through a coordinated training process, as shown in equation (3): (3)

[0031] wherein, represents the overall loss function of the framework, combining the losses from the feature learning and classifier learning branches. The loss associated with prototype supervised contrastive learning in the feature learning branch. Cross Entropy loss refers to the cross-entropy loss in the classifier learning branch. The weighting coefficient decreases gradually as the number of epochs increases.

[0032] In the proposed framework, dual-branch prototype contrastive learning uses a fully convolutional network (FCN) architecture, which integrates RepVGG as the backbone. Specifically, the RepVGG architecture is used to process each input image, and its fully connected layers are replaced by convolutional layers to adapt to the requirements of FCN. This modification enables the network to output spatially dense predictions suitable for the task. The resulting image representation is denoted as , which is located in a high-dimensional space wherein and are the dimensions derived from the RepVGG output.

[0033] Although FCN architectures exhibit good robustness, they can have certain limitations in identifying elongated and complex structures. To this end, the present application introduces an EMA module and integrates it into the backbone network, extracts multi-scale features from micro to macro through multi-scale parallel branches, and then assigns dynamic weights to different scale feature maps - with semantic importance as the core basis to integrate information, such as assigning higher weights to large-area bleeding features that have strong indications for severe DR, and enhancing attention to microaneurysm cluster features that are critical for mild DR. This module can effectively establish short-range and long-range feature dependency relationships while avoiding information loss caused by channel dimension reduction, accurately derive cross-space and channel attention weights, and thus highlight the feature representation of key lesion areas, providing more discriminative image feature support for subsequent feature learning and classification tasks.

[0034] The multi-scale attention EMA module is derived from the improvement of the side effects of dimension reduction and the limitations of single-scale features in existing attention mechanisms. It realizes efficient feature enhancement through parallel substructure design and cross-space information aggregation. As shown in Figure 3 , the overall processing flow of EMA mainly includes three steps: feature grouping, parallel subnetwork feature encoding, and cross-space attention aggregation.

[0035] In the feature grouping stage, for the feature map output by the backbone network, EMA divides it into sub-features along the channel dimension, each sub-feature being to realize independent learning and distribution optimization of different semantic features.

[0036] In the parallel subnetwork feature encoding stage, to capture multi-scale spatial structure information, EMA sets two parallel branches: 1x1 branch and 3x3 branch. The 1x1 branch uses a 1x1 convolution component to model local cross-channel interactions; the 3x3 branch uses a 3x3 convolution to expand the receptive field and enhance the feature capture of small and elongated structures. At the same time, EMA reshapes the grouped channel dimension to the batch dimension, avoiding channel dimension reduction operations in conventional convolutions, preserving complete channel information and ensuring the richness of feature description.

[0037] In the cross-space attention aggregation stage, EMA performs 2D global average pooling on the output features of the two parallel branches to encode global spatial information, and then generates two spatial attention maps through matrix dot product operation and Softmax activation function. Among them, the first attention map aggregates spatial information of different scales, and the second attention map preserves accurate spatial position information. Finally, the output of each sub-feature is calculated by aggregating the two attention weight values and the Sigmoid function, realizing the capture of pixel-level pair relationship and the highlighting of global context, and outputting enhanced features with the same size as the input feature map, providing accurate support for subsequent DR lesion feature learning.

[0038] The feature learning branch of the dual-branch prototype contrast learning uses a prototype-based supervised contrast learning method to improve learning efficiency and data processing. This method particularly addresses the challenges posed by sample imbalance and high-dimensional data sets. Inspired by prototype supervised contrast learning, this branch integrates a projection head layer that can map image features to vector representations in a metric space . To effectively represent the centroids of different lesion severity levels in the feature space, the prototypes for each class need to be initialized. The prototypes for class are obtained by calculating the mean of all sample features within that class, as shown in equation (4): (4)

[0039] where is the number of samples in class , is the feature vector of the th sample in class . These prototypes serve as centroids in the feature space, encouraging samples from the same class to cluster around them.

[0040] The feature learning process is dominated by the prototype supervised contrast loss function , which aims to minimize the distance between each sample and its corresponding class prototype while maximizing the distance between the sample and other class prototypes. As shown in equation (5): (5)

[0041] where denotes the prototype representation of class , denotes the feature vector of the current image in the metric space, denotes the temperature scaling factor, is the total number of classes in the dataset. This loss function effectively enhances the similarity of intra-class samples and the difference between inter-class samples.

[0042] The calculation of the prototype supervised contrastive loss for each mini-batch is shown in equation (6): (6)

[0043] wherein, is the number of samples in each batch. By minimizing on mini-batches, the encoder weights are constantly optimized, improving the model's ability to distinguish between different lesion classes.

[0044] This method only calculates the distance between samples and prototypes, without the need to retain all samples in memory, thus significantly reducing memory usage and computational overhead. Its advantages are reflected in two aspects: on the one hand, by ensuring that all classes are equally represented during the learning process, the problem of sample imbalance is alleviated; on the other hand, the learning process is simplified, making the model more robust to changes in data volume and data dimension. Through the above method, the feature learning branch of the dual-branch prototype contrastive learning not only improves the classification accuracy, but also effectively adapts to complex patterns in the data, thereby enhancing the model's sensitivity and robustness to different levels of lesions.

[0045] The classifier learning branch of the dual-branch prototype contrastive learning uses the cross-entropy loss method, whose core goal is to optimize the class-specific features using the updated prototypes from the feature learning branch. Subsequently, these optimized features will be used to adjust the encoder, a process that not only corrects the weights of the classifier learning branch, but also effectively reduces classification bias, thereby improving classification accuracy.

[0046] In the proposed model, the classifier branch processes the image feature representation using the encoder weights optimized by the feature learning branch to extract more detailed features, as shown in equation (7): (7)

[0047] wherein, is the input image, represents the encoder trained and optimized by the feature learning branch. This process ensures that the features extracted from the original data can best reflect the information useful for the classification task.

[0048] The features output by the encoder are processed by the GAP layer and passed to the fully connected classification layer , which is responsible for mapping the optimized features to a new feature space to generate the logit used for classification, as shown in equation (8): (8)

[0049] wherein, This represents the logarithmic probability, which is the model's confidence prediction for each class. This step is crucial in the classification process because it directly affects the generation of the final classification result.

[0050] After obtaining the logarithmic probability, apply The function normalizes it into a probability distribution: (9)

[0051] in, Representing the classification result, the softmax function ensures that the sum of the probabilities of all classes is 1, thus allowing the model's output to be interpreted as a probability estimate for each class. Finally, utilizing from... Calculate cross-entropy loss based on the probability obtained from the function. : (10)

[0052] in, It's a real label. This refers to the predicted probability. The loss function measures the deviation between the predicted probability distribution and the actual label distribution. By optimizing this loss function, the fully connected classification layer can be further tuned. The weights are adjusted to reduce prediction errors.

[0053] Step 4: The model outputs the detection results, including four lesion types and a five-level severity rating of diabetic retinopathy (DR). The four lesion types are microaneurysms (MA), hard exudates (EX), soft exudates (SE), and hemorrhage (HE). In the five-level DR rating, 0-4 represent negative, mild, moderate, severe, and proliferative, respectively.

[0054] Example 2: Experiments were conducted on the IDRiD and DDR public datasets for lesion segmentation and DR grading, and compared with other network models. The DDR dataset is a general, high-quality dataset comprising three sub-datasets: DR grading, lesion detection, and lesion segmentation. The DR grading task provides 12,522 color fundus images: 6,260 for training, 2,503 for validation, and 3,759 for testing. This dataset contains some low-quality images with degradation factors such as uneven illumination, underexposure, and overexposure. The segmentation task provides 757 fundus images with pixel-level annotations for EX, MA, and HE. The IDRiD dataset, released at the 2018 International Biomedical Imaging (ISBI) Challenge Workshop, provides data and labels for three sub-challenges: lesion segmentation, disease grading, and lesion detection. The DR grading task provides 516 color fundus images, with 413 for training and 103 for testing. The segmentation task provides 81 fundus images with pixel-level annotations.

[0055] The experiment is based on Python 3.8.13 and PyTorch 1.12.0 framework, and the training environment is configured as 4 NVIDIA GeForce GTX 1080Ti GPUs (single card memory 11GB) to meet the computing needs of large-scale neural networks. In the training process of IDRiD and DDR datasets, the batch size is set to 16 to balance the computational load and training efficiency. The resolution of the input image is uniformly adjusted to 640x640, which can ensure sufficient extraction of detailed features by the model and maintain the computational practicability of the training process. To enhance the generalization ability and robustness of the model, data augmentation techniques are used in the training, including random horizontal flipping, random vertical flipping, and random rotation in the range of -90° to 90°. The Adam optimizer is selected for the model, and the total training epoch is 300. The initial learning rate is set to 5e-3, and the cosine annealing strategy is used to dynamically adjust it during the training process to optimize the learning efficiency and improve the stability of model convergence.

[0056] The embodiment quantitatively compares the lesion segmentation performance with other latest methods on IDRiD and DDR public datasets. The segmentation comparison results on the two datasets are shown in Tables 1 and 2.

[0057] Table 1 Performance comparison with other models on IDRiD dataset (the best result is marked in bold)

[0058] Table 1 shows the quantitative comparison results on the IDRiD dataset. The compared methods include 5 current popular deep learning models. From the data, the AUPR values of the invented method for EX, HE, MA, and SE lesion segmentation are 0.559, 0.688, 0.882, and 0.750, respectively: The performance is the best in EX and MA lesion segmentation tasks, and the AUPR value of MA is higher than that of other comparison models, which benefits from the model's ability to accurately extract high-frequency features in the wavelet domain and capture the edge details of MA, a kind of small lesion; In HE and SE lesion segmentation tasks, it ranks second, with a difference of only 1 percentage point from the optimal model, and the overall segmentation effect is better than most of the comparison models.

[0059] To more intuitively show the performance of the segmentation network, the model provides qualitative segmentation results on the DDR dataset. The segmentation results of MA, HE, EX, and SE and the comparison with the labels are as follows: Figure 4The segmentation results of different types of lesions are highly consistent with the labels, and the model shows precise segmentation ability for the characteristics of each type of lesion: for MA with small diameter and small gray scale difference with the surrounding tissue, the segmentation result can accurately locate each tiny lesion without obvious omission or missegmentation, which benefits from the model's mining of wavelet domain high-frequency features, effectively capturing the edge texture details of MA; for HE, whose boundary is easily confused with blood vessels, the segmentation result can clearly distinguish the blood region from the normal blood vessels, avoiding background interference, which confirms the sensitive capture ability of the gray scale mutation feature; for EX with wide distribution range and irregular shape, the segmentation area completely covers the lesion range without edge fracture, which reflects the advantage of focusing on the core features of large-scale lesions; for SE with low contrast and fuzzy boundary, the segmentation result can still accurately outline its diffusion shape, further verifying the model's feature extraction ability for low-contrast lesions. It is worth noting that: in this paper, the segmented multiple lesion information is finally merged into the original image for data preprocessing, making the lesions in the retinal image more obvious, thereby facilitating the subsequent DR classification.

[0060] Table 2 Ablation experiment verification results of MCA-Unet on DDR dataset

[0061] Based on the DDR dataset, the MCA-Unet model was subjected to systematic ablation experiment, focusing on verifying the performance optimization effect of the semi-wavelet collaborative attention module integrated into the model. Table 2 compares the performance of the model with and without the integration of the HWCA module. Compared with the baseline model without the integration of HWCA, the HWCA module has a clear optimization effect on the segmentation performance of MCA-Unet model: based on the baseline model without the integration of HWCA, after adding the HWCA module, the AUPR of EX is improved by 0.032, the HE is improved by 0.010, the MA is improved by 0.016, and the SE is improved by 0.028, the comprehensive index mAUC is improved by 0.038, the mIOU is improved by 0.0123, and the mDice is improved by 0.0219. This highlights the importance of the HWCA module in complex fundus lesion segmentation. By mining the high-frequency features in the wavelet domain, it can accurately capture the detailed information of tiny lesions such as MA, and at the same time, by means of selective operation, it can filter out the background interference of normal fundus tissue, effectively improving the segmentation accuracy of lesions such as EX and SE. Due to the characteristics of the DDR dataset, such as multiple types of lesions and strong background interference, the optimization effect of the HWCA module for different lesion segmentation pain points is particularly critical, directly promoting the model to achieve stable improvement in single lesion and comprehensive index, fully verifying the necessity of the module for MCA-Unet model.

[0062] This embodiment compares the DR classification performance of the DDR and IDRiD datasets with other popular network models, and the experimental results are shown in Tables 3 and 4.

[0063] Table 3 Performance comparison with other models on DDR dataset (best results are marked in bold)

[0064] Table 3 presents the performance of each model on the DDR dataset. According to the data in the table, the proposed model performs well in core evaluation indicators: the accuracy is 0.907, the precision is 0.702, and the F1 score is 0.664. Compared with the suboptimal model DR-Net (ACC=0.905, Precision=0.651, F1=0.643), the proposed model's precision is improved by 5.1 percentage points, and the F1 score is improved by 2.1 percentage points, which verifies that the proposed model has better recognition ability for minority class lesions in the class imbalance scenario. In addition, compared with the classic Transformer model, the F1 score of the proposed model is improved by 9.9 percentage points and 10.9 percentage points, respectively, due to the proposed model's ability to more accurately capture the multi-scale features of key lesions such as microaneurysms and hard exudates, avoiding the problem of insufficient attention to local subtle lesions in traditional Transformer models.

[0065] Table 4 Ablation experiment of prototype contrast learning

[0066] This embodiment conducts an ablation experiment on the double-branch prototype contrast model. The role of the double-branch structure and the multi-scale attention module in the unbalanced sample learning scenario is verified. The experimental results are shown in Table 4. The model relies only on the FCN architecture of the RepVGG backbone network, and the performance is relatively limited; when the double-branch structure is enabled, the accuracy is improved by 0.011, and the precision and F1 score are increased by 0.078 and 0.067, respectively, verifying the effectiveness of the feature learning branch in achieving intra-class compactness and inter-class separability through prototype supervised contrast learning, enhancing unbalanced sample learning, and making the feature representation of different DR stages more discriminative; when the double-branch structure and the EMA module are enabled at the same time, the accuracy is further improved by 0.015, and the precision and F1 score are increased to 0.048 and 0.039, respectively, fully proving that the EMA module effectively enhances the feature response of key lesion areas. The double-branch structure is the basic architecture for improving DR classification performance, and the loss coordination mechanism solves the classification bias caused by sample imbalance; the EMA module is an enhancement component of the backbone network, which further releases the potential of the model by precisely focusing on key lesion features; the organic combination of the two achieves dual optimization of feature discriminability and classification robustness in the DR classification task.

[0067] The above merely describes the preferred embodiments of the present application and is not used to limit the present application, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or equivalently replace some technical features thereof. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for retinopathy detection by fusing multi-focus segmentation and contrastive learning, characterized in that, The method comprises the following steps: Step 1, inputting a fundus image where H is high, W is wide, and 3 represents RGB channels. Step 2, performing lesion segmentation on the fundus image using an improved multi-scale cross-attention U-shaped network, wherein the improved multi-scale cross-attention U-shaped network is based on a U-Net architecture expansion and comprises an encoder, multi-scale cross-jump connection and a bidirectional decoder, and the improved multi-scale cross-attention U-shaped network introduces a semi-wavelet collaborative attention module to optimize feature extraction capability; Step 3, performing contrastive learning on the segmentation image using a double-branch prototype contrast model, wherein the double-branch prototype contrast model comprises a backbone network and a double-branch network, the backbone network introduces a multi-scale attention module to enhance the recognition of key image regions, and the double-branch network comprises a feature learning branch and a classifier learning branch; The training process is realized by minimizing the overall loss function; Step 4, outputting the detection result, including four types of lesions and five levels of diabetic retinopathy (DR) severity, and the four types of lesions are microaneurysm (MA), hard exudate (EX), soft exudate (SE) and hemorrhage (HE); In the five levels of DR, 0-4 represent negative, mild, moderate, severe and proliferative, respectively.

2. The method of claim 1, wherein the method further comprises: The half-wavelet cooperative attention module contains two parallel branches of channel attention and spatial attention, two feature maps of the decoder, up-sampling features and deconvolution features After splicing as query features , the feature maps from different scales of the encoder are denoted as , the encoder feature maps are denoted as , and the decoder feature maps are denoted as are regarded as the combination of channel and spatial attention; For the channel attention branch, a global average pooling layer is used For spatial compression to obtain global spatial information, the vector is then transformed to embed the global spatial information into the vector by the following operation as shown in equation (1): （1） where , and are the weights of the three linear layers, is the ReLU activation function; Generating vectors for recalibration or excitation of a microphone array The channel-wise dependencies need to be encoded as in equation (2): （2） where the activation function represents the importance of the th channel; For the spatial attention branch, the input feature map is processed through semi-wavelet attention. , Process and and The number of feature channels is reduced to the same amount. Semi-wavelet attention for input feature maps Input features Divided into two parts along the channel dimension: Preserve normal domain features and pass them directly to retain basic context information; Used for wavelet domain feature processing; Wavelet domain feature conversion and processing pair Performing discrete wavelet transform to obtain wavelet domain features ; converting Inputting into a dual attention unit, weighting the wavelet domain features through channel attention and spatial attention to obtain weighted wavelet domain features , ; inversely converting the wavelet domain features on the weighted wavelet domain features Performing inverse wavelet transform to reshape it into the same spatial size as to obtain weighted normal domain features ; concatenating with to extract residual features through a convolution layer and a ReLU layer , adjusting the channels of the input features through a 1x1 convolution, and adding the residual features to obtain final output features .

3. The method of claim 1, wherein the method further comprises: The double-branch prototype contrast model adopts a full convolutional network architecture and integrates RepVGG as the backbone network.

4. The method of claim 3, wherein the method further comprises: The multi-scale attention module processing process comprises three steps of feature grouping, parallel sub-network feature coding and cross-space attention aggregation: The feature grouping stage is applied to the feature maps output by the backbone network The multi-scale attention module divides it into sub-features along the channel dimension Each sub-feature is ; In the parallel sub-network feature coding stage, the multi-scale attention module sets two parallel branches: a 1x1 branch and a 3x3 branch, the 1x1 branch uses a 1x1 convolution component to model local cross-channel interaction, and the 3x3 branch expands the receptive field through a 3x3 convolution to enhance the feature capture of small and slender structures; In the cross-space attention aggregation stage, the multi-scale attention module encodes global spatial information by performing 2D global average pooling on the output features of the two parallel branches, and then generates two spatial attention maps through matrix point multiplication operation and Softmax activation function, the first attention map aggregates spatial information of different scales, and the second attention map retains accurate spatial position information; the output of each sub-feature is obtained by aggregating the two attention weight values and Sigmoid function calculation.

5. The method of claim 4, wherein the method further comprises: The feature learning branch integrates a projection head layer that maps image features to vector representations in a metric space ; prototypes for each class are initialized, classes prototypes are obtained by computing the mean of all sample features within that class, as shown in equation (4): （4） wherein is the number of samples in the class , is the feature vector of the th sample in the class . The feature learning process is supervised by a prototype contrastive loss function The dominant, as shown in equation (5): （5） wherein represents a class prototype representation, represents a feature vector of a current image in the metric space, represents a temperature scaling factor, is the total number of classes in the dataset; For each mini-batch, the calculation method of the prototype supervised contrast loss is shown in formula (6): （6） wherein, is the number of samples per batch.

6. The method of claim 5, wherein the method further comprises: The classifier learning branch adopts a cross-entropy loss method, and the image feature representation is processed by using the encoder weight optimized by the feature learning branch, and the specific process is shown in formula (7): （7） wherein, is an input image, denotes an encoder trained and optimized through a feature learning branch, the features output by the encoder are passed to a fully connected classification layer after being processed by a GAP layer which is responsible for mapping the optimized features to a new feature space, generating logit for classification The specific formula is as follows: （8） where, represents the log probabilities, i.e. the model's confidence prediction for each class; after obtaining the log probabilities, the function is applied to normalize them to a probability distribution: （9） where, represents the classification result, the softmax function ensures that the sum of probabilities of all classes is 1; finally, the cross-entropy loss is calculated using the probabilities obtained from :​ （10） wherein, is the true label, is the predicted probability.

7. The method of claim 5, wherein the method further comprises: The overall loss function combines the losses from the feature learning and classifier learning branches: （3） where the weighting coefficients decrease gradually as the number of epochs increases.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps in the retinopathy detection method fusing multi-lesion segmentation and contrast learning according to any one of claims 1-7.

9. A computer device comprising a processor, a memory and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps in the retinopathy detection method fusing multi-lesion segmentation and contrast learning according to any one of claims 1-7.

Images