A multi-instance diabetic retinopathy grading method based on a dual attention mechanism

Abstract

The application discloses a diabetic retinopathy grading method based on multi-instance learning and attention feature modeling, and belongs to the technical field of medical image analysis. The method comprises the following steps: acquiring fundus image data and constructing a training sample set and a test sample set; dividing a single fundus image into multiple image instances to form an instance set; performing feature extraction on each image instance, and weighting and modeling the channel information and spatial saliency of the instance features through an attention mechanism; adaptively aggregating the instance set to obtain image-level feature representation; completing diabetic retinopathy grading prediction based on the image-level feature, and iteratively optimizing the model parameters. The application can effectively highlight the key lesion area features under weak annotation conditions, reduce irrelevant instance interference, and improve the accuracy, stability and generalization ability of diabetic retinopathy grading.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence-assisted diagnosis, specifically to a method for grading diabetic retinopathy based on multi-instance learning and attention feature modeling. Background Technology

[0002] With the accelerating aging of the global population and changes in lifestyle, the number of people with diabetes continues to grow, currently exceeding 589 million worldwide. Diabetic retinopathy is one of the most common microvascular complications of diabetes and a significant cause of vision impairment and even blindness in adults. Clinically, fundus imaging is commonly used to screen diabetic patients for retinal abnormalities, and the severity of diabetic retinopathy is graded based on the characteristics of the lesions in the fundus images.

[0003] To improve screening efficiency and reduce the workload of manual image interpretation, various automatic grading methods for diabetic retinopathy based on computer vision and deep learning have been proposed in the existing technology. Among them, methods based on convolutional neural networks achieve automatic discrimination of diabetic retinopathy grades by extracting and classifying features from fundus images; some methods further divide a single fundus image into multiple local regions or image instances, and complete the grade prediction based on the set of instances.

[0004] However, diabetic retinopathy typically presents as localized, small-scale, and scattered lesions in fundus images. Some lesions are subtle and difficult to detect, making accurate identification and annotation challenging. With only image-level annotation, after dividing an image into multiple local instances, some instances may not contain valid lesion information. These irrelevant instances can interfere with feature aggregation and grading decisions, affecting the accuracy and stability of the grading results. Furthermore, existing technologies lack effective differentiation in handling the importance of different instances and feature dimensions, making it difficult to simultaneously consider both local lesion details and overall structural information, thus limiting the grading performance and generalization ability of diabetic retinopathy. Summary of the Invention

[0005] To address the problems existing in the prior art, this disclosure provides an automatic grading method for diabetic retinopathy based on bi-branch attention and multi-instance learning. This method can effectively jointly model local discriminative information and global semantic features under weak supervision, thereby improving the accuracy and stability of the grading results.

[0006] The method of the present invention includes the following steps:

[0007] Step S1: Acquire fundus image data and construct training and testing sample sets. All images are uniformly resized to 384×384 pixels and pixel values ​​are normalized. Data augmentation operations, including rotation, horizontal flipping, and vertical flipping, are performed on the images in the training sample set. The augmented images are then input together with the original images into the subsequent feature extraction branch.

[0008] Step S2: Divide each preprocessed image into multiple fixed-size instance blocks to form an instance set. Instance partitioning uses a grid or sliding window method, with each instance covering a local area of ​​the image. The instance blocks are uniformly adjusted to a specified size, and the standardized pixel values ​​are used as input for the multi-instance learning branch.

[0009] Step S3: Input the instance set into the feature extraction branch and extract the feature vector of each instance using a visual Transformer or convolutional network. Then, input the instance features into the dual attention module to calculate channel attention and spatial attention. Spatial attention generates weights through convolution operations, and channel attention generates weights through fully connected layers. The two types of weights are then weighted and fused with the instance features to output an enhanced instance feature vector.

[0010] Step S4: Input the enhanced instance feature vector into the adaptive aggregation module, calculate the weight of each instance by using the Soft Top-k adaptive aggregation mechanism, perform weighted processing on key instances, and sum or average the weighted instance features to obtain the image-level feature representation, which is output as the high-dimensional feature vector of the entire image.

[0011] Step S5: Input the image-level feature vector into the multilayer perceptron classifier to integrate the high-dimensional features and map them to the category space, outputting a diabetic retinopathy grading result based on multi-instance aggregation. Simultaneously, the global label vector CLS introduced in the feature extraction stage is used for grading prediction by the global classification head to obtain a prediction result based on global semantic features. The multi-instance branch prediction result is fused with the global branch prediction result to obtain the final diabetic retinopathy grading result. During the training phase, the gradient descent algorithm is used to optimize the model parameters of the classifier and feature extraction branches end-to-end, completing model training.

[0012] Through the above steps, the method disclosed in this invention can highlight key lesion areas under weak supervision and reduce interference from irrelevant instances; the dual-branch attention mechanism enhances the representation ability of local fine-grained features while maintaining global semantic information; the Soft Top-k adaptive aggregation mechanism enables adaptive selection of instance-level discriminative information, significantly improving the accuracy, stability and generalization ability of diabetic retinopathy grading; the overall method takes into account both local and global features, achieving efficient, reliable and interpretable DR grading diagnosis. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of the overall framework structure of the multi-instance attention fusion network proposed in this embodiment of the invention;

[0015] Figure 3 This is a schematic diagram of the internal structure of the dual attention module in an embodiment of the present invention;

[0016] Figure 4 This is a schematic diagram of the structure of the Soft Top-k multi-instance aggregation module in an embodiment of the present invention, which is used to perform adaptive weighted fusion of instance features;

[0017] Figure 5 This is a schematic diagram illustrating the performance comparison of models on the APTOS2019 dataset in an embodiment of the present invention;

[0018] Figure 6 This is a schematic diagram showing the performance comparison of the models on the RFMiD2020 dataset in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and beneficial effects of the embodiments of the present invention clearer, the specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of them. Various modifications, substitutions, or equivalent transformations made by those skilled in the art based on the disclosure of the present invention without departing from the spirit and substance of the present invention should fall within the protection scope of the present invention. Furthermore, the technical features involved in each embodiment can be combined with each other without conflict.

[0020] Figure 1 This is a basic flowchart illustrating a multi-instance attention fusion method for grading diabetic retinopathy provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes the following steps:

[0021] Step S1: Acquire fundus image data and construct training and testing sample sets. The acquired fundus image data is organized and labeled, and divided into training and testing sample sets according to a preset ratio. All input images are uniformly adjusted to 384×384 pixels, and pixel values ​​are normalized to eliminate differences in brightness and contrast between different images. During the training phase, data augmentation operations are performed on the images in the training sample set, including random rotation, horizontal flipping, and vertical flipping, to expand the sample size and improve the model's generalization ability. The augmented images and the original images are used together as input for subsequent feature extraction branches.

[0022] Step S2: Divide the preprocessed image into multiple instance blocks to form an instance set. Divide each preprocessed image into a grid of fixed size or use a sliding window method for local cropping, ensuring that each instance block covers a local area of ​​the image. The instance blocks are uniformly adjusted to a specified size and standardized to form the input instance set for the multi-instance learning branch. Through this method, the original image is transformed from a single overall sample into an instance bag structure composed of multiple local instances, providing a foundation for subsequent multi-instance feature modeling.

[0023] Step S3: Extract features from the instance set and perform dual attention enhancement. The instance set obtained in Step S2 is input into the feature extraction branch, which uses a visual Transformer network or a convolutional neural network to extract the corresponding feature vector representation for each instance block. During feature extraction, a global label vector (CLS) is introduced to represent the global semantic information of the entire image. Subsequently, the feature vectors of each instance are input into the dual attention module for feature enhancement. The dual attention module includes a spatial attention submodule and a channel attention submodule, wherein: spatial attention generates a spatial weight map through convolution operations, weighting the instance features in the spatial dimension; channel attention generates channel weights through fully connected layers, recalibrating the instance features in the channel dimension; the spatial attention weights and channel attention weights are then weighted and fused with the instance features respectively to obtain the enhanced instance feature vector.

[0024] Step S4: Adaptively aggregate the enhanced instance features to generate an image-level feature representation. The enhanced instance feature vector obtained in Step S3 is input into the adaptive aggregation module. This module employs a Soft Top-k adaptive aggregation mechanism, assigning weights based on the importance of each instance. First, attention scores are calculated for all instances, and key instances are selected for weighting according to the Soft Top-k mechanism, giving higher weights to key regions with lesion features. Then, the weighted instance features are summed or averaged to obtain an image-level feature representation, which is output as the high-dimensional semantic feature vector of the entire image.

[0025] Step S5: Perform classification prediction and fuse multi-branch results. The image-level feature vector obtained in Step S4 is input into a multilayer perceptron classifier to integrate high-dimensional features and map them to the class space, obtaining a hierarchical prediction result based on multi-instance aggregation. Simultaneously, the global label vector CLS obtained in the feature extraction stage of Step S3 is input into the global classification head to obtain a prediction result based on global semantic features. The multi-instance branch prediction result is fused with the global branch prediction result to obtain the final diabetic retinopathy grading result. During the training phase, the gradient descent algorithm is used to perform end-to-end optimization of the feature extraction branch, adaptive aggregation module, and classifier parameters to complete model training.

[0026] Figure 2 This is a schematic diagram of a multi-instance attention fusion network for grading diabetic retinopathy provided in an embodiment of the present invention. It illustrates the overall processing flow from fundus image input to final grading result output, including key modules such as image preprocessing and sample construction, multi-instance segmentation and feature extraction, dual attention enhancement, adaptive aggregation, and multilayer perceptron classifier prediction. It intuitively demonstrates the data flow and information interaction relationship between the modules, providing a reference for understanding the technical solution of the present invention.

[0027] In one embodiment, step S1 specifically includes the following operations:

[0028] Step S11: Obtain the labeled fundus color image dataset. Let the original image be: ,in, and These represent the image height and width, respectively, with 3 indicating the RGB three channels. Each image is accompanied by a label indicating the level of diabetic retinopathy. For subsequent supervised training;

[0029] Step S12: Normalize the size of all images:

[0030] ,

[0031] Ensure that the images input to the network are of consistent size to facilitate batch processing and instance partitioning;

[0032] Step S13: Perform channel normalization on the adjusted image:

[0033] ,

[0034] in, The mean and standard deviation of the RGB channels in the dataset are shown below; the normalized image is also shown. Used as input for feature extraction networks;

[0035] Step S14: Data Augmentation. During the training phase, to improve the model's robustness to changes such as image rotation, horizontal or vertical flipping, random data augmentation is performed on each training image. Specifically, this includes randomly rotating the image. The images are then randomly flipped horizontally or vertically. The enhanced images, along with the original images, form the training set, which is used to train the network, thereby expanding the sample size and improving the model's generalization ability.

[0036] In one embodiment, step S2 specifically includes the following operations:

[0037] Step S21: Divide the image into instance blocks. The preprocessed fundus image obtained in step S1... Evenly divided into A set of fixed-size image blocks, each image block being an image instance:

[0038] ;

[0039] The instance division method can adopt grid division or sliding window method, so that each instance covers different local areas of the image, so as to fully preserve the local structural features of potential lesion areas;

[0040] Step S22: Instance Normalization and Feature Mapping. For each image instance, size normalization is performed, and its instance features are mapped to a uniform dimension through linear mapping or embedding layers. From the feature space, we obtain the instance embedding vector:

[0041] ;

[0042] These instance embedding vectors form the initial instance embedding sequence, providing a foundation for subsequent feature modeling;

[0043] Step S23: Construct an instance feature sequence with location information. A global label vector CLS is introduced before the instance embedding vector sequence: CLS vectors are used to represent the global semantic information of the entire image. Simultaneously, the embedding vector of each instance is added to its corresponding positional encoding vector to preserve the spatial location information of each image instance in the original image, forming an initial instance feature sequence:

[0044] ;

[0045] Step S24: Global Dependency Modeling and Feature Extraction. The initial instance feature sequence is input into the feature extraction and modeling module. This module includes a feature modeling network based on a self-attention mechanism. It models the global correlation between instances and extracts high-level semantic features through a multi-head self-attention structure and a nonlinear mapping unit. The output is an enhanced sequence containing global semantic label features and local instance features. The global label features are used for subsequent global branch prediction, and the local instance features are used as input to the dual-attention enhancement and multi-instance aggregation module.

[0046] like Figure 3 As shown, the dual attention module structure and enhanced instance features are illustrated. The generation integrates spatial and channel information.

[0047] In one embodiment, step S3 specifically includes the following operations:

[0048] Step S31: Spatial attention modeling. For each instance embedding vector obtained in step S2, the features corresponding to... Convolutional processing is performed to extract local contextual information and generate a spatial attention map. Normalized by the Sigmoid activation function, it represents the importance of different spatial locations;

[0049] Step S32: Spatial Enhancement Feature Generation. Generate the spatial attention map. The spatially enhanced features are obtained by element-wise multiplication with the input instance features:

[0050] ;

[0051] Step S33: Channel Attention Modeling. Perform global average pooling on the input instance features to obtain channel statistical vectors, and generate channel attention weights through a bottleneck mapping structure containing non-linear activations. This is used to characterize the discriminative contribution of each channel to the hierarchical task;

[0052] Step S34: Channel Enhancement Feature Generation. This involves assigning channel attention weights... The channel-enhanced features are obtained by multiplying them channel-by-channel with the input instance features:

[0053] ;

[0054] Step S35: Dual attention fusion. Spatial augmentation features. With channel enhancement features The features are concatenated along the channel dimension and then fused using convolutional methods and residual connections to obtain the final dual-attention enhanced instance features:

[0055] ,

[0056] This enhanced instance feature is used for subsequent multi-instance adaptive aggregation.

[0057] like Figure 4 As shown, the Soft Top-k adaptive aggregation process is demonstrated, which generates image-level features by weighted summation of key instances. .

[0058] In one embodiment, step S4 specifically includes the following operations:

[0059] Step S41: Unify instance feature dimensions. Perform linear mapping and nonlinear transformation on the dual-attention enhanced instance features obtained in Step S3 to obtain instance feature representations with unified dimensions:

[0060] ;

[0061] Step S42: Instance Importance Scoring. The features of each instance are modeled using an attention scoring network to generate a corresponding instance importance score.

[0062] ;

[0063] Step S43: Soft Top-k weighting. Score instances based on their importance. The system performs SoftTop-k adaptive weighted filtering on multiple instance features, and normalizes the importance scores using a temperature-controlled Softmax function to obtain the weights for each instance. This results in instances related to key lesions receiving higher weights, while instances in the background region receive lower weights.

[0064] Step S44: Image-level feature generation. The instance features are weighted and summed according to their respective weights to obtain the image-level feature representation:

[0065] ,

[0066] in, This represents the image-level feature representation, which serves as the high-dimensional semantic vector input to the classifier for the entire image.

[0067] In one embodiment, step S5 specifically includes the following operations:

[0068] Step S51: Multilayer perceptron classifier prediction. The image-level feature vector obtained in step S4 is used for prediction. Input a multilayer perceptron classifier (MLP) and output the diabetic retinopathy grading prediction result;

[0069] Step S52: Global Branch Prediction. Input the global label vector CLS obtained in steps S2-S3 into the global classification head to obtain the prediction result based on global semantic features;

[0070] Step S53: Fusion and Model Training. The multi-instance branch prediction results are fused with the global branch prediction results to obtain the final classification result. During the training phase, the gradient descent algorithm is used to optimize the feature extraction branch, adaptive aggregation module, and classifier parameters end-to-end to complete model training.

[0071] See Figure 5 and Figure 6 As shown, the multi-instance attention fusion method proposed in this embodiment achieves superior grading performance on both the APTOS2019 and RFMiD2020 datasets. Overall, it outperforms several existing diabetic retinopathy grading methods in key evaluation metrics such as accuracy, area under the curve, and consistency evaluation index. Experimental results demonstrate that this invention, by introducing a dual-attention enhancement mechanism and an adaptive instance aggregation strategy, effectively highlights key lesion region features and improves image-level feature representation capabilities, thereby significantly enhancing the accuracy and stability of diabetic retinopathy grading. This verifies the effectiveness and practicality of the proposed technical solution.

[0072] In summary, this invention proposes a multi-instance attention fusion method for grading diabetic retinopathy. It models and aggregates local lesion regions in fundus images using a multi-instance learning framework to obtain robust and discriminative image-level feature representations, enabling accurate grading of the severity of diabetic retinopathy. Specifically, firstly, input fundus image data is acquired, and training and testing sample sets are constructed. A single fundus image is uniformly divided into several image instances to form an instance set. Then, features are extracted from each image instance, and a dual attention mechanism is introduced to jointly model the spatial saliency and channel dependency of instance features, thereby obtaining enhanced instance-level feature representations. Further, based on instance importance scores, a Soft Top-k weighting strategy is used to adaptively select and normalize the weights of key instances, achieving effective aggregation of multi-instance features and generating image-level feature representations with global discriminative capabilities. Finally, the grading prediction of diabetic retinopathy is completed based on these image-level features.

[0073] This invention introduces an attention-based multi-instance adaptive aggregation mechanism between the instance level and the image level, enabling the model to focus on key regions related to the lesion and effectively suppress interference from irrelevant or noisy instances. This improves the accuracy and stability of diabetic retinopathy grading under weak supervision. The method of this invention has good generalization ability and interpretability, and can be widely applied in computer-aided diagnosis of diabetic retinopathy in clinical settings.

[0074] Furthermore, the present invention also provides a computing device comprising a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform any of the aforementioned multi-instance attention fusion methods for grading diabetic retinopathy.

[0075] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the steps defined in any of the aforementioned multi-instance attention fusion methods for grading diabetic retinopathy.

[0076] Those skilled in the art will understand that embodiments of the present invention can be implemented in pure hardware, pure software, or a combination of hardware and software. The present invention can also be implemented as a computer program product, comprising one or more computer-readable storage media storing computer-executable program instructions for implementing the method steps described in the embodiments of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A multi-instance attention fusion method for grading diabetic retinopathy, characterized in that, The method includes acquiring an input fundus image, extracting and enhancing features from local instances of the image through a multi-instance learning branch and a dual attention module, and then weighted and fused the enhanced instance features through an adaptive aggregation module to obtain an image-level high-dimensional feature representation. The image-level feature vector is then input into a multilayer perceptron classifier for integration and mapping, outputting the diabetic retinopathy grading result. The method specifically includes the following steps: Step S1: Acquire fundus image data and construct training and testing sample sets; adjust all images to 384×384 pixels and perform pixel value normalization; perform data augmentation on the images in the training sample set and input the augmented images and the original images into the subsequent feature extraction branch. Step S2: Divide a single fundus image into multiple image instances to form an instance set. The division is done using a fixed-size grid or a sliding window method. Each instance covers a local area, and the instances are uniformly sized and standardized. Step S3: Extract features from each image instance and perform weighted modeling of the channel information and spatial saliency of the instance features through a dual attention mechanism to obtain enhanced instance features; wherein spatial attention calculates weights through convolution, and channel attention calculates weights through a fully connected layer, and the two types of weights are fused into the instance features; Step S4: Adaptively aggregate the enhanced instance feature set to obtain image-level feature representation; the aggregation uses a Soft Top-k adaptive aggregation mechanism to weight key instances and sums the weighted instance features to obtain the final image-level vector representation; Step S5: Predict the grading of diabetic retinopathy based on image-level features and iteratively optimize the model parameters; by inputting image-level features into a multilayer perceptron classifier, outputting grading results, and using the gradient descent algorithm to update the parameters of each layer of the model during the training phase, the model training is completed.

2. The multi-instance attention fusion method for grading diabetic retinopathy according to claim 1, characterized in that, Step S1 includes: acquiring fundus image data and constructing a training sample set and a test sample set; uniformly adjusting the size of the fundus images to make the image size 384×384 pixels, and normalizing the image pixel values; performing data augmentation on the fundus images in the training sample set, and using the augmented images and the original images together as inputs for subsequent feature extraction branches.

3. The multi-instance attention fusion method for grading diabetic retinopathy according to claim 1, characterized in that, In step S2, dividing a single fundus image into multiple image instances specifically includes: Step S21: Input fundus image The image is evenly divided into several fixed-size blocks, with each block serving as an image instance, forming an instance set. ,in, Indicates the number of image instances; Step S22: Perform size normalization and linear mapping on each image instance, embed it into a feature space of uniform dimension, and obtain the instance embedding sequence; Step S23: Introduce a global label vector CLS before the instance embedding sequence, and add the instance embedding sequence to the position encoding vector to retain the spatial position information of each image instance, forming an initial instance feature sequence for subsequent feature modeling and attention fusion.

4. The multi-instance attention fusion method for grading diabetic retinopathy according to claim 1, characterized in that, Step S3 includes: Step S31: Perform spatial attention modeling on the instance features corresponding to each image instance obtained in Step S2, wherein the instance features are represented as... The spatial attention modeling includes: sequentially performing convolution, batch normalization, and non-linear activation on instance features to extract local contextual information; and then generating a spatial attention map through channel compression convolution. The values ​​are then normalized using the Sigmoid function to represent the importance of different spatial locations. Step S32: Transfer the spatial attention map The spatially enhanced features are obtained by element-wise multiplication with the input instance features: ； Step S33: Perform channel attention modeling on the input instance features, wherein the channel attention modeling includes: performing global average pooling on the instance features to obtain channel statistical vectors; generating channel attention weights through a bottleneck mapping structure containing non-linear activations. This is used to characterize the discriminative contribution of each channel to the hierarchical task; Step S34: Adjust the channel attention weights The channel-enhanced features are obtained by multiplying them channel-by-channel with the input instance features: ； Step S35: Spatial augmentation features With channel enhancement features The features are concatenated along the channel dimension and then fused using convolutional methods and residual connections to obtain the final dual-attention enhanced instance features: This is used for subsequent multi-instance aggregation.

5. The multi-instance attention fusion method for grading diabetic retinopathy according to claim 1, characterized in that, Step S4 includes: Step S41: Perform linear mapping and nonlinear transformation on the dual-attention enhanced instance features obtained in Step S3 to obtain instance feature representations with a unified dimension: ； Step S42: Model the features of each instance using an attention scoring network to generate the corresponding instance importance score: ； Step S43: Based on the instance importance score, select instances with higher weights using a Soft Top-k weighting strategy, and normalize them using temperature-based Softmax to obtain the instance weights. ; Step S44: Perform a weighted summation of the instance features according to the instance weights to obtain the image-level feature representation: ； in Representing image-level feature representation, Indicates the first The feature vector corresponding to each instance Indicates the first The weight coefficient of each instance, where 𝑁 represents the total number of instances.

6. The multi-instance attention fusion method for grading diabetic retinopathy according to claim 1, characterized in that, Step S5 includes: The image-level features obtained in step S4 are input into a multilayer perceptron classifier, which outputs a diabetic retinopathy grading result based on multi-instance aggregation. At the same time, the global label vector introduced in step S2 is input into a global classification head, which outputs a grading prediction result based on global semantic features. The multi-instance branch prediction result and the global branch prediction result are fused to obtain the final diabetic retinopathy grading result.

7. A computing device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, causes the processor to perform the multi-instance attention fusion method for grading diabetic retinopathy as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the multi-instance attention fusion method for grading diabetic retinopathy as described in any one of claims 1 to 6.

Images