A method and device for diabetic retinopathy segmentation based on feature interaction
By embedding a feature interaction module into the image segmentation model SAM, the feature interaction between lesion types and between lesions and the background is enhanced, which solves the problem of insufficient accuracy in the segmentation of diabetic retinopathy images in the prior art and achieves higher accuracy in lesion identification and segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG UNIV
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-09
AI Technical Summary
Existing image segmentation models struggle to effectively model the relationships between multiple lesions in diabetic retinopathy scenarios, and their ability to interact with the features of lesions and the background is insufficient, resulting in low accuracy in lesion identification and segmentation, especially in small-scale, low-contrast lesions.
In each ViT block coding layer of the image segmentation model SAM, a feature interaction module for modeling semantic relationships of multiple lesions is embedded. Through image subdomain partitioning, lesion interaction feature extraction, lesion-background interaction feature extraction, and interaction feature fusion, the feature interaction capabilities between lesion types and between lesions and the background are enhanced.
It improves the accuracy of automatic segmentation of multiple lesions in fundus color images, suppresses background noise interference, improves segmentation performance in cases of blurred lesion boundaries and insufficient contrast, and enhances the distinction between lesion areas and background areas.
Smart Images
Figure CN122176311A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and device for segmenting diabetic retinopathy based on feature interaction. Background Technology
[0002] Diabetic retinopathy is a common and serious retinal complication of diabetes, and a major cause of vision loss and even blindness. With the continued increase in the number of diabetic patients, early screening and accurate identification of diabetic retinopathy have become crucial tools in clinical ophthalmology.
[0003] Fundus color images are crucial for diagnosing diabetic retinopathy (DR). DR typically presents in fundus color images as various lesion types, including microaneurysms, hemorrhages, soft exudates, and hard exudates. These lesions vary significantly in morphology, size, boundary, grayscale distribution, and spatial distribution. Furthermore, some lesions are small in size and have low contrast, making them easily confused with complex background tissue, thus increasing the difficulty of lesion identification and segmentation. Therefore, achieving automatic and accurate segmentation of various lesions in fundus color images is a key issue in the intelligent assisted diagnosis of DR.
[0004] In existing technologies, most image analysis methods for diabetic retinopathy employ traditional image processing techniques or deep learning segmentation networks. Traditional methods typically rely on manually designed features, are highly sensitive to changes in image quality, illumination, and lesion morphology, and have limited generalization capabilities. While segmentation methods based on convolutional neural networks have improved lesion recognition accuracy to some extent, their limited receptive field in convolution operations hinders their ability to model global contextual relationships between multiple lesions, making it difficult to simultaneously consider both fine-grained lesion features and global dependencies.
[0005] In recent years, large-scale models based on the visual Transformer have demonstrated strong global feature modeling capabilities in image segmentation. In particular, the Segment Anything Model (SAM) exhibits good transferability and general segmentation potential, providing a new technical path for medical image segmentation. However, existing SAM models are primarily designed for general natural images and still have the following shortcomings in diabetic retinopathy scenarios: First, they lack a dedicated modeling mechanism for the relationships between multiple lesion categories, making it difficult to fully utilize the co-occurrence information and dependencies between different lesions; second, the boundaries between lesions and the background are complex, making it difficult for general feature extraction methods to effectively suppress background interference; and third, there is still room for improvement in the segmentation accuracy of small-scale, low-contrast lesions in fundus images. Summary of the Invention
[0006] To address the problems existing in the prior art, this invention provides a method and device for segmenting diabetic retinopathy based on feature interaction. While retaining the global modeling capability of the image segmentation model SAM, it further enhances the feature interaction capability between different lesion types of diabetic retinopathy and between lesions and the background of fundus color images, so as to improve the accuracy and reliability of automatic segmentation of multiple lesions in fundus color images.
[0007] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0008] A feature-interaction-based segmentation method for diabetic retinopathy includes the following steps: Step S1: Collect fundus color images and corresponding black and white lesion images from different diabetic patients, and perform mask annotation on all lesion types corresponding to retinal lesions on the black and white lesion images to obtain fundus color image-mask annotation pairs; Step S2: Construct a feature interaction module for multi-lesion semantic relationship modeling, embed it into each ViT block encoding layer of the image encoder of the image segmentation model SAM, and obtain the image segmentation model FIASAM with the feature interaction module fused. Step S3: Use the fundus color image in the fundus color image-mask annotation pair as the input of the image segmentation model FIASAM of the fusion feature interaction module, and use the corresponding mask annotation as the training label to train the image segmentation model FIASAM of the fusion feature interaction module until the composite loss function of fusion cross-entropy loss and mask overlap consistency loss converges, thus completing the training of the image segmentation model FIASAM of the fusion feature interaction module. Step S4: Collect color images of the fundus of diabetic patients and input them into the trained image segmentation model SAM with fusion feature interaction module to generate image segmentation results containing all lesion types.
[0009] Further, step S1 includes the following sub-steps: Step S1.1: Provide each diabetic patient with a color fundus image and a black and white lesion image corresponding to each type of retinal lesion, and unify the resolution of all color fundus images and black and white lesion images; Step S1.2: Set pixel identifiers for different lesion types and generate mask information for the lesion types; Step S1.3: Mask the lesion type on the black and white lesion image with uniform resolution for each diabetic patient according to the mask information of the lesion type; Step S1.4: Integrate the mask annotation results of all lesion types for each diabetic patient to obtain a comprehensive annotation map containing multiple lesion types; Step S1.5: Combine the fundus color image of each diabetic patient with the comprehensive annotation map to form a fundus color image-mask annotation pair, and combine the fundus color image-mask annotation pairs of all diabetic patients to form a lesion annotation dataset.
[0010] Furthermore, the feature interaction module includes: an image subdomain division unit, a lesion interaction feature extraction unit, a lesion and background interaction feature extraction unit, an interaction feature fusion unit, and a multi-lesion interaction feature stitching unit; The image sub-region division unit is used to divide the fundus color image extracted by global semantic features into image sub-regions of different lesion types according to the mask information, and extract the image sub-region set and image background sub-region set corresponding to each lesion type. The lesion interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and other lesion types based on the image sub-region set corresponding to each lesion type. The lesion-background interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and the image background based on the image sub-region set and the image background sub-region set corresponding to each lesion type. The interactive feature fusion unit is used to perform weighted fusion of the semantic interaction features between the target lesion type and other lesion types, as well as the semantic interaction features between the target lesion type and the image background, to obtain the enhanced features of the target lesion type. The multi-lesion interactive feature stitching unit is used to stitch together the enhanced features of all target lesion types.
[0011] Furthermore, the extraction process of semantic interaction features between the target lesion type and other lesion types is specifically as follows: i: Obtain the guiding features, response features, and content features of the lesion type based on the set of image sub-regions corresponding to each lesion type; ii: Select a lesion type as the target lesion type, calculate the directional semantic effect strength of the target lesion type on the lesion type based on the guiding features of the target lesion type and the response features of each other lesion type, and combine the content features of the lesion type to obtain the semantic interaction features between the target lesion type and the lesion type; iii: The semantic interaction features between the target lesion type and each of the other lesion types are aggregated by weighted importance to obtain the semantic interaction features between the target lesion type and the other lesion types.
[0012] Furthermore, the semantic interaction features between the target lesion type and other lesion types are represented as follows:
[0013] in, This indicates the total number of lesion types. express index, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features with other lesion types, Indicates the type of target lesion The guiding characteristics, Indicates lesion type The response characteristics, Indicates lesion type Content characteristics, Represents the normalization factor. This indicates the transpose operation. This indicates a normalization operation. Indicates the type of target lesion With lesion type The importance weights of semantic interaction features between them.
[0014] Furthermore, the process of extracting the semantic interaction features between the target lesion type and the image background is as follows: i: Obtain the guiding features of the lesion type based on the set of image sub-regions corresponding to each lesion type, and obtain the response features and content features of the image background based on the set of image background sub-regions; ii: Select a lesion type as the target lesion type, calculate the directional semantic effect strength of the target lesion type on the image background based on the guiding features of the target lesion type and the response features of the image background, and obtain the semantic interaction features between the target lesion type and the image background by combining the content features of the image background.
[0015] Furthermore, the semantic interaction features between the target lesion type and the image background are represented as follows:
[0016] in, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features between the image and the background Indicates the type of target lesion The guiding characteristics, Represents the image background The response characteristics, Represents the image background Content characteristics, Represents the normalization factor. This indicates a normalization operation. This indicates the transpose operation.
[0017] Furthermore, each ViT block coding layer includes a first-layer regularization unit, a multi-head attention unit, a residual unit, a second-layer regularization unit, a multilayer perceptron unit, and a feature fusion unit connected in sequence. The input of the feature interaction module is connected to the output of the residual unit, and the output of the feature interaction module is connected to the input of the feature fusion unit.
[0018] Furthermore, the composite loss function Specifically:
[0019] in, Represents the cross-entropy loss function. , This represents the number of pixels in a color image of the fundus. express index, This indicates the total number of lesion types. express index, Represents pixels Lesion type The tag value, Represents pixels Lesion type The predicted probability, express Weighting factors; This represents the mask overlap consistency loss function. , Indicates the smoothing term. express Weighting factors.
[0020] Furthermore, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the aforementioned feature-interaction-based diabetic retinopathy segmentation method.
[0021] Compared with the prior art, the present invention has the following beneficial effects: This invention presents a feature-interaction-based segmentation method for diabetic retinopathy. By embedding a feature interaction module for modeling semantic relationships among multiple lesions into each ViT block encoding layer of the image encoder in the SAM image segmentation model, this method enables the establishment of corresponding interactive semantic mapping relationships between different lesion types and between lesion types and the image background during the image encoding process of fundus color images. It also models the dependencies between different lesion types, thereby achieving feature interaction and information fusion between different lesion types and between lesions and the background. This feature interaction module fully explores the co-occurrence features, spatial connections, and contextual dependencies among different lesion types, which is beneficial for improving the accuracy of multi-lesion segmentation in complex fundus color images. Simultaneously, it helps suppress background noise and interference from non-lesion regions, improving segmentation results in cases of blurred lesion boundaries and insufficient contrast, and enhancing the distinguishability between lesion regions and background regions. This invention selectively models the association of sub-region sets based on lesion type, rather than performing pairwise association calculations on all image blocks, avoiding a large amount of redundant computation and significantly reducing computational costs while ensuring feature extraction quality.
[0022] In summary, this invention adapts and fine-tunes the image segmentation model SAM for medical tasks. While retaining the global context modeling capability of the image segmentation model SAM, it further enhances the adaptability to the multi-lesion segmentation task of diabetic retinopathy, thereby improving the segmentation accuracy of the image segmentation model SAM in medical fundus color image scenes. Attached Figure Description
[0023] Figure 1 This is a flowchart of the feature-interaction-based diabetic retinopathy segmentation method of the present invention; Figure 2 This is a schematic diagram illustrating the construction of the feature interaction module for multi-lesion semantic relationship modeling in this invention; Figure 3 This is a schematic diagram of semantic interaction feature extraction in the feature interaction module of the present invention; Figure 4 This is a schematic diagram of the FIASAM image segmentation model that integrates the feature interaction module of the present invention. Figure 5 This diagram illustrates a comparison of the segmentation results of the feature-interaction-based diabetic retinopathy segmentation method of this invention with other large-scale basic segmentation models. Figure 6 This diagram illustrates a comparison of the segmentation results between the feature-interaction-based diabetic retinopathy segmentation method of this invention and the SAM segmentation method incorporating an attention mechanism. Detailed Implementation
[0024] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.
[0025] like Figure 1 This is a flowchart of the feature-interaction-based diabetic retinopathy segmentation method of the present invention, which includes the following steps: Step S1: Collect fundus color images and corresponding black-and-white lesion images from different diabetic patients. Mask all lesion types corresponding to retinal lesions on the black-and-white lesion images to obtain fundus color image-mask label pairs. This includes the following sub-steps: Step S1.1: Provide each diabetic patient with a color fundus image and a black and white lesion image corresponding to each type of retinal lesion. Unify the resolution of all fundus color images and black and white lesion images to 1280×1280 to eliminate size differences caused by different acquisition devices, imaging conditions and storage formats. Among them, retinal lesions correspond to 4 lesion types, including: microaneurysm, hemorrhage, soft exudate and hard exudate.
[0026] Step S1.2: Set pixel identifiers for different lesion types, such as setting the black and white lesion images of microaneurysms, hemorrhages, soft exudates, and hard exudates to grayscale values of 50, 100, 150, and 200 respectively, to generate lesion mask information that can characterize the spatial distribution relationship of multiple types of lesions.
[0027] Step S1.3: Mask the lesion type on the black and white lesion image with uniform resolution for each diabetic patient according to the lesion type mask information.
[0028] Step S1.4: Integrate the mask annotation results of all lesion types for each diabetic patient to obtain a comprehensive annotation map containing multiple lesion types, so as to simultaneously describe the location, extent, morphology and mutual adjacency of various lesions in the comprehensive annotation map.
[0029] Step S1.5: Combine the fundus color image of each diabetic patient with the comprehensive annotation map to form a fundus color image-mask annotation pair, and combine the fundus color image-mask annotation pairs of all diabetic patients to form a lesion annotation dataset.
[0030] Step S2: Construct a feature interaction module for multi-lesion semantic relationship modeling, embedding it into each ViT block encoding layer of the image encoder of the image segmentation model SAM, resulting in the image segmentation model FIASAM with fused feature interaction module. This allows for the establishment of corresponding interactive semantic mapping relationships between different lesion types and between lesion types and the image background during the image encoding process of fundus color images, and models the dependencies between different lesion types, thereby achieving feature interaction and information fusion between different lesion types and between lesions and the background. Through this feature interaction module, the co-occurrence features, spatial connections, and contextual dependencies between different lesion types can be fully explored, which is beneficial to improving the accuracy of multi-lesion segmentation in complex fundus color images. At the same time, it helps to suppress background noise and interference from non-lesion areas, thereby improving the segmentation effect in cases of blurred lesion boundaries and insufficient contrast, and improving the distinguishability between lesion areas and background areas. This invention selectively models the association of sub-region sets based on lesion type, rather than performing pairwise association calculations on all image blocks, avoiding a large amount of redundant calculations and greatly reducing the computational cost while ensuring the quality of feature extraction.
[0031] like Figure 2 The feature interaction module for multi-lesion semantic relationship modeling of the present invention includes: an image subdomain division unit, a lesion interaction feature extraction unit, a lesion and background interaction feature extraction unit, an interaction feature fusion unit, and a multi-lesion interaction feature splicing unit.
[0032] The image sub-region division unit is used to divide the fundus color image extracted by global semantic features into image sub-regions of different lesion types according to the mask information, and to extract the image sub-region set and image background sub-region set corresponding to each lesion type.
[0033] The lesion interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and other lesion types based on the image sub-region set corresponding to each lesion type, such as... Figure 3 : i: Construct three semantic mapping mechanisms specific to each lesion type based on the image sub-region set corresponding to each lesion type, which are used to map the global semantic features of the image sub-region set corresponding to each lesion type. Projecting these features onto an independent semantic representation space corresponding to the lesion type yields guiding features for each lesion type. Response characteristics and content features :
[0034]
[0035]
[0036] Among them, guiding features Used to express lesion type Active focus direction and response characteristics during the interaction process Used to express lesion type Sensitivity to other categories of semantic input, content features Used to preserve lesion type Its core structural information, texture information, and local context information; Lesion type The guiding projection matrix, response projection matrix, and content projection matrix.
[0037] ii: Select a lesion type as the target lesion type, and calculate the directional semantic effect strength of the target lesion type on the other lesion types based on the guiding features of the target lesion type and the response features of each of the remaining lesion types, in order to characterize the target lesion type. Another lesion type in the current semantic context Based on the degree of dependence, the differentiation of needs and information retrieval tendencies, and combined with the content characteristics of lesion types, the semantic interaction characteristics between the target lesion type and another lesion type are obtained. iii: The semantic interaction features between the target lesion type and each of the other lesion types are aggregated using an importance-weighted method to obtain the semantic interaction features between the target lesion type and the other lesion types:
[0038] in, This indicates the total number of lesion types. express index, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features with other lesion types, Indicates the type of target lesion The guiding characteristics, Indicates lesion type The response characteristics, Indicates lesion type Content characteristics, Represents the normalization factor. This indicates the transpose operation. This indicates a normalization operation. Indicates the type of target lesion With lesion type The importance weights of semantic interaction features are used to adjust the importance of interactions between different lesion types.
[0039] The lesion-background interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and the image background based on the image sub-region set corresponding to each lesion type and the image background sub-region set, such as... Figure 3 : i: Obtain the guiding features of the lesion type based on the set of image sub-regions corresponding to each lesion type, and obtain the response features and content features of the image background based on the set of image background sub-regions; ii: Select a lesion type as the target lesion type. Calculate the directional semantic influence strength of the target lesion type on the image background based on the guiding features of the target lesion type and the response features of the image background. Combine this with the content features of the image background to obtain the semantic interaction features between the target lesion type and the image background.
[0040] in, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features between the image and the background Indicates the type of target lesion The guiding characteristics, Represents the image background The response characteristics, Represents the image background Content characteristics, Represents the normalization factor. This indicates a normalization operation. This indicates the transpose operation.
[0041] The interactive feature fusion unit is used to weightedly fuse the semantic interaction features between the target lesion type and other lesion types, as well as the semantic interaction features between the target lesion type and the image background, to obtain the enhanced features of the target lesion type. :
[0042] in, and These are all learnable balancing parameters that control the contribution of foreground-foreground interaction and foreground-background interaction to the final features, thereby adaptively adjusting the class imbalance problem at the feature level.
[0043] The multi-lesion interactive feature stitching unit is used to stitch together the enhancement features of all target lesion types.
[0044] like Figure 4Each ViT block coding layer includes a first-layer regularization unit, a multi-head attention unit, a residual unit, a second-layer regularization unit, a multilayer perceptron unit, and a feature fusion unit, all connected sequentially. The input of the feature interaction module is connected to the output of the residual unit, and the output of the feature interaction module is connected to the input of the feature fusion unit. The residual unit outputs global semantic features that characterize the overall semantics of the fundus color image. These global semantic features simultaneously contain information about multiple lesion regions and background regions, enabling the feature interaction module to continuously receive global visual semantics during image encoding and to model and mine the relationships between multiple lesions. The enhanced features concatenated from the output of the feature interaction module are fused with the global semantic features output from the multilayer perceptron unit via the feature fusion module to form the output of the ViT block coding layer. This continuously optimizes the feature representation capability of the ViT block coding layer within the encoder, thereby forming a segmentation model suitable for segmenting multiple lesions in diabetic retinopathy. Simultaneously, user-provided cue boxes are fed into the cue encoder and converted into cue embeddings to indicate the target regions that the FIASAM image segmentation model, which integrates the feature interaction module, needs to focus on segmenting. Subsequently, the output of the last ViT block coding layer in the image encoder is fused with the cue embedding and further supplemented with local spatial information by convolution operation. Then, they are jointly input into the mask decoder to generate the final lesion type prediction mask.
[0045] Step S3: The fundus color image from the fundus color image-mask annotation pair is used as input to the FIASAM image segmentation model of the fusion feature interaction module. The corresponding mask annotations are used as training labels to train the FIASAM image segmentation model. The Nadam optimizer is used to optimize the parameters of the FIASAM image segmentation model, and the predetermined batch size is set to 4, the learning step size is set to 0.0006, and the training parameters are updated through backpropagation until the composite loss function of fusion entropy loss and mask overlap consistency loss converges, thus completing the training of the FIASAM image segmentation model. The FIASAM image segmentation model of the fusion feature interaction module of this invention retains the powerful segmentation capabilities of the original SAM image segmentation model while achieving effective adaptation and performance improvement for fundus medical image segmentation tasks.
[0046] The composite loss function in this invention Specifically:
[0047] in, Represents the cross-entropy loss function. , This represents the number of pixels in a color image of the fundus. express index, This indicates the total number of lesion types. express index, Represents pixels Lesion type The tag value, Represents pixels Lesion type The predicted probability, express Weighting factors; This represents the mask overlap consistency loss function. , This represents a smoothing term used to avoid a denominator of 0. express Weighting factors.
[0048] Step S4: Collect color images of the fundus of diabetic patients and input them into the trained image segmentation model SAM with fusion feature interaction module to generate image segmentation results containing all lesion types.
[0049] To verify the effectiveness of the feature interaction-based segmentation method for diabetic retinopathy of this invention, the image segmentation model FIASAM, which integrates the feature interaction module of this invention, was compared with existing large-scale basic segmentation models on the FGADR dataset. These large-scale basic segmentation models include: SAM, MedSAM, SAM(LoRA), SAMUnet, and HQ-SAM. SAM is a benchmark segmentation model first proposed by Meta and trained on a massive dataset. MedSAM is an improved version of SAM, optimized for medical image segmentation, trained using a large amount of medical image data, improving its performance in complex medical problems. SAM(LoRA) incorporates low-rank techniques to optimize the model's parameter update process, enabling SAM(LoRA) to adapt more efficiently to new datasets and exhibiting greater flexibility during training. SAMUnet combines SAM and UNet; UNet's encoder-decoder structure enhances SAM's feature extraction capabilities, improving segmentation accuracy, especially when handling complex structures. HQ-SAM is a model focused on high-quality segmentation, improving segmentation detail through the design of learnable "high-quality output labels."
[0050] Table 1: Comparison of lesion type segmentation performance of image segmentation models on the FGADR dataset
[0051] Table 1 compares the segmentation performance of different image segmentation models on the FGADR dataset. Among the evaluation metrics, mDice and mJacc reflect the degree of overlap between the lesion region segmented by the lesion prediction mask and the lesion region labeled by the ground truth mask; higher values indicate higher segmentation accuracy. mHD reflects the average distance between the boundary of the lesion prediction mask and the boundary of the ground truth mask; lower values indicate more accurate lesion boundary localization. As shown in Table 1, the FIASAM image segmentation model fused with the feature interaction module of this invention achieves the best results in all three metrics, with mDice reaching 80.63%, mJacc reaching 73.48%, and mHD reduced to 12.19 mm. Compared to other image segmentation models, the FIASAM image segmentation model fused with the feature interaction module of this invention further improves boundary characterization ability while maintaining high region segmentation consistency, indicating that this invention can achieve more accurate and stable segmentation of lesion regions.
[0052] Table 2: Comparison of mAUC-PR (%) of image segmentation models for multi-lesion types on the FGADR dataset
[0053] Because medical images commonly suffer from class imbalance, meaning there is a significant difference in pixel ratio between the segmented target and the background, mAUC-PR is used to reflect the ability of an image segmentation model to identify lesions under different threshold conditions in the case of imbalanced sample distribution. A higher value indicates better model performance. Table 2 shows that the FIASAM image segmentation model, which integrates the feature interaction module of this invention, achieves mAUC-PR of 83.18%, 84.83%, 81.74%, and 89.47% for soft exudates, hemorrhages, microaneurysms, and hard exudates, respectively, all of which are superior to the baseline image segmentation model SAM's 75.42%, 73.37%, 75.17%, and 84.53%. This result indicates that the feature interaction module introduced in this invention can maintain high lesion recognition performance even under class imbalance conditions, and has good feature representation ability and sensitivity for different types of lesions, thereby effectively improving the segmentation accuracy in multi-lesion scenarios.
[0054] like Figure 5 This diagram illustrates a comparison of the segmentation results of the feature-interaction-based diabetic retinopathy segmentation method of this invention with other large-scale basic segmentation models. It can be seen that the present invention performs excellently across all lesion types and is the segmentation result closest to the actual lesion mask. In contrast, SAM and HQ-SAM showed serious missed detections in hard exudates and microaneurysms, and the segmentation edges were mottled. Therefore, the present invention can effectively improve the accuracy of multi-lesion segmentation in complex scenarios.
[0055] To verify the effectiveness of the feature interaction module in this invention, a comparative experiment was conducted with the attention mechanism commonly used in feature extraction. In the image encoder of the original image segmentation model SAM, a cross-attention mechanism and a self-attention mechanism were embedded, respectively, referring to the deployment location of the feature interaction module in this invention. The experimental results are shown in Table 3. The image segmentation model SAM with embedded cross-attention and self-attention mechanisms showed very limited improvement in several metrics, and even performed worse than the original image segmentation model SAM. This is because the self-attention mechanism, by modeling the global dependencies between image pixels, can capture global information within the image to some extent, but it cannot distinguish differences between categories, resulting in no significant performance improvement in complex multi-lesion category scenarios. The feature interaction mode in the cross-attention mechanism is usually unidirectional and does not perform reverse calculations, which greatly limits its performance in complex image structures and class imbalance situations. Compared with traditional self-attention and cross-attention mechanisms, the feature interaction module achieves bidirectional foreground-foreground and foreground-background interactions by modeling the interactions between lesion categories and the asymmetric semantic dependence between lesion categories and background. By introducing the computation of category-guided features and category-response features, the feature interaction module can adaptively adjust the category imbalance, thereby significantly improving the performance of the image segmentation model SAM that integrates the feature interaction module in multi-lesion image segmentation tasks.
[0056] Table 3: Comparison of segmentation performance between SAM image segmentation model with different attention mechanisms and the present invention
[0057] like Figure 6 This paper presents the feature map visualization results generated by Grad-CAM for the feature interaction-based diabetic retinopathy segmentation method of this invention and the SAM segmentation method incorporating an attention mechanism. It can be seen that while the two SAM models, one incorporating cross-attention and the other self-attention, show some attention in the overall image, they fail to accurately focus on the lesion region, instead focusing more on the background area. In contrast, the FIASAM image segmentation model of this invention, which integrates a feature interaction module, accurately focuses attention on the core location of the lesion, clearly displaying a high response value for this region in the feature map. This indicates that FIASAM performs excellently in focusing on important features, effectively focusing on key regions in the task and improving its performance in image segmentation.
[0058] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the feature-interaction-based diabetic retinopathy segmentation method of the present invention.
[0059] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0060] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0061] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for segmenting diabetic retinopathy based on feature interaction, characterized in that, Includes the following steps: Step S1: Collect fundus color images and corresponding black and white lesion images from different diabetic patients, and perform mask annotation on all lesion types corresponding to retinal lesions on the black and white lesion images to obtain fundus color image-mask annotation pairs; Step S2: Construct a feature interaction module for multi-lesion semantic relationship modeling, embed it into each ViT block encoding layer of the image encoder of the image segmentation model SAM, and obtain the image segmentation model FIASAM with the feature interaction module fused. Step S3: Use the fundus color image in the fundus color image-mask annotation pair as the input of the image segmentation model FIASAM of the fusion feature interaction module, and use the corresponding mask annotation as the training label to train the image segmentation model FIASAM of the fusion feature interaction module until the composite loss function of fusion cross-entropy loss and mask overlap consistency loss converges, thus completing the training of the image segmentation model FIASAM of the fusion feature interaction module. Step S4: Collect color images of the fundus of diabetic patients and input them into the trained image segmentation model FIASAM with fusion feature interaction module to generate image segmentation results containing all lesion types.
2. The method for segmenting diabetic retinopathy based on feature interaction according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S1.1: Provide each diabetic patient with a color fundus image and a black and white lesion image corresponding to each type of retinal lesion, and unify the resolution of all color fundus images and black and white lesion images; Step S1.2: Set pixel identifiers for different lesion types and generate mask information for the lesion types; Step S1.3: Mask the lesion type on the black and white lesion image with uniform resolution for each diabetic patient according to the mask information of the lesion type; Step S1.4: Integrate the mask annotation results of all lesion types for each diabetic patient to obtain a comprehensive annotation map containing multiple lesion types; Step S1.5: Combine the fundus color image of each diabetic patient with the comprehensive annotation map to form a fundus color image-mask annotation pair, and combine the fundus color image-mask annotation pairs of all diabetic patients to form a lesion annotation dataset.
3. The method for segmenting diabetic retinopathy based on feature interaction according to claim 2, characterized in that, The feature interaction module includes: an image subdomain division unit, a lesion interaction feature extraction unit, a lesion and background interaction feature extraction unit, an interaction feature fusion unit, and a multi-lesion interaction feature stitching unit; The image sub-region division unit is used to divide the fundus color image extracted by global semantic features into image sub-regions of different lesion types according to the mask information, and extract the image sub-region set and image background sub-region set corresponding to each lesion type. The lesion interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and other lesion types based on the image sub-region set corresponding to each lesion type. The lesion-background interaction feature extraction unit is used to extract semantic interaction features between the target lesion type and the image background based on the image sub-region set and the image background sub-region set corresponding to each lesion type. The interactive feature fusion unit is used to perform weighted fusion of the semantic interaction features between the target lesion type and other lesion types, as well as the semantic interaction features between the target lesion type and the image background, to obtain the enhanced features of the target lesion type. The multi-lesion interactive feature stitching unit is used to stitch together the enhanced features of all target lesion types.
4. The method for segmenting diabetic retinopathy based on feature interaction according to claim 3, characterized in that, The extraction process of semantic interaction features between the target lesion type and other lesion types is as follows: i: Obtain the guiding features, response features, and content features of the lesion type based on the set of image sub-regions corresponding to each lesion type; ii: Select a lesion type as the target lesion type, calculate the directional semantic effect strength of the target lesion type on the lesion type based on the guiding features of the target lesion type and the response features of each other lesion type, and combine the content features of the lesion type to obtain the semantic interaction features between the target lesion type and the lesion type; iii: The semantic interaction features between the target lesion type and each of the other lesion types are aggregated by weighted importance to obtain the semantic interaction features between the target lesion type and the other lesion types.
5. A method for segmenting diabetic retinopathy based on feature interaction according to claim 4, characterized in that, The semantic interaction features between the target lesion type and other lesion types are represented as follows: in, This indicates the total number of lesion types. express index, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features with other lesion types, Indicates the type of target lesion The guiding characteristics, Indicates lesion type The response characteristics, Indicates lesion type Content characteristics, Represents the normalization factor. This indicates the transpose operation. This indicates a normalization operation. Indicates the type of target lesion With lesion type The importance weights of semantic interaction features between them.
6. The method for segmenting diabetic retinopathy based on feature interaction according to claim 3, characterized in that, The process of extracting the semantic interaction features between the target lesion type and the image background is as follows: i: Obtain the guiding features of the lesion type based on the set of image sub-regions corresponding to each lesion type, and obtain the response features and content features of the image background based on the set of image background sub-regions; ii: Select a lesion type as the target lesion type, calculate the directional semantic effect strength of the target lesion type on the image background based on the guiding features of the target lesion type and the response features of the image background, and obtain the semantic interaction features between the target lesion type and the image background by combining the content features of the image background.
7. A method for segmenting diabetic retinopathy based on feature interaction according to claim 6, characterized in that, The semantic interaction features between the target lesion type and the image background are represented as follows: in, An index representing the type of target lesion. Indicates the type of target lesion Semantic interaction features between the image and the background Indicates the type of target lesion The guiding characteristics, Represents the image background The response characteristics, Represents the image background Content characteristics, Represents the normalization factor. This indicates a normalization operation. This indicates the transpose operation.
8. The method for segmenting diabetic retinopathy based on feature interaction according to claim 1, characterized in that, Each ViT block coding layer includes a first-layer regularization unit, a multi-head attention unit, a residual unit, a second-layer regularization unit, a multilayer perceptron unit, and a feature fusion unit connected in sequence. The input of the feature interaction module is connected to the output of the residual unit, and the output of the feature interaction module is connected to the input of the feature fusion unit.
9. A method for segmenting diabetic retinopathy based on feature interaction according to claim 1, characterized in that, The composite loss function Specifically: in, Represents the cross-entropy loss function. , This represents the number of pixels in a color image of the fundus. express index, This indicates the total number of lesion types. express index, Represents pixels Lesion type The tag value, Represents pixels Lesion type The predicted probability, express Weighting factors; This represents the mask overlap consistency loss function. , Indicates the smoothing term. express Weighting factors.
10. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the feature-interaction-based diabetic retinopathy segmentation method as described in any one of claims 1-9.