Robust polyp segmentation method based on improved sam-med2d

Abstract

The application discloses a robust polyp segmentation method based on improved SAM-Med2D, comprising the following steps: S1, generating a record file for a given data set; S2, constructing a SAM-Med2D model, keeping the aspect ratio for a non-square image; S3, extracting multi-scale features of the image through a double-branch encoder; S4, performing fusion processing on the multi-scale features in a mask decoder; S5, generating a preliminary segmentation result based on the fused features and prompt coding; S6, constructing a lightweight reverse attention refinement module; S7, performing loss calculation and parameter optimization on the SAM-Med2D model; S8, ending polyp segmentation of the current image and entering the next image, and repeating steps S1 to S7. Through the fusion of the ViT-CNN double-branch structure and the reverse attention mechanism, the application realizes the robust segmentation of the polyp region with high precision and low calculation cost in a complex endoscope scene.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a robust polyp segmentation method based on improved SAM-Med2D. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, medical image segmentation has become a key task in computer-assisted diagnosis and treatment. During gastrointestinal endoscopy, accurate polyp segmentation facilitates early detection and intervention of diseases such as colorectal cancer, and possesses high clinical application value. Existing medical image segmentation methods have evolved from convolutional neural network (CNN)-based methods (such as U-Net and its variants, U-Net++, and Attention U-Net), to Transformer-based models (such as TransUNet and Swin-Unet), and finally to hybrid models that combine these two architectures (such as TransFuse and MC-Trans), achieving some improvements in segmentation accuracy. However, these methods are generally limited by model size, insufficient feature extraction and representation capabilities, and limited generalization and scalability due to their design for specific tasks.

[0003] As a general visual foundation model pre-trained on large-scale datasets, the Segment Anything Model (SAM) has demonstrated strong zero-shot and few-shot generalization capabilities in natural image segmentation tasks, providing a new technical path for general image segmentation. However, due to the differences between medical images and natural images in terms of image features, distribution, and representation, the performance of SAM may degrade when directly applied to medical image segmentation tasks. To enhance the adaptability of SAM in medical image scenarios, a variety of adaptation schemes have emerged, such as SAMed, MedSAM, MSA, and SAMUS. Among them, SAM-Med2D, as an extended model for medical image segmentation, is trained on medical datasets and uses Adapter technology to fine-tune the Visual Transformer (ViT) encoder in SAM. It supports a variety of interactive prompting methods based on points, boxes, and masks, thereby enhancing its adaptability in various two-dimensional medical image segmentation tasks.

[0004] Although existing adaptation methods have achieved certain results in multiple medical scenarios, they still suffer from performance degradation in complex tasks such as endoscopic polyp segmentation. The main reasons include: (1) endoscopic images often contain interference factors such as uneven illumination, complex morphology, and high tissue structure similarity; (2) the high cost of medical image data annotation and the limited number of samples restrict the training effect of large models; (3) the ViT encoder has structural bias in small sample scenarios or structural modeling tasks, which affects the feature expression ability. In particular, if the original SAM-Med2D model is directly trained and parameterized on the public polyp segmentation dataset, its overall performance will decline to varying degrees, and its performance will be worse than that of the original pre-trained model without fine-tuning. This phenomenon indicates that the model may have forgotten the general representation ability originally obtained through large-scale medical data pre-training during the fine-tuning process, that is, the typical "catastrophic forgetting" problem has occurred.

[0005] Therefore, how to provide a robust polyp segmentation method based on improved SAM-Med2D is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0006] One object of the present invention is to propose a robust polyp segmentation method based on an improved SAM-Med2D. The present invention combines a dual-branch cross-attention module, a LoRA-based fine-tuning strategy, and a reverse attention refinement module to enhance feature expression capabilities under lightweight fine-tuning constraints, thereby improving the adaptability and segmentation performance of the SAM architecture in complex medical image scenarios.

[0007] A robust polyp segmentation method based on improved SAM-Med2D according to an embodiment of the present invention includes the following steps:

[0008] S1. For a given dataset, generate a record file in JSON format to describe the correspondence between images and mask labels;

[0009] S2. Build the SAM-Med2D model, uniformly scale the original image and mask labels to 256 × 256 pixels through bilinear interpolation, and use edge padding strategy to maintain the aspect ratio for non-square images;

[0010] S3, extracting multi-scale features of the image through a dual-branch encoder;

[0011] S4. In the mask decoder, a cross-attention mechanism and a feature fusion module are used to fuse multi-scale features.

[0012] S5, generating preliminary segmentation results based on fusion features and hint coding;

[0013] S6. Build a lightweight reverse attention refinement module to correct errors in the initial segmentation results;

[0014] S7, using a multi-objective loss joint optimization strategy to perform loss calculation and parameter optimization on the SAM-Med2D model;

[0015] S8. End the polyp segmentation of the current image, proceed to the next image, and repeat steps S1 to S7.

[0016] Optionally, the data set includes a training data set and a test data set, the training data set is a single image corresponding to a single mask label, and the test data set is a single mask label corresponding to a single original image.

[0017] Optionally, the dual-branch encoder includes a ViT branch and a CNN branch, as well as a fine-tuning strategy. The output of the ViT branch that integrates the LoRA parameters includes deep semantic features and shallow intermediate features. The deep semantic features are used to characterize the overall morphological structure of the polyp, and the shallow intermediate features retain the edge information and mucosal boundary features in the image. The CNN branch extracts multi-scale features of the image through strip convolution operations. The strip convolution decomposes the convolution kernel in the horizontal and vertical directions respectively, thereby reducing the computational complexity while maintaining the original receptive field size, and can adapt to polyp areas of different sizes.

[0018] Optionally, the ViT branch: in the SAM-Med2D model, calls the pre-trained visual Transformer backbone network ViT-b and loads the corresponding weight parameters. In the ViT branch, all parameters remain frozen to retain its semantic modeling capabilities obtained in large-scale medical image pre-training;

[0019] Optionally, the CNN branch uses the MSCAN-Tiny network released by SegNeXt as a lightweight convolutional neural network branch, the parameters in the lightweight convolutional neural network branch remain trainable, and its multi-scale structure is designed to extract fine-grained feature information of the mucosal surface in the image;

[0020] Optionally, the fine-tuning strategy introduces a LoRA parameter fine-tuning strategy in the multi-head attention layer of the Transformer module, introducing only a small number of trainable parameters for the query and value projection matrices, thereby achieving a balance between feature adaptability and parameter efficiency while keeping the backbone parameters frozen;

[0021] Optionally, the S4 specifically includes:

[0022] S41. First, a bidirectional cross-attention mechanism is introduced between the image branch and the CNN branch in the original structure, including cross-attention from image to CNN and cross-attention from CNN to image.

[0023] S42. Secondly, in the feature fusion module, the image embedding, ViT intermediate layer output and CNN branch features are aligned in spatial size and channel dimension, and the three types of features are fused by pixel-by-pixel addition to construct a complementary representation that integrates global semantic features and local detail information, which is used for feature analysis and region discrimination in subsequent segmentation tasks.

[0024] Optionally, the S5 specifically includes:

[0025] S51. Hint coding supports both sparse and dense hints. The sparse hints include point hints and box hints, which are converted into corresponding embedding vectors through position encoding. The dense hints include mask hints, which are feature encoded through a convolutional encoder.

[0026] S52. In the improved mask decoder, the hint embedding and fusion features are input into the multi-layer perceptron network together, and the polyp segmentation mask and its corresponding IoU score are output.

[0027] Optionally, the S6 specifically includes:

[0028] S61. Let the initial mask be M. First, activate M with a Sigmoid function to obtain a pixel-level probability map. Then, construct a reverse attention map by inverting the operation. The reverse attention map is used to indicate the spatial distribution of low-confidence areas in the prediction, which serves as a weighted reference for the error area.

[0029] S62, introduce the spatial attention mechanism to the fusion feature to enhance the local response, and then multiply it element-by-element with the reverse attention map after bilinear upsampling to further highlight the area with errors in the initial mask. The resulting feature is then input into the channel attention module to perform feature recalibration in the channel dimension, and finally fused with the spatial attention feature through the residual connection method to generate a refined mask. ;

[0030] S63, in order to balance the information of the initial mask and the refined mask, an interpolation fusion strategy is used to obtain the final prediction mask .

[0031] Optionally, the S7 specifically includes:

[0032] S71. Use the AdamW optimizer and set different weight decay coefficients for different model parameters to suppress model overfitting;

[0033] S72. The Dice loss function, Focal loss function and MaskIoU loss function are jointly used to optimize the model. The Dice loss function is used to enhance the pixel overlap between the predicted mask and the true label, the Focal loss function is used to improve the recognition ability of difficult-to-distinguish samples, and the MaskIoU loss function is used to constrain the consistency between the predicted mask confidence and the actual intersection-over-union ratio. The model performance evaluation indicators include Dice score and IoU score.

[0034] The beneficial effects of the present invention are:

[0035] First, by introducing the ViT-CNN dual-branch encoder and cross-attention fusion module, the global semantic features and local structural details are fully integrated, which significantly improves the representation ability of multi-scale and multi-morphological polyp areas in endoscopic images, thereby enhancing the adaptability of the segmentation model to complex backgrounds and areas with blurred boundaries.

[0036] Secondly, the LoRA low-rank adaptive fine-tuning strategy is adopted to significantly reduce the computing resource consumption during training while maintaining stable model performance, thereby improving the adaptation efficiency of large models in specific medical scenarios.

[0037] Finally, by integrating a lightweight reverse attention refinement module, the model effectively strengthens its focus on low-confidence boundary regions, further correcting initial segmentation errors and improving segmentation accuracy and robustness. In summary, this invention significantly improves the segmentation of polyp regions in medical images while ensuring computational efficiency, and has excellent practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0039] Figure 1 Flowchart of the robust polyp segmentation method based on improved SAM-Med2D proposed in the present invention;

[0040] Figure 2 This is the overall structural flow chart of the robust polyp segmentation method based on improved SAM-Med2D proposed in the present invention;

[0041] Figure 3 Schematic diagram of the internal structure of the ViT branch encoder corresponding to the dual-branch encoder of the robust polyp segmentation method based on the improved SAM-Med2D proposed in the present invention;

[0042] Figure 4The double-branch encoder corresponding to the improved SAM-Med2D-based robust polyp segmentation method proposed in the application is a CNN branch encoder, and a schematic diagram of an internal structure of the CNN branch encoder is shown in the figure;

[0043] Figure 5 An internal structure schematic diagram of an improved mask decoder in the improved SAM-Med2D-based robust polyp segmentation method proposed in the application is shown in the figure;

[0044] Figure 6 The qualitative effect comparison of the improved SAM-Med2D-based robust polyp segmentation method proposed in the application on five public data sets is shown in the figure. DETAILED DESCRIPTION

[0045] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams and only show the basic structure of the application in a schematic manner, and thus only show the components related to the application.

[0046] REFERENCE Figure 1-3 The improved SAM-Med2D-based robust polyp segmentation method comprises the following steps:

[0047] S1. For a given data set, a JSON format record file is generated to describe the correspondence between the image and the mask label;

[0048] S2. A SAM-Med2D model is constructed, and the original image and the mask label are uniformly scaled to 256x256 pixels by bilinear interpolation, and an edge padding strategy is used to maintain the aspect ratio for non-square images;

[0049] S3. The multi-scale features of the image are extracted by a double-branch encoder;

[0050] S4. In the mask decoder, a cross-attention mechanism and a feature fusion module are used to fuse and process the multi-scale features;

[0051] S5. The preliminary segmentation result is generated based on the fused features and the prompt encoding;

[0052] S6. A lightweight reverse attention refinement module is constructed to correct the errors of the preliminary segmentation result;

[0053] S7. A multi-objective loss joint optimization strategy is used to calculate the loss and optimize the parameters of the SAM-Med2D model;

[0054] S8. The polyp segmentation of the current image is ended, and the next image is entered, and steps S1 to S7 are repeatedly executed.

[0055] The present invention constructs an improved SAM-Med2D segmentation process, systematically integrating data preprocessing, feature extraction, attention fusion, segmentation prediction and optimization refinement mechanism, to achieve high-precision, end-to-end segmentation processing of polyp images, and improve the robustness and generalization ability of the model in complex scenarios.

[0056] In this embodiment, the dataset includes a training dataset and a test dataset. The training dataset is a single image corresponding to a single mask label, and the test dataset is a single mask label corresponding to a single original image, specifically:

[0057] In the JSON file corresponding to the training dataset, the format of each record is:

[0058] ". / TrainDataset / images / image1.png:. / TrainDataset / masks / mask1.png";

[0059] In the JSON file corresponding to the test dataset, the format of each record is:

[0060] ". / TestDataset / masks / mask1.png:. / TestDataset / images / image1.png".

[0061] By clearly distinguishing the image-mask pairing methods of the training set and the test set, the present invention optimizes the data loading and model evaluation processes, and improves the training efficiency and the stability of the inference stage.

[0062] In this embodiment, S3 specifically includes:

[0063] S31, ViT branch: In the SAM-Med2D model, the pre-trained visual Transformer backbone network ViT-b is called and the corresponding weight parameters are loaded. In the ViT branch, all parameters remain frozen to preserve the semantic modeling capabilities obtained in large-scale medical image pre-training;

[0064] S32, CNN branch: The MSCAN-Tiny network released by SegNeXt is used as a lightweight convolutional neural network branch. The parameters in the lightweight convolutional neural network branch remain trainable, and its multi-scale structure is designed to extract fine-grained feature information of the mucosal surface in the image. Let F be the input feature, To output features, the process can be formally expressed as:

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] Among them, DW and SC represent depth-wise separable convolution and strip convolution respectively. It is the result obtained after the input feature undergoes depth-wise separable convolution. Represents the composite operation of two strip convolutions. Attn is the attention feature map obtained by residual fusion and then convolution. The convolution kernel size H is taken as 7, 11 and 21 respectively. represents the element-wise product operation, Represents the dot product operator;

[0070] S33, Fine-tuning Strategy: Introducing the LoRA parameter fine-tuning strategy in the multi-head attention layer of the Transformer module, only introducing a small number of trainable parameters for the query and value projection matrices, thereby achieving a balance between feature adaptability and parameter efficiency while keeping the backbone parameters frozen;

[0071] S34. For a pre-trained weight matrix , its updated value Can be decomposed into two low-rank matrices and The product of , where the rank r satisfies , A is initialized by Gaussian distribution, B is initialized by zero, and the LoRA calculation formula is:

[0072] ;

[0073] Among them, F is the input feature, is the output feature, is the updated weight matrix;

[0074] S35. Based on this, the LoRA parameters are injected into the self-attention calculation of the Transformer module in the following form:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] Among them, Q, K, V represent the query, key and value in the attention calculation process respectively, b represents the bias term, and d is the dimension of the input feature. , and Represent the frozen pre-trained projection matrix, , , Represent the query weight matrix, key weight matrix and value weight matrix respectively, are trainable LoRA parameters, represents the activation function;

[0080] S36. The output of the ViT branch that integrates LoRA parameters includes deep semantic features and shallow intermediate features. The deep semantic features are used to characterize the overall morphological structure of the polyp, and the shallow intermediate features retain the edge information and mucosal boundary features in the image. The CNN branch extracts multi-scale features of the image through strip convolution operations. The strip convolution decomposes the convolution kernel in the horizontal and vertical directions respectively, reducing the computational complexity while maintaining the original receptive field size, and can adapt to polyp areas of different sizes.

[0081] This paper introduces the dual-branch structure of ViT and CNN, and combines it with the efficient fine-tuning strategy of LoRA parameters to effectively integrate global semantics and local structural features, thereby improving the fineness and adaptability of segmentation while ensuring computational efficiency.

[0082] In this embodiment, the S4 specifically includes:

[0083] S41. First, a bidirectional cross-attention mechanism is introduced between the image branch and the CNN branch in the original structure, which specifically includes the following two parts:

[0084] Cross-Attention from Image to CNN, denoted as CrossAttnIC, with features extracted from ViT branch As a query, , CNN branch features As keys and values, respectively and :

[0085] ;

[0086] Cross-Attention from CNN to image, denoted as CrossAttnCI, is based on CNN branch features. For query, record it as , ViT features are keys and values, respectively and ;

[0087] ;

[0088] in, and Represent the channel dimensions of the attention heads in the ViT branch and the CNN branch respectively;

[0089] S42. Secondly, in the feature fusion module, the image embedding, ViT intermediate layer output and CNN branch features are aligned in spatial size and channel dimension, and the three types of features are fused by pixel-by-pixel addition to construct a complementary representation that integrates global semantic features and local detail information, which is used for feature analysis and region discrimination in subsequent segmentation tasks.

[0090] The present invention improves the information interaction capability of features from different sources by constructing a bidirectional cross-attention mechanism and a multi-feature fusion structure, thereby enhancing the model's ability to discriminate polyp areas.

[0091] In this embodiment, the S5 specifically includes:

[0092] S51. Hint coding supports both sparse and dense hints. The sparse hints include point hints and box hints, which are converted into corresponding embedding vectors through position encoding. The dense hints include mask hints, which are feature encoded through a convolutional encoder.

[0093] S52. In the improved mask decoder, the hint embedding and fusion features are input into the multi-layer perceptron network together, and the polyp segmentation mask and its corresponding IoU score are output.

[0094] The present invention combines sparse and dense prompt encoding methods to improve the model's adaptability to human-labeled or automatically generated prompt information, enabling the model to have good segmentation consistency and accuracy in a variety of interaction scenarios.

[0095] In this embodiment, S6 specifically includes:

[0096] S61. Let the initial mask be M. First, activate M with Sigmoid function to obtain pixel-level probability map, and then construct the reverse attention map by inversion operation. , the reverse attention map is used to indicate the spatial distribution of low confidence areas in the prediction, which serves as a weighted reference for the error area;

[0097] S62, introduce the spatial attention mechanism to the fusion feature to enhance the local response, and then multiply it element-by-element with the reverse attention map after bilinear upsampling to further highlight the area with errors in the initial mask. The resulting feature is then input into the channel attention module to perform feature recalibration in the channel dimension, and finally fused with the spatial attention feature through the residual connection method to generate a refined mask. , the formal process is expressed as follows:

[0098] ;

[0099] ;

[0100] Among them, F is the fusion feature, SA, RA and CA represent the spatial attention, reverse attention and channel attention modules respectively. It is an intermediate feature, and Head represents the prediction head structure used to output the mask;

[0101] S63, in order to balance the information of the initial mask and the refined mask, an interpolation fusion strategy is used to obtain the final prediction mask , the calculation formula is as follows:

[0102] ;

[0103] in, For fusion weight hyperparameters, they can be adjusted according to task requirements to control the impact of the degree of refinement on the final result.

[0104] The present invention combines the reverse attention mechanism with a multi-layer attention structure to enhance the error correction capability of low-confidence areas and improve the accuracy and refinement of segmentation boundaries.

[0105] In this embodiment, the S7 specifically includes:

[0106] S71. Use the AdamW optimizer and set different weight decay coefficients for different model parameters;

[0107] S72. Use multiple loss functions for joint optimization, including Dice loss function, Focal loss function and MaskIoU loss function. Among them, the Dice loss function is used to maximize the pixel overlap between the predicted mask and the true label, the Focal loss function is used to enhance the recognition ability of difficult samples, and the MaskIoU loss function is used to constrain the consistency of the predicted confidence and the true intersection over union ratio. The indicators for evaluating model performance include Dice score and IoU score. The total calculation formula of the loss function is:

[0108] .

[0109] By combining multiple loss functions and fine-grained control strategies for optimizer parameters, the present invention effectively improves the model's ability to handle complex situations such as category imbalance and blurred boundaries, thereby enhancing overall training stability and segmentation performance.

[0110] Example 1:

[0111] To verify the feasibility of the present application in implementation, the present application is applied to a certain typical endoscopic image segmentation task, and the target is to solve the problems of low segmentation accuracy and poor model generalization ability of existing polyp segmentation methods in processing complex background, multi-scale lesions and blurred boundary conditions. The selected images come from five mainstream polyp segmentation evaluation benchmarks, including CVC-ClinicDB, Kvasir, CVC-ColonDB, ETIS and EndoScene. These datasets cover endoscopic images of different types, resolutions and lesion complexity, and are widely used for performance evaluation of polyp segmentation tasks, and have high representativeness. The method of the present application maximizes the preservation of spatial structure information by uniformly processing the image and label to the same size in the input stage and maintaining the original aspect ratio; then a ViT-CNN dual-branch structure is used to extract multi-scale features, and a cross-attention fusion mechanism is used to fuse global and local features, which significantly enhances the model's ability to recognize multi-morphology lesions; At the same time, the LoRA fine-tuning strategy is introduced, which greatly reduces the model training cost and further improves the deployment efficiency; in the prediction stage, a lightweight reverse attention module is used to refine the boundary region, achieving higher-precision segmentation output; finally, a multi-objective joint loss is used for optimization, which enhances the robustness of the model in the unbalanced sample and small target scene.

[0112] Table 1 Comparison of the present application method and other methods on five public datasets

[0113] ;

[0114] In actual application tests, the present application method is compared with more than 20 current mainstream segmentation algorithms, covering traditional CNN-based segmentation methods (such as UNet, PraNet, SANet, etc.) and various improved methods based on the SAM framework (such as SAM-L, SAM-Adapter, AutoSAM, MedSAM, etc.). As can be seen from Table 1, the present application achieves the best performance on four of the five datasets, which are:

[0115] Kvasir dataset: Dice score of 93.5, IoU of 88.8, significantly better than all existing methods, among which the traditional optimal method Polyp-PVT is 91.7 / 86.4, and the SAM framework optimal ASPS-L is 91.4 / 84.8;

[0116] CVC-ColonDB dataset: Dice score of 90.6, IoU of 83.8, significantly higher than ASPS-T (88.5 / 79.2) and SAMed (81.2 / 66.5);

[0117] ETIS dataset: Dice score is 88.4, IoU is 80.9, far exceeding SAM-L (55.1 / 50.7) and AutoSAM (40.2 / 30.8);

[0118] EndoScene dataset: Dice score of 93.1, IoU of 87.4, also surpassing all comparison models, including ASPS-L (91.4 / 84.8) and Polyp-PVT (90.2 / 83.7) with higher parameter counts;

[0119] CVC-ClinicDB dataset: Although slightly lower than ASPS-L (95.0 / 90.5) and Polyp-PVT (93.7 / 88.9), our method still ranked third with a Dice score of 92.5 and an IoU score of 86.9, maintaining strong competitiveness.

[0120] It's important to note that while ASPS-L boasts leading advantages in certain metrics, it's built on the MSCAN-Large network, which uses approximately five times as many parameters as the MSCAN-Tiny. This consumes significant computational resources and makes it unsuitable for deployment in resource-constrained medical settings. The MSCAN-Tiny architecture and LoRA fine-tuning strategy employed in this paper significantly reduce model parameters and inference time while maintaining performance, improving the practicality and flexibility of model deployment.

[0121] In summary, this example demonstrates the outstanding performance of the present invention in the task of endoscopic polyp image segmentation. Whether in small object recognition, boundary refinement, or cross-scale adaptation, the present invention demonstrates exceptional robustness and accuracy. Combined with its advantages across five datasets, the present invention demonstrates its broad potential for application in medical image-assisted diagnosis.

[0122] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A robust polyp segmentation method based on improved SAM-Med2D, characterized by: The steps include: S1. For a given dataset, generate a record file in JSON format to describe the correspondence between images and mask labels; S2. Build the SAM-Med2D model, uniformly scale the original image and mask labels to 256 × 256 pixels through bilinear interpolation, and use edge padding strategy to maintain the aspect ratio for non-square images; S3, extracting multi-scale features of the image through a dual-branch encoder; The dual-branch encoder includes a ViT branch and a CNN branch, as well as a fine-tuning strategy. The ViT branch outputs deep semantic features and shallow intermediate features after fusing LoRA parameters. The deep semantic features are used to characterize the overall morphological structure of the polyp, and the shallow intermediate features retain the edge information and mucosal boundary features in the image. The CNN branch extracts multi-scale features of the image through strip convolution operations. The strip convolution decomposes the convolution kernel in the horizontal and vertical directions respectively, reducing computational complexity while maintaining the original receptive field size, and can adapt to polyp areas of different sizes. The ViT branch: In the SAM-Med2D model, the pre-trained visual Transformer backbone network ViT-b is called and the corresponding weight parameters are loaded. In the ViT branch, all parameters remain frozen to retain the semantic modeling capabilities obtained in large-scale medical image pre-training; The CNN branch uses the MSCAN-Tiny network released by SegNeXt as a lightweight convolutional neural network branch. The parameters in the lightweight convolutional neural network branch remain trainable, and its multi-scale structure is designed to extract fine-grained feature information of the mucosal surface in the image; The fine-tuning strategy introduces the LoRA parameter fine-tuning strategy in the multi-head attention layer of the Transformer module, introducing only a small number of trainable parameters for the query and value projection matrices, thereby achieving a balance between feature adaptability and parameter efficiency while keeping the backbone parameters frozen; S4. In the mask decoder, a cross-attention mechanism and a feature fusion module are used to fuse multi-scale features. S5, generating preliminary segmentation results based on fusion features and hint coding; S6. Build a lightweight reverse attention refinement module to correct errors in the initial segmentation results; S7, using a multi-objective loss joint optimization strategy to perform loss calculation and parameter optimization on the SAM-Med2D model; S8. End the polyp segmentation of the current image, proceed to the next image, and repeat steps S1 to S7.

2. The robust polyp segmentation method based on improved SAM-Med2D according to claim 1, characterized in that: The dataset includes a training dataset and a test dataset. The training dataset is a single image corresponding to a single mask label, and the test dataset is a single mask label corresponding to a single original image.

3. The robust polyp segmentation method based on improved SAM-Med2D according to claim 1, characterized in that: The S4 specifically includes: S41. First, a bidirectional cross-attention mechanism is introduced between the image branch and the CNN branch in the original structure, including cross-attention from image to CNN and cross-attention from CNN to image. S42. Secondly, in the feature fusion module, the image embedding, ViT intermediate layer output and CNN branch features are aligned in spatial size and channel dimension, and the three types of features are fused by pixel-by-pixel addition to construct a complementary representation that integrates global semantic features and local detail information, which is used for feature analysis and region discrimination in subsequent segmentation tasks.

4. The robust polyp segmentation method based on improved SAM-Med2D according to claim 1, characterized in that: The S5 specifically includes: S51. Hint coding supports both sparse and dense hints. The sparse hints include point hints and box hints, which are converted into corresponding embedding vectors through position encoding. The dense hints include mask hints, which are feature encoded through a convolutional encoder. S52. In the improved mask decoder, the hint embedding and fusion features are input into the multi-layer perceptron network together, and the polyp segmentation mask and its corresponding IoU score are output.

5. The robust polyp segmentation method based on improved SAM-Med2D according to claim 1, characterized in that: The S6 specifically includes: S61, set the initial mask to , first of all Perform Sigmoid function activation to obtain a pixel-level probability map, and then construct a reverse attention map through the inversion operation. The reverse attention map is used to indicate the spatial distribution of low-confidence areas in the prediction, which serves as a weighted reference for the error area. S62, introduce the spatial attention mechanism to the fusion feature to enhance the local response, and then multiply it element-by-element with the reverse attention map after bilinear upsampling to further highlight the area with errors in the initial mask. The resulting feature is then input into the channel attention module to perform feature recalibration in the channel dimension, and finally fused with the spatial attention feature through the residual connection method to generate a refined mask. ; S63, in order to balance the information of the initial mask and the refined mask, an interpolation fusion strategy is used to obtain the final prediction mask .

6. The robust polyp segmentation method based on improved SAM-Med2D according to claim 1, characterized in that: The S7 specifically includes: S71. Use the AdamW optimizer and set different weight decay coefficients for different model parameters to suppress model overfitting; S72. The Dice loss function, Focal loss function and MaskIoU loss function are jointly used to optimize the model. The Dice loss function is used to enhance the pixel overlap between the predicted mask and the true label, the Focal loss function is used to improve the recognition ability of difficult-to-distinguish samples, and the MaskIoU loss function is used to constrain the consistency between the predicted mask confidence and the actual intersection-over-union ratio. The model performance evaluation indicators include Dice score and IoU score.

Images