A MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction

Abstract

This application provides a method for constructing an MSMP-SAM image encoder based on multi-pooling feature adaptive extraction, relating to the field of general image data processing technology. It processes images through image analysis and region segmentation. This application constructs a multi-pooling feature adaptive extraction module (MSMP), where multi-scale feature extraction can simultaneously cover diffuse large-area lesions and small local lesions in diabetic retinopathy (DR), effectively solving the problem of significant scale differences in DR lesions. Multi-pooling can take into account the detailed texture and global structural information of different types of lesions, improving the segmentation completeness and localization accuracy of diverse lesions. Through the collaborative design of multi-pooling and multi-scale, this application fundamentally improves the model's feature perception and segmentation capabilities for DR, significantly enhancing segmentation accuracy in medical imaging scenarios and further strengthening the model's clinical adaptability and practical value.

Description

Technical Field

[0001] This application relates to the field of image semantic segmentation, and in particular to a large-model segmentation method based on adaptive extraction of multi-pooling features, namely MSMP-SAM. Background Technology

[0002] Diabetic retinopathy (DR) is one of the most common chronic microvascular complications of diabetes and the leading cause of irreversible blindness in working-age populations worldwide. With the increasing prevalence of diabetes, the number of people affected by DR continues to expand, making it a major public health issue. The disease progresses insidiously, often without obvious symptoms in its early stages, making it easily overlooked by patients. As blood sugar remains poorly controlled over a long period, retinal microvascular endothelial cells are damaged, leading to progressive microvascular occlusion and leakage, which in turn induces pathological changes such as microaneurysms, hemorrhages, and exudations. Later stages can result in serious complications such as retinal detachment and neovascular glaucoma. Once it reaches an advanced stage, visual impairment is completely irreversible. Therefore, precise monitoring and intervention are crucial for preventing blindness caused by DR.

[0003] Traditional convolutional segmentation models and deep learning methods mostly rely on massive, finely annotated medical datasets, limiting their generalization ability and making them poorly adaptable to low-contrast, high-quality clinical fundus images. This further widens the gap in the clinical application of automatic segmentation of DR lesions. SAM, as a representative large-scale visual model, precisely addresses the core shortcomings of traditional DR segmentation models, becoming an ideal model for segmenting multiple fundus lesions. Yin Jianqin's "A Multi-Task Whole-Heart CT Segmentation Method Based on Pooling Transformer (CN202310609850.0)" explores the roles of average pooling and max pooling as self-attention mechanisms in cardiac CT segmentation tasks. Feng Tian's "A Retinal Vessel Segmentation Method Based on Cross-Attention and Bi-branch Pooling Fusion (CN202311752136.3)" introduces average pooling and max pooling layers to input features into standard convolutional layers, reducing the loss of vascular spatial information. However, the above studies only involve average pooling and max pooling, without considering soft pooling, and all belong to deep learning methods. Pooling design under the SAM large model has not received sufficient attention and research. Summary of the Invention

[0004] The purpose of this application is to address the technical problem that pooling design under the SAM large model has not received sufficient attention and research in the prior art.

[0005] To address the shortcomings of existing technologies, this application provides the following technical solution:

[0006] A large-scale MSMP-SAM model segmentation method based on adaptive extraction of multi-pooling features includes the following steps:

[0007] S1: Data Acquisition and Preprocessing: The data is a public dataset of diabetic retinopathy, and then the image data of diabetic retinopathy from different patients are preprocessed.

[0008] S2: Constructing the Multi-Pooling Feature Adaptive Extraction Module (MSMP)

[0009] S2 includes the following steps:

[0010] S21: Feature map output by the visual Transformer multi-head attention mechanism based on the SAM image encoder;

[0011] S22: MLP is used to process the feature map of the pooling branch to obtain the feature map channel weights at each sampling scale, and then the upper and lower limits of the weights are determined based on the feature map channel weights.

[0012] S23: Introduce trainable feature vectors to perform weighted fusion of the upper and lower bounds of the feature map channel weights to obtain channel importance. ;

[0013] S24: For different sampling scales Upsampling is performed to restore the shape of the feature map, and the upsampled feature maps are then concatenated to obtain the final fused feature map. ;

[0014] S3: Introduce the MSMP constructed in S2 into the SAM model to build the MSMP-SAM model and train it;

[0015] S4: Visualization results of automatic identification and segmentation of diabetic retinopathy based on the trained MSMP-SAM model.

[0016] Preferably, the preprocessing in S1 includes data validity detection, removal of invalid data, unification of valid image format, standardization conversion of labeled mask images, matching of valid images with labeled mask images, splitting of subsets, and splitting of training and test sets.

[0017] Preferably, in step S1, color image data of diabetic retinopathy from patients in the public dataset FGADR is first extracted, and then the extracted image data is cleaned, the validity of each data is checked, and all invalid data is removed.

[0018] Preferably, after removing invalid data, step S1 performs dimensionality standardization on the valid image data, uniformly adjusting it to the 1024×1024 pixel specification commonly used in the SAM model to obtain uniformly sized color fundus images. The labeled mask image is then standardized and converted, and the pixel values ​​of the labeled mask are redefined according to preset rules to obtain standardized labeled mask images. The uniformly sized color fundus images are matched one by one with the standardized labeled mask images to generate a preprocessed color image dataset M.

[0019] Preferably, in S1, the new dataset M is split into four sub-datasets: microaneurysm sub-dataset, hemorrhage sub-dataset, soft exudate sub-dataset, and hard exudate sub-dataset, with each sub-dataset serving as a dedicated dataset for independent input to the model.

[0020] In step S1, the new dataset M is randomly divided into a training set and a test set in a ratio of 8:2. At the same time, each sub-dataset is simultaneously split in a ratio of 8:2 to ensure the consistency of the distribution of each lesion type sub-dataset in the training set and the test set.

[0021] Preferably, the specific steps of S21 are as follows:

[0022] S211. Employing the visual Transformer inherent in the SAM image encoder, after the multi-head attention mechanism of each visual Transformer, the input feature map is processed. Perform multi-scale convolution kernel operations to obtain feature maps at three scales. ,in The size is , The size is , Indicates the sampling scale;

[0023] S212, Feature map after multi-scale convolution kernel operation as described in S211 Perform multi-pooling operations;

[0024] Multi-pooling operations are divided into three branches: max pooling, average pooling, and soft pooling. The feature maps after the three pooling operations are represented as follows: , .

[0025] Preferably, the specific steps of S22 are as follows:

[0026] S221: Use MLP to process the feature map described in S21 , Further processing to obtain Feature map channel weights at different scales , , Specifically, it is expressed as follows:

[0027]

[0028] in, Represents the ReLU activation function. This represents the Sigmoid activation function, which maps input values ​​to the range of 0 to 1. and They represent Weight matrices of two fully connected layers at the sampling scale;

[0029] S222: Feature map channel weights based on S221 , , The maximum value among the three is selected as the upper bound of the feature map weights. The minimum value among the three is selected as the lower bound of the feature map weights. .

[0030] Preferably, steps S23 and S24 are as follows:

[0031] Introduce two sizes of Trainable feature vectors and The upper and lower bounds of the feature map channel weights described in S222 are weighted and fused to obtain the channel importance. Specifically, it is expressed as follows:

[0032]

[0033] in, and Each element in the array is between 0 and 1, and and The sum of corresponding elements in the array is equal to 1;

[0034] Then, at different sampling scales Upsampling is performed to restore the shape of the feature map to... The upsampled feature maps are then concatenated to obtain the final fused feature map. Specifically, it is expressed as follows:

[0035]

[0036] in, Represents learnable scale weights, satisfying .

[0037] Preferably, step S3 includes the following steps:

[0038] S31: The multi-pooling feature adaptive extraction module MSMP described in S2 is introduced into the image encoder of the SAM model, and after being arranged with the multi-head attention mechanism of each visual Transformer, it forms the MSMP-SAM model.

[0039] S32: The preprocessed image is input to the image encoder, and the image embedding is output through image block embedding and multiple multi-layer visual Transformers described in S31;

[0040] S33: The prompt encoder converts manually entered box prompts into automatic prompt embeddings;

[0041] S34: The image embedding described in S32 and the cue embedding described in S33 are dimensionally aligned and spatially fused through the Conv layer, and then input into the mask decoder to generate a prediction mask;

[0042] S35: Compare the prediction mask described in S34 with the multi-lesion labeling mask described in S14, and perform iterative training using a linear combination of focus loss, cross-entropy loss, and dice loss, as specifically shown below:

[0043]

[0044] S36: Select Nadam as the optimizer, set the batch size to 8, and the learning rate to 0.0006. Update the adapter parameters to minimize the multi-task loss function value.

[0045] S37: Set a threshold for loss value fluctuation. When the loss value fluctuation is lower than this threshold, the model has reached convergence and training stops.

[0046] S38: Calculate key metrics such as Dice, IoU, AUC-PR, and HD95, record the model weights corresponding to the best performance, and save the current model state.

[0047] Preferably, the specific steps of S4 are as follows:

[0048] S41: Input the preprocessed test set images into the trained MSMP-SAM model, perform forward inference, automatically output the pixel-level segmentation mask of diabetic retinopathy, and overlay the original fundus image to generate a visualization of the lesion area, clearly distinguishing microaneurysms, hemorrhages, soft exudates, hard exudates and background areas.

[0049] S42: Based on the segmentation mask output by the model and the ground truth labeled mask, calculate multiple core indicators for medical image segmentation, including Dice, IoU and HD95, and statistically analyze the quantitative performance of overall lesions and four types of single diseases.

[0050] Compared with the prior art, this application has at least the following beneficial effects:

[0051] 1. This application constructs a multi-pooling feature adaptive extraction module MSMP, in which multi-scale feature extraction can simultaneously cover diffuse large-area lesions and small local lesions in DR, effectively solving the problem of huge differences in the scale of DR lesions; while multi-pooling can take into account the detailed texture and global structural information of different types of lesions, improving the segmentation integrity and localization accuracy of diverse lesions.

[0052] 2. This application implements fine-tuning of the adapter module of the SAM large model, breaking through the original fixed-scale feature extraction paradigm of SAM and constructing a multi-scale feature branch architecture: the small-scale branch can achieve accurate localization of small lesions such as microaneurysms and microbleeds, while the large-scale branch can complete the complete segmentation of diffuse lesion areas such as exudation and edema.

[0053] 3. To address the issues of native SAM models being sensitive to noise and lacking robustness in complex fundus backgrounds, this application introduces a multi-pooling feature complementarity extraction mechanism to synergistically enhance lesion features in different receptive fields. This enables the model to stably identify lesions even in complex fundus backgrounds, effectively reducing missegmentation and missed segmentation rates. Through the collaborative design of multi-pooling and multi-scale methods, the model's feature perception and segmentation capabilities for DR are fundamentally improved, significantly enhancing segmentation accuracy in medical imaging scenarios and further strengthening the model's clinical adaptability and practical value. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating an embodiment of the MSMP-SAM large model segmentation method based on adaptive extraction of multi-pooling features in this application;

[0055] Figure 2 yes Figure 1 A flowchart illustrating step S1 of an embodiment;

[0056] Figure 3 yes Figure 1 A flowchart illustrating step S2 of an embodiment;

[0057] Figure 4 yes Figure 3 A flowchart illustrating step S21 of an embodiment;

[0058] Figure 5 yes Figure 3 A flowchart illustrating step S22 of an embodiment;

[0059] Figure 6 This is a model architecture overview diagram of an embodiment of the MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction in this application;

[0060] Figure 7 yes Figure 6 A schematic diagram of the multi-pooling feature adaptive extraction module structure designed in each Transformer layer;

[0061] Figure 8 This is a visualization of the algorithm comparison results of an embodiment of the MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction in this application. Figure 1 ;

[0062] Figure 9 This is the loss function iteration curve plotted during the training process of the MSMP-SAM large model based on multi-pooling feature adaptive extraction in this application;

[0063] Figure 10 yes Figure 7 Lower bound of feature map weights The switching frequency distribution diagram. Detailed Implementation

[0064] Please see Figure 1 This application provides a large-model segmentation method for MSMP-SAM based on adaptive extraction of multi-pooling features, which includes the following steps:

[0065] S1: Obtain the public dataset FGADR for diabetic retinopathy and preprocess the image data of diabetic retinopathy from different patients;

[0066] Specifically, in one implementation, please refer to Figure 2 The S1 step is as follows:

[0067] S11. Extract color image data of diabetic retinopathy from patients in the public dataset FGADR;

[0068] S12. Clean all extracted image data, check the validity of each data (including but not limited to invalid cases such as image damage, blurred content, and missing annotation information), and remove all invalid data;

[0069] S13. Perform dimensionality standardization on the cleaned effective image data and uniformly adjust it to the 1024×1024 pixel specification commonly used in the SAM model;

[0070] S14. Standardize the labeled mask image and redefine the pixel values ​​of the labeled mask according to preset rules: set the pixel values ​​of the background area to (0,0,0), the pixel values ​​of the microaneurysm area to (128,128,128), the pixel values ​​of the hemorrhage area to (128,0,0), the pixel values ​​of the soft exudate area to (0,128,0), and the pixel values ​​of the hard exudate area to (0,0,128).

[0071] S15. Match the color fundus images after size unification in step S13 with the standardized labeled mask images in step S14 according to the patient dimension to generate a preprocessed color image dataset M.

[0072] S16. Based on the new dataset obtained in step S15, split it into 4 sub-datasets according to lesion type: microaneurysm sub-dataset, hemorrhage sub-dataset, soft exudate sub-dataset, and hard exudate sub-dataset. Each sub-dataset is used as the dedicated dataset for independent input to the model.

[0073] S17. The new dataset M obtained in step S15 is randomly divided into a training set and a test set in a ratio of 8:2. At the same time, each sub-dataset in step S16 is also split in a corresponding ratio of 8:2 to ensure the consistency of the distribution of each lesion type sub-dataset in the training set and the test set.

[0074] S2: Construct the Multi-Pooling Feature Adaptive Extraction Module (MSMP);

[0075] In one implementation, please refer to Figure 3 The specific steps of S2 are as follows:

[0076] S21. The feature map output by the visual Transformer multi-head attention mechanism based on the SAM image encoder is first generated by multi-scale convolution kernel operation to generate feature maps of different sampling scales. Then, the feature maps of each scale are subjected to three types of multi-pooling operations in parallel: max pooling, average pooling, and soft pooling, to obtain the feature map of the corresponding branch.

[0077] For details, please refer to Figure 4 The specific steps of S21 are as follows:

[0078] S211. Employing the visual Transformer inherent in the SAM image encoder, after the multi-head attention mechanism of each visual Transformer, the input feature map is processed. Perform multi-scale convolution kernel operations to obtain feature maps at three scales. ,in The size is , The size is , Indicates the sampling scale;

[0079] S212, Feature map after multi-scale convolution kernel operation as described in S211 Perform multi-pooling operations. Multi-pooling operations consist of three branches: max pooling, average pooling, and soft pooling. The feature maps after each of these three pooling operations are represented as follows: , ;

[0080] S22. The feature maps of the above three types of pooling branches are processed using MLP to obtain the feature map channel weights at each sampling scale, and then the upper and lower limits of the weights are determined based on the channel weights at each scale.

[0081] For details, please refer to Figure 5 The specific steps of S22 are as follows:

[0082] S221. Use MLP to process the feature map described in S21. , Further processing to obtain Feature map channel weights at different scales , , Specifically, it is expressed as follows:

[0083]

[0084] in, This represents the ReLU activation function. This represents the Sigmoid activation function, which maps input values ​​to the range of 0 to 1. and They represent Weight matrices of two fully connected layers at the sampling scale;

[0085] S222. Based on the feature map channel weights described in S221 , , The maximum value among the three is selected as the upper bound of the feature map weights. The minimum value among the three is selected as the lower bound of the feature map weights. .

[0086] S23. Introduce two sizes of Trainable feature vectors and The upper and lower bounds of the feature map channel weights described in S222 are weighted and fused to obtain the channel importance. Specifically, it is expressed as follows:

[0087]

[0088] in, and Each element in the array is between 0 and 1, and and The sum of corresponding elements in the array is equal to 1;

[0089] S24. For different sampling scales Upsampling is performed to restore the shape of the feature map to... The upsampled feature maps are then concatenated to obtain the final fused feature map. Specifically, it is expressed as follows:

[0090]

[0091] in, Represents learnable scale weights, satisfying .

[0092] S3: Introduce the feature extraction module MSMP constructed in S2 into the SAM model to build the MSMP-SAM model and train it.

[0093] In one embodiment, S3 includes the following steps:

[0094] S31. The multi-pooling feature adaptive extraction module MSMP described in S2 is introduced into the image encoder of the SAM model, and after being arranged with the multi-head attention mechanism of each visual Transformer, it forms the MSMP-SAM model.

[0095] S32. The preprocessed image is input into the image encoder, and the image embedding is output through image block embedding and multiple multi-layer visual Transformers as described in S31;

[0096] S33. The prompt encoder converts manually entered box prompts into automatic prompt embeddings;

[0097] S34. The image embedding described in S32 and the cue embedding described in S33 are dimensionally aligned and spatially fused through the Conv layer, and then input into the mask decoder to generate a prediction mask;

[0098] S35. Compare the prediction mask described in S34 with the multi-lesion labeling mask described in S14, and perform iterative training using a linear combination of focus loss, cross-entropy loss, and dice loss, as specifically shown below:

[0099]

[0100] S36. Select Nadam as the optimizer, set the batch size to 8, and the learning rate to 0.0006. Update the adapter parameters to minimize the multi-task loss function value.

[0101] S37. Set a threshold for loss value fluctuation. When the loss value fluctuation is lower than this threshold, the model has reached convergence and training stops.

[0102] S38. Calculate the key metrics Dice, IoU, AUC-PR, and HD95, record the model weights corresponding to the best performance, and save the current state of the model.

[0103] S4: Visualization results of automatic identification and segmentation of diabetic retinopathy based on the trained MSMP-SAM model.

[0104] In one embodiment, step S4 is as follows:

[0105] S41. Input the preprocessed test set images into the trained MSMP-SAM model, perform forward inference, automatically output the pixel-level segmentation mask of diabetic retinopathy, and overlay the original fundus image to generate a visualization of the lesion area, clearly distinguishing microaneurysms, hemorrhages, soft exudates, hard exudates and background areas.

[0106] S42. Based on the segmentation mask output by the model and the ground truth labeled mask, calculate multiple core indicators for medical image segmentation, including Dice, IoU and HD95, and statistically analyze the quantitative performance of the overall lesion and four types of single diseases.

[0107] To verify the effectiveness of the above-mentioned scheme in this application, the above-mentioned scheme is compared with the current mainstream SAM-like segmentation model on FGADR, and the specific results are shown in Table 1.

[0108] Table 1 shows the comparison and segmentation results.

[0109]

[0110] Table 1 compares the performance of the method provided in this application with other DR lesion segmentation models on the FGADR dataset. The evaluation metrics include mDice value, mIoU value, mAUC-PR, and mHD95 value. These four metrics are commonly used in medical image segmentation and computer vision, with mDice value, mIoU value, and mAUC-PR being better the closer they are to 1. The results show that the method in this embodiment outperforms other comparative models in all four metrics.

[0111] Specifically, in this embodiment, mDice reaches 0.7098, which is 0.0822 to 0.4269 higher than other methods; mIoU value is 0.6354, which is 0.1847 to 0.4715 higher than other methods; mAUC-PR is 0.6847, which is 0.2295 higher than the suboptimal method; and mHD95 distance is 40.39, which is 27.47 mm lower than the suboptimal method.

[0112] This demonstrates that the method described in this embodiment achieves competitive segmentation results in the field of DR lesion segmentation. DR fundus images contain multiple lesions, including both large-area diffuse lesions and small lesions. To visualize the segmentation results of various SAM algorithms, the segmented hard exudates, hemorrhages, soft exudates, and microaneurysms are marked with red, green, orange, and sky blue, respectively. Figure 8 It can be observed that the original SAM segmentation is relatively complete for large diffuse lesions, but it is prone to missing small lesions and has rough boundaries, making it poorly adaptable to complex DR lesions. Compared with the original SAM, MedSAM has significantly improved boundary accuracy, but it still has some missed detections and missegments for multiple lesions with large scale differences. The MSMP-SAM model can accurately segment both large diffuse lesions and small lesions simultaneously, with smoother boundaries, more robust to noise, and the best overall segmentation effect.

[0113] Fundus images contain various lesions, and the location and size of these lesions vary significantly. By comparing the segmentation results of multiple SAM algorithms, the fine-tuned SAM large model can more accurately segment small lesions in the image, distinguish different lesion types, and generate more accurate segmentation results. The specific results are shown in Table 1.

[0114] To further verify the effectiveness of the multi-pooling feature adaptive extraction module, a further comparative experiment was conducted with the original SAM model in four lesion areas: microaneurysm, hemorrhage, soft exudate, and hard exudate. The specific results are shown in Table 2.

[0115] Table 2 Comparison of the effectiveness of MSMP-SAM in different lesion types

[0116]

[0117] The results show that MSMP does indeed improve the SAM model, with the module resulting in superior segmentation metrics for all four lesion regions compared to the SAM model. Specifically, the DSC of this application was improved by 0.1081, 0.1519, 0.1639, and 0.2239 compared to the SAM model; the HD95 was reduced by 26.26 mm, 36.47 mm, 37.65 mm, and 18.52 mm compared to the SAM model.

[0118] Figure 6This is a schematic diagram of the MSMP-SAM large-scale model in the diabetic retinopathy segmentation method based on SAM and multi-pooling feature extraction proposed in Embodiment 1 of the present invention. In this embodiment, the encoder of the MSMP-SAM large-scale model includes several Transformer layers, and a multi-pooling feature adaptive extraction module is embedded after the multi-head attention sublayer in each Transformer layer. The module containing the flame marker has been fine-tuned, while the module containing the snowflake marker indicates that the original SAM module has been retained.

[0119] Figure 7 yes Figure 6 A schematic diagram of the multi-pooling feature adaptive extraction module structure designed in each Transformer layer. (See diagram below.) Figure 7 As shown, in this embodiment, in the input feature map After performing multi-scale convolution kernels, max pooling, average pooling, and soft pooling in sequence, the data is then input into three MLP networks. The maximum value among the channel weights of the three feature maps is taken as the upper bound of the feature map weights. The minimum value is the lower bound of the feature map weights. The final feature map is obtained by weighted fusion of the two and then upsampling. .

[0120] To compare the impact of three single pooling methods (max pooling, average pooling, and soft pooling) with the multi-pooling fusion strategy of this invention on model convergence and stability, this study plots iterative curves of the loss function based on the model training process. The results are as follows: Figure 9 As shown, soft pooling (orange curve) and average pooling (blue curve) exhibit a gradual decrease in loss and slow convergence speed in the early stages of training (1–100 epochs). While average pooling achieves a lower loss value in the later stages of training (250–275 epochs), it shows significant loss oscillations in the 275–300 epoch stage, indicating insufficient model stability. In contrast, the multi-pooling fusion design proposed in this invention (red curve) shows a rapid decrease in loss in the early stages of training and enters a stable convergence state earlier. It not only converges faster but also maintains a lower final loss value. The training process is smooth without significant oscillations, and its overall convergence and stability are significantly better than single pooling methods.

[0121] Due to the channel weights of the three feature maps , , The maximum value in is The minimum value is , The smaller value in the range. Therefore, only the occurrence times of average pooling and soft pooling need to be recorded to accurately track the multi-pooling strategy. The dynamic value range. From Figure 10As can be seen, the lower bound of the feature map channel It adaptively switches frequently between soft pooling and average pooling. The ratio of trigger times for average pooling to soft pooling is approximately 7:3. Figure 10 The handover frequency distribution shown in the image is... Figure 9 The loss iteration curve (red curve) corresponding to the multi-pooling fusion strategy fully verifies that this design can indeed ensure that the loss function decreases faster and is more stable in the later stages of convergence.

[0122] In summary, this application constructs a multi-pooling feature adaptive extraction module (MSMP), where multi-scale feature extraction can simultaneously cover diffuse large-area lesions and small local lesions in DR, effectively solving the problem of significant differences in lesion scale in DR. Meanwhile, multi-pooling can take into account the detailed texture and global structural information of different types of lesions, improving the segmentation completeness and localization accuracy of diverse lesions. Through the collaborative design of multi-pooling and multi-scale, this application fundamentally improves the model's feature perception and segmentation capabilities for DR, significantly enhancing segmentation accuracy in medical imaging scenarios and further strengthening the model's clinical adaptability and practical value.

Claims

1. A large-scale MSMP-SAM model segmentation method based on multi-pooling feature adaptive extraction, characterized in that: Includes the following steps: S1: Data Acquisition and Preprocessing: The data is a public dataset of diabetic retinopathy, and then the image data of diabetic retinopathy from different patients are preprocessed. S2: Construct the Multi-Pooling Feature Adaptive Extraction Module (MSMP); S2 includes the following steps: S21: Feature map output by the visual Transformer multi-head attention mechanism based on the SAM image encoder; S22: MLP is used to process the feature map of the pooling branch to obtain the feature map channel weights at each sampling scale, and then the upper and lower limits of the weights are determined based on the feature map channel weights. S23: Introduce trainable feature vectors to perform weighted fusion of the upper and lower bounds of the feature map channel weights to obtain channel importance. ; S24: For different sampling scales Upsampling is performed to restore the shape of the feature map, and the upsampled feature maps are then concatenated to obtain the final fused feature map. ; S3: Introduce the MSMP constructed in S2 into the SAM model to build the MSMP-SAM model and train it; S4: Visualization results of automatic identification and segmentation of diabetic retinopathy based on the trained MSMP-SAM model.

2. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 1, characterized in that: The preprocessing in S1 includes data validity detection, removal of invalid data, standardization of valid image format, standardization of labeled mask images, matching of valid images with labeled mask images, splitting of subsets, and splitting of training and test sets.

3. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 2, characterized in that: In step S1, color image data of diabetic retinopathy from patients in the public dataset FGADR is first extracted. Then, the extracted image data is cleaned, the validity of each data is checked, and all invalid data is removed.

4. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 3, characterized in that: After removing invalid data, S1 performs dimensionality standardization on the valid image data, uniformly adjusting it to the 1024×1024 pixel specification commonly used in the SAM model to obtain uniformly sized color fundus images. It also performs standardization transformation on the labeled mask image, redefines the pixel values ​​of the labeled mask according to preset rules to obtain standardized labeled mask images, and matches the uniform color fundus images with the standardized labeled mask images one by one to generate a preprocessed color image dataset M.

5. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 4, characterized in that: In S1, the new dataset M is split into four sub-datasets: microaneurysm sub-dataset, hemorrhage sub-dataset, soft exudate sub-dataset, and hard exudate sub-dataset. Each sub-dataset is used as a dedicated dataset for independent input to the model. In step S1, the new dataset M is randomly divided into a training set and a test set in a ratio of 8:

2. At the same time, each sub-dataset is simultaneously split in a ratio of 8:2 to ensure the consistency of the distribution of each lesion type sub-dataset in the training set and the test set.

6. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 1, characterized in that: The specific steps of S21 are as follows: S211. Employing the visual Transformer inherent in the SAM image encoder, after the multi-head attention mechanism of each visual Transformer, the input feature map is processed. Perform multi-scale convolution kernel operations to obtain feature maps at three scales. ,in The size is , The size is , Indicates the sampling scale; S212, Feature map after multi-scale convolution kernel operation as described in S211 Perform multi-pooling operations; Multi-pooling operations are divided into three branches: max pooling, average pooling, and soft pooling. The feature maps after the three pooling operations are represented as follows: , .

7. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 6, characterized in that: The specific steps of S22 are as follows: S221: Use MLP to process the feature map described in S21 , Further processing to obtain Feature map channel weights at different scales , , Specifically, it is expressed as follows: in, Represents the ReLU activation function. This represents the Sigmoid activation function, which maps input values ​​to the range of 0 to 1. and They represent Weight matrices of two fully connected layers at the sampling scale; S222: Feature map channel weights based on S221 , , The maximum value among the three is selected as the upper bound of the feature map weights. The minimum value among the three is selected as the lower bound of the feature map weights. .

8. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 7, characterized in that: Steps S23 and S24 are as follows: Introduce two sizes of Trainable feature vectors and The upper and lower bounds of the feature map channel weights described in S222 are weighted and fused to obtain the channel importance. Specifically, it is expressed as follows: in, and Each element in the array is between 0 and 1, and and The sum of corresponding elements in the array is equal to 1; Then, at different sampling scales Upsampling is performed to restore the shape of the feature map to... The upsampled feature maps are then concatenated to obtain the final fused feature map. Specifically, it is expressed as follows: in, Represents learnable scale weights, satisfying .

9. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 1, characterized in that: S3 includes the following steps: S31: The multi-pooling feature adaptive extraction module MSMP described in S2 is introduced into the image encoder of the SAM model, and after being arranged with the multi-head attention mechanism of each visual Transformer, it forms the MSMP-SAM model. S32: The preprocessed image is input to the image encoder, and the image embedding is output through image block embedding and multiple multi-layer visual Transformers described in S31; S33: The prompt encoder converts manually entered box prompts into automatic prompt embeddings; S34: The image embedding described in S32 and the cue embedding described in S33 are dimensionally aligned and spatially fused through the Conv layer, and then input into the mask decoder to generate a prediction mask; S35: Compare the prediction mask described in S34 with the multi-lesion labeling mask described in S14, and perform iterative training using a linear combination of focus loss, cross-entropy loss, and dice loss, as specifically shown below: S36: Select Nadam as the optimizer, set the batch size to 8, and the learning rate to 0.0006. Update the adapter parameters to minimize the multi-task loss function value. S37: Set a threshold for loss value fluctuation. When the loss value fluctuation is lower than this threshold, the model has reached convergence and training stops. S38: Calculate key metrics such as Dice, IoU, AUC-PR, and HD95, record the model weights corresponding to the best performance, and save the current model state.

10. The MSMP-SAM large model segmentation method based on multi-pooling feature adaptive extraction according to claim 1, characterized in that: The specific steps of S4 are as follows: S41: Input the preprocessed test set images into the trained MSMP-SAM model, perform forward inference, automatically output the pixel-level segmentation mask of diabetic retinopathy, and overlay the original fundus image to generate a visualization of the lesion area, clearly distinguishing microaneurysms, hemorrhages, soft exudates, hard exudates and background areas. S42: Based on the segmentation mask output by the model and the ground truth labeled mask, calculate multiple core indicators for medical image segmentation, including Dice, IoU and HD95, and statistically analyze the quantitative performance of overall lesions and four types of single diseases.

Images