Sam automatic mixed prompt segmentation method and system based on lesion structure characteristics
By constructing an Automatic Hybrid Hint (AHP) module for lesion structural features and embedding it into the SAM model, the problem of insufficient adaptability of SAM for lesion features in diabetic retinopathy image segmentation is solved, achieving efficient and automatic lesion segmentation and improving segmentation accuracy and model generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- AFFILIATED HOSPITAL OF NANTONG UNIV
- Filing Date
- 2026-04-17
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, the Segment Anything Model (SAM) lacks prior knowledge of lesions in the segmentation of diabetic retinopathy images, cannot automatically adapt to the structural features of different lesions, and the single prompting strategy is difficult to balance the segmentation accuracy of high-density small lesions and low-density large lesions. Manual annotation is inefficient and highly subjective, and cannot achieve fully automatic segmentation.
An automatic hybrid suggestion module (AHP) based on lesion structural features was constructed. Automatic hybrid suggestions were generated through connected component analysis, pixel proportion allocation algorithm and weighted KMeans clustering strategy, and embedded into the SAM model to achieve adaptive suggestions based on lesion location and morphological constraints.
It has achieved a fully automated segmentation process, which significantly reduces the complexity of clinical applications and labor costs, improves the segmentation accuracy of microaneurysms and hard exudates, maintains the model's generalization ability, and solves the problem of lesion class imbalance.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for automatic SAM (Synthetic Awareness and Assistive Technology) segmentation based on lesion structural features. Background Technology
[0002] Diabetic retinopathy is a common and serious retinal complication of diabetes, and a major cause of vision loss and even blindness. With the continued increase in the number of diabetic patients, early screening and accurate identification of diabetic retinopathy have become crucial tools in clinical ophthalmology.
[0003] Accurate segmentation of various lesions in fundus images, such as microaneurysms, hemorrhages, and hard exudates, is crucial for early diagnosis and disease grading. However, different lesions vary significantly in morphology, size, number, and spatial distribution, posing a significant challenge to automated segmentation. In recent years, the Segment Anything Model (SAM), as a large-scale visual model, has demonstrated strong zero-shot generalization ability in natural image segmentation, but its direct application to diabetic fundus lesion segmentation has obvious limitations: SAM lacks prior knowledge about lesions, relies on manual input of points, boxes, etc., and cannot automatically adapt to the structural features of different lesions. Secondly, fundus lesions exhibit severe imbalance problems; microaneurysms are numerous but occupy a very small percentage of pixels, while hard exudates cover a large area but are sparsely distributed, making it difficult for a single prompting strategy to simultaneously consider both types of features. Furthermore, manual annotation prompts in clinical applications are inefficient and highly subjective, making fully automated segmentation difficult.
[0004] Most existing automatic suggestion generation methods are based on global image features or simple location priors, ignoring the structural differences in lesion categories in terms of pixel proportion and spatial density. This results in a tradeoff between achieving high-density small lesions and low-density large lesions in terms of segmentation accuracy. Therefore, how to construct a method that can automatically perceive lesion structural features and generate adaptive hybrid suggestions has become a key issue in improving the practicality of SAM in the field of medical image segmentation. Summary of the Invention
[0005] This invention provides an automatic SAM (Synthetic Aspect-Oriented Detection and Segmentation) method and system based on lesion structural features, which can solve the problems in the prior art.
[0006] A first aspect of the present invention provides an automatic SAM (Symptom Awareness and Modulation) segmentation method based on lesion structural features, comprising:
[0007] S1. Obtain color fundus images of different diabetic patients and black and white mask images corresponding to the color fundus images. The black and white mask images contain lesion information annotations. The obtained data are divided into training set and test set according to a preset ratio.
[0008] S2. Construct an Automatic Hybrid Hint Module (AHP) oriented to lesion category prior information to generate automatic hint embedding, wherein the lesion category prior information is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories;
[0009] S3. Embed the Automatic Hybrid Prompt (AHP) module into the SAM large model to construct the AHPSAM model, and train the AHPSAM model based on the training set;
[0010] S4. Input the color fundus images of diabetic patients into the trained AHPSAM model, and output the image segmentation results containing all lesion types.
[0011] The Automatic Hybrid Hint Module (AHP) constructed in S2, which is based on prior information about lesion categories, includes:
[0012] Connectivity analysis of lesion regions is performed on the black and white mask images in the training set. Connectivity regions for each lesion category are extracted, and the minimum bounding rectangle of each connected region is calculated as a bounding box cue.
[0013] Based on the pixel distribution characteristics and spatial density in the prior information of the lesion categories, the pixel proportion allocation algorithm PPA is used to determine the number of point prompts that should be allocated to each lesion category.
[0014] Based on the number of point prompts, a multi-point prompt is generated by selecting representative locations within the lesion area using a weighted KMeans clustering strategy.
[0015] The bounding box hints and multi-point hints are combined to generate an automatic hint embedding that includes prior information about the lesion location and morphological constraint information.
[0016] The pixel proportion allocation algorithm (PPA) is used to determine the number of point prompts that should be allocated to each lesion category, including:
[0017] For each batch in the training set, the lesion category is statistically analyzed for the black and white mask image corresponding to each color fundus image in the batch, and the number of pixels of each lesion category in each color fundus image and the total number of pixels of all lesion categories are calculated.
[0018] Based on the ratio of the number of pixels of each lesion category to the total number of pixels, the pixel proportion of each lesion category in each color fundus image is determined. According to the product of the preset number of initial prompts and the pixel proportion, the number of prompts to be allocated to each lesion category in each color fundus image is rounded up to obtain the number of image-level prompts allocated.
[0019] The number of image-level cue points allocated to the same lesion category in all color fundus images within a batch is averaged and rounded to obtain the batch-level cue point allocation for each lesion category in the current batch.
[0020] Based on the number of point cues, a multi-point cues are generated by selecting representative locations within the lesion area using a weighted KMeans clustering strategy, including:
[0021] For each color fundus image in the batch, extract the set of pixel coordinates corresponding to each lesion category in the corresponding black and white mask image;
[0022] For each set of pixel coordinates, the nearest neighbor algorithm is used to calculate the average distance between each pixel in the set and its preset number of nearest neighbor pixels. The sum of the average distance and the preset minimum constant is used as the sampling weight of each pixel.
[0023] Based on the sampling weights, a weighted clustering operation is performed on each set of pixel coordinates. The number of clusters is set to the batch-level prompt point allocation number for each lesion category. The cluster center of each cluster is obtained by minimizing the squared error within the weighted cluster.
[0024] Each cluster center is projected to the nearest actual pixel in the corresponding pixel coordinate set, which serves as the cue point set for each lesion category in the image. The cue point sets for all lesion categories in the color fundus image are then merged to form a multi-point cue.
[0025] The minimization of the weighted cluster-in-square error includes:
[0026]
[0027] in, This represents the number of clusters for lesion category k. This indicates the position of the lesion pixel in the image coordinate system. Let represent the set of pixels in color fundus image b that are assigned to t clusters within lesion category k. express Sampling weights, Let t be the centroid of the lesion category k.
[0028] S3 involves constructing the AHPSAM model and training it based on the training set, including:
[0029] Color fundus images from the training set are input into the image encoder of the AHPSAM model. The image encoder adopts a visual transformer architecture and extracts multi-scale image feature representations through multi-layer transformer blocks.
[0030] The bounding box cues and multipoint cues generated by the Automatic Hybrid Cueing (AHP) module are input into the cue encoder of the AHPSAM model. The cue encoder performs positional encoding on the two-dimensional coordinates and point type labels of the point cues, as well as the coordinates of the top left and bottom right corners of the bounding box cues, to generate cue feature embeddings.
[0031] The mask decoder receives the multi-scale image feature representation and the cue feature embedding, fuses the two types of features through a cross-attention mechanism, and generates a predicted segmentation mask through multi-layer convolutional upsampling;
[0032] Calculate the segmentation loss between the predicted segmentation mask and the pixel-level labeled mask image. Based on the segmentation loss, update the trainable parameters in the AHPSAM model through the backpropagation algorithm. Repeat the above training process until the model converges.
[0033] A second aspect of the present invention provides an automatic SAM (Synthetic Awareness and Modulation) segmentation system based on lesion structural features, comprising:
[0034] The data acquisition unit acquires color fundus images of different diabetic patients and black-and-white mask images corresponding to the color fundus images. The black-and-white mask images contain lesion information annotations. The acquired data is divided into training set and test set according to a preset ratio.
[0035] An automatic suggestion module construction unit is used to construct an automatic hybrid suggestion module (AHP) based on prior information of lesion categories to generate automatic suggestion embeddings. The prior information of lesion categories is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories.
[0036] The model training unit embeds the Automatic Hybrid Hints (AHP) module into the SAM large model to construct the AHPSAM model, and trains the AHPSAM model based on the training set.
[0037] The image segmentation unit inputs color fundus images of diabetic patients into the trained AHPSAM model and outputs image segmentation results containing all lesion types.
[0038] A third aspect of the present invention provides an electronic device, comprising:
[0039] processor;
[0040] Memory used to store processor-executable instructions;
[0041] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.
[0042] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] This invention utilizes an Automatic Hybrid Hints (AHP) module. This module first adaptively determines the number of hint points based on the pixel proportion of lesion categories in a black-and-white mask image. Then, it generates corresponding point hint embeddings based on the spatial density of lesions, and combines these with bounding box hints to form an automatic hybrid hint. This brings significant clinical and technical advantages to the SAM (Segmentation and Analysis) model. This module achieves a fully automated segmentation process: AHP can automatically calculate and generate hybrid hints based on the structural features of lesions, eliminating the need for any manual point selection or bounding box selection. This fundamentally eliminates the reliance on manual hints in traditional SAM, significantly reducing the complexity and labor costs of clinical applications.
[0045] Secondly, by integrating dual cues based on pixel ratio and spatial density, this solution possesses excellent structural adaptability: for lesions with small pixel ratio but dense distribution, such as microaneurysms, it automatically enhances local point cues to capture details; for lesions with large pixel ratio but sparse distribution, such as hard exudates, it automatically introduces global box cues to ensure regional integrity, thereby effectively solving the industry pain point that a single cue strategy cannot take into account multiple types of heterogeneous lesions.
[0046] Furthermore, this mechanism alleviates the imbalance in pixels and quantity among different categories by explicitly modeling the prior distribution of lesions. It can maintain stable segmentation performance even for lesion categories with few samples. AHP is embedded in SAM as a lightweight module without destroying its original pre-trained weights, thus preserving the model's good generalization ability across devices and datasets. Attached Figure Description
[0047] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention;
[0048] Figure 2 A schematic diagram of the AHPSAM model constructed according to an embodiment of the present invention;
[0049] Figure 3 A diagram showing the comparison of the effects of different point sampling methods;
[0050] Figure 4 This is a diagram comparing the segmentation effects of different models. Detailed Implementation
[0051] Figure 1 This is a flowchart illustrating the SAM automatic mixed prompting segmentation method based on lesion structural features according to an embodiment of the present invention. Figure 1 As shown, the method includes:
[0052] S1. Obtain color fundus images of different diabetic patients and black and white mask images corresponding to the color fundus images. The black and white mask images contain lesion information annotations. The obtained data are divided into training set and test set according to a preset ratio.
[0053] S2. Construct an Automatic Hybrid Hint Module (AHP) oriented to lesion category prior information to generate automatic hint embedding, wherein the lesion category prior information is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories;
[0054] S3. Embed the Automatic Hybrid Prompt (AHP) module into the SAM large model to construct the AHPSAM model, and train the AHPSAM model based on the training set;
[0055] S4. Input the color fundus images of diabetic patients into the trained AHPSAM model, and output the image segmentation results containing all lesion types.
[0056] The Automatic Hybrid Hint Module (AHP) constructed in S2, which is based on prior information about lesion categories, includes:
[0057] Connectivity analysis of lesion regions is performed on the black and white mask images in the training set. Connectivity regions for each lesion category are extracted, and the minimum bounding rectangle of each connected region is calculated as a bounding box cue.
[0058] Based on the pixel distribution characteristics and spatial density in the prior information of the lesion categories, the pixel proportion allocation algorithm PPA is used to determine the number of point prompts that should be allocated to each lesion category.
[0059] Based on the number of point prompts, a multi-point prompt is generated by selecting representative locations within the lesion area using a weighted KMeans clustering strategy.
[0060] The bounding box hints and multi-point hints are combined to generate an automatic hint embedding that includes prior information about the lesion location and morphological constraint information.
[0061] In one optional implementation, a pixel proportion allocation algorithm (PPA) is used to determine the number of point prompts to be allocated to each lesion category, including:
[0062] For each batch in the training set, the lesion category is statistically analyzed for the black and white mask image corresponding to each color fundus image in the batch, and the number of pixels of each lesion category in each color fundus image and the total number of pixels of all lesion categories are calculated.
[0063] Based on the ratio of the number of pixels of each lesion category to the total number of pixels, the pixel proportion of each lesion category in each color fundus image is determined. According to the product of the preset number of initial prompts and the pixel proportion, the number of prompts to be allocated to each lesion category in each color fundus image is rounded up to obtain the number of image-level prompts allocated.
[0064] The number of image-level cue points allocated to the same lesion category in all color fundus images within a batch is averaged and rounded to obtain the batch-level cue point allocation for each lesion category in the current batch.
[0065] In this specific embodiment, for each image in each batch, the number of pixels for each lesion category is calculated. Total number of pixels of all lesions Lesion type In the image The pixel ratio in Accordingly, the number of point hints assigned to this category is:
[0066] ,
[0067] in, This represents the preset initial number of point prompts, where ⌈·⌉ indicates rounding up. Further calculations are performed to determine the number of point prompts allocated to each lesion category within this batch:
[0068] ,
[0069] Where B is the batch size, and Round(·) represents the rounding operation.
[0070] In one optional implementation, based on the number of point cues, a multi-point cue is generated by selecting representative locations within the lesion region using a weighted KMeans clustering strategy, including:
[0071] For each color fundus image in the batch, extract the set of pixel coordinates corresponding to each lesion category in the corresponding black and white mask image;
[0072] For each set of pixel coordinates, the nearest neighbor algorithm is used to calculate the average distance between each pixel in the set and its preset number of nearest neighbor pixels. The sum of the average distance and the preset minimum constant is used as the sampling weight of each pixel.
[0073] Based on the sampling weights, a weighted clustering operation is performed on each set of pixel coordinates. The number of clusters is set to the batch-level prompt point allocation number for each lesion category. The cluster center of each cluster is obtained by minimizing the squared error within the weighted cluster.
[0074] Each cluster center is projected to the nearest actual pixel in the corresponding pixel coordinate set, which serves as the cue point set for each lesion category in the image. The cue point sets for all lesion categories in the color fundus image are then merged to form a multi-point cue.
[0075] The minimization of the weighted cluster-in-square error includes:
[0076] ,
[0077] in, This represents the number of clusters for lesion category k. This indicates the position of the lesion pixel in the image coordinate system. Let represent the set of pixels in color fundus image b that are assigned to t clusters within lesion category k. express Sampling weights, Let t be the centroid of the lesion category k.
[0078] In this specific embodiment, image information corresponding to lesion category k is extracted from image b, wherein the set of pixel coordinates of the lesion region is... as follows:
[0079] ,
[0080] in, This indicates the position of the lesion pixel in the image coordinate system (two-dimensional coordinates).
[0081] for The KNN (K-Nearest Neighbors) algorithm is used to calculate the local density of each pixel within the set. For any point... ,make To and Recent The nearest neighbor set formed by n points is defined as follows: To the nearest neighbor set All inside Average distance of pixels As sampling weights, they are represented as follows:
[0082]
[0083] High-density areas tend to have shorter distances between neighbors, while low-density areas have longer distances between neighbors.
[0084] In the set The above execution includes The weighted KMeans of each cluster are used to obtain the cluster centers by minimizing the weighted squared error:
[0085] ,
[0086] in, This represents the number of clusters for lesion category k. This indicates the position of the lesion pixel in the image coordinate system. Let represent the set of pixels in color fundus image b that are assigned to t clusters within lesion category k. express Sampling weights, Let t be the centroid of the lesion category k.
[0087] in, This represents the number of clusters for lesion category k. The image is obtained by finding the t-th cluster centroid of lesion category k. Category Point hint set:
[0088]
[0089] By merging all categories of dot hints, the image can be obtained. Point hint set:
[0090]
[0091] Furthermore, this set of point hints of size B can be represented as:
[0092]
[0093] Figure 2 This is a schematic diagram of the AHPSAM model constructed according to an embodiment of the present invention. Step S3, which involves constructing the AHPSAM model and training it based on the training set, includes:
[0094] Color fundus images from the training set are input into the image encoder of the AHPSAM model. The image encoder adopts a visual transformer architecture and extracts multi-scale image feature representations through multi-layer transformer blocks.
[0095] The bounding box cues and multipoint cues generated by the Automatic Hybrid Cueing (AHP) module are input into the cue encoder of the AHPSAM model. The cue encoder performs positional encoding on the two-dimensional coordinates and point type labels of the point cues, as well as the coordinates of the top left and bottom right corners of the bounding box cues, to generate cue feature embeddings.
[0096] The mask decoder receives the multi-scale image feature representation and the cue feature embedding, fuses the two types of features through a cross-attention mechanism, and generates a predicted segmentation mask through multi-layer convolutional upsampling;
[0097] The segmentation loss between the predicted segmentation mask and the pixel-level labeled mask is calculated. This segmentation loss is calculated using a dynamic loss function. This dynamic loss function identifies difficult sample pixels whose predicted probability is lower than a preset dynamic threshold based on the predicted probability value of each pixel. Only the prediction error of these difficult sample pixels is calculated using logarithmic loss and normalized, allowing the model training process to focus more on segmenting difficult lesion regions. During the prediction probability calculation, weighted Gaussian noise inversely proportional to the pixel proportion of each lesion category is added to the original prediction logic value to enhance the learning weight for small-area lesion categories.
[0098] Based on the segmentation loss, the trainable parameters in the AHPSAM model are updated using the backpropagation algorithm, and the above training process is repeated until the model converges.
[0099] To verify the effectiveness of the Automatic Hybrid Hint (AHP) module, which is based on prior information about lesion categories, for lesion structural features, in addition to the aforementioned weighted K-Means method, several alternative point selection strategies were explored, including random sampling, DBSCAN clustering, and farthest point sampling (FPS). Figure 3 As shown, however, these methods have obvious limitations, especially when the distribution of lesion pixels is uneven: (1) Random sampling is prone to generating multiple points in high-density lesion areas, leading to redundancy, such as Figure 3 As shown in (a), region I has several yellow warning points. It also performs poorly in sparse, fragmented lesions, leaving areas without any warning points, as shown in region II (green lesion area). (2) DBSCAN is sensitive to density-related parameters, making it difficult to accurately control the number of warning points. For example... Figure 3 (b) shows that sparse lesions are usually classified as noise, therefore there are no cue points, increasing the cost of parameter adjustment and computation. (3) FPS mainly generates cue points along the lesion boundary, such as Figure 3 As shown in (c), this method results in insufficient coverage of representative internal lesion areas, wasting limited cue points. Figure 3 As shown in (d), the weighted K-Means prompting strategy assigns only one prompting point to each large lesion area, avoiding redundant prompts. In addition, it adaptively adjusts the prompting allocation according to the pixel distribution, ensuring balanced coverage in dense and sparse lesion areas.
[0100] To verify the effectiveness of the SAM automatic hybrid prompting segmentation method based on lesion structural features of the present invention, the segmentation performance of the method was compared with that of the basic model SAM on various lesion categories, as shown in Table 1.
[0101] Table 1 Comparison of segmentation results of different models
[0102]
[0103] The results in Table 1 show that AHPSAM's segmentation performance is significantly better than SAM's. In terms of DSC metrics, AHPSAM achieved segmentation rates of 78.39%, 81.07%, 78.35%, and 83.25% for microaneurysms, hemorrhages, soft exudates, and hard exudates, respectively, with an average of 80.27%, significantly higher than SAM's 55.62%. This indicates that AHPSAM has a higher degree of overlap with the actual annotations and better segmentation results. In terms of HD95 metrics, AHPSAM's average value was 17.92 mm, significantly lower than SAM's 67.86 mm. This indicates that the lesion boundaries obtained by AHPSAM segmentation are closer to the actual boundaries, resulting in more accurate localization. Figure 4 This is a comparative diagram showing the segmentation effects of different models. For example... Figure 4 As shown, after introducing AHP, the identification and segmentation of the three types of small discrete lesions (red - hard exudate, green - hemorrhage, blue - microaneurysm) can be significantly increased. This indicates that AHP greatly improves the model's sensitivity to small and sparse lesions, making the segmentation results more consistent with the real mask.
[0104] This invention also provides an automatic SAM (Synthetic Awareness and Modulation) segmentation system based on lesion structural features, comprising:
[0105] The data acquisition unit acquires color fundus images of different diabetic patients and black-and-white mask images corresponding to the color fundus images. The black-and-white mask images contain lesion information annotations. The acquired data is divided into training set and test set according to a preset ratio.
[0106] An automatic suggestion module construction unit is used to construct an automatic hybrid suggestion module (AHP) based on prior information of lesion categories to generate automatic suggestion embeddings. The prior information of lesion categories is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories.
[0107] The model training unit embeds the Automatic Hybrid Hints (AHP) module into the SAM large model to construct the AHPSAM model, and trains the AHPSAM model based on the training set.
[0108] The image segmentation unit inputs color fundus images of diabetic patients into the trained AHPSAM model and outputs image segmentation results containing all lesion types.
[0109] A third aspect of the present invention provides an electronic device, comprising:
[0110] processor;
[0111] Memory used to store processor-executable instructions;
[0112] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.
[0113] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.
[0114] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.
Claims
1. A SAM automatic hybrid prompting and segmentation method based on lesion structural features, characterized in that, include: S1. Obtain color fundus images of different diabetic patients and black and white mask images corresponding to the color fundus images. The black and white mask images contain lesion information annotations. The obtained data are divided into training set and test set according to a preset ratio. S2. Construct an Automatic Hybrid Hint Module (AHP) oriented to lesion category prior information to generate automatic hint embedding, wherein the lesion category prior information is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories; S3. Embed the Automatic Hybrid Prompt (AHP) module into the SAM large model to construct the AHPSAM model, and train the AHPSAM model based on the training set; S4. Input the color fundus images of diabetic patients into the trained AHPSAM model, and output the image segmentation results containing all lesion types.
2. The method according to claim 1, characterized in that, The Automatic Hybrid Hint Module (AHP) constructed in S2, which is based on prior information about lesion categories, includes: Connectivity analysis of lesion regions is performed on the black and white mask images in the training set. Connectivity regions for each lesion category are extracted, and the minimum bounding rectangle of each connected region is calculated as a bounding box cue. Based on the pixel distribution characteristics and spatial density in the prior information of the lesion categories, the pixel proportion allocation algorithm PPA is used to determine the number of point prompts that should be allocated to each lesion category. Based on the number of point prompts, a multi-point prompt is generated by selecting representative locations within the lesion area using a weighted KMeans clustering strategy. The bounding box hints and multi-point hints are combined to generate an automatic hint embedding that includes prior information about the lesion location and morphological constraint information.
3. The method according to claim 2, characterized in that, The pixel proportion allocation algorithm (PPA) is used to determine the number of point prompts that should be allocated to each lesion category, including: For each batch in the training set, the lesion category is statistically analyzed for the black and white mask image corresponding to each color fundus image in the batch, and the number of pixels of each lesion category in each color fundus image and the total number of pixels of all lesion categories are calculated. Based on the ratio of the number of pixels of each lesion category to the total number of pixels, the pixel proportion of each lesion category in each color fundus image is determined. According to the product of the preset number of initial prompts and the pixel proportion, the number of prompts to be allocated to each lesion category in each color fundus image is rounded up to obtain the number of image-level prompts allocated. The number of image-level cue points allocated to the same lesion category in all color fundus images within a batch is averaged and rounded to obtain the batch-level cue point allocation for each lesion category in the current batch.
4. The method according to claim 2, characterized in that, Based on the number of point cues, a multi-point cues are generated by selecting representative locations within the lesion area using a weighted KMeans clustering strategy, including: For each color fundus image in the batch, extract the set of pixel coordinates corresponding to each lesion category in the corresponding black and white mask image; For each set of pixel coordinates, the nearest neighbor algorithm is used to calculate the average distance between each pixel in the set and its preset number of nearest neighbor pixels. The sum of the average distance and the preset minimum constant is used as the sampling weight of each pixel. Based on the sampling weights, a weighted clustering operation is performed on each set of pixel coordinates. The number of clusters is set to the batch-level prompt point allocation number for each lesion category. The cluster center of each cluster is obtained by minimizing the squared error within the weighted cluster. Each cluster center is projected to the nearest actual pixel in the corresponding pixel coordinate set, which serves as the cue point set for each lesion category in the image. The cue point sets for all lesion categories in the color fundus image are then merged to form a multi-point cue.
5. The method according to claim 1, characterized in that, The minimization of the weighted cluster-in-square error includes: ， in, This represents the number of clusters for lesion category k. This indicates the position of the lesion pixel in the image coordinate system. Let represent the set of pixels in color fundus image b that are assigned to t clusters within lesion category k. express Sampling weights, Let t be the centroid of the lesion category k.
6. The method according to claim 1, characterized in that, S3 involves constructing the AHPSAM model and training it based on the training set, including: Color fundus images from the training set are input into the image encoder of the AHPSAM model. The image encoder adopts a visual transformer architecture and extracts multi-scale image feature representations through multi-layer transformer blocks. The bounding box cue and multipoint cue generated by the Automatic Hybrid Cueing Module (AHP) are input into the cue encoder of the AHPSAM model. The cue encoder performs position encoding on the two-dimensional coordinates and point type label of the point cue, and the coordinates of the upper left and lower right corners of the bounding box cue, to generate cue feature embedding. The mask decoder receives the multi-scale image feature representation and the cue feature embedding, fuses the two types of features through a cross-attention mechanism, and generates a predictive segmentation mask through multi-layer convolutional upsampling; Calculate the segmentation loss between the predicted segmentation mask and the pixel-level labeled mask image. Based on the segmentation loss, update the trainable parameters in the AHPSAM model using the backpropagation algorithm. Repeat the above training process until the model converges.
7. A SAM automatic hybrid prompting and segmentation system based on lesion structural features, used to implement the method as described in any one of claims 1-6, characterized in that, include: The data acquisition unit acquires color fundus images of different diabetic patients and black-and-white mask images corresponding to the color fundus images. The black-and-white mask images contain lesion information annotations. The acquired data is divided into training set and test set according to a preset ratio. An automatic suggestion module construction unit is used to construct an automatic hybrid suggestion module (AHP) based on prior information of lesion categories to generate automatic suggestion embeddings. The prior information of lesion categories is used to characterize the differences in pixel ratio and spatial distribution density of different lesion categories. The model training unit embeds the Automatic Hybrid Hints (AHP) module into the SAM large model to construct the AHPSAM model, and trains the AHPSAM model based on the training set. The image segmentation unit inputs color fundus images of diabetic patients into the trained AHPSAM model and outputs image segmentation results containing all lesion types.
8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.