A medical image segmentation method based on SAM and a prompter

Abstract

This invention belongs to the field of image segmentation technology, specifically disclosing a medical image segmentation method based on SAM and cues. The method first preprocesses the original medical image, sequentially passing it through a SAM image encoder and feature enhancement blocks to obtain enhanced features. Then, positional and level codes are merged and input into a Transformer encoder to generate multi-scale features for cues. These features are then passed through a Transformer decoder to generate enhanced cue features, which are then input into a cue linear projection layer and combined with the multi-scale features to form a sparse cue. This sparse cue is then concatenated with intermediate features generated at each layer of the SAM image encoder and input into the SAM image decoder to obtain a mask image. Finally, the model is trained and used to segment the medical image to be segmented, ultimately obtaining a segmented image. This method overcomes the problem that the type, position, and number of SAM cues significantly affect the SAM generation results, and has the advantages of short training time, high training accuracy, strong universality, and convenient operation.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically a medical image segmentation method based on SAM and a prompter. Background Technology

[0002] With the rapid development of computer technology, medical imaging technology has seen significant advancements and widespread application, such as computed tomography (CT) and magnetic resonance imaging (MRI). The medical images generated by these technologies provide doctors with crucial tools for diagnosing diseases and support medical researchers in pathological analysis. The rapid growth of medical imaging has resulted in a massive amount of data, posing new challenges for storage, processing, and analysis. Effectively utilizing this data relies on further development of computer and artificial intelligence technologies. By applying artificial intelligence, especially machine learning and deep learning techniques, medical images can be analyzed more precisely, uncovering deeper information hidden within the data. This not only helps doctors make more accurate diagnoses but also plays a crucial role in treatment planning and disease monitoring.

[0003] In computer vision, segmentation refers to the process of classifying each pixel in an image into a specific category. This typically includes two types: semantic segmentation and instance segmentation. Semantic segmentation assigns each pixel in an image to a predefined category without distinguishing between different entities within the same category; while instance segmentation distinguishes not only categories but also different instances within the same category. Image segmentation technology is particularly important in the medical field. Accurate image segmentation helps doctors identify and quantify lesions, such as tumors, inflammation, or other abnormal structures; it can also accurately identify the boundaries of organs such as the heart, liver, and brain, and track changes in lesions before and after treatment. Furthermore, it is an indispensable tool in medical education, used to train medical students to understand complex anatomical structures and pathological changes.

[0004] Chinese invention patent application CN108038862A discloses an interactive intelligent medical image segmentation modeling method for achieving target region segmentation, optimizing segmentation results, and improving modeling efficiency. However, at the segmentation level, this method heavily relies on user input and interaction, meaning the final segmentation quality and efficiency are affected by the user's experience and skill level. For users without sufficient training, achieving optimal segmentation results is difficult. Furthermore, when processing large or complex medical image data, the process of manually marking and adjusting contours is very time-consuming, resulting in low efficiency in fast-paced clinical environments. Additionally, the lack of analysis of the overall information of the initial image in practical applications leads to poor performance in high-dimensional data analysis. Summary of the Invention

[0005] The purpose of this invention is to provide a medical image segmentation method based on SAM (Segment Anything Model) and a prompter, in order to solve the problems of weak framework universality, long training time, and low accuracy of existing medical image segmentation methods.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A medical image segmentation method based on SAM and a prompter includes the following steps:

[0008] Step 1: Obtain the original medical images, preprocess the original medical images, and construct the training dataset;

[0009] Step 2: Build a medical image segmentation model based on the SAM model. This medical image segmentation model includes a SAM image encoder, feature enhancement blocks, a cue module, and a SAM image decoder.

[0010] The SAM image encoder employs a multi-layer Transformer Block structure.

[0011] The cue module includes a Transformer encoder, a Transformer decoder, and a cue linear projection layer;

[0012] The SAM image decoder employs a multi-layer Transformer decoder structure.

[0013] The processing procedure of the preprocessed original medical image in the medical image segmentation model is as follows:

[0014] The preprocessed raw medical image first passes through a SAM image encoder composed of multiple Transformer Blocks to obtain intermediate features generated by each layer. Then, the feature enhancement block recursively accepts the features output by each Transformer Block and performs enhancement processing.

[0015] The enhanced features output from the feature enhancement block are input into the cue module and processed as follows: First, the enhanced features input to the cue module are flattened after merging position encoding and level encoding, and then input into the Transformer encoder. The Transformer encoder extracts the multi-scale features of the high-level cue, and then passes through the Transformer decoder. Combined with the initialization parameters, the enhanced cue features of each layer are recursively generated. Then, the enhanced cue features are input into the cue linear projection layer, and finally combined with the multi-scale features generated by the Transformer encoder to form a sparse cue.

[0016] The sparse cue symbols output by the cue module are concatenated with the intermediate features generated by each layer of the SAM image encoder and then input into the SAM image decoder.

[0017] In the SAM image decoder, the Transformer decoder block is used to enable interaction between the total image features generated by the encoder and the cue embedding generated by the cue generator to obtain the mask image;

[0018] Step 3: Train the medical image segmentation model built in Step 2 based on the training dataset in Step 1. Then, use the trained medical image segmentation model to segment the medical image to be segmented, and finally obtain the segmented image.

[0019] Preferably, in step 1, the preprocessing process includes uniform image size and data normalization, as detailed below:

[0020] First, the image with the largest pixel value in the original medical images is selected as the reference matrix. The other original medical images are resized to match the reference matrix by filling the blank areas with zeros.

[0021] Then, divide the pixel values ​​of all original medical images by 255 to ensure that the pixel values ​​of all original medical images are uniformly between [0,1].

[0022] Preferably, the SAM image encoder includes an embedding layer for converting the input image into 16*16 blocks, and multiple Transformer Blocks;

[0023] The input image is segmented into fixed-size blocks through an embedding layer. Each block is flattened and mapped into a vector. Position encoding is added to the embedding vector of each block to add positional information to the feature vector of each block.

[0024] Each Transformer Block consists of an attention layer with 256-dimensional input features and an MLP layer that amplifies the features by 4 times to 1024 dimensions. The attention layer uses a self-attention mechanism.

[0025] The blocks with added location information from the embedding layer will pass through each Transformer Block. Through the self-attention mechanism, the model can calculate the similarity between any two blocks in the image and combine the information of each block in a weighted manner based on the similarity.

[0026] After each self-attention layer, a residual connection is added to help avoid the gradient vanishing problem;

[0027] At the end of each Transformer Block, an MLP layer amplifies the features by a factor of 4 to 1024 dimensions, and performs further non-linear mapping to capture more complex visual information.

[0028] Preferably, the feature enhancement block includes a feature aggregator and a feature splitter;

[0029] The feature aggregator receives intermediate feature maps from the outputs of each Transformer Block in the SAM image encoder and learns representative semantic features. The specific processing steps are as follows:

[0030] In the SAM image encoder, the feature maps output by each Transformer Block are first reduced to 32 channels using a 1×1 convolutional-ReLU block, and then spatial information is added using a 3×3 convolutional-ReLU block. After that, downsampling is used to generate downsampled features, which are then merged with features from the previous layer. Then, they are passed through a 3×3 convolutional-ReLU block, and finally through a fusion convolutional layer consisting of two 3×3 convolutional layers and one 1×1 convolutional layer. The fusion convolutional layer is used to restore the channel dimension.

[0031] The feature splitter receives features from the feature aggregator and upsamples them using multiple transposed convolutional layers to obtain enhanced features.

[0032] Preferably, in step 2, the Transformer encoder consists of N stacked self-attention layers and feedforward layers;

[0033] The process by which the Transformer encoder extracts multi-scale features of high-level cues is shown below:

[0034]

[0035] Among them, PE i Represents the positional encoding of the i-th layer, LE i This represents the level encoding of the i-th layer, and Cat(·) represents the connection of the tensor along the channel dimension. i LE i Intermediate features from the i-th layer of the SAM encoder merged into This indicates the generation of multiple intermediate features, where T_enc represents the Transformer encoder layer;

[0036] The process by which the multi-scale features extracted by the Transformer encoder are combined with the multi-scale features through the Transformer decoder and the cue linear projection layer to form sparse cues is shown below:

[0037]

[0038]

[0039] in, This represents the zero-initialized learnable parameters, along with multi-scale features from various blocks of the Transformer encoder. The feature parameters are obtained recursively from the input Transformer decoding layer T_dec. mlp prompt It is a two-layer MLP used to obtain cue linear embeddings e i sin represents the sine function. Indicates a sparse prompt.

[0040] Preferably, the process of training the medical image segmentation model in step 3 is as follows:

[0041] First, by combining the SAM and the prompter module, the loss function of the medical image segmentation model is obtained. The expression of the loss function is:

[0042]

[0043] N p Indicates the number of prompt groups; This represents the cross-entropy loss calculated between the predicted category and the target. The binary cross-entropy loss represents the difference between the predicted mask and the mask of the matched ground truth instance, including both the coarse-grained and fine-grained predicted masks; a i This indicates that the match is positive; where, and The calculation formula is:

[0044]

[0045] Where y represents a binary label for a certain category, and the value of y is 0 or 1; This represents the probability of the corresponding category predicted by the neural network. The value is between (0, 1);

[0046] When y = 0, the above equation only contains the second term. The closer y is to 0, the smaller the loss; when y = 1, only the first term in the above equation has a value, at which point... The closer the value is to 1, the smaller the loss.

[0047] Finally, the Adam optimization algorithm is used to find the minimum value of the loss function.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] As described above, the medical image segmentation method based on SAM and prompts of this invention realizes medical image feature aggregation and segmentation within the SAM framework, fully considering the characteristics of high-dimensional medical image data. The medical image obtains intermediate features through the SAM encoder, and adaptively learns the optimal embedding prompt using a category-related prompt mechanism to locate objects and infer their semantic categories and instance masks. This overcomes the problem that the type, position, and number of SAM prompts significantly affect the SAM generation results, enabling more accurate and rapid segmentation of medical images. The method of this invention has the advantages of short training time, high training accuracy, strong universality, and convenient operation; moreover, it is more coherent and theoretically guided than the method of manually obtaining prompts for segmentation. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0051] Figure 1 This is a flowchart of a medical image segmentation method based on SAM and a prompter in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the medical image segmentation method based on SAM and prompter in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the feature enhancement block in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the prompter in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0056] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example

[0058] like Figures 1 to 4As shown, this invention proposes a medical image segmentation method based on SAM and cues. It can realize the aggregation and segmentation of medical image features under the SAM framework. At the same time, it uses the category-related cue mechanism to adaptively learn the best embedding cue, locate objects and infer their semantic category and instance mask. The method has strong universality, short training time and high accuracy.

[0059] The following describes the specific process of medical image segmentation using lung X-ray images:

[0060] Step 1: Obtain the original medical images, preprocess them, and construct a training dataset. The preprocessing process includes unifying image size and normalizing data.

[0061] Step 1.1: Save the original medical image to the hard drive in a certain image format. For example, take a lung X-ray image as an example. The image size is 128*128*1 pixels.

[0062] Step 1.2: Read the original medical images and select the image with the largest pixel value as the reference matrix. Adjust the size of the other original medical images by padding the blank areas with zeros to match the reference matrix. Then, normalize the pixel values ​​of the original medical images by dividing all pixel values ​​of the original medical images by 255 to ensure that the pixel values ​​of all original medical images are uniform between [0,1] or [-1,1], so as to ensure the numerical consistency of the input data of the model, help accelerate training and improve accuracy, and finally obtain the training dataset.

[0063] Step 2: Build a medical image segmentation model based on the SAM model. This medical image segmentation model includes a SAM image encoder, feature enhancement blocks, a cue module, and a SAM image decoder.

[0064] The SAM image encoder consists of an embedding layer that transforms the image into 16*16 blocks and twelve Transformer Blocks. Each Transformer Block includes an attention layer with 256-dimensional input features and a multilayer perceptron (MLP) layer that magnifies the features by 4 times to 1024 dimensions.

[0065] The number of layers in the Transformer Block can be adjusted according to the amount of data.

[0066] The SAM encoder takes the input sequence and divides it into N learnable blocks (N is a hyperparameter) through embedding layers and positional encoding. These blocks are then fed into an attention layer, where each block serves as both a query, key, and value. The attention mechanism captures the dependencies between blocks. A feedforward layer then amplifies the features by a factor of four using a multilayer perceptron. Residual connections and layer normalization ensure training stability. The SAM encoder ultimately encodes the input sequence into an intermediate feature map. The specific processing steps are as follows:

[0067] First, the preprocessed image is segmented into fixed-size blocks through an embedding layer. Each block is flattened and mapped to a vector. Positional encoding is added to the embedding vector of each block to add positional information to the feature vector of each block, ensuring that the model can understand the relative positions between different blocks.

[0068] The blocks with added location information from the embedding layer pass through each Transformer Block layer. Through a self-attention mechanism, the model can calculate the similarity between any two blocks in the image and weightedly combine the information of each block based on this similarity. This allows it to capture long-range dependencies and effectively transfer information between multiple regions in the image, while maintaining 256-dimensional output features. Residual connections are added after each attention layer to avoid the vanishing gradient problem.

[0069] At the end of each Transformer Block, a multi-MLP layer is passed to amplify the features by 4 times to 1024 dimensions, and further non-linear mapping is performed to capture more complex visual information.

[0070] The feature enhancement block recursively accepts features output from each Transformer Block of the SAM image encoder and enhances them, such as... Figure 3 As shown.

[0071] The feature enhancement block includes a feature aggregator and a feature splitter. The feature aggregator aims to learn representative semantic features from the feature maps output from each Transformer Block in the SAM image encoder. First, a 1×1 convolutional-ReLU block reduces the number of channels in the feature map to 32. Then, a 3×3 convolutional-ReLU block is used to add spatial information. After downsampling, downsampled features are generated and merged with features from the previous layer through a 3×3 convolutional-ReLU block. Finally, a fusion convolutional layer restores the channel dimension. This fusion convolutional layer consists of two 3×3 convolutional layers and one 1×1 convolutional layer.

[0072] The feature splitter receives features from the feature aggregator and upsamples them using multiple transposed convolutional layers to obtain enhanced features, which are then delivered to the prompter module.

[0073] like Figure 4 As shown, the cue module includes a Transformer encoder, a Transformer decoder, and a cue linear projection layer. The Transformer encoder consists of N stacked self-attention layers and feedforward layers, which are used to generate multi-scale features for the cue. The Transformer decoder consists of M stacked cross-attention layers and feedforward layers, which are used to generate enhanced cue features.

[0074] The specific processing procedure of the cue module is as follows: First, the enhanced features input to the cue module are flattened after merging positional and level encodings, and then input into the Transformer encoder. The Transformer encoder extracts high-level multi-scale features of the cue, as shown below.

[0075]

[0076] Among them, PE i Represents the positional encoding of the i-th layer, LE i This represents the level encoding of the i-th layer, and Cat(·) represents the connection of the tensor along the channel dimension. i LE i Intermediate features from the i-th layer of the SAM encoder merged into This indicates the generation of multiple intermediate features, where T_enc represents the Transformer encoder layer.

[0077] The Transformer encoder extracts multi-scale features of high-level cues and delivers them to the Transformer decoder. Combined with initialization parameters, it recursively generates enhanced cue features for each layer. These features are then input to the linear projection layer of the cue, and finally combined with the multi-scale features generated by the Transformer encoder to form sparse cues, as shown below.

[0078]

[0079] in, This represents the zero-initialized learnable parameters, along with multi-scale features from various blocks of the Transformer encoder. The feature parameters are obtained recursively from the input Transformer decoding layer T_dec. mlpprompt is a two-layer MLP used to obtain cue linear embeddings e i sin represents the sine function. Indicates a sparse prompt.

[0080] Finally, the sparse cue output by the cue module is concatenated with the intermediate features generated by each layer of the SAM image encoder and then input into the SAM image decoder.

[0081] In the SAM image decoder, the Transformer decoder block is used to enable interaction between the total image features generated by the encoder and the cue embedding generated by the cue generator to obtain the mask image. The specific process is as follows:

[0082] The SAM decoder takes feature maps concatenated from the SAM encoder and cue module. Using a cross-attention mechanism, it compares the similarity between the SAM decoder's query and the keys and values ​​of the SAM encoder, strengthening important features and suppressing unimportant ones. Finally, it performs feature transformation through a feedforward neural network. In the final step of the SAM decoder, the output passes through a linear layer and a softmax operation to generate class probabilities for each pixel, thus producing the segmentation result.

[0083] Step 3: Train the medical image segmentation model built in Step 2 based on the training dataset in Step 1. Then, use the trained medical image segmentation model to segment the medical image to be segmented, and finally obtain the segmented image.

[0084] The training process for the medical image segmentation model is as follows: First, by combining the SAM and cue module, the loss function of the medical image segmentation model is obtained. The expression for the loss function is as follows:

[0085]

[0086] N p Indicates the number of prompt groups; This represents the cross-entropy loss calculated between the predicted category and the target. The binary cross-entropy loss represents the difference between the predicted mask and the mask of the matched ground truth instance, including both the coarse-grained and fine-grained predicted masks; a i This indicates that the match is confirmed as positive. Among them, and The calculation formula is:

[0087]

[0088] Where y represents a binary label for a certain category, and the value of y is 0 or 1; This represents the probability of the corresponding class predicted by the neural network. Because softmax processing has been performed, The value is between (0, 1);

[0089] When y = 0, the above equation only contains the second term. The closer y is to 0, the smaller the loss; when y = 1, only the first term in the above equation has a value, at which point... The closer the value is to 1, the smaller the loss.

[0090] Taking a lung X-ray image as an example, the number of groups N is indicated. p The value is set to 5, corresponding to pneumonia, tuberculosis, tumor, COPD, and pulmonary embolism, with a corresponding prediction category number of 5. Calculate the cross-entropy loss between the probability of each class and the true class. The cross-entropy loss is calculated between the probability of each point corresponding to its class under one-hot encoding of the predicted mask image and the probability of each point corresponding to the true class being 1. The loss calculated for each class is averaged to obtain the final loss value. Taking the true mask of a lung X-ray image as an example, y represents the probability of each pixel in the true mask being 1. This represents the probability that the predicted mask is 1.

[0091] Once the loss function of the medical image segmentation model is obtained, the Adam optimization algorithm can be used to find the minimum value of the loss function.

[0092] Finally, the trained medical image segmentation model was used to segment and evaluate the lung X-ray image, and the best segmentation result was obtained through comparison and calculation.

[0093] This invention uses the Dice coefficient to evaluate the segmentation effect. The higher the Dice value, the better the segmentation effect. The calculation formula is as follows:

[0094]

[0095] In this metric, TP (True Positive) is the number of pixels correctly labeled as foreground (object of interest), FP (False Positive) is the number of pixels incorrectly labeled as foreground, and FN (False Negative) is the number of pixels incorrectly labeled as background. This metric considers both precision and recall, and is a balanced method for measuring segmentation performance.

[0096] This invention fully considers the characteristics of high-dimensional medical image data. The medical image obtains intermediate features through the SAM encoder, and uses a category-related cue mechanism to adaptively learn the best embedding cue, locate objects and infer their semantic categories and instance masks. This overcomes the problem that the type, position and number of SAM cue can significantly affect the results of SAM generation, and can segment medical images more accurately and quickly.

[0097] The embodiments of the present invention are only used to illustrate the technical solutions of the present invention and are not intended to limit it. For those skilled in the art, it will be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for medical image segmentation based on SAM and a prompter, characterized in that, The method comprises the following steps: Step 1, obtaining an original medical image, preprocessing the original medical image, and constructing a training data set; Step 2, building a medical image segmentation model based on a SAM model, the medical image segmentation model comprising a SAM image encoder, a feature enhancement block, a prompter module, and a SAM image decoder; The SAM image encoder adopts a multi-layer Transformer Block structure; The prompter module comprises a Transformer encoder, a Transformer decoder, and a prompt linear projection layer; The SAM image decoder adopts a multi-layer Transformer decoder structure; The processing process of the preprocessed original medical image in the medical image segmentation model is as follows: The preprocessed original medical image is first subjected to the SAM image encoder composed of multiple layers of Transformer Block to obtain the intermediate features generated by each layer, and then the feature enhancement block recursively accepts the features output by each layer of Transformer Block and performs enhancement processing; The enhanced features output by the feature enhancement block are input into the prompter module and processed as follows: First, the enhanced features input into the prompter module are combined with position encoding and level encoding, then flattened, and then input into the Transformer encoder, the Transformer encoder extracts multi-scale features of high-level prompts, and then the Transformer decoder is used, combined with initialization parameters, to recursively generate enhanced prompt features of each layer; then input into the prompt linear projection layer, and finally combined with the multi-scale features generated by the Transformer encoder to form sparse prompts; The sparse prompts output by the prompter module are spliced with the intermediate features generated by each layer of the SAM image encoder and input into the SAM image decoder; In the SAM image decoder, the Transformer decoder block is used to interact between the total image features generated by the encoder and the prompt embedding generated by the prompter to obtain a mask image; Step 3, training the medical image segmentation model built in step 2 based on the training data set in step 1, then using the trained medical image segmentation model to segment the medical image to be segmented, and finally obtaining a segmented image.

2. The method of claim 1, wherein the method is a method of medical image segmentation based on SAM and a prompter. In step 1, the preprocessing process comprises uniform image size and data normalization processing, and the specific process is as follows: First, select an image with the maximum pixel value in the original medical image as a reference matrix, and adjust the size of other original medical images by filling zero in the blank area to match the reference matrix; Then, divide the pixel values of all original medical images by 255 to ensure that the pixel values of all original medical images are uniform in [0, 1].

3. The method of claim 1, wherein the method further comprises: The SAM image encoder comprises an embedding layer for converting the input image into a 16*16 block, and multiple layers of Transformer Block; The input image is divided into fixed-size blocks by the embedding layer, each block is flattened and mapped into a vector; the position encoding is added to the embedding vector of each block to add position information to the feature vector of each block; Each Transformer Block includes an attention layer with an input feature of 256 dimensions and an MLP layer that enlarges the feature by 4 times to 1024 dimensions, and the attention layer adopts a self-attention mechanism; The blocks with added position information from the embedding layer will pass through each Transformer Block, and through the self-attention mechanism, the model can calculate the similarity between any two blocks in the image and combine the information of each block according to the similarity; After each layer of self-attention, a residual connection is added to help avoid the problem of gradient vanishing; At the end of each Transformer Block, a layer of MLP is used to enlarge the feature by 4 times to 1024 dimensions for further nonlinear mapping to capture more complex visual information.

4. The method of claim 1, wherein the method further comprises: The feature enhancement block includes a feature aggregator and a feature disassembler; The feature aggregator is used to receive the intermediate feature maps output by the Transformer Block in the SAM image encoder, and learn representative semantic features, the specific process is as follows: The feature maps output by the Transformer Block in the SAM image encoder are first reduced from the original number of channels to 32 using a 1x1 convolution-reLU block, then 3x3 convolution-reLU block is used to increase spatial information; After down-sampling, the down-sampled features are combined with the features from the previous layer, then a 3x3 convolution-reLU block is used, and finally a fusion convolution layer composed of two 3x3 convolution layers and a 1x1 convolution layer is used to restore the channel dimension; The feature disassembler is used to receive the features from the feature aggregator and perform up-sampling processing using multiple transpose convolution layers to obtain enhanced features.

5. The method of claim 1, wherein the method further comprises: In step 2, the Transformer encoder in the prompter module is composed of N layers of stacked self-attention layers and feedforward layers; The process of the Transformer encoder extracting multi-scale features of high-level prompts is as follows: where PE i denotes the position encoding of the i-th layer, LE i denotes the level encoding of the i-th layer, Cat(·) denotes the concatenation of tensors along the channel dimension, and PE i , LE i are merged into denotes generating multi-layer intermediate features, and T_enc denotes a Transformer encoder layer.​ The Transformer decoder is composed of M layers of stacked cross-attention layers and feedforward layers; The process of combining the multi-scale features extracted by the Transformer encoder with the sparse prompt formed by the Transformer decoder and the prompt linear projection layer is as follows: wherein, denote zero-initialized learnable parameters, and inputting the Transformer decoding layer T_dec recursively to obtain the feature parameters mlp prompt is a two-layer MLP to obtain the prompt linear embedding e i ; sin denotes a sine function, denote sparse prompts.

6. The method of claim 1, wherein the method further comprises: The process of training the medical image segmentation model in step 3 is as follows: First, combine the SAM and the prompter module to obtain the loss function of the medical image segmentation model, the expression of the loss function is: Where, N p Indicates the number of prompt groups; This represents the cross-entropy loss calculated between the predicted category and the target. The binary cross-entropy loss represents the difference between the predicted mask and the mask of the matched ground truth instance, including both the coarse-grained and fine-grained predicted masks; a i This indicates that the match is positive; where, and The calculation formula is: where y represents a binary label on a certain class, and y has a value of 0 or 1; denotes a probability of a corresponding class predicted by a neural network, the value of y is between (0, 1); When y = 0, only the second term in the above equation has a value, and when The closer to 0, the smaller the loss; when y = 1, only the first term in the above equation has a value, and when The closer to 1, the smaller the loss. Finally, the Adam optimization algorithm is used to solve the minimum value of the loss function.

Images