A medical image segmentation method, system and product based on SAM

By introducing intuitive perception fine-tuning adapter and multi-scale domain migration adapter in SAM, the problem of unsatisfactory consistency and accuracy in medical image segmentation is solved, and more efficient medical image segmentation effect and better adaptability are achieved.

CN118485682BActive Publication Date: 2024-12-03NANKAI UNIV +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410919002.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-12-03
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

The consistency and accuracy of SAM in medical image segmentation results are not ideal, and its adaptability to medical images is not high, resulting in performance degradation.

Method used

Introducing intuitive perception fine-tuning adapter and multi-scale domain migration adapter, by fusing shared projection features and multi-head self-attention features in the image encoder, adopting triple attention mechanism and spatial attention mechanism, combining multi-scale convolutional structures to enhance the model pair Feature expression and adaptability of medical images.

Benefits of technology

It significantly improves the accuracy and efficiency of medical image segmentation, enhances the adaptability and generalization capabilities of the model, can better handle complex medical image structures, and reduces information loss in feature transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118485682B_ABST
    Figure CN118485682B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of medical image segmentation technology, and specifically discloses a medical image segmentation method, system and product based on SAM, the method comprising: inputting a medical image into an image encoder to obtain an image coding vector; the image encoder comprises a sequentially connected block embedding layer and a plurality of coding blocks, the image encoder is embedded with an intuitive perception fine-tuning adapter, the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each coding block; inputting a prompt content into a prompt encoder to obtain a prompt coding vector; splicing the image coding vector and the prompt coding vector into a mask decoder to obtain an image segmentation result. The present invention can process medical image data more accurately and effectively, improve the accuracy and efficiency of medical diagnosis and treatment, and realize the wide application of SAM in the field of medical image processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method, system and product. Background Art

[0002] In the field of medical image segmentation, accurate image segmentation is crucial for computer-aided diagnosis. Since the UNet model was proposed, a variety of improved versions based on this architecture have emerged, such as UNet++ and UNet3+, which effectively improve the segmentation performance by enhancing the information transmission capability of jump connections. With the successful application of Visual Transformer (ViT), researchers began to explore the integration of Transformer structure into medical image segmentation. For example, TransUNet introduces Transformer after the convolutional layer to extract global information from low-resolution feature maps. SwinUNet uses SwinTransformer to build a U-shaped architecture of the full Transformer. TransFuse builds CNN and Transformer encoders at the same time, making full use of the advantages of both, reducing the network depth while capturing high-level and low-level information.

[0003] Although these models have demonstrated superiority on specific tasks, they often require task-specific network design, which limits their scalability and transferability across different medical segmentation tasks. To overcome these limitations, researchers explored the fine-tuning of SAM (Segment Anything Model). SAM is a pioneering image segmentation method developed by the Meta AI research team that leverages large datasets and advanced machine learning techniques to achieve superior zero-shot and few-shot learning capabilities. SAM introduces a new approach to image segmentation by combining three core components: a promptable segmentation task, a segmentation model, and a data engine. These components work together to enable the model to dynamically segment images based on text or spatial prompts. 1) Promptable segmentation task: This task design enables SAM to interpret and respond to a variety of prompts. These prompts can be as simple as a text description or as complex as a spatial point on an image. The goal of the model is to generate valid segmentation masks based on a given prompt while seamlessly handling ambiguity and changes in the input. 2) Segmentation model architecture: The model architecture is divided into three main parts: Image encoder: Processes images using visual transformers and creates comprehensive image embeddings. Hint Encoder: Converts hints into a format that the model can interpret, either textual or spatial. Mask Decoder: Combines the inputs from the image encoder and hint encoder to generate accurate segmentation masks. 3) Data Engine: To support the high demand for data for SAM, a robust data engine was developed. This engine uses an iterative process to collect and refine segmentation masks. It starts with a manually assisted annotation phase and proceeds through semi-automatic and fully-automatic phases, using the model itself to generate high-quality masks that are then used to further train and refine SAM.

[0004] As the first visual base model, SAM has been applied to a variety of downstream tasks such as image restoration and detection. However, the performance of SAM in medical image segmentation still needs to be improved. With the increase in data and model size, the paradigm of pre-training large models and migrating them to downstream tasks is becoming increasingly popular. In the process of transfer learning, researchers proposed the parameter-efficient fine-tuning technique (PEFT). PEFT transforms the learned general features into task-specific features by introducing a small number of trainable parameters on the pre-trained large model, such as visual cue tuning (VPT) and Pro-tuning.

[0005] For example, the Chinese invention patent with application publication number CN117132776A discloses a method for constructing a multi-organ image segmentation model and a segmentation method, wherein a control module is added to the pre-trained model to obtain a first model, wherein the control module is connected to the multi-head attention module Multi-headAttention in the encoder layer encoder, and is used to process the features of the multi-head attention mechanism module Multi-headAttention and fuse them with the subsequent fully connected module. Another example is the Chinese invention patent with application publication number CN117496135A discloses an intelligent segmentation algorithm for arbitrary shapes with efficient tuning of large models, which uses an adapter parallel to the ViT backbone, which is inserted into the ViT backbone in parallel, and can simultaneously adjust the multi-head attention layer and the forward propagation layer, and utilizes a small part of the parameters added by the adapter; by increasing the amount of some parameters, the frozen ViT backbone is adjusted, and semantically related features are learned without destroying the generalization of its parameters, so that it can adapt to various downstream tasks and achieve the effect of efficient tuning. These methods all add an adapter to each layer of the encoder during model fine-tuning to adjust the features obtained at each layer. However, as the number of layers increases, information in the target domain is inevitably lost. In addition, SAM is mainly trained on natural images, which are very different from medical images in nature. Medical images usually have low contrast and diverse anatomical structures, which makes the direct application of these models to medical tasks result in a significant drop in performance, and the above methods have not been able to solve this problem well. Summary of the invention

[0006] The present invention aims to solve the problems of unsatisfactory consistency and accuracy of the segmentation results of SAM in medical image segmentation and low adaptability to medical images. To this end, the present invention provides a medical image segmentation method, system and product based on SAM, which can process medical image data more accurately and effectively, improve the accuracy and efficiency of medical diagnosis and treatment, and realize the wide application of SAM in the field of medical image processing.

[0007] The present invention provides a medical image segmentation method based on SAM, and the technical solution adopted is as follows:

[0008] Inputting a medical image into an image encoder to obtain an image encoding vector; the image encoder comprises a block embedding layer and a plurality of encoding blocks connected in sequence, the image encoder is embedded with an intuitive perception fine-tuning adapter, and the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each encoding block;

[0009] Input the prompt content into the prompt encoder to obtain a prompt encoding vector;

[0010] The image coding vector and the hint coding vector are concatenated and input into a mask decoder to obtain an image segmentation result.

[0011] Furthermore, the processing process of the intuitive perception fine-tuning adapter includes: the multi-head self-attention features are normalized, down-sampled and activated, and then merged with the shared projection features to obtain merged features, and the merged features are normalized, triple attention mechanism, and up-sampled to obtain intuitive features.

[0012] Furthermore, the processing process of the triple attention mechanism includes: capturing the interaction between the C dimension and the W dimension, the interaction between the C dimension and the H dimension, and the interaction between the W dimension and the H dimension of the merged feature through three branches, and then averaging the three interaction results.

[0013] Furthermore, the three branches are respectively a first branch, a second branch and a third branch;

[0014] The processing process of the first branch includes: rotating the merged feature 90° counterclockwise along the H axis, then passing through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function, and then rotating 90° clockwise along the H axis to obtain a first interaction result;

[0015] The processing process of the second branch includes: rotating the merged feature 90° counterclockwise along the W axis, then passing through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function, and then rotating 90° clockwise along the W axis to obtain a second interaction result;

[0016] The processing process of the third branch includes: passing the merged features through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function to obtain an attention weight, and multiplying the attention weight by the merged features to obtain a third interaction result.

[0017] Furthermore, a multi-scale domain migration adapter is embedded in the encoding block, and the multi-scale domain migration adapter calculates attention weights according to the input features of the multi-layer perceptron of the encoding block. The attention weights are multiplied by the output features of the multi-layer perceptron, so as to capture multi-scale local features.

[0018] Furthermore, the processing process of the multi-scale domain migration adapter includes: reducing the dimension of the input features of the multi-layer perceptron in the channel dimension through a convolution operation, and then after passing through an activation function, inputting three convolution kernels of different sizes, merging the three features obtained, and then normalizing, applying a GELU activation function, and restoring the channel dimension through a 1×1 convolution kernel, and then passing through a spatial attention mechanism to obtain an attention weight.

[0019] Furthermore, the three convolution kernels of different sizes are 1×1, 3×3, and 5×5;

[0020] The processing process of the spatial attention mechanism includes: generating attention weights by performing global average pooling, global maximum pooling and 7×7 convolution operations on the features after restoring the channel dimension.

[0021] Furthermore, the image encoder adopts a VisionTransformer architecture pre-trained based on Masked Autoencoder, including 1 block embedding layer and 12 Transformer encoding blocks.

[0022] The present invention also provides a medical image segmentation system based on SAM, which adopts the following technical solution: comprising: an image encoder, a prompt encoder and a mask decoder, wherein the mask decoder is connected to the image encoder and the prompt encoder respectively,

[0023] The image encoder is used to obtain an image encoding vector according to a medical image generator; the image encoder comprises a block embedding layer and a plurality of encoding blocks connected in sequence, the image encoder is embedded with an intuitive perception fine-tuning adapter, and the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each encoding block;

[0024] The prompt encoder is used to obtain a prompt encoding vector according to the prompt content;

[0025] The mask decoder is used to obtain an image segmentation result according to the spliced ​​image coding vector and the hint coding vector.

[0026] The present invention also provides a computer program product, comprising a computer program, which implements the above-mentioned SAM-based medical image segmentation method when executed by a processor.

[0027] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0028] 1. The intuitive perception fine-tuning adapter designed by the present invention introduces additional branch paths into each Transformer encoding block. Compared with the traditional image encoder that only uses serialized layer outputs, the present invention can directly integrate the initial and current feature information of each encoding block. The intuitive perception fine-tuning adapter enables each encoding block to perceive the detailed features of the target domain more deeply, so the present invention is more suitable for processing high-complexity medical image segmentation tasks. Through this intuitive perception method, the model not only retains more useful image information, but also reduces information loss during feature transmission.

[0029] 2. The multi-scale domain transfer adapter designed by the present invention processes features by using convolution kernels of different sizes. Compared with the traditional single-scale feature processing method, it can more comprehensively capture multi-scale features from micro to macro. The multi-scale processing method enables the model to understand images at different resolutions, thereby improving the model's recognition accuracy and generalization ability for complex medical image structures. In addition, the spatial attention mechanism is used in the multi-scale domain transfer adapter to further enhance the expressiveness of the features, allowing the model to focus on the most critical parts of the image, so the present invention has higher segmentation accuracy.

[0030] 3. The present invention uses the intuitive perception fine-tuning adapter and the multi-scale domain migration adapter in combination, so that the overall SAM model architecture can effectively utilize the advanced attention mechanism and multi-scale convolution structure, which significantly improves the expressiveness of features and the accuracy of segmentation compared with the existing medical image segmentation methods. Through the intuitive perception fine-tuning adapter and the multi-scale domain migration adapter, the present invention not only improves the efficiency and accuracy of medical image segmentation, but also has better adaptability and generalization ability, which makes it have significant advantages in practical medical applications.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is an overall flow chart provided by the present invention.

[0034] Figure 2 It is a flow chart of the intuitive perception fine-tuning adapter provided by the present invention.

[0035] Figure 3 It is a flow chart of the multi-scale domain migration adapter provided by the present invention.

[0036] Figure 4 It is a schematic diagram of the image segmentation result provided by the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0038] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0039] As the first visual base model, SAM, including image encoder, cue encoder and mask decoder, has been applied to a variety of downstream tasks such as image restoration and detection. However, the performance of SAM in medical image segmentation still needs to be improved. With the increase of data and model size, the paradigm of pre-training large models and migrating them to downstream tasks is becoming increasingly popular. In the process of transfer learning, researchers proposed PEFT, which transforms the learned general features into task-specific features by introducing a small number of trainable parameters on the pre-trained large model. The existing PEFT generally adds an adapter to each layer of the encoder during model fine-tuning to adjust the features obtained in each layer. However, as the number of layers increases, the information of the target domain will inevitably be lost. In addition, SAM is mainly trained on natural images, which are very different from medical images in nature. Medical images usually have low contrast and diverse anatomical structures, which makes the direct application of these models to medical tasks result in a significant drop in performance, and existing fine-tuning methods have not been able to solve this problem well.

[0040] In order to make the image encoder of the SAM model better adapt to specific medical image segmentation tasks, the present invention introduces two types of adapters: an intuitive perception fine-tuning adapter and a multi-scale domain migration adapter. The intuitive perception fine-tuning adapter adds an additional branch path in each Transformer encoding block, allowing the model to perceive the image content of the target domain and perform fine-tuning in each encoding block. This method not only improves the expressiveness of features, but also reduces the loss of information during multi-layer transmission. The multi-scale domain migration adapter processes features through three parallel convolution paths, which can capture local features of different scales, effectively migrate the learned features from the natural image domain to the medical image domain, and enhance the generalization ability of the model.

[0041] Combine the following Figure 1-Figure 4 The present invention is further described in detail, and a medical image segmentation method, system and product based on SAM are described as follows:

[0042] In this embodiment, Figure 1 As shown, a medical image segmentation method based on SAM is provided, comprising the following steps:

[0043] Step 1: Transform the image encoder of SAM by embedding the Intuitive Perceptual Fine-tuning Adapter (IPF) and the Multi-scale Domain Transfer Adapter (MDT) into the original image encoder.

[0044] In this embodiment, the original image encoder uses the VisionTransformer architecture based on Masked Autoencoder pre-training, which is specifically used to extract high-dimensional features from input medical images. Vision Transformer is a model based on the attention mechanism that can capture the global dependencies of images to better understand the image content. The original image encoder includes 1 block embedding layer and 12 Transformer encoding blocks connected in sequence. We embed 1 intuitive perception fine-tuning adapter and 12 multi-scale domain transfer adapters ( Figure 1 Only one multi-scale domain migration adapter is shown in the figure).

[0045] After the medical image is input into the image encoder, it is divided into features with M image blocks after passing through the block embedding layer. , and then processed step by step by 12 Transformer encoding blocks.

[0046] For the lth Transformer encoding block, l=1,2,...,12, the feature As input, the Transformer encoding block processes:

[0047] feature After layer normalization and multi-head self-attention, the multi-head self-attention feature is obtained. The multi-head self-attention feature and the shared projection feature are input into the intuitive perception fine-tuning adapter to obtain the intuitive feature. The intuitive feature and the feature After adding, feature F1 is obtained. After layer normalization, feature F1 is obtained. Feature F2 is input into the multi-layer perceptron and the multi-scale domain transfer adapter respectively to obtain feature F3 and attention weight respectively. The result of multiplying feature F3 and attention weight is added to feature F1 to obtain feature .

[0048] feature As the input of the l+1th Transformer encoding block. Features is the output of the image encoder, that is, the image coding vector.

[0049] An additional branch path is introduced in the intuitive perception fine-tuning adapter to fuse the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each encoding block, thereby achieving intuitive perception and fine-tuning of the target domain image content, thereby significantly improving the quality and adaptability of feature representation.

[0050] like Figure 2 As shown, the processing process of the intuitive perception fine-tuning adapter includes: the multi-head self-attention feature is normalized, downsampled and activated, and then merged with the shared projection feature to obtain a merged feature. The merged feature is normalized, triple attention mechanism, upsampled and dimensioned to obtain an intuitive feature. The processing process can be expressed as follows:

[0051]

[0052] in, represents the multi-head self-attention feature, represents the shared projection feature, Represents intuitive characteristics, Responsible for downsampling of features, , Responsible for upsampling of features, , Represents the feature matrix, r represents the intermediate dimension of the sampling process, which is generally much smaller than the initial dimension d, and TA represents the triple attention mechanism.

[0053] feature After being processed by the shared projection layer, the shared projection features are mapped through the fully connected operation .

[0054] The triple attention mechanism uses spatial and channel attention to enable the merged features to be transmitted to the target area faster and more accurately. The triple attention mechanism mainly includes three branches, which capture the cross-channel interaction between the C dimension and the W dimension, the cross-channel interaction between the C dimension and the H dimension, and the interaction between the W dimension and the H dimension of the merged feature respectively, and then the triple attention mechanism averages the three interaction results. The three branches are named the first branch, the second branch, and the third branch respectively. The specific processing process of the triple attention mechanism is:

[0055] The first branch establishes an interaction between the H dimension and the C dimension. The processing process includes: rotating the merged feature with a dimension of (C×H×W) 90° counterclockwise along the H axis, and the dimension after rotation becomes (W×H×C); then passing through the pooling layer, the dimension becomes (2×H×C); then passing through a standard convolution layer with a kernel size of 7, and then passing through a batch normalization layer, the dimension becomes (1×H×C); then passing through a sigmoid activation function, and finally rotating 90° clockwise along the H axis to obtain a first interaction result consistent with the dimension of the merged feature.

[0056] The second branch establishes an interaction between the C dimension and the W dimension. The processing process includes: rotating the merged feature with a dimension of (C×H×W) 90° counterclockwise along the W axis, and the dimension after rotation becomes (H×C×W); then passing through the pooling layer, the dimension becomes (2×C×W); then passing through a standard convolution layer with a kernel size of 7, and then passing through a batch normalization layer, the dimension becomes (1×C×W); then passing through a sigmoid activation function, and finally rotating 90° clockwise along the W axis to obtain a second interaction result consistent with the dimension of the merged feature.

[0057] The third branch uses the traditional calculation of spatial attention weights and establishes an interaction between the H dimension and the W dimension. The processing process includes: passing the merged feature with a dimension of (C×H×W) through a pooling layer, and the dimension becomes (2×H×W); then passing through a standard convolution layer with a kernel size of 7, a batch normalization layer and a sigmoid activation function to obtain an attention weight with a shape of (1×H×W), and multiplying the attention weight by the merged feature to obtain the third interaction result.

[0058] The average values ​​of the first interaction result, the second interaction result, and the third interaction result are calculated to obtain the output features of the triple attention mechanism.

[0059] The architectural feature of the intuitive perceptual fine-tuning adapter is that it can directly access and adjust the initial image features in each pre-trained encoding block to better adapt to the target task. This design is mainly achieved through the following steps:

[0060] Image segmentation and initial feature extraction: Medical images are segmented into multiple small image patches, each of which is converted into shared projection features through a shared linear projection layer. The shared projection features constitute the basic unit of image encoder processing.

[0061] Feature processing and inter-layer transfer: Each encoding block of the image encoder uses multi-head self-attention and layer regularization to process these features layer by layer. On this basis, the branch added by IPF merges the multi-head self-attention features of each encoding block with the shared projection features, enhancing the continuity and relevance of the features.

[0062] Intuitive feature path and triple attention mechanism: When processing each image block feature, in addition to the standard Transformer processing flow, IPF also uses shared projection features to generate intuitive features, which are reintroduced and fused in each encoding block. In addition, IPF uses a triple attention mechanism to perform in-depth spatial and channel analysis on the merged features, which helps to capture the details of the target domain more accurately.

[0063] Each encoding block has a multi-scale domain migration adapter embedded in it. The multi-scale domain migration adapter calculates the attention weight based on the input features of the multi-layer perceptron of the encoding block. The attention weight is multiplied by the output features of the multi-layer perceptron to capture multi-scale local features. The multi-scale domain migration adapter is composed entirely of convolutional layers, such as Figure 3 As shown, based on the advantages of convolution operations in capturing local features and realizing hierarchical learning, it is particularly suitable for processing image data.

[0064] The processing process of the multi-scale domain migration adapter includes: reducing the dimension of feature F2 in the channel dimension through a convolution operation, which helps to reduce the complexity of the feature and prepare for subsequent multi-scale processing. Then, after the activation function, three convolution kernels of different sizes are input. The three convolution kernels of different sizes are 1×1, 3×3 and 5×5, respectively, which are optimized for features of different scales. Through this design, the multi-scale domain migration adapter can capture important features of the image from details to the whole, especially the features of different scales from pixel level to organ level in medical images. The three features obtained by the three convolution kernels of different sizes are merged, and then reactivated by normalization and GELU activation function to unify and enhance the expressiveness of the features. Then the channel dimension is restored by a 1×1 convolution kernel. Finally, the attention weight is generated by the spatial attention mechanism through global average pooling, global maximum pooling and 7×7 convolution operation. The attention weight will dynamically adjust the response of the feature according to the importance of different positions in the feature map, so as to ensure that the model can focus on the most critical parts of the medical image.

[0065] Step 2: Input the prompt content into the prompt encoder to obtain the prompt encoding vector.

[0066] Step 3: Concatenate the image coding vector and the prompt coding vector and input them into the mask decoder to obtain the image segmentation result.

[0067] When training SAM, the present invention freezes the parameters of the original image encoder, that is, it is no longer updated during the training process; fine-tunes the parameters of the intuitive perception fine-tuning adapter and the multi-scale domain migration adapter. In this way, the rich visual features learned on large-scale datasets can be utilized without training from scratch. The resolution of all input images is uniformly adjusted to 256×256. The pre-trained SAM of the ViT-Base version is selected as the base model. The learning rate is set to 0.0001 and the maximum training cycle is set to 200. The learning rate decay adopts a linear warm-up and cosine annealing strategy with a warm-up cycle of 10 cycles. The batch size is set to 8. The AdamW optimizer is used for optimization, and the weight decay is set to 1e-8. In order to enhance the training stability and improve the segmentation performance of the model, the loss function adopts a joint loss function combining dice loss and binary cross entropy loss. Then the medical image segmentation effect of the present invention is verified. All experiments were performed on a GeForce RTX 3090 equipped with 24GB of memory.

[0068] The present invention uses LiTS, ISIC, Kvasir, BUSI, CXML and FML datasets to verify the effect of this method (Tran-SAM) and multiple existing visual fine-tuning methods. We added different visual fine-tuning strategies to the image encoder of SAM and evaluated the ability of different strategies to transfer SAM to medical image segmentation tasks. We fully fine-tuned SAM and used this result as a baseline reference. The experimental results are shown in Table 1. The performance indicator is the Dice coefficient, which is the ratio of the overlap between the predicted foreground area and the true foreground area to their average.

[0069] Table 1 Comparison results of fine-tuning strategies

[0070]

[0071] As can be seen from Table 1, the performance of LoRA and its improved version GLoRA is not satisfactory compared with other fine-tuning strategies. On the Kvasir and CXML datasets, their performance is even lower than the full fine-tuning strategy. For VPT, we adopted two versions: VPT_shallow only introduced learnable tags in the first layer of Transformer, while VPT_deep added learnable tags in each layer. However, VPT did not perform well on most tasks, and even showed significant performance degradation in some tasks, performing worse than the full fine-tuning strategy. Although SSF achieved performance improvements on ISIC, Kvasir, and FML datasets, the improvement was limited, and the robustness of the strategy was poor, showing performance degradation on the other three datasets. For AdaptFormer, we conducted experiments on three versions: the base parameter version (Adapt_b, 0.47M), the large parameter version (Adapt_l, 28.36M), and the huge parameter version (Adapt_h, 57.02M). In the experiments, Adapt_b achieved performance improvements on all six datasets. As the model size increases, the performance of most datasets is further improved. However, the performance of LiTS and CXML datasets decreases slightly as the number of parameters increases. This shows that AdaptFormer does have some effectiveness, but the stability of its performance improvement decreases as the number of parameters increases. In contrast, Trans-SAM achieves more significant performance improvements than other methods on the six datasets.

[0072] like Figure 4 As shown, Figure 4 (a) and Figure 4 (b) in the figure are the original image and label respectively; Figure 4 (c)-(f) in the figure are the segmentation effects of TransUNet, TransFuse, UCTransNet and Trans-SAM respectively. Figure 4 It can be seen that our method performs well in segmenting the target region, even in the face of challenging scenarios, such as low contrast between the target tissue and its surrounding structures. This ability is of particular significance in medical imaging, where subtle changes in tissue appearance can convey crucial diagnostic information. The remarkable performance of our method can be attributed to the incorporation of the attention mechanism, which effectively reduces over-segmentation, and it allows the model to focus on relevant regions and mitigate the impact of irrelevant features and noise, thereby improving the overall robustness and generalizability.

[0073] The present invention also provides a medical image segmentation system based on SAM, such as Figure 1As shown, the adopted technical solution is as follows: based on SAM implementation, it includes: an image encoder, a prompt encoder and a mask decoder, and the mask decoder is connected to the image encoder and the prompt encoder respectively.

[0074] The image encoder is used to obtain an image coding vector according to a medical image generator; the image encoder includes a block embedding layer and multiple coding blocks connected in sequence, the image encoder is embedded with an intuitive perception fine-tuning adapter, and the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each coding block; the coding block is embedded with a multi-scale domain migration adapter, and the multi-scale domain migration adapter calculates the attention weight according to the input features of the multi-layer perceptron of the coding block, and the attention weight is multiplied with the output features of the multi-layer perceptron to capture multi-scale local features.

[0075] The prompt encoder is used to obtain a prompt encoding vector according to the prompt content.

[0076] The mask decoder is used to obtain an image segmentation result according to the spliced ​​image coding vector and the hint coding vector.

[0077] The working process of this system is:

[0078] The medical image is input into the image encoder, and the medical image is divided into features with M image blocks after passing through the block embedding layer. , and then processed step by step by 12 Transformer encoding blocks.

[0079] feature After being processed by the shared projection layer, the shared projection features are mapped through the fully connected operation .

[0080] For the lth Transformer encoding block, l=1,2,...,12, the feature As input, the Transformer encoding block processes:

[0081] feature After layer normalization and multi-head self-attention, the multi-head self-attention feature is obtained. The multi-head self-attention feature and the shared projection feature are input into the intuitive perception fine-tuning adapter to obtain the intuitive feature. The intuitive feature and the feature After adding, feature F1 is obtained. After layer normalization, feature F1 is obtained. Feature F2 is input into the multi-layer perceptron and the multi-scale domain transfer adapter respectively to obtain feature F3 and attention weight respectively. The result of multiplying feature F3 and attention weight is added to feature F1 to obtain feature .

[0082] feature As the input of the l+1th Transformer encoding block. Features is the output of the image encoder, i.e., the image coding vector. The image encoder transmits the image coding vector to the mask decoder.

[0083] The processing process of the intuitive perception fine-tuning adapter includes: the multi-head self-attention features are normalized, down-sampled and activated, and then merged with the shared projection features to obtain merged features. The merged features are normalized, triple attention mechanism, up-sampled and dimensioned to obtain intuitive features.

[0084] The specific processing process of the triple attention mechanism is:

[0085] The first branch establishes an interaction between the H dimension and the C dimension. The processing process includes: rotating the merged feature with a dimension of (C×H×W) 90° counterclockwise along the H axis, and the dimension after rotation becomes (W×H×C); then passing through the pooling layer, the dimension becomes (2×H×C); then passing through a standard convolution layer with a kernel size of 7, and then passing through a batch normalization layer, the dimension becomes (1×H×C); then passing through a sigmoid activation function, and finally rotating 90° clockwise along the H axis to obtain a first interaction result consistent with the dimension of the merged feature.

[0086] The second branch establishes an interaction between the C dimension and the W dimension. The processing process includes: rotating the merged feature with a dimension of (C×H×W) 90° counterclockwise along the W axis, and the dimension after rotation becomes (H×C×W); then passing through the pooling layer, the dimension becomes (2×C×W); then passing through a standard convolution layer with a kernel size of 7, and then passing through a batch normalization layer, the dimension becomes (1×C×W); then passing through a sigmoid activation function, and finally rotating 90° clockwise along the W axis to obtain a second interaction result consistent with the dimension of the merged feature.

[0087] The third branch uses the traditional calculation of spatial attention weights and establishes an interaction between the H dimension and the W dimension. The processing process includes: passing the merged feature with a dimension of (C×H×W) through a pooling layer, and the dimension becomes (2×H×W); then passing through a standard convolution layer with a kernel size of 7, a batch normalization layer and a sigmoid activation function to obtain an attention weight with a shape of (1×H×W), and multiplying the attention weight by the merged feature to obtain the third interaction result.

[0088] The average values ​​of the first interaction result, the second interaction result, and the third interaction result are calculated to obtain the output features of the triple attention mechanism.

[0089] The processing process of the multi-scale domain migration adapter includes: reducing the dimension of feature F2 in the channel dimension through a convolution operation. Then after the activation function, three convolution kernels of different sizes are input, and the three convolution kernels of different sizes are 1×1, 3×3 and 5×5 respectively. The three features obtained by the three convolution kernels of different sizes are merged, and then reactivated by normalization and GELU activation function. The channel dimension is restored by a 1×1 convolution kernel. Finally, the attention weight is generated through the spatial attention mechanism through global average pooling, global maximum pooling and 7×7 convolution operations.

[0090] The prompt content is input into the prompt encoder to obtain a prompt encoding vector. The prompt encoder transmits the prompt encoding vector to the mask decoder.

[0091] The mask decoder calculates the image segmentation result according to the splicing result of the image coding vector and the hint coding vector.

[0092] The present invention also provides a computer program product, comprising a computer program, which implements the above-mentioned SAM-based medical image segmentation method when executed by a processor.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method based on SAM, characterized in that: include: Inputting a medical image into an image encoder to obtain an image encoding vector; the image encoder comprises a block embedding layer and a plurality of encoding blocks connected in sequence, the image encoder is embedded with an intuitive perception fine-tuning adapter, and the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each encoding block; The processing process of the intuitive perception fine-tuning adapter includes: the multi-head self-attention feature is normalized, down-sampled and activated, and then merged with the shared projection feature to obtain a merged feature, and the merged feature is normalized, triple attention mechanism, and up-sampled to obtain an intuitive feature; The processing process of the triple attention mechanism includes: capturing the interaction between the C dimension and the W dimension, the interaction between the C dimension and the H dimension, and the interaction between the W dimension and the H dimension of the merged feature through three branches, and then averaging the three interaction results; The encoding block has a multi-scale domain migration adapter embedded therein, and the multi-scale domain migration adapter calculates attention weights according to input features of the multi-layer perceptron of the encoding block, and the attention weights are multiplied by output features of the multi-layer perceptron; The input features of the multilayer perceptron are subjected to a convolution operation to reduce the dimension in the channel dimension, and then after the activation function, three convolution kernels of different sizes are input, and the three features obtained are merged, and then normalized, GELU activation function, 1×1 convolution kernel is used to restore the channel dimension, and then the attention weight is obtained through the spatial attention mechanism; Input the prompt content into the prompt encoder to obtain a prompt encoding vector; The image coding vector and the hint coding vector are concatenated and input into a mask decoder to obtain an image segmentation result.

2. A medical image segmentation method based on SAM as claimed in claim 1, characterized in that: The three branches are respectively a first branch, a second branch and a third branch; The processing process of the first branch includes: rotating the merged feature 90° counterclockwise along the H axis, then passing through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function, and then rotating 90° clockwise along the H axis to obtain a first interaction result; The processing process of the second branch includes: rotating the merged feature 90° counterclockwise along the W axis, then passing through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function, and then rotating 90° clockwise along the W axis to obtain a second interaction result; The processing process of the third branch includes: passing the merged features through a pooling layer, a convolution layer, a normalization layer, and a sigmoid activation function to obtain an attention weight, and multiplying the attention weight by the merged features to obtain a third interaction result.

3. The medical image segmentation method based on SAM as claimed in claim 1, characterized in that: The three convolution kernels of different sizes are 1×1, 3×3, and 5×5; The processing process of the spatial attention mechanism includes: generating attention weights by performing global average pooling, global maximum pooling and 7×7 convolution operations on the features after restoring the channel dimension.

4. The medical image segmentation method based on SAM as claimed in claim 1, characterized in that: The image encoder adopts the Vision Transformer architecture based on Masked Autoencoder pre-training, including 1 block embedding layer and 12 Transformer encoding blocks.

5. A medical image segmentation system based on SAM, characterized in that: A method for performing a medical image segmentation method based on SAM as claimed in any one of claims 1 to 4, comprising: an image encoder, a hint encoder and a mask decoder, wherein the mask decoder is connected to the image encoder and the hint encoder respectively. The image encoder is used to obtain an image encoding vector according to a medical image generator; the image encoder comprises a block embedding layer and a plurality of encoding blocks connected in sequence, the image encoder is embedded with an intuitive perception fine-tuning adapter, and the intuitive perception fine-tuning adapter fuses the shared projection features of the output of the block embedding layer into the multi-head self-attention features of each encoding block; The processing process of the intuitive perception fine-tuning adapter includes: the multi-head self-attention feature is normalized, down-sampled and activated, and then merged with the shared projection feature to obtain a merged feature, and the merged feature is normalized, triple attention mechanism, and up-sampled to obtain an intuitive feature; The processing process of the triple attention mechanism includes: capturing the interaction between the C dimension and the W dimension, the interaction between the C dimension and the H dimension, and the interaction between the W dimension and the H dimension of the merged feature through three branches, and then averaging the three interaction results; The encoding block has a multi-scale domain migration adapter embedded therein, and the multi-scale domain migration adapter calculates attention weights according to input features of the multi-layer perceptron of the encoding block, and the attention weights are multiplied by output features of the multi-layer perceptron; The input features of the multilayer perceptron are subjected to a convolution operation to reduce the dimension in the channel dimension, and then after the activation function, three convolution kernels of different sizes are input, and the three features obtained are merged, and then normalized, GELU activation function, 1×1 convolution kernel is used to restore the channel dimension, and then the attention weight is obtained through the spatial attention mechanism; The prompt encoder is used to obtain a prompt encoding vector according to the prompt content; The mask decoder is used to obtain an image segmentation result according to the spliced ​​image coding vector and the hint coding vector.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the medical image segmentation method based on SAM described in any one of claims 1 to 4 is implemented.

Citation Information

Cited By

  • Method for detecting retinopathy hard exudate cluster set under OCT (optical coherence tomography) image

    CN122453824A