Tumor CT image segmentation method based on SAM large model

Abstract

The invention discloses a tumor CT image segmentation method based on an SAM large model, and belongs to the field of image segmentation. The method comprises the following steps: respectively inputting a CT image to be segmented into a Unet encoder and an SAM encoder, and carrying out multi-scale cross-model fusion on output features of the Unet encoder and the SAM encoder to obtain a target fusion feature map; performing segmentation prediction based on the output characteristics of the Unet encoder by using the Unet decoder, and generating a sparse prompt point set of each category based on a segmentation prediction result; inputting the sparse prompt point set of each category into an SAM prompt encoder to obtain sparse prompt position codes and dense prompt position codes; and inputting the target fusion feature map, the sparse prompt position code and the dense prompt position code into an SAM decoder to obtain a final segmentation result. According to the scheme, the defects that in an existing segmentation method, manual prompt is relied on, and missing detection and misjudgment of small targets and multiple targets are remarkably reduced can be overcome.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a tumor CT image segmentation method based on the SAM large model. Background Technology

[0002] The main goal of CT image segmentation is to automate the segmentation of tumor CT images using deep learning methods. This assists doctors in accurately identifying and segmenting lesion areas in CT images, thereby aiding in diagnosis, surgical planning, and treatment evaluation, and improving the accuracy and efficiency of clinical treatment. A key objective of current tumor CT image segmentation tasks is to achieve more precise segmentation of irregular, small, or numerous organs or tumor lesions.

[0003] In the field of image segmentation, most models employ deep learning frameworks like U-Net or the SAM segmentation framework to perform segmentation tasks, achieving pixel-level image classification using an encoder-decoder structure. The encoder extracts deep features by encoding the image, compressing spatial resolution to capture high-level semantic information; the decoder restores the image resolution through upsampling or convolution operations to recognize image features. This structure effectively captures both local and global image features, making it suitable for high-precision segmentation tasks.

[0004] However, current tumor CT image segmentation models can effectively segment tumors and visceral regions in most cases, achieving a level consistent with or even surpassing that of clinicians. But for some smaller tumors with unclear margins on CT images, problems such as missed detections, misjudgments, and margin deformities still exist, resulting in unsatisfactory segmentation outcomes that fail to meet actual clinical reference requirements.

[0005] Therefore, there is an urgent need to provide a new method for tumor CT image segmentation. Summary of the Invention

[0006] To address the issues of missed detections, misjudgments, and edge deformities in current tumor CT image segmentation models for CT images with small tumors and unclear edges, this invention provides a tumor CT image segmentation method based on the SAM large model.

[0007] On the one hand, a tumor CT image segmentation method based on a large SAM model is provided, the method comprising: The CT image to be segmented is input into the Unet encoder and the SAM encoder respectively. The output features of the Unet encoder and the SAM encoder are fused across multiple scales to obtain the target fusion feature map. The Unet decoder is used to perform segmentation prediction based on the output features of the Unet encoder, and a sparse cue point set for each category is generated based on the segmentation prediction results. The sparse cue point set of each category is input into the SAM cue encoder to obtain the sparse cue location code and the dense cue location code; The target fusion feature map, the sparse cue location code, and the dense cue location code are input into the SAM decoder to obtain the final segmentation result.

[0008] On the other hand, a tumor CT image segmentation device based on a large SAM model is provided, which is based on the steps described in any embodiment of the method in the specification. The device includes: The fusion unit is used to input the CT image to be segmented into the Unet encoder and the SAM encoder respectively, and to perform multi-scale cross-model fusion of the output features of the Unet encoder and the SAM encoder to obtain the target fusion feature map. The cue unit is used to perform segmentation prediction based on the output features of the Unet encoder using the Unet decoder, and to generate a sparse cue point set for each category based on the segmentation prediction results. The overlay unit is used to input the sparse cue point set of each category into the SAM cue encoder to obtain the sparse cue position code and the dense cue position code; The segmentation unit is used to input the target fusion feature map, the sparse cue location code, and the dense cue location code into the SAM decoder to obtain the final segmentation result.

[0009] On the other hand, a computer device is provided, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for executing the computer program stored in the memory to implement the steps of the method described above.

[0010] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the method described above.

[0011] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0012] The technical solution provided by this invention can bring at least the following beneficial effects: This method integrates the SAM segmentation model with the traditional Unet segmentation network. It fuses multi-layer features from the Unet encoder into the SAM encoder output to enhance local detail awareness while maintaining global semantic consistency. Furthermore, it generates corresponding high-probability class points from Unet and feeds them into the SAM prompt encoder, achieving high-quality automatic segmentation of tumors and internal organs. This avoids the missed detections and misjudgments caused by manual prompts in practical applications, while also improving the segmentation accuracy for small and multiple targets. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a tumor CT image segmentation method based on a large SAM model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a tumor CT image segmentation model architecture provided in an embodiment of the present invention; Figure 3 This is a structural diagram of a tumor CT image segmentation device based on a large SAM model provided in an embodiment of the present invention; Figure 4 This is a hardware architecture diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0016] The following describes the specific implementation of the above concept.

[0017] Please refer to Figure 1 This invention provides a tumor CT image segmentation method based on a large SAM model, the method comprising: Step 100: Input the CT image to be segmented into the Unet encoder and the SAM encoder respectively, and perform multi-scale cross-model fusion of the output features of the Unet encoder and the SAM encoder to obtain the target fusion feature map; Step 102: Use the Unet decoder to perform segmentation prediction based on the output features of the Unet encoder, and generate a sparse cue point set for each category based on the segmentation prediction results; Step 104: Input the sparse cue point set of each category into the SAM cue encoder to obtain the sparse cue position code and the dense cue position code; Step 106: Input the target fusion feature map, sparse cue location code, and dense cue location code into the SAM decoder to obtain the final segmentation result.

[0018] In this embodiment of the invention, the SAM segmentation model and the traditional Unet segmentation network are integrated. Multi-layer features from the Unet encoder are fused into the SAM encoder output, and corresponding high-probability points for each category are generated from the Unet and fed back into the SAM prompt encoder, achieving high-quality automatic segmentation of tumors and internal organs. This overcomes the shortcomings of existing SAM segmentation methods, such as reliance on manual prompts and the potential for missed detections and misjudgments of small or multiple targets.

[0019] The following description Figure 1 The execution method of each step is shown.

[0020] For step 100: In some implementation methods, reference may be made to Figure 2 The model architecture diagram shows that the CT image to be segmented is input into the Unet encoder and the SAM encoder, respectively, including: The CT image to be segmented is input into the Unet encoder to extract features from the interpolated three-channel image using the multi-scale downsampling coding layer and bottleneck layer of the Unet encoder. The CT image to be segmented is input into the SAM encoder through three-channel bilinear interpolation to extract a high-resolution image.

[0021] In this embodiment, the CT image to be segmented is a grayscale image of (B, 1, H, W). The Unet encoder is used to progressively downsample the CT image and extract rich multi-scale features, ultimately obtaining the bottleneck layer features with a shape of (B, 1024, H / 16, W / 16). With the SAM encoder frozen, the interpolated three-channel image is input into the SAM encoder to extract a high-resolution image with a shape of (B, 256, 64, 64).

[0022] In some implementation methods, reference may be made to Figure 2 The model architecture diagram shows that the output features of the Unet encoder and SAM encoder are fused across multiple scales to obtain the target fused feature map, including: The three feature maps output from the third downsampling coding layer, the fourth downsampling coding layer, and the bottleneck layer of the Unet encoder are channel-projected respectively, and bilinear interpolation is used to unify each projected feature map to the same spatial scale as the SAM encoder output. The three projected feature maps are stitched together along the channel dimension, and a lightweight convolutional aggregator is used to perform multi-scale feature fusion on the stitched image to obtain a fused feature map. The fused feature map is used as the key and value representations of the cross-attention layer, and the high-resolution image extracted by the SAM encoder is used as the query representation to obtain the target fused feature map.

[0023] In this embodiment, U-Net's Layer features , Layer features Bottleneck layer features First, channel projection is performed using 1×1 convolution, and then bilinear interpolation is used to unify the spatial scale to match the output of the SAM encoder. : in, For the i-th projected feature map, Indicates linear projection. This indicates the scale alignment of bilinear interpolation.

[0024] Subsequently, the three projected features are concatenated along the channel dimension, and multi-scale feature fusion is achieved through a lightweight convolutional aggregator to obtain a fused feature map. : in, This indicates a feature concatenation operation. Composed of convolution, batch normalization, and nonlinear activation functions, it is used to compress redundant information and learn cross-scale contextual relationships.

[0025] Based on this, feature maps are fused. As a key-value representation, the SAM encoder extracts high-resolution images. The query representation is input to the cross-attention layer for interaction, resulting in the target fusion feature map: in, These are learnable linear mappings, This is the scaling factor.

[0026] In this embodiment, the three feature maps output from the third downsampling coding layer, the fourth downsampling coding layer, and the bottleneck layer of the Unet encoder are concatenated after channel projection rows. A lightweight convolutional aggregator is then used to perform multi-scale feature fusion on the concatenated image to obtain a fused feature map. The outputs of the third and fourth downsampling coding layers can increase the detail information of small targets. The fusion of the outputs of the Unet encoder and the SAM encoder using a cross-attention layer can improve the attention to the small target features extracted by Unet.

[0027] Regarding step 102: In some implementations, step 102 may include: The Unet decoder is used to perform segmentation prediction based on the output features of the Unet encoder. A normalized exponential function is applied to the segmentation prediction to calculate the pixel probability map for each foreground category. For each foreground category, the pixel probability map for each category Above, select the one with the highest confidence level. An initial sparse cue point set is generated from 100 pixels. If the maximum confidence of a certain category is lower than a fixed threshold, empty placeholder cue points are used as the initial sparse cue point set. The pixel position encodings in the initial sparse cue point set are scaled and mapped according to the image ratio to obtain a sparse cue point set for each foreground category carrying category label values.

[0028] In this embodiment, the sparse cue point set is determined in the following way: The pixel probability map for each foreground category is as follows: In the formula, c is the foreground category number, and C is the number of foreground categories. For segmentation prediction, k is the foreground category variable; The initial set of sparse cue points is: In the formula, Encode the position of the i-th pixel in category c; Scaling mapping can be performed as follows: In the formula, The position encoding of the i-th pixel in category c after scaling and mapping. and The width and height of the CT image to be segmented, Provide the feature space of the encoder for SAM; The sparse cue set is: In the formula, Let c be the set of sparse cue points for the c-th foreground category. The foreground category label value for this sparse cue point set.

[0029] In this embodiment, the bottleneck layer feature map is progressively passed through the Unet upsampling decoder, and skip links are used with the encoding side features to generate preliminary multi-class segmentation with the same size as the input features. Based on the number of classes required at runtime, the first n channels are truncated to obtain the corresponding segmentation prediction. A normalized exponential function is applied to the segmentation prediction to calculate the pixel probability map for each foreground class. For each class, several pixel positions with the highest probability are selected and scaled to 1024×1024 to generate sparse cue points and corresponding labels; if there are no high-probability points, a default cue is used at the center position.

[0030] Regarding step 104: refer to Figure 2 The model architecture freezes the SAM cue encoder of the large SAM model, and feeds the generated sparse cue point set of each category into the SAM cue encoder to obtain sparse cue position encoding and dense cue position encoding.

[0031] Specifically: in, This refers to the Prompt Encoder operation in SAM. Encoding sparse cue positions, Encodes the location of dense prompts.

[0032] In this embodiment, by first using the Unet decoder to generate a sparse cue point set for each category, and then using the SAM cue encoder to generate sparse cue location codes and dense cue location codes based on the sparse cue point set for each category, the extraction of small target locations can be greatly improved.

[0033] Regarding step 106: In some implementations, step 106 may include: The target fusion feature map, sparse cue location code, and dense cue location code are input into the bidirectional Transformer module of the SAM decoder to obtain the updated mask token representation; For each foreground category, the corresponding mask token representation is mapped to dynamic weights using a hypernetwork MLP, and then linearly combined with the upsampled spatial features to generate mask predictions for each foreground category. The IoU prediction head outputs the IoU quality score for each foreground category; By stitching together channels with zero background, the final confidence maps for each channel are obtained. The multi-class mask prediction is restored to the original input size through bilinear interpolation to obtain the final segmentation result.

[0034] This solution constructs a novel tumor CT image segmentation framework that integrates a large segmentation model with the traditional Unet segmentation network. It achieves feature fusion across different models, channels, and resolutions, enabling more efficient extraction of global semantic features from CT images. Ultimately, it automates image segmentation based on the SAM model and improves the robustness and accuracy of segmentation for different organs and tumor types.

[0035] Compared with existing technologies, the segmentation framework proposed in this invention establishes a more convenient, accurate, and stable automatic segmentation method for tumors. It improves the accuracy of current tumor image segmentation and its usability in some unconventional situations. Experimental results show that the proposed model achieves approximately 93% accuracy in visceral / tumor segmentation, leading existing models by 1% in accuracy, and maintains good usability and robustness across different tumor types and organs.

[0036] Please refer to Figure 3 This invention provides a tumor CT image segmentation device based on a large SAM model, used to implement the steps of any method embodiment in the specification. The device includes: The fusion unit 301 is used to input the CT image to be segmented into the Unet encoder and the SAM encoder respectively, and to perform multi-scale cross-model fusion of the output features of the Unet encoder and the SAM encoder to obtain the target fusion feature map. The cue unit 302 is used to perform segmentation prediction based on the output features of the Unet encoder using the Unet decoder, and to generate a sparse cue point set for each category based on the segmentation prediction results. The overlay unit 303 is used to input the sparse cue point set of each category into the SAM cue encoder to obtain the sparse cue position code and the dense cue position code; The segmentation unit 304 is used to input the target fusion feature map, sparse cue location code and dense cue location code into the SAM decoder to obtain the final segmentation result.

[0037] It should be noted that the above device embodiments and method embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0038] Embodiments of this application also provide a computer device, please refer to... Figure 4The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the tumor CT image segmentation method based on the SAM large model provided in the above method embodiments.

[0039] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the tumor CT image segmentation method based on the SAM large model provided in the above-described method embodiments.

[0040] Embodiments of this application also provide a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium and executes the computer program, causing the computer device to perform any of the tumor CT image segmentation methods based on the SAM large model described in the above embodiments.

[0041] For ease of description, the above devices or apparatuses are described separately according to their functions, divided into various modules or units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0042] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0043] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0044] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A tumor CT image segmentation method based on a large SAM model, characterized in that, include: The CT image to be segmented is input into the Unet encoder and the SAM encoder respectively. The output features of the Unet encoder and the SAM encoder are fused across multiple scales to obtain the target fusion feature map. The Unet decoder is used to perform segmentation prediction based on the output features of the Unet encoder, and a sparse cue point set for each category is generated based on the segmentation prediction results. The sparse cue point set of each category is input into the SAM cue encoder to obtain the sparse cue location code and the dense cue location code; The target fusion feature map, the sparse cue location code, and the dense cue location code are input into the SAM decoder to obtain the final segmentation result.

2. The method as described in claim 1, characterized in that, The step of inputting the CT image to be segmented into the Unet encoder and the SAM encoder respectively includes: The CT image to be segmented is bilinearly interpolated in three channels and input into the Unet encoder, so that the multi-scale downsampling coding layer and bottleneck layer of the Unet encoder can be used to extract features from the interpolated three-channel image. The interpolated three-channel image is input into the SAM encoder to extract a high-resolution image.

3. The method as described in claim 1, characterized in that, The step of performing multi-scale cross-model fusion of the output features of the Unet encoder and the SAM encoder to obtain the target fused feature map includes: The three feature maps output from the third downsampling coding layer, the fourth downsampling coding layer, and the bottleneck layer of the Unet encoder are channel-projected respectively, and bilinear interpolation is used to unify each projected feature map to the same spatial scale as the SAM encoder output. The three projected feature maps are stitched together along the channel dimension, and a lightweight convolutional aggregator is used to perform multi-scale feature fusion on the stitched image to obtain a fused feature map. The fused feature map is used as the key and value representations of the cross-attention layer, and the high-resolution image extracted by the SAM encoder is used as the query representation to obtain the target fused feature map.

4. The method as described in claim 1, characterized in that, The step of using the Unet decoder to perform segmentation prediction based on the output features of the Unet encoder, and generating a sparse cue point set for each category based on the segmentation prediction results, includes: The Unet decoder is used to perform segmentation prediction based on the output features of the Unet encoder. A normalized exponential function is applied to the segmentation prediction to calculate the pixel probability map for each foreground category. For each foreground category, select the pixel with the highest confidence level from the corresponding pixel probability map. For each pixel, an initial sparse cue point set is generated. If the maximum confidence of a certain category is lower than a fixed threshold, then empty placeholder cue points are used as the initial sparse cue point set. The pixel position encodings in the initial sparse cue point set are scaled and mapped according to the image ratio to obtain a sparse cue point set for each foreground category carrying category label values.

5. The method as described in claim 4, characterized in that, The sparse cue point set is determined in the following way: The pixel probability map for each foreground category is as follows: In the formula, c is the foreground category number, and C is the number of foreground categories. For the segmentation prediction, k is the foreground category variable; The initial set of sparse cue points is: In the formula, Encode the position of the i-th pixel in category c; Scaling mapping can be performed as follows: In the formula, The position encoding of the i-th pixel in category c after scaling and mapping. and The width and height of the CT image to be segmented, Provide the feature space of the encoder for SAM; The sparse cue set is: In the formula, Let c be the set of sparse cue points for the c-th foreground category. The foreground category label value for this sparse cue point set.

6. The method according to any one of claims 1-5, characterized in that, The step of inputting the target fusion feature map, the sparse cue location code, and the dense cue location code into the SAM decoder to obtain the final segmentation result includes: The target fusion feature map, the sparse cue location code, and the dense cue location code are input into the bidirectional Transformer module of the SAM decoder to obtain the updated mask token representation; For each foreground category, the corresponding mask token representation is mapped to dynamic weights using a hypernetwork MLP, and then linearly combined with the upsampled spatial features to generate mask predictions for each foreground category. The IoU prediction head outputs the IoU quality score for each foreground category; By stitching together channels with zero background, the final confidence maps for each channel are obtained. The multi-class mask prediction is restored to the original input size through bilinear interpolation to obtain the final segmentation result.

7. A tumor CT image segmentation device based on a large SAM model, used to implement the steps of the method described in any one of claims 1-6, characterized in that, include: The fusion unit is used to input the CT image to be segmented into the Unet encoder and the SAM encoder respectively, and to perform multi-scale cross-model fusion of the output features of the Unet encoder and the SAM encoder to obtain the target fusion feature map. The cue unit is used to perform segmentation prediction based on the output features of the Unet encoder using the Unet decoder, and to generate a sparse cue point set for each category based on the segmentation prediction results. The overlay unit is used to input the sparse cue point set of each category into the SAM cue encoder to obtain the sparse cue position code and the dense cue position code; The segmentation unit is used to input the target fusion feature map, the sparse cue location code, and the dense cue location code into the SAM decoder to obtain the final segmentation result.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

Images