Gastric cancer lesion region segmentation method for 3D medical image
By adopting the method of enhanced decoders and dual attention bips in 3D medical image segmentation, the problem of missing spatial feature information and automatic prompts are solved, and the accurate and complete segmentation of the gastric cancer lesions area is improved. accuracy.
Patent Information
- Application Number
- CN202411320337.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-24
AI Technical Summary
The prior art has problems with the loss of spatial feature information and the lack of automatic prompting methods in 3D medical image segmentation, especially in the segmentation of gastric cancer lesions.
Using the method of feature enhancement decoder and dual attention prompter, by integrating the original image features and utilizing the spatial and channel self-attention mechanism, the gastric cancer lesion area segmentation of 3D medical images is achieved, and an automatic prompting mechanism is provided.
It improves the accuracy and completeness of 3D medical image segmentation, reduces misjudgments, and effectively prevents the fatigue and dazzling problems of naked eye recognition.
Smart Images
Figure CN119359740A_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the technical field of computer image processing, and in particular to a method for segmenting gastric cancer lesion regions for 3D medical images. [Background Technology]
[0002] CT medical image segmentation can accurately depict the lesion area, assist doctors in disease diagnosis, and play an important role in modern medicine. 3D medical imaging is a common mode, and its segmentation is also an important part of medical image segmentation.
[0003] In recent years, Meta has launched a pre-trained model for 2D image segmentation, the Segment Anything Model (SAM), which has powerful segmentation performance. Therefore, researchers have also tried to apply it to the field of 3D segmentation. A simple application method is to cut 3D into multiple 2D slices, or use pseudo-3D technology to integrate the entire 3D image into SAM. Although these methods have made some progress, they still have limitations, mainly in the following two aspects.
[0004] (1) There is a problem of loss of spatial feature information. Pseudo-3D processing is an approximation of the expected 3D image processing, which will lead to the loss of spatial feature information. This loss is likely to reduce the segmentation performance. Specifically, it manifests as inaccurate segmentation boundaries and incomplete regions, especially for hollow organs such as the stomach.
[0005] (2) The problem of lack of automatic hinting methods. There is a lack of automatic hinting methods for SAM in 3D medical image segmentation. SAM was proposed for 2D image segmentation and supports multiple hints, including frontier points, background points, boxes, and masks. However, the implementation of these interactive segmentation techniques is more challenging in the real-world 3D image segmentation environment. Moreover, these hinting methods require a lot of external expert knowledge. Although SEEM proposed a frontier point hinting method for 3D medical image segmentation, its actual implementation in real-world scenarios remains a major challenge. Therefore, there is an urgent need to develop an automatic hinting mechanism for SAM in 3D medical segmentation. In addition, there has been no research in the field of gastric cancer tumor segmentation in SAM. [Summary of the invention]
[0006] The present invention overcomes the shortcomings of the prior art and provides a method for segmenting gastric cancer lesion areas for 3D medical images, which can accurately and completely segment lesion areas from CT images, and can automatically prompt SAM in three-dimensional medical segmentation, assist doctors in making rapid diagnoses, improve efficiency and accuracy, avoid misjudgments, and prevent problems such as fatigue or dizziness in naked eye recognition.
[0007] To achieve the above object, the present invention provides a gastric cancer lesion region segmentation method for 3D medical images, comprising:
[0008] S1, decompose the 3D medical image into a series of block feature maps and input them into the image encoder for processing;
[0009] S2, inputting the 3rd layer data, the 6th layer data and the 9th layer data processed by the image encoder into the feature enhancement decoder;
[0010] S3, the 12th layer data obtained by the image encoder is input into the dual attention prompter to integrate the prior knowledge of the dual attention mechanism, and then input into the feature enhancement decoder after processing;
[0011] S4, the feature enhancement decoder concatenates the data after feature enhancement on the 4 layers of data, and the concatenated data passes through the prediction layer to obtain the final image segmentation result.
[0012] As described above, a gastric cancer lesion region segmentation method for 3D medical images is used. In S1, pseudo 3D patching and patch embedding are used to segment the gastric cancer lesion region of the shape The shape of the 3D medical image decomposition map is a series of block feature maps of H×W×D×C, where They represent the initial height, width and depth of the image respectively; N represents the number of channels; H, W, D represent the height, width and depth of the feature map respectively, and C represents the channel dimension.
[0013] In the gastric cancer lesion area segmentation method for 3D medical images as described above, the image encoder in S2 includes 12 transformation network layers, and the transformation network layer includes 1 attention layer, 1 multi-layer perceptron layer and 2 normalization layers.
[0014] As described above, a gastric cancer lesion region segmentation method for 3D medical images sets the attention mechanism parameters of the attention layer and the multi-layer perceptron parameters of the multi-layer perceptron layer, and fine-tunes the multi-layer perceptron parameters by scaling the parallel adapter. The calculation formula is:
[0015] Adapter(X) = ReLU(XW down )W up
[0016] in, represents the feature map; ReLU(·) represents the activation function; and All represent mapping weights, and l represents the dimension;
[0017] The calculation of the entire transformation network layer is as follows, and the output feature map of the i-th transformation network layer is expressed as:
[0018]
[0019] Among them, Norm(·) represents the normalization layer; Attention * (·) and MLP * (·) denotes the attention and multilayer perceptron layers with frozen parameters, respectively; s denotes the scaling factor, is an intermediate variable.
[0020] As described above, in the gastric cancer lesion area segmentation method for 3D medical images, the dual attention prompter in S3 includes spatial attention and channel attention.
[0021] As described above, a gastric cancer lesion region segmentation method for 3D medical images, spatial attention is a pixel-level spatial self-attention, and the number of tags belonging to Ki,SA and Vi,SA is reduced to a constant level through a linear layer. The calculation formula of spatial attention is:
[0022]
[0023] in, represents the weight parameter of spatial self-attention; Linear(·) represents a linear mapping that reduces the number of tokens to a constant n; Norm(·) represents a normalization layer; Softmax(·) represents a softmax function;
[0024] Channel attention aims to extract key information based on channels. The calculation formula of channel attention is:
[0025]
[0026] in, The weight parameter representing the channel self-attention is the spatial attention and With channel attention, ie, shared;
[0027] Connect the features of the two attention modules and use the residual connection before the output, then we have:
[0028]
[0029] in represents the feature map containing automatic prompt information; W down,SA , Represents the linear mapping parameters.
[0030] In the gastric cancer lesion region segmentation method for 3D medical images as described above, the feature enhancement decoder in S4 includes 4 original feature enhancers and 1 prediction layer.
[0031] In the method for gastric cancer lesion area segmentation in 3D medical images as described above, the original feature enhancer is divided into two different branches. The task of the first branch is to extract the original image features using a convolution block, including a 3D convolution and an instance normalization layer; the second branch processes the feature map by upsampling to match the size of the original image features; the features of the two branches are connected and integrated through convolution blocks to enhance the features.
[0032] In the above-mentioned gastric cancer lesion region segmentation method for 3D medical images, the enhanced features of the four original feature enhancers are connected in the prediction layer, and the calculation formula of the feature enhancement decoder is:
[0033]
[0034] Y=Predict(Concat(E1,…,E j ,…,E4))##
[0035] Among them, Z i represents the feature map from the i-th layer of the image encoder, i = 3, 6, 9, 12; I represents the original image; E j represents the output of the j-th enhancer, j = 1, 2, 3, 4; Y represents the predicted segmentation; Upsample(·) represents upsampling; ConvBlock(·) represents convolutional block; Concat(·) represents concatenated features; Predict(·) represents the prediction layer; Represents an intermediate variable.
[0036] Compared with the prior art, the gastric cancer lesion area segmentation method for 3D medical images of the present invention has the following beneficial effects:
[0037] 1. The present invention is provided with a feature enhancement decoder, which enhances the encoded feature map by integrating the original image, and uses each feature map in the three transformation layers (3, 6, 9, 12) to capture the image semantics at different depths, thereby greatly alleviating the problem of spatial feature information loss.
[0038] 2. The present invention sets up a dual attention mechanism, which provides prior information by combining the spatial self-attention mechanism (Linformer) and the channel self-attention mechanism (channel), thereby providing position information of the positioning segmentation area and reducing the computational overhead of global self-attention on 3D medical images.
[0039] 3. It can efficiently and accurately identify CT images, avoid misjudgment, and effectively prevent problems such as fatigue or dizziness caused by naked eye recognition. [Drawings]
[0040] Figure 1 This is a structural framework diagram of the present invention. [Specific implementation method]
[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.
[0042] Figure 1 is the overall framework of DEAP-3DSAM; Figure 1 As shown in the figure, part a represents the feature augmentation decoder; part b illustrates the original feature enhancer; part c illustrates the dual attention prompter; part d represents the adapter; part e represents the convolution block; and part f represents the prediction layer.
[0043] like Figure 1 As shown, a gastric cancer lesion region segmentation method for 3D medical images, specifically for 3D CT medical images, includes:
[0044] S1. Decompose the 3D medical image into a series of block feature maps and input them into the image encoder for processing.
[0045] Specifically, Figure 1 As shown in the left part of part a, DEAP-3DSAM utilizes pseudo 3D patching, similar to the 3DSAM adapter. After pseudo 3D patching and patch embedding, the shape of the 3D image becomes H×W×D×C. Represent the initial height, width and depth of the image respectively; N is the number of channels; H, W, D represent the height, width and depth of the feature map respectively; C represents the channel dimension. It is then input into the image encoder. The DEAP-3DSAM image encoder contains 12 identical transformation network layers (Transformer layers). Each layer contains an attention layer, a multi-layer perceptron (MLP) layer, and two normalization layers. In order to use the pre-trained parameters of SAM as much as possible, the parameters of the attention mechanism and the MLP layer are fixed, and a scaled parallel adapter is added for fine-tuning. He et al. proposed the Scale Parallel Adapter, which is one of the best choices for fine-tuning the Transformer layer. They proved that adapters placed in parallel are better than adapters placed sequentially, and adapters parallel to the feedforward network are more effective than adapters aligned with the multi-head attention mechanism. Therefore, the image encoder uses the Scale Parallel Adapter in the MLP layer, such as Figure 1 As shown in part d, the calculation formula is:
[0046] Adapter(X) = ReLU(XW down )W up #
[0047] in represents the feature map; ReLU(·) represents the activation function; and Both represent mapping weights, and l is the smaller dimension.
[0048] In summary, the calculation of the entire Transformer layer is as follows. The output feature map of the i-th Transformer layer is expressed as
[0049]
[0050] Among them, Norm(·) represents the normalization layer; Attention * (·) and MLP / (·) denotes attention and MLP layers with frozen parameters; s denotes the scaling factor, is an intermediate variable.
[0051] In order to exploit image semantics at different depths, the feature maps of each third layer of the image encoder (specifically Z3, Z6, Z9, and Z 12 ) is input to the decoder.
[0052] S2. Input the 3rd layer data, 6th layer data and 9th layer data processed by the image encoder into the feature enhancement decoder.
[0053] Specifically, SAM designs a customized 3D mask decoder called feature enhancement decoder, such as Figure 1 As shown on the right side of part a in the figure. The feature enhancement decoder consists of 4 original feature enhancers and 1 prediction layer. Each original feature enhancer is divided into 2 different branches. The task of the first branch is to extract the original image features using the convolution block (Conv Block), including 3D convolution and instance normalization layers, as shown in Figure 1 As shown in part e; the second branch processes the feature map, which needs to be upsampled to match the size of the original image features. In order to integrate the semantics from each layer, the enhanced features of all enhancers are connected together in the prediction layer. Therefore, the upsampling rate must be moderate; too large a scale will significantly increase the parameters in the prediction layer. Here we use 2H, 2W, and 2D. Finally, the features of the two branches are connected and integrated through convolution blocks. In summary, the calculation formula of the feature enhancement decoder is:
[0054]
[0055] Y=Predict(Concat(E1,…,E j ,…,E4))##
[0056] Among them, Zi represents the feature map from the i-th layer of the image encoder, i = 3, 6, 9, 12; I represents the original image; E j represents the output of the j-th enhancer, j = 1, 2, 3, 4; Y represents the predicted segmentation; Upsample(·) represents upsampling; ConvBlock(·) represents the convolution block; Concat(v) represents the concatenated feature; Predict(·) represents the prediction layer; Represents an intermediate variable.
[0057] S3. The 12th layer data processed by the image encoder is input into the dual attention prompter, and the prior knowledge is integrated through the spatial self-attention and channel self-attention mechanisms, and then input into the feature enhancement decoder after processing.
[0058] Specifically, a unique dual attention mechanism is introduced as an automatic reminder encoder, called the dual attention reminder. Figure 1 As shown in part c, the dual attention prompter consists of spatial attention and channel attention. Spatial attention is the spatial self-attention at the pixel level. In order to solve the huge computational challenge, self-attention with linear complexity is adopted in Linfomer. Specifically, the linear layer converts the K i,SA and V i,SA The number of tags is reduced to a constant level. Therefore, the calculation formula of spatial attention is as follows:
[0059]
[0060] in, represents the weight parameter of spatial self-attention; Linear(·) represents a linear mapping that reduces the number of tokens to a constant n; Norm(·) represents a normalization layer; Softmax(·) represents a softmax function.
[0061] Channel attention aims to extract key information based on channels. This mechanism enables the model to focus on more prominent aspects of channel features. Specifically, the calculation formula of channel attention is as follows:
[0062]
[0063]
[0064] in, Represents the weight parameter of channel self-attention. Inspired by the efficient paired attention in UNETR++, the spatial attention and With channel attention, ie, Sharing can better integrate the two attention modules and reduce the computational complexity.
[0065] Finally, the features of the two attention modules are connected and the residual connection is used before the output, then:
[0066]
[0067] in, represents the feature map containing automatic prompt information; W down,SA , represents the linear mapping parameters. Note that only Z 12 Merge automatic prompt information.
[0068] S4, the feature enhancement decoder concatenates the data after feature enhancement on the 4 layers of data, and obtains the image segmentation result through the prediction layer of the feature enhancement decoder.
[0069] This paper proposes the DEAP-3DSAM architecture, an innovative 3D encoder-enhanced and automatically prompted SAM designed for 3D medical image segmentation. Existing SAM-based 3D medical imaging methods usually rely on pseudo-3D techniques, resulting in the loss of spatial features, which hinders their effectiveness. DEAP-3DSAM addresses this limitation by integrating original image features through a feature-enhanced decoder, thereby enriching the spatial feature representation. In addition, popular SAM-based methods rely on manual prompts, which poses a challenge for application in real-world environments and requires external expert knowledge to use. DEAP-3DSAM is different from this norm. By combining dual attention prompters, it uses spatial and channel self-attention mechanisms to autonomously identify relevant prompt information in feature maps. This achieves accurate and complete segmentation of lesion areas in 3D medical images.
Claims
1. A gastric cancer lesion region segmentation method for 3D medical images, characterized in that Included are: S1, decompose the 3D medical image into a series of block feature maps and input them into the image encoder for processing; S2, inputting the 3rd layer data, the 6th layer data and the 9th layer data processed by the image encoder into the feature enhancement decoder; S3, the 12th layer data obtained by the image encoder is input into the dual attention prompter to integrate the prior knowledge of the dual attention mechanism, and then input into the feature enhancement decoder after processing; S4, the feature enhancement decoder concatenates the data after feature enhancement on the 4 layers of data to obtain the image segmentation result.
2. The method for segmenting gastric cancer lesion region for 3D medical images according to claim 1, characterized in that: In S1, the shape is transformed into The shape of the 3D medical image decomposition map is a series of block feature maps of H×W×D×C, where They represent the initial height, width and depth of the image respectively, N represents the number of channels; H, W, D represent the height, width and depth of the feature map respectively, and C represents the channel dimension.
3. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 1, characterized in that The image encoder in S2 consists of 12 transformation network layers, which include 1 attention layer, 1 multi-layer perceptron layer and 2 normalization layers.
4. The method for segmenting gastric cancer lesion region for 3D medical images according to claim 3, characterized in that Set the attention mechanism parameters of the attention layer and the multilayer perceptron parameters of the multilayer perceptron layer, and fine-tune the multilayer perceptron parameters by scaling the parallel adapter, then: Adapter(X)=ReLU(XW down )IN up in, represents the feature map; ReLU(·) represents the activation function; and All represent mapping weights, and l represents the dimension; The calculation of the entire transformation network layer is as follows, and the output feature map of the i-th transformation network layer is expressed as: Among them, Norm(·) represents the normalization layer; Attention * (·) and MLP * (·) denotes the attention and multilayer perceptron layers with frozen parameters, respectively; s denotes the scaling factor, is an intermediate variable.
5. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 4, characterized in that The dual attention cues in S3 include spatial attention and channel attention.
6. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 5, characterized in that Spatial attention is a pixel-level spatial self-attention, and the number of tags belonging to Ki,SA and Vi,SA is reduced to a constant level through a linear layer. The calculation formula of spatial attention is: in, represents the weight parameter of spatial self-attention; Linear(·) represents a linear mapping that reduces the number of tokens to a constant n; Norm(·) represents a normalization layer; Softmax(·) represents a softmax function; Channel attention aims to extract key information based on channels. The calculation formula of channel attention is: in, The weight parameter representing the channel self-attention is the spatial attention and With channel attention, ie, shared; Connect the features of the two attention modules and use the residual connection before the output, then we have: in represents the feature map containing automatic prompt information; W down,SA , Represents the linear mapping parameters.
7. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 6, characterized in that The feature enhancement decoder in S4 consists of 4 original feature enhancers and 1 prediction layer.
8. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 7, characterized in that The original feature enhancer is divided into two different branches. The task of the first branch is to extract the original image features using convolution blocks, including 3D convolution and instance normalization layers; the second branch processes the feature map by upsampling to match the size of the original image features; the features of the two branches are connected and integrated through convolution blocks to enhance the features.
9. The method for segmenting gastric cancer lesion area for 3D medical images according to claim 8, characterized in that The enhanced features of the four original feature enhancers are connected in the prediction layer, and the calculation formula of the feature enhancement decoder is: Y=Predict(Concat(E1,…,E j ,…,E4))## Among them, Z i represents the feature map from the i-th layer of the image encoder, i = 3, 6, 9, 12; I represents the original image; E j represents the output of the j-th enhancer, j = 1, 2, 3, 4; Y represents the predicted segmentation; Upsample(·) represents upsampling; ConvBlock(·) represents convolutional block; Concat(·) represents concatenated features; Predict(·) represents the prediction layer; Represents an intermediate variable.