A CT image segmentation method based on Transformer and convolutional attention mechanism
By combining convolutional attention mechanisms and Transformer encoders, the problem of acquiring local and global features in medical image segmentation is solved, achieving more efficient CT image segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-01
- Publication Date
- 2026-03-10
AI Technical Summary
Existing medical image segmentation methods rely on too much prior medical knowledge, which is prone to human error. Furthermore, convolutional neural networks suffer from limited receptive fields and an inability to capture global feature relationships, resulting in poor segmentation performance.
By combining convolutional attention mechanism and Transformer encoder, local feature information is obtained through CNN encoder, global feature information is obtained by combining deformable attention Transformer layer, and the target result is restored through decoder. Skip connection is used to preserve low-level detailed features.
It improves the accuracy and efficiency of medical image segmentation, overcomes the limitations of a single architecture, and achieves better segmentation results.
Smart Images
Figure CN115908811B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image analysis technology, and in particular to a CT image segmentation method based on Transformer and convolutional attention mechanisms. Background Technology
[0002] Medical image segmentation is of paramount importance for clinical diagnosis and is a crucial step in the quantitative analysis of medical images. Computed Tomography (CT) and Magnetic Resonance (MR) imaging are currently the most widely used radiological imaging techniques in diagnosis, treatment, and clinical research. Therefore, CT and MR segmentation methods are the mainstream research direction in the field of medical image segmentation. Traditional segmentation methods suffer from unavoidable problems, such as excessive reliance on prior medical knowledge and human error in evaluation. To address these issues, this paper proposes a method for accurate medical image segmentation using deep learning technology.
[0003] With the widespread application of computers and the development of artificial intelligence technology, deep learning's Convolutional Neural Network (CNN) method has achieved remarkable results in the field of image analysis. Image segmentation has long been a challenge in medical image analysis. Compared with ordinary images, medical images have characteristics such as a large grayscale range and unclear boundaries. U-Net was proposed in this context as a general network structure that combines downsampling and upsampling, and combines low-level information with high-level information, thus improving the segmentation effect. Since the introduction of U-Net, CNN has become the main method for solving this type of problem. However, due to the locality of convolution operations, CNN has always suffered from problems such as limited receptive field and inability to obtain global feature relationships. Many people have devoted themselves to increasing the convolution kernel and using methods such as feature pyramids to obtain global feature information, thereby improving its global modeling ability. Although these methods have improved global relationship modeling to some extent, they are still inevitably limited by the receptive field of the CNN convolution kernel.
[0004] Currently, most methods for medical image segmentation still rely on CNN-based network architectures. However, many studies have also transferred the mature Transformer model from natural language processing to the image domain. These methods mainly fall into two categories: one uses the Transformer's self-attention mechanism to replace CNNs, employing a pure Transformer structure to extract image features. A typical example is the Vision Transformer (ViT), which outperforms the CNN-based ResNet in image recognition tasks. The other category combines the Transformer's self-attention mechanism with common CNN architectures. TransUNet is one such successful method combining CNNs and Transformers. It first uses a CNN to extract local features, then feeds the extracted final layer features into a Transformer for global feature relationship modeling, and finally uses deconvolution to upsample the feature map containing both local and global features to reconstruct the target result. These methods have achieved good results in medical image processing tasks, becoming a new research and application direction in medical image segmentation. Summary of the Invention
[0005] The technical problem this invention aims to solve is to provide a CT image segmentation method based on Transformer and convolutional attention mechanisms, which can automatically mark target regions in CT images, thereby facilitating their use in diagnosis, treatment, and clinical research. To solve the above problem, the technical solution is as follows:
[0006] This invention provides a CT image segmentation method based on Transformer and convolutional attention mechanisms, the method comprising the following steps:
[0007] S1, Data Preprocessing Module: Acquires CT image data, filters irrelevant regions by voxel truncation, standardizes the filtered CT image data, and then divides it into training, validation and test sets.
[0008] S2, CNN encoder module: It uses multi-layer convolutional operations to perform preliminary encoding of CT images, and uses a convolutional attention module to implement the attention mechanism, adding attention to the main regions of semantic segmentation and obtaining local feature information of CT images; where "convolutional attention module" is abbreviated as CBAM; CNN encoder module is the convolutional neural network encoder module.
[0009] S3, Transformer encoder module: connects the feature information of the last two layers of the CNN encoder and feeds it into the Transformer encoder for processing to obtain the global feature information of the CT image. The result of the global feature information of the CT image is added to the residual of the last two layers of feature information passed from the CNN encoder to enhance the information lost during the transmission of the last two layers of feature information of the CNN encoder.
[0010] S4, Decoder Module: This module restores the output information of the Transformer encoder to the size of the last two feature maps of the CNN encoder. The last feature map is deconvolved and added to the restored output of the previous layer. Then, another deconvolution operation restores the sum to the size of the first two feature maps of the CNN encoder. Skip connections are then added accordingly to finally reconstruct the labeled CT image. The input data for the decoder module is the output of the Transformer encoder. First, it is restored to feature maps based on the size of the output sequence. Then, a transposed convolution operation is used to progressively upsample the feature maps to the input resolution (D*H*W). Furthermore, skip connections are added between the encoder and decoder to retain more low-level detail features of the image, achieving better segmentation.
[0011] Furthermore, the CNN encoder module described in step S2 further includes:
[0012] S21, a Conv_IN_ReLU block is used to perform preliminary processing on the CT image, including a convolutional layer with a 7×7 kernel, a normalization layer and an activation layer, to obtain an intermediate feature map containing rich local information;
[0013] S22 uses a convolutional attention module to infer attention mappings along both channel and spatial dimensions, emphasizing important features and reducing the influence of unnecessary features, thereby focusing on the key parts to be segmented by the task and enhancing the performance of the CNN encoder.
[0014] S23, the intermediate feature map that incorporates convolutional attention is fed into ResConvBlock, and after processing by multiple residual convolutional blocks, a feature map with lower resolution is obtained.
[0015] S24. Following the methods in S22 and S23, after two convolutional attention modules and ResConvBlock processing, a higher-resolution high-level feature map is obtained, which contains global information of the CT image.
[0016] Furthermore, in step S22, the convolutional attention module divides the injection of attention into two processes: a channel attention module along the spatial axis and a spatial attention module along the channel axis. The intermediate feature map F∈R is input to the convolutional attention module. C*H*WThe convolutional attention module derives a 1-dimensional channel attention map M. c ∈R C*1*1 and 2D spatial attention mapping M s ∈R 1*H*W The specific process is as follows:
[0017] S221, the channel attention module along the spatial axis aggregates information from intermediate feature maps along the spatial axis through average pooling and max pooling operations, generating two distinct channel feature information. and Let these represent the average pooling feature and the max pooling feature across the space, respectively. Then, these two channel feature information are forwarded to a shared feedforward network. Finally, the shared feedforward network is applied to the result of each pooling feature, and element-wise summation is used to output the feature vector. The calculation method is as follows:
[0018]
[0019] In step S222, the channel attention map obtained in step S221 is multiplied element-wise with the input data. Through step S221, the channel attention is propagated downwards. The process can be summarized as follows:
[0020]
[0021] S223, the spatial attention module along the channel axis first aggregates information from the intermediate feature maps of the channel attention along the channel axis through average pooling and max pooling operations, generating two 2D maps: and Let these represent the average pooling feature and the max pooling feature across channels, respectively. Then, these features are concatenated and convolved through a standard 7×7 convolutional layer to generate a 2D spatial attention feature map. Finally, the attention map is normalized using the sigmoid function, calculated as follows:
[0022]
[0023] S224, the attention map obtained in step S223 is multiplied element-wise with F' obtained in step S222. This operation adds spatial attention. The result of this step includes both channel attention and spatial attention, which emphasizes the important features of the input intermediate feature map and reduces the influence of unnecessary features. The whole process can be summarized as follows:
[0024]
[0025] Furthermore, the Transformer encoder module described in step S3 further includes:
[0026] S31: Expand and concatenate the feature information of the last two layers of the CNN encoder, and add position encoding as input to the Transformer;
[0027] S32 passes the input information to the Deformable Transformer Layer to model global information. The Transformer encoder contains six Deformable Transformer Layers. Unlike the standard Transformer Layer, the Deformable Transformer Layer mechanism is introduced here. In the standard Transformer, self-attention focuses on all locations in the feature map. However, in image segmentation tasks, it's unnecessary to focus on all points, otherwise, the convergence speed would be slow and the computational complexity high. Deformable self-attention, on the other hand, selects a reference point in the feature map and only focuses on the eight offset positions around that point, instead of spreading attention globally. This can significantly improve model performance in image segmentation tasks.
[0028] S33, the result after passing through 6 deformable attention Transformer layers, is added to the residual information of the input information of the Transformer encoder to enhance the local feature information lost or missing during the processing and transmission in the deformable attention Transformer layers, and obtain a result with the same dimension as the input. This result contains the local feature information and global feature information of the input image.
[0029] The beneficial effects of the CT image segmentation method provided by this invention are:
[0030] I. The CT image segmentation method provided by this invention applies Transformer to 3D medical image segmentation tasks, which improves the accuracy compared to traditional CNN architecture methods.
[0031] Second, a deformable self-attention mechanism is used, which distributes attention to a local area around the sampling point rather than the whole in the image segmentation task, thus improving the efficiency of model training.
[0032] Third, both the CNN architecture and the Transformer architecture alone have some shortcomings in medical image segmentation tasks. This invention combines the two simple architectures to improve their respective shortcomings, and uses the residual connection between the CBAM and CNN encoders and the Transformer encoder to make the invention more efficient and achieve better segmentation. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a schematic diagram of the CT image segmentation method based on Transformer and convolutional attention mechanism of the present invention;
[0035] Figure 2 This is a schematic diagram of the process of obtaining channel attention using the Channel Attention Model of CBAM introduced in the CNN encoder module of the CT image segmentation method based on Transformer and convolutional attention mechanism of the present invention.
[0036] Figure 3 This is a schematic diagram of the process of obtaining spatial attention using the Spatial Attention Model of CBAM introduced in the CNN encoder module of the CT image segmentation method based on Transformer and convolutional attention mechanism of the present invention.
[0037] Figure 4 This is a data annotation example diagram of one embodiment of the CT image segmentation method based on Transformer and convolutional attention mechanism of the present invention. Detailed Implementation
[0038] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned objectives, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings.
[0039] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0040] Please refer to Figures 1 to 3 This embodiment presents a CT image segmentation method based on Transformer and convolutional attention mechanisms, which includes the following steps:
[0041] S1, Data Preprocessing Module: Acquires CT image data, filters irrelevant regions by voxel truncation, standardizes the filtered CT image data, and then divides it into training, validation and test sets.
[0042] S2, CNN encoder module: It uses multi-layer convolutional operations to perform preliminary encoding of CT images, and uses a convolutional attention module to implement the attention mechanism, adding attention to the main regions of semantic segmentation and obtaining local feature information of CT images.
[0043] S3, Transformer encoder module: connects the feature information of the last two layers of the CNN encoder and feeds it into the Transformer encoder for processing to obtain the global feature information of the CT image. The result of the global feature information of the CT image is added to the residual of the last two layers of feature information passed from the CNN encoder to enhance the information lost during the transmission of the last two layers of feature information of the CNN encoder.
[0044] S4, Decoder Module: This module restores the output information of the Transformer encoder to the size of the last two feature maps of the CNN encoder. The last feature map is deconvolved and added to the restored output of the previous layer. Then, another deconvolution operation restores the sum to the size of the first two feature maps of the CNN encoder. Skip connections are then added accordingly to finally reconstruct the labeled CT image. The input data for the decoder module is the output of the Transformer encoder. First, it is restored to feature maps based on the size of the output sequence. Then, a transposed convolution operation is used to progressively upsample the feature maps to the input resolution (D*H*W). Furthermore, skip connections are added between the encoder and decoder to retain more low-level detail features of the image, achieving better segmentation.
[0045] Preferably, the CNN encoder module described in step S2 further includes:
[0046] S21, a Conv_IN_ReLU block is used to perform preliminary processing on the CT image, including a convolutional layer with a 7×7 kernel, a normalization layer and an activation layer, to obtain an intermediate feature map containing rich local information;
[0047] S22 uses a convolutional attention module to infer attention mappings along both channel and spatial dimensions, emphasizing important features and reducing the influence of unnecessary features, thereby focusing on the key parts to be segmented by the task and enhancing the performance of the CNN encoder.
[0048] S23, the intermediate feature map that incorporates convolutional attention is fed into ResConvBlock, and after processing by multiple residual convolutional blocks, a feature map with lower resolution is obtained.
[0049] S24. Following the methods in S22 and S23, after two convolutional attention modules and ResConvBlock processing, a higher-resolution high-level feature map is obtained, which contains global information of the CT image.
[0050] Preferably, in step S22, the convolutional attention module divides the injection of attention into two processes: a channel attention module along the spatial axis and a spatial attention module along the channel axis. The intermediate feature map F∈R input to the convolutional attention module... C*H*W The convolutional attention module derives a 1-dimensional channel attention map M. c ∈R C*1*1 and 2D spatial attention mapping M s ∈R 1*H*W The specific process is as follows:
[0051] S221, the channel attention module along the spatial axis aggregates information from intermediate feature maps along the spatial axis through average pooling and max pooling operations, generating two distinct channel feature information. and Let these represent the average pooling feature and the max pooling feature across the space, respectively. Then, these two channel feature information are forwarded to a shared feedforward network. Finally, the shared feedforward network is applied to the result of each pooling feature, and element-wise summation is used to output the feature vector. The calculation method is as follows:
[0052]
[0053] In step S222, the channel attention map obtained in step S221 is multiplied element-wise with the input data. Through step S221, the channel attention is propagated downwards. The process can be summarized as follows:
[0054]
[0055] S223, the spatial attention module along the channel axis first aggregates information from the intermediate feature maps of the channel attention along the channel axis through average pooling and max pooling operations, generating two 2D maps: and Let these represent the average pooling feature and the max pooling feature across channels, respectively. Then, these features are concatenated and convolved through a standard 7×7 convolutional layer to generate a 2D spatial attention feature map. Finally, the attention map is normalized using the sigmoid function, calculated as follows:
[0056]
[0057] S224, the attention map obtained in step S223 is multiplied element-wise with F' obtained in step S222. This operation adds spatial attention. The result of this step includes both channel attention and spatial attention, which emphasizes the important features of the input intermediate feature map and reduces the influence of unnecessary features. The whole process can be summarized as follows:
[0058]
[0059] Preferably, the Transformer encoder module described in step S3 further includes:
[0060] S31: Expand and concatenate the feature information of the last two layers of the CNN encoder, and add position encoding as input to the Transformer;
[0061] S32 passes the input information to the Deformable Transformer Layer to model global information. The Transformer encoder contains six Deformable Transformer Layers. Unlike standard Transformer Layers, Deformable Transformer Layers introduce a deformable self-attention mechanism. In standard Transformers, self-attention focuses on all locations in the feature map. However, in image segmentation tasks, it's unnecessary to focus on all points, otherwise, convergence speed would be slow and computational complexity high. Deformable self-attention selects a reference point in the feature map and only focuses on the eight offset locations around that point, instead of spreading attention globally. This can significantly improve model performance in image segmentation tasks.
[0062] S33, the result after passing through 6 deformable attention Transformer layers, is added to the residual information of the input information of the Transformer encoder to enhance the local feature information lost or missing during the processing and transmission in the deformable attention Transformer layers, and obtain a result with the same dimension as the input. This result contains the local feature information and global feature information of the input image.
[0063] like Figure 4As shown, in step S1, the BTCV (Beyond The Cranial Vault) abdominal organ segmentation dataset was acquired. This dataset contains 30 abdominal CT scans, derived from a colorectal cancer chemotherapy trial and a recurrent abdominal hernia study. The image volume ranges from 512×512×85 to 512×512×198, with a field of view of approximately 280×280×280 mm. 3 -500×500×650mm 3 The planar resolution is 0.54×0.54mm. 2 -0.98×0.98mm 2 The slice thickness ranged from 2.5 mm to 5.0 mm. The dataset included images of 13 organs: spleen (Sp), right kidney (R_Ki), left kidney (L_Ki), gallbladder (Gb), esophagus (Es), liver (Li), stomach (St), aorta (Ao), inferior vena cava (IVC), portal and splenic veins (PSV), pancreas (Pa), right adrenal gland (R_AG), and left adrenal gland (L_AG). The data was labeled and validated by radiologists. An example of the data labeling is shown below. Figure 4 As shown, after normalizing the BTCV dataset, it was randomly divided into 15 training sets, 6 validation sets, and 9 test sets. The validation set was used to select the hyperparameters of the model.
[0064] The loss function of this method is the sum of Dice loss and cross-entropy loss, as shown in the formula below, where X represents the real data, Y represents the predicted result, |X| represents the number of elements in X, |Y| represents the number of elements in Y, and |X∩Y| represents the intersection between X and Y.
[0065] loss=Dice_loss+CrossEntropy_loss
[0066]
[0067] The CT image segmentation method of the present invention was successfully applied to BTCV data and segmentation was achieved. Table 1 shows the Dice results of the CT image segmentation method and other methods.
[0068] Table 1. Experimental Comparison Results
[0069]
[0070] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0071] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations made to these embodiments without departing from the principles and spirit of the present invention still fall within the protection scope of the present invention.
Claims
1. A CT image segmentation method based on a Transformer and a convolution attention mechanism, characterized in that, The method comprises the following steps: S1, a data preprocessing module: acquiring CT image data, filtering irrelevant regions according to a voxel value truncation method, performing standardization processing on the filtered CT image data, and then dividing a training set, a validation set and a test set; S2, a CNN encoder module: using a multi-layer convolution operation to preliminarily encode the CT image, using a convolution attention module to realize an attention mechanism, adding attention to a main region of semantic segmentation, and acquiring local feature information of the CT image; S3, a Transformer encoder module: connecting the last two layers of feature information of the CNN encoder, transmitting the last two layers of feature information to the Transformer encoder for processing, acquiring global feature information of the CT image, adding the global feature information of the CT image to a residual error of the last two layers of feature information transmitted from the CNN encoder, and enhancing information lost in a transmission process of the last two layers of feature information of the CNN encoder; S4, a decoder module: restoring output information of the Transformer encoder to a size of the last two layers of feature maps of the CNN encoder, adding the last layer of feature maps after a deconvolution operation to a restoration result of an output of a previous layer, restoring the addition result to a size of the first two layers of feature maps of the CNN encoder by using a deconvolution operation, and corresponding jump connection, and finally restoring a CT image with labels; The CNN encoder module in step S2 further comprises: S21, using a Conv_IN_ReLU block to preliminarily process the CT image, including a convolution layer with a 7*7 convolution kernel, a normalization layer and an activation layer, to obtain an intermediate feature map containing rich local information; S22, using a convolution attention module to infer an attention map along two dimensions of channels and space, emphasize important features, reduce the influence of unnecessary features, thereby focus on key parts to be segmented by the task, and enhance the performance of the CNN encoder; S23, transmitting the intermediate feature map fused with the convolution attention to a ResConvBlock, and obtaining a feature map with a lower resolution through processing of multiple residual convolution blocks; S24, according to the method in S22 and S23, obtaining a high-level feature map with a lower resolution through twice convolution attention module and ResConvBlock processing, wherein the high-level feature map contains global information of the CT image; In step S22, the convolution attention module divides the injection of attention into two processes, a channel attention module along the spatial axis and a spatial attention module along the channel axis, and the intermediate feature map input by the convolution attention module The convolution attention module derives a 1-dimensional channel attention map and a 2-dimensional spatial attention map The specific process is as follows: S221, the channel attention module of the spatial axis aggregates the information of the intermediate feature map along the spatial axis through average pooling and max pooling operations to generate two different channel feature information and respectively represent the average pooling feature and the max pooling feature across the space; then the two channel feature information are forwarded to a shared forward network respectively; finally, the results after applying the shared forward network to each of the pooled features are summed element by element and output a feature vector, and the calculation method is: ； S222, multiplying the channel attention map obtained in step S221 with input data element by element, and through step S221, the channel attention is downward propagated, and the process is summarized as: ； S223, the spatial attention module of the channel axis first aggregates the information contained in the channel attention intermediate feature map along the channel axis through average pooling and max pooling operations to generate two 2-dimensional maps: and , representing the average-pooled features and the max-pooled features across channels, respectively; then the information is concatenated and convolved through a convolution layer with a standard 7x7 convolution kernel to produce a 2-dimensional spatial attention feature map; finally, the final attention map is standardized through a sigmoid function, and the calculation method is: ； S224, map the attention obtained in step S223 with the attention obtained in step S222 The element-wise multiplication is performed to add the spatial attention, and the result obtained in this step contains the channel attention and the spatial attention, which emphasizes the important features of the input intermediate feature map and reduces the influence of unnecessary features. The whole process can be summarized as follows: 。 2. The CT image segmentation method of claim 1, wherein, The Transformer encoder module in step S3 further comprises: S31, expanding and splicing the last two layers of feature information of the CNN encoder, and adding position encoding as input of the Transformer; S32, transmitting the input information to a deformable attention Transformer layer to model global information, and the Transformer encoder comprises 6 deformable attention Transformer layers. S33, the result after passing through the 6-layer deformable attention Transformer layer is added with the incoming information residual of the Transformer encoder, the local feature information lost or missing in the processing and transmission process in the deformable attention Transformer layer is enhanced, and a result with the same dimension as the input is obtained, which contains the local feature information and the global feature information of the input image.
Citation Information
Patent Citations
Pulmonary nodule image detection method and system based on CT image
CN113888466A