A lung CT image segmentation method based on hybrid Swin Transformer U-Net
By hybridizing the Swin Transformer U-Net model and the adaptive attention module, the difficult problem of segmenting lung infection lesions on CT images was solved, and high-precision segmentation of lung infection areas was achieved, adapting to irregular and low-contrast characteristics.
Patent Information
- Application Number
- CN202211412454.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-11
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2042-11-11
AI Technical Summary
Existing technologies have difficulty in accurately segmenting lung infection lesions on CT images, especially due to the problems of missegmentation and lack of labeled datasets caused by irregular infection boundaries, different sizes and shapes, and low contrast characteristics.
A hybrid Swin Transformer U-Net model is used, combined with an adaptive attention module and a pre-activation residual module, to achieve accurate segmentation of lung infection areas through data enhancement and feature fusion.
The segmentation accuracy of lung infection areas is improved, missegmentation is reduced, the model adapts to infection areas of different scales and shapes, and the performance of the model on small-scale datasets is improved.
Smart Images
Figure CN117274147B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image segmentation and relates to a lung CT image segmentation method based on a hybrid Swin Transformer U-Net. Background Art
[0002] Medical images play a vital role in helping healthcare providers reach patients for diagnosis and treatment. Studying medical images primarily relies on visual interpretation by radiologists. However, this often takes a significant amount of time, depends on the radiologist's experience, and is highly subjective. To overcome these limitations, the use of computer-assisted systems has become essential. Computerized medical image segmentation plays a vital role in medical imaging applications. It has a wide range of applications in diverse fields, including diagnosis, pathology localization, anatomical structure research, treatment planning, and computer-integrated surgery. However, due to the variability and complexity of human anatomy, medical image segmentation remains a challenging problem.
[0003] The current standard for diagnosing COVID-19 is a real-time reverse transcription polymerase chain reaction (RT-PCR) swab test. However, RT-PCR diagnostic results take several hours to process, and the test has a high false negative rate, often requiring repeat testing. Compared to RT-PCR, chest computed tomography (CT) imaging can effectively screen for COVID-19 with high sensitivity and is easy to use in clinical settings.
[0004] The application of deep learning technology in medical diagnosis can improve disease detection rates and efficiency, and has achieved significant success in the field of medical image recognition. To diagnose lung cancer, lung tumors, and lung nodules, many researchers have studied deep learning-based lung CT image recognition methods. CT image recognition has proven to be very useful for diagnosing lung diseases. Accurately segmenting lung infection areas from CT images is crucial for the quantification and diagnosis of lung diseases, including COVID-19.
[0005] However, accurately segmenting lung infection lesions on CT images remains a challenging task due to the following facts: 1. Infections on CT images have irregular boundaries, varying sizes and shapes, and exhibit a fuzzy appearance and low contrast. This can easily lead to the omission of small ground-glass lesions or over-segmentation of infections on CT images. 2. Labeled datasets are lacking, and large-scale infection annotations provided by clinicians are not readily available. Summary of the Invention
[0006] The purpose of the present invention is to provide a lung CT image segmentation method based on a hybrid Swin Transformer U-Net, which is used to accurately segment the lung infection area from the CT image.
[0007] The present invention specifically comprises the following steps:
[0008] Step 1: Data preprocessing and data enhancement:
[0009] A large number of publicly available CT images of lung infections were collected, and data augmentation was performed to expand the number of samples and normalize these images; these images were used as training sets for the model; data augmentation specifically involved randomly cropping, inverting, rotating, scaling, and offsetting the images.
[0010] Step 2: Build the segmentation model HySwinUNet:
[0011] The segmentation model HySwinUNet is constructed based on the encoder-decoder structure of U-Net. The segmentation model HySwinUNet includes an encoder, an adaptive attention module, a decoder, and skip connections. The basic unit of HySwinUNet is the Swin Transformer Block, which serves as the backbone network of U-Net.
[0012] In the encoder, the input image is divided into 4×4 patches through patch partitioning. After linear embedding, the dimension of the vector will become the preset value; the dimension is C and the resolution is The input value is fed into two consecutive Swin converters to perform representation learning, while the feature dimension and resolution remain unchanged during the process. The Swin converter module is responsible for feature representation learning. After learning, patch merging is performed to downsample and increase the dimension, reducing the spatial size by 1 / 2 and increasing the feature dimension to twice the original, thus forming a hierarchical design. The above process is repeated three times in the encoder, and the propagation process of each layer is pre-activated through the pre-activated residual block (PRB).
[0013] During the encoding process, the adaptive attention module (ADM) is used to locate the feature information of the region of interest (RoI), suppress the feature information of the irrelevant regions, effectively extract the feature information, and more accurately segment the lesion area; thereby increasing the weight ratio of the target area features and improving the network segmentation accuracy;
[0014] A symmetric decoder is constructed based on the Swin converter block; feature maps of adjacent dimensions are reshaped into higher-resolution feature maps through upsampling, and the feature dimension is correspondingly reduced to half of the original dimension; the extracted contextual features are fused with the multi-scale features of the encoder through skip connections to compensate for the spatial information loss caused by downsampling and restore valuable spatial information.
[0015] The pre-activated residual module is used at the entry point of the encoding phase and the exit point of the decoding phase. The pre-activated residual module initializes the Transformer as a convolutional network and uses convolutional layers to extract local intensity features to avoid large-scale preprocessing of the Transformer, making the training of the Swin converter easier. Since mis-segmented areas are usually located on the boundaries of the region of interest (RoI), high-resolution contextual information plays a vital role in segmentation. The module performs an element-wise addition with the original input after two consecutive batch normalizations (BN), activation functions (ReLU), and convolution operations (Conv). The convolution operation is performed after passing through the ReLU layer, and the above operations do not change the dimension and resolution of the feature map. The pre-activated residual module makes the information smoother during the forward and backward propagation of the network.
[0016] The Swin converter module is constructed based on a moving window and includes two consecutive Swin converters. Each Swin converter includes a multi-head self-attention module (MSA) and a multi-layer perceptron (MLP). In addition, a layer normalization (LN) layer is used before each MSA module and MLP module. Based on the multi-head self-attention module, the Swin converter proposes a window-based multi-head self-attention module (W-MSA) and a moving window-based multi-head self-attention module (SW-MSA). The calculation formula is as follows:
[0017] in, and z l Represent the outputs of the l-th layer W-MSA and MLP respectively; and z l+1 They represent the outputs of the l+1th layer SW-MSA and MLP respectively; the self-attention calculations of W-MSA and SM-MSA are: in, Represents the query, key, and value matrices; M 2 and d represent the number of patches in the window and the dimension of the query or key respectively; the value of B is taken from the bias matrix
[0018] The input channels of the adaptive attention module (ADM) are merged into dual attention inputs through 3×3 convolutions with dilation rates of 1 and 3, respectively, to identify different global information through two different attention mechanisms. A C×H×W matrix is obtained by obtaining channel data based on global average pooling and pixel-wise correlation to obtain a C×1×1 matrix. After concatenation, the matrix is normalized using the sigmoid function. In addition, a fully connected layer (FC) is used to generate more nonlinear features. Finally, a Softmax operation is applied to the channels, and cross-channel attention can adaptively select receptive fields of different sizes.
[0019] Step 3: Set the training strategy and loss function to train the model;
[0020] The preprocessed dataset is divided into training, test, and validation sets. Random initialization and the Adam optimization algorithm are used. Batch size, epoch, and appropriate learning rate are set, while regularization is used to prevent overfitting. The segmentation model HySwinUNet uses the backpropagation algorithm to update the weights and biases in the model. The loss function is used to update the parameters during the training iteration.
[0021] Step 4: Verify the trained network model: Input the segmented validation set into the trained segmentation model HySwinUNet. The output will segment the lesion portion of the lung CT image to obtain a segmented image. The model is evaluated by comparing the CT image segmented by the expert with the image segmented by the trained network model of the present invention.
[0022] After verification, any lung CT image is input into the segmentation model HySwinUNet, and the lung CT image with the lesion segmented is output.
[0023] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0024] This paper effectively combines Swin Transformer and U-Net to enhance the functionality and flexibility of the traditional encoder-decoder architecture, and applies it to the field of medical image segmentation to achieve automatic segmentation of the lung infection part of lung CT, and can accurately segment the lung infection area from CT images.
[0025] Because the Transformer has no inductive bias for images, it performs poorly on small datasets. Even on pre-trained ImageNet, the Transformer may not perform as well as the residual network. The HySwinUNet model combines convolution and Transformer, adding a pre-activated residual block (PRB) module. It utilizes the inductive bias of convolutional images to avoid large-scale preprocessing. Information can be directly transferred from one module to any other module in both forward and backward propagation of the network, reducing the training burden and enabling better network training.
[0026] The Adaptive Dual-attention Module is used to obtain multi-scale global features by integrating two attention mechanisms, thereby increasing the weight ratio of target area features, aiming to more accurately segment irregular, low-contrast lesion areas in COVID-19 CT images. In CT images, infected areas may have discontinuous boundaries and irregular shapes, resulting in blurred appearance and low-contrast characteristics. Channel and pixel information is important for obtaining representative features of the region of interest. Therefore, the Adaptive Dual-attention Module is used to extract feature information and increase the weight ratio of target area features. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a structural diagram of the HySwinUNet network of the present invention;
[0028] Figure 2 This is a structural diagram of the pre-activation of the remaining modules of the present invention;
[0029] Figure 3 This is a diagram illustrating the implementation of the Swin converter module;
[0030] Figure 4 Implementation diagram for the adaptive attention module. DETAILED DESCRIPTION
[0031] The present invention will be further described below with reference to the accompanying drawings:
[0032] A lung CT image segmentation method based on a hybrid Swin Transformer U-Net specifically includes the following steps:
[0033] Step 1: Data preprocessing and data enhancement;
[0034] In this example, CT images of COVID-19 collected by the Italian Society of Medical and Interventional Radiology (SIRM) and from the MosMedData dataset were used to train the model. The images were randomly cropped, flipped, rotated, scaled, and offset to expand the dataset, increase the number of training samples, and improve the robustness of the model. Finally, these images were normalized.
[0035] Step 2: Build the segmentation model HySwinUNet:
[0036] like Figure 1 As shown in the figure, the segmentation model HySwinUNet is based on the encoder-decoder structure of U-Net, including an encoder, an adaptive attention module, a decoder, and skip connections. The basic unit of HySwinUNet is the Swin Transformer Block, which serves as the backbone network of U-Net.
[0037] For the encoder, the segmentation process of the model will first divide the input image into 4×4 small blocks (patch) through block segmentation (PatchPartition). After linear embedding (Linear Embedding), the dimension of the vector will become a pre-set value; the dimension is C and the resolution is The input value is fed into two consecutive Swin transformers to perform representation learning, during which the feature dimension and resolution remain unchanged. The Swin transformer module is responsible for feature representation learning. Patch merging is used to downsample and increase the dimension, reducing the spatial size by 1 / 2 and increasing the feature dimension to twice the original, thus forming a hierarchical design. During the propagation of each layer, it will pass through the pre-activation residual block in advance. This procedure will be repeated three times in the encoder.
[0038] During the encoding process, in order to effectively extract feature information and more accurately segment the lesion area, an adaptive dual-attention module is used to locate the feature information of the region of interest (RoI) and suppress feature information of irrelevant regions, thereby increasing the weight ratio of the target region features and improving the network segmentation accuracy.
[0039] For the decoder, a symmetric decoder is constructed based on the Swin transformer block. Feature maps of adjacent dimensions are reshaped into higher-resolution feature maps through upsampling, and the feature dimension is correspondingly reduced to half of the original dimension. The extracted contextual features are fused with the multi-scale features of the encoder through skip connections to compensate for the spatial information loss caused by downsampling and restore valuable spatial information.
[0040] like Figure 2 As shown in the figure, since the Transformer has no inductive bias for images, it performs poorly on small-scale datasets. Even for pre-trained ImageNet, the Transformer may not perform as well as the residual network. The HySwinUNet model combines the power of convolution and Transformer. The pre-activation residual block initializes the Transformer as a convolutional network and uses the convolution layer to extract local intensity features to avoid large-scale preprocessing of the Transformer, making the training of the Swin converter easier. Since the mis-segmented area is usually located on the boundary of the region of interest (RoI), high-resolution contextual information plays a vital role in segmentation. The module performs an element-wise addition with the original input after two consecutive batch normalizations (BN), activation functions (ReLU) and convolution operations (Conv). The Conv convolution operation is performed after passing through the ReLU layer, which does not change the dimension and resolution of the feature map. The pre-activation residual block makes the information smoother during the forward and backward propagation of the network. The pre-activation residual block is used at the entry of the encoding stage and the exit of the decoding stage.
[0041] like Figure 3 As shown in the figure, unlike the traditional multi-head self-attention module (MSA), the Swin converter module is constructed based on a moving window. It consists of two consecutive Swin converters, each of which consists of a multi-head self-attention module (Multi-head Self-attention Module) and a multi-layer perceptron (MLP). In addition, a layer normalization (LN) layer is used before each MSA module and MLP module. Based on the multi-head self-attention module, the Swin converter proposes a window-based multi-head self-attention module (W-MSA) and a moving window-based multi-head self-attention module (SW-MSA). The calculation formula is as follows: in, and z l Represent the outputs of the l-th layer W-MSA and MLP respectively; and z l+1They represent the outputs of the l+1th layer SW-MSA and MLP respectively; the self-attention of W-MSA and SM-MSA is: in, Represents the query, key, and value matrices; M 2 and d represent the number of patches in the window and the dimension of the query or key respectively; the value of B is taken from the bias matrix
[0042] Infected areas on lung CT images may have discontinuous boundaries and irregular shapes, resulting in a blurred appearance and low contrast. Channel and pixel information are crucial for obtaining representative features of the region of interest. Therefore, an adaptive dual-attention module is used to extract more comprehensive and discriminative feature information, increasing the weight of target region features to identify lesion boundaries. This module uses a global average pooling algorithm to capture boundary discontinuities in lung CT lesions and a pixel-by-pixel correlation algorithm to address shape irregularities.
[0043] The structure of the adaptive attention module is as follows Figure 4 As shown in the figure, the input channels will pass through 3×3 convolutions with dilation rates of 1 and 3 respectively, which are combined into the input of dual attention, and different global information is found through two different attention mechanisms; the C×H×W matrix obtains channel data based on global average pooling and pixel-wise correlation to obtain a C×1×1 matrix, which is merged through concatenation and normalized using the sigmoid function; in addition, a fully connected layer (FC) is used to generate more nonlinear features; finally, a softmax operation is applied to the channel to output a feature map of the original size; cross-channel attention can adaptively select receptive fields of different sizes;
[0044] Step 3: Set the training strategy and loss function to train the model;
[0045] The preprocessed dataset was divided into training, test, and validation sets in a ratio of 5:3:2. Random initialization and the Adam optimization algorithm were used. Batch size, epoch, and appropriate learning rate were set, and regularization was used to prevent overfitting. The backpropagation algorithm was used in the HySwinUNet network model to update the weights and biases in the network. The loss function was used to update the parameters during the training iterations.
[0046] The HySwinUNet network model is trained according to the set training strategy. During the training phase, HySwinUNet is trained in an end-to-end manner using the objective function. The loss function is used to update the parameters in the iterative process. In terms of the choice of loss function, a combination of Dice Loss and Binary Cross Entropy Loss is used to train all networks. Therefore, the loss function is Loss = αL Dice +βL BCE ;
[0047] Where y is the true probability of sample i, is the predicted probability of sample i; L Dice and L BCE Denote dice loss and binary cross entropy loss respectively; Loss represents the final loss function, dice loss and binary cross entropy loss are combined into one term and given to L Dice More weights can better handle the problem of class imbalance; the value of α is 0.9 and the value of β is 0.1.
[0048] Step 4: Verify the trained network model;
[0049] The segmented validation set is fed into the trained HySwinUNet network model. The output segmentates the lesion portion of the lung CT image to obtain a segmented image. The model is evaluated by comparing the CT image segmented by the expert with the image segmented by the trained network model of the present invention.
[0050] Four widely adopted evaluation criteria are used to measure the performance of the HySwinUNet model; the evaluation metrics are as follows:
[0051] Dice similarity coefficient: DSC is used to measure the similarity between the predicted lung infection and the facts, where V Seg Represents the area segmented by the model algorithm, V GT Represents the true segmentation area; TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively;
[0052] Sensitivity: SEN represents the percentage of correctly segmented lung infections;
[0053] Specificity: SPE represents the percentage of correctly segmented non-infected areas;
[0054] Positive predictive value (Precision): PRE indicates the accuracy of lung infection area segmentation.
[0055] After verification, any lung CT image is input into the segmentation model HySwinUNet, and the lung CT image with the lesion segmented is output.
[0056] The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention, but the above is only a preferred embodiment of the present invention, which can be fully applied to various fields suitable for the present invention. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described here.
Claims
1. A lung CT image segmentation method based on a hybrid Swin Transformer U-Net, characterized by: The specific steps include: Step 1: Data preprocessing and data enhancement: Collect a large number of publicly available lung infection CT images, perform data augmentation, expand the sample size, and normalize these images to serve as the training set for the model. Step 2: Build the segmentation model HySwinUNet: The segmentation model HySwinUNet is constructed based on the encoder-decoder structure of U-Net. The segmentation model HySwinUNet includes an encoder, an adaptive attention module, a decoder, and skip connections; In the encoder, the input image is divided into 4×4 blocks by block segmentation. After linear embedding, the dimension of the vector will become the preset value; the dimension is C and the resolution is The input value is fed into two consecutive Swin converters to perform representation learning, and the feature dimension and resolution remain unchanged during the process; the Swin converter module is responsible for feature representation learning. After learning, block merging, downsampling and dimension increase are performed to reduce the spatial size by 1 / 2 and increase the feature dimension to twice the original, thus forming a hierarchical design; the above process is repeated three times in the encoder, and in the process of propagation at each layer, it passes through the pre-activation residual module in advance; During the encoding process, an adaptive attention module is used to locate the feature information of the region of interest, suppress the feature information of the irrelevant region, effectively extract the feature information, and more accurately segment the lesion area; thereby increasing the weight ratio of the target area features and improving the network segmentation accuracy; A symmetric decoder is constructed based on the Swin transformer block. Feature maps of adjacent dimensions are reshaped into higher-resolution feature maps through upsampling, and the feature dimension is correspondingly reduced to half of the original dimension. The extracted contextual features are fused with the multi-scale features of the encoder through skip connections to compensate for the spatial information loss caused by downsampling and restore valuable spatial information. Step 3: Set the training strategy and loss function; The preprocessed dataset is divided into training, test, and validation sets. Random initialization and the Adam optimization algorithm are used. Batch size, epoch, and appropriate learning rate are set, while regularization is used to prevent overfitting. The segmentation model HySwinUNet uses the backpropagation algorithm to update the weights and biases in the model. The loss function is used to update the parameters during the training iteration; Step 4: Verify the trained network model: Input the segmented validation set into the trained segmentation model HySwinUNet. The output will segment the lesion portion of the lung CT image to obtain a segmented image. The model is evaluated by comparing the CT image segmented by the expert with the image segmented by the trained network model of the present invention. After verification, any lung CT image is input into the segmentation model HySwinUNet, and the lung CT image with the lesion segmented is output.
2. The lung CT image segmentation method based on hybrid Swin Transformer U-Net according to claim 1, characterized in that: The data enhancement specifically includes: randomly cropping, reversing, rotating, scaling and offsetting the image.
3. The lung CT image segmentation method based on hybrid Swin Transformer U-Net according to claim 1, characterized in that: The pre-activated residual module is used at the entrance of the encoding stage and the exit of the decoding stage. The pre-activated residual module initializes the Transformer as a convolutional network and uses the convolution layer to extract local intensity features to avoid large-scale preprocessing of the Transformer, making the training of the Swin converter easier; since the mis-segmented area is usually located on the boundary of the region of interest, high-resolution contextual information plays a vital role in segmentation; the module performs an element-wise addition with the original input after two consecutive batch normalizations (BN), activation function ReLU and convolution operation Conv; passing through the ReLU layer before performing the Conv convolution operation will not change the dimension and resolution of the feature map, making the information smoother during the forward and backward propagation of the network.
4. The lung CT image segmentation method based on hybrid Swin Transformer U-Net according to claim 1, characterized in that: The Swin converter module is constructed based on a moving window and includes two consecutive Swin converters. Each Swin converter includes a multi-head self-attention module MSA and a multi-layer perceptron MLP. In addition, a layer normalization layer LN is used before each MSA module and MLP module. Based on the multi-head self-attention module, the Swin converter proposes a window-based multi-head self-attention module W-MSA and a moving window-based multi-head self-attention module SW-MSA. The calculation formula is as follows: in, and z l Represent the outputs of the l-th layer W-MSA and MLP respectively; and z l+1 They represent the outputs of the l+1th layer SW-MSA and MLP respectively; the self-attention calculations of W-MSA and SW-MSA are: in, represents the query, key, and value matrix; M 2 and d represent the number of patches in the window and the dimension of the query or key respectively; the value of B is taken from the bias matrix 5. The lung CT image segmentation method based on hybrid Swin Transformer U-Net according to claim 1, characterized in that: The input channels of the adaptive attention module are combined into dual attention inputs through 3×3 convolutions with dilation rates of 1 and 3, respectively. Different global information is extracted through two different attention mechanisms. The C×H×W matrix is converted into a C×1×1 matrix after obtaining channel data based on global average pooling and pixel-by-pixel correlation. The matrix is concatenated and then normalized using a sigmoid function. In addition, a fully connected layer is used to generate more nonlinear features. Finally, the Softmax operation is applied to the channel, and the cross-channel attention can adaptively select receptive fields of different sizes.