Lung CT image segmentation method based on Transform and U-Net

Abstract

The invention discloses a lung CT image segmentation method based on Transform and U-Net, and the method comprises the steps: firstly obtaining a pulmonary nodule CT scanning image, carrying out the preprocessing, then constructing a segmentation model MCSTU-Net, enabling the model to extract abundant features through the fusion of a convolutional neural network and a Swin Transform module, and enabling the segmentation model to obtain a segmentation model MCSTU-Net. Enhancing and fusing feature information from different levels through a multiple attention fusion module; dividing the data set into a training set, a test set and a verification set, training and verifying the segmentation model, and finally performing focus segmentation on the pulmonary nodule CT scanning image by using the trained and verified model. According to the model and the method constructed by the invention, lesion tissues in the lung CT image can be segmented more accurately, the effectiveness of each module is verified through an ablation experiment, and doctors are assisted to diagnose and treat lung diseases.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation and relates to a lung CT image segmentation method based on Transformer and U-Net. Background Art

[0002] When patients with lung diseases go to the hospital for treatment, doctors will first use medical testing equipment to examine the patient's lungs. In medicine, computed tomography (CT) is considered a key tool for diagnosing and evaluating the condition of patients with lung diseases. CT can quickly generate high-quality scan images, and interstitial and solid lesions in the lungs can be better displayed. The medical images generated by the testing instrument are used to segment the region of interest, make a preliminary diagnosis, evaluate the progression of the disease, and determine whether the patient should be admitted to the intensive care unit or discharged.

[0003] The process of annotating a medical image into several regions of interest is medical image segmentation, which aims to segment organs or tissues in a medical image into regions suitable for specific medical applications. Medical image segmentation has many different application scenarios, such as anatomical structure research, disease diagnosis, tissue volume quantification, pathological localization, treatment planning, and clinical surgery. Purely manual segmentation of medical images generally requires a lot of energy and time from experts. Manual annotation of a CT scan image takes a doctor about four hours and is easily affected by subjective experience. The high risk of occupational burnout and occupational death among medical workers requires automation of the annotation process, which will improve productivity and minimize errors. Research on automatic processing and analysis methods of medical images has always been a hot topic for researchers. Researchers have proposed various automatic segmentation systems, including traditional medical image segmentation methods and artificial intelligence-based methods. In recent years, medical image processing and analysis methods based on deep learning have made significant breakthroughs, among which the performance of convolutional neural networks (CNN) in various medical image processing and analysis tasks has exceeded that of traditional algorithms. With the development of deep learning technology, the use of deep learning-based methods has become the first choice in the field of image segmentation and has achieved the best results. Among the existing medical image segmentation methods, the fully convolutional neural network (FCNN) with a U-shaped structure is dominant, such as U-Net, UNet++, and Res-UNet. Due to its high efficiency and good generalization, many FCNN-based methods have been used for lung image segmentation.

[0004] In recent years, in order to further improve the accuracy and efficiency of image segmentation, some studies have begun to try to combine Transformer and U-Net to take advantage of both. However, while these network architectures improve segmentation performance, they still have some defects, such as: ① TransUNet, the computational complexity of its model is high, and for high-resolution images, especially medical images, the computing resource requirements are very large, which may lead to slow training and inference, or even unusable when computing resources are limited. ② Swin-Unet, developed based on the Transformer of the ShiftedWindow self-attention mechanism, lacks the intuitive interpretability of traditional convolutional networks. In tasks such as medical image segmentation, especially in clinical settings, the interpretability of the model is crucial. Doctors need to understand the decision-making process of the model in order to trust the segmentation results of the model. Therefore, the complexity and opacity of its decision-making process limit its application in some high-risk fields (such as medicine). ③ DS-TransUNet, its dual Swin Transformer design increases the computational complexity and memory requirements of the model. In high-resolution images (such as 3D medical images), the computation and memory consumption of the model will increase significantly. Although the local window attention mechanism is used to reduce the computational complexity, there may still be bottlenecks when processing large-scale images. Usually a large amount of data is required for effective training, especially in medical image segmentation tasks, where labeled data is often scarce and expensive.

[0005] Based on the above, it is still a challenging task to accurately segment lung disease infected tissue (such as lung nodules) in medical images: 1. Lack of annotated datasets: It is difficult to generate more lung image datasets without long-term accumulation; 2. Category imbalance: The sizes of lung lesions vary, and most infected tissues occupy a much smaller area than non-infected tissues. These smaller lesion tissues are the information of interest in the segmentation task. Using this data to train a deep learning model will not produce a good segmentation effect and will fall into a local minimum; 3. The segmentation performance of the model is difficult to improve: The boundary between infected tissue and healthy tissue is unclear and there is no obvious boundary. Moreover, the infected tissues are different in position and texture, which makes the segmentation of the infected part of the lung disease more challenging. Therefore, the over-segmentation and under-segmentation of the segmentation prediction map of the model are relatively common, resulting in low segmentation performance of the model. Summary of the invention

[0006] The first object of the present invention is to provide a lung CT image segmentation method based on Transformer and U-Net, which is used to accurately segment the lesion part in the medical image.

[0007] The specific implementation steps of the present invention are as follows:

[0008] Step S1, obtaining a CT scan image of a lung nodule, performing preprocessing and enhancement, and obtaining a data set;

[0009] Step S2: Construct segmentation model MCSTU-Net:

[0010] The segmentation model MCSTU-Net includes an encoder, a decoder, a skip connection, and a MCST (Mixed Convolution and Swin Transformer) module;

[0011] The encoder includes a first ResNet module, a second ResNet module, a third ResNet module and a fourth ResNet module connected in series, and the decoder includes a first Multi-Attention Fusion Module (MAF), a second Multi-Attention Module and a third Multi-Attention Module connected in series; a jump connection is used between the first ResNet module and the first Multi-Attention Module, a jump connection is used between the second ResNet module and the second Multi-Attention Module, and a jump connection is used between the third ResNet module, the fourth ResNet module and the third Multi-Attention Module, respectively, and an MCST module is connected in series on the jump connection;

[0012] The MCST module includes a skip connection branch, a Swin Transformer branch and a mixed dilated convolution branch; the skip connection branch includes a 1×1 convolution layer, the Swin Transformer branch includes a 1×1 convolution layer, a Patch Partition layer, a Linear Embedding layer, two Swin Transformer blocks and a Patch Expanding layer in series; the mixed dilated convolution branch includes a 1×1 convolution layer and three 3×3 dilated convolution layers with different expansion rates in series;

[0013] Step S3: Divide the data set into training set, test set and validation set, set the training strategy and loss function, and train the segmentation model MCSTU-Net:

[0014] Step S4: Verify the trained segmentation model MCSTU-Net:

[0015] The validation set is input into the trained segmentation model MCSTU-Net. After segmentation by the model, the lesion part in the medical image is segmented out. The segmented image is compared and evaluated with the lesion area judged by the expert to verify the segmentation model MCSTU-Net.

[0016] Step S5: Input any lung nodule CT scan image into the verified segmentation model MCSTU-Net to segment the lesion part.

[0017] Furthermore, the MCST module takes the output of the ResNet module at the same level as input, and obtains the feature map F after skip connection branch processing. RB , after being processed by the Swin Transformer branch, the feature map F is obtained ST , after processing with the mixed dilated convolution branch, the feature map F is obtained C ; F RB 、F ST 、F C After feature fusion, it is input into the 3×3 convolution layer, and then the activation function is processed to obtain the output of the MCST module, as follows:

[0018] Output MCST =ReLU(Conv 3×3 (Cat(F RB , F C , F ST )))

[0019] Among them: Cat() represents the concatenation operation, ReLU() is the ReLU activation function, Conv 3×3 It is a convolution operation with a convolution kernel of 3×3.

[0020] Furthermore, each of the multiple attention modules has the same structure, including a channel attention module and a spatial attention module; the multiple attention modules obtain the output of the MCST module at the same level. And the output of the previous multi-attention module Will After being processed by the channel attention module and the initial The element-by-element product (Hadamard Product) is performed, and then the intermediate feature F' is obtained after the 4×4 deconvolution operation; the intermediate feature F' is processed by the spatial attention module and the initial intermediate feature F' and the intermediate feature F' after the 3×3 convolution operation. Perform element-wise product and then add it to the 4×4 deconvolution operation. Element-wise addition is performed and the output of the multiple attention modules is obtained after the ReLU activation function.

[0021] Furthermore, the formula of the multiple attention module is as follows:

[0022]

[0023]

[0024] M S =σ(Conv 3×3 (Cat(AvgPool(F′), MaxPool(F′))))

[0025]

[0026] in, represents the features output by the kth MCST module, Deconv represents the high-level features from the k+1th MCST module, k∈{1, 2, 3}; 4×4 Indicates the deconvolution operation with a convolution kernel of 4×4; M S represents the 2D spatial attention map generated by the spatial attention module; M C represents the one-dimensional channel attention map generated by the channel attention module; MLP() represents a multi-layer perceptron; AvgPool() and MaxPool() represent average pooling and maximum pooling operations respectively; Conv 3×3 Represents a convolutional layer, and its subscript indicates the size of the convolution kernel; ReLU() indicates the ReLU activation function; Cat() indicates a concatenation operation; σ() indicates a Sigmoid activation function; and * indicates a Hadamard product operation.

[0027] The second object of the present invention is to provide a lung CT image segmentation system for implementing the above method, comprising the following modules:

[0028] Data acquisition module, acquiring CT scan images of lung nodules;

[0029] A data preprocessing module is used to preprocess the CT scan images of lung nodules;

[0030] The image segmentation module uses the trained and verified segmentation model MCSTU-Net to segment the preprocessed lung nodule CT scan images.

[0031] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the above method.

[0032] A fourth object of the present invention is to provide a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the above method is implemented.

[0033] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0034] The present invention is based on the U-Net architecture, and is composed of an encoder, a decoder and a key module MCST, which effectively fuses CNN and Swin Transformer modules, models long-distance dependencies and obtains local information of a larger receptive field, and fuses rich feature information to achieve the purpose of further enriching feature information. In the network architecture of the present invention, an MCST module is set in the jump connection part, and an MCST module consists of three parallel branches, namely a mixed hole convolution branch, a Swin Transformer branch and a jump connection branch. The mixed hole convolution branch in the MCST module obtains a larger receptive field through a hole convolution operation, and the Swin Transformer branch uses self-attention to model long-distance dependencies, obtains rich feature information in different ways, and outputs it to the corresponding level of the decoder after fusion.

[0035] The decoder of the present invention uses a Multi-Attention Fusion module (MAF) to extract feature information of two different scales. MAF includes a Channel Attention (CA) submodule and a Spatial Attention (SA) submodule, and the two submodules are connected together using a series structure.

[0036] In summary, through the network model architecture design of the present invention, the effectiveness and reliability of the lung image segmentation method are improved, and the problem of large size changes of infected tissues is addressed. By using different calculation methods of dilated convolution and Swin Transformer, the network model has both the bias induction ability of CNN and the ability of Transformer to model long-distance relationships, and obtains rich feature information from different angles. At the same time, the calculation amount of the model is minimized, the segmentation performance is significantly improved, and the boundary between infected tissues and healthy tissues is strengthened. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural diagram of the network of the lung CT image segmentation method of the present invention using Transformer and U-Net;

[0038] Figure 2 It is the structural diagram of MCST module;

[0039] Figure 3 It is the structural diagram of MAF module;

[0040] Figure 4 For medical images used for testing;

[0041] Figure 5 for Figure 4 Image after lesion segmentation. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings:

[0043] A lung CT image segmentation method based on Transformer and U-Net specifically includes the following steps:

[0044] Step 1: Preprocess and enhance the medical image data:

[0045] This example uses three datasets to train the model: a pulmonary nodule CT scan image dataset - LIDC-IDRI dataset, and two pulmonary CT image datasets of patients with lung diseases - Coronacases dataset and MosMedData dataset. Axial two-dimensional slices and their corresponding segmented images are generated based on the original three-dimensional CT scan and annotation information, and the CT slices and their corresponding segmented images are saved as png format images, with a slice size of 512×512 pixels. The generated images are cropped to include images of the lung area and the corresponding segmented images, and then the dataset is improved by cropping and enlarging to make it suitable for deep learning models, removing slices that do not contain pulmonary nodules in the LIDC-IDRI dataset, and removing some slices that do not contain pneumonia pulmonary lesions in the two lung disease datasets. The images generated by the first step will still contain a large amount of irrelevant background information, which will increase the amount of calculation on the one hand, and aggravate the problem of class imbalance on the other hand. The slices generated by this step are not fixed in size. The image sizes generated in the second step are all adjusted to 384×384 pixels and normalized. First, the grayscale value of the original image is normalized to between 0 and 1, and then it is uniformly multiplied by 255 to unify the pixel value to between 0 and 255. These images are still saved in png format. The pulmonary nodule dataset is divided into a training set and a validation set in a ratio of 8:2, and the pulmonary disease dataset is divided into a training set and a test set in a ratio of 8:2.

[0046] Step 2: Build a lung segmentation model based on Transformer and U-Net:

[0047] The segmentation model MCSTU-Net built based on the encoder-decoder structure of U-Net consists of three parts: ResNet-50, MCST and MAF;

[0048] like Figure 1 As shown, the overall network structure of MCSTU-Net can be displayed.

[0049] The encoder consists of the first four stages of ResNet-50, and the decoder consists of three modules connected in series that utilize the attention mechanism;

[0050] In the encoder:

[0051] ResNet-50 can be divided into four stages, and the kernel size of each convolutional layer is 3×3. Assuming an input image of size 224×224×3, the feature map size change process of the image is:

[0052] (1) Convolutional layer conv1: uses a 64-channel convolution kernel with a 7×7 filter, a stride of 2, and a padding of 3. The output feature map size is 112×112×64.

[0053] (2) Convolutional block conv2_x (first stage): First, it passes through a maximum pooling layer with a convolution kernel of 3×3 and a stride of 2, and the output feature map size is 56×56×64. Then, it passes through three consecutive residual units, and the output feature map size is 56×56×256;

[0054] (3) Convolutional layer conv3_x (second stage): four consecutive residual units, the size of the output feature map is 28×28×512;

[0055] (4) Convolutional layer conv4_x (third stage): six consecutive residual units, the size of the output feature map is 14×14×1024;

[0056] (5) Convolutional layer conv5_x (the fourth stage): three consecutive residual units, the size of the output feature map is 7×7×2048;

[0057] Then the features from different levels of the encoder are input into the MCST module respectively, which uses different calculation methods to obtain information from different angles. The jump connection branch can make up for the feature values ​​lost during the encoder downsampling process and the calculation process of other branches of the module. The mixed hole convolution branch will further extract context information by expanding the receptive field. The Swin Transformer branch is to model long-distance dependencies. These information are fused to obtain richer feature information.

[0058] The Swin Transformer branch includes a 1×1 convolutional layer, a Patch Partition layer, a Linear Embedding layer, two consecutive Swin Transformer blocks and a PatchExpanding layer in series; in two consecutive Swin Transformer blocks, window-based MSA (W-MSA) and sliding window-based MSA (Shifted Window-based MSA, (S)W-MSA) modules are used alternately. The first Swin Transformer block obtains the output of the Linear Embedding layer, and inputs it into the W-MSA after the layer normalization LN operation. The output of the W-MSA is connected to the output residual of the original Linear Embedding layer, and then input into the multilayer perceptron MLP through the layer normalization LN operation. Then, it is connected to the input residual of the multilayer perceptron MLP and output to the second Swin Transformer block. The second Swin Transformer block obtains the output of the first Swin Transformer block, and inputs it into the (S)W-MSA after the layer normalization LN operation. The output of the (S)W-MSA is connected to the output residual of the first Swin Transformer block, and then input into the multilayer perceptron MLP through the layer normalization LN operation. Then, it is connected to the input residual of the multilayer perceptron MLP and output to the Patch Expanding layer.

[0059] Based on this window partitioning mechanism, the calculation formula of the Swin Transformer block is as follows:

[0060]

[0061]

[0062]

[0063]

[0064] Among them, z i-1 represents the input of the block, LN represents layer normalization, W-MSA represents performing window multi-head self-attention within a fixed window, represents the output after the residual connection between the attention result and the input; MLP represents multi-layer perceptron; SW-MSA represents shifted window multi-head self-attention, which enables the information between windows to interact; z i represents the output of the MLP module of the i-th block.

[0065] The calculation formula of self-attention is as follows:

[0066]

[0067] Among them, Q, K, M denotes the query matrix, key matrix and value matrix respectively. 2 represents the number of patches in the window and d represents the dimension of the query or key. The values ​​in B are taken from the bias matrix Where M is the side length of the window.

[0068] The hybrid dilated convolution and Swin Transformer are used to reduce the amount of computation and trainable parameters, while expanding the receptive field and realizing the ability to model long-distance dependencies. This module combines Swin Transformer and CNN. It has the fast convergence of CNN and the powerful representation ability of Transformer. Inspired by the hybrid dilated convolution, the input features pass through multiple dilated convolution layers with different dilation rates (in this embodiment, the dilation rates r are 2, 3, and 5, respectively) to achieve the purpose of obtaining a larger receptive field. The feature map processed by a jump connection branch is denoted as F RB The feature map obtained after SwinTransformer branch processing is denoted as F ST The feature map obtained by processing the mixed dilated convolution branch is denoted as F C . F RB 、F ST 、F C After feature fusion, it is input into the 3×3 convolution layer, and then the activation function is processed to obtain the output of the MCST module, as follows:

[0069] Output MCST =ReLU(Conv 3×3 (Cat(F RB , F C , F ST )))

[0070] Among them: Cat() represents the concatenation operation, ReLU() is the ReLU activation function, Conv 3×3 It is a convolution operation with a convolution kernel of 3×3.

[0071] like Figure 2 As shown, the specific structure of the MCST module can be shown, which includes the fused CNN and SwinTransformer.

[0072] In the decoder:

[0073] According to the height and width of the feature information, the smaller feature information is called low-level feature, and the relatively larger one is called high-level feature. Each MAF module receives feature information of two different scales. This module not only uses the channel attention mechanism, but also adopts the spatial attention mechanism. This module uses the attention mechanism to enhance the feature information that deserves more attention. The operation process of the MAF module is: Compared with low-level features, high-level features More channels, advanced features First, through a channel attention subcomponent, the M of the one-dimensional channel attention map is inferred C , then the high-level features According to the channel attention map M C Extract interesting features, i.e. high-level features With channel attention map M C Perform element-wise multiplication, and the feature map obtained by deconvolution is denoted as F′, which has the same size as the low-level features. Keeping the same, F′ is processed by the spatial attention submodule to obtain a two-dimensional spatial attention map M S , in order to extract the key information in the feature space, the feature F′ is combined with the two-dimensional space attention map M S Perform element multiplication to obtain new features and add this new feature to the low-level features. Perform Hadamard product operation to extract low-level features At the same time, the initial high-level features Perform upsampling operations to make it consistent with low-level features The size of is kept consistent, and finally it is element-wise added to the feature map obtained after being processed by the attention submodule to merge the two features together. The output result is processed by the ReLU activation function as the input of the low-level MAF module.

[0074] The formula of the MAF module is as follows:

[0075]

[0076]

[0077] M S =σ(Conv 3×3 (Cat(AvgPool(F′), MaxPool(F′))))

[0078]

[0079] represents the features output by the kth MCST module, Deconv represents the high-level features from the k+1th MCST module, k∈{1, 2, 3}; 4×4 Indicates the deconvolution operation with a convolution kernel of 4×4; M S represents the one-dimensional channel attention map generated by the SA subcomponent; M C represents the 2D spatial attention map generated by the CA subcomponent; MLP() represents a multi-layer perceptron; AvgPool() and MaxPool() represent average pooling and maximum pooling operations, respectively; Conv 3×3 Represents a convolutional layer, and its subscript indicates the size of the convolution kernel; ReLU() indicates the ReLU activation function; Cat() indicates a concatenation operation; σ() indicates a Sigmoid activation function; and * indicates a Hadamard product operation.

[0080] like Figure 3 As shown, the specific structure of the MAF module can be displayed.

[0081] Finally, the CT image is converted into a segmented CT image by upsampling, and the segmented CT image is the same as the input image in terms of size and dimension;

[0082] Step 3: Set the training strategy and loss function to train the model:

[0083] The preprocessed dataset is divided into training set, test set and validation set; the model is trained using a combination of Dice loss and BCE loss, and a regularization strategy is used to prevent overfitting; the MCSTU-Net network model also uses a transfer learning strategy to help the model transition from natural image datasets to lung disease datasets; the loss function is used to update parameters during the training iteration process;

[0084] Compared with natural scenes, medical images have their own particularities, and the variability of object shapes and appearances is much smaller. Therefore, the prediction model obtained using an inappropriate loss function may have anatomical anomalies, holes, voids, or high inaccuracies near organ boundaries. Choosing a suitable loss function can not only improve the performance of the model, but also speed up the convergence of the model.

[0085] Dice loss is a loss function designed based on the Dice coefficient. Dice loss is more flexible in dealing with the problem of imbalance of samples of different categories in experimental data. Its mathematical definition is the formula:

[0086]

[0087] Among them, GT is the true segmentation result and P is the predicted segmentation result.

[0088] The loss curve of the binary cross-entropy (BCE) loss function is relatively smooth and can converge quickly, but it is easily affected by the class imbalance problem. Wei et al. proposed a variant of the binary cross-entropy loss function to alleviate the problem of imbalanced samples of different categories in experimental data.

[0089] The weighted BCE loss function is defined as follows:

[0090]

[0091] Among them, the value of l is 0 or 1, indicating that the label is positive or negative; H and W represent the width and height of the slice; g i，j represents the true value at row i and column j in the image pixel value matrix; p i，j represents the predicted value at row i and column j in the image pixel value matrix; ψ represents the parameters in the model; Pr(p i，j =l) is the predicted probability; a i，j Indicates the corresponding pixel weight.

[0092] The loss function used in the present invention is defined as the weighted sum of Dice loss and binary cross entropy loss (BCE), and its mathematical definition formula is as follows:

[0093]

[0094] Step 4: Verify the trained network model:

[0095] The segmented validation set is input into the trained segmentation model MCSTU-Net. After segmentation by the model, the lesion part in the medical image is segmented out. The segmented image is compared and evaluated with the lesion area judged by the expert to verify the network model.

[0096] Four widely adopted evaluation criteria are used to measure the performance of the MCSTU-Net model; the evaluation indicators are as follows:

[0097] Dice Similarity Coefficient (DSC): DSC is used to measure the similarity between the predicted lesion area and the fact. TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

[0098]

[0099] Sensitivity: SEN represents the percentage of correctly segmented lesion areas;

[0100]

[0101] Specificity: SPE represents the percentage of non-lesion areas that are correctly segmented;

[0102]

[0103] Positive Predictive Value (Precision): PPV reflects the percentage of correctly segmented infected area pixels in all prediction results;

[0104]

[0105] Step 5: Input any lung nodule CT scan image into the verified model to segment the lesion.

[0106] Select medical images for testing, such as Figure 4 As shown, this is the standard result of segmenting the image. Figure 5 This is the result after model segmentation. It can be seen that the MCSTU-Net model can accurately segment the lesion area. Such results provide strong help for doctors in diagnosis and treatment.

[0107] To further verify the effectiveness of the method of the present invention, the lung image segmentation method of the present invention is compared with the segmentation method proposed in the prior art, and the results are shown in Tables 1 to 3. The method of the present invention can improve DSC (Dice Similarity Coefficient), SEN (Sensitivity), SPE (Specificity), PPV (Positive Predictive Value) and segmentation speed in lung image segmentation. All comparative experiments are completed on the same set of data, and three different data sets are compared, namely: MosMedData, Coronacases, and non-homologous test sets.

[0108] Table 1 Quantitative evaluation results of segmentation methods for MosMedData dataset

[0109]

[0110] Table 2 Quantitative evaluation results of segmentation methods for the Coronacases dataset

[0111]

[0112] Table 3 Quantitative evaluation results of non-homologous test set segmentation methods

[0113]

[0114] In the drawings of the embodiments disclosed in the present invention, only the structures involved in the embodiments disclosed in the present invention are involved, but the above is only a preferred embodiment of the present invention, which can be fully applied to various fields suitable for the present invention. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrations shown and described here.

Claims

1. A lung CT image segmentation method based on Transformer and U-Net, characterized in that: The method comprises the following steps: Step S1, obtaining a CT scan image of a lung nodule and obtaining a data set after preprocessing; Step S2: Construct segmentation model MCSTU-Net: The segmentation model MCSTU-Net includes an encoder, a decoder, a skip connection, and a MCST module; The encoder includes a first ResNet module, a second ResNet module, a third ResNet module and a fourth ResNet module connected in series, and the decoder includes a first multiple attention module, a second multiple attention module and a third multiple attention module connected in series; a jump connection is used between the first ResNet module and the first multiple attention module, a jump connection is used between the second ResNet module and the second multiple attention module, and a jump connection is used between the third ResNet module, the fourth ResNet module and the third multiple attention module, respectively, and an MCST module is connected in series on the jump connection; The MCST module includes a skip connection branch, a Swin Transformer branch and a mixed dilated convolution branch; the skip connection branch includes a 1×1 convolution layer, the Swin Transformer branch includes a 1×1 convolution layer, a Patch Partition layer, a Linear Embedding layer, two consecutive Swin Transformer blocks and a Patch Expanding layer in series; the mixed dilated convolution branch includes a 1×1 convolution layer and three 3×3 dilated convolution layers with different expansion rates in series; Step S3, divide the data set into a training set, a test set and a validation set, set the training strategy and loss function, and train the segmentation model MCSTU-Net; Step S4: Verify the trained segmentation model MCSTU-Net: The validation set is input into the trained segmentation model MCSTU-Net. After segmentation by the model, the lesion part in the medical image is segmented out. The segmented image is compared and evaluated with the lesion area judged by the expert to verify the segmentation model MCSTU-Net. Step S5: Input any lung nodule CT scan image into the verified segmentation model MCSTU-Net to segment the lesion part.

2. The lung CT image segmentation method based on Transformer and U-Net according to claim 1, characterized in that: The MCST module takes the output of the ResNet module at the same level as input and obtains the feature map F after processing by the jump connection branch. RB , after being processed by the Swin Transformer branch, the feature map F is obtained SR , after processing with the mixed dilated convolution branch, the feature map F is obtained C ; F RB 、F ST 、F C After feature fusion, it is input into the 3×3 convolutional layer and then processed by the activation function to obtain the output of the MCST module.

3. The lung CT image segmentation method based on Transformer and U-Net according to claim 1, characterized in that: The two consecutive Swin Transformer blocks perform W-MSA and (S)W-MSA operations respectively.

4. The lung CT image segmentation method based on Transformer and U-Net according to claim 3, characterized in that: The first Swin Transformer block obtains the output of the Linear Embedding layer, and inputs it into the W-MSA after layer normalization LN operation. The output of the W-MSA is connected with the output residual of the original Linear Embedding layer, and then input into the multi-layer perceptron MLP after layer normalization LN operation. Then, it is connected with the input residual of the multi-layer perceptron MLP and output to the second Swin Transformer block. The second Swin Transformer block obtains the output of the first Swin Transformer block, and after layer normalization LN operation, it is input into the (S)W-MSA. The output of the (S)W-MSA is connected with the output residual of the first Swin Transformer block and then input into the multilayer perceptron MLP through layer normalization LN operation. It is then connected with the input residual of the multilayer perceptron MLP and output to the Patch Expanding layer.

5. The lung CT image segmentation method based on Transformer and U-Net according to claim 1, characterized in that: Each multi-attention module has the same structure, including a channel attention module and a spatial attention module; the multi-attention module obtains the output of the MCST module at the same level And the output of the previous multi-attention module Will After being processed by the channel attention module and the initial Perform element-by-element multiplication and then perform a 4×4 deconvolution operation to obtain the intermediate feature F'; The intermediate feature F' after processing by the spatial attention module is compared with the initial intermediate feature F' and the 3×3 convolution operation. Perform element-wise product and then add it to the 4×4 deconvolution operation. Perform element-by-element addition and obtain the output of the multiple attention modules after the ReLU activation function.

6. The lung CT image segmentation method based on Transformer and U-Net according to claim 5, characterized in that: The formula of the multi-attention module is as follows: M S =σ(Conv 3×3 (Coat(AvgPool(F'),MaxPool(F')))) in, represents the features output by the kth MCST module, Deconv represents the high-level features from the k+1th MCST module, k∈{1, 2, 3}; 4×4 Indicates the deconvolution operation with a convolution kernel of 4×4; M S represents the 2D spatial attention map generated by the spatial attention module; M C represents the one-dimensional channel attention map generated by the channel attention module; MLP() represents a multi-layer perceptron; AvgPool() and MaxPool() represent average pooling and maximum pooling operations respectively; Conv 3×3 Represents a convolutional layer, and its subscript represents the size of the convolution kernel; ReLU( ) represents the ReLU activation function; Cat( ) represents the concatenation operation; σ( ) represents the Sigmoid activation function; * represents the Hadamard product operation.

7. A lung CT image segmentation system implementing the method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Data acquisition module, acquiring CT scan images of lung nodules; A data preprocessing module is used to preprocess the CT scan images of lung nodules; The image segmentation module uses the trained and verified segmentation model MCSTU-Net to segment the preprocessed lung nodule CT scan images.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images