A lung CT image segmentation method based on a transformer and a convolutional neural network

Abstract

A lung CT image segmentation method based on a Transformer and a convolutional neural network, first, data augmentation and data preprocessing are performed on the data set, then a deep neural network model of the application is constructed, the network model includes a backbone network, a fused Swin Transformer and CNN module and a multiple attention fusion module, the image is input into the network model, first, the backbone network is used for coding to obtain multi-level features, the multi-level features are respectively input into the fused Swin Transformer and CNN module to model long-distance relationships and obtain a larger receptive field, then the multiple attention fusion module is used for feature enhancement and fusion, finally, an upsampling operation is used to restore the feature map to the input size. After training the network model through the transfer learning strategy, the lung CT image to be processed is input into the trained model to obtain the segmentation result of the image. The patent can train a network model for automatically segmenting lung CT images with more accurate segmentation results under a small amount of annotated data sets, can improve the productivity of the work, and has high application value in clinical medical treatment.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation based on deep learning technology, specifically involving a lung CT image segmentation method based on Transformer and Convolutional Neural Networks (CNN).

[0002] Background Technology and Significance

[0003] Lung diseases are relatively common, with high incidence and mortality rates. It is estimated that in 2020, there were 19.3 million newly diagnosed cancer cases globally, and nearly 10 million people died from cancer. Lung cancer was the second most common cancer, accounting for approximately 11.4% of all cancers. Lung cancer also has the highest mortality rate among cancers, with an estimated 1.8 million deaths, accounting for about 18% of all cancer deaths. According to research by Peking University and the China-Japan Friendship Hospital, the incidence of community-acquired pneumonia in China in 2016 was approximately 7 per 1,000, with about 1.4 million people experiencing at least one episode of community-acquired pneumonia, totaling approximately 1.48 million cases. Medical institutions generally use computed tomography (CT) scans to generate medical images of the lungs of patients with lung diseases and segment regions of interest within the images for diagnosis and monitoring of the patient's condition. Manual segmentation of medical images requires a significant amount of time from medical experts and is easily influenced by subjective experience; automating this process can improve productivity and standardize segmentation criteria.

[0004] Currently, although convolutional neural network (CNN)-based methods have achieved good results in the segmentation of pneumonia lesions in lung CT images, they still cannot fully meet the stringent segmentation accuracy requirements of medical applications, exhibiting problems such as overfitting and undersegmentation. Due to the inherent limitations of CNNs, CNN-based methods lack the ability to model long-range dependencies. Traditional Vision Transformers are computationally expensive, and Transformers lack some inductive biases inherent in CNNs, thus requiring extensive data training to achieve better performance. The Swin Transformer, based on Transformers, can acquire hierarchical features, and its computational complexity is linearly related to the spatial resolution of the image. The Swin Transformer has achieved state-of-the-art performance in many areas of vision tasks, and its success in image tasks demonstrates its application potential in medical image segmentation.

[0005] Medical images differ from natural images. Different organs and tissues have unique characteristics, requiring different imaging methods. Due to the diversity of imaging methods and their multimodal nature, no ideal segmentation method is suitable for all tissues and organs. Currently, deep learning-based methods have achieved good results in the automatic segmentation of lung CT images. However, these methods still cannot fully meet the stringent requirements for segmentation accuracy in medical applications. This task faces the following challenges:

[0006] (1) Lack of labeled datasets. Deep learning techniques require sufficient datasets for training and testing models; otherwise, the generalization and accuracy of the models will be affected. Although the development of computer-aided diagnostic methods has accelerated, obtaining well-characterized image data remains a common limitation, as the task of identifying and collecting appropriate images for any specific research activity is a laborious and costly process. Researchers developing automated detection methods need the opinions of experienced radiologists, or more accurately, a group of radiologists regarding the location of lesions in images.

[0007] (2) Class imbalance. The size of lung lesions varies, and most infected tissue occupies a much smaller area than non-infected tissue. These smaller lesions are the information of interest for segmentation tasks. Using such data to train a deep learning model will not yield good segmentation results and will get stuck in local minima.

[0008] (3) The boundary between infected and healthy tissue is unclear and there is no clear boundary. Moreover, infected tissues differ in location and texture, which makes the segmentation of infected tissues more challenging. Therefore, oversegmentation and undersegmentation are common in the model's segmentation prediction map, resulting in low segmentation performance of the model. Summary of the Invention

[0009] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a lung CT image segmentation method based on Transformer and convolutional neural network.

[0010] To address these issues, this invention leverages the advantages of Swing Transformer and Convolutional Neural Networks to improve the accuracy of lung lesion tissue segmentation in patients with lung diseases. Addressing the challenges of lesion tissue segmentation in lung CT images, this invention provides a lung CT image segmentation method based on Transformer and Convolutional Neural Networks, used to accurately segment lung lesion tissue from lung CT images.

[0011] A lung CT image segmentation method based on Transformer and convolutional neural networks includes the following steps:

[0012] Step 1) Input the dataset;

[0013] Input the LIDC-IDRI dataset of lung nodules CT scans;

[0014] Input a dataset of labeled lung CT images of pneumonia patients, denoted as Pneumonia CT-Mask;

[0015] Input the lung CT image of the pneumonia patient to be segmented, denoted as pneumoniaCTCT.

[0016] Step 2) Data augmentation and data preprocessing;

[0017] According to the training requirements and characteristics of the dataset of this invention, preprocessing of the dataset is necessary. Because the amount of annotated data is relatively small, while deep learning models require a large amount of data for training, converting 3D CT scan images into 2D slices can increase the data volume. Furthermore, doctors diagnose conditions based on CT axial slices; therefore, this invention designs a model for segmenting 2D slices.

[0018] Step 1: Generate axial 2D slices and their corresponding segmented images based on the original 3D CT scan and annotation information.

[0019] The second step involves cropping the images generated in the first step to include the lung region and the corresponding segmented images. Then, the dataset is improved by cropping and enlarging to make it suitable for deep learning models. This involves removing slices from the LIDC-IDRI dataset that do not contain lung nodules, and removing some slices from the pneumonia CT-Mask dataset that do not contain pneumonia lesions. The images generated in the first step still contain a large amount of irrelevant background information, increasing computational cost and exacerbating class imbalance. The slices generated in this step have variable sizes.

[0020] Step 3: All images generated in Step 2 are resized to 384×384 pixels and normalized to avoid adverse effects on training from a few outliers. First, the grayscale values ​​of the original images are unified to between 0 and 1, then multiplied by 255 to normalize the pixel values ​​to between 0 and 255. These images are saved as PNGs. The lung nodule dataset is divided into training and validation sets in an 8:2 ratio, while the pneumonia dataset is divided into training, validation, and test sets in a 6:2:2 ratio.

[0021] Step 3) Construct the network model;

[0022] We construct our invented segmentation model, MCSTU-Net. MCSTU-Net is a U-shaped network consisting of an encoder, a decoder, and skip connection modules. The encoder of MCSTU-Net comprises the first four stages of ResNet-34, specifically the first four ResNet-34 Blocks (RBs). A module is added to the skip connection portion of MCSTU-Net: a Mixed CNN and Swin Transformer module (MCST). The decoder of MCSTU-Net consists of three cascaded modules utilizing an attention mechanism: the Multi-Attention Fusion module (MAF).

[0023] Images used to train the MCSTU-Net model are first input to the encoder. The ResNet-34 encoder extracts hierarchical features at different stages using a series of convolutional layers and successive downsampling layers. These hierarchical features have different receptive fields. Features from different levels of the encoder are then input to the MCST module, which uses different computational methods to obtain information from different perspectives. Skip connections compensate for feature values ​​lost during encoder downsampling and other branch computations. Hybrid dilated convolution branches further extract contextual information by expanding the receptive field. The Swin Transformer branch models long-range dependencies. This fusion of information yields richer feature information. In the decoder, based on the height and width of the feature information, smaller features are called low-level features, and relatively larger ones are called high-level features. Each MAF module receives feature information at two different scales. This module utilizes not only channel attention but also spatial attention mechanisms. Through these attention mechanisms, the MAF module enhances more noteworthy feature information. This invention uses 1×1 convolution to reduce the dimensionality of the features output by the last MAF module in the calculation process, resulting in a feature map with 1 channel. Then, it uses bilinear interpolation to perform a 4x upsampling operation to restore the feature map to the input size as the segmentation prediction result.

[0024] 3.1. The MCST module proposed in this invention:

[0025] This module is used to model long-range dependencies and acquire local information with a large receptive field, fusing rich feature information to further enrich the feature set. An MCST module consists of three parallel branches: a hybrid dilated convolution branch, a Swin Transformer branch, and a skip connection branch. Each branch begins with a 1×1 convolutional layer, reducing the input to half the channels of the previous layer. The hybrid dilated convolution branch in the MCST module obtains a larger receptive field through dilated convolution operations, the Swin Transformer branch uses Swin Transformer blocks to model long-range dependencies, and the skip connection branch prevents degradation issues in the MCSTU-Net model. Rich feature information obtained through different methods is fused and output to the corresponding layer of the decoder. The MCST module combines Swin Transformer and CNN, possessing the fast convergence of CNN and the powerful representational capabilities of Transformer.

[0026] Compared to ordinary convolutional layers, dilated convolutions can achieve a larger receptive field without increasing computational cost or time. However, improper stacking of dilated convolutional layers can lead to a mesh effect, resulting in the loss of some information. To address this issue, multiple dilated convolutional layers with different dilation rates can be cascaded. Since the area of ​​lung lesions varies greatly—some lesions are very small, while others can spread throughout the entire lung—the dilation rate of the dilated convolutional layers should not be too large, as this will degrade model performance. At the same time, the dilation rate cannot be too small, as this would cause the dilated convolution to degenerate into a traditional convolution. Therefore, the dilation rate setting must also meet the condition that the dilation rates of the cascaded dilated convolutional layers cannot have a common divisor greater than 1. Because the input image size of the MCSTU-Net model is planned to be 384×384 pixels, the dilation rates of the dilated convolutional layers are 2, 3, and 5 respectively, and their convolutional kernels are all 3×3. The feature map after processing by the hybrid dilated convolutional branches is denoted as... .

[0027] The Swin Transformer branch has a powerful ability to model long-distance relationships and is used to extract global information. The specific operation flow of the Swin Transformer branch is described below. This branch receives a feature image as input from the encoder and first performs dimensionality reduction on it through a convolutional layer to obtain... Then, a patching layer is used to segment the input feature image into non-overlapping patches. Since this branch's computation requires sequence embedding, the patch size is 4×4. Through this partitioning method, the feature dimension of each patch becomes 4×4×C = 16C. Because the computational cost of this branch increases exponentially with the data dimension, the feature dimension cannot be too large. A linear embedding layer projects the original features to an arbitrary dimension (denoted as V). The dimensions of the MCST module are set to 128, 256, 512, and 1024 respectively. The resolution is... A V-dimensional tokenized patch is input into two consecutive Swing Transformer blocks for representation learning, where the feature dimension and resolution remain unchanged. The patch expansion layer regenerates a map upsampled to 4 times the resolution based on the feature information of adjacent dimensions, and then restores the resolution of the feature map to the input resolution. The feature map obtained after processing by the Swing Transformer branch is denoted as . .

[0028] The feature map processed by a skip connection branch is denoted as... The three features obtained from the three branches, each with the same spatial resolution but containing different key information, are concatenated using a stitching operation. Then, they are passed through a 3×3 convolutional layer. The purpose of this layer is to reduce the number of channels to one-third of the original without changing the spatial resolution of the feature map. The final output is a feature map rich in information.

[0029] The mathematical expression of the fused CNN and Swing Transformer modules is Equation (1).

[0030]

[0031] in:

[0032] , , As mentioned above;

[0033] Indicates a splicing operation;

[0034] It is the ReLU activation function;

[0035] It is a convolution operation with a 3×3 kernel.

[0036] 3.2. The MAF module proposed in this invention:

[0037] Each MAF module receives feature information at two different scales. In the decoder, this method classifies the smaller feature information as low-level features and the relatively larger one as high-level features, based on the height and width of the feature information. To refine this feature information, this invention designs a module called the Multi-Attention Fusion module (MAF). This module contains a Channel Attention (CA) submodule and a Spatial Attention (SA) submodule, which are connected in a cascaded structure. In these two attention submodules, average pooling and max pooling operations are used simultaneously to obtain different feature information from different perspectives.

[0038] The detailed operation process of the MAF module is as follows: Compared to low-level features, high-level features... More channels, advanced features First, a channel attention sub-component is used to infer a one-dimensional channel attention graph. Then advanced features According to the channel attention diagram Extracting noteworthy features, i.e., high-level features. With regard to the channel diagram The feature map obtained by performing element-wise multiplication is denoted as the feature map obtained by deconvolution. Its size and basic features Maintain consistency. A two-dimensional spatial attention map is obtained through processing by the spatial attention submodule. In order to extract key information from the feature space, features Attention diagram in two-dimensional space Performing element-wise multiplication yields a new feature, which is then combined with the lower-level feature. Perform the Hadamard product operation to extract low-level features. This contains noteworthy information. Simultaneously, the initial high-level features... Perform an upsampling operation to make it the same as the low-level features. The dimensions are kept consistent. Finally, element-wise addition is performed between this feature map and the feature map obtained after processing by the attention submodule, fusing the two features together. The result is then processed by the ReLU activation function and used as the input to the low-level MAF module. It's important to note that this method uses a 1×1 convolution to reduce the dimensionality of the features output from the last MAF module during computation, resulting in a feature map with 1 channel. Then, bilinear interpolation is used for a 4x upsampling operation to restore the feature map to the input size as the segmentation prediction result.

[0039] The mathematical model for the multi-attention fusion module is shown below:

[0040]

[0041] in:

[0042] Indicates the first The characteristics output by each MCST module Indicates from the Advanced features of the MCST module, ;

[0043] This indicates a deconvolution operation with a 4×4 kernel;

[0044] This represents a one-dimensional channel attention diagram generated by the SA sub-component;

[0045] This represents a two-dimensional spatial attention graph generated by the CA sub-component.

[0046] This represents a multilayer perceptron;

[0047] and These represent average pooling and max pooling operations, respectively.

[0048] This indicates a convolutional layer, and its subscript indicates the size of the convolutional kernel.

[0049] Represents the ReLU activation function;

[0050] Indicates a splicing operation;

[0051] This represents the Sigmoid activation function;

[0052] This indicates the Hadamard product operation.

[0053] 3.3 Loss Function

[0054] Compared to natural scenes, medical images have their own unique characteristics, with much less variability in object shape and appearance. Therefore, predictive models obtained using inappropriate loss functions may exhibit anatomically significant errors, such as holes or cavities near organ boundaries, or high inaccuracies. Choosing a suitable loss function can not only improve model performance but also accelerate convergence. Implementing a medical image segmentation network requires selecting a suitable architecture and training it to optimize network weights, which necessitates the use of an appropriate loss function. Cross-entropy and Dice loss, as well as their variations and combinations, are widely used in segmentation networks.

[0055] Dice loss is a loss function designed based on the Dice coefficient. Dice loss is more flexible in dealing with imbalanced samples of different classes in experimental data. Its mathematical definition is formula (6):

[0056]

[0057] Where GT is the true segmentation result and P is the predicted segmentation result.

[0058] The binary cross-entropy (BCE) loss function has a relatively smooth loss curve and converges quickly, but it is susceptible to class imbalance. This invention uses a variant of the binary cross-entropy loss function to alleviate the problem of imbalance between different classes in the experimental data.

[0059] The weighted BCE loss function is defined in formula (7):

[0060]

[0061] in:

[0062] The value is 0 or 1, indicating that the label is positive or negative;

[0063] and This represents the width and height of the slice;

[0064] This represents the true value at row i and column j in the image pixel value matrix;

[0065] This represents the predicted value at row i and column j in the image pixel value matrix;

[0066] Represents the parameters in the model;

[0067] It is the predicted probability;

[0068] This represents the weight of the corresponding pixel.

[0069] The loss function used in this invention is defined as a weighted sum of Dice loss and binary cross-entropy (BCE) loss, and its mathematical definition is given in formula (8):

[0070]

[0071] Step 4) Transfer learning strategy;

[0072] In the field of deep learning-based medical image processing, transfer learning is also an effective method to alleviate data shortages. This strategy has been widely applied to lung CT image segmentation tasks. The main idea of ​​transfer learning can be summarized as: learning knowledge related to the current task from other related tasks, and transferring the learned knowledge to the target task through fine-tuning or model pre-training. Currently, there are many backbone networks trained using large natural image datasets (such as ImageNet), such as ResNet and VGGNet. Researchers have found that transferring these trained parameters to the medical image processing domain is effective and can improve the performance of deep learning models for medical image segmentation tasks. Generally, the higher the similarity between the source and target domains, the better the effect of transfer learning. By analyzing the characteristics of lung CT images, it was found that the existing LIDC-IDRI dataset is suitable as a pre-training set for this model.

[0073] This invention uses a two-stage cross-domain transfer learning strategy to train MCSTU-Net.

[0074] First, the encoder of the MCSTU-Net model consists of the first four residual blocks of the ResNet-34 network, which its design team has pre-trained on ImageNet. During MCSTU-Net model initialization, the decoder first loads the weights of the first four feature extraction modules of ResNet-34. This stage of transfer learning is model-level transfer learning, which has the advantages of being simple to use, saving pre-training time, and improving the performance of the MCSTU-Net model.

[0075] Then, the MCSTU-Net model was pre-trained using the lung nodule dataset, and the pre-trained MCSTU-Net model parameters were saved. Images in the ImageNet database are all natural images, which differ significantly from medical images. Medical images often exhibit low contrast at organ and tissue boundaries and class imbalance. In CT images, there is a very high similarity between features on pneumonia lesions and lung nodules. This stage of our paper utilizes data-level transfer learning to help the model transition from the natural image dataset to the pneumonia dataset.

[0076] Finally, the parameters of the MCSTU-Net model trained through the second stage of transfer learning are loaded, and the MCSTU-Net model is trained using the pneumonia dataset.

[0077] Step 5) Training strategy;

[0078] The general experimental setup for this method is as follows: the initial input image resolution is set to 384×384 pixels. The Adam optimizer is used to update the parameters of the MCSTU-Net model, with a momentum value of 0.9. The initial learning rate is 0.0001, trained for 100 epochs, and then reduced to 0.1 times its original value (0.00001) every 50 epochs. The batch size is set to 10, and the patch size is 4×4 pixels. The model is pre-trained using a lung nodule dataset.

[0079] First, the MCSTU-Net model is pre-trained using the lung nodule dataset. At this stage, the MCSTU-Net decoder loads the parameters trained on ImageNet using ResNet-34. It's worth noting that when training the lung nodule dataset using transfer learning, the slices in the dataset are divided into training and validation sets in an 8:2 ratio. The validation set is used to search for the best model and save the optimal model parameters. Then, the MCSTU-Net model is trained using the pneumonia dataset. In the initial stage, the parameters pre-trained on the lung nodule dataset are loaded. Finally, the model performance is tested using their respective test sets.

[0080] Step 6) Evaluation indicators;

[0081] When evaluating the performance of network models used for medical image segmentation, the similarity between the predicted and actual results is typically used. Four labels are defined: True Negative (TN), False Positive (FP), False Negative (FN), and True Positive (TP). A True Positive indicates that both the predicted and actual results show overlapping areas of normal tissue; a True Negative indicates that both the predicted and actual results show areas of diseased tissue; a False Positive indicates that the predicted result shows normal tissue while the actual result shows diseased tissue; and a False Negative indicates that the predicted result shows diseased tissue while the actual result shows normal tissue. This method uses four commonly used evaluation metrics to compare the performance of the methods and models involved in the experiment: DICE similarity coefficient, sensitivity, specificity, and positive prediction rate. By calculating these evaluation metrics, our method is compared with state-of-the-art methods for lung CT image segmentation to evaluate its performance.

[0082] The Dice Similarity Coefficient (DSC) is commonly used to measure the similarity between the actual and predicted results. Its specific definition is given in formula (9). It is defined as the proportion of twice the number of correctly segmented lesions in the actual and predicted results, with a value between 0 and 1. The closer the value is to 1, the better the result.

[0083]

[0084] Sensitivity (SEN) is also known as recall. It is the percentage of infected tissue that is correctly segmented relative to the actual lesion tissue. Its formula is shown in (10).

[0085]

[0086] Specificity (SPE) represents the percentage of normal tissue that is correctly segmented. Its formula is shown in (11).

[0087]

[0088] The Positive Predictive Value (PPV) reflects the percentage of correctly segmented infected region pixels out of all predictions. Its formula is shown in (12).

[0089]

[0090] Step 7) Use the pre-trained network model;

[0091] Save the weight values ​​of the trained MCSTU-Net network model. When using it, build the network model and load the weight values ​​obtained after training. Input the original lung CT image of the patient with lung disease, and finally get the segmented image.

[0092] Compared with the prior art, the present invention, employing the above technical solution, has the following advantages:

[0093] 1. A lung CT image segmentation method based on Transformer and Convolutional Neural Network (CNN) was invented. This method integrates CNN and Transformer, enabling the MCSTU-Net network model to simultaneously possess the fast convergence of CNN and the powerful representational capabilities of Transformer, thus more accurately segmenting lesions in lung CT images. Data augmentation techniques and transfer learning strategies are used to alleviate the problem of insufficient dataset.

[0094] 2. An MCST module integrating convolutional neural networks and Swing Transformers was invented. By utilizing the different computational methods of dilated convolution and Swing Transformers, the MCSTU-Net network model can simultaneously possess the bias induction ability of CNNs and the ability of Transformers to model long-distance relationships, thereby acquiring rich feature information from different angles. At the same time, the computational load of the model is minimized, enabling more accurate segmentation of lesions of different locations and sizes.

[0095] 3. A multi-attention fusion module (MAF) was designed, which utilizes channel attention and spatial attention mechanisms to enhance feature information and fuses feature information from different levels of the encoder, making the segmentation of lesion tissue boundaries in lung CT images more accurate. Attached Figure Description

[0096] Figure 1 This is a flowchart of the specific method proposed in this invention;

[0097] Figure 2 This is a diagram of the MCSTU-Net network structure proposed in this invention;

[0098] Figure 3 This is a structural diagram of the MCST module proposed in this invention;

[0099] Figure 4 This is a structural diagram of the MAF module proposed in this invention;

[0100] Figure 5 This is a structural diagram of the attention submodule in the MAF module proposed in this invention. Detailed Implementation

[0101] The present invention will be further described below with reference to the accompanying drawings:

[0102] like Figure 1 As shown, the lung CT image segmentation method based on Transformer and convolutional neural network of the present invention specifically includes the following steps:

[0103] Step 1) Input the dataset;

[0104] Input the LIDC-IDRI dataset of lung nodules CT scans;

[0105] Input a dataset of labeled lung CT images of pneumonia patients, denoted as Pneumonia CT-Mask;

[0106] Input the lung CT images of the pneumonia patient to be segmented, denoted as pneumonia CT.

[0107] Step 2) Data augmentation and data preprocessing;

[0108] According to the training requirements and characteristics of the dataset presented in this invention, preprocessing of the dataset is necessary. Because the amount of annotated data is relatively small, while deep learning models require a large amount of data for training, converting 3D CT scan images into 2D slices can increase the data volume. Furthermore, doctors diagnose conditions based on CT axial slices; therefore, this paper proposes a model for segmenting 2D slices.

[0109] Step 1: Generate axial 2D slices and their corresponding segmented images based on the original 3D CT scan and annotation information, and save the CT slices and their corresponding segmented images as PNG images. At this time, the slice size is 512×512 pixels.

[0110] The second step involves cropping the images generated in the first step to include the lung region and the corresponding segmented images. Then, the dataset is improved by cropping and enlarging to make it suitable for deep learning models. This involves removing slices from the LIDC-IDRI dataset that do not contain lung nodules, and removing some slices from the pneumonia CT-Mask dataset that do not contain pneumonia lesions. The images generated in the first step still contain a large amount of irrelevant background information, increasing computational cost and exacerbating class imbalance. The slices generated in this step have variable sizes.

[0111] Step 3: All images generated in Step 2 are resized to 384×384 pixels and normalized to avoid adverse effects on training from a few outliers. First, the grayscale values ​​of the original images are unified to between 0 and 1, then multiplied by 255 to normalize the pixel values ​​to between 0 and 255. These images are still saved in PNG format. The lung nodule dataset is divided into training and validation sets in an 8:2 ratio, while the pneumonia dataset is divided into training, validation, and test sets in a 6:2:2 ratio.

[0112] Step 3) Construct the network model;

[0113] We construct our invented segmentation model, MCSTU-Net. For example... Figure 2 As shown, MCSTU-Net is a U-shaped network architecture consisting of an encoder, a decoder, and skip connection modules. The encoder of MCSTU-Net comprises the first four stages of ResNet-34, i.e., the first four ResNet-34 Blocks (RBs). Since Transformer-based models require large-scale pre-training to maximize performance, this method does not use the Swing Transformer as the backbone network. A module is added to the skip connection part of MCSTU-Net, namely the Mixed CNN and Swing Transformer module (MCST). The decoder of MCSTU-Net consists of three cascaded modules utilizing attention mechanisms, namely the Multi-Attention Fusion module (MAF).

[0114] The images used to train the model are first input to the encoder. The ResNet-34 encoder extracts hierarchical features at different stages using a series of convolutional layers and successive downsampling layers. These hierarchical features have different receptive fields. Then, features from different levels of the encoder are input to the MCST module. This module uses different calculation methods to obtain information from different perspectives. Skip connection branches compensate for feature values ​​lost during encoder downsampling and calculations in other branches of this module. Hybrid dilated convolution branches further extract contextual information by expanding the receptive field. The Swin Transformer branch models long-range dependencies. This fusion of information yields richer feature information. In the decoder, based on the height and width of the feature information, smaller features are called low-level features, and relatively larger ones are called high-level features. Each MAF module receives feature information at two different scales. This module utilizes not only channel attention but also spatial attention mechanisms to enhance more noteworthy feature information. This invention uses 1×1 convolution to reduce the dimensionality of the features output by the last MAF module in the calculation process, resulting in a feature map with 1 channel. Then, it uses bilinear interpolation to perform a 4x upsampling operation to restore the feature map to the input size as the segmentation prediction result.

[0115] 3.1. The MCST module proposed in this invention:

[0116] This module is used to model long-distance dependencies and acquire local information with a large receptive field, and to fuse rich feature information to further enrich the feature information. For example... Figure 3 As shown, an MCST module consists of three parallel branches: a hybrid dilated convolution branch, a Swin Transformer branch, and a skip connection branch. Each branch begins with a 1×1 convolutional layer, reducing the input to half the channels of the previous layer. The hybrid dilated convolution branch in the MCST module obtains a larger receptive field through dilated convolution operations, the Swin Transformer branch models long-range dependencies using Swin Transformer blocks, and the skip connection branch prevents model degradation. Rich feature information obtained through different methods is fused and output to the corresponding layer of the decoder. This module integrates Swin Transformer and CNN, possessing the fast convergence of CNN and the powerful representational capabilities of Transformer.

[0117] The feature map processed by a skip connection branch is denoted as... .

[0118] Compared to ordinary convolutional layers, dilated convolutions can achieve a larger receptive field without increasing computational cost or time. However, improper stacking of dilated convolutional layers can lead to a mesh effect, resulting in the loss of some information. To address this issue, multiple dilated convolutional layers with different dilation rates can be cascaded. Since the area of ​​lung lesions varies greatly—some lesions are very small, while others can spread throughout the entire lung—the dilation rate of the dilated convolutional layers should not be too large, as this will degrade the model's performance. At the same time, the dilation rate cannot be too small, as this would cause the dilated convolution to degenerate into a traditional convolution. Therefore, the dilation rate setting must also meet the condition that the dilation rates of the cascaded dilated convolutional layers cannot have a common divisor greater than 1. Because the planned input image size for this model is 384×384 pixels, the dilation rates of the dilated convolutional layers in this branch are 2, 3, and 5 respectively, and their convolutional kernels are all 3×3.

[0119] The Swin Transformer branch has a powerful ability to model long-distance relationships and is used to extract global information. The specific operation flow of the Swin Transformer branch is described below. This branch receives a feature image as input from the encoder and first performs dimensionality reduction on it through a convolutional layer to obtain... Then, a patching layer is used to segment the input feature image into non-overlapping patches. Since this branch's computation requires sequence embedding, the patch size is 4×4. Through this partitioning method, the feature dimension of each patch becomes 4×4×C = 16C. Because the computational cost of this branch increases exponentially with the data dimension, the feature dimension cannot be too large. A linear embedding layer projects the original features to an arbitrary dimension (denoted as V). In this model setup, the dimensions of the MCST modules from top to bottom are set to 128, 256, 512, and 1024 respectively. The resolution is... A V-dimensional tokenized patch is input into two consecutive SwinTransformer blocks for representation learning, where the feature dimension and resolution remain unchanged. The patch expansion layer regenerates a map upsampled to 4 times the resolution based on the feature information of adjacent dimensions, and then restores the resolution of the feature map to the input resolution. The feature map obtained after processing by the SwinTransformer branch is denoted as . .

[0120] The three features obtained from the three branches, each with the same spatial resolution but containing different key information, are concatenated using a stitching operation. Then, they are passed through a 3×3 convolutional layer. The purpose of this layer is to reduce the number of channels to one-third of the original without changing the spatial resolution of the feature map. The final output is a feature map rich in information.

[0121] The mathematical expression of the fused CNN and Swing Transformer modules is Equation (1).

[0122]

[0123] in:

[0124] , , As mentioned above;

[0125] Indicates a splicing operation;

[0126] It is the ReLU activation function;

[0127] It is a convolution operation with a 3×3 kernel.

[0128] 3.2. The MAF module proposed in this invention:

[0129] Each MAF module receives feature information at two different scales. In the decoder, this method refers to the smaller feature information as low-level features and the relatively larger one as high-level features, based on the height and width of the feature information. To refine this feature information, this invention designs a module, namely the Multi-Attention Fusion module (MAF), whose structure is as follows: Figure 4 As shown, this module contains a Channel Attention (CA) submodule and a Spatial Attention (SA) submodule, which are connected together in a cascaded structure. Figure 5 As shown, in these two attention sub-modules, average pooling and max pooling operations are used simultaneously to obtain different feature information from different perspectives.

[0130] The detailed operation process of the MAF module is as follows: Compared to low-level features, high-level features... More channels, advanced features First, a channel attention sub-component is used to infer a one-dimensional channel attention graph. Then advanced features According to the channel attention diagram Extracting noteworthy features, i.e., high-level features. With regard to the channel diagram The feature map obtained by performing element-wise multiplication is denoted as the feature map obtained by deconvolution. Its size and basic features Maintain consistency. A two-dimensional spatial attention map is obtained through processing by the spatial attention submodule. In order to extract key information from the feature space, features Attention diagram in two-dimensional space Performing element-wise multiplication yields a new feature, which is then combined with the lower-level feature. Perform the Hadamard product operation to extract low-level features. This contains noteworthy information. Simultaneously, the initial high-level features... Perform an upsampling operation to make it the same as the low-level features. The dimensions are kept consistent. Finally, element-wise addition is performed between this feature map and the feature map obtained after processing by the attention submodule, fusing the two features together. The result is then processed by the ReLU activation function and used as the input to the low-level MAF module. It's important to note that this method uses a 1×1 convolution to reduce the dimensionality of the features output from the last MAF module during computation, resulting in a feature map with 1 channel. Then, bilinear interpolation is used for a 4x upsampling operation to restore the feature map to the input size as the segmentation prediction result.

[0131] The mathematical model for the multi-attention fusion module is shown below:

[0132]

[0133] in:

[0134] Indicates the first The characteristics output by each MCST module Indicates from the Advanced features of the MCST module, ;

[0135] This indicates a deconvolution operation with a 4×4 kernel;

[0136] This represents a one-dimensional channel attention diagram generated by the SA sub-component;

[0137] This represents a two-dimensional spatial attention graph generated by the CA sub-component.

[0138] This represents a multilayer perceptron;

[0139] and These represent average pooling and max pooling operations, respectively.

[0140] This indicates a convolutional layer, and its subscript indicates the size of the convolutional kernel.

[0141] Represents the ReLU activation function;

[0142] Indicates a splicing operation;

[0143] This represents the Sigmoid activation function;

[0144] This indicates the Hadamard product operation.

[0145] 3.3 Loss Function

[0146] Implementing a medical image segmentation network requires selecting a suitable architecture and training it to optimize network weights, which necessitates the use of an appropriate loss function. Cross-entropy and Dice loss, along with their variations and combinations, are widely used in segmentation networks. However, these losses neglect high-level features or structures associated with the objects of interest, such as their shape or topology. They also penalize all errors equally, regardless of their nature.

[0147] Compared to natural scenes, medical images have their own unique characteristics, with much less variability in the shape and appearance of objects. Therefore, predictive models obtained using inappropriate loss functions may exhibit anatomical anomalies, such as holes or cavities near organ boundaries, or high inaccuracies. Choosing a suitable loss function can not only improve model performance but also accelerate model convergence.

[0148] Dice loss is a loss function designed based on the Dice coefficient. Dice loss is more effective in dealing with imbalanced samples of different classes in experimental data. Its mathematical definition is formula (6):

[0149]

[0150] Where GT is the true segmentation result and P is the predicted segmentation result.

[0151] The binary cross-entropy (BCE) loss function has a relatively smooth loss curve and converges quickly, but it is susceptible to class imbalance. This paper presents a variant of the binary cross-entropy loss function to mitigate the problem of imbalance between different classes in experimental data.

[0152] The weighted BCE loss function is defined in formula (7):

[0153]

[0154] in:

[0155] The value is 0 or 1, indicating that the label is positive or negative;

[0156] and This represents the width and height of the slice;

[0157] This represents the true value at row i and column j in the image pixel value matrix;

[0158] This represents the predicted value at row i and column j in the image pixel value matrix;

[0159] Represents the parameters in the model;

[0160] It is the predicted probability;

[0161] This represents the weight of the corresponding pixel.

[0162] The loss function used in this method is defined as a weighted sum of Dice loss and binary cross-entropy (BCE) loss, and its mathematical definition is given in formula (8):

[0163]

[0164] Step 4) Transfer learning strategy;

[0165] In the field of deep learning-based medical image processing, transfer learning is also an effective method to alleviate data shortages. This strategy has been widely applied to lung CT image segmentation tasks. The main idea of ​​transfer learning can be summarized as: learning knowledge related to the current task from other related tasks, and transferring the learned knowledge to the target task through fine-tuning or model pre-training. Currently, there are many backbone networks trained using large natural image datasets (such as ImageNet), such as ResNet and VGGNet. Researchers have found that transferring these trained parameters to the medical image processing domain is effective and can improve the performance of deep learning models for medical image segmentation tasks. Generally, the higher the similarity between the source and target domains, the better the effect of transfer learning. By analyzing the characteristics of lung CT images, it was found that the existing LIDC-IDRI dataset is suitable as a pre-training set for this model.

[0166] This method uses a two-stage cross-domain transfer learning strategy to train MCSTU-Net.

[0167] First, the encoder of the MCSTU-Net model consists of the first four residual blocks of the ResNet-34 network, which has already been pre-trained on ImageNet. During model initialization, the MCSTU-Net decoder first loads the weight values ​​of the first four feature extraction modules of ResNet-34. This stage of transfer learning is model-level transfer learning, which has the advantages of being simple to use, saving pre-training time, and improving model performance.

[0168] Then, the MCSTU-Net model was pre-trained using the lung nodule dataset, and the pre-trained model parameters were saved. Images in the ImageNet database are all natural images, which differ significantly from medical images. Medical images often exhibit low contrast at organ and tissue boundaries and class imbalance. In CT images, there is a very high similarity between features on pneumonia lesions and lung nodules. This stage of our paper utilizes data-level transfer learning to help the model transition from the natural image dataset to the pneumonia dataset.

[0169] Finally, the model parameters trained after the second stage of transfer learning are loaded, and the MCSTU-Net model is trained using the pneumonia dataset.

[0170] Step 5) Training strategy;

[0171] The general experimental setup for this method is as follows: the initial input image resolution is set to 384×384 pixels. The Adam optimizer is used to update the model parameters, with a momentum value of 0.9. The initial learning rate is 0.0001, and training is performed for 100 epochs. Every 50 epochs, the learning rate is reduced to 0.1 times its original value, i.e., the learning rate becomes 0.00001. The batch size is set to 10, and the patch size is 4×4. The deep learning framework used is PyTorch 2.0, and training is performed on an NVIDIA GeForce RTX 3090 GPU.

[0172] First, the MCSTU-Net model is pre-trained using the lung nodule dataset. At this stage, the MCSTU-Net decoder loads the parameters trained on ImageNet using ResNet-34. It's worth noting that during transfer learning training on the lung nodule dataset, the slices in the dataset are divided into training and validation sets in an 8:2 ratio. The validation set is used to search for the best model and save the optimal model parameters. Then, the MCSTU-Net model is trained on the pneumonia dataset. In the initial stage, the parameters pre-trained on the lung nodule dataset are loaded. Finally, the performance of the MCSTU-Net model is tested using their respective test sets.

[0173] Step 6) Evaluation indicators;

[0174] When evaluating the performance of network models used for medical image segmentation, the similarity between the predicted and actual results is typically used. Four labels are defined: True Negative (TN), False Positive (FP), False Negative (FN), and True Positive (TP). A True Positive indicates that both the predicted and actual results show overlapping areas of normal tissue; a True Negative indicates that both the predicted and actual results show areas of diseased tissue; a False Positive indicates that the predicted result shows normal tissue while the actual result shows diseased tissue; and a False Negative indicates that the predicted result shows diseased tissue while the actual result shows normal tissue. This method uses four commonly used evaluation metrics to compare the performance of the methods and models involved in the experiment: DICE similarity coefficient, sensitivity, specificity, and positive prediction rate. By calculating these evaluation metrics, our method is compared with state-of-the-art methods for lung CT image segmentation to evaluate its performance.

[0175] The Dice Similarity Coefficient (DSC) is commonly used to measure the similarity between the actual and predicted results. Its specific definition is given in formula (9). It is defined as the proportion of twice the number of correctly segmented lesions in the actual and predicted results, with a value between 0 and 1. The closer the value is to 1, the better the result.

[0176]

[0177] Sensitivity (SEN) is also known as recall. It is the percentage of infected tissue that is correctly segmented relative to the actual lesion tissue. Its formula is shown in (10).

[0178]

[0179] Specificity (SPE) represents the percentage of normal tissue that is correctly segmented. Its formula is shown in (11).

[0180]

[0181] The Positive Predictive Value (PPV) reflects the percentage of correctly segmented infected region pixels out of all predictions. Its formula is shown in (12).

[0182]

[0183] Step 7) Use the pre-trained network model;

[0184] Save the weight values ​​of the trained MCSTU-Net network model. When using it, build the network model and load the weight values ​​obtained after training. Input the original lung CT image of the pneumonia patient, and finally get the segmented image.

[0185] The accompanying drawings of the embodiments disclosed in this invention only involve structures relevant to the embodiments of this disclosure. However, the above description is merely a preferred embodiment of the present invention, and it can be fully applied to various fields suitable for the present invention. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A lung CT image segmentation method based on Transformer and convolutional neural networks, characterized in that, Includes the following steps: First, input the dataset; Input the LIDC-IDRI dataset of lung nodules CT scans; Input a dataset of labeled lung CT images of pneumonia patients, denoted as Pneumonia CT-Mask; Input the lung CT images of the pneumonia patient to be segmented, denoted as pneumonia CT; Secondly, data augmentation and data preprocessing; Furthermore, construct a network model; The segmentation model MCSTU-Net is constructed. MCSTU-Net is a U-shaped network consisting of an encoder, a decoder, and skip connection modules. The encoder of MCSTU-Net consists of the first four stages of ResNet-34, namely the first four residual network blocks (RB) of ResNet-34. An MCST module, a fused CNN and Swing Transformer module, is added to the skip connection part of MCSTU-Net. The decoder of MCSTU-Net consists of three cascaded modules that utilize the attention mechanism, namely the Multi-Attention Fusion (MAF) module. The images used to train the model are first input to the encoder. The ResNet-34 encoder extracts hierarchical features at different stages using a series of convolutional layers and successive downsampling layers. These hierarchical features have different receptive fields. Then, the features from different levels of the encoder are input to the MCST module, which uses different calculation methods to obtain information from different perspectives. Skip connection branches can compensate for feature values ​​lost during encoder downsampling and calculations in other branches of this module. Hybrid dilated convolution branches further extract contextual information by expanding the receptive field. The Transformer branch is used to model long-range dependencies, and this information is fused to obtain richer feature information. In the decoder part, based on the height and width of the feature information, smaller features are called low-level features, and relatively larger ones are called high-level features. Each MAF module receives feature information at two different scales. This module not only utilizes the channel attention mechanism but also the spatial attention mechanism. Through the attention mechanism, this module enhances the more noteworthy feature information. The features output by the last MAF module in the calculation process are reduced in dimensionality using a 1×1 convolution to obtain a feature map with 1 channel. Then, a 4x upsampling operation is performed using bilinear interpolation to restore the feature map to the input size as the segmentation prediction result. Building the MCST module: The MCST module is used to model long-range dependencies and acquire local information with a large receptive field, fusing rich feature information to further enrich the feature information. An MCST module consists of three parallel branches: a hybrid dilated convolution branch, a Swing Transformer branch, and a skip connection branch. Each branch starts with a 1×1 convolutional layer, reducing the input to half the channels of the previous layer. The hybrid dilated convolution branch in the MCST module obtains a larger receptive field through dilated convolution operations, the Swing Transformer branch uses Swing Transformer blocks to model long-range dependencies, and the skip connection branch prevents model degradation. The rich feature information obtained in different ways is fused and output to the corresponding layer of the decoder. The dilation rate settings must also meet the following requirements: the dilation rates of cascaded dilated convolutional layers cannot have a common divisor greater than 1; the input image size for the MCSTU-Net model is 384×384 pixels, the dilation rates of the dilated convolutional layers are 2, 3, and 5 respectively, and their convolutional kernels are all 3×3; the feature map processed by the hybrid dilated convolutional branches is denoted as... ; The specific operation process of the Swin Transformer branch is as follows: A feature image input from the encoder is first passed through a convolutional layer to reduce its dimensionality. Then, a patching layer is used to segment the input feature image into non-overlapping patches. This branch requires sequence embedding, and the patch size is 4×4. Through this partitioning method, the feature dimension of each patch becomes 4×4×C = 16C. A linear embedding layer projects the original features to an arbitrary dimension, denoted as V. The dimensions of the MCST modules are set to 128, 256, 512, and 1024 respectively. The resolution is... A V-dimensional tokenized patch is input into two consecutive Swing Transformer blocks for representation learning, where the feature dimension and resolution remain unchanged. The patch expansion layer regenerates a map upsampled to 4 times the resolution based on the feature information of adjacent dimensions, and then restores the resolution of the feature map to the input resolution. The feature map obtained after processing by the Swing Transformer branch is denoted as... ; The feature map processed by a skip connection branch is denoted as... The three features with the same spatial resolution but different key information obtained from the three branches are concatenated using a concatenation operation. Then, they are passed through a convolutional layer with a kernel size of 3×3. The purpose of this convolutional layer is to reduce the number of channels to one-third of the original without changing the spatial resolution of the feature map. The final output is a feature map with rich information. Build the MAF module; Each MAF module receives feature information at two different scales. In the decoder, based on the height and width of the feature information, the smaller feature information is called low-level feature, and the relatively larger one is called high-level feature. To refine this feature information, a multi-attention fusion module (MAF) is designed. The MAF module contains a channel attention (CA) submodule and a spatial attention (SA) submodule, which are connected together in a cascaded structure. In these two attention submodules, average pooling and max pooling operations are used simultaneously to obtain different feature information from different perspectives. The operation flow of the MAF module is as follows: Compared to low-level features, high-level features... More channels, advanced features First, a channel attention sub-component is used to infer a one-dimensional channel attention graph. Then advanced features According to the channel attention diagram Extracting noteworthy features, i.e., high-level features. With regard to the channel diagram The feature map obtained by performing element-wise multiplication is denoted as the feature map obtained by deconvolution. Its size and basic features Maintain consistency. A two-dimensional spatial attention map is obtained through processing by the spatial attention submodule. In order to extract key information from the feature space, features Attention diagram in two-dimensional space Performing element-wise multiplication yields a new feature, which is then combined with the lower-level feature. Perform the Hadamard product operation to extract low-level features. Information worth noting; at the same time, initial high-level features Perform an upsampling operation to make it the same as the low-level features. The size is kept consistent. Finally, element-wise addition is performed between the feature map obtained after processing by the attention submodule to fuse the two features together. The result is then processed by the ReLU activation function and used as the input of the low-level MAF module. It should be noted that this method uses a 1×1 convolution to reduce the dimensionality of the features output by the last MAF module in the calculation process, resulting in a feature map with 1 channel. Then, it uses bilinear interpolation to perform a 4x upsampling operation to restore the feature map to the input size as the segmentation prediction result. Choose a loss function; the loss function is a weighted sum of the Dice loss and the binary cross-entropy loss (BCE). Finally, the pre-trained MCSTU-Net network model is used; Save the weight values ​​of the trained MCSTU-Net network model. When using it, build the network model and load the weight values ​​obtained after training. Input the original lung CT image of the patient with lung disease, and finally get the segmented image.

2. The lung CT image segmentation method based on Transformer and convolutional neural network as described in claim 1, characterized in that, The data augmentation and data preprocessing specifically include: Step 1: Generate axial 2D slices and their corresponding segmented images based on the original 3D CT scan and annotation information; Step 2: Crop the images generated in Step 1 to include the lung region and the corresponding segmented images. Then, improve the dataset by cropping and enlarging to make it suitable for deep learning models. Remove slices from the LIDC-IDRI dataset that do not contain lung nodules and remove some slices from the pneumonia dataset Pneumonia CT-Mask that do not contain pneumonia lung lesions. Step 3: All images generated in Step 2 are resized to 384×384 pixels and normalized to avoid adverse effects on training from individual abnormal samples; first, the grayscale values ​​of the original images are unified to between 0 and 1, then multiplied by 255 to normalize the pixel values ​​to between 0 and 255, and these images are saved in PNG format; the lung nodule dataset is divided into training and validation sets in an 8:2 ratio, and the pneumonia dataset is divided into training, validation, and test sets in a 6:2:2 ratio.

3. The lung CT image segmentation method based on Transformer and convolutional neural network as described in claim 2, characterized in that, The mathematical expression of the fused CNN and Swing Transformer modules is Equation (1); in: Indicates a splicing operation; It is the ReLU activation function; It is a convolution operation with a 3×3 kernel; The mathematical model for the multi-attention fusion module is shown below: in: Indicates the first The characteristics output by each MCST module Indicates from the Advanced features of the MCST module, ; This indicates a deconvolution operation with a 4×4 kernel; This represents a one-dimensional channel attention diagram generated by the SA sub-component; This represents a two-dimensional spatial attention graph generated by the CA sub-component. Represents a multilayer perceptron; and These represent average pooling and max pooling operations, respectively. This indicates a convolutional layer, and its subscript indicates the size of the convolutional kernel. Represents the ReLU activation function; Indicates a splicing operation; This represents the Sigmoid activation function; This indicates the Hadamard product operation.

4. The lung CT image segmentation method based on Transformer and convolutional neural network as described in claim 3, characterized in that, The mathematical definition of Dice loss is formula (6): Where GT is the true segmentation result and P is the predicted segmentation result; The weighted binary cross-entropy loss function (BCE) is defined in formula (7): in: The value is 0 or 1, indicating that the label is positive or negative; and This represents the width and height of the slice; This represents the true value at row i and column j in the image pixel value matrix; This represents the predicted value at row i and column j in the image pixel value matrix; Represents the parameters in the model; It is the predicted probability; This indicates the corresponding pixel weight; The loss function is defined as the weighted sum of the Dice loss and the binary cross-entropy loss (BCE), and its mathematical definition is given in formula (8): 。 5. The lung CT image segmentation method based on Transformer and convolutional neural network as described in claim 4, characterized in that, The MCSTU-Net network model is trained using transfer learning and training strategies, as detailed below; The LIDC-IDRI dataset was used as the pre-training set for the MCSTU-Net model. MCSTU-Net is trained using a two-stage cross-domain transfer learning strategy. First, the encoder of the MCSTU-Net model is composed of the first four residual blocks of the ResNet-34 network. Its design team has pre-trained the ResNet-34 model on ImageNet. When the MCSTU-Net model is initialized, the MCSTU-Net decoder will first load the weight values ​​of the first four feature extraction modules of ResNet-34. Then, the MCSTU-Net model was pre-trained using the lung nodule dataset, and the pre-trained model parameters were saved. The images in the ImageNet database are all natural images, which are quite different from medical images. Medical images have low contrast at the boundaries of organs and tissues and are class imbalanced. In CT images, there is a high degree of similarity between the features on lung lesions and lung nodules. This stage of data-level transfer learning helped the MCSTU-Net model transition from the natural image dataset to the pneumonia dataset. Finally, the parameters of the MCSTU-Net model trained by the second stage of transfer learning are loaded, and the MCSTU-Net model is trained using the pneumonia dataset. Training strategies; The initial input image resolution was set to 384×384 pixels; the Adam optimizer was used to update the parameters of the MCSTU-Net model, and the momentum was set to 0.9; the initial learning rate was 0.0001, and the training lasted for 100 epochs. Every 50 epochs, the learning rate was reduced to 0.1 times its original value, i.e., the learning rate became 0.00001; the batch size was set to 10, and the patch size was 4×4 pixels. First, the MCSTU-Net model is pre-trained using the lung nodule dataset. At this point, the MCSTU-Net decoder will first load the parameters trained on ImageNet by ResNet-34. It should be noted that when training the lung nodule dataset using transfer learning, the slices in the lung nodule dataset are divided into training and validation sets in an 8:2 ratio. The validation set is used to search for the best model and save the best model parameters. Then, the MCSTU-Net model is trained using the pneumonia dataset. In the initial stage, the parameters pre-trained on the lung nodule dataset are loaded. Finally, the model performance is tested using their respective test sets.

6. The lung CT image segmentation method based on Transformer and convolutional neural network as described in claim 5, characterized in that, Four evaluation metrics were used to measure the performance of the MCSTU-Net model; The performance of network models used for medical image segmentation is measured by the similarity between the predicted results and the actual results; four labels are defined: true negative (TN), false positive (FP), false negative (FN), and true positive (TP). A true positive indicates an area where both the predicted and actual results are normal tissue overlapping; a true negative indicates an area where both the predicted and actual results are diseased tissue; a false positive indicates an area where the predicted result is normal tissue but the actual result is diseased tissue; and a false negative indicates an area where the predicted result is diseased tissue but the actual result is normal tissue. Four commonly used evaluation metrics were used to compare the performance of the methods and models involved in the experiment: DICE similarity coefficient, sensitivity, specificity, and positive predictive value. Evaluation metrics were calculated. The Dice similarity coefficient (DSC) is used to measure the similarity between the actual and predicted results. Sensitivity (SEN) is also known as recall; it is the percentage of infected tissue that is correctly segmented relative to the actual lesion tissue, as shown in formula (10). Specific SPE represents the percentage of normal tissue that is correctly segmented, as shown in formula (11). The positive prediction value (PPV) reflects the percentage of correctly segmented infected region pixels out of all predictions; its formula is shown in (12). 。

Images