Lung nodule segmentation algorithm based on transunet

Abstract

This invention relates to the field of segmentation algorithm technology, and in particular to a lung nodule segmentation algorithm based on TransUNet. The steps include: replacing the Transformer layers in the TransUNet network with MaxViT networks to form an MM-TransUNet network as the segmentation model; training the segmentation model using the MM-TransUNet network; after each training round, saving the segmentation model parameters for that round; comparing the detection results obtained by the segmentation model with the actual results to calculate the loss; adjusting the parameters of the segmentation model based on the loss; reading the optimal parameters saved during the training phase; substituting the optimal parameters into the segmentation model to obtain the optimal lung nodule segmentation model; applying the MM-TransUNet network as the segmentation model can effectively improve the segmentation accuracy of lung nodules; MM-TransUNet combines residual mechanisms, CNNs, and multi-axial self-attention mechanisms as an encoder module, playing a role in more completely extracting lung nodule features.

Description

Technical Field

[0001] This invention relates to the field of segmentation algorithm technology, and in particular to a lung nodule segmentation algorithm based on TransUNet. Background Technology

[0002] Studies have shown that pulmonary nodules are closely related to lung cancer and are one of the early symptoms of lung cancer. Early diagnosis and treatment of pulmonary nodules can greatly reduce the mortality rate of lung cancer. Therefore, implementing a high-precision algorithm for automatic pulmonary nodule segmentation has valuable clinical value and practical significance. However, due to the random location, varying size, and different shapes of pulmonary nodules, the segmentation accuracy of existing deep learning-based pulmonary nodule segmentation algorithms is still not high enough.

[0003] U-Net is one of the most classic models in the field of medical image segmentation. It is called U-Net because of its U-shaped symmetrical structure. The U-Net network is based on CNN and adopts an encoder-decoder structure, which includes convolutional layers and pooling layers on the left and upsampling layers on the right, as well as a skip connection design connecting the two ends. Compared with other network structures (such as FCN), U-Net does not need to perform supervision and loss calculation directly on high-level feature maps, which helps to improve the segmentation accuracy of the model.

[0004] Currently, most lung nodule segmentation models still use the U-Net architecture as their basic structure. This involves using an encoder to extract high-level features from the image, a decoder to upsample these features to obtain the segmentation result, and skip connections to fuse features from different levels in the encoder and decoder, ultimately achieving automated lung nodule segmentation. However, U-Net's encoder is too simple, limiting its feature extraction capabilities and making it unable to adapt to the random location and size variations of lung nodules in images. Furthermore, although U-Net's skip connections can connect high-level and low-level features, they still miss information at certain levels, causing the model to be unable to distinguish between relevant and irrelevant information. Therefore, although U-Net has become a highly representative work in the field of medical image segmentation, it still has significant room for improvement. Summary of the Invention

[0005] The present invention aims to solve the above problems, thereby providing a TransUNet-based lung nodule segmentation algorithm that can accurately segment lung nodules and effectively identify image noise.

[0006] The invention solves the aforementioned problem by employing the following technical solution:

[0007] A lung nodule segmentation algorithm based on TransUNet, comprising the following steps:

[0008] S1: Obtain the lung CT image dataset and perform lung parenchyma segmentation, region of interest extraction, and data augmentation operations on the lung CT image dataset;

[0009] S2: Divide the lung CT image dataset after data augmentation into a training set and a test set;

[0010] S3: Construct a segmentation model based on the TransUNet network structure and train the segmentation model using the training set; after each round of training, save the segmentation model parameters for this round, compare the detection results obtained by the segmentation model with the actual results to calculate the loss, and adjust the parameters of the segmentation model according to the loss;

[0011] S4: Read the optimal parameters saved during the training phase, and substitute the optimal parameters into the segmentation model to obtain the optimal lung nodule segmentation model;

[0012] S5. The optimal lung nodule segmentation model is tested using a test set, and the model accuracy of the optimal lung nodule segmentation model is evaluated using evaluation metrics.

[0013] In step S3, the Transformer layer in the TransUNet network is replaced with the MaxViT network to form the MM-TransUNet network as a segmentation model. The specific process of training the segmentation model using the MM-TransUNet network is as follows:

[0014] S301: Input the images in the training set into the encoder, and first use the residual mechanism in conjunction with the convolutional neural network to extract shallow features from the input images layer by layer to extract shallow feature maps;

[0015] S302: The shallow feature map is processed by the MaxViT module to fully perceive the deep context information, further learn deep features, and extract the deep feature map of the image;

[0016] S303: Input the shallow feature map and deep feature map into the decoder, connect and upsample the shallow feature map and deep feature map, perform feature fusion layer by layer, and finally output the lung nodule segmentation image;

[0017] S304: Calculate the loss between the lung nodule segmentation image output in step S303 and the actual lung nodule segmentation image using the loss function, adjust the parameters in the MM-TransUNet network, update the MM-TransUNet network model, and return to step S301 for a new round of training until all images in the training set have been trained.

[0018] Preferably, in step S1, the specific steps for segmenting the lung parenchyma in the lung CT image dataset are as follows:

[0019] S101: The original CT image is binarized and segmented using the OTSU algorithm to obtain a binary image;

[0020] S102: The cavity in the lung parenchyma is filled using the flooding method to obtain a filled image; the noise and small air bubbles in the binarized image are removed using the opening operation to obtain a denoised image;

[0021] S103: Subtract the denoised image from the filled image to obtain a primary lung parenchyma mask image containing small connected regions;

[0022] S104: Opening operation removes small connected regions in the initial lung parenchyma mask image to obtain the final lung parenchyma mask image;

[0023] S105: Perform an AND operation between the final lung parenchyma mask image and the original CT image to obtain the lung parenchyma image.

[0024] Preferably, step S1 further includes,

[0025] S106: Extract the region of interest containing lung nodules from the lung parenchyma image; locate the lung nodules using the lung nodule location information in the dataset, and then crop the lung parenchyma image with the lung nodules as the center point to generate input image blocks;

[0026] S107: Expand the image source by randomly flipping or rotating the image patches. Use the expanded image patches and the original image patches together as the training and test datasets. Specifically, flipping involves randomly selecting image patches and flipping them horizontally or vertically, and the label images are also flipped accordingly. Specifically, rotating involves randomly selecting image patches and rotating them clockwise by 90°, 180°, or 270°, and the label images are also rotated accordingly.

[0027] Preferably, in step S301, at the beginning of the encoder, MM-TransUNet first uses a three-layer ResNet-50 network based on CNN and residual mechanism. The shallow feature map output by each layer is used as the input to the next layer. Each ResNet-50 network layer combines the input features directly with the output features through skip connections to form a residual block. The core idea of ​​the residual block can be represented as follows:

[0028] output = F(input) + input

[0029] Where F(*) represents a series of operations performed by the network on the input;

[0030] Then, the shallow feature map output by the third residual block is divided into blocks and positional encoding is added.

[0031] Preferably, in step S302, the shallow feature map after adding position encoding is processed by MaxViT module 12 times. The specific process of each MaxViT module processing is as follows:

[0032] The feature map is input into the MBConv block. After passing through a 1x1 convolutional layer and a 3x3 convolutional layer, the feature map is input into the channel attention module to enhance relevant features and weaken irrelevant features in the feature map, resulting in an enhanced feature map. Then, the enhanced feature map is convolved with the input feature map point by point to restore the dimension and output the enhanced feature map.

[0033] The enhanced feature map is input to the block attention processing module, which splits it into several complementary and overlapping image blocks. Then, a self-attention mechanism is applied to each split image block to achieve interaction within a local range. The image blocks that have completed local interaction are then input into a feedforward neural network for full connection so that the output feature map has the same size as the final enhanced feature map.

[0034] The feature map output by the block attention processing module is input into the table attention processing module. The input feature map is aggregated into different local grids in the form of global expansion. A self-attention mechanism is performed on the local space within each grid to achieve global interaction. Finally, all local grids are input into the feedforward neural network for full connection so that the output feature map has the same size as the final enhanced feature map.

[0035] Preferably, in step S303, the decoder in MM-TransUNet adopts a skip connection structure, and an MSCA attention mechanism is added to each skip connection.

[0036] The specific decoding steps of the decoder are as follows:

[0037] S3031: The shallow feature maps output by each layer of the three-layer ResNet-50 network are first processed by the multi-scale cross-axial attention mechanism (MSCA) to extract important features from the shallow feature maps. After processing by the MSCA attention mechanism, the first, second, and third layers of the ResNet-50 network output the first, second, and third important feature maps, respectively. The first, second, and third important feature maps are then input into the decoder.

[0038] S3032: The decoder integrates the important feature map of the third layer with the deep feature map using a 3x3 convolutional layer and performs linear activation using the ReLU function, and then upsamples to generate the initial segmentation image;

[0039] S3032: The important feature map of the second layer is integrated with the initial segmentation image using a 3x3 convolutional layer and linearly activated using the ReLU function. Then, it is upsampled to generate an intermediate segmentation image.

[0040] S3033: The first layer of important feature maps and the intermediate segmentation image are integrated using a 3x3 convolutional layer and linearly activated using the ReLU function, and then upsampled to generate the final segmentation image;

[0041] S3034: Upsample the final segmented image and then apply the ReLU function for linear activation to obtain the lung nodule segmentation image.

[0042] Preferably, in step S3031, the specific steps for applying the MSCA attention mechanism to process the shallow feature map are as follows: Input a shallow feature map F with shape H×W×C, and perform multi-scale convolution operations on the feature map in parallel along the X-axis and Y-axis respectively; the convolution kernels on the X-axis are set to 1×7, 1×11, and 1×21, and the convolution kernels on the Y-axis are set to 7×1, 11×1, and 21×1; after the multi-scale convolution operations, the convolution outputs of the input feature map F along the X-axis and Y-axis and F1 are obtained. x and F y Then calculate F. x and F y Cross-attention; for cross-attention on the X-axis, first, F... x As a key and value matrix, and F y As a query matrix, F is obtained. T For attention along the Y-axis, F... y As a key and value matrix, and F x As a query matrix, F is obtained. B Finally, F B F T By integrating F, we can obtain the first layer of important feature maps, the second layer of important feature maps, or the third layer of important feature maps.

[0043] Preferably, in step S304, when calculating the loss between the detection result and the actual result, the DiceBCE loss function is used. The DiceBCE loss function model is as follows:

[0044] L total =(1-α)L BCE +αL Dice

[0045] Where α is the weight parameter of the composite loss function, L BCE Let L be the binary cross-entropy loss function. Dice The Dice loss function;

[0046] L BCE The function expression is as follows:

[0047]

[0048] Where G(r,c) is the true value of the pixel at coordinate (r,c), and S(r,c) is the predicted value of the pixel at coordinate (r,c).

[0049] L Dice The function expression is as follows:

[0050]

[0051] Where X represents the true value and Y represents the segmentation result.

[0052] Preferably, in step S5, the evaluation indicators include Dice coefficient, MIoU coefficient, Recall coefficient and Precision coefficient.

[0053] The formula for the Dice coefficient is:

[0054]

[0055] The formula for the MioU coefficient is:

[0056]

[0057] The formula for the recall coefficient is:

[0058]

[0059] The formula for the Precision coefficient is as follows:

[0060]

[0061] In this context, TP represents a true example, FN represents a false negative, FP represents a false positive, and TN represents a true negative.

[0062] The present invention, which adopts the above technical solution, has the following prominent features compared with the prior art:

[0063] To address the issue of poor segmentation accuracy due to the varying shapes and sizes of lung nodules, the application of the MM-TransUNet network as a segmentation model can effectively improve the segmentation accuracy of lung nodules. MM-TransUNet combines residual mechanisms, CNNs, and multi-axial self-attention mechanisms as an encoder module, which plays a role in extracting lung nodule features more completely. At the same time, it improves the traditional skip connection module by adding a multi-scale cross-axial attention mechanism in the skip connections of each layer, thereby better connecting shallow feature maps and deep feature maps. Attached Figure Description

[0064] Figure 1 This is a flowchart of the training and testing process of the lung nodule segmentation algorithm based on TransUNet in this embodiment of the invention;

[0065] Figure 2 This is a flowchart of the lung parenchyma segmentation, region of interest extraction, and data augmentation operations performed on a lung CT image dataset in this embodiment of the invention.

[0066] Figure 3 This is a list of detailed information in the XML file in the embodiments of the present invention;

[0067] Figure 4 This is the overall network architecture diagram of the MM-TransUNet model in this embodiment of the invention;

[0068] Figure 5 This is an encoder architecture diagram in an embodiment of the present invention;

[0069] Figure 6 This is a structural diagram of the residual block in an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram of the MaxViT module processing feature maps in an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the block attention processing module and the table attention processing module processing feature maps in an embodiment of the present invention;

[0072] Figure 9 This is a diagram of the decoder architecture in this invention;

[0073] Figure 10 This is an architecture diagram of the MSCA attention mechanism in this invention when processing feature maps;

[0074] Figure 11 This is a graph showing how various metrics of the MM-TransUNet model change with the number of training rounds during the training phase.

[0075] Figure 12 This is a comparison chart of the segmentation results of the MM-TransUNet model and other open-source models for lung nodules, using samples 1, 2, 3, and 4.

[0076] in, Figure 3 In the diagram, Downsmple represents the downsampling operation, and Upsmple represents the upsampling operation. Detailed Implementation

[0077] The present invention will be further described below with reference to embodiments, which are intended only to provide a better understanding of the invention. Therefore, the examples given do not limit the scope of protection of the present invention.

[0078] like Figure 1 , Figure 4 As shown, this embodiment discloses a lung nodule segmentation algorithm based on TransUNet, the steps of which include:

[0079] S1: Obtain the lung CT image dataset and perform lung parenchyma segmentation, region of interest extraction, and data augmentation operations on the lung CT image dataset.

[0080] In this embodiment, the lung CT image dataset uses the LIDC-IDRI dataset, which contains lung CT data from 1018 research cases. Each case's CT data consists of several CT images, each with a size of 512×512. The lung nodules on each CT image are labeled by four experienced radiologists. To avoid errors caused by forced consensus, the labeling and diagnosis work of this database is divided into two stages: first, the four physicians independently label the lung nodules; then, each physician reviews the labeling results of the other physicians.

[0081] The LIDC-IDRI dataset stores contour, category, and other feature information of lung nodules in XML files. Detailed information in the XML files includes, for example, [details of the XML files would be inserted here]. Figure 3 As shown in Table 1, the detailed annotation of lung nodules in the LIDC-IDRI dataset is as follows.

[0082]

[0083] Table 1. Detailed information on lung nodule annotation in the LIDC-IDRI dataset.

[0084] Since the lung nodule information in LIDC-IDRI is annotated by physicians, and manual annotation inevitably involves a certain degree of subjectivity, to ensure the accuracy and reliability of the data used, the experimental work in this embodiment is limited to samples that have been jointly identified and annotated as lung nodules by four physicians, totaling 928 cases. To make full use of these samples, each slice in the 3D image containing lung nodules is treated as an independent sample, totaling 4946 slices. To improve the quality of the input images, thereby improving the model's data processing effect and efficiency, a series of preprocessing operations are required on the dataset before applying it to the model. In addition to the lung parenchyma, CT images in the LIDC-IDRI database also contain other trunk information, such as lung organs and pulmonary vessels; therefore, to avoid other irrelevant information affecting lung nodule segmentation and to improve the accuracy and efficiency of lung nodule segmentation, preprocessing is necessary.

[0085] like Figure 2 As shown, the preprocessing operation flow is as follows;

[0086] S101: The original CT image is binarized using the OTSU algorithm to obtain a binarized image.

[0087] S102: Use the flooding method to fill the cavities in the lung parenchyma to obtain a filled image; use the opening operation to remove noise and small bubbles from the binarized image to obtain a denoised image.

[0088] S103: Subtract the denoised image from the filled image to obtain a primary lung parenchyma mask image containing small connected regions.

[0089] S104: The opening operation removes small connected regions in the initial lung parenchyma mask image to obtain the final lung parenchyma mask image.

[0090] S105: Perform an AND operation between the final lung parenchyma mask image and the original CT image to obtain the lung parenchyma image.

[0091] S106: The lung parenchyma image obtained in step S105 has a resolution of 512×512 pixels. Considering that the area occupied by lung nodules in the lung region is very small, directly inputting the entire image into the model for processing will result in a very low signal-to-noise ratio, leading to class-polarity imbalance and affecting the lung nodule segmentation accuracy. Therefore, it is necessary to extract the region of interest containing lung nodules from the lung parenchyma image. By reading the information provided by the XML file in the LIDC-IDRI dataset, the location of the lung nodules is located. Then, the original image is cropped using the lung nodule as the center point to generate an image patch with a size of 64×64 pixels. This image patch will be used as the input data for the segmentation model to facilitate subsequent analysis and processing.

[0092] S107: To improve the generalization and robustness of the lung nodule segmentation model and avoid overfitting, the data needs to be augmented and expanded. The original data is flipped and rotated, and the expanded images are used together with the original images as input data for training and testing the segmentation model. Flipping specifically involves randomly selecting image blocks and flipping them horizontally or vertically, and the label images are also flipped accordingly. Rotation specifically involves randomly selecting image blocks and rotating them clockwise by 90°, 180°, or 270°, and the label images are also rotated accordingly.

[0093] S2: Divide the lung CT image dataset after data augmentation into training and test sets.

[0094] S3: Construct a segmentation model based on the TransUNet network structure and train the segmentation model using the training set; after each round of training, save the segmentation model parameters for this round, compare the detection results obtained by the segmentation model with the actual results to calculate the loss, and adjust the parameters of the segmentation model according to the loss.

[0095] In step S3, the Transformer layer in the TransUNet network is replaced with the MaxViT network to form the MM-TransUNet network as the segmentation model. The specific process of training the segmentation model using the MM-TransUNet network is as follows:

[0096] S301: Input the images from the training set into the encoder. The encoder architecture is as follows: Figure 5 As shown, the residual mechanism is first used in conjunction with the convolutional neural network (CNN) to extract shallow features from the input image layer by layer, and the shallow feature map is extracted.

[0097] Preferably, in step S301, MM-TransUNet adds a residual mechanism to the convolutional neural network and places it in the first three layers of the encoder, thereby improving the model's learning ability and accelerating its convergence speed, while also avoiding gradient explosion or gradient vanishing. At the beginning of the encoder, a three-layer ResNet-50 network based on CNN and the residual mechanism is used first, with the shallow feature map output from each layer serving as the input to the next layer. The residual block structure of the ResNet-50 network is as follows: Figure 6 As shown, the ResNet-50 network is one of the variants of the ResNet network. Compared with the traditional CNN network, ResNet adds a residual mechanism, which enables the model to better solve the problems of gradient vanishing and gradient explosion in neural networks, and provides feasibility for the development of neural networks to higher and wider.

[0098] The core idea of ​​ResNet is to combine input features directly with output features through skip connections to form residual blocks. By setting skip connections, the network can learn residual information instead of relying entirely on information passed from layer to layer. At the same time, the residual mechanism can better realize gradient propagation, making the network more stable and faster during the training phase. By setting the residual learning mechanism, the learning and expressive capabilities of the network can be improved, the problems of gradient vanishing and gradient exploding can be solved, and the convergence speed of the network can be accelerated.

[0099] The core idea of ​​residual blocks can be expressed as a formula.

[0100] output = F(input) + input

[0101] Where F(*) represents a series of operations performed by the network on the input; the residual mechanism enables the model to better solve the problems of gradient vanishing and gradient explosion.

[0102] Then, the shallow feature map output by the third residual block is divided into blocks and positional encoding is added.

[0103] S302: The shallow feature map is processed by the MaxViT module (MaxViT Block) to fully perceive the deep context information, further learn deep features, and extract the deep feature map of the image.

[0104] MaxViT is a hierarchical Transformer architecture that cleverly combines attention mechanisms with convolutional operations and introduces an independent multi-axis attention module. The multi-axis attention module consists of two parts: a local attention mechanism that focuses on local features and an extended global attention mechanism that can capture global contextual information. This design gives the MaxViT model the ability to handle perceptual tasks with linear complexity. By setting local and global attention, MaxViT achieves a form of local and global interaction, thus performing well in image segmentation and object detection.

[0105] Furthermore, in step S302, the shallow feature map after adding position encoding is processed by the MaxViT module 12 times, such as... Figure 7 As shown, the MaxViT module adopts a serial structure, which offers good flexibility and simplicity. The modules can be easily disassembled or combined, providing better scalability. The specific process of each MaxViT module processing step is as follows:

[0106] S3021: The feature map is input into the MBConv module, which is a moving flippable convolutional block that combines attention and residual structure. The feature map is convolved through a 1x1 convolutional layer. The input feature map, after being processed by convolutional dimensionality upscaling, will be processed by depthwise separable convolution (3x3 convolutional layer). The input channel attention module (SE) enhances relevant features and weakens irrelevant features in the feature map to obtain an enhanced feature map. Then, the enhanced feature map is convolved with the input feature map point by point to restore the dimension and output the enhanced feature map.

[0107] S3022: As Figure 7 , Figure 8 As shown, the enhanced feature map is input into the Block Attention module. The Block Attention module divides the input feature map into complementary and overlapping image blocks. The splitting operation can be represented as follows:

[0108]

[0109] After splitting the input image into several complementary and overlapping image blocks, a self-attention mechanism (Block-SA) is applied to each split image block P×P to achieve interaction within a local range. Then, the image blocks that have completed local interaction are input into a feedforward neural network (FNN) for full connection so that the output feature map has the same size as the final enhanced feature map.

[0110] S3023: The feature map output by the block attention processing module is input into the grid attention processing module. The grid attention processing module aggregates the input image into different local grids in the form of global expansion. The specific operation can be represented as follows:

[0111]

[0112] The input feature maps are aggregated into different local grids in the form of a global expansion. For each grid (size...), ... The local space within the local area performs a self-attention mechanism (Grid-SA) to achieve global interaction. Finally, all local grids are input into the feedforward neural network and fully connected so that the output feature map has the same size as the final enhanced feature map.

[0113] S303: Input the shallow feature map and deep feature map into the decoder, connect and upsample the shallow feature map and deep feature map, perform feature fusion layer by layer, and finally output the lung nodule segmentation image;

[0114] Preferably, in step S303, the decoder in MM-TransUNet adopts a skip connection structure, and an MSCA attention mechanism is added to each skip connection.

[0115] like Figure 9 The specific decoding steps of the decoder are as follows:

[0116] S3031: The shallow feature maps output by each layer in the three-layer ResNet-50 network are first processed by the multi-scale cross-axial attention mechanism, namely the MSCA attention mechanism, to extract important features from the shallow feature maps. The MSCA attention mechanism is a method that embeds multi-scale features into axial attention and combines horizontal and vertical axial attention, aiming to better segment lesion regions or organs with various sizes and shapes.

[0117] After processing by the MSCA attention mechanism, the first, second, and third layers of the ResNet-50 network output the first, second, and third important feature maps, respectively. These important feature maps are then input into the decoder.

[0118] like Figure 10In step S3031, the specific steps for applying the MSCA attention mechanism to process shallow feature maps are as follows: Input a shallow feature map F with shape H×W×C, and perform multi-scale convolution operations on the feature map in parallel along the X-axis and Y-axis respectively; the convolution kernels on the X-axis are set to 1×7, 1×11, and 1×21, and the convolution kernels on the Y-axis are set to 7×1, 11×1, and 21×1; after the multi-scale convolution operations, the convolution outputs F and F along the X-axis and Y-axis are obtained. x and F y ;

[0119]

[0120]

[0121] Then calculate F. x and F y Cross-attention; for cross-attention on the X-axis, first, F... x As a key and value matrix, and F y As a query matrix, F is obtained. T ;

[0122] F T =MHCA y (F y ,F x ,F x )

[0123] Among them, MHCA y (*,*,*) indicates multi-head cross attention along the X-axis.

[0124] As for attention on the Y-axis, F y As a key and value matrix, and F x As a query matrix, F is obtained. B ;

[0125] F B =MHCA x (F x ,F y ,F y )

[0126] Among them, MHCA x (*,*,*) indicates multi-head cross attention along the Y-axis.

[0127] Finally, F B F T By integrating F, we obtain either a first-layer important feature map, a second-layer important feature map, or a third-layer important feature map. The integration process formula is as follows:

[0128] F out =Conv1×1 (F T )+Conv 1×1 (F B )+F

[0129] By adding the MSCA attention mechanism to the feature maps in skip connections, the neural network's ability to perceive specific channels and spatial dimensions can be enhanced, enabling it to extract richer and more useful features more effectively, thereby strengthening the function of skip connections.

[0130] S3032: The decoder integrates the important feature map of the third layer with the deep feature map using a 3x3 convolutional layer and performs linear activation using the ReLU function, and then upsamples to generate the initial segmentation image;

[0131] S3032: The important feature map of the second layer is integrated with the initial segmentation image using a 3x3 convolutional layer and linearly activated using the ReLU function. Then, it is upsampled to generate an intermediate segmentation image.

[0132] S3033: The first layer of important feature maps and the intermediate segmentation image are integrated using a 3x3 convolutional layer and linearly activated using the ReLU function, and then upsampled to generate the final segmentation image;

[0133] S3034: Upsample the final segmented image and then apply the ReLU function for linear activation to obtain the lung nodule segmentation image.

[0134] S304: Calculate the loss between the lung nodule segmentation image output in step S303 and the actual lung nodule segmentation image using the loss function, adjust the parameters in the MM-TransUNet network, update the MM-TransUNet network model, and return to step S301 for a new round of training until all images in the training set have been trained.

[0135] Furthermore, considering that the binary cross-entropy loss function is the most common loss function in binary semantic segmentation, using it alone will lead to insufficient ability of the model to learn features and poor segmentation accuracy; the Dice loss function is not conducive to model training. When the target region contained in the image is small, even if only a few pixels are predicted incorrectly, the loss function value may fluctuate significantly. This will not only affect the stability of the model during training, but may also have a negative impact on the model's convergence performance.

[0136] To fully utilize the characteristics of the binary cross-entropy loss function and the Dice loss function, this embodiment combines them to form a new loss function, DiceBCE. The DiceBCE loss function, through a linear combination of the Dice and BCE loss functions, possesses both the stability of backpropagation and the ability to focus more on the lung nodule region. Using the DiceBCE loss function to guide the model's learning can improve the accuracy of lung nodule segmentation while accelerating the convergence speed.

[0137] The DiceBCE loss function model is as follows:

[0138] L total =(1-α)L BCE +αL Dice

[0139] Where α is the weight parameter of the composite loss function, different values ​​of the weight parameter will affect the segmentation model's focus on different learning content, and thus affect the segmentation accuracy of lung nodules; L BCE Let L be the binary cross-entropy loss function. Dice The Dice loss function;

[0140] L BCE The function expression is as follows:

[0141]

[0142] Where G(r,c) is the true value of the pixel at coordinate (r,c), and S(r,c) is the predicted value of the pixel at coordinate (r,c).

[0143] L Dice The function expression is as follows:

[0144]

[0145] Where X represents the true value and Y represents the segmentation result.

[0146] S4: Read the optimal parameters saved during the training phase, and substitute the optimal parameters into the segmentation model to obtain the optimal lung nodule segmentation model;

[0147] S5. The optimal lung nodule segmentation model is tested using a test set, and the model accuracy of the optimal lung nodule segmentation model is evaluated using evaluation metrics.

[0148] In the field of medical image segmentation, researchers typically employ a series of evaluation metrics to assess and compare the segmentation performance of different models. These metrics are designed to closely align with the specific needs and objectives of the segmentation task, comprehensively reflecting the model's segmentation capabilities. Lung nodule segmentation is a pixel-level image analysis task, with the goal of accurately classifying each pixel. For the lung nodule segmentation task, this embodiment uses the following four evaluation metrics to assess the model's accuracy.

[0149] (1) Dice coefficient, which evaluates the degree of overlap between the model's predicted output and the true label. Its value ranges from [0,1]. The closer the value is to 1, the higher the degree of overlap between the model's lung nodule segmentation results and the true label. The formula for the Dice coefficient is:

[0150]

[0151] (2) MIoU coefficient, which is the ratio of the intersection area to the union area, and its value ranges from [0,1]. The closer the value is to 1, the better the segmentation effect; the formula for the MioU coefficient is,

[0152]

[0153] (3) Recall coefficient, which represents the proportion of lung nodules correctly detected by the model out of all real lung nodules; the formula for the recall coefficient is:

[0154]

[0155] (4) Precision coefficient, which represents the proportion of pixels predicted as lung nodules by the model that are also lung nodules in reality; the formula for the Precision coefficient is as follows.

[0156]

[0157] In this context, TP represents a true example, FN represents a false negative, FP represents a false positive, and TN represents a true negative.

[0158] In the specific experiment, this embodiment uses RMSprop as the model optimization algorithm, which is an algorithm that adaptively adjusts the learning rate to minimize the loss function and optimize the model parameters; the initial learning rate is set to 0.001, and the experimental details are shown in Table 2.

[0159]

[0160] Table 2 Experimental Details

[0161] As shown in Table 2, the initial learning rate was set to 0.001. Training with a relatively large initial learning rate aims to accelerate network convergence. An ExponentialLR learning rate scheduler was created, and the learning rate decay rate (gamma) was set to 0.9, ensuring that the learning rate is updated according to the set decay rate at the end of each epoch. Gradually reducing the learning rate during training allows for better model optimization. The exponential decay setting ensures that the learning rate gradually decreases in the later stages of training, facilitating more precise parameter tuning.

[0162] Furthermore, in order to eliminate the potential impact of differences in the experimental environment on the experimental results and to ensure the reliability of the experiment, this chapter adopted a uniform experimental environment setting for all experiments; the detailed hardware configuration of the experimental environment can be found in Table 3.

[0163]

[0164] Table 3 Hardware Parameters

[0165] The experiments in this chapter use the PyTorch framework to implement the proposed model. The PyTorch framework's source code is intuitive and easy to understand, and its design is concise and efficient. Therefore, PyTorch was chosen as the framework for building the model. The specific software parameters for the experiment are shown in Table 4.

[0166]

[0167]

[0168] Table 4 Software Parameters

[0169] To verify the effectiveness of the MM-TransUNet model in the lung nodule segmentation task and to demonstrate its ability to handle the poor segmentation accuracy caused by the different shapes and scales of lung nodules, this section uses MM-TransUNet to perform segmentation experiments on lung nodule images in the LIDC-IDRI dataset. Figure 11 The graph shows the changes in various metrics of MM-TransUNet during the training phase. As can be seen from the graph, the loss during the training phase gradually decreases with the increase of training epochs, eventually converging to a small value; while the Dice and IoU metrics, which represent the segmentation accuracy, gradually increase with the increase of epochs, and eventually tend to a constant value; MM-TransUNet will save the parameters of the epoch with the highest segmentation accuracy and load them during the testing phase; in the testing phase, the Dice coefficient of MM-TransUNet reached 85.82%, proving that it can perform the lung nodule segmentation task well.

[0170] Furthermore, to demonstrate the positive impact of the MaxViT module and the MSCA attention mechanism module in MM-TransUNet on the accuracy of lung nodule segmentation, this embodiment sets up corresponding ablation experiments to complete the effectiveness test of each module; the test results are shown in Table 5.

[0171]

[0172]

[0173] Table 5. Results of MM-TransUNet ablation experiments

[0174] Table 5 shows the lung nodule segmentation accuracy after adding different modules at different locations in the TransUNet model, based on the TransUNet model. These metrics were evaluated on the LIDC-IDRI dataset. Baseline Method 1 uses the TransUNet model. Compared to Baseline Method 1, integrating MaxViT, MSCA-1, MSCA-2, and MSCA-3 progressively into the baseline method improved all metrics to varying degrees. MSCA-1, MSCA-2, and MSCA-3 refer to adding the MSCA module to the skip connections in the first, second, and third layers, respectively. MaxViT refers to replacing the ViT block in the baseline method with the MaxViT block. As shown in Table 5, Method 2, after using the MaxViT module, improved all metrics compared to the baseline method. Comparing methods 1, 3, 4, and 5, it can be seen that adding MSCA modules to the skip connections in the baseline method significantly improves various metrics, and the magnitude of these improvements generally increases with the number of MSCA modules. When MSCA-1, MSCA-2, and MSCA-3 are all integrated into the baseline method, the metric improvements are essentially the greatest. While the baseline method's skip connection design is simple and can fuse low-level and high-level features from the encoder and decoder, it also introduces some redundant information. Adding MSCA modules to the skip connections enhances the model's perception of specific channels and spatial dimensions, enabling more effective extraction of richer and more useful features.

[0175] Integrating MSCA, MSCA-1, MSCA-2, and MSCA-3 into the baseline method resulted in the greatest improvement across all metrics, with the Dice coefficient increasing by 2.17% and MIoU by 1.71%. Overall, in practical applications, MaxViT exhibits good scalability and effectively extracts features when processing high-resolution images, enabling the neural network to further learn deeper features and fully perceive deeper contextual information. Furthermore, the skip connections improved by the MSCA attention mechanism, after extracting richer and more effective features from the feature map, are fed into the decoder, thus better fusing lower-level features from the encoder with higher-level features from the corresponding decoder, ultimately improving various segmentation metrics.

[0176] To determine the optimal value of the α weight in the DiceBCELoss mixture function designed in this chapter, different α values ​​were set in the experiment, and MM-TransUNet was trained and evaluated. By observing the performance of MM-TransUNet on the test set, the best-performing value was selected as the weight parameter α in the loss function. The impact of different α values ​​on the model segmentation accuracy is shown in Table 6.

[0177]

[0178] Table 6. The effect of different values ​​of α on accuracy.

[0179] As shown in Table 6, the value of α has some impact on the segmentation accuracy of the model. When the value of α is 0.5, the Dice coefficient can reach the optimal value of 85.82%. The Dice coefficient is one of the most important indicators for measuring the segmentation accuracy of the model. It represents the degree of overlap between the model's prediction and the true value. Therefore, in this embodiment, 0.5 is selected as the value of α.

[0180] To further verify the effectiveness of MM-TransUNet in lung nodule segmentation, we conducted a multi-model comparison experiment based on the LIDC-IDRI dataset. The selected comparison models included U-Net, U-Net++, U-Net3+, Res-UNet, TransUNet, RISU-Net, and REMU-Net. The final results of the comparison experiment are shown in Table 7. Compared with existing excellent open-source models and other improved models proposed by other scholars, MM-TransUNet achieved the best performance in Dice coefficient, which is the most important indicator for measuring the model's segmentation accuracy, representing the degree of overlap between the model's output and the true label. At the same time, MM-TransUNet also achieved the best values ​​in MIoU and Precision metrics. Therefore, this proves the effectiveness of MM-TransUNet in improving the accuracy of lung nodule segmentation and also effectively demonstrates the gains brought to the model by combining Transformer and CNN.

[0181]

[0182] Table 7 Comparative Experiment Results

[0183] Figure 12 The results of different open-source models on lung nodule segmentation are presented intuitively. To better illustrate the various situations encountered in lung nodule segmentation, four lung nodule CT images of different shapes and sizes were selected here, which can better reproduce the lung nodule segmentation task under different conditions, making the comparative experiment highly representative.

[0184] Sample 1 selected an irregularly shaped, medium-sized lung nodule. Its shape not only had an inward concavity but also an outward convex elongated strip at the bottom. The segmentation results showed that MM-TransUNet could learn its features in detail and reproduce its concave and convex shape well, while other models were lacking in this respect. For example, U-Net failed to segment the outward convex structure at the bottom, and the remaining models that reproduced the outward convex structure were also significantly different from the gold standard.

[0185] Sample 2 selected small-sized lung nodules. The segmentation of small-sized lung nodules is particularly susceptible to the influence of noise. For example, the U-Net3+ model incorrectly identifies other noise as lung nodules, while the U-Net model identifies lung nodules as noise, resulting in segmentation results that are too small. The MM-TransUNet model can distinguish lung nodules from similar noise just right, and the segmentation results are neither too large nor too small.

[0186] Sample 3 selected large, irregularly shaped lung nodules with a concave center. The segmentation results show that the selected models can basically capture the concave trend of the lung nodules, but the degree of restoration varies. The U-Net model's concavity is too large, resulting in distortion, while the TransUNet model's segmentation results show that the concavity is not obvious. The MM-TransUNet segmentation results achieve better restoration in terms of the degree of concavity.

[0187] Sample 4 selected an irregularly shaped, large lung nodule with surface noise. The segmentation results show that, due to the surface noise, other models identified the protruding lung nodule below as noise and failed to segment it, which is a significant distortion. MM-TransUNet, however, still managed to generate good segmentation results despite the noise interference. Although it cannot yet completely restore every detail of the lung nodule, it effectively distinguishes between the nodule and the noise.

[0188] In conclusion, based on the visualized segmentation results, MM-TransUNet achieves high segmentation accuracy when dealing with lung nodules of different shapes, and its segmentation results are closer to the gold standard compared to other excellent models.

[0189] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes made based on the description and drawings of the present invention are included within the scope of the present invention.

Claims

1. A lung nodule segmentation algorithm based on TransUNet, comprising the following steps: S1: Obtain the lung CT image dataset and perform lung parenchyma segmentation, region of interest extraction, and data augmentation operations on the lung CT image dataset; S2: Divide the lung CT image dataset after data augmentation into a training set and a test set; S3: Construct a segmentation model based on the TransUNet network structure and train the segmentation model using the training set; after each round of training, save the segmentation model parameters for this round, compare the detection results obtained by the segmentation model with the actual results to calculate the loss, and adjust the parameters of the segmentation model according to the loss; S4: Read the optimal parameters saved during the training phase, and substitute the optimal parameters into the segmentation model to obtain the optimal lung nodule segmentation model; S5. The optimal lung nodule segmentation model is tested using a test set, and the model accuracy of the optimal lung nodule segmentation model is evaluated using evaluation metrics. The feature is that, in step S3, the Transformer layer in the TransUNet network is replaced with the MaxViT network to form the MM-TransUNet network as a segmentation model, and the specific process of training the segmentation model using the MM-TransUNet network is as follows; S301: Input the images in the training set into the encoder, and first use the residual mechanism in conjunction with the convolutional neural network to extract shallow features from the input images layer by layer to extract shallow feature maps; S302: The shallow feature map is processed by the MaxViT module to fully perceive the deep context information, further learn deep features, and extract the deep feature map of the image; S303: Input the shallow feature map and deep feature map into the decoder, connect and upsample the shallow feature map and deep feature map, perform feature fusion layer by layer, and finally output the lung nodule segmentation image; S304: Calculate the loss between the lung nodule segmentation image output in step S303 and the actual lung nodule segmentation image using the loss function, adjust the parameters in the MM-TransUNet network, update the MM-TransUNet network model, and return to step S301 for a new round of training until all images in the training set have been trained.

2. The lung nodule segmentation algorithm based on TransUNet according to claim 1, characterized in that: In step S1, the specific steps for segmenting the lung parenchyma in the lung CT image dataset are as follows: S101: The original CT image is binarized and segmented using the OTSU algorithm to obtain a binary image; S102: The cavity in the lung parenchyma is filled using the flooding filling method to obtain a filled image; The opening operation is used to remove noise and small bubbles from the binarized image to obtain a denoised image; S103: Subtract the denoised image from the filled image to obtain a primary lung parenchyma mask image containing small connected regions; S104: Opening operation removes small connected regions in the initial lung parenchyma mask image to obtain the final lung parenchyma mask image; S105: Perform an AND operation between the final lung parenchyma mask image and the original CT image to obtain the lung parenchyma image.

3. The lung nodule segmentation algorithm based on TransUNet according to claim 2, characterized in that: Step S1 also includes, S106: Extract the region of interest containing lung nodules from the lung parenchyma image; locate the lung nodules using the lung nodule location information in the dataset, and then crop the lung parenchyma image with the lung nodules as the center point to generate input image blocks; S107: Expand the image source by randomly flipping or rotating the image patches. Use the expanded image patches together with the original image patches as the training and test datasets. Specifically, flipping involves randomly selecting image patches and flipping them horizontally or vertically, and the label images are also flipped accordingly. Specifically, rotating involves randomly selecting image patches and rotating them clockwise by 90°, 180°, or 270°, and the label images are also rotated accordingly.

4. The lung nodule segmentation algorithm based on TransUNet according to claim 1, characterized in that: In step S301, at the beginning of the encoder, MM-TransUNet first uses a three-layer ResNet-50 network based on CNN and residual mechanism. The shallow feature map output by each layer is used as the input to the next layer. Each ResNet-50 network layer combines the input features directly with the output features through skip connections to form residual blocks. The core idea of ​​residual blocks can be represented as follows: output = F(input) + input Where F(*) represents a series of operations performed by the network on the input; Then, the shallow feature map output by the third residual block is divided into blocks and positional encoding is added.

5. The lung nodule segmentation algorithm based on TransUNet according to claim 4, characterized in that: In step S302, the shallow feature map after adding position encoding is processed by MaxViT module 12 times. The specific process of each MaxViT module processing is as follows: The feature map is input into the MBConv block. After passing through a 1x1 convolutional layer and a 3x3 convolutional layer, the feature map is input into the channel attention module to enhance relevant features and weaken irrelevant features in the feature map, resulting in an enhanced feature map. Then, the enhanced feature map is convolved with the input feature map point by point to restore the dimension and output the enhanced feature map. The enhanced feature map is input to the block attention processing module, which splits it into several complementary and overlapping image blocks. Then, a self-attention mechanism is applied to each split image block to achieve interaction within a local range. The image blocks that have completed local interaction are then input into a feedforward neural network for full connection so that the output feature map has the same size as the final enhanced feature map. The feature map output by the block attention processing module is input into the table attention processing module. The input feature map is aggregated into different local grids in the form of global expansion. A self-attention mechanism is performed on the local space within each grid to achieve global interaction. Finally, all local grids are input into the feedforward neural network for full connection so that the output feature map has the same size as the final enhanced feature map.

6. The lung nodule segmentation algorithm based on TransUNet according to claim 5, characterized in that: In step S303, the decoder in MM-TransUNet adopts a skip connection structure, and an MSCA attention mechanism is added to each skip connection. The specific decoding steps of the decoder are as follows: S3031: The shallow feature maps output by each layer of the three-layer ResNet-50 network are first processed by the multi-scale cross-axial attention mechanism (MSCA) to extract important features from the shallow feature maps. After processing by the MSCA attention mechanism, the shallow feature maps output by the first, second, and third layers of the ResNet-50 network are processed into important feature maps of the first, second, and third layers, respectively. These important feature maps are then input into the decoder. S3032: The decoder integrates the important feature map of the third layer with the deep feature map using a 3x3 convolutional layer and performs linear activation using the ReLU function, and then upsamples to generate the initial segmentation image; S3032: The important feature map of the second layer is integrated with the initial segmentation image using a 3x3 convolutional layer and linearly activated using the ReLU function. Then, it is upsampled to generate an intermediate segmentation image. S3033: The first layer of important feature maps and the intermediate segmentation image are integrated using a 3x3 convolutional layer and linearly activated using the ReLU function, and then upsampled to generate the final segmentation image; S3034: Upsample the final segmented image and then apply the ReLU function for linear activation to obtain the lung nodule segmentation image.

7. The lung nodule segmentation algorithm based on TransUNet according to claim 6, characterized in that: In step S3031, the specific steps for applying the MSCA attention mechanism to process shallow feature maps are as follows: Input a shallow feature map F with shape H×W×C, and perform multi-scale convolution operations on the feature map in parallel along the X-axis and Y-axis respectively; the convolution kernels on the X-axis are set to 1×7, 1×11, and 1×21, and the convolution kernels on the Y-axis are set to 7×1, 11×1, and 21×1; after the multi-scale convolution operations, the convolution outputs F and F along the X-axis and Y-axis are obtained. x and F y Then calculate F. x and F y Cross-attention; for cross-attention on the X-axis, first, F... x As a key and value matrix, and F y As a query matrix, F is obtained. T For attention along the Y-axis, F... y As a key and value matrix, and F x As a query matrix, F is obtained. B ; Finally, F B F T By integrating F, we can obtain the first layer of important feature maps, the second layer of important feature maps, or the third layer of important feature maps.

8. The lung nodule segmentation algorithm based on TransUNet according to claim 6, characterized in that: In step S304, the DiceBCE loss function is used to calculate the loss between the detection result and the actual result. The DiceBCE loss function model is as follows: L total =(1-α)L BCE +αL Dice Where α is the weight parameter of the composite loss function, L BCE Let L be the binary cross-entropy loss function. Dice The Dice loss function; L BCE The function expression is as follows: Where G(r,c) is the true value of the pixel at coordinate (r,c), and S(r,c) is the predicted value of the pixel at coordinate (r,c). L Dice The function expression is as follows: Where X represents the true value and Y represents the segmentation result.

9. The lung nodule segmentation algorithm based on TransUNet according to claim 1, characterized in that: In step S5, the evaluation metrics include Dice coefficient, MIoU coefficient, Recall coefficient, and Precision coefficient. The formula for the Dice coefficient is: The formula for the MioU coefficient is: The formula for the recall coefficient is: The formula for the Precision coefficient is as follows: In this context, TP represents a true example, FN represents a false negative, FP represents a false positive, and TN represents a true negative.

Images