Dual branch network 3D image segmentation method for lung nodule CT scan

Abstract

The application discloses a kind of double-branch network 3D image segmentation methods for lung nodule CT scanning, belong to the application of depth learning technique in lung nodule medical image processing field.It first constructs the segmentation model of double-branch network 3D image, which includes feature extraction branch for extracting medical image features, main feature decoding branch for decoding the image features extracted by feature extraction branch by layer upsampling, boundary feature decoding branch for extracting and enhancing the edge information of lung nodule CT scanning, and fusion module for fusing the output of main feature decoding branch and boundary feature decoding branch;The segmentation model is trained based on the training data set collected, and the 3D image segmentation model for lung nodule CT scanning is obtained.The application effectively overcomes the limitations of high variability and image noise by combining deep learning algorithm and innovative double-branch network structure, improves the accuracy and generalization ability of segmentation, especially in lung nodule boundary recognition, the application method is especially suitable for limited sample size, through efficient feature extraction and edge attention mechanism, the accurate segmentation of lung nodule is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the application of deep learning technology in the field of medical image processing of lung nodules, and in particular to a dual-branch network 3D image segmentation method for lung nodule CT scans. BACKGROUND

[0002] The diagnosis and treatment of lung diseases largely depend on the accurate detection and segmentation of lung nodules. Traditional image processing techniques, such as threshold segmentation, region growing or edge detection, have been used for such tasks, but they often rely on manually set parameters and prior knowledge, making it difficult to cope with the high variability of lung nodules and noise in CT images.

[0003] With the increase in computing power and the availability of large datasets, deep learning, especially convolutional neural networks (CNN), has shown unprecedented performance in medical image segmentation tasks. They can automatically learn complex features in image data and perform efficient feature extraction and pattern recognition. However, for small and fuzzy targets such as lung nodules, conventional CNN segmentation networks still face problems of insufficient segmentation accuracy and generalization ability.

[0004] In addition, existing deep learning models usually require a large amount of labeled data for training, and high-quality medical image labeling is both expensive and time-consuming. In the case of small samples, the performance of these models will usually be greatly reduced. At the same time, the edge region of lung nodules is difficult to be accurately segmented by the network due to its low contrast with the surrounding tissue, which limits the practical application value of the model. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application proposes a dual-branch network 3D image segmentation method for lung nodule CT scans, aiming to improve the accuracy of lung nodule segmentation.

[0006] The technical solution adopted by the present application is:

[0007] The dual-branch network 3D image segmentation method for lung nodule CT scans includes:

[0008] Step 1, construct a dual-branch network 3D image segmentation model:

[0009] The segmentation model includes a feature extraction branch, a main feature decoding branch, a boundary feature decoding branch and a fusion module;

[0010] Among them, the feature extraction branch is used to extract the image features of the input lung nodule CT scan (3D medical image), and the feature extraction branch includes a plurality of cascaded down-sampling convolution blocks and a bottleneck layer, and the number of up-sampling convolution blocks is defined as N;

[0011] The main feature decoding branch decodes the image features extracted by the feature extraction branch through layer-by-layer upsampling to generate a segmentation map of the lung nodule CT scan. The input of the main feature decoding branch is the output of the bottleneck layer of the feature extraction branch. The main feature decoding branch includes N cascaded downsampling convolutional blocks.

[0012] The boundary feature decoding branch is used to extract and enhance the edge information of pulmonary nodules in CT scans. The input of the boundary feature decoding branch is the output of the bottleneck layer of the feature extraction branch. The boundary feature decoding branch includes N cascaded boundary feature decoders based on boundary attention modules.

[0013] The input of each upsampled convolutional block in the main feature decoding branch also includes the output of the downsampled convolutional block of the corresponding layer. That is, for the i-th upsampled convolutional block in the main feature decoding branch, its input includes the branch input of the main feature decoding branch (the branch input of the first upsampled convolutional block is the output of the bottleneck layer, and the branch input of the remaining upsampled convolutional blocks is the output of the previous downsampled convolutional block), as well as the output of the (N-i+1)-th downsampled convolutional block in the feature extraction branch; where i = 1, 2, ..., N;

[0014] The input of each boundary feature decoder in the boundary feature decoding branch also includes the output of the upsampled convolutional block of the corresponding layer. That is, for the j-th boundary feature decoder in the boundary feature decoding branch, its input includes the branch input of the boundary feature decoding branch (the branch input of the first boundary feature decoder is the output of the bottleneck layer, and the branch input of the remaining boundary feature decoders is the output of the previous boundary feature decoder), as well as the output of the (N-j+1)-th upsampled convolutional block of the main feature decoding branch; where j = 1, 2, ..., N;

[0015] The fusion module is used to fuse the outputs of the main feature decoding branch and the boundary feature decoding branch to obtain the final lung nodule segmentation result of the CT scan of lung nodules;

[0016] Step 2: Train the segmentation model based on the collected training dataset. During training, optimize the segmentation model parameters based on the difference between the predicted results and the actual segmentation mask to obtain a 3D image segmentation model for CT scans of lung nodules.

[0017] In this invention, the main feature decoding branch and the boundary feature decoding branch share the image features extracted by the feature extraction branch. An upsampling module gradually restores the spatial dimension of the image while retaining important feature information. A boundary attention module improves the edge clarity and accuracy of the segmentation map. The fusion module effectively integrates the information generated by the two decoding branches to produce a final, accurate segmentation result. During training, the segmentation model of this invention automatically learns how to effectively extract features from medical images and generate accurate segmentation results through end-to-end training.

[0018] Furthermore, the downsampling convolutional blocks of the feature extraction branch sequentially include: a 3D convolutional layer, a ReLU activation function layer, and a max pooling layer.

[0019] Furthermore, the upsampling convolutional blocks of the main feature decoding branch sequentially include: a transposed convolutional layer, a 3D convolutional layer, and a ReLU activation function layer. The transposed convolutional layer transposes the input of the main feature decoding branch's branch input to obtain a feature map of the same dimension as the output of the (N-i+1)th downsampling convolutional block of the feature extraction branch. This feature map is then added to the output of the (N-i+1)th downsampling convolutional block and fed into the 3D convolutional layer. The ReLU activation function layer then passes through the 3D convolutional layer to obtain the output of the i-th upsampling convolutional block of the main feature decoding branch. The main feature decoding branch uses standard upsampling and convolution operations to gradually restore the spatial dimension of the image while preserving important feature information.

[0020] Furthermore, the structure of each boundary feature decoder in the boundary feature decoding branch is as follows:

[0021] The attention heatmap of the j-th boundary feature decoder is obtained by calculating the attention heatmap of the output of the (N-j+1)-th upsampled convolutional block of the main feature decoding branch. The input of the j-th boundary feature decoder is transposed and convolved by the transposed convolutional layer, and then added to the attention heatmap of the j-th boundary feature decoder to obtain the output of the j-th boundary feature decoder.

[0022] The boundary feature decoder enhances the extraction of edge and detail features through a specific attention mechanism, thereby improving the model's sensitivity to image edges and segmentation accuracy.

[0023] Furthermore, the fusion module employs a three-dimensional dilated spatial convolutional pooling pyramid, which enhances the model's adaptability to changes in nodule size by using multiple 3D dilated convolutions with different dilation rates in parallel.

[0024] Furthermore, in step 2, the loss function of the segmentation model during training is set to a combination of Dice loss and boundary loss, and the main feature decoding branch and the boundary feature decoding branch are trained under supervision based on Dice loss and boundary loss respectively.

[0025] Furthermore, the Dice loss is set to be a cross-entropy-based Dice loss.

[0026] The technical solution provided by this invention brings at least the following beneficial effects:

[0027] Traditional lung nodule detection and segmentation techniques face challenges such as high variability and image noise. This invention addresses these limitations by combining deep learning algorithms with an innovative dual-branch network structure, effectively improving segmentation accuracy and generalization ability, particularly excelling in lung nodule boundary recognition. This method is especially suitable for situations with limited sample sizes, achieving precise lung nodule segmentation through efficient feature extraction and edge attention mechanisms. This invention is used for precise segmentation of three-dimensional (3D) medical images, particularly for improving the accuracy and efficiency of lung nodule segmentation. Its application can enhance the automation level of medical image analysis, assisting medical experts in making more accurate diagnoses and treatment plans. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of the structure of the 3D dual-branch network (DualNet) used in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the Encoder (downsampling) module structure used in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the Decoder (upsampling) module structure used in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the bottleneck module structure used in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the EA boundary feature enhancement module structure used in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0035] This invention proposes a dual-branch network 3D image segmentation method for CT scans of lung nodules. This method can effectively improve the accuracy of lung nodule segmentation, especially in the boundary region. Figure 1 As shown, the method in this embodiment of the invention employs a dual-branch 3D medical image segmentation network—DualNet—with a U-shaped encoding and decoding structure. The DualNet structure includes two key decoding branches: a main feature decoding branch and a boundary feature decoding branch. The main feature decoding branch is responsible for the overall mask prediction of lung nodules, while the boundary feature decoding branch focuses on capturing the edge information of the nodules. Both branches share the same feature extraction encoder. Figure 1 The diagram shows several cascaded feature extraction branches of feature encoders.

[0036] like Figure 1 As shown, in this embodiment of the invention, the feature extraction branch includes several encoders (Encoder_1 to Encoder_4) and a bottleneck layer. In this embodiment, it specifically includes four encoders, each with the same network structure, such as... Figure 2 As shown, each Encoder sequentially includes a three-dimensional convolutional layer (preferably with a kernel size of 3×3×3), a ReLU activation function layer, and a max pooling layer. The features extracted by the four Encoders are denoted as F1 to F4. The main feature decoding branch includes several main feature decoders. In this embodiment, there are specifically four, meaning the number of main feature decoders is the same as the number of Encoders, denoted as Decoder_1 to Decoder_4. Furthermore, each main feature decoder has the same network structure, such as... Figure 3As shown, it sequentially includes: a transposed convolutional layer, a 3D convolutional layer (the preferred kernel size is 3×3×3), and a ReLU activation function layer; the output of the feature extraction branch is input to the main feature decoding branch via the bottleneck layer. The input of each Decoder in the main feature decoding branch includes the output of the Encoder at the same level (in the direction of forward propagation). The input of the first Encoder (Decoder_4) in the main feature decoding branch also includes the output of the bottleneck layer, that is, the input of Decoder_4 is the output of the bottleneck layer and the output of Encoder_4; the input of Decoder_3 is the output of Decoder_4 and the output of Encoder_3. The Decoder first performs a transposed convolution on the main input (the first one is the output of the bottleneck layer, and the rest are the outputs of the previous level) through the transposed convolutional layer to obtain a feature map with the same dimension as the output of the Encoder at the same level. Then, the output feature maps of the transposed convolutional layers are added to the output feature map of the Encoder at the same level, and then passed through the 3D convolutional layer and the ReLU activation function layer to obtain the output feature map of each Decoder. In this embodiment of the invention, the outputs of the four decoders in the main feature decoding branch are denoted as decoding features M1 to M4, that is, the output feature maps of all decoders constitute the prediction segmentation result.

[0037] See Figure 4 In this embodiment of the invention, the bottleneck layer used is a cascaded three-dimensional convolutional layer and a ReLU activation function layer. The boundary feature decoding branch also includes several boundary feature decoders (EAs), the number of which is the same as the Encoder. Each EA has the same network structure, and its input includes the output of the Decoder at the same level (in the forward propagation direction) and the output of the previous Decoder. The input of the first EA of the boundary feature decoding branch is the output of the bottleneck layer and the output of Decoder_4. Figure 5 As shown, the specific structure of each EA is as follows: the outputs of the same level Decoder (M1 to M4) are processed by attention heat calculation to obtain the corresponding attention heat map, and then added to the boundary branch feature map of the transposed convolutional layer to obtain the output of each EA, denoted as E1 to E4. For the first EA, its input boundary branch feature map is the output of the bottleneck layer. Starting from the second EA, its input boundary branch feature map is the output of the previous EA.

[0038] The output of the main feature decoding branch (the last Decoder, i.e.) Figure 1The outputs of Decoder_1 and the boundary feature decoding branch (the output of the last EA) are concatenated and then fed into 3D ASPP (three-dimensional dilated spatial convolutional pooling pyramid) for fusion processing to obtain the final accurate segmentation result.

[0039] During model training, the training supervision of the main feature decoding branch is obtained by extracting the lung nodule mask, and the training supervision of the boundary feature decoding branch is obtained by extracting the edges of the lung nodule mask. That is, during model training, the training loss includes boundary loss and Dice loss based on cross-entropy. The total loss can be set as a weighted sum of the two, and the weight coefficients are empirical values, set based on the actual processing scenario.

[0040] In this embodiment of the invention, the feature extraction branch of the constructed segmentation model includes several upsampled convolutional blocks and a bottleneck; the main feature decoding branch includes several downsampled convolutional blocks, while the boundary feature decoding branch uses a boundary attention module, which includes several boundary feature decoders based on the boundary attention module, and the number of upsampled convolutional blocks, downsampled convolutional blocks, and boundary feature decoders is the same. During segmentation model training, the degree of attention to edge information is dynamically adjusted based on the difference between the predicted result and the actual segmentation mask to improve the model's recognition accuracy for blurred boundary regions. The 3D ASPP module processes feature information from different depth levels, and enhances the model's adaptability to changes in nodule size by using 3D dilated convolutions with different dilation rates in parallel.

[0041] Example

[0042] (1) System settings:

[0043] Utilize a computing platform with high-performance GPUs to support the training and inference computations of deep learning models. Choose a suitable deep learning framework for medical image processing, such as TensorFlow or PyTorch, to build and train the model.

[0044] (2) Data preparation:

[0045] Collect a dataset of CT scans of lung nodules, ensuring it includes labeled lung nodule information. Preprocess the dataset, including noise removal, image intensity normalization, and window width and level adjustment.

[0046] (3) Model Architecture:

[0047] Design a deep learning model, such as... Figure 1As shown, it includes: a feature extraction branch, a main feature decoding branch, a boundary feature decoding branch, and a dilated spatial convolutional pooling pyramid. The loss function is a combination of Dice loss and boundary loss, which supervise the main feature extraction branch and the boundary feature extraction branch respectively. The Adam optimizer is selected to optimize the model parameters, and appropriate learning rates and decay strategies are set.

[0048] (4) Model training:

[0049] Train the model on the training set, utilizing batch processing and data augmentation techniques to improve model performance. Validate the model using the validation set to adjust model parameters and structure, avoiding overfitting.

[0050] (5) Model testing and evaluation:

[0051] Model performance was evaluated on an independent test set using quantitative metrics such as Dice coefficient, average surface distance (ASD), sensitivity (SEN), overlap between segmentation results and true segmentation (DSC), and positive predictive value (PPV).

[0052] Furthermore, after obtaining the segmentation results based on the model, mathematical morphological operations such as opening and closing operations can be applied to optimize the segmentation results, removing noise and small regions. The segmentation results can then be fine-tuned based on feedback from clinical experts.

[0053] To further demonstrate the effectiveness of the segmentation model proposed in this embodiment, the constructed segmentation model was compared with common networks in the field of image segmentation, including 3DUnet, Vnet, denseNet, LinkNet (Unet+ResNet), and nnunet. All networks use the same framework, sampling strategy, and preprocessing and post-processing as the segmentation model constructed in this embodiment. Experimental results are shown in Table 1.

[0054] Table 1. Mean ± Standard Deviation of Results for Various Segmentation Methods

[0055] Network architecture DSC (%) ASD (mm) SEN (%) PPV (%) 3DUnet 75.89±12.99 0.31±0.39 87.16±12.9 70.81±17.57 Vnet 77.51±11.40 0.29±0.31 88.83±12.34 71.42±14.78 DenseNet 78.55±12.49 0.27±0.35 86.01±15.22 75.79±14.73 linkNet 77.84±21.74 1.79±7.52 77.98±24.52 82.52±21.55 nnunet 80.30±11.34 0.26±0.45 85.40±13.27 78.69±14.49 DualNet 82.74±10.19 0.19±0.21 89.35±11.79 79.64±13.34

[0056] Due to the effective edge supervision provided by its dual-branch structure design, DualNet in this embodiment of the invention performs exceptionally well on the ASD (Advanced Distance Score) metric. DualNet's performance on ASD is also outstanding, with an average ASD of 0.19 mm ± 0.21, significantly outperforming other comparative models. For example, the closest model, NNUNet, has an ASD of 0.26 mm ± 0.45, while ResNet's average ASD is as high as 1.79 mm ± 7.52. This indicates that DualNet, provided by this embodiment of the invention, significantly improves the edge localization accuracy of lung nodules compared to other models. DualNet can more accurately identify the boundaries of lung nodules, especially in boundary recognition, and maintains good performance even with limited training samples.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0058] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for segmenting 3D images of lung nodules using a dual-branch network in CT scans, characterized in that, Includes the following steps: Step 1: Construct a dual-branch network 3D image segmentation model: The segmentation model includes a feature extraction branch, a main feature decoding branch, a boundary feature decoding branch, and a fusion module; The feature extraction branch is used to extract the image features of the input lung nodule CT scan. The feature extraction branch includes several cascaded downsampling convolutional blocks and a bottleneck layer. The number of upsampling convolutional blocks is defined as N. The main feature decoding branch decodes the image features extracted by the feature extraction branch through layer-by-layer upsampling to generate a segmentation map of the lung nodule CT scan. The input of the main feature decoding branch is the output of the bottleneck layer of the feature extraction branch. The main feature decoding branch includes N cascaded downsampling convolutional blocks. The boundary feature decoding branch is used to extract and enhance the edge information of pulmonary nodules in CT scans. The input of the boundary feature decoding branch is the output of the bottleneck layer of the feature extraction branch. The boundary feature decoding branch includes N cascaded boundary feature decoders based on boundary attention modules. For the i-th upsampled convolutional block of the main feature decoding branch, its input includes: the branch input of the main feature decoding branch and the output of the N-i+1-th downsampled convolutional block of the feature extraction branch; wherein, the branch input of the first upsampled convolutional block of the main feature decoding branch is the output of the bottleneck layer, and the branch input of the remaining upsampled convolutional blocks is the output of the previous downsampled convolutional block, i = 1, 2, ..., N; For the j-th boundary feature decoder in the boundary feature decoding branch, its input includes: the branch input of the boundary feature decoding branch and the output of the (N-j+1)-th upsampled convolutional block of the main feature decoding branch; wherein, the branch input of the first boundary feature decoder in the boundary feature decoding branch is the output of the bottleneck layer, and the branch input of the remaining boundary feature decoders is the output of the previous boundary feature decoder, j = 1, 2, ..., N; The fusion module is used to fuse the outputs of the main feature decoding branch and the boundary feature decoding branch to obtain the final lung nodule segmentation result of the CT scan of lung nodules; Step 2: Train the segmentation model based on the collected training dataset. During training, optimize the segmentation model parameters based on the difference between the predicted results and the actual segmentation mask to obtain a 3D image segmentation model for CT scans of lung nodules.

2. The method as described in claim 1, characterized in that, The downsampling convolutional blocks of the feature extraction branch consist of: a 3D convolutional layer, a ReLU activation function layer, and a max pooling layer.

3. The method as described in claim 1, characterized in that, The upsampling convolutional blocks of the main feature decoding branch sequentially include: a transposed convolutional layer, a 3D convolutional layer, and a ReLU activation function layer. The transposed convolutional layer is used to transpose the input of the main feature decoding branch branch to obtain a feature map of the same dimension as the output of the (N-i+1)th downsampling convolutional block of the feature extraction branch. This feature map is then added to the output of the (N-i+1)th downsampling convolutional block and fed into the 3D convolutional layer. After passing through the ReLU activation function layer, the output of the ith upsampling convolutional block of the main feature decoding branch is obtained.

4. The method as described in claim 1, characterized in that, The structure of each boundary feature decoder in the boundary feature decoding branch is as follows: The attention heatmap of the j-th boundary feature decoder is obtained by calculating the attention heatmap of the output of the (N-j+1)-th upsampled convolutional block of the main feature decoding branch. The input of the j-th boundary feature decoder is transposed and convolved by the transposed convolutional layer, and then added to the attention heatmap of the j-th boundary feature decoder to obtain the output of the j-th boundary feature decoder.

5. The method as described in claim 1, characterized in that, The bottleneck layer consists of cascaded three-dimensional convolutional layers and ReLU activation function layers.

6. The method as described in claim 3, 4, or 5, characterized in that, The kernel size of the 3D convolutional layer is 3×3×3.

7. The method as described in claim 1, characterized in that, The fusion module employs a three-dimensional, hollow spatial convolutional pooling pyramid.

8. The method as described in claim 1, characterized in that, In step 2, the loss function of the segmentation model during training is set to a combination of Dice loss and boundary loss. The main feature decoding branch and the boundary feature decoding branch are trained under supervision based on Dice loss and boundary loss respectively.

9. The method as described in claim 8, characterized in that, The Dice loss is set to be based on cross-entropy.

Images