A pulmonary nodule segmentation method based on residual ECA channel attention UNet combined with TRW-S
By combining the sequential tree-weighted information transfer algorithm of residual ECA channel attention UNet and Markov random field, the problem of edge blur and inconsistency in pulmonary nodule CT image segmentation is solved, and clearer boundaries and consistency are achieved. The segmentation effect improves the detection ability of small lesions.
Patent Information
- Application Number
- CN202210290186.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-23
AI Technical Summary
The existing UNet and its improved network have problems of blurred edges and inconsistent lesion areas in the segmentation of pulmonary nodules, making it difficult to effectively obtain the connection between channels, resulting in poor segmentation effect.
Using residual ECA channel attention UNet network combined with Markov random field sequential tree weighted information transfer algorithm (TRW-S), the residual ECA channel attention module and ASPP hollow convolution pyramid are improved, and the network architecture is improved. Residual structure is added to the decoder, and edge smoothing is performed using the sequential tree weighted information transfer algorithm based on Markov random field.
The marginal continuity and regional consistency of lung nodule segmentation are improved, the model's detection ability of small lesions is enhanced, information loss and gradient explosion risks are reduced, network model convergence is faster and better, and segmentation effect is improved.
Smart Images

Figure CN114677511B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image segmentation, and specifically relates to a lung nodule segmentation method combining a residual attention UNet network with a sequential tree weighted information propagation (TRW-S) algorithm. Background Art
[0002] Lung cancer is the most common cancer in the world today, accounting for 11.6% of all cancers. In 2019, lung cancer ranked first among Chinese men, and smoking is the most important cause of lung cancer. Malignant lung nodules are a precursor to lung tumors. The characteristics of CT images are high-density white shadows or ground-glass shadows in the lungs with a diameter of less than 3 cm. Using segmentation algorithms to segment lung nodules will assist doctors in judging the condition. The segmented lesions can also be used to further classify the benign and malignant lesions. Early detection, identification, tracking, and removal of malignant lung nodule lesions are important means of preventing lung cancer. Manual screening and identification of lung nodules often consumes a lot of time and energy of radiologists. The use of machine algorithms can assist doctors in diagnosis and reduce the workload of doctors.
[0003] The use of end-to-end deep learning networks for image segmentation has become the mainstream of image semantic segmentation. Based on Unet and its improved full convolutional neural network, through training, pulmonary nodules can be classified at the pixel level, and finally a binary image is output to mark the pulmonary nodule area and its edge. The full convolutional network combines high-dimensional position information and low-dimensional edge information to complete the task of semantic segmentation through a decoding-encoding mode. However, the disadvantage is that due to the influence of the edge volume effect, the CT images of pulmonary nodules are often segmented by Unet and its improved network, and some lesions are small or the difference between lesion tissue and surrounding tissue is small, and pulmonary nodules are difficult to be completely segmented. By introducing a processed public data set and comparing it with the existing general medical image segmentation method, it can be found that UNet and its improved method cannot obtain a good pulmonary nodule segmentation effect; the fundamental reason is that these networks have large fluctuations during training, and the connection between channels cannot be well obtained, and the final result has blurred edge demarcation, and even there are some misjudgment problems inside the lesion area; the purpose of the present invention is to solve the above technical problems. Summary of the invention
[0004] The purpose of the present invention is to solve the technical problems of blurred edge demarcation and inconsistent internal segmentation of lesion areas when using UNet and its existing UNet improved network to segment lung nodules.
[0005] A pulmonary nodule segmentation method based on residual ECA channel attention UNet combined with TRW-S, which includes the following steps:
[0006] Step 1: Read lung CT images and pre-process the images;
[0007] Step 2: Cut out the lung nodule image and generate the corresponding mask;
[0008] Step 3: Divide the acquired lung nodule images and corresponding masks into data sets to obtain training sets, validation sets, and test sets;
[0009] Step 4: Build the residual ECA channel attention UNet deep learning network;
[0010] Step 5: Input the obtained training set and validation set into the constructed deep learning network for training;
[0011] Step 6: Input the obtained training set into the trained neural network to obtain the prediction graph;
[0012] Step 7: Input the predicted image into the Markov random field based sequential tree weighted information transfer algorithm (TRW-S) for edge smoothing;
[0013] Step 8: Output the predicted image.
[0014] In step 1, a lung CT image in a specified format in a target data set is read; and a window level and a window width of the CT image are adjusted.
[0015] In step 2, the portion of the CT image with the lung nodules is cut out, and a mask image corresponding to the lung nodules is generated.
[0016] In step 4, the constructed residual ECA channel attention UNet deep learning network is specifically:
[0017] Input layer → convolution module → the first residual ECA channel attention module (composed of two 3×3 convolutions, a combination of a BN layer and an ECA channel attention module combined with residual connections) → the first maximum pooling layer → the second residual ECA channel attention module → the second maximum pooling layer → the third residual ECA channel attention module → the third maximum pooling layer → the fourth residual ECA channel attention module → the fourth maximum pooling layer → the dilated convolution pyramid module → the first double upsampling, the feature map formed by the first double upsampling and the feature map formed by the fourth residual ECA channel attention module are channel fused → the first ECA channel attention module (composed of two 3×3 convolutions, The BN layer is composed of a BN layer and an ECA channel attention module) → the second double upsampling, the feature map formed by the second double upsampling is channel-fused with the feature map formed by the third residual ECA channel attention module → the second ECA channel attention module → the third double upsampling, the feature map formed by the third double upsampling is channel-fused with the feature map formed by the second residual ECA channel attention module → the third ECA channel attention module → the fourth double upsampling, the feature map formed by the fourth double upsampling is channel-fused with the feature map formed by the first residual ECA channel attention module → the fourth ECA channel attention module → convolution module → sigmoid function → prediction graph.
[0018] In step 5, when training the neural network, a loss function combining binary cross entropy and confidence is used, Nadam is used as the optimization function, and the intersection-over-union ratio is used as the evaluation index of the model.
[0019] In step 7, a Markov random field is constructed for the prediction graph, and the binary terms are smoothed to improve the edge continuity and internal consistency of the segmentation effect, which specifically includes the following steps:
[0020] 7-1) Construct a Markov random field in the grayscale prediction graph output by the deep learning network, treat each pixel of the image as a node in the Markov random field, and use the pixel grayscale value as the unary weight of the node;
[0021] 7-2) Construct a sequential tree weighted information transfer algorithm (TRW-S) based on Markov random fields to perform binary smoothing on the segmentation edges, where the unary weight is the pixel value of the segmentation result generated by the segmentation network.
[0022] Compared with the prior art, the present invention has the following technical effects:
[0023] 1) After the existing technology uses a fully convolutional neural network to segment the lung nodules, it often uses a hard threshold method to directly binarize the segmentation results, which does not take into account the influence between each pixel; the present invention constructs a sequential tree weighted information transfer algorithm based on a Markov random field in the image output by the deep learning network, and better determines the boundary of the lung nodule through information transfer between pixels. At the same time, for some gray areas inside the lesion area (i.e., the inaccurate segmentation area), it can also rely on the influence of the surrounding lesion pixels to adjust it, and finally achieve the purpose of high consistency within the segmentation area and reasonable and clear segmentation boundaries;
[0024] 2) The present invention improves the backbone network architecture of UNet, uses a residual network combined with ECA channel attention in its backbone network, and uses ASPP to improve the network;
[0025] 3) Since the addition of a residual structure to the decoder of the network has little effect on the segmentation result, the present invention adds a residual structure to the encoding part of the backbone network of UNet, and uses the residual structure to directly transmit the input information to the output, thereby reducing the loss of information; for the network, the problem of gradient explosion is reduced, thereby making the constructed network model converge faster and better;
[0026] 4) Adding the ECA channel attention mechanism to the network constructed by the present invention can enable the model to better focus on the information of important channels, thereby improving the training effect of the model; using the ASPP hole convolution pyramid in the model can enable the model to better obtain multi-scale information and improve the detection effect of small lesions. Finally, batch normalization is inserted to prevent the model from overfitting. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0028] Figure 1 is a flow chart of the present invention;
[0029] Figure 2 is a structural diagram of a neural network in the present invention;
[0030] Figure 3 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION
[0031] like Figure 1 to Figure 2 As shown, a lung nodule segmentation method comprises the following steps:
[0032] 1. Read lung CT images in dcm format from the LIDC-IDRI dataset
[0033] 2. Adjust the window level of the CT image to 1600 and the window width to 450
[0034] 3. Read the XML annotation file, and cut out the part with lung nodules in the CT image based on the annotation information. The size is 64*64. Read the XML annotation file, and generate the mask image corresponding to the lung nodule based on the edge segmentation information;
[0035] 4. Divide the images captured in step 3 into data sets
[0036] 5. Build the residual ECA channel attention UNet deep learning network
[0037] 6. Input the training set and validation set divided in step 5 into the network constructed in step 6 for training
[0038] 7. Input the test set into the network trained in step 7 to obtain the prediction graph
[0039] 8. Input the output graph in step 7 into the Markov random field based sequential tree weighted information transfer algorithm (TRW-S) for edge smoothing
[0040] 9. Output the predicted image;
[0041] In step 1, the LIDC-IDRI dataset was collected by the National Cancer Institute of the United States. There are a total of 1018 research cases, each of which has approximately 200-400 CT images and is annotated by four professional doctors. The size of each CT image is 512*512. This method selects 3mm-33mm lung nodule slices containing the complete outline of the node for segmentation.
[0042] In step 3, the node part is cut into a 64*64 image with the node as the center according to the location information marked in the XML file in the data set, and the corresponding boundary image is generated according to the coordinate value of the node outline information. The boundary is filled by the algorithm to obtain the corresponding mask image. Finally, 10,454 lung nodule images and corresponding masks are cut out.
[0043] In step 4, the data set is divided into 8:1:1 ratios, where the training set has 8363 pairs of images, the validation set has 1045 pairs of images, and the test set has 1046 pairs of images.
[0044] In step 5, construct the residual ECA channel attention UNet full convolutional neural network:
[0045] The input size of the lung nodule image is 64×64. After a 1×1 convolution, A is obtained. It enters a residual ECA channel attention module. Specifically, A is subjected to two 3×3 convolutions, a BN layer and an ECA channel attention module to obtain B. A is subjected to a 1×1 convolution and element-wise addition with B to obtain B1. It is subjected to a maximum pooling layer to obtain B2. The feature map size becomes 32×32 and the number of channels becomes 128. B2 is subjected to a residual ECA channel attention module to obtain C1. C1 is subjected to a maximum pooling layer to obtain C2. The feature map size becomes 16×16 and the number of channels becomes 256. C2 is subjected to a residual ECA channel attention module to obtain D1. D1 is subjected to a maximum pooling layer to obtain D2. The feature map size becomes 8×8 and the number of channels becomes 512. D2 is obtained by a residual ECA channel attention module to obtain E1, and E1 is obtained by a maximum pooling layer to obtain E2. The feature map size becomes 4×4, and the number of channels becomes 1024. E2 obtains multi-scale information through a dilated pyramid convolution module to obtain F. F is upsampled twice to obtain E-1, and E-1 and E1 are channel-fused and then pass through two layers of 3×3 convolutions, and then the BN layer and ECA channel attention to obtain E-2. E-2 is upsampled twice to obtain D-1, and D-1 and D1 are channel-fused and then pass through two layers of 3×3 convolutions, and then the BN layer and ECA channel attention to obtain D-2. D-2 is upsampled twice to obtain C-1, and C-1 and C1 are channel-fused and then pass through two layers of 3×3 convolutions, and then the BN layer and ECA channel attention to obtain C-2. C-2 is upsampled twice to obtain B-1, and B-1 and B1 are channel-fused and then pass through two layers of 3×3 convolutions, and then the BN layer and ECA channel attention to obtain B-2. After 1×1 convolution and sigmoid function, B-2 obtains A-1 as the prediction result.
[0046] In step 6, a loss function combining binary cross entropy and confidence was used, Nadam was used as the optimization function, intersection-over-union ratio was used as the evaluation index of the model, batch_size was set to 32, and the learning rate was 1e-4. After 200 rounds of training, the effect reached the optimal value. The model hyperparameters were adjusted according to the accuracy of the validation set.
[0047] In step 8, a Markov random field is constructed for the output image, and the binary terms are smoothed to improve the edge continuity and internal consistency of the segmentation effect, which specifically includes the following steps:
[0048] 8-1) constructing a Markov random field in the grayscale prediction graph output by the network constructed by the present invention, taking each pixel of the image as a node of the Markov random field, and using the pixel grayscale value as the unary weight of the node;
[0049] 8-2) Construct a sequential tree weighted information transfer algorithm (TRW-S) based on Markov random fields to perform binary smoothing on the segmentation edges, where the unary weight is the pixel value of the segmentation result generated by the segmentation network.
Claims
1. A pulmonary nodule segmentation method based on residual ECA channel attention UNet combined with TRW-S, characterized in that: It includes the following steps: Step 1: Read lung CT images and pre-process the images; Step 2: Cut out the lung nodule image and generate the corresponding mask; Step 3: Divide the acquired lung nodule images and corresponding masks into data sets to obtain training sets, validation sets, and test sets; Step 4: Build the residual ECA channel attention UNet deep learning network; Step 5: Input the obtained training set and validation set into the constructed deep learning network for training; Step 6: Input the obtained test set into the trained network to obtain the prediction graph; Step 7: Input the predicted image into the Markov random field based sequential tree weighted information transfer algorithm TRW-S for edge smoothing; Step 8: Output the predicted image; In step 4, the constructed residual ECA channel attention UNet deep learning network is specifically: Input layer → convolution module → first residual ECA channel attention module → first maximum pooling layer → second residual ECA channel attention module → second maximum pooling layer → third residual ECA channel attention module → third maximum pooling layer → fourth residual ECA channel attention module → fourth maximum pooling layer → dilated convolution pyramid module → first double upsampling; The feature map formed by the first two-fold upsampling is channel-fused with the feature map formed by the fourth residual ECA channel attention module → the first ECA channel attention module → the second two-fold upsampling; The feature map formed by the second double upsampling is channel-fused with the feature map formed by the third residual ECA channel attention module → second ECA channel attention module → third double upsampling; The feature map formed by the third two-fold upsampling is channel-fused with the feature map formed by the second residual ECA channel attention module → the third ECA channel attention module → the fourth two-fold upsampling; The feature map formed by the fourth double upsampling is channel-fused with the feature map formed by the first residual ECA channel attention module → the fourth ECA channel attention module → convolution module → sigmoid function → prediction map.
2. The method according to claim 1, characterized in that In step 1, a lung CT image in a specified format in a target data set is read; and a window level and a window width of the CT image are adjusted.
3. The method according to claim 1, characterized in that In step 2, the portion of the CT image with the lung nodules is cut out, and a mask image corresponding to the lung nodules is generated.
4. The method according to claim 1, characterized in that: In step 5, when training the neural network, a loss function combining binary cross entropy and confidence is used, Nadam is used as the optimization function, and the intersection-over-union ratio is used as the evaluation index of the model.
5. The method according to claim 1, characterized in that In step 7, a Markov random field is constructed for the prediction graph, and the binary terms are smoothed to improve the edge continuity and internal consistency of the segmentation effect, which specifically includes the following steps: 7-1) Construct a Markov random field in the grayscale prediction graph output by the deep learning network, treat each pixel of the image as a node in the Markov random field, and use the pixel grayscale value as the unary weight of the node; 7-2) Construct a sequential tree weighted information transfer algorithm TRW-S based on Markov random fields to perform binary smoothing on the segmentation edges, where the unary weight is the pixel value of the segmentation result generated by the segmentation network.
Citation Information
Patent Citations
Medical image automatic segmentation method based on multi-path attention fusion
CN111681252A
Unet network brain tumor MRI image segmentation method for improving attention module
CN113554669A
