A lung nodule image detection method based on deep learning

Abstract

The present invention discloses a lung nodule detection method based on deep learning, comprising the following steps: target detection of lung nodules, edge segmentation of lung nodules, and three-dimensional reconstruction of lung nodules. The lung nodule detection method based on deep learning of the present invention proposes a fast two-stage nodule detection, segmentation and classification CAD system based on 2DCNN, which can not only detect the malignancy degree of nodules, but also detect their morphological features; introduces a segmentation attention module to improve the interpretability of the model and enhance the feature extraction of small nodules; replaces the original backbone network with a lightweight network to simplify the network, reduce the number of model parameters, and lower the computational cost; after obtaining a complete nodule slice, crops the nodules to be trained, solving the problem of unbalanced positive and negative sample distribution.

Description

Technical Field

[0001] The present invention relates to a novel deep - learning - based pulmonary nodule image detection method, belonging to the field of computer - aided diagnosis technology. Background Art

[0002] Most early cancer lesions are pulmonary nodules. Therefore, the detection of pulmonary nodules is crucial for the diagnosis of early cancer. Traditional pulmonary nodule detection is carried out by doctors screening one by one through CT images, which is very time - consuming and error - prone.

[0003] With the development of deep learning and image detection technology, a large number of computer - aided diagnosis (CAD) systems based on deep learning have been developed. This system mainly helps doctors identify and locate the position of pulmonary nodules through image recognition and target detection technology, greatly shortening the search time. However, currently, most CAD systems are only used for the detection of pulmonary nodules, and CAD systems for detection, classification, and segmentation are extremely rare.

[0004] Currently, there are three major difficulties in deep - learning - based pulmonary nodule detection:

[0005] 1. The positive - negative distribution of samples is extremely unbalanced, and the sample data of pulmonary nodules of different sizes is not balanced. Conventional CT image diagnosis technology adjusts by stretching the picture size, which will cause the network to show inconsistent performance when facing pulmonary nodules of different sizes;

[0006] 2. The current three - dimensional convolutional neural network (CNN) method requires a high computational cost, which is a heavy burden on researchers;

[0007] 3. It is difficult to obtain the dataset because radiologists are required to annotate pulmonary nodule images. Summary of the Invention

[0008] The present invention provides an end - to - end deep - learning - based pulmonary nodule detection method that reduces computational cost and solves the problem of reduced accuracy caused by the inconsistent sizes of pulmonary nodules. This method achieves high accuracy and fast speed in pulmonary nodule target detection, and can use one - stage target detection to achieve a relatively high - precision segmentation of the edges of pulmonary nodules, effectively solving the imbalance of positive - negative sample distribution.

[0009] The technical solution adopted by the present invention to solve the above - mentioned technical problems is as follows:

[0010] A deep - learning - based pulmonary nodule image detection method, comprising the following steps:

[0011] S1. Obtain the CT pulmonary nodule image to be detected;

[0012] S2. Use the improved Yolo X model network to perform object detection on the pulmonary nodules in the CT pulmonary nodule images to be detected, and obtain pulmonary nodule pictures;

[0013] S3. Crop the selected pulmonary nodule images, use the improved Unet network to perform edge segmentation on the pulmonary nodule images, and obtain the pulmonary nodule pictures at the edges;

[0014] S4. Restore the output pulmonary nodule pictures of the improved Unet network in S3 to the size before inputting into the network, paste the pulmonary nodule pictures back to the original positions of the output pictures of the improved Yolo X model network in S2, obtain the results of pulmonary nodule prediction, and perform three-dimensional reconstruction on the pulmonary nodule pictures to obtain the final pulmonary nodule display results.

[0015] Furthermore, the implementation process of the improved Yolo X model specifically includes the following:

[0016] S1. Based on the lightweight Mobilenet-V3, replace the three output features for multi-scale fusion in CSPDarknet53 with three features of Mobilenet-V3, and perform multi-scale fusion on the three output features of the pulmonary nodules from large to small;

[0017] S2. Use SplitAttention to perform weight allocation on the multi-scale fused features of the pulmonary nodules, divide the features into several blocks to perform weight allocation of channel attention respectively, and then splice each block of pulmonary nodule features to obtain the multi-scale fused features of the pulmonary nodules.

[0018] Furthermore, the improved Unet specifically includes:

[0019] S1. Add an SPP module at the last step of downsampling. The SPP module outputs the features through max pooling of three different sizes, stacks the output results in the channel dimension to obtain the final pulmonary nodule features, and then uses a 1*1 convolution to restore the number of channels to the original feature channel number to obtain pulmonary nodule features with a deeper receptive field.

[0020] Furthermore, the three-dimensional reconstruction specifically includes:

[0021] S1. Obtain the bounding box information of the pulmonary nodules;

[0022] S2. Calculate the intersection over union of adjacent bounding box information. If the threshold condition is met, it is judged as the same pulmonary nodule. If the threshold condition is not met, it is judged as a new pulmonary nodule.

[0023] Compared with the prior art, the novel deep learning-based lung nodule detection method of the present invention proposes a fast two-stage nodule detection, segmentation and classification CAD (computer-aided diagnosis) system based on 2DCNN, which can not only detect the malignancy degree of nodules, but also detect their morphological features; introduce a segmentation attention module to improve the interpretability of the model and enhance the feature extraction of small nodules; replace the original backbone network with a lightweight network to simplify the network, reduce the number of model parameters and lower the computational cost; after obtaining the complete nodule slices, crop the nodules to be trained, solving the problem of unbalanced positive and negative sample distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 FIG. is the working diagram of the MSA-Yolo structure of the deep learning-based lung nodule detection method of the present invention.

[0025] Figure 2 FIG. is the working diagram of the SPP-Unet structure of the deep learning-based lung nodule detection method of the present invention.

[0026] Figure 3 FIG. is the flowchart of the lung nodule detection system of the deep learning-based lung nodule detection method of the present invention.

[0027] Figure 4 FIG. is the schematic diagram of the CSPDarknet53 structure of the deep learning-based lung nodule detection method of the present invention.

[0028] Figure 5 FIG. is the schematic diagram of the Mobilenet-V3 structure of the deep learning-based lung nodule detection method of the present invention.

[0029] Figure 6 FIG. is the segmentation attention structure diagram of the deep learning-based lung nodule detection method of the present invention.

[0030] Figure 7 FIG. is the schematic diagram of the SPP structure of the deep learning-based lung nodule detection method of the present invention.

[0031] Figure 8 FIG. is the schematic diagram of the structure of the lung nodule detection system of the deep learning-based lung nodule detection method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0033] As Figures 1 to 8 shown, a deep learning-based lung nodule detection method includes the following steps:

[0034] S1. Target detection of pulmonary nodules (MAS-Yolo): Through two-dimensional CT images of the lungs, the target detection network can accurately locate the positions of pulmonary nodules. Using YoloX as the basic framework and making a series of reasonable and effective modifications to it, a new network MSA-Yolo is obtained, which is more suitable for target detection in this scenario than the original version.

[0035] The original YoloX is more suitable for target detection tasks with changing scenarios. Therefore, the backbone feature extraction network is CSPDarknet53, which is relatively deep and has good feature extraction capabilities. However, when faced with a single CT image of a pulmonary nodule, this network appears too lengthy and consumes too much time when faced with a large number of prediction targets. Therefore, a more lightweight Mobilenet-V3 is adopted as a replacement. This network combines the advantages of its first two generations while maintaining a higher accuracy rate. Replace the three output features for multi-scale fusion in CSPDarknet53 with three features of Mobilenet-V3 for multi-scale fusion.

[0036] Decoupling is very important for feature interpretability. Since different outputs of the backbone are responsible for nodes of different sizes, after performing a concatenation operation on the backbone outputs, a split attention block is introduced to split the outputs. The split attention mechanism performs three steps of operation: splitting the features into several blocks, using the attention mechanism for weight reallocation, dividing the features, obtaining attention, and then fusing. On the one hand, it brings an improvement in the model's learning ability, and on the other hand, it enables the network to selectively choose important targets when outputting.

[0037] S2. Edge segmentation of pulmonary nodules (SPP-Unet): The pulmonary nodule images cropped within the detection frame obtained through MSA-Yolo reduce the proportion of irrelevant background in the entire image as much as possible, so that the image only contains pulmonary nodules and a small part of the background, solving the problem of unbalanced positive and negative samples in the segmentation process. To obtain a better segmentation effect again and enhance the receptive field of the network, as Figure 2 shown, add an SPP structure to the last layer of the Unet downsampling. Perform max pooling on the features in several sizes such as 3*3, 5*5, and 7*7. Stack all the pooled features and the original features in the channels, and then use a 3*3 convolution for channel feature interaction to make the number of channels and the size of the features conform to the network structure. In different pooling processes, the features extracted by the network obtain a larger receptive field, enabling it to obtain better performance. The final structure schematic diagram.

[0038] S3. Three-dimensional reconstruction of pulmonary nodules (NSS): Since pulmonary nodules are three-dimensional tissues, after detection and edge segmentation, three-dimensional reconstruction is required to determine the attribution of nodule slices. The intersection over union (IOU) of the target detection boxes of adjacent pulmonary nodules is used to calculate whether they belong to the same nodule. The formula for calculating the IOU is as follows, where (left1, right1, top1, bottom1) represents the coordinate information of the first nodule, (left2, right2, top2, bottom2) represents the coordinate information of the second nodule, and S3 represents the intersection area of the prediction boxes of the two nodules:

[0039]

[0040] If the IOU value is greater than 0.5, it is considered that these two nodules belong to the same nodule, and the information of the second nodule is added to the nodule list; if the IOU value is less than 0.5, it is considered that these two nodules do not belong to the same nodule, and the information of the second nodule is used as the initial information of the new nodule to create a new list. The pseudo-code of its implementation process is as follows, and the final output N will contain the information of each nodule.

[0041]

[0042]

[0043] In the first stage, MAS-Yolo is an improved version of Yolox. The block attention mechanism is used in it to improve the interpretability of the model for nodules of different sizes. At the same time, the lightweight network backbone Mobilenet V3 is used to replace the original CSPDarknet to simplify the network as much as possible to reduce the number of parameters. At the same time, a nodule selection scheme NSS is used to reconstruct all slices of the same nodule.

[0044] In the second stage, SPP-Unet is an improved version of Unet. After the pulmonary nodule detection and classification by MSA-Yolo, a large amount of background information can be removed, and edge detection is only performed on the target boxes containing pulmonary nodules. At this time, the effect is far better than the segmentation effect of the entire CT image. At the same time, the introduction of SPP can better enhance the perception of the network.

[0045] The entire detection and segmentation process is shown in the following figure: The picture input passes through MSA-Yolo to obtain the pulmonary nodule target boxes and category information. After cropping, the picture containing only pulmonary nodules is sent to SPP-Unet for edge detection, and then through NSS for pulmonary nodule reconstruction to obtain the final output effect.

[0046] Embodiment

[0047] On the Luna16 pulmonary nodule public dataset, MSA-Yolo (ours) has the highest sensitivity and F1 score. The remaining accuracy and MAP are only 3% and 0.46% lower than those of the best network models.

[0048] Table 1 Performance comparison of CAD detection systems on the Luna16 dataset

[0049]

[0050] In the problem of pulmonary nodule detection, MSA-Yolo has achieved good results, and the effectiveness of the MSA link has been proved through ablation experiments, and its MAP has obtained the highest score of 94.15%.

[0051] Table 2 Performance comparison of CAD detection systems on the LCS dataset

[0052]

[0053] In the overall process of pulmonary nodule detection and segmentation, MAS-Yolo + SPP-Unet has achieved the best results, and the accuracy of its segmentation ranks first among all networks.

[0054] Table 3 Performance comparison of CAD for pulmonary nodule segmentation

[0055]

[0056] In the entire MSA-Yolo + SPP-Unet, the target recognition and edge segmentation of pulmonary nodules can be accurately achieved. Through extensive testing in a certain medical center and put into practical application, this can greatly reduce the workload of radiologists.

[0057] The present invention proposes a fast two-stage nodule detection, segmentation and classification CAD system based on 2DCNN for the deep learning-based pulmonary nodule detection method. It can not only detect the malignancy degree of nodules, but also detect their morphological features; introduce a segmentation attention module to improve the interpretability of the model and enhance the feature extraction of small nodules; replace the original backbone network with a lightweight network to simplify the network and reduce the number of model parameters and calculation costs; after obtaining the complete nodule slices, crop the nodules to be trained to solve the problem of unbalanced positive and negative sample distribution.

[0058] Finally, it should be noted that the above embodiments only illustrate the technical solutions of the present invention and do not limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting lung nodule images based on deep learning, characterized in that: It includes the following steps: S1. Obtain the CT lung nodule image to be detected; S2. Use the improved Yolo X model network to perform object detection on the lung nodules in the CT lung nodule image to be detected, and obtain the lung nodule pictures; S3. Crop the selected lung nodule image, use the improved Unet network to perform edge segmentation on the lung nodule image, and obtain the lung nodule pictures at the edge; S4. Restore the output lung nodule pictures of the improved Unet network in S3 to the size before inputting into the network, paste the lung nodule pictures back to the original positions of the output pictures of the improved Yolo X model network in S2, obtain the results of lung nodule prediction, and perform three-dimensional reconstruction on the lung nodule pictures to obtain the final lung nodule display results; The implementation process of the improved Yolo X model specifically includes the following: S21. Based on the lightweight Mobilenet-V3, replace the three output features for multi-scale fusion in CSPDarknet53 with the three features of Mobilenet-V3, and perform multi-scale fusion on the three output features of the lung nodules from large to small; S22. Use SplitAttention to allocate weights to the multi-scale fused features of the lung nodules, divide the features into several blocks and perform weight allocation of channel attention respectively, and then splice each block of lung nodule features to obtain the multi-scale fused features of the lung nodules; The improved Unet specifically includes: Add an SPP module in the last step of downsampling. The SPP module outputs the features through max pooling of three different sizes, stacks the output results in the channel dimension to obtain the final lung nodule features, and then uses a 1*1 convolution to restore the number of channels to the original feature channels to obtain the lung nodule features with a deeper receptive field.

2. The method for detecting lung nodule images based on deep learning according to claim 1, characterized in that: The three-dimensional reconstruction specifically includes: S41. Obtain the bounding box information of the lung nodules; S42. Calculate the intersection over union of adjacent bounding box information. If the threshold condition is met, it is judged as the same lung nodule. If the threshold condition is not met, it is judged as a new lung nodule; S41. MAS-Yolo for object detection of lung nodules: Through the two-dimensional CT image of the lung, use the object detection network to accurately locate the position of the lung nodules. Use YoloX as the basic framework and modify it to obtain a new network MSA-Yolo.

3. The method for detecting lung nodule images based on deep learning according to claim 1, characterized in that: Since different outputs of the backbone are responsible for nodes of different sizes, after performing a concatenation operation on the backbone outputs, introduce a SplitAttention block to split the outputs. The SplitAttention mechanism divides the features into several blocks and uses the attention mechanism to reallocate weights, divides the features, and then fuses them after obtaining the attention.

4. The method for detecting lung nodule images based on deep learning according to claim 1, characterized in that: Add an SPP structure to the last layer of Unet downsampling. Max pool the features with sizes of 3*3, 5*5, 7*7, stack all the pooled features and the original features in the channel dimension, and then use a 3*3 convolution for channel feature interaction to make the number of channels and size of the features conform to the network structure.

5. The method for detecting lung nodule images based on deep learning according to claim 1, characterized in that: After detection and edge segmentation, 3D reconstruction is required to determine the attribution of lung nodule slices. The intersection over union (IOU) of the target detection boxes of adjacent lung nodules is used to calculate whether they belong to the same lung nodule. The formula for calculating the IOU is as follows, where (left1, right1, top1, bottom1) represents the coordinate information of the first lung nodule, (left2, right2, top2, bottom2) represents the coordinate information of the second lung nodule, and S3 represents the intersection area of the prediction boxes of the two lung nodules: If the IOU value is greater than 0.5, it is considered that these two lung nodules belong to the same nodule, and the information of the second lung nodule is added to the list of this lung nodule; if the IOU value is less than 0.5, it is considered that these two lung nodules do not belong to the same nodule. Using the information of the second lung nodule as the initial information of the new nodule, a new list of lung nodules is created, and the final output N will contain the information of each lung nodule.

Images