A Deep Learning-Based Fully Automated Segmentation Method and System for Lung Nodules in Lung CT Images

Abstract

This invention discloses a fully automated lung nodule segmentation system and method based on deep learning for lung CT images, including a feature extraction unit, a pixel-level detection unit, and a pixel-level segmentation unit. The feature extraction unit is used to extract image feature maps from lung CT images; the pixel-level detection unit is used to predict the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes; the pixel-level segmentation unit is used to crop candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes, and predict and output lung nodule segmentation results based on the candidate nodule regions. This invention can solve the inconsistency problem between lung nodule detection and segmentation in multi-task networks, and can achieve high segmentation performance while maintaining minimal model parameters, thus realizing fully automated lung nodule segmentation.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medicine and computer science, and specifically relates to a fully automated method and system for lung nodule segmentation in lung CT images based on deep learning. Background Technology

[0002] Lung cancer has become one of the leading causes of cancer death worldwide. Early diagnosis of lung cancer can be achieved through monitoring, analysis, and treatment using low-dose computed tomography (CT) scans of the chest, significantly improving patient survival rates. In CT images, early-stage lung cancer often presents as pulmonary nodules. Accurate segmentation of these nodules is crucial for physicians, as it provides direct evidence, including key information such as the nodule's size, shape, and location, guiding subsequent diagnosis and treatment. However, even for experienced physicians, accurately identifying pulmonary nodules from a large number of CT images remains a time-consuming task. Furthermore, the existence of numerous small nodules, and the different morphological characteristics of different nodule types, further increases the difficulty for physicians in accurately identifying and diagnosing nodules, potentially leading to missed or false diagnoses. To address this issue, computer-assisted lung cancer screening systems can be used for early automated diagnosis of cancerous lesions (i.e., nodules), ensuring accurate segmentation of pulmonary nodules while reducing the workload of physicians.

[0003] The core of the aforementioned computer-aided system lies in its lung nodule segmentation method. This method should possess good segmentation accuracy and a low rate of manual intervention to meet the practical needs of clinical diagnosis. However, compared to a whole-body lung CT image (approximately 19×29×34 cm), early-stage lung nodules are typically relatively small, generally between 3 and 30 millimeters in size. Therefore, to effectively suppress false-positive nodules caused by extensive irrelevant background (e.g., lung parenchyma, bronchi, and blood vessels), the first step is to screen the entire CT image for lung nodule candidates, followed by further segmentation of the regions containing individual candidate nodules. Current methods primarily employ semi-automatic lung nodule segmentation, which includes a manual step where the physician manually screens for candidate nodules, followed by a deep learning-based automatic segmentation algorithm to further refine the segmentation of candidate nodules in the local CT image identified by the physician. This combination of manual and automated methods reduces the workload of physicians while ensuring accurate identification and segmentation of small-sized lung nodules.

[0004] The aforementioned semi-automatic lung nodule segmentation method suffers from the problem that, due to the unclear number and distribution of lung nodules in each patient, manual screening of the entire CT image still consumes significant time and manpower, requiring highly experienced physicians to ensure screening quality. To address this issue, the traditional solution is to employ a multi-task fully automatic lung nodule segmentation method to eliminate the need for physicians to manually screen the entire CT image. Specifically, by introducing a deep learning-based automatic lung nodule detection method, candidate nodule regions can be automatically identified and cropped. Compared to manual screening, this method can eliminate useless background outside the nodule region to the greatest extent possible for subsequent segmentation tasks, thereby reducing the workload of physicians and improving segmentation performance. However, this method has limitations. It typically uses independent detection and segmentation networks in series, and poor detection performance directly affects subsequent segmentation performance. Furthermore, multiple networks result in a large overall model parameter set and long training time, impacting the overall system performance. To address this, some studies have proposed merging the detection and segmentation tasks into a single multi-task network and training it end-to-end. However, this simple combination ignores the inconsistencies between different tasks. The detection task predefines a large number of fixed bounding boxes to predict whether a lung nodule is contained within the bounding box at the object level, while the segmentation task predicts whether a pixel belongs to a lung nodule at the pixel level. These two tasks have different optimization objectives, leading to severe conflicts in the shared feature representations of the merged multi-task network, thus affecting the detection and segmentation performance of lung nodules. Although this method retains many independent parameters in each branch to reduce the proportion of shared features, mitigating the inconsistency between multiple tasks to some extent, it in turn increases the overall model size. Summary of the Invention

[0005] The purpose of this invention is to provide a fully automatic lung nodule segmentation method and system based on deep learning for lung CT images, which can achieve high lung nodule segmentation performance while maintaining minimal model parameters.

[0006] To achieve the above-mentioned objectives, this invention provides a fully automated lung nodule segmentation system based on deep learning for lung CT images, including a feature extraction unit, a pixel-level detection unit, and a pixel-level segmentation unit.

[0007] The feature extraction unit is used to extract image feature maps of lung CT images. It includes a feature extraction and fusion module, a multi-scale feature encoder, and a convolutional feature decoder. The feature extraction and fusion module extracts multi-scale feature maps by combining the positional information of low-level features and the semantic information of high-level features. The multi-scale feature encoder encodes the multi-scale feature maps separately, and the convolutional feature decoder decodes and fuses the encoded multi-scale feature maps to obtain the image feature maps.

[0008] The pixel-level detection unit is used to predict the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes, wherein the candidate nodule detection boxes are obtained by combining the nodule category probability map, the nodule size map and the nodule position offset map;

[0009] The pixel-level segmentation unit is used to crop candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes, and predict and output lung nodule segmentation results based on the candidate nodule regions.

[0010] Preferably, the feature extraction and fusion module includes a bottom-up feature extraction part and a top-down feature fusion part.

[0011] The low-to-high feature extraction part includes N consecutive downsampling convolution groups. Each downsampling convolution group includes a downsampling convolution module and an attention module. The input map is downsampled by the downsampling convolution module to obtain a feature map, and then the attention module is used to perform feature calibration in spatial and channel dimensions.

[0012] The top-down feature fusion part includes N consecutive upsampling convolutional modules and N fusion modules. One downsampling convolutional group corresponds to one upsampling convolutional module and one fusion module at one scale. Within each scale, the output of the downsampling convolutional group and the input of the upsampling convolutional module form a skip connection structure to fuse the low-level features output by the downsampling convolutional group with the high-level features output by the previous upsampling convolutional module and input them into the upsampling convolutional module of the current size for upsampling. The upsampling features are then processed by the fusion module to obtain a feature map of one scale.

[0013] Preferably, each fusion module includes multiple dilated convolutional layers with progressively increasing dilation rates. By expanding the network's receptive field, upsampled features are fused to adapt to target regions of different scales, resulting in a fused feature map.

[0014] Preferably, the multi-scale feature encoder includes multiple consecutive deformable Transformer layers. The deformable Transformer layers establish long-distance dependencies on the multi-scale feature maps to achieve feature encoding, thereby obtaining the encoded multi-scale feature maps.

[0015] Preferably, the convolutional feature decoder includes N upsampling convolutional modules. The input of each upsampling convolutional module is fused with the feature maps of different scales output by the multi-scale feature encoder through a skip connection structure, and the image feature map is obtained by sequential upsampling.

[0016] Preferably, the pixel-level detection unit includes three parallel single-layer convolutional layers and a non-maximum suppression module.

[0017] The three parallel single-layer convolutional layers predict the category probability map, size map, and position offset map of the same size as the input lung CT image based on the image feature map, and then combine the category probability map, size map, and position offset map to obtain the candidate nodule detection box.

[0018] The non-maximum suppression module excludes overlapping detection results based on all candidate nodule detection boxes to obtain the final candidate nodule detection boxes.

[0019] Preferably, the pixel-level segmentation unit includes a single-layer convolutional layer and a post-cropping strategy. The post-cropping strategy first crops candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes. Then, the cropped candidate nodule regions are scaled to a fixed large size to increase image resolution. The single-layer convolutional layer segments the fixed-size candidate nodule regions. The segmentation results are then rescaled to the size of the original candidate nodule regions. The rescaled segmentation results are stitched together at the corresponding positions in the original lung CT image to obtain the final lung nodule segmentation result.

[0020] Preferably, the feature extraction unit, pixel-level detection unit, and pixel-level segmentation unit need to undergo parameter optimization before being applied. The optimization process includes:

[0021] Collect lung CT images and annotate lung nodules, including the 3D world coordinates of the nodule center, nodule diameter, and nodule segmentation mask;

[0022] Positive sample points, negative sample points, and ignored points are sampled from each lung CT image. Positive sample points are the pixels closest to the center of the nodule; negative sample points are pixels outside the nodule region; and the remaining pixels are considered ignored.

[0023] During training, positive and negative sample points are used to calculate the classification loss for the node category probability map output by the pixel-level detection unit, and the re-focal loss function is used. Positive sample points are used to calculate the regression loss for both the nodule size map and the nodule position offset map, and the smooth L1 loss is used to calculate the regression loss for the nodule segmentation results output by the pixel-level segmentation unit. Cross-entropy is used to calculate the segmentation loss for the nodule segmentation results output by the pixel-level segmentation unit.

[0024] Preferably, the classification loss L class for:

[0025]

[0026]

[0027] Where T represents all points in the lung CT image, p t F represents the probability map of nodule categories classThe classification probability of pixel t, where α represents the weight used to adjust the classification loss, γ represents the sensitivity to the predicted probability value, and w t This represents the proportion of classification loss used to adjust different sample categories, where v represents a prediction probability threshold, P represents the set of positive sample points, N represents the set of negative sample points, and I represents the set of negligible points. The top negative sample point set N with the largest classification loss represents the top negative sample point set N. A difficult sample to form, a new point set

[0028] The nodule size diagram F d The corresponding smooth L1 loss function for:

[0029]

[0030] Where d represents the predicted value of the nodule diameter, The true value representing the diameter of the nodule;

[0031] The position offset map F offset The corresponding smooth L1 loss function for:

[0032]

[0033] Where f represents the predicted offset, This represents the true value of the center position of the nodule to be regressed, and (z,y,x) represents the offset from the center point of the nodule.

[0034] To achieve the above-mentioned objectives, this invention also provides a fully automated lung nodule segmentation method based on deep learning for lung CT images. The method uses the aforementioned fully automated lung nodule segmentation system for lung CT images and includes the following steps:

[0035] Acquire lung CT images to be detected and input them into the fully automated lung nodule segmentation system;

[0036] After extracting the image feature map of the lung CT image using the feature extraction unit, the pixel-level detection unit predicts the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes. Then, the pixel-level segmentation unit crops the candidate nodule region from the image feature map based on the identified candidate nodule detection boxes and predicts and outputs the lung nodule segmentation result based on the candidate nodule region.

[0037] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0038] The present invention proposes a fully automatic lung nodule segmentation system and method based on deep learning for lung CT images. Through the combined action of feature extraction unit, pixel-level detection unit and pixel-level segmentation unit, it can solve the problem of inconsistency between lung nodule detection and segmentation in multi-task networks. It can achieve high segmentation performance while maintaining minimal model parameters, and realize fully automatic lung nodule segmentation. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the structure of the fully automated lung nodule segmentation system based on deep learning in lung CT images provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of the feature extraction unit and the extraction of multi-scale feature maps provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of the deformable Transformer layer provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of the pixel-level detection unit and lung nodule detection provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of the pixel-level segmentation unit and lung nodule segmentation provided in an embodiment of the present invention;

[0045] Figure 6 This is a training schematic diagram of the fully automatic segmentation system provided in an embodiment of the present invention;

[0046] Figure 7 This is a flowchart of a fully automated lung nodule segmentation method based on deep learning for lung CT images provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0048] The technical concept of this invention is as follows: To address the inconsistency between detection and segmentation tasks and the large overall model parameters in existing lung nodule segmentation methods, this invention proposes a novel, highly shareable, fully automated lung nodule segmentation system that achieves high lung nodule segmentation performance while maintaining minimal model parameters. Specifically, to address the difficulty of detecting small nodules, a Transformer-based lung nodule detection architecture is proposed, effectively improving detection performance without increasing the model's parameter size. Furthermore, a post-pruning strategy is designed to effectively fuse global and local CT information, further enhancing lung nodule segmentation performance. Experimental verification on datasets demonstrates that this invention outperforms existing methods in both lung nodule detection and segmentation.

[0049] like Figure 1 As shown in the embodiment, the fully automatic lung nodule segmentation system for lung CT images includes a feature extraction unit, a pixel-level detection unit, and a pixel-level segmentation unit. The feature extraction unit is connected to the pixel-level detection unit and the pixel-level segmentation unit. The feature extraction unit is used to extract image feature maps from the lung CT images. The pixel-level detection unit is used to predict the bounding box of each pixel based on the image feature maps to locate and identify candidate nodules and obtain candidate nodule detection boxes. The pixel-level segmentation unit is used to crop candidate nodule regions from the image feature maps based on the identified candidate nodule detection boxes and predict and output lung nodule segmentation results based on the candidate nodule regions.

[0050] The construction process of the above-mentioned fully automated lung nodule segmentation system for lung CT images includes:

[0051] (a) Constructing an annotated dataset of lung nodules in annotated lung CT images:

[0052] A dataset of lung CT images was collected, in which each lung nodule was independently annotated by four radiologists. The annotations for each lung nodule included the 3D world coordinates of the nodule center, the nodule diameter, and a nodule segmentation mask. For each lung nodule, the annotation information was only selected for inclusion in the annotation dataset if it was independently identified by at least three radiologists. To ensure the accuracy of the segmentation mask, for each lung nodule in the annotation dataset, only voxels labeled by two or more radiologists were recognized as lung nodule regions. The annotation dataset was divided into training, validation, and test sets according to a specific ratio.

[0053] In the specific experiment, 888 CT images were collected as the total dataset. Among them, 1186 lung nodules were independently identified by at least three radiologists and selected for the annotation dataset. Each lung nodule was annotated in detail with the coordinates of the nodule center, its diameter, and a segmentation mask. For the segmentation mask of each lung nodule in the annotation dataset, only voxels labeled by two or more radiologists were identified as lung nodule regions. The annotation dataset was divided into training, validation, and test sets in a 5:1:4 ratio.

[0054] (b) Constructing a feature extraction unit for extracting image feature maps:

[0055] like Figure 2 As shown, the feature extraction unit comprises three main parts: a feature extraction and fusion module, a multi-scale feature encoder, and a convolutional feature decoder. The feature extraction and fusion module extracts multi-scale feature maps by combining the positional information of low-level features and the semantic information of high-level features. The multi-scale feature encoder encodes the multi-scale feature maps separately, and the convolutional feature decoder decodes and fuses the encoded multi-scale feature maps to obtain the image feature map. Specifically, the input is a 3D lung CT image, where D, H, and W represent the depth, height, and width of the image, respectively.

[0056] The feature extraction and fusion module is designed similarly to a feature pyramid structure, effectively combining the locational information of low-level features with the semantic information of high-level features, which aids in the identification of small nodules. This module comprises two parts: a bottom-up feature extraction part and a top-down feature fusion part. The bottom-up feature extraction part includes N (e.g., 4) consecutive downsampling convolutional groups. Each downsampling convolutional group includes a downsampling convolutional module and an attention module (scSE module). The downsampling convolutional module downsamples the input image to obtain a feature map, and the attention module performs spatial and channel dimension feature calibration, thereby enhancing the network's feature extraction capability.

[0057] The top-down feature fusion section includes N (e.g., 4) consecutive upsampling convolutional modules and N (e.g., 4) fusion modules. One downsampling convolutional group corresponds to one upsampling convolutional module and one fusion module at one scale. Within each scale, the output of the downsampling convolutional group and the input of the upsampling convolutional module form a skip connection structure to fuse the low-level features output by the downsampling convolutional group with the high-level features output by the previous upsampling convolutional module and input them into the upsampling convolutional module of the current size for upsampling. The upsampling features are then processed by the fusion module to obtain a feature map of one scale.

[0058] The fusion module enables the network to fully learn effective information from small nodules with a large scale range. Specifically, it includes multiple dilated convolutional layers with progressively increasing dilation rates, such as three dilated convolutions with dilation rates of 1, 2, and 5 respectively. By expanding the network's receptive field, it fuses upsampled features to adapt to target regions of different scales, resulting in high-resolution feature maps at different scales. in l represents different layers of the network, C l This indicates the number of channels in the feature map of this layer.

[0059] The multi-scale feature encoder consists of multiple consecutive deformable Transformer layers. By establishing long-distance dependencies on the multi-scale feature maps through the deformable Transformer layers, feature encoding is achieved, which overcomes the limitations of convolutional neural networks. Furthermore, features effective for small nodule detection are extracted to obtain the encoded multi-scale feature maps.

[0060] The deformable Transformer model focuses only on key sampling points, enabling the network to efficiently process multi-scale, high-resolution feature maps from feature extraction and fusion modules. like Figure 3 As shown, each deformable Transformer layer consists of a deformable attention module (DeAttention module), a feedforward module, and two normalization modules. This is used to integrate multi-scale feature maps. Before inputting the multi-scale feature encoder, it needs to be converted into a one-dimensional sequence. In the form of, However, directly flattening the feature map leads to the loss of crucial spatial location information in image segmentation. To address this issue, sine and cosine functions of different frequencies are used to encode the location coordinates of each dimension in the input feature map, as shown below:

[0061] PE {D,H,W} (pos, 2k) = sin(pos / 10000) 2k / (C / 3) )

[0062] PE {D,H,W} (pos, 2k+1) = cos(pos / 10000) 2k / (C / 3) )

[0063] Where pos represents the position in each dimension of the feature map, and k represents the channel index of the feature map. The three-dimensional position encoding PE... D PE H and PE W The concatenation results in the final position code P, which is related to the sequence. By combining these elements, spatial information is added to each location, helping the network distinguish image features at different locations within the sequence.

[0064]

[0065] To reduce the complexity of the computational model and accelerate network convergence, the DeAttention module uniformly samples a series of reference points P. q And only a small subset of key locations near these points in the feature map are calculated, instead of all locations. For each query element q, let... Indicates that it is in The corresponding content features, The normalized coordinates of the reference point are calculated by the DeAttention module as follows:

[0066]

[0067]

[0068] Where K represents the number of sampling keypoints, and N represents the number of attention heads. A(z) q ) nlqk ∈[0,1] and Represent the attention weight and sampling offset of the k-th sampling point in the l-th feature layer of the n-th attention head, respectively, both derived by adjusting z. q The linear mapping is used to obtain the result, where The normalized coordinates have been readjusted. This adapts the input feature map of layer l to the input feature map of layer l. Concat(·) represents the concatenation operation, Φ(·) and Ψ(·) represent the linear mapping layers respectively, where Ψ(·) is used to perform weighted fusion of the outputs of all attention heads.

[0069] The output of the DeAttention module is passed through a normalization module, then fed into a feedforward module, and finally through another normalization module to generate the output of a single deformable Transformer layer. To prevent gradient vanishing, skip connections are used between each module. Six deformable Transformer layers are stacked together to form a multi-scale feature encoder for multi-scale feature extraction. Finally, feature maps from different layers output by the multi-scale feature encoder are obtained.

[0070] The convolutional feature decoder includes N (e.g., 4) upsampling convolutional modules. The input of each upsampling convolutional module is fused with the feature maps of different scales output by the multi-scale feature encoder through a skip connection structure. The image feature maps extracted by the entire feature extraction unit are obtained by sequential upsampling.

[0071] (c) Constructing pixel-level detection units for lung nodule detection:

[0072] like Figure 4 As shown, the pixel-level detection unit includes three parallel single-layer convolutional layers and a non-maximum suppression module. The three parallel single-layer convolutional layers predict class probability maps of the same size as the input lung CT image based on the image feature maps. Dimensions and position offset diagram The category probability map is a binary classification probability map, where each voxel represents the probability p that the voxel point is the center point of the lung nodule. In the size map, each voxel represents the diameter d of the lung nodule. In the position offset map, the three channels of each voxel represent the position offsets z, y, and x from the center point of the lung nodule, respectively, to obtain a more accurate nodule center point. Finally, the category probability map, size map, and position offset map are combined to obtain candidate nodule detection boxes. The non-maximum suppression module eliminates overlapping detection results based on all candidate nodule detection boxes to obtain the final candidate nodule detection boxes for subsequent segmentation tasks.

[0073] The specific steps are as follows: First, select the J points with the highest probability values ​​from the category probability map to form a set of candidate positive samples. Each candidate sample includes its corresponding classification probability value. and location coordinates The dimension drawing can be obtained from the position coordinates. and position offset map Diameter of the corresponding nodule and offset vector Finally, the prediction of the candidate samples can be written as After eliminating overlap based on nonmaximum suppression using 3D IoU, the final accurate lung nodule detection value is obtained and used for subsequent segmentation tasks.

[0074] (d) Construct pixel-level segmentation units for lung nodule segmentation:

[0075] like Figure 5As shown, the pixel-level segmentation unit includes a single convolutional layer and a post-cropping strategy. It shares the same global image feature map as the pixel-level detection unit, which is the image feature map extracted from the entire CT image by the feature extraction unit, to better predict the semantic information of the foreground and background. A single convolutional layer is used to segment lung nodules based on the global image feature map to reduce the complexity of the entire system and decrease the inconsistency between the detection and segmentation tasks. To further improve segmentation accuracy, especially for smaller lung nodules, a post-cropping strategy is introduced. The post-cropping strategy first crops candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes. Then, the cropped candidate nodule regions are scaled to a fixed large size to increase image resolution. The single convolutional layer segments the fixed-size candidate nodule regions, and the segmentation results are rescaled to the original candidate nodule region size. The rescaled segmentation results are then stitched together at the corresponding positions in the original lung CT image to obtain the final lung nodule segmentation probability map. This method serves as the result of lung nodule segmentation. It allows the segmentation task to comprehensively consider both global and local CT information of lung nodules, thereby improving the performance of lung nodule segmentation.

[0076] (e) Training of the fully automated lung nodule segmentation system:

[0077] During the network training phase, this invention employs a novel positive and negative sample point sampling strategy. Unlike anchor-based target detection methods that select training positive and negative sample boxes based on the overlap between the anchor box and the ground truth box, this strategy selects training positive and negative sample points based on the distance from all pixels in the input lung CT image to the center point of the labeled lung nodule. This strategy categorizes sample points in the image into three types: positive sample points, negative sample points, and negligible points. Positive sample points are the pixels closest to the center point of the nodule; negative sample points are pixels outside the nodule region; and the remaining pixels are considered negligible.

[0078] The specific operation steps are as follows: 1) Calculate the distance from all pixels in the image to the center point of all lung nodules based on the true values. 2) Consider the k pixels closest to the center as the center points of the nodules and select them as the positive sample point set P. Since the tissue textures within the nodule region are relatively similar and easily confused, in order to effectively distinguish them from the positive sample points, select points whose distance is greater than the radius of the center point as the negative sample point set N. Finally, select all points between the positive and negative samples as the negligible point set I. 3) Since the proportion of the lung nodule region in the entire image is very small, there is an extreme imbalance between the positive and negative samples. The online hard sample mining strategy OHEM is adopted to retain the negative sample point set N with the largest classification loss. One point is designated as a difficult sample, and the remaining points are merged into the negligible point set I to form a new point set. They do not participate in network training.

[0079] During training, such as Figure 6 As shown, for the node category probability map output by the pixel-level detection unit, in order to further balance positive and negative samples, positive sample points and negative sample points are used, and the classification loss L is calculated using the re-focal loss function. class for:

[0080]

[0081]

[0082] Where T represents all points in the lung CT image, p t F represents the probability map of nodule categories class The classification probability of pixel t, where α represents the weight used to adjust the classification loss, γ represents the sensitivity to the predicted probability value, and w t This represents the proportion of classification loss used to adjust different sample categories, and v represents a prediction probability threshold.

[0083] For both the nodule size map and the nodule position offset map, positive sample points are used and the regression loss is calculated using smooth L1 loss. The calculation is performed only on all positive sample points to regress the diameter d of the target and the offset (z,y,x) from the target center point.

[0084] Nodule size diagram F d The corresponding smooth L1 loss function for:

[0085]

[0086] Where d represents the predicted value of the nodule diameter, The true value representing the diameter of the nodule;

[0087] Position offset diagram F offset The corresponding smooth L1 loss function for:

[0088]

[0089] Where f represents the predicted offset, This represents the true value of the center location of the nodule to be regressed.

[0090] Ultimately, the detection network can pass the total loss function L. det Optimize:

[0091] L det =L class +L size +L offset

[0092] Segmentation probability map output by pixel-level segmentation units The segmentation loss L is calculated using the cross-entropy loss function for all voxels within the labeled lung nodules. seg .

[0093]

[0094] in, p represents the segmentation prediction probability value. s N represents the truth value of the split. all σ represents the number of all voxels, and σ(·) represents the sigmoid function.

[0095] Ultimately, the entire network passes through a total loss function L with a balancing parameter λ. total End-to-end optimization was performed to train the fully automated lung nodule segmentation system.

[0096] L total =L det +λL seg

[0097] In the specific experiments, the model was trained on a GTX 2080 graphics card using the PyTorch platform. During training, a stochastic gradient descent optimizer was used to optimize the network parameters, with momentum set to 0.9 and weight decay coefficient set to 0.0001. The number of training iterations was set to 170, the batch size to 4, and a learning rate decay strategy was employed: the initial learning rate was 0.001, and when the number of iterations exceeded 100 and 150, the learning rate was reduced to 0.1 times its original value. Furthermore, a series of data augmentation techniques were used to enhance the model's robustness, including horizontal flipping, rotation, and swapping. During network training, the hyperparameters were set to K=4, N=6, J=800, k=7, α=0.75, γ=2, v=0.9, and λ=1.

[0098] After training, a fully automated lung nodule segmentation system was obtained.

[0099] A well-trained fully automated lung nodule segmentation system can achieve fully automated segmentation of lung nodules in lung CT images, such as... Figure 7 As shown, the specific process is as follows:

[0100] Acquire lung CT images to be detected and input them into the fully automated lung nodule segmentation system;

[0101] After extracting the image feature map of the lung CT image using the feature extraction unit, the pixel-level detection unit predicts the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes. Then, the pixel-level segmentation unit crops the candidate nodule region from the image feature map based on the identified candidate nodule detection boxes and predicts and outputs the lung nodule segmentation result based on the candidate nodule region.

[0102] The specific experiment is as follows:

[0103] (a) Evaluation criteria:

[0104] The performance of the proposed algorithm in lung nodule detection was evaluated using the official evaluation metric FROC curve of the LUNA16 dataset. Here, the ordinate represents the detection sensitivity, and the abscissa represents the average number of false positives per CT image (FPs, FP / Scan). Sensitivity is defined as:

[0105] Sensitivity = TP / (TP + FN)

[0106] Where TP represents the number of true positives and FN represents the number of false negatives. LUNA16 selected the average sensitivity of FPs at 1 / 8, 1 / 4, 1 / 2, 1, 2, 4, and 8 as the comprehensive evaluation index CPM. A higher value indicates better detection performance of the model.

[0107]

[0108] For segmentation performance evaluation, the segmentation metrics Dice similarity coefficient and Intersection over Union (IoU) are used to measure the similarity between the lung nodule segmentation results predicted by the model and the ground truth.

[0109]

[0110]

[0111] Where X and Y represent the predicted segmentation result and the actual segmentation result, respectively.

[0112] (b) Ablation Experiments: To evaluate the effectiveness of each key component and training strategy used in the fully automated lung nodule segmentation system, ablation experiments were conducted. The network parameters of the pixel-level segmentation units were fixed, and the network was optimized using only the detection loss. The base model consisted only of convolutional layers, without the scSE module, positive and negative sample point sampling strategy, and multi-scale feature encoder. The base model was compared with the following improved methods: 1) base model + scSE module; 2) base model + scSE module + positive and negative sample point sampling strategy; 3) base model + scSE module + positive and negative sample point sampling strategy + multi-scale feature encoder. The experimental results are shown in Table 1.

[0113] Table 1 Comparison of detection performance under different modules and training strategies

[0114]

[0115] Experimental results show that by introducing the scSE module, positive and negative sample sampling strategy, and multi-scale feature encoder, superior lung nodule detection performance can be achieved compared to the basic model. In particular, the sensitivity of all FPs is improved after adding the multi-scale feature encoder, especially at FP / s = 0.125, 0.25, and 0.5, demonstrating the effectiveness of the multi-scale feature encoder in detecting small nodules.

[0116] (c) Comprehensive Performance Comparison Experiment: To evaluate the accuracy of lung nodule detection in the fully automated lung nodule segmentation system, this invention was compared with several state-of-the-art lung nodule detection methods, including anchor-based LeakyNoisy-OR, Deeplung, NoduleNet, DeepSEED, and SANet, as well as non-anchor-based CPM-Net and SCPM-Net. The experimental results are shown in Table 2.

[0117] Table 2 Comparison of detection performance of different lung nodule detection methods

[0118]

[0119] Among them, LeakyNoisy-OR refers to the method in "Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network". Deeplung refers to the method in "Deeplung: Deep3d dual path nets for automated pulmonary nodule detection and classification". NoduleNet refers to the method in "Nodulenet: Decoupled false positive reduction for pulmonary nodule detection and segmentation". DeepSEED refers to the method in "DeepSEED: 3Dsqueeze-and-excitation encoder-decoder convolutional neural networks for pulmonary nodule detection". SANet refers to the method in "SANet: Aslice-aware network for pulmonary nodule detection". CPM-Net refers to the method in "CPM-Net: A3D center-points matching network for pulmonary nodule detection in CT scans". SCPM-Net refers to the method described in "SCPM-Net: An anchor-free 3D lung nodule detection network using sphere representation and center points matching".

[0120] Experimental results show that this invention achieves the optimal CPM value compared to state-of-the-art anchor-based and anchorless lung nodule detection methods. Furthermore, this invention outperforms other methods in sensitivity across almost all FPs, validating its superiority in detecting both small and large lung nodules.

[0121] (d) Comprehensive Comparison Experiment on Segmentation Performance: To evaluate the accuracy of the fully automated lung nodule segmentation system, two evaluation methods were selected: Local segmentation performance was evaluated at the individual lung nodule level, i.e., segmentation indices were calculated in each local CT image containing a lung nodule. This evaluation method comprehensively measures the sensitivity of lung nodule detection and the segmentation performance for individual lung nodules; Global segmentation performance was evaluated at the individual CT image level, i.e., segmentation indices were calculated in the entire CT image containing all lung nodules and the background. This evaluation method comprehensively measures the sensitivity of lung nodule detection and the segmentation performance at the global scale. This invention was compared with various lung nodule segmentation methods, including semi-automatic segmentation methods with manual intervention such as 3DUNet, DS-CMSF, and HAUNet, and fully automated segmentation methods such as SCPM+HAUNet and NoduleNet. The experimental results are shown in Table 3.

[0122] Table 3. Comparison of the overall performance of different lung nodule segmentation methods

[0123]

[0124]

[0125] In this document, 3D UNet represents the method described in "3D U-Net: learning dense volumetric segmentation from sparse annotation," DS-CMSF represents the method described in "A cascaded multi-stage framework for automatic detection and segmentation of pulmonary nodules in developing countries," and HAUNet represents the method described in "HAUNet-3D: a novel hierarchical attention 3DUNet for lung nodule segmentation." For a semi-automatic segmentation method with manual intervention, evaluated at the individual lung nodule level, this invention achieved the highest segmentation score, validating the superiority of the segmentation network for segmenting individual lung nodules in local CT scans. For a fully automatic segmentation method evaluated at the individual lung nodule level, this invention also outperformed the comparative methods in all metrics, demonstrating its superior overall performance in lung nodule detection and segmentation. In evaluations at the individual CT image level, despite considering the detection of partially predicted false positive and false negative nodules, this invention still outperformed the comparative methods, validating its superiority in detection and segmentation tasks. Furthermore, by comparing the model parameters of different lung nodule segmentation methods, the model parameters of the present invention are almost smaller than those of all the comparative methods, demonstrating the superiority of the present invention in the training and deployment of the lung nodule segmentation system.

[0126] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fully automated lung nodule segmentation system based on deep learning for lung CT images, characterized in that, It includes a feature extraction unit, a pixel-level detection unit, and a pixel-level segmentation unit; The feature extraction unit is used to extract image feature maps from lung CT images, and includes feature extraction. The system includes a fusion module, a multi-scale feature encoder, and a convolutional feature decoder. The feature extraction and fusion module extracts multi-scale feature maps by combining the positional information of low-level features and the semantic information of high-level features. The multi-scale feature encoder encodes the multi-scale feature maps separately, and the convolutional feature decoder decodes and fuses the encoded multi-scale feature maps to obtain the image feature map. The pixel-level detection unit is used to predict the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes, wherein the candidate nodule detection boxes are obtained by combining the nodule category probability map, the nodule size map and the nodule position offset map; The pixel-level segmentation unit is used to crop candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes, and predict and output lung nodule segmentation results based on the candidate nodule regions. The feature extraction and fusion module includes a bottom-up feature extraction part and a top-down feature fusion part. The low-to-high feature extraction part includes N consecutive downsampling convolution groups. Each downsampling convolution group includes a downsampling convolution module and an attention module. The input map is downsampled by the downsampling convolution module to obtain a feature map, and then the attention module is used to perform feature calibration in spatial and channel dimensions. The top-down feature fusion part includes N consecutive upsampling convolutional modules and N fusion modules. One downsampling convolutional group corresponds to one upsampling convolutional module and one fusion module at one scale. Within each scale, the output of the downsampling convolutional group and the input of the upsampling convolutional module form a skip connection structure to fuse the low-level features output by the downsampling convolutional group with the high-level features output by the previous upsampling convolutional module and input them into the upsampling convolutional module of the current size for upsampling. The upsampling features are then processed by the fusion module to obtain a feature map of one scale. The pixel-level segmentation unit includes a single-layer convolutional layer and a post-cropping strategy. The post-cropping strategy first crops candidate nodule regions from the image feature map based on the identified candidate nodule detection boxes. Then, the cropped candidate nodule regions are scaled to a fixed large size to increase image resolution. The single-layer convolutional layer segments the fixed-size candidate nodule regions. The segmentation results are then rescaled to the size of the original candidate nodule regions. The rescaled segmentation results are stitched together at the corresponding positions in the original lung CT image to obtain the final lung nodule segmentation result.

2. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 1, characterized in that, Each fusion module includes multiple dilated convolutional layers with progressively increasing dilation rates. By expanding the network's receptive field, upsampled features are fused to adapt to target regions of different scales, resulting in a fused feature map.

3. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 1, characterized in that, The multi-scale feature encoder includes multiple consecutive deformable Transformer layers. By establishing long-distance dependencies on the multi-scale feature maps through the deformable Transformer layers, feature encoding is achieved, resulting in encoded multi-scale feature maps.

4. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 1, characterized in that, The convolutional feature decoder includes N upsampling convolutional modules. The input of each upsampling convolutional module is fused with the feature maps of different scales output by the multi-scale feature encoder through a skip connection structure, and the image feature map is obtained by sequential upsampling.

5. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 1, characterized in that, The pixel-level detection unit includes three parallel single-layer convolutional layers and a non-maximum suppression module. The three parallel single-layer convolutional layers predict the category probability map, size map, and position offset map of the same size as the input lung CT image based on the image feature map, and then combine the category probability map, size map, and position offset map to obtain the candidate nodule detection box. The non-maximum suppression module excludes overlapping detection results based on all candidate nodule detection boxes to obtain the final candidate nodule detection boxes.

6. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 1, characterized in that, The feature extraction unit, pixel-level detection unit, and pixel-level segmentation unit need to undergo parameter optimization before being applied. The optimization process includes: Collect lung CT images and annotate lung nodules, including the 3D world coordinates of the nodule center, nodule diameter, and nodule segmentation mask; Positive sample points, negative sample points, and ignored points are sampled from each lung CT image. Positive sample points are the pixels closest to the center of the nodule; negative sample points are pixels outside the nodule region; and the remaining pixels are considered ignored. During training, positive and negative sample points are used to calculate the classification loss for the node category probability map output by the pixel-level detection unit, and the re-focal loss function is used. Positive sample points are used to calculate the regression loss for both the nodule size map and the nodule position offset map, and the smooth L1 loss is used to calculate the regression loss for the nodule segmentation results output by the pixel-level segmentation unit. Cross-entropy is used to calculate the segmentation loss for the nodule segmentation results output by the pixel-level segmentation unit.

7. The fully automated lung nodule segmentation system based on deep learning for lung CT images according to claim 6, characterized in that, The classification loss L class for: in, Represents all points in a lung CT image. Probability map of nodule categories medium pixel The classification probability, This represents the weights used to adjust the classification loss. This indicates the sensitivity to the predicted probability value. This indicates the proportion of classification loss used to adjust different sample categories. This represents a prediction probability threshold. P Represents the set of positive sample points. N Represents the set of negative sample points. I Indicates a set of points that can be ignored. Represents the negative sample point set The largest loss in the middle classification A difficult sample to form, a new point set ; The nodule size diagram The corresponding smooth L1 loss function for: in, This represents the predicted value of the nodule diameter. The true value representing the diameter of the nodule; The position offset map The corresponding smooth L1 loss function for: in, This indicates the predicted offset. This represents the true value of the center location of the nodule to be regressed. , , ) represents the offset from the center point of the nodule.

8. A fully automated segmentation method for lung nodules in lung CT images based on deep learning, characterized in that, The method uses the fully automated lung nodule segmentation system for lung CT images according to any one of claims 1-7, and includes the following steps: Acquire lung CT images to be detected and input them into the fully automated lung nodule segmentation system; After extracting the image feature map of the lung CT image using the feature extraction unit, the pixel-level detection unit predicts the bounding box of each pixel based on the image feature map to locate and identify candidate nodules and obtain candidate nodule detection boxes. Then, the pixel-level segmentation unit crops the candidate nodule region from the image feature map based on the identified candidate nodule detection boxes and predicts and outputs the lung nodule segmentation result based on the candidate nodule region.

Images