Intelligent image processing system for pulmonary nodules

Abstract

The present invention provides an intelligent image processing system for lung nodules, comprising: a module M1: acquiring 3D lung CT image data; a module M2: preprocessing the acquired 3D lung CT image data to obtain preprocessed 3D lung CT image data; a module M3: constructing a one-stage lung nodule 3D instance segmentation model, and training the one-stage lung nodule 3D instance segmentation model to obtain a trained one-stage lung nodule 3D instance segmentation model; a module M4: using the preprocessed lung nodule CT image data to generate lung nodule segmentation and detection results according to the trained one-stage lung nodule 3D instance segmentation model; a module M5: determining the central position and diameter of the lung nodule according to the lung nodule segmentation and detection results, and outlining the lung nodule boundary.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent medical image processing, and in particular to an intelligent image processing system for pulmonary nodules. Background Art

[0002] Lung nodule recognition is one of the specific applications in the field of deep learning computer vision. General computer vision tasks can be used for lung nodule recognition, but because lung nodule recognition tasks focus more on positioning information and have specific imaging patterns, there are many specialized computer vision models to handle lung nodule recognition tasks. Among them, the lung nodule recognition method based on instance segmentation can directly provide the boundary contour of the lung nodule, which is convenient for doctors to view. Instance segmentation not only distinguishes voxel points of different categories, but also distinguishes whether voxel points of the same category belong to the same instance.

[0003] The model used for pulmonary nodule instance segmentation can outline the nodule boundary while giving the nodule location coordinates, providing richer nodule information. The pulmonary nodule instance segmentation task is divided into one-stage pulmonary nodule instance segmentation and two-stage pulmonary nodule instance segmentation. The one-stage model can output the detection of nodule boundaries end-to-end; the two-stage model first detects potential nodule areas, and then further detects and locates the candidate areas. E. Kopelowitz et al.'s "Lung Nodules Detection and Segmentation Using 3D Mask-RCNN" proposed a 3D pulmonary nodule instance segmentation scheme based on Mask R-CNN, but the shared feature map of the detection branch and segmentation branch of the Mask R-CNN model caused their respective performance to decline compared with the single-task model. H. Tang et al.'s "NoduleNet: Decoupled False Positive Reduction for Pulmonary Nodule Detection and Segmentation" proposed a model called NoduleNet, which decouples the detection branch and the segmentation branch. The model achieved good instance segmentation performance on the LIDC-IDRI dataset.

[0004] At present, most of the existing lung nodule instance segmentation models are based on Mask R-CNN, which adopts a two-stage approach, making the model structure more complicated, requiring complex post-processing, and poor interpretability. Existing lung nodule recognition models do not make good use of three-dimensional (3D) spatial information, ignore the relationship between lesions and lung environment, and most of them have poor shape representation capabilities for three-dimensional entities.

[0005] Patent document CN111062955A (application number: 202010188475.3) discloses a lung CT image data segmentation method and system, which belongs to the field of medical image processing and artificial intelligence technology. The method includes: marking the lung contour and target area, and performing numerical clipping and normalization processing on the image data; using a first neural network for training and learning to obtain a lung contour segmentation model; extracting the lung contour, determining the lung region of interest, and performing numerical clipping and normalization processing; clipping the lung region of interest, and using a second neural network for training and learning to obtain a lung target area segmentation model; according to the image data and the lung target area segmentation model, the lung target area is segmented. The system includes a labeling module, a first clipping and normalization module, a first training and learning module, a second clipping and normalization module, a second training and learning module, and a detection module. The present invention improves the processing efficiency of lung CT image data and can quickly segment the target area in the lung CT image data. Summary of the invention

[0006] In view of the defects in the prior art, an object of the present invention is to provide an intelligent image processing system for lung nodules.

[0007] According to the present invention, an intelligent image processing system for pulmonary nodules includes:

[0008] Module M1: Acquisition of 3D lung CT image data;

[0009] Module M2: preprocessing the acquired 3D lung CT image data to obtain preprocessed 3D lung CT image data;

[0010] Module M3: constructing a one-stage pulmonary nodule 3D instance segmentation model, and training the one-stage pulmonary nodule 3D instance segmentation model to obtain a trained one-stage pulmonary nodule 3D instance segmentation model;

[0011] Module M4: Generate lung nodule segmentation and detection results using the preprocessed lung nodule CT image data according to the trained one-stage lung nodule 3D instance segmentation model;

[0012] Module M5: Determine the center position and diameter of the lung nodule based on the lung nodule segmentation and detection results, and outline the boundary of the lung nodule;

[0013] The one-stage lung nodule 3D instance segmentation model is a lung nodule real segmentation model that uses 3D lung CT image data as input. The 3D data is used as the input of the one-stage lung nodule 3D instance segmentation model to extract CT frame information, and the lung nodule segmentation and detection results are obtained simultaneously in an end-to-end manner.

[0014] Preferably, in the module M2: standard voxel conversion, lung window selection and lung parenchyma extraction are performed on the patient's CT image data to obtain first-stage lung nodule CT image data with consistent voxel standards and brightness.

[0015] Preferably, the standard voxel conversion uses trilinear interpolation to restore the actual spatial size represented by each voxel to a preset value.

[0016] Preferably, the lung window selection adopts: adjusting the width of the lung window by truncation so that the brightness range of the lung image data is within a preset range.

[0017] Preferably, the lung parenchyma extraction adopts:

[0018] Module M2.1: Filter lung CT data to remove noise from the data;

[0019] Module M2.2: Binarize the filtered lung CT data to distinguish between areas with high brightness and areas with low brightness;

[0020] Module M2.3: Through graphics processing, the area outside the lung parenchyma is set to 0, and the maximum connected component of the geometrically non-enclosed area with a value of 0 is set to 1, so as to separate the area outside the CT field of view and the background of the lung parenchyma area; then the whole image is inverted to extract the lung parenchyma area, and image 1 is obtained;

[0021] Module M2.4: Obtain the contour of the lung parenchyma area by calculating the convex hull of the area with the current value of 1, and include the nodule area in the lung parenchyma to obtain image 2;

[0022] Module M2.5: Using images 1 and 2, filter out the connected components with values ​​of 0 in the convex hull of the lung parenchyma that meet the preset requirements;

[0023] Module M2.6: Ensure that both lungs are located in the connected components of the lung parenchyma through expansion and closing operations;

[0024] Module M2.7: Use the lung parenchyma connected components to mask the original image and obtain the final lung parenchyma area.

[0025] Preferably, the one-stage lung nodule 3D instance segmentation model adopts: a pre-extraction module, a first-type dense connection module, a down-sampling module, a second-type dense connection module, an up-sampling module and a region recommendation module;

[0026] Module S1: In the pre-extraction module, 3D data is pre-extracted and down-sampled to obtain an initial feature map;

[0027] Module S2: The initial feature map is extracted through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map;

[0028] Module S3: extract features from the extracted feature map through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map, and repeatedly trigger module S3 to make the feature semantics of the feature map reach the highest level of abstraction;

[0029] Module S4: restore the features of the feature map with the highest level of feature semantics through a combination module of the second type of dense connection block and the upsampling block to obtain a restored feature map, and repeatedly trigger module S4 until the resolution of the feature map is restored to the resolution when it was input into module S3;

[0030] The feature semantics reaching the highest level of abstraction means that the resolution reaches a preset value.

[0031] Preferably, the first-stage lung nodule 3D instance segmentation model adopts:

[0032] In the pre-extraction module: a 3*3*3 convolution operation is used to perform preliminary feature perception on the input feature map, and feature extraction is performed through a 3*3*3 standard convolution module, and finally a 2*2*2 maximum pooling is used to extract the most critical information that meets the preset requirements in each neighborhood to obtain a preliminary feature map;

[0033] In the first type of densely connected module: the preliminary feature map is passed through a 1*1*1 standard convolution module, and then through a 3*3*3 standard convolution module for feature extraction to obtain a 16-channel level 1 feature map, which is cascaded with the preliminary feature map, and then repeatedly passed through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map and the level 1 feature map, and repeatedly triggered until a preset level feature map is output;

[0034] In the downsampling module, the channels of the feature map with a preset number of levels are halved by a 1*1*1 standard convolution, and a 2*2*2 maximum pooling is performed;

[0035] In the second type of densely connected module, the preliminary feature map is passed through a 1*1*1 standard convolution module, and then through a 3*3*3 standard convolution module for feature extraction to obtain a 16-channel level 1 feature map, which is cascaded with the initial feature map, and then repeatedly passed through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map, and repeatedly triggered until a preset number of levels of feature maps are obtained and the cascade results of 16-channel feature maps at each level are output;

[0036] In the upsampling module, the concatenation results of the 16-channel feature maps at each level are subjected to 2*2*2 deconvolution and activated to obtain a feature map of restored features;

[0037] In the region recommendation module, the concatenation result of the preset level 16-channel feature map is used to generate a 128-channel feature map through a 3*3*3 standard convolution module, and then a 61-channel feature map is generated through a 1*1*1 standard convolution module as the final segmentation and detection result;

[0038] The standard convolution module includes batch normalization, nonlinear activation, and convolution operations of preset size.

[0039] Preferably, the one-stage lung nodule 3D instance segmentation model loss function includes positive and negative classification loss, positive sample bounding box regression loss and positive sample segmentation loss.

[0040] Preferably, the pulmonary nodule segmentation and detection results include: 60-channel anchor frame information and 1-channel nodule heat map;

[0041] The anchor frame information indicates that each position has a preset number of anchor frames of different sizes, and each anchor frame has a preset number of attributes, including three-dimensional center coordinates, anchor frame diameter, and confidence that the anchor frame contains a nodule;

[0042] The nodule heatmap indicates the confidence that each location belongs to the lung nodule region.

[0043] Preferably, in the module M5,

[0044] Non-maximum suppression module: All anchor frames are sorted by confidence, the largest anchor frame is taken out and added to the positive anchor frame set, and the anchor frames whose intersection-over-union ratio with the current anchor frame is below the set threshold are retained; the remaining anchor frames are further sorted by confidence, and repeatedly triggered to finally obtain a set of positive anchor frames whose intersection-over-union ratios between each pair are below the set threshold, and the detected non-overlapping nodule lesions are obtained;

[0045] Nodule mask generation module: The nodule resolution is restored to the size of the input data of the first-stage lung nodule 3D instance segmentation model through trilinear interpolation, and the threshold of mask binarization is selected according to the binary method to minimize the error between the binarized result and the real labeled mask. The mask of each anchor frame and each frame is then filled into a convex set to ensure the integrity of the mask.

[0046] Nodule boundary outlining module: used to generate and outline boundaries based on anchor box information and mask information.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention performs instance segmentation of lung nodules based on a 3D one-stage method, achieving both detection and segmentation tasks end-to-end, and the system is more concise;

[0049] 2. The present invention designs preprocessing, data block selection, anchor frame design, false positive attenuation, and mask post-processing solutions suitable for pulmonary nodule image recognition to improve the system's processing capabilities for pulmonary nodules;

[0050] 3. Through the graphics processing method suitable for lung nodule image recognition, the lung parenchyma is accurately extracted; by designing 12 sizes of anchor frames, the potential maximum intersection-overlap ratio of each nodule with all anchor frames is the same as much as possible, that is, the intersection-overlap ratio consistency; the false positives are attenuated through the non-maximum suppression module, and non-overlapping nodule lesions are obtained at the same time; the lung nodule segmentation results are obtained by generating nodule masks, which is convenient for doctors to check the location and shape of nodules;

[0051] 4. The present invention combines the results of the detection task and the segmentation task to obtain the final result of instance segmentation, which effectively suppresses false positives. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0053] Figure 1 This is the one-stage lung nodule 3D instance segmentation model structure.

[0054] Figure 2 It is the submodule structure of the one-stage lung nodule 3D instance segmentation model. DETAILED DESCRIPTION

[0055] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0056] In view of the problems that the existing lung nodule detection model has a complicated structure, requires complicated post-processing, and has poor interpretability; it does not make good use of three-dimensional spatial information, and has insufficient recognition capabilities for lung nodules with unclear boundaries and low density, as well as lung tissue areas with shapes similar to nodules, the present invention provides a three-dimensional (3D) lung nodule intelligent image processing method, that is, a one-stage lung nodule 3D instance segmentation diagnosis model using 3D lung CT image data as input. Using 3D data as the input of the model can better extract CT inter-frame information; the results of lung nodule segmentation and detection are obtained simultaneously in an end-to-end manner, with the characteristics of both segmentation models and detection models, and the outline of nodule boundaries is generated in a simpler way.

[0057] Example 1

[0058] According to the present invention, an intelligent image processing system for pulmonary nodules includes:

[0059] Module M1: Acquisition of 3D lung CT image data;

[0060] Module M2: preprocessing the acquired 3D lung CT image data to obtain preprocessed 3D lung CT image data;

[0061] Module M3: constructing a one-stage pulmonary nodule 3D instance segmentation model, and training the one-stage pulmonary nodule 3D instance segmentation model to obtain a trained one-stage pulmonary nodule 3D instance segmentation model;

[0062] Module M4: Generate lung nodule segmentation and detection results using the preprocessed lung nodule CT image data according to the trained one-stage lung nodule 3D instance segmentation model;

[0063] Module M5: Determine the center position and diameter of the lung nodule based on the lung nodule segmentation and detection results, and outline the boundary of the lung nodule;

[0064] The one-stage lung nodule 3D instance segmentation model is a lung nodule real segmentation model that uses 3D lung CT image data as input. The 3D data is used as the input of the one-stage lung nodule 3D instance segmentation model to extract CT frame information, and the lung nodule segmentation and detection results are obtained simultaneously in an end-to-end manner.

[0065] Specifically, in the module M2: standard voxel conversion, lung window selection and lung parenchyma extraction are performed on the patient's CT image data to obtain first-stage lung nodule CT image data with consistent voxel standards and brightness.

[0066] Specifically, the standard voxel conversion adopts: trilinear interpolation to restore the actual space size represented by each voxel to 1mm*1mm*1mm.

[0067] Specifically, the lung window selection adopts: in order to ensure the brightness consistency of the CT image, the lung window width is adjusted by truncation so that the brightness range of the lung image data is between -1024 and 1024.

[0068] Specifically, the lung parenchyma extraction adopts: extracting the lung parenchyma in the lung image data by using a graphics processing method;

[0069] Module M2.1: Filter lung CT data to remove noise from the data;

[0070] Module M2.2: Binarize the filtered lung CT data to distinguish between areas with high brightness and areas with low brightness;

[0071] Module M2.3: Through graphics processing, the area outside the lung parenchyma is set to 0, and the maximum connected component of the geometrically non-enclosed area with a value of 0 is set to 1, so as to separate the area outside the CT field of view and the background of the lung parenchyma area; then the whole image is inverted to extract the lung parenchyma area, and image 1 is obtained;

[0072] Module M2.4: Obtain the contour of the lung parenchyma area by calculating the convex hull of the area with the current value of 1, and include the nodule area in the lung parenchyma to obtain image 2;

[0073] Module M2.5: Using images 1 and 2, filter out the connected components with values ​​of 0 in the convex hull of the lung parenchyma that meet the preset requirements;

[0074] Module M2.6: Ensure that both lungs are located in the connected components of the lung parenchyma through expansion and closing operations;

[0075] Module M2.7: Use the lung parenchyma connected components to mask the original image and obtain the final lung parenchyma area.

[0076] Specifically, the one-stage lung nodule 3D instance segmentation model adopts: a pre-extraction module, a first-type dense connection module, a down-sampling module, a second-type dense connection module, an up-sampling module, and a region recommendation module;

[0077] Module S1: In the pre-extraction module, 3D data is pre-extracted and down-sampled to obtain an initial feature map;

[0078] Module S2: The initial feature map is extracted through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map;

[0079] Module S3: extract features from the extracted feature map through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map. Repeat the triggering of module S3 three times to make the feature semantics of the feature map reach the highest level of abstraction.

[0080] Module S4: restore the features of the feature map with the highest level of feature semantics through a combination module of the second type of dense connection block and the upsampling block to obtain the restored feature map. Repeat the triggering of module S4 three times until the resolution of the feature map is restored to the resolution when it was input into module S3.

[0081] The feature semantics reaching the highest level of abstraction means that the resolution reaches a preset value.

[0082] Specifically, Figure 1 As shown, the first-stage lung nodule 3D instance segmentation model adopts:

[0083] In the pre-extraction module: a 3*3*3 convolution operation is used to perform preliminary feature perception on the input feature map, and feature extraction is performed through a 3*3*3 standard convolution module, that is, a combination of batch normalization, nonlinear activation, and 3*3*3 convolution operations, for further feature extraction. Finally, a 2*2*2 maximum pooling is used to extract the most critical information that meets the preset requirements in each neighborhood to obtain a preliminary feature map, such as Figure 2 As shown; for example: 2*2*2 max pooling is to cut the feature map into many 2*2*2 small blocks with a step size of 2, and each 2*2*2 small block is replaced by the largest number among these 8 numbers, that is, to extract the most critical information. Therefore, after 2*2*2 max pooling, the length, width and height of the feature map are reduced by half;

[0084] In the first type of densely connected module: the preliminary feature map passes through a 1*1*1 standard convolution module, i.e., a bottleneck layer, to reduce the number of channels and the amount of model calculation, and then extracts features through a 3*3*3 standard convolution module to obtain a 16-channel level 1 feature map, which is cascaded with the preliminary feature map, and then repeatedly passes through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map and the level 1 feature map, and is repeatedly triggered until a preset level feature map is output, such as Figure 2 As shown;

[0085] In the downsampling module, the channels of the feature map with a preset number of levels are halved by a 1*1*1 standard convolution, and a 2*2*2 maximum pooling is performed; the halving of the feature map channels is used to reduce the amount of calculation, and the maximum pooling is used to extract the most obvious features of the local area, such as Figure 2 As shown;

[0086] In the second type of densely connected module, the preliminary feature map is passed through a 1*1*1 standard convolution module, and then through a 3*3*3 standard convolution module for feature extraction to obtain a 16-channel level 1 feature map, which is cascaded with the initial feature map, and then repeatedly passed through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map, and repeatedly triggered until a preset number of levels of feature maps are obtained and the cascade results of 16-channel feature maps at each level are output;

[0087] In the upsampling module, the concatenation results of the 16-channel feature maps at each level are deconvolved 2*2*2 and activated to obtain a feature map of restored features; compared with the method of upsampling by interpolation, deconvolution performs upsampling by learning, which can better restore features, such as Figure 2 As shown;

[0088] In the region recommendation module, the concatenation result of the preset level 16-channel feature map is used to generate a 128-channel feature map through a 3*3*3 standard convolution module, and then a 61-channel feature map is generated through a 1*1*1 standard convolution module as the final segmentation and detection result; Figure 2 As shown in the figure, in the one-stage lung nodule 3D instance segmentation model, the size of the anchor box is carefully designed through clustering according to the distribution of nodules, and finally 12 different anchor box sizes are obtained. The detection results are obtained based on these 12 anchor boxes of different sizes.

[0089] Each convolutional layer is followed by batch normalization-LeakyReLU activation function.

[0090] Specifically, the loss function of the one-stage lung nodule 3D instance segmentation model is divided into several parts for calculation. The loss function consists of positive and negative classification loss, positive sample bounding box regression loss, and positive sample segmentation loss.

[0091] Specifically, the pulmonary nodule segmentation and detection results include: 60-channel anchor frame information and 1-channel nodule heat map;

[0092] The anchor frame information indicates that there are 12 anchor frames of different sizes at each position, and each anchor frame has 5 attributes, including the three-dimensional center coordinates, the anchor frame diameter, and the confidence that the anchor frame contains the nodule;

[0093] The nodule heatmap indicates the confidence that each location belongs to the lung nodule region.

[0094] Specifically, in the module M5,

[0095] Non-maximum suppression module: All anchor frames are sorted by confidence, the largest anchor frame is taken out and added to the positive anchor frame set, and the anchor frames whose intersection-over-union ratio with the current anchor frame is below the set threshold are retained; the remaining anchor frames are further sorted by confidence, and repeatedly triggered to finally obtain a set of positive anchor frames whose intersection-over-union ratios between each pair are below the set threshold, and the detected non-overlapping nodule lesions are obtained;

[0096] The non-maximum suppression module is used to reduce overlapping anchor boxes and find the closest anchor box match of the nodule target to prevent multiple matches.

[0097] Nodule mask generation module: The nodule resolution is restored to the size of the input data of the first-stage pulmonary nodule 3D instance segmentation model (i.e., the size of the preprocessed pulmonary nodule CT image) through trilinear interpolation. The threshold of mask binarization is then selected based on the binary method to minimize the error between the binarized result and the true labeled mask. The mask of each anchor frame and each frame is then filled into a convex set to ensure the integrity of the mask.

[0098] Nodule boundary outlining module: used to generate and outline boundaries based on anchor box information and mask information.

[0099] Example 2

[0100] Embodiment 2 is a preferred embodiment of Embodiment 1

[0101] The present invention provides a method for intelligent image processing of pulmonary nodules, comprising:

[0102] Step S1: Select the data required for model training, verification and testing;

[0103] Step S2: Perform standard voxel conversion, lung window selection, and lung parenchyma extraction on the patient's CT image data to obtain voxel standards with consistent brightness, which can be used to input data into the system;

[0104] Step S3: building a first-stage lung nodule 3D instance segmentation model network structure;

[0105] Step S4: using the 3D data to input a first-stage 3D instance segmentation model and simultaneously obtaining the results of lung nodule segmentation and detection in an end-to-end manner;

[0106] Step S5: Determine the central position and diameter of the lung nodule, and outline the boundary of the lung nodule for the convenience of doctors to check; for each nodule, its boundary coordinates are stored frame by frame.

[0107] Specifically, step S1 adopts: the model uses the LUNA-16 data set for training, verification and testing. LUNA-16 has a total of 888 CT scan data of different patients, divided into 10 sub-data sets, 9 of which are used for training (subset0-9 excluding subset5), and the remaining 1 is used for testing. The CT scan data of each patient contains about 200-300 frames of 2D slices, and the frame spacing varies from 0.625 to 2.5 mm. The number of nodules in each CT scan data is different, the size is between 3-30 mm, and it is marked by the center coordinates and diameter.

[0108] Specifically, step S2 adopts: data preprocessing including standard voxel conversion, lung window selection, lung parenchyma extraction and other processes. Voxel standardization uses trilinear interpolation to restore the actual space size represented by each voxel to 1mm*1mm*1mm. To ensure the brightness consistency of CT images, the model adjusts the lung window width by truncation so that the brightness range of lung image data is between -1024 and 1024.

[0109] Lung parenchyma extraction is divided into 7 sub-steps:

[0110] Step S2.1: filtering the lung CT data to remove noise from the data;

[0111] Step S2.2: binarizing the filtered lung CT data to distinguish between areas with high brightness and areas with low brightness;

[0112] Step S2.3: Through graphics processing, the area outside the lung parenchyma is set to 0, so as to obtain the preliminary lung parenchyma area. Specifically, the maximum connected component of the geometrically non-enclosed area with a value of 0 is first set to 1 to separate the area outside the CT field of view and the background of the lung parenchyma area; then the whole image is inverted to extract the lung parenchyma area;

[0113] Step S2.4: Obtain the contour of the lung parenchyma area by calculating the convex hull of the area whose current value is 1, and include the nodule area in the lung parenchyma;

[0114] Step S2.5: Using the image generated in step S2.3 and the image generated in step S2.4, the smaller connected components with values ​​of 0 in the convex hull of the lung parenchyma can be filtered out, thereby further refining the image obtained in step S2.3 and making full use of the characteristic of step S2.4 that the nodule area can be included in the lung parenchyma;

[0115] Step S2.6: Ensure that both lungs are located in the connected components of the lung parenchyma through expansion and closing operations;

[0116] Step S2.7: Use the lung parenchyma connected component to mask the original image to obtain the final lung parenchyma region.

[0117] The step S3 adopts: a one-stage lung nodule 3D instance segmentation model with 3D U-Net as the backbone, each convolution block is implemented using a densely connected block, which maximizes the flow of information in the network and facilitates better feature extraction; the lateral connection between the feature maps of the same resolution on both sides of the U-Net ensures the flow and fusion of semantic information between the encoder and the decoder.

[0118] The one-stage lung nodule 3D instance segmentation model takes 3D data blocks as input and a 61-channel feature map as output, including 60-channel anchor box information and 1-channel nodule heat map. The anchor box information indicates that there are 12 anchor boxes of different sizes at each position, and each anchor box has 5 attributes: the three-dimensional center coordinates, the anchor box diameter, and the confidence that the anchor box contains a nodule; the nodule heat map indicates the confidence that each position belongs to the nodule area. The rectangular 3D data block data input to the model uses 32 consecutive frames of each patient's CT image to avoid the input of complete CT data causing excessive model calculation, but it also fully retains the complete information of the axial section, which is conducive to obtaining the overall association between the nodule and the lung environment.

[0119] The entire model is composed of 6 different types of sub-modules, namely the pre-extraction module, the first type of dense connection module, the downsampling module, the second type of dense connection module, the upsampling module, and the region proposal module. The model is divided into a downsampling path and an upsampling path, located at Figure 1 The two different types of densely connected blocks are the forms of densely connected blocks used in the model downsampling path and upsampling path. After the 3D data is fed into the model, it first enters the pre-extraction block for feature pre-extraction and downsampling; then the combination of the first type of densely connected blocks and the downsampling blocks is repeatedly used to extract features, and the feature map obtained by each first type of densely connected block is fused with the feature map of the corresponding size in the upsampling path. When the resolution of the feature map is small enough, the feature semantics reaches the highest level of abstraction, and the model enters the upsampling path. In the upsampling path, the combination of the upsampling module and the second type of densely connected blocks is repeatedly used to restore the positioning information of high-level semantic features. In the process of combining the upsampling module and the second type of densely connected blocks, each feature map obtained from the upsampling module is fused with the feature map of the corresponding size in the downsampling path and then enters the second type of densely connected block to achieve the correspondence between image information and semantic information. The size of each dimension of the final feature map is half the size of the original image, and it is sent to the region proposal layer to extract the anchor box information and the corresponding nodule heat map to obtain the final detection and segmentation results.

[0120] The step S4 adopts: a three-dimensional rectangular block is used as the input of the model during model training, and each data block is expanded frame by frame with a specific nodule as the center, and the lung parenchyma area in the 16 frames before and after it is obtained to form a rectangular block, so as to ensure that the input data has nodules. For the case where the center of the nodule is adjacent to the top or bottom of the CT image and less than 32 frames, 32 frames are expanded from the top or bottom of the CT in the same direction, and the number of frames is preferentially guaranteed to meet the requirements. Since some areas of other nodules may be included in the rectangular block, the input distribution is optimized during training: other nodules contained in the data block corresponding to each nodule are included in the statistics again. After distribution optimization, when randomly acquiring data blocks, the more concentrated the nodule distribution is, the higher the sampling probability is, which is beneficial for the positioning of the model learning nodule dense area. In order to make U-Net downsampling not produce loss of accuracy, the data block needs to be slightly cropped and filled in the lung parenchyma area to ensure that each dimension is a multiple of 32.

[0121] Data enhancement in deep learning can enrich the data pattern, simulate various situations in the image, and avoid model overfitting. Common data enhancement methods include: data translation, data rotation, data flipping, data occlusion, adding noise, etc. For the first-stage lung nodule 3D instance segmentation lung nodule recognition model, the targets are mostly spheres with small diameters, and the model input contains a complete lung parenchyma section in the axial direction. Therefore, data enhancement using 3D cuboid data blocks translated along the frame is used to ensure model performance while avoiding excessive training complexity.

[0122] The data labels in training need to be transformed from the LUNA-16 dataset annotations. For nodule position annotation, a specific mathematical transformation is performed to facilitate model fitting: the true value of the nodule coordinates is converted into a relative displacement value with the anchor frame coordinates, and the true value of the nodule diameter is converted into a logarithmic ratio value with the anchor frame diameter.

[0123] The one-stage lung nodule 3D instance segmentation model is a multi-task model. The loss function is calculated in several parts. The formula is as follows:

[0124] loss = λ 1 l negcls +λ 2 l poscls +λ 3 l posreg +λ 4 l posmask +penalty

[0125] Among them, loss represents the loss function, l negcls represents the negative classification loss function, λ 1 represents its weight, l poscls represents the positive classification loss function, λ 2 Indicates its weight; l posreg represents the positive sample bounding box regression loss function, λ 3 Indicates its weight. posmask represents the positive sample segmentation loss function, λ 4 Represents its weight. Penalty represents the regularization penalty term.

[0126] The loss function consists of positive and negative classification loss, positive sample bounding box regression loss, and positive sample segmentation loss. For the classification of positive and negative samples, the classic binary cross entropy loss is used to calculate the error, and the formula is as follows:

[0127]

[0128] Among them, l cls represents the classification loss function, y represents the probability truth label (0 or 1), represents the predicted probability;

[0129] For the positive sample border regression loss function, a smooth L1 function is used to calculate the error of the border regression. The formula is as follows:

[0130]

[0131] Among them, l posreg represents the positive sample border regression loss function, x represents the true value of the border regression, r x Represents the predicted value of bounding box regression;

[0132] For the positive sample segmentation loss function, the overlap between the mask prediction of the anchor box with a confidence greater than 0.5 in the detection task and the true mask is measured, and the formula is as follows:

[0133]

[0134] Among them, a∈PosAnchor represents the anchor box with a confidence greater than 0.5 in the detection task, and mask a Represents the segmentation mask prediction value within the anchor box, gt a Represents the true value of the mask of anchor box a;

[0135] In step 5, the system infers that the CT scan of subset 5 in the LUNA-16 dataset is used as the input data block of the inference process by sliding the window with a window width of 32 and a step length of 16 along the data frame direction. In the inference process, the positive and negative classification of all anchor boxes is given with a confidence threshold of 0.5, and the positive anchor box is mask predicted and post-processed.

[0136] The post-processing of the system mainly includes three parts: non-maximum suppression (NMS), nodule mask generation, and nodule boundary delineation. Non-maximum suppression is mainly used to reduce overlapping anchor boxes, find the closest anchor box match to the nodule target, and prevent multiple matches. For the generation of nodule masks, firstly, the nodule resolution is restored to the size of the network input data through trilinear interpolation, and then the threshold of mask binarization is selected according to the dichotomy method to minimize the error between the binarized result and the mask of the real annotation. The binarization process of fixed threshold may cause the mask area to be incompletely connected, so the mask of each anchor box in each frame is complemented into a convex set to ensure the integrity of the mask. The final nodule boundary is generated according to the anchor box information and mask information: the mask in the positive anchor box is regarded as the mask of the nodule area. If the diameter of the nodule area mask in the positive anchor box is less than 2mm, the anchor box will be marked as a negative sample, and the outer edge points of the mask in the positive anchor box are taken as the nodule contour. For each nodule, its boundary coordinates are stored frame by frame.

[0137] The system for intelligent image processing of pulmonary nodules provided by the present invention can be implemented by the steps and procedures in the method for intelligent image processing of pulmonary nodules provided by the present invention. Those skilled in the art can understand the method for intelligent image processing of pulmonary nodules as a preferred example of the system for intelligent image processing of pulmonary nodules.

[0138] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0139] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An intelligent image processing system for pulmonary nodules, characterized in that: include: Module M1: Acquisition of 3D lung CT image data; Module M2: preprocessing the acquired 3D lung CT image data to obtain preprocessed 3D lung CT image data; Module M3: constructing a one-stage pulmonary nodule 3D instance segmentation model, and training the one-stage pulmonary nodule 3D instance segmentation model to obtain a trained one-stage pulmonary nodule 3D instance segmentation model; Module M4: Generate lung nodule segmentation and detection results using the preprocessed lung nodule CT image data according to the trained one-stage lung nodule 3D instance segmentation model; Module M5: Determine the center position and diameter of the lung nodule based on the lung nodule segmentation and detection results, and outline the boundary of the lung nodule; The one-stage pulmonary nodule 3D instance segmentation model is a pulmonary nodule real segmentation model that uses 3D lung CT image data as input, uses 3D data as input of the one-stage pulmonary nodule 3D instance segmentation model to extract CT frame-to-frame information, and simultaneously obtains pulmonary nodule segmentation and detection results in an end-to-end manner; In the module M2: the patient's CT image data is subjected to standard voxel conversion, lung window selection, and lung parenchyma extraction to obtain first-stage lung nodule CT image data with consistent voxel standards and brightness; The first-stage lung nodule 3D instance segmentation model adopts: a pre-extraction module, a first-type dense connection module, a down-sampling module, a second-type dense connection module, an up-sampling module and a region recommendation module; Module S1: In the pre-extraction module, 3D data is pre-extracted and down-sampled to obtain an initial feature map; Module S2: The initial feature map is extracted through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map; Module S3: extract features from the extracted feature map through a combination module of the first type of dense connection block and the downsampling block to obtain the extracted feature map, and repeatedly trigger module S3 to make the feature semantics of the feature map reach the highest level of abstraction; Module S4: restore the features of the feature map with the highest level of feature semantics through a combination module of the second type of dense connection block and the upsampling block to obtain a restored feature map, and repeatedly trigger module S4 until the resolution of the feature map is restored to the resolution when it was input into module S3; The feature semantics reaching the highest level of abstraction means that the resolution reaches a preset value; The first-stage lung nodule 3D instance segmentation model adopts: In the pre-extraction module: a 3*3*3 convolution operation is used to perform preliminary feature perception on the input feature map, and feature extraction is performed through a 3*3*3 standard convolution module, and finally a 2*2*2 maximum pooling is used to extract the most critical information that meets the preset requirements in each neighborhood to obtain a preliminary feature map; In the first type of densely connected module: the preliminary feature map is passed through a 1*1*1 standard convolution module, and then through a 3*3*3 standard convolution module for feature extraction to obtain a 16-channel level 1 feature map, which is cascaded with the preliminary feature map, and then repeatedly passed through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map and the level 1 feature map, and repeatedly triggered until a preset level feature map is output; In the downsampling module, the channels of the feature map with a preset number of levels are halved by a 1*1*1 standard convolution, and a 2*2*2 maximum pooling is performed; In the second type of densely connected module, the preliminary feature map is passed through a 1*1*1 standard convolution module, and then through a 3*3*3 standard convolution module for feature extraction to obtain a 16-channel level 1 feature map, which is cascaded with the initial feature map, and then repeatedly passed through a 1*1*1 standard convolution block and a 3*3*3 standard convolution block to obtain a 16-channel level 2 feature map, which is cascaded with the preliminary feature map, and repeatedly triggered until a preset number of levels of feature maps are obtained and the cascade results of 16-channel feature maps at each level are output; In the upsampling module, the concatenation results of the 16-channel feature maps at each level are subjected to 2*2*2 deconvolution and activated to obtain a feature map of restored features; In the region recommendation module, the concatenation result of the preset level 16-channel feature map is used to generate a 128-channel feature map through a 3*3*3 standard convolution module, and then a 61-channel feature map is generated through a 1*1*1 standard convolution module as the final segmentation and detection result; The standard convolution module includes batch normalization, nonlinear activation, and convolution operations of preset size.

2. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: The standard voxel conversion adopts: trilinear interpolation to restore the actual space size represented by each voxel to a preset value.

3. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: The lung window selection adopts: adjusting the width of the lung window by truncation so that the brightness range of the lung image data is within a preset range.

4. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: The lung parenchyma extraction adopts: Module M2.1: Filter lung CT data to remove noise from the data; Module M2.2: Binarize the filtered lung CT data to distinguish between areas with high brightness and areas with low brightness; Module M2.3: Through graphics processing, the area outside the lung parenchyma is set to 0, the maximum connected component of the geometrically non-enclosed area with a value of 0 is set to 1, and the area outside the CT field of view and the background of the lung parenchyma area are separated; Then the whole image is inverted to extract the lung parenchyma area to obtain image 1; Module M2.4: Obtain the contour of the lung parenchyma area by calculating the convex hull of the area with the current value of 1, and include the nodule area in the lung parenchyma to obtain image 2; Module M2.5: Using images 1 and 2, filter out the connected components with values ​​of 0 in the convex hull of the lung parenchyma that meet the preset requirements; Module M2.6: Ensure that both lungs are located in the connected components of the lung parenchyma through expansion and closing operations; Module M2.7: Use the lung parenchyma connected components to mask the original image and obtain the final lung parenchyma area.

5. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: The loss function of the one-stage lung nodule 3D instance segmentation model includes positive and negative classification loss, positive sample bounding box regression loss, and positive sample segmentation loss.

6. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: The pulmonary nodule segmentation and detection results include: 60-channel anchor frame information and 1-channel nodule heat map; The anchor frame information indicates that each position has a preset number of anchor frames of different sizes, and each anchor frame has a preset number of attributes, including three-dimensional center coordinates, anchor frame diameter, and confidence that the anchor frame contains a nodule; The nodule heatmap indicates the confidence that each location belongs to the lung nodule region.

7. The intelligent image processing system for pulmonary nodules according to claim 1, characterized in that: In the module M5, Non-maximum suppression module: All anchor frames are sorted by confidence, the largest anchor frame is taken out and added to the positive anchor frame set, and the anchor frames whose intersection-over-union ratio with the current anchor frame is below the set threshold are retained; the remaining anchor frames are further sorted by confidence, and repeatedly triggered to finally obtain a set of positive anchor frames whose intersection-over-union ratios between each pair are below the set threshold, and the detected non-overlapping nodule lesions are obtained; Nodule mask generation module: The nodule resolution is restored to the size of the input data of the first-stage lung nodule 3D instance segmentation model through trilinear interpolation, and the threshold of mask binarization is selected according to the binary method to minimize the error between the binarized result and the real labeled mask. The mask of each anchor frame and each frame is then filled into a convex set to ensure the integrity of the mask. Nodule boundary outlining module: used to generate and outline boundaries based on anchor box information and mask information.

Images