Automatic detection method of regions of interest in lung CT based on multi-task convolutional model

Abstract

A method for automatically detecting regions of interest (ROIs) in lung CT images based on a multi-task convolutional model. CT scans are parsed to obtain a tensor-like image. This image is then coarsely segmented, resampled, and pixel-normalized to produce a three-dimensional CT image voxel matrix containing only the lung region. The three-dimensional image is then sliced ​​and several resulting two-dimensional cross-sectional images are fed into a two-dimensional convolutional model to obtain rough ROI outlines. Three-dimensional image blocks are then captured centered around the rough ROI outlines. A three-dimensional classifier is used to determine the probability of each ROI being a true ROI. After screening, the resulting three-dimensional image blocks containing true ROIs are fed into a three-dimensional convolutional model trained using multiple tasks to obtain precise ROI outlines. These outlines are then rendered in different colors based on the probability of the ROI being a true ROI. This method can rapidly process large batches of CT images and improve ROI detection efficiency.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method for automatically detecting regions of interest in lung CT (Computed Tomography) based on a multi-task convolutional model. Background Art

[0002] With the development of medical technology, more and more lung CT images need to be processed in a timely manner. Among them, the detection of regions of interest, or ROIs (Region Of Interest), is the first step in lung CT image processing. In the field of digital image processing, ROI refers to the specific area of ​​the image that technicians are interested in. If the traditional method of manually inspecting CT images layer by layer to determine whether there are ROIs is used, it will add a large burden to doctors, because doctors need to refer to multiple images and constantly compare them to make judgments when looking for ROIs, which is a time-consuming and laborious task. However, using convolutional networks to automatically locate ROIs and outline ROIs can reduce the processing time of CT images while ensuring accuracy, making it possible to process CT images in large quantities, freeing doctors from tedious work and improving the overall operating efficiency of the hospital.

[0003] Current algorithms for automatic ROI detection in lung CT images lack versatility and are often limited to processing CT images of a certain thickness. They generally generalize poorly to CT images of varying thicknesses or scanned by different machines. Furthermore, most algorithms train convolutional networks using only ROI masks or anchor boxes as semantic labels. Consequently, the convolutional model focuses solely on voxel-based visual features in CT images, ignoring clinical biological characteristics and hindering accurate ROI delineation. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention proposes a method for automatic detection of lung CT regions of interest based on a multi-task convolutional model. By training different models for different layer thicknesses, the problem of inconsistent information content in CT images of different layer thicknesses can be effectively solved. In addition, when training the three-dimensional semantic segmentation model, the ROI mask and the semantic labels of the ROI benign and malignant are used simultaneously, combining the imaging and biological characteristics of the ROI, which is conducive to accurately outlining the ROI contour. By obtaining the ROI contour through a convolutional model based on an encoder-decoder structure, large quantities of CT images can be quickly processed, thereby improving the efficiency of ROI detection.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to an automatic detection method for lung CT regions of interest based on a multi-task convolutional model. The method comprises the following steps: analyzing a CT scan to obtain a tensor-shaped image, performing lung coarse segmentation, resampling, and pixel value normalization on the image to obtain a three-dimensional CT image voxel matrix containing only the lung region; slicing the three-dimensional image, inputting the cross-sectional slices into a two-dimensional semantic segmentation network based on an encoder-decoder structure to obtain several rough ROI outlines; intercepting three-dimensional image blocks with the rough ROI outline as the center, and using a three-dimensional classifier to give the probability of each ROI being a true ROI. After screening, the obtained three-dimensional image blocks containing the true ROI are input into a three-dimensional convolutional model obtained by multi-task training to obtain accurate ROI outlines, and rendering the outlines in different colors according to the probability that the ROI is a true ROI.

[0007] The three-dimensional image is obtained by combining the two-dimensional slice matrices in multiple DICOM (Digital Imaging and Communications in Medicine) files into a three-dimensional image based on the position coordinates of the slices, and then preprocessing the three-dimensional image. Specifically, the three-dimensional image is obtained by using digital image processing technology to find a rough anchor frame of the lung area and cropping the non-lung portion of the image according to the anchor frame; sampling the pixel spacing of the image to (0.7 mm, 0.7 mm, 1.5 mm) along the (x, y, z) axes respectively; and normalizing the pixel values ​​according to the lung window [-1024, 400] and the mediastinum window [-160, 240] to obtain two channels corresponding to the lung window and the mediastinum window, respectively.

[0008] The two-dimensional semantic segmentation network is based on the U2-Net neural network, which has an encoder-decoder structure as a whole. At the same time, each unit of the encoder and decoder is also an encoder-decoder structure. The encoder gradually expands the receptive field of the convolution layer through several downsampling processes, extracts high-dimensional semantic features in the feature map, and obtains the rough position of the nodule in the current input; the decoder gradually samples the feature map to the same resolution as the input through several upsampling processes, extracts low-dimensional visual features in the feature map, and obtains the mask of the nodule in the input image.

[0009] The output of each layer of the encoder and decoder {e i |i∈[1,N]} and {d j The resolutions of |j∈[1,N]} are all different. That is, the encoder-decoder structure uses multiple layers of downsampling and upsampling to enable the model to extract ROI features at different resolutions, which can effectively handle the problem of inconsistent ROI sizes in different cases. N is the number of layers of the encoder and decoder.

[0010] There is a short-circuit connection between the corresponding layers of the encoder and decoder, that is, the input of the decoder of the jth layer is Concat(e j ,Up(d j -1)), where: Concat means concatenating the feature maps by channel dimension, and Up means upsampling the feature maps to twice the original resolution using interpolation. This allows the decoder to simultaneously process the features containing high-dimensional semantic information output by the previous layer decoder and the features containing low-dimensional visual information output by the encoder at the same height, which is beneficial to improving the accuracy of ROI contour delineation while ensuring the accuracy of ROI position.

[0011] The probability that the ROI is a true ROI is determined by cutting a 96×96×48 3D image block with the rough outlines of several ROIs output by the 2D semantic segmentation model as the center. The 3D image block is then input into a 3D classifier based on a residual network to obtain the probability that the ROI is a true ROI.

[0012] The three-dimensional classifier based on the residual network is a three-dimensional variant based on the residual network, and the residual structure is specifically: y=F(x,{W i})+x, where: subscript i refers to the i-th layer of the network, x is the input of the current layer, y is the output of the current layer, {W i} is the parameter of the i-th layer, For a function form such as Residual connections can partially address the problem of training degradation after network deepening. The bottleneck layer maps feature maps to a high-dimensional space, processes them, and then compresses them to a low-dimensional space. This not only preserves the multi-dimensional features that better express semantic information, but also reduces the network's training parameters and eases the training difficulty.

[0013] The screening means that ROIs with a probability less than a certain threshold are judged as false positives, and their contours will not be output to the final mask.

[0014] The precise ROI outline is obtained by inputting a 128×128×64 three-dimensional image block centered on the filtered ROI into a three-dimensional semantic segmentation network based on an encoder-decoder structure. The outline of each ROI is then output to an RT STRUCT file, which uses different colors to render the outline of each ROI according to the probability output by the classifier.

[0015] The three-dimensional semantic segmentation network is a three-dimensional variant of the above-mentioned two-dimensional semantic segmentation network, in which only the two-dimensional operations are changed into corresponding three-dimensional operations, that is, the two-dimensional convolution, pooling, and normalization layers are changed into three-dimensional convolution, pooling, and normalization layers respectively.

[0016] The multi-task training means that when training the three-dimensional semantic segmentation network, in addition to the main task of outputting the ROI mask, an auxiliary task of judging whether the ROI is benign or malignant is added. Specifically, in the three-dimensional semantic segmentation network, in addition to the encoder that extracts features from the input CT image and the decoder that outputs the ROI mask, a classifier is added to judge whether the current ROI is benign or malignant. The classifier takes the output of each layer of the encoder as input, and through appropriate downsampling and convolution operations, it finally obtains the probability value representing the current ROI is benign. During the training process, the parameters of the encoder are updated by the back-propagation gradients from the decoder and the classifier at the same time;

[0017] The loss function of the three-dimensional classifier and the classification part in multi-task training is Focal Loss, specifically: Among them: y, They refer to the true label and predicted value of the benign and malignant ROI respectively. The hyperparameter α is used to control the weight of positive and negative samples. When α>0.5, the positive sample contributes more to the model parameters. The loss function controls the weight of difficult and easy samples through the hyperparameter γ. When γ>1, the weight of difficult samples is greater during gradient backpropagation, that is, the model pays more attention to difficult samples.

[0018] The loss function of the two-dimensional and three-dimensional semantic segmentation networks is Focal Loss + Dice Loss, specifically: Among them: p, where i represents the true label and predicted value of the ROI mask, respectively. The subscript i represents the value of the i-th pixel in the mask. Focal Loss is a pixel-wise loss function that calculates and sums the loss function for each pixel. This form of loss function has a more optimized gradient and facilitates smooth model training. Dice Loss is a regional loss function that only affects the foreground region in the mask. It is consistent with the final evaluation metric and can alleviate class imbalance. However, if the foreground region is too small, the model optimization process will become unstable. Combining these two loss functions can ensure that the model outputs a ROI mask with a more optimized Dice coefficient while ensuring smooth training.

[0019] The training datasets used were two in-hospital datasets. The first dataset contained 989 samples with a CT slice thickness of 5 mm. The second dataset contained 172 samples with a CT slice thickness of less than 1.5 mm. Two models were trained on the two datasets. In practice, when the input CT image slice thickness was greater than 3 mm, the corresponding model trained on the first dataset was used; otherwise, the model trained on the second dataset was used.

[0020] Technical Effects

[0021] The present invention uses both the ROI mask and the benign and malignant labels as true labels for training a three-dimensional semantic segmentation model, ensuring that the model learns both the visual features of the CT image through the ROI mask and the biological features of the CT image through the benign and malignant labels of the ROI. Furthermore, the present invention uses three steps, namely, rough contouring, false positive suppression, and precise contouring, and corresponding models, to balance the efficiency of locating the ROI in three-dimensional CT images with the accuracy of ROI contouring, while reducing false positives in ROI positioning. Finally, the present invention trains different models for CT images of different layer thicknesses, improving the accuracy of ROI contouring in CT images of different layer thicknesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of an embodiment;

[0023] Figure 2 This is a schematic diagram of the image of ROI in the two-dimensional slice of CT scan;

[0024] Figure 3 Schematic diagram of the lung area coarse segmentation process;

[0025] Figure 4 This is a diagram of the three-dimensional semantic segmentation network structure used in the present invention;

[0026] Figure 5 Schematic diagram of two-dimensional comparison of the original image (left), the real ROI mask (center), and the ROI mask output based on the present invention (right);

[0027] Figure 6 Schematic diagram of three-dimensional comparison between the real ROI mask (left) and the ROI mask output based on the present invention (right). DETAILED DESCRIPTION

[0028] like Figure 1 As shown in FIG, this embodiment relates to an automatic detection process of lung CT regions of interest based on a multi-task convolutional model. Figure 2 The DICOM format lung CT scan shown uses a 2D encoder-decoder semantic segmentation model to scan the entire lung area and find the rough outline of the ROI. A 3D residual structure classifier then judges each ROI and gives the probability that the ROI is a true ROI. Finally, a 3D encoder-decoder semantic segmentation model accurately outlines the ROI whose probability is greater than a certain threshold.

[0029] The detection method specifically comprises the following steps:

[0030] The first step is to parse the CT image in DICOM format and combine multiple two-dimensional grayscale images into a three-dimensional matrix according to coordinates.

[0031] The second step is to use digital image processing technology to find the lung area. Specifically, the grayscale image is thresholded and binarized; the connected domain is found, and the two largest connected domains are retained. The connected domain is the lung area, and the area is cropped to remove the part outside the lung, such as Figure 3 shown.

[0032] In the third step, the pixel spacing of the image was sampled to (0.7 mm, 0.7 mm, 1.5 mm) along the (x, y, z) axes, and the 3D image pixel values ​​were normalized using the lung window [-1024, 400] and the mediastinum window [-160, 240], respectively. The specific operation was to subtract the window lower limit from the pixel value and then divide it by the window width.

[0033] The fourth step is to slice the 3D image, divide it into several images of size 7×512×256, and resample it into images of size 7×512×256. The number of channels 7 represents seven consecutive 2D slices.

[0034] In the fifth step, the sliced ​​images are input into the two-dimensional semantic segmentation network of the encoder-decoder structure to obtain the rough outline of the ROI.

[0035] In the sixth step, a 96×96×48 3D image block is captured with each ROI obtained in the fifth step as the center, and the block is input into the 3D classifier based on the residual structure to give the probability P that each ROI is a true ROI.

[0036] In the seventh step, a 3D graphic block of size 128×128×64 is intercepted with the ROI in the sixth step whose probability of being true is greater than a certain threshold as the center, and input into the 3D semantic segmentation network of the encoder-decoder structure to give the precise outline of each ROI. The 3D semantic segmentation network structure is as follows Figure 4 shown.

[0037] In the eighth step, the precise contours obtained in the seventh step are written into the DICOM format output file RT STRUCT. Different colors are assigned to each ROI according to the probability P corresponding to the sixth step. Specifically, the contours with 0.7<P≤0.8 are green, the contours with 0.8<P≤0.9 are yellow, and the contours with P>0.9 are red. The two-dimensional and three-dimensional comparisons of the ROI mask output by the model and the ROI mask annotated by the doctor are shown in Figure 2. Figure 5 and Figure 6 shown.

[0038] Compared with the existing technology, this method achieved an average Dice of 0.7026 on a test set of 119 thick-slice CT images (slice thickness greater than 3 mm), and an average Dice of 0.7046 on a test set of 21 thin-slice CT images (slice thickness less than or equal to 3 mm). The software takes an average of less than one minute to process thick-slice CT images and less than two minutes to process thin-slice CT images on a single NVIDIA GeForce GTX 3090Ti GPU.

[0039] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. An automatic detection of regions of interest in lung CT based on a multi-task convolutional model, characterized by: The CT scan is parsed to obtain a tensor-like image, which is then subjected to coarse lung segmentation, resampling, and pixel value normalization to obtain a three-dimensional CT image voxel matrix containing only the lung area. The three-dimensional image is then sliced ​​and the cross-sectional slices are input into a two-dimensional semantic segmentation network based on an encoder-decoder structure to obtain several rough ROI outlines. Three-dimensional image blocks are then intercepted with the rough ROI outline as the center, and the probability of each ROI being the true ROI is given by a three-dimensional classifier. After screening, the obtained three-dimensional image blocks containing the true ROI are input into a three-dimensional convolutional model obtained by multi-task training to obtain the precise ROI outline, and the outline is rendered in different colors according to the probability of the ROI being the true ROI. The two-dimensional semantic segmentation network is based on the U2-Net neural network, which has an encoder-decoder structure as a whole. At the same time, each unit of the encoder and decoder also has an encoder-decoder structure. The encoder gradually expands the receptive field of the convolution layer through several downsampling steps, extracts high-dimensional semantic features in the feature map, and obtains the rough location of the nodule in the current input; the decoder gradually samples the feature map to the same resolution as the input through several upsampling steps, extracts low-dimensional visual features in the feature map, and obtains the mask of the nodule in the input image; The three-dimensional classifier is a three-dimensional variant based on the residual network, and its residual structure is specifically as follows: , where: subscript Refers to the network layer, is the input of the current layer, is the output of the current layer, For the The parameters of the layer, For a function form: ,The bottleneck layer processes the feature map by mapping it to a high-dimensional space and then compressing it to a low-dimensional space; The multi-task training described herein refers to the addition of an auxiliary task, i.e., determining whether the ROI is benign or malignant, in addition to the main task of outputting a ROI mask when training a 3D semantic segmentation network. Specifically, in addition to an encoder that extracts features from an input CT image and a decoder that outputs a ROI mask, a classifier is added to the 3D semantic segmentation network to determine whether the current ROI is benign or malignant. The classifier uses the outputs of each layer of the encoder as input and, through appropriate downsampling and convolution operations, ultimately obtains a probability value indicating whether the current ROI is benign. During training, the encoder parameters are simultaneously updated by back-propagated gradients from the decoder and the classifier.

2. The automatic detection of lung CT regions of interest based on a multi-task convolutional model according to claim 1 is characterized in that: The output of each layer of the encoder and decoder and The resolutions are all different, that is, the encoder-decoder structure is down-sampling and up-sampling in multiple layers, where N is the number of encoder and decoder layers; There is a short-circuit connection between the corresponding layers of the encoder and decoder, that is, the input of the decoder of the jth layer is ,in: Indicates that the feature maps are spliced ​​according to the channel dimension. It refers to using interpolation to upsample the feature map to twice the original resolution.

3. The automatic detection of lung CT regions of interest based on a multi-task convolutional model according to claim 1 is characterized in that: The precise ROI outline is obtained by taking the size of the ROI as the center after the screening. The three-dimensional image block is input into the three-dimensional semantic segmentation network based on the encoder-decoder structure to obtain the accurate outline of the ROI; then the outline of each ROI is output to the RTSTRUCT file, which uses different colors to render the outline of each ROI according to the probability size output by the classifier.

4. The automatic detection of lung CT regions of interest based on a multi-task convolutional model according to claim 3 is characterized in that: The three-dimensional semantic segmentation network is a three-dimensional variant of the above-mentioned two-dimensional semantic segmentation network, in which only the two-dimensional operations are changed into corresponding three-dimensional operations, that is, the two-dimensional convolution, pooling, and normalization layers are changed into three-dimensional convolution, pooling, and normalization layers respectively.

5. The automatic detection of lung CT regions of interest based on a multi-task convolutional model according to claim 1 is characterized in that: The loss function of the three-dimensional classifier and the classification part in multi-task training is FocalLoss, specifically: ,in: Refers to the true label and predicted value of ROI benign and malignant, respectively, and hyperparameters Used to control the weights of positive and negative samples. , positive samples contribute more to the model parameters; the loss function is adjusted by hyperparameters Control the weight of difficult and easy samples, when , the weight of difficult samples is greater during gradient backpropagation, that is, the model pays more attention to difficult samples.

6. The automatic detection of lung CT regions of interest based on a multi-task convolutional model according to claim 1 or 3, characterized in that: The loss function of the two-dimensional and three-dimensional semantic segmentation networks is FocalLoss+DiceLoss, specifically: , ,in: The actual label and predicted value of the ROI mask are respectively, Indicates the first FocalLoss is a pixel loss function that calculates the loss function for each pixel and sums the results. DiceLoss is a regional loss function that is only related to the foreground area in the mask. It is consistent with the final evaluation index and alleviates the problem of class imbalance. However, when the foreground area is too small, the model optimization process will be unstable.

Images