A 3D deep convolutional neural network approach for CT images

By applying a 3D deep convolutional neural network with forward and reverse input in liver cancer images, combined with deep learning transfer learning and data enhancement technology, the problem of difficulty in classifying small and medium samples in early diagnosis of liver cancer is solved, and efficient identification of liver cancer images is achieved. diagnosis.

CN112614084BActive Publication Date: 2024-11-05SHENYANG INST OF COMPUTING TECH CO LTD THE CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910881070.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-18
Publication Date
2024-11-05
Estimated Expiration
2039-09-18

AI Technical Summary

Technical Problem

Early diagnosis of liver cancer is difficult to accurately carry out. Traditional imaging diagnosis relies on physician experience and is not very accurate. The existing convolutional neural network is not used enough in liver cancer imaging research, which leads to difficulty in classifying liver cancer in small samples.

Method used

Using a 3D deep convolutional neural network method for forward and reverse bidirectional input of CT images, combined with transfer learning of deep learning, the lung cancer data is used to train the model and fine-tune it to solve the classification problem of small sample liver cancer. Through data preprocessing and data augmentation technology, Multi-layer convolutional and pooled layer structures are constructed, and trained with forward and reverse input images to capture complex features.

Benefits of technology

The classification accuracy of small sample liver cancer is improved, the problem of underfitting small samples is overcome, and efficient identification and diagnosis of liver cancer images is achieved through transfer learning and data enhancement technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112614084B_ABST
    Figure CN112614084B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for a 3D deep convolutional neural network with positive and negative bidirectional input for CT images. The method proposes a 3D deep convolutional neural network model with positive and negative bidirectional input for medical image formats. The model adds two input layers on top of the 3D deep convolutional neural network model, corresponding to the positive and negative inputs of the time dimension of the CT image, respectively, and can retain information on the lung slice dimension and extract more image features. The present invention is not only trained on lung cancer image data, but also uses a small amount of liver cancer image data to fine-tune the model, and finally applies the model to liver cancer CT images, which not only has a fast convergence speed, but also can achieve a high accuracy rate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing and medical auxiliary diagnosis, and in particular to a method of a 3D deep convolutional neural network for positive and negative bidirectional input of CT images. Background Art

[0002] Liver cancer is a relatively serious malignant tumor, and its incidence rate is second only to lung cancer. In the early stage of liver cancer, patients generally do not feel any discomfort, but due to the insidious nature of liver cancer, once diagnosed, it is often in the middle and late stages. Therefore, liver cancer is difficult to treat and has poor efficacy. Generally, the survival time after onset is only six months, and the 5-year survival rate is only 2%-16%. Early diagnosis and research of liver cancer is of great significance to improving human health.

[0003] Traditional imaging diagnosis is done through corresponding equipment. Even for an experienced physician, it takes about 10 minutes to conduct a thorough examination of the patient. The physician can assess the malignancy of the tumor based on its morphology, but the accuracy depends largely on the physician's experience, and different doctors may give different predictions.

[0004] The latest advances in deep learning have enabled computer vision models to help doctors diagnose a variety of problems, and in some cases, the models have shown competitiveness with physicians. As of March 2019, there have been no reports of convolutional neural networks applied to imaging studies of liver cancer, but in the diagnosis model research of other diseases, it has achieved a level that cannot be matched by previous two-layer neural networks. Summary of the invention

[0005] In order to better classify small sample liver cancer, the present invention provides a method of 3D deep convolutional neural network with positive and negative bidirectional input for CT images, which can better capture complex features and solve the problem of small sample liver cancer CT image classification based on deep learning transfer learning method.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention comprises the following steps:

[0007] A method for a 3D deep convolutional neural network with forward and reverse bidirectional input of CT images, comprising the following steps:

[0008] A 3D deep convolutional neural network model with positive and negative bidirectional input was established using CT image data of lung cancer and liver cancer. The lung cancer CT image data was divided into a training set and a test set to adjust the model parameters.

[0009] The constructed model is transferred to a small sample of liver cancer disease with similarity for prediction, and the model parameters are further adjusted to obtain the final neural network model, which is used to identify the probability of disease in the image.

[0010] It also includes the pre-processing steps:

[0011] a. Collect CT image data of lung cancer and liver cancer, and convert the image format into a format that can be recognized by computers;

[0012] b. Use a two-dimensional Gaussian filter to denoise the image;

[0013] c. Use linear function to normalize the image;

[0014] d. Screening based on the ratio of the cancer area to the background area in the image to obtain the suspected layer where the lung or liver area is located; further adjusting the threshold of the ratio of the cancer area to the background area to determine the layer where the lung or liver area is located; manually marking the obtained image to obtain a positive sample image of lung or liver disease;

[0015] e. Use data augmentation methods to balance the image datasets of lung or liver diseases.

[0016] The format recognizable by the computer is UNIT8.

[0017] The data enhancement method includes image scaling, translation and rotation, which are used to expand the positive or negative samples of the lung or liver, so that the ratio of positive and negative samples of the lung data and the liver data is balanced.

[0018] The structure of the 3D deep convolutional neural network model with forward and reverse bidirectional input includes multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer, and receives forward input images and reverse input images to train the model.

[0019] Both the forward and reverse sequence images are processed through the convolution layer, pooling layer, convolution layer, pooling layer, and then through the convolution layer, pooling layer, convolution layer, convolution layer, and pooling layer. Finally, they are processed through three fully connected layers and one output layer to output the recognition probability.

[0020] The method of migrating the built model to a liver cancer disease with a similar small sample for prediction includes inputting liver cancer image data into a neural network model to further adjust model parameters.

[0021] It also includes: selecting lung cancer or liver cancer CT images and inputting them into the final neural network model to identify the probability of the image being diseased.

[0022] The present invention has the following advantages and beneficial effects:

[0023] 1. Bidirectional input: forward 3D image and reverse 3D image of the same image. The so-called forward and reverse refer to the third dimension of the image, that is, the time dimension. The reverse 3D image is the reversal of the forward 3D image in the time dimension, which can better capture the complex medical image features.

[0024] 2. Solved the classification problem of liver cancer with small samples; the deep learning model aims to learn a mapping function containing a large number of parameters. The parameter fitting ability learned from small sample data is very limited. Therefore, considering the similarity between lung cancer images and liver cancer images, large sample data is used to learn the model parameters, and then small samples are used to fine-tune the parameters. This method can solve the problem of underfitting of small samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the structure of the 3D deep convolutional neural network model with positive and negative bidirectional input;

[0026] Figure 2 3D deep convolutional neural network model diagram for positive and negative bidirectional input.

[0027] Specific implementation method

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings, and the steps are as follows:

[0029] like Figure 1 As shown, the modeling steps are:

[0030] Step 1: Clean the image. The original datasets (lung data and liver data) are all in Digital Imaging and Communications in Medicine (DICOM) format, so we first use the image format to process the data. The raw data is first converted to Hounsfield units (HU), which is a standard quantitative scale for describing radiation density. Each tissue has its own specific HU range, which is the same for different people. To prepare the data for the deep network, we convert the image from HU to UINT8 (data format), and then normalize the data to between 0 and 255. The original size of the image is 512×512 pixels.

[0031] Step 2: Denoising. CT images are two-dimensional, so a two-dimensional Gaussian filter is needed to denoise the image.

[0032] Step 3: Normalization. Use a linear function to normalize the image.

[0033] The normalization formula is: y = (x-MinValue) / (MaxValue-MinValue), where x and y are the pixel values ​​before and after the conversion, respectively, and MaxValue and MinValue are the maximum and minimum values ​​of the pixels in the image, respectively.

[0034] Step 4: CT image serialization processing. To predict diseases based on CT image data, in order to make the training data more targeted, we first count the approximate layers of the lung or liver areas in the CT image, then select the layers that meet the requirements, and finally manually mark them by doctors to obtain positive samples.

[0035] CT chest images have the characteristics of serialization. In order to extract more correlation information between adjacent layers, in the process of network training, the entire CT sequence needs to be treated as a training sample and assigned a label (divided into diseased and non-disease according to medical markers). A large number of CT images can be obtained through CT imaging devices, but there is some noise in these CT images, that is, some layers of images only contain a small part of the lung or liver image. Therefore, in order to make the training data more targeted, here we first count the approximate layers where the lung or liver area contained in the CT image is located, and then select the layers that meet the requirements. In the screening process, the ratio of the lung area to the background area in the image can be used to select the threshold. For example, assuming that the CT data has 190-320 different layers, among so many layers, some layers may contain a relatively small proportion of lung area. Therefore, the threshold is manually set to select 128 layers containing lungs or liver from each CT data (considering the laboratory equipment and experimental results, 128 layers are found to be the best)

[0036] Step 5: Data enhancement and sample balance. The data enhancement method is used to balance the data set. For the positive sample data set, three methods are used: image scaling (scaling is to scale the image to different sizes), translation (translation is to translate with an affine matrix to change the geometric spatial position of the pixels in the image, the purpose is to record and move the pixel area of ​​the lesion node with a greater impact to the next frame) and rotation (rotate 30-90 degrees with the center of the image, the center of x, the center of y) to expand the positive or negative samples of the lung or liver, and finally make the positive and negative sample ratios of the lung data and liver data 1:1. Finally, a set of sample images suitable for model processing is obtained.

[0037] like Figure 2 The figure shows a 3D deep convolutional neural network model with forward and reverse bidirectional input. The structure of the model includes multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer. It receives forward input images and reverse input images to train the model.

[0038] (Each image in the image set obtained in step 5 is processed using the following steps 6-13)

[0039] After the image processing process from step 1 to step 5, a 3D image matrix with a size of 128*512*512 (128 slice thickness, 512*512 is the image pixel size) can be obtained.

[0040] The forward 3D image and the reverse 3D image are processed through steps 6 to 10 respectively.

[0041] Step 6: Use the time series as the third dimension and the forward 3D image and reverse 3D image of the same image as the input of the model. The image size is: 1*128*512*512. (1 is single channel, 128 is slice thickness, 512*512 is image size).

[0042] Step 7: The 3D image passes through the first convolution layer with a convolution kernel of 3*3*3, a padding value of 1, and a stride of 2. The image size is: 16*128*256*256 (the formula is: O=(I-K+2P) / S+1, where O is the size of the output image, I is the size of the input image, K is the kernel size of the convolution layer, S is the moving step, and P is the padding number. For example: (512-3+2*1) / 2+1=256, 16 is 16 convolution kernels).

[0043] Step 8: The 3D images are passed through the first pooling layer with a stride of 2 and a pooling size of 2*2*2. The image size is: 16*64*256*256 (the formula is: O = (I-Ps) / S+1, where O is the size of the output image, I is the size of the input image, S is the moving step, and Ps is the pooling layer size. For example: ((128-2) / 2+1=64, 16 is 16 convolution kernels).

[0044] Step 9: The 3D images are passed through the second convolution layer with a convolution kernel of 3*3*3, a padding value of 1, and a stride of 2. The image size is: 32*64*128*128 ((256–3+2*1) / 2+1=128, 32 is 32 convolution kernels).

[0045] Step 10: The 3D images are respectively passed through the second pooling layer with a step size of 2 and a pooling size of 2*2*2. The image size is: 32*32*128*128 ((64-2) / 2+1=32, 32 is 32 convolution kernels).

[0046] After steps 6 to 10, the forward and reverse directions are operated separately, and steps 11 to 12 are combined operations:

[0047] Step 11: Input the output of step 10 into a convolution layer with a convolution kernel of 3 and a pooling layer with a stride of 2. The image size is: 128*16*64*64 (64*64 image size: (128–3+2*1) / 2+1=64, 16 slice thickness: (32-2) / 2+1=16, 128 is 128 convolution kernels).

[0048] Step 12: Input the output of step 11 into 2 convolutional layers with a convolution kernel of 3*3*3 and a pooling layer with a stride of 2. The image size is: 256*8*32*32 (32*32 image size: (64–3+2*1) / 2+1=32, 8 slice thickness: (16-2) / 2+1=8, 256 is 256 convolution kernels).

[0049] Step 13: After the first fully connected layer, the image size is: 1*2097152. (The length of the fully connected layer output vector is equal to the number of neurons. For example: 256*8*32*32=2097152)

[0050] Step 14: After the second fully connected layer, the image size is 1*512 (512 represents the number of hidden layers).

[0051] Step 15: After the third fully connected layer, the image size is 1*64 (64 represents the number of hidden layers).

[0052] All image data obtained in step 15 are processed as follows:

[0053] Step 16: After the last output layer (only one node), the output result is y, and the value range of y is [0,1], which represents the probability of lung cancer.

[0054] In this embodiment, lung disease or liver disease is used as the processing object for preprocessing and modeling to obtain a better probability of disease. Since the lung cancer model and the liver cancer model have the same structure but different parameters, a large amount of lung cancer data is first used to train the model to obtain a better model parameter result, and then a small amount of liver cancer data is input into the model to fine-tune the model parameters to obtain a liver cancer classification model.

Claims

1. A method for a 3D deep convolutional neural network with forward and reverse bidirectional input of CT images, characterized in that The following steps are involved: A 3D deep convolutional neural network model with positive and negative bidirectional input was established using CT image data of lung cancer and liver cancer. The lung cancer CT image data was divided into a training set and a test set to adjust the model parameters. The built model is transferred to a similar small sample of liver cancer for prediction, and the model parameters are further adjusted to obtain the final neural network model, which is used to identify the probability of disease in the image; It also includes the pre-processing steps: a. Collect CT image data of lung cancer and liver cancer, and convert the image format into a format that can be recognized by computers; b. Use a two-dimensional Gaussian filter to denoise the image; c. Use linear function to normalize the image; d. Screening based on the ratio of the cancer area to the background area in the image to obtain the suspected layer where the lung or liver area is located; further adjusting the threshold of the ratio of the cancer area to the background area to determine the layer where the lung or liver area is located; manually marking the obtained image to obtain a positive sample image of lung or liver disease; e. Use data augmentation methods to balance the image datasets of lung or liver diseases; The data enhancement method includes image scaling, translation and rotation, which is used to expand the positive or negative samples of the lung or liver, so that the ratio of positive and negative samples of the lung data and the liver data is balanced; The structure of the 3D deep convolutional neural network model with forward and reverse bidirectional input includes multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer, and receives forward input images and reverse input images to train the model.

2. A method for a 3D deep convolutional neural network with forward and reverse bidirectional input of CT images according to claim 1, characterized in that The format recognizable by the computer is UNIT8.

3. The method of a 3D deep convolutional neural network for positive and negative bidirectional input of CT images according to claim 1, characterized in that: Both the forward and reverse sequence images are processed through the convolution layer, pooling layer, convolution layer, pooling layer, and then through the convolution layer, pooling layer, convolution layer, convolution layer, and pooling layer. Finally, they are processed through three fully connected layers and one output layer to output the recognition probability.

4. The method of a 3D deep convolutional neural network for positive and negative bidirectional input of CT images according to claim 1, characterized in that The method transfers the constructed model to a liver cancer disease with a similar small sample for prediction, including inputting liver cancer image data into a 3D deep convolutional neural network model with positive and negative bidirectional inputs to further adjust model parameters.

5. The method of a 3D deep convolutional neural network for positive and negative bidirectional input of CT images according to claim 1, characterized in that It also includes: selecting lung cancer or liver cancer CT images and inputting them into the final neural network model to identify the probability of the image being diseased.