Neural network prediction method for colorectal cancer treatment effect based on MRI and CT images

Abstract

The present invention relates to a neural network prediction method for the treatment effect of colorectal cancer based on MRI and CT images. The method first segments the lesion through a deep learning network, and then automatically extracts the region of interest (ROI) according to the segmentation result. Finally, MRI and CT features are fused through channels, and a convolutional neural network is used to perform pCR classification. The method of the present invention can reduce the tediousness and time-consumingness of manual segmentation of tumor areas in predicting patient pCR.

Description

Technical Field

[0001] The invention relates to a neural network prediction method for colorectal cancer treatment effect based on MRI and CT images. Background Art

[0002] The patent with publication number CN111210909A discloses a deep neural network-based automatic diagnosis system for rectal cancer T staging, including a deep neural network model, which includes: feature extraction network, region generation network, pooling layer, classification and regression layer; first, using the ResNet-50 model, the layer type of the entire input image is learned and judged, the image projection orientation is learned, and a layer recognition module is established; then, using the ResNet101 model as the basic network, a target detection model is trained at each image layer to delineate the tumor area and judge the tumor T stage. The system has consistent processing results and considerable accuracy, which is easy to integrate and apply on a large scale

[0003] The patent with publication number CN113345576A proposes a method for diagnosing lymph node metastasis of rectal cancer based on deep learning multimodal CT, including: preprocessing the data of the multimodal CT images of rectal cancer, and extracting image features from the cropped fixed-size 3D plain scan CT images and 3D enhanced CT images through the newly constructed Mlenet (multimodal Lenet convolutional neural network) convolutional neural network. The feature map is spliced ​​to form a new feature map, and the new feature map is input into the new Mlenet convolutional neural network for binary classification prediction to obtain a binary classification prediction result. The invention can effectively extract features from multimodal CT images and greatly improve the accuracy of lymph node metastasis prediction.

[0004] Disadvantage 1: The patent publication number is CN111210909A. This method is easily affected by the training data set when outlining the lesion area, which can easily lead to misjudgment of the lesion area.

[0005] Disadvantage 2: The patent with publication number CN113345576A only uses multimodal images of CT and cannot make good use of the rich image information contained in CT and MRI images.

[0006] At present, machine learning methods have been applied to predict pathological complete response (pCR) of cancer patients after non-invasive treatment using radiomic features. Wang et al. used omics features and clustering methods of CT images to predict patients' pCR, which also had good results. Zhuang et al. used radiomics and traditional machine learning methods to evaluate the performance of a comprehensive model based on CT and T2-weighted (T2W) MRI-based radiomic signals in pCR prediction. The prediction results showed that the performance was better than that of MRI or CT alone. With the rapid development of deep learning, the use of deep learning to predict pCR has gradually become popular. Fu et al. combined manually extracted MRI omics features with features extracted by deep learning, and used the least absolute shrinkage and selection operator LASSO logistic regression model to predict whether the pathological complete response of patients with locally advanced rectal cancer (LARC) after treatment was achieved, and achieved good accuracy. Jin et al. used a multi-task deep learning network to segment and classify MRI images with different weights before and after non-invasive treatment, and achieved good results in predicting pCR. Although researchers have achieved good performance, it is cumbersome to manually extract omics features, and the extraction of omics features also requires the corresponding doctor-marked lesion areas to achieve.

[0007] Manual identification of tumors and residual lesions after treatment is an error-prone and tedious process for radiologists; therefore, the adoption of automated systems is critical. Traditional machine learning methods use CT and MRI combined models to evaluate PCR, but most current deep learning studies use a single image type and cannot make good use of the different types of information contained in CT and MRI images. The purpose of medical image fusion is to improve image quality while preserving image features and to improve the clinical applicability of images in diagnosing and evaluating medical problems. On organs without peristalsis, Bashir et al. used CT and MRI images of the brain and performed multimodal fusion using PCA dimensionality reduction; Zhizhong et al. used gradient similarity filters to perform multimodal fusion on CT and MRI images of the brain. However, due to the peristalsis of most organs, traditional MRI and CT image fusion techniques have the possibility of feature artifacts. Summary of the invention

[0008] The purpose of the present invention is to provide a neural network prediction method for the treatment effect of colorectal cancer based on MRI and CT images, which can reduce the tediousness and time-consumingness of manual segmentation of tumor areas in predicting patient pCR.

[0009] To achieve the above-mentioned purpose, the technical solution of the present invention is: a neural network prediction method for the treatment effect of colorectal cancer based on MRI and CT images. First, the tumor area of ​​the input image is segmented by a deep learning network; then, the region of interest ROI is automatically extracted according to the segmentation result; finally, MRI and CT features are fused through channels, and a convolutional neural network is used for pCR classification.

[0010] In one embodiment of the present invention, the deep learning network is a CE-Net network.

[0011] In one embodiment of the present invention, the CE-Net network includes an encoding end network and a decoding end network, the encoding end network includes 5 semantic modules, semantic modules 1-4 have one input and two outputs, wherein the input of semantic module 1 is a preprocessed input image, one output is used as the input of semantic module 2 after downsampling operation, and the other output is used as a side output to the decoding layer of the corresponding decoding end network; the output of semantic module 5 is used as the input of the context extractor; the decoding end network includes 5 convolution modules, convolution modules 1-4 have two inputs and one output, one input comes from the output of the encoding layer of the corresponding encoding end network, and the other input comes from the upsampled features; convolution module 5 has only one input, which receives the output from the context extractor.

[0012] In one embodiment of the present invention, the context extractor is composed of a dense dilated convolution module and a residual multi-core pooling module. In the dense dilated convolution module, five branches are cascaded together, and the dilated convolution receptive fields of each branch are 3, 7, 9, and 19, respectively. The remaining branch is directly passed on, and the dilated convolution expansion sizes of the four dilated convolution branches increase from 1 to 1, 3, and 5. 1×1 convolution is used for linear activation in the four dilated convolution branches; the residual multi-core pooling module relies on multiple different receptive fields to encode and detect global targets, and the residual multi-core pooling module uses four different sizes of pooling kernels for downsampling: 2×2, 3×3, 5×5, and 6×6. The four branches output four feature maps of different sizes, and then the number of feature channels is changed to 1 after 1×1 convolution after the four branches, and then bilinear interpolation is performed on the feature maps of different sizes to make them have the same size as the module input feature map. Finally, the original feature map is spliced ​​and fused with the feature map after bilinear interpolation.

[0013] In one embodiment of the present invention, the input image needs to be processed by a Laplacian Gaussian filter before segmentation. The formula of the Laplacian Gaussian filter is as follows:

[0014]

[0015] Where σ is the Gaussian standard deviation, LoG(x,y) is the pixel point after filtering by the Gaussian Laplacian filter, x, y are the horizontal pixel and vertical pixel values ​​respectively, and e is a natural constant.

[0016] In an embodiment of the present invention, the region of interest ROI is automatically extracted according to the segmentation result, that is, the tumor region and its surrounding area of ​​the image are located and cropped using a binary mask obtained by segmentation to obtain the region of interest ROI.

[0017] In one embodiment of the present invention, the convolutional neural network is a CNN multimodal classification network, comprising 1 fusion layer, 3 convolutional layers, 3 maximum pooling layers, 2 fully connected layers and 1 output layer.

[0018] The present invention also provides a neural network prediction system for colorectal cancer treatment effects based on MRI and CT images, including a segmentation module and a classification module;

[0019] The segmentation module uses the CE-Net network to automatically segment the tumor area of ​​the input image, and uses the binary mask obtained by the segmentation to locate and crop the tumor area of ​​the image and its surrounding areas to obtain the region of interest ROI;

[0020] The classification module uses interpolation on the region of interest ROI to restore the cropped image to images of different sizes, and then uses a convolutional neural network to perform pCR classification.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. A deep learning model integrating rectal tumor segmentation and pCR classification was established, which provided new ideas and methods for clinicians to manually segment tumor areas and determine whether LARC patients are in pCR after non-invasive treatment.

[0023] 2. The context extractor of CE-Net is utilized. The context extractor can extract multi-scale features and the connection between features to the greatest extent, greatly improving the accuracy of network segmentation of lesion areas.

[0024] 3. By adopting a multimodal approach and making full use of the rich information features of images of different categories, the performance and generalization of the model's disease classification have been greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the overall framework of the present invention.

[0026] Figure 2 This is the CE-Net network structure of the present invention.

[0027] Figure 3This is the DAC module structure of the present invention.

[0028] Figure 4 This is the RMP module structure of the present invention.

[0029] Figure 5 This is the CNN network structure of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0031] The present invention provides a neural network prediction method for the treatment effect of colorectal cancer based on MRI and CT images. First, the tumor area of ​​the input image is segmented by a deep learning network; then, the region of interest (ROI) is automatically extracted according to the segmentation result; finally, MRI and CT features are fused through channels, and a convolutional neural network is used for pCR classification.

[0032] The present invention also provides a neural network prediction system for colorectal cancer treatment effects based on MRI and CT images, including a segmentation module and a classification module;

[0033] The segmentation module uses the CE-Net network to automatically segment the tumor area of ​​the input image, and uses the binary mask obtained by the segmentation to locate and crop the tumor area of ​​the image and its surrounding areas to obtain the region of interest ROI;

[0034] The classification module uses interpolation on the region of interest ROI to restore the cropped image to images of different sizes, and then uses a convolutional neural network to perform pCR classification.

[0035] The following are specific examples of the present invention.

[0036] The present invention provides a neural network prediction method for colorectal cancer treatment effect based on MRI and CT images. Figure 1 As shown. In the present invention, a model for predicting pCR is proposed, which is end-to-end segmentation followed by classification. This model is mainly composed of two main modules: a segmentation module and a classification module. The segmentation module is used to automatically segment the tumor area, using the CE-Net network for segmentation, and obtaining more deep image features through the context extractor of the network to avoid the loss of some spatial information. Then, the binary mask obtained by segmentation is used to locate and crop the tumor and its surrounding areas in the image, thereby obtaining a square ROI area. Interpolation is used to restore the cropped image to images of different sizes, and finally a convolutional neural network is used to perform probability prediction on the image, and finally the probability of a patient's pCR is obtained by integrating and voting images of different frames of the same patient. This can reduce the tediousness and time-consuming nature of manual segmentation of the tumor area in predicting the patient's pCR.

[0037] The characteristics of the present invention and the main algorithms adopted are:

[0038] 1. Gaussian Laplace filter preprocessing

[0039] As an excellent edge detection operator, the Laplace operator has been widely used in edge detection. This method detects edges by finding the zero crossing point of the second-order inverse of the image. Since the Laplace operator detects edges by performing differential operations on the image, it is sensitive to discrete points and noise. Therefore, the image is first subjected to Gaussian convolution filtering for noise reduction, and then the Laplace operator is used for edge detection, which can improve the robustness of the operator to noise and discrete points.

[0040] In order to enhance the edge texture expression of the abdominal image, the present invention uses a Gaussian Laplacian filter for processing, then combines the original image with the filtered image, and simultaneously expands the Gaussian smoothing component of the operator to reduce the noise component after superposition.

[0041] The formula for the Laplacian of Gaussian filter is as follows:

[0042]

[0043] Where σ is the Gaussian standard deviation, LoG(x,y) is the pixel point after filtering by the Gaussian Laplacian filter, x, y are the horizontal pixel and vertical pixel values ​​respectively, and e is a natural constant.

[0044] 2. CE-Net segmentation network

[0045] The network structure is as follows Figure 2As shown in the figure, the segmentation network uses the CE-Net network for segmentation. Common deep learning-based segmentation will cause some useful information to be lost due to frequent convolution and pooling, making the segmentation effect unsatisfactory. Therefore, a context extractor is added to the U-Net network and some convolution and encoding and decoding methods are changed, which is the CE-Net network. The encoding end network includes 5 semantic modules. Semantic modules 1-4 have one input and two outputs, where the input of module 1 is an image that has been preprocessed. One output of the output part is downsampled as the input of the next semantic module, and the other output is output as a side to the corresponding decoding layer; the output of semantic module 5 is used as the input of the context extractor. The decoding end network includes 5 convolution modules. Modules 1-4 have two inputs and one output, one input comes from the output of the encoding layer, and the other input comes from the upsampled features. The last module has only one input and receives the output from the context extractor. The input of the semantic module in the encoding layer is downsampled before entering the module. The input undergoes a 3*3 convolution operation with the number of channels unchanged. Then, it undergoes another 3*3 convolution operation to double the number of channels. The output is used as the input of the next semantic module and also as the input of the side connection. The convolution layer has two inputs. After the two inputs are integrated, they go through two sets of 1*1 convolution operations and one set of 3*3 deconvolution operations.

[0046] The context extractor consists of a dense atrous convolution module (DAC) and a residual multi-kernel pooling module (RMP). Figure 3 As shown, in the DAC module, five branches are cascaded together, and the receptive fields of the dilated convolutions of each branch are 3, 7, 9, and 19 respectively, and the remaining branch is passed directly. The advantage of dilated convolution is that it increases the receptive field without losing image information through pooling operations, so that each convolution output contains a wider range of information. The dilated convolution sizes of the four dilated convolution branches increase from 1 to 1, 3, and 5. 1×1 convolution is used for linear activation in the four dilated convolution branches. Generally speaking, the features of larger target areas can be better extracted by convolutions with large receptive fields, and the features of smaller target areas can be better extracted by convolutions with small receptive fields. In this way, by cascading dilated convolutions with different receptive fields together, the DAC module can extract features of objects of different sizes. As shown Figure 4As shown in the figure, RMP mainly relies on multiple different receptive fields to encode and detect global targets. RMP uses four pooling kernels of different sizes for downsampling: 2×2, 3×3, 5×5, and 6×6, and the four branches output four feature maps of different sizes. Then, after the four branches, a 1×1 convolution is performed to change the number of feature channels to 1, and then bilinear interpolation is performed on the feature maps of different sizes to make them have the same size as the module input feature map. Finally, the original feature map is spliced ​​and fused with the bilinearly interpolated feature map.

[0047] 3. CNN multimodal classification network

[0048] The binary mask obtained by the segmentation module network is used to locate and crop the tumor and its surrounding area in the image to obtain a square ROI area. Bilinear interpolation is used to restore the cropped image to images of different sizes. Figure 5 As shown in the figure, a convolutional neural network (CNN) is used to extract and classify the fused features of the cropped CT and MRI images. The classification model mainly contains two types of layers, including convolutional layers for extracting information imaging features and fully connected layers for mapping between convolutional features and patient pCR. It contains 1 fusion layer, 3 convolutional layers, 3 maximum pooling layers, and then 2 fully connected layers and 1 output layer. The fusion layer performs initial convolution on the CT and MRI of the same part of the same patient and then fuses the feature channels; the convolutional layer further convolves the fused features to extract multidimensional features; 3 maximum pooling layers are inserted between the convolutional layer and the fully connected layer to reduce the number of model parameters used to control overfitting; rectified linear units (Relu) are used as nonlinear activation functions for convolutional layers and fully connected layers; batch normalization is added to the network to enhance the generalization ability of the model; the output layer contains a function node, namely the Sigmoid function, which is used to calculate the pCR probability of the input image data.

[0049] Results Analysis

[0050] Due to the irregularity of rectal tumors, the size and shape of tumors in different patients vary greatly, so rectal tumor segmentation is very challenging. The present invention compares the segmentation effects of the U-Net network with the CE-Net network. When using U-Net to extract colorectal tumors, some tumor pixels will be misjudged as non-tumor pixels, which reduces the accuracy of some tumor predictions. The CE-Net network has better prediction effects on tumor boundaries because its context extractor enables the network to learn to extract rich feature information. Table 1 shows the segmentation performance indicators of U-Net and CE-Net on patients' MRI and CT. It can be seen that in the segmentation module, the segmentation effects of the same network for patients' MRI and CT images are slightly different, and CE-Net has better segmentation performance.

[0051] Table 1: Results of segmentation indicators

[0052]

[0053] The classification model idea proposed in the present invention is to perform convolution voting classification on the ROI area of ​​CT and MRI images automatically cropped by the fusion segmentation module, so as to avoid misclassification caused by insufficient features or insufficient discrimination of a single CT or MRI image. In order to verify that the fusion of MRI and CT can have better performance, the present invention divides the experiment into three models: CT model, MRI model, and CT and MRI fusion model for comparative verification. In all experiments, the bilateral statistical significance level was set to 0.05. As can be seen from Table 2, from the performance of the comprehensive model, the accuracy, sensitivity and specificity of the fusion model are 0.826 (0.713-0.894), 0.712 (0.592-0.885) and 0.856 (0.748-0.943), respectively, which are better than the performance indicators of the single modality model.

[0054] Table 2: Results of some classification indicators

[0055]

[0056] The advantages of the method of the present invention are as follows:

[0057] Advantage 1: A deep learning model integrating rectal tumor segmentation and pCR classification was established, which provides a new method for clinicians to manually segment tumor areas and determine whether LARC patients are in pCR after non-invasive treatment.

[0058] Advantage 2: The context extractor of CE-Net is utilized. The context extractor can extract multi-scale features and the connection between features to the greatest extent, greatly improving the accuracy of network segmentation of lesion areas.

[0059] Advantage 3: It adopts a multimodal approach and makes full use of the rich information features of images of different categories, which greatly improves the performance and generalization of the model's disease classification.

[0060] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A neural network prediction method for colorectal cancer treatment effect based on MRI and CT images, characterized in that: First, the tumor area of ​​the input image is segmented through a deep learning network. Then, the region of interest (ROI) is automatically extracted based on the segmentation results. Finally, MRI and CT features are fused through channels, and a convolutional neural network is used to classify pCR. The deep learning network is a CE-Net network; The CE-Net network includes an encoding end network and a decoding end network, the encoding end network includes 5 semantic modules, and semantic modules 1-4 have one input and two outputs, wherein the input of semantic module 1 is a preprocessed input image, one output is used as the input of semantic module 2 after downsampling operation, and the other output is output as a side to the decoding layer of the corresponding decoding end network; one output of semantic module 2 is used as the input of semantic module 3 after downsampling operation, and the other output is output as a side to the decoding layer of the corresponding decoding end network; one output of semantic module 3 is used as the input of semantic module 4 after downsampling operation, and the other output is output as a side to the decoding layer of the corresponding decoding end network; one output of semantic module 4 is used as the input of semantic module 5 after downsampling operation, and the other output is output as a side to the decoding layer of the corresponding decoding end network; the output of semantic module 5 is used as the input of the context extractor; the decoding end network includes 5 convolution modules, convolution modules 1-4 have two inputs and one output, one input comes from the output of the encoding layer of the corresponding encoding end network, and the other input comes from the upsampled features; convolution module 5 has only one input, which receives the output from the context extractor; The context extractor is composed of a dense dilated convolution module and a residual multi-core pooling module. In the dense dilated convolution module, five branches are cascaded together, and the dilated convolution receptive fields of each branch are 3, 7, 9, and 19 respectively. The remaining branch is directly passed on, and the dilated convolution expansion sizes of the four dilated convolution branches increase from 1 to 1, 3, and 5; 1×1 convolution is used for linear activation in the four dilated convolution branches; the residual multi-core pooling module relies on multiple different receptive fields to obtain global targets for encoding detection, and the residual multi-core pooling module uses four different sizes of pooling kernels for downsampling: 2×2, 3×3, 5×5, and 6×6. The four branches output feature maps of four different sizes, and then the number of feature channels is changed to 1 after the four branches through 1×1 convolution, and then the feature maps of different sizes are bilinearly interpolated to make them have the same size as the module input feature map. Finally, the original feature map is spliced ​​and fused with the feature map after bilinear interpolation; The convolutional neural network is a CNN multimodal classification network, which includes 1 fusion layer, 3 convolutional layers, 3 maximum pooling layers, 2 fully connected layers and 1 output layer.

2. The neural network prediction method for colorectal cancer treatment effect based on MRI and CT images according to claim 1, characterized in that: The input image needs to be processed by the Gaussian Laplace filter before segmentation. The formula of the Gaussian Laplace filter is as follows: Where σ is the Gaussian standard deviation, LoG(x,y) is the pixel point after filtering by the Gaussian Laplacian filter, x, y are the horizontal pixel and vertical pixel values ​​respectively, and e is a natural constant.

3. The neural network prediction method for colorectal cancer treatment effect based on MRI and CT images according to claim 1, characterized in that: The method of automatically extracting the region of interest ROI according to the segmentation result is to use the binary mask obtained by segmentation to locate and crop the tumor region and its surrounding area in the image to obtain the region of interest ROI.

4. A neural network prediction system for colorectal cancer treatment effects based on MRI and CT images, characterized in that: Includes segmentation module and classification module; The segmentation module uses the CE-Net network to automatically segment the tumor area of ​​the input image, and uses the binary mask obtained by the segmentation to locate and crop the tumor area of ​​the image and its surrounding areas to obtain the region of interest ROI; The CE-Net network includes an encoding end network and a decoding end network. The encoding end network includes 5 semantic modules. The semantic modules 1-4 have one input and two outputs, wherein the input of the semantic module 1 is a preprocessed input image, one output is used as the input of the semantic module 2 after a downsampling operation, and the other output is used as a side output to the decoding layer of the corresponding decoding end network; one output of the semantic module 2 is used as the input of the semantic module 3 after a downsampling operation, and the other output is used as a side output to the decoding layer of the corresponding decoding end network; one output of the semantic module 3 is used as the input of the semantic module 4 after a downsampling operation, and the other output is used as a side output to the decoding layer of the corresponding decoding end network; one output of the semantic module 4 is used as the input of the semantic module 5 after a downsampling operation, and the other output is used as a side output to the decoding layer of the corresponding decoding end network; the output of the semantic module 5 is used as the input of the context extractor; the decoding end network includes 5 convolution modules. The convolution modules 1-4 have two inputs and one output, one input comes from the encoding of the corresponding encoding end network, and the other output comes from the decoding of the corresponding decoding end network. The output of the coding layer, and the other input comes from the upsampled features; the convolution module 5 has only one input, which receives the output from the context extractor; the context extractor is composed of a dense dilated convolution module and a residual multi-core pooling module. In the dense dilated convolution module, five branches are cascaded together, and the dilated convolution receptive field of each branch is 3, 7, 9, and 19 respectively. The remaining branch is directly passed on, and the dilated convolution expansion size of the four dilated convolution branches increases from 1 to 1, 3, and 5; 1×1 convolution is used in the four dilated convolution branches. Linear activation; The residual multi-core pooling module relies on multiple different receptive fields to obtain global targets for encoding detection. The residual multi-core pooling module uses four pooling kernels of different sizes for downsampling: 2×2, 3×3, 5×5 and 6×6. The four branches output four feature maps of different sizes, and then the number of feature channels is changed to 1 after 1×1 convolution after the four branches. Then, bilinear interpolation is performed on the feature maps of different sizes to make them have the same size as the module input feature map. Finally, the original feature map is spliced ​​and fused with the feature map after bilinear interpolation; The classification module uses interpolation on the region of interest ROI to restore the cropped image to images of different sizes, and then uses a convolutional neural network to perform pCR classification; The convolutional neural network is a CNN multimodal classification network, which includes 1 fusion layer, 3 convolutional layers, 3 maximum pooling layers, 2 fully connected layers and 1 output layer.

Images