Colorectal cancer prediction model based on CNN and Transformer

Abstract

The invention provides a colorectal cancer peritoneal metastasis prediction model combining a convolutional neural network (CNN) and a Transformer model, and aims to improve the accuracy and efficiency of early detection and diagnosis of colorectal cancer. By collecting abdominal CT image data of a colorectal cancer patient before an operation and processing the data, a deep learning network based on CNN and Swin-Transformer is constructed, and the model can automatically extract and learn important features in the image. And after pre-training the model, further training the model so as to accurately predict the peritoneal metastasis risk of the patient. According to the model, higher prediction accuracy, automatic feature extraction and a more efficient diagnosis process are realized, and application in the fields of colorectal cancer risk prediction and prevention and control is facilitated.

Description

Technical field:

[0001] The present invention relates to a colorectal cancer peritoneal metastasis prediction model combining a convolutional neural network (CNN) and a Transformer model, which is used to improve the accuracy and efficiency of early detection and diagnosis of colorectal cancer. The model can be applied in the risk prediction, prevention and control of colorectal cancer. Background technology:

[0002] In the past decade, the field of quantitative image analysis has experienced significant development, enabling computer technology to automatically extract key quantitative features from medical radiological images. In this process, many researchers have used traditional manual radiomics methods to conduct in-depth analysis of medical images with the aim of predicting chemotherapy response, disease prognosis, and molecular typing in patients with locally advanced rectal cancer. However, the implementation process of radiomics methods is complex and time-consuming, especially the accurate tumor labeling process, which to some extent limits its widespread application in clinical practice.

[0003] With the rise of deep learning (DL) technology, this emerging field of machine learning has shown great potential to solve the above problems. In recent years, a variety of models based on deep learning have been successfully applied to the classification tasks of medical images. They can provide radiologists with additional preoperative information and assist them in formulating more accurate and personalized treatment plans. The core advantage of deep learning is that it can automatically learn and extract highly discriminative features from medical images through advanced neural network architectures such as convolutional neural networks (CNN) and Transformer. Compared with traditional machine learning methods and radiomics technology, deep learning models have shown higher accuracy in capturing the underlying patterns and complex characteristics of tumors and diseases, thereby significantly improving the prediction and diagnostic accuracy of medical image analysis.

[0004] The reason why deep learning models can achieve this breakthrough is mainly due to their powerful feature learning capabilities and high degree of automation. By building a complex neural network structure, deep learning models can automatically discover and learn features that are critical for disease diagnosis and treatment decisions from large-scale medical imaging data. This process not only greatly reduces the reliance on professional knowledge and manual intervention, but also improves the efficiency and accuracy of the analysis process. Therefore, deep learning technology has broad application prospects in the field of medical image analysis, and is expected to provide clinicians with more powerful and accurate auxiliary diagnostic tools. Summary of the invention:

[0005] In order to predict the risk of peritoneal metastasis in patients with colorectal cancer, the present invention discloses a colorectal cancer prognosis peritoneal metastasis prediction model based on the combination of CNN and Transformer.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A method for predicting peritoneal metastasis of colorectal cancer based on deep learning. In the first stage, training set, validation set and test set are established, and a deep learning model based on CNN and Transformer is established; in the second stage, the deep learning classification model we constructed is trained on the constructed dataset. The training is performed on the training set, the validation set is used to guide the selection of parameters, and the test set is used to test the prediction effect.

[0008] Step 1: We first collected a series of raw abdominal computed tomography (CT) image datasets of colorectal cancer patients before surgery. In this process, we focused on identifying and selecting the largest cross-sectional image of the tumor, and then cropped the surrounding area based on the center of the tumor to determine the region of interest (ROI) in the image. In order to meet the input requirements of the deep learning model, we used linear interpolation technology to resize these ROI images to ensure that the resolution of all images was unified to 256×256 pixels. In addition, we also used a stratified random sampling method to scientifically divide the original dataset into training, validation, and test sets, providing a structured data processing basis for subsequent model training and evaluation.

[0009] Step 2: In the construction of the deep learning network, we designed a composite network architecture that integrates CNN and Transformer. In the initial stage, the network adopted the first three stages of the ResNet-50 model to extract the shallow feature maps of the image. The downsampling multiples of these feature maps are consistent with the Patch Partition strategy in Swin-Transformer, ensuring the consistency and effectiveness of feature extraction. On this basis, we made innovative improvements to the Swin-TransformerBlock in the Swin-Transformer network and introduced a local perception enhancement module based on deep separable convolution. This module can use the contextual information in the feature map to dynamically correct the Value vector in the self-attention mechanism, thereby enhancing the model's perception of local features.

[0010] Step 3: In order to fully utilize the advantages of the pre-trained model, we used the images and labels in the ImageNet dataset to pre-train the deep learning model that combines CNN and Transformer. This step not only accelerates the convergence speed of the model, but also improves the model's generalization ability to new data.

[0011] Step 4: Based on the pre-training, we further used the colorectal cancer CT image dataset obtained in step 1 to conduct targeted training on the network model. In this process, we adjusted the number of categories in the last classification layer of the model according to the number of categories in the colorectal cancer database to ensure that the model can adapt to specific classification tasks. For example, in the specific implementation of this study, we set the number of categories in the classification layer to 2 to adapt to the binary classification problem of colorectal cancer peritoneal metastasis.

[0012] Step 5: Finally, after the model training is completed, we sample the ROI area of ​​the new colorectal cancer CT image to be identified to a size of 256×256 pixels through linear interpolation and input it into the trained network model. The model can output the corresponding category label based on the input image features, thereby providing clinicians with predictive information on the patient's peritoneal metastasis risk.

[0013] 1. According to claim 1, the region of interest acquisition method, linear interpolation processing and data set division are characterized in that, in step 1, the region of interest is acquired with the tumor as the center and extended 5 pixels outside the tumor in four directions of up, down, left and right. For linear interpolation, quadratic linear interpolation is used. For length and width less than 256 pixels, a part of the pixels are deleted by scaling. For length and width greater than 256 pixels, pixels are supplemented by quadratic linear interpolation. For data set division, stratified random sampling is used to stratify and randomly divide the original data set into training set, validation set and test set in a ratio of 3:1:1, so as to prepare for training classifiers and evaluating model performance.

[0014] 2. According to the prediction model of colorectal cancer prognosis peritoneal metastasis based on CNN and Transformer combination in claim 1, it is characterized in that in the step 1, the first three stages of ResNet-50 are used to extract shallow feature maps, and a three-stage Swin-Transformer model is used for further feature extraction. The first stage is composed of a pre-embedding layer composed of a fully connected layer and an improved Swin-Transformer module, and the second and third stages are composed of a downsampling layer and an improved Swin-Transformer module. The downsampling layer is composed of a convolutional layer, which reduces the length and width by one half and doubles the number of channels. The improved Swin-Transformer module adds a local perception enhancement module after the Value vector on the window multi-head attention mechanism. The module and the window multi-head self-attention module are on two parallelizable branches. Through efficient separable convolution operations, the spatial attention weight coefficient in the input feature map is extracted, and then the Value vector is corrected through a point-by-point multiplication interaction path.

[0015] Beneficial effects:

[0016] 1. By combining CNN and Transformer models, we can take advantage of their strengths to better capture the potential patterns and complex characteristics of tumors and diseases.

[0017] 2. Utilize deep learning technology to automatically learn distinguishing features from medical images, reducing the tedious and time-consuming manual labeling.

[0018] 3. The model can quickly process and analyze large amounts of medical images, provide additional preoperative information, and help radiologists develop more effective treatment plans. Description of the drawings:

[0019] Figure 1 A flowchart of the training and prediction of the deep learning model provided by the present invention.

[0020] Figure 2 A network structure diagram of a colorectal cancer prognosis peritoneal metastasis prediction model based on a combination of CNN and Transformer is established for the present invention.

[0021] Figure 3 It is a ROC diagram after the model is executed in the implementation mode of the present invention. Specific implementation method:

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the present invention.

[0023] Taking the people who came to a cancer hospital for colorectal cancer testing within five years as an example, a colorectal cancer prognosis peritoneal metastasis prediction model based on the combination of CNN and Transformer was established.

[0024] The present invention implements a sample flow chart of a colorectal cancer prognosis peritoneal metastasis prediction model based on a combination of CNN and Transformer, as shown in FIG. Figure 1 As shown, the following steps are included:

[0025] Step 1: We first collected a series of raw abdominal computed tomography (CT) image datasets of patients with colorectal cancer before surgery. In this process, we focused on identifying and selecting the largest cross-sectional image of the tumor, and then cropped the surrounding area based on the center of the tumor to determine the region of interest (ROI) in the image. In order to meet the input requirements of the deep learning model, we used linear interpolation technology to resize these ROI images to ensure that the resolution of all images was unified to 256×256 pixels. In addition, we also used a stratified random sampling method to scientifically divide the original dataset into training, validation, and test sets.

[0026] Step 2: In the construction of the deep learning network, we designed a composite network architecture that integrates CNN and Transformer. In the initial stage, the network uses the first three stages of the ResNet-50 model to extract shallow feature maps of the image. The downsampling multiples of these feature maps are consistent with the Patch Partition strategy in Swin-Transformer, ensuring the consistency and effectiveness of feature extraction.

[0027] ResNet-50 is a classic deep residual network for image classification and feature extraction. Its structure consists of multiple residual blocks, each of which contains multiple convolutional layers and identity shortcuts. Through cross-layer connections and residual learning mechanisms, it can effectively solve the gradient vanishing and gradient exploding problems in the training process of deep neural networks. The key to ResNet-50 is the introduction of residual modules, which makes the network deeper and easier to train. The feature extraction process of inputting CT images into ResNet-50 is as follows: First, the CT image is used as input, and after a series of convolution and pooling operations, the spatial information of the image is gradually extracted and compressed to obtain a high-dimensional feature representation. Then, these features are processed by a series of residual blocks, each of which transforms and enhances the features through cross-layer connections and nonlinear activation functions (such as ReLU). Inside the residual block, jump connections (identity shortcuts) and residual learning mechanisms are involved, and their mathematical expressions are as follows:

[0028]

[0029] Where x and y represent the input vector and output vector of the residual block respectively. is the residual mapping to be learned, W iis the neural network weight of the i-th layer. Resnet can build Blocks through basic blocks, and different Resnets may have different numbers. For example, Resnet-50 consists of 4 Blocks, each with 3, 4, 6, and 3 basic blocks respectively.

[0030] Step 3: For the backbone network in the feature extraction stage, the Swin-Transformer Block is improved based on the Swin-Transformer network. A local perception enhancement module based on deep separable convolution is used to dynamically modify the Value vector in the self-attention mechanism using the context information in the feature map. The architecture of the model is as follows: Figure 2 shown.

[0031] Swin-Transformer is a Transformer-based image classification model. Its core is to use a layered window self-attention mechanism to process large-scale images. The process of inputting chest fat images into Swin-Transformer includes the following steps: first, the image is divided into small patches and downsampled, then concatenated and LayerNorm processed in the depth direction, and then the self-attention mechanism is applied to calculate the correlation between the query, key and value, and a deep linear transformation is performed through the fully connected layer. The formula of the self-attention mechanism is:

[0032]

[0033] Where (Q), (K), (V) represent query, key, and value matrices, and (d_k) is the dimension of the key. This process enables Swin-Transformer to effectively extract features such as chest fat, providing an important basis for subsequent analysis and application.

[0034] Step 4: Using the image data and labels in ImageNet as a basis, pre-train the deep learning model based on CNN and Transformer. Use the data set obtained in step 1 to train the network model obtained in step 2, and modify the number of categories in the last classification layer to be equal to the number of categories in the colorectal cancer database obtained in step 1 (for example, the number is changed to 2 in the specific implementation, because the number of categories in the colorectal cancer database obtained in step 1 is 2 in the specific implementation, and obtain a trained network model for the colorectal cancer peritoneal metastasis database obtained in step 1.

[0035] Step 5: Linearly interpolate and sample the region of interest of the maximum cross-sectional CT image of colorectal cancer to be identified to a size of 256×256 pixels, and input it into the network model trained in step 3. The trained network model outputs the category label of the input image, which is the peritoneal metastasis risk of the patient corresponding to the image.

[0036] The colorectal cancer prognosis peritoneal metastasis prediction model based on the combination of CNN and Transformer in the embodiment of the present invention has a high accuracy in predicting peritoneal metastasis of colorectal cancer. The ROC result is shown in the figure below: Figure 3 shown.

[0037] The above is a detailed introduction to the implementation rules of the present invention in conjunction with the accompanying drawings. The specific implementation methods herein are only used to help understand the method of the present invention. For ordinary technicians in this technical field, according to the ideas of the present invention, changes and modifications can be made in the specific implementation methods and application scopes. Therefore, this document should not be understood as a limitation of the present invention.

Claims

1. The method for obtaining the region of interest, the linear interpolation processing and the data set division according to claim 1, characterized in that: In step 1, the acquisition of the region of interest is carried out by taking the tumor as the center and extending 5 pixels outside the tumor in four directions, upward, downward, left and right. For linear interpolation, quadratic linear interpolation is used. For length and width less than 256 pixels, some pixels are deleted by scaling, and for length and width greater than 256 pixels, pixels are supplemented by quadratic linear interpolation. For the division of the data set, stratified random sampling is used to stratify and randomly divide the original data set into training set, validation set and test set in a ratio of 3:1:1, in preparation for training the classifier and evaluating the model performance.

2. The colorectal cancer prognosis peritoneal metastasis prediction model based on the combination of CNN and Transformer according to claim 1, characterized in that: In step 1, the first three stages of ResNet-50 are used to extract shallow feature maps, and a three-stage Swin-Transformer model is used for further feature extraction. The first stage consists of a pre-embedding layer composed of a fully connected layer and an improved Swin-Transformer module, and the second and third stages are composed of a downsampling layer and an improved Swin-Transformer module. The downsampling layer is composed of a convolutional layer, which reduces the length and width by half and doubles the number of channels. The improved Swin-Transformer module adds a local perception enhancement module after the Value vector on the window multi-head attention mechanism. The module and the window multi-head self-attention module are on two parallelizable branches. Through efficient separable convolution operations, the spatial attention weight coefficient in the input feature map is extracted, and then the Value vector is corrected through a point-by-point multiplication interaction path.

Images