Method and device for predicting conversion therapy by using preoperative CT (Computed Tomography) image

Abstract

The invention relates to a method for predicting a conversion therapy by using a preoperative CT (Computed Tomography) image, which belongs to the field of image processing and comprises the following steps of: preprocessing data to obtain an image for training a deep learning model; adding multiple model iterations on the basis of knowledge distillation, and introducing label smoothing, cosine dynamic learning rate adjustment and model noise to obtain a progressive distillation method for training the model; and performing multi-round training and testing on the model which adopts the OfficientNet and loads the ImageNet according to the training set and the testing set which are divided according to the set proportion, and performing prediction through the final model after the multi-round training and testing to obtain a prediction result. According to the invention, the computed tomography sign of the primary tumor can be combined with artificial intelligence, the response of the patient in the late gastric cancer to the conversion therapy is predicted, and a good diagnosis effect is generated.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for predicting conversion therapy using preoperative CT images. Background Art

[0002] The latest epidemiological data show that gastric cancer ranks fifth in incidence and fourth in mortality among malignant tumors worldwide; due to the lack of obvious specificity in the clinical status of early gastric cancer, most gastric cancer patients are often diagnosed only when the disease progresses. Due to factors such as extensive distant metastasis, peritoneal metastasis or local progression, some patients with advanced gastric cancer cannot undergo radical resection surgery and can only receive conservative treatment to delay disease progression. How to better treat such patients and improve prognosis remains a challenge; in recent years, with the in-depth understanding of the occurrence, development and biological behavior characteristics of gastric cancer, as well as the development and application of new anticancer drugs, the treatment concepts and strategies of advanced gastric cancer have gradually undergone profound changes. At present, various strategies to improve survival rates have emerged in an endless stream, among which conversion therapy has been proven to be effective in patients with unresectable advanced gastric cancer and has improved survival rates.

[0003] However, due to differences and heterogeneity in tumor biology, the implementation of conversion therapy is not precise, and not all patients can benefit from conversion therapy; in clinical practice, histopathological examination is still the gold standard for evaluating response, but histopathological examination can only be obtained after surgery, delaying the timely adjustment of treatment. Therefore, how to predict the response of patients with advanced gastric cancer to conversion therapy and evaluate the feasibility of implementing conversion therapy are issues that need to be considered at present. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and device for predicting conversion therapy using preoperative CT images, thereby solving the shortcomings of the prior art.

[0005] The object of the present invention is achieved by the following technical solution: a method for predicting conversion therapy using preoperative CT images, the method comprising: Step 1: Preprocess the data to obtain images for training the deep learning model; Step 2: Add multiple model iterations based on knowledge distillation, introduce label smoothing, cosine dynamic learning rate adjustment and model noise, and obtain a progressive distillation method for training the model; Step 3: Perform multiple rounds of training and testing on the EfficientNet network pre-trained on the ImageNet dataset according to the training set and test set divided into the set ratio, and obtain the prediction results by predicting the final model after multiple rounds of training and testing.

[0006] The data preprocessing in step 1 includes: S11, check the original abdominal CT slices. If the patient's gastric cancer tumor has metastasized outside the abdominal cavity, delete the patient's image and do not use the patient's image for model training; S12, review the original abdominal CT slices, delete the patient images with low image quality, and do not use the patient images for model training; S13, check the original abdominal CT slices, delete all images of the patient except the portal venous phase, and only keep the portal venous phase images for model training; S14, convert all images to a window width of 35 and a window position of 250; S15. Scale the obtained image to 256×256 pixels, randomly crop it to 224×224 pixels, and then perform data augmentation processing of random flipping and rotation.

[0007] The step 2 specifically includes the following contents: S21, initialize and train the teacher model. After obtaining the teacher model, the teacher model outputs the soft label prediction of the image and passes the soft label to the student model for imparting knowledge. The teacher model freezes itself in the process of transferring knowledge, that is, the soft label prediction does not change; S22, the first round of progressive distillation iteration, model noise is added to student model 1 during the training process, and after the training is completed, student model 1 outputs a soft label for the KLD loss function to calculate the loss, and the two losses are multiplied by their respective coefficients to obtain the final loss of student model 1; S23, re-using the student model 1 as the teacher model, outputting soft labels to be passed to the student model 2 for imparting knowledge; S24, student model 2 receives the knowledge transfer from student model 1, learns, and finally serves as the student model after the teacher’s teaching.

[0008] The step three specifically includes the following contents: S31. The training model uses EffcientNet and loads the ImageNet pre-trained model. In the training process summary, the stochastic gradient descent method is used as the optimizer, the initial learning rate is 0.001, the momentum is 0.9, the weight decay is 0.0005, and the training rounds are NM rounds. During this period, the cosine annealing dynamic learning rate adjustment strategy is used, gradually decaying from 0.001, decaying according to the cosine function, the lowest point is 0.00001, and the lowest point is reached in the M rounds; S32, the trained model outputs the predicted probability that all images of each patient in the test set are classified into class 1 according to the CT images, and calculates the average of the 10 highest probabilities as the prediction result. If the average is greater than the threshold, it is classified into class 1, otherwise it is classified into class 0, where class 1 represents the poor response class and class 0 represents the good response class; S33. Prediction is performed using the final model after multiple rounds of training and testing to obtain prediction results.

[0009] A device for predicting conversion therapy using preoperative CT images, the device comprising: a data preprocessing module, a progressive distillation module and a training and testing module; The data preprocessing module is used to preprocess the data to obtain images for training the deep learning model; The progressive distillation module is used to add multiple model iterations on the basis of knowledge distillation, and introduce label smoothing, cosine dynamic learning rate adjustment and model noise to obtain a progressive distillation method for training the model; The training and testing module is used to perform multiple rounds of training and testing on the EfficientNet network pre-trained on the ImageNet dataset according to the training set and test set divided into a set ratio, and obtain the prediction result by predicting the final model after multiple rounds of training and testing.

[0010] The data preprocessing module specifically includes the following contents: Check the original abdominal CT slices. If the patient's gastric cancer has metastasized outside the abdominal cavity, delete the patient's image and do not use it for model training. Review the original abdominal CT slices, delete patient images with low image quality, and do not use patient images for model training; Check the original abdominal CT slices, delete all images of the patient except the portal venous phase, and only keep the portal venous phase images for model training; Convert all images to a window width of 35 and a window level of 250; The resulting image is scaled to 256×256 pixels, randomly cropped to 224×224 pixels, and then randomly flipped and rotated for data augmentation.

[0011] The progressive distillation module specifically includes the following contents: Initialize and train the teacher model. After obtaining the teacher model, the teacher model outputs the soft label prediction for the image and passes the soft label to the student model for knowledge transfer. The teacher model freezes itself during the process of knowledge transfer, that is, the soft label prediction does not change. In the first round of progressive distillation iteration, model noise is added to student model 1 during the training process. After training, student model 1 outputs a soft label for KLD loss function to calculate the loss. The two losses are multiplied by their respective coefficients to obtain the final loss of student model 1. The student model 1 is used as the teacher model again, and the output soft labels are passed to the student model 2 for knowledge transfer; Student model 2 receives the knowledge transfer from student model 1, learns, and finally serves as the student model after teacher teaching again.

[0012] The training and testing module specifically includes the following contents: The training model uses EffcientNet and loads the ImageNet pre-training model. In the training process, the stochastic gradient descent method is used as the optimizer, the initial learning rate is 0.001, the momentum is 0.9, the weight decay is 0.0005, and the training rounds are NM rounds. During this period, the cosine annealing dynamic learning rate adjustment strategy is used to gradually decay from 0.001, and decay according to the cosine function. The lowest point is 0.00001, and the lowest point is reached in the M rounds; The trained model outputs the predicted probability that all images of each patient in the test set are classified into class 1 based on the CT image, and calculates the average of the 10 highest probabilities as the prediction result. If the average is greater than the threshold, it is classified into class 1, otherwise it is classified into class 0, where class 1 represents the poor response class and class 0 represents the good response class. The prediction results are obtained by performing predictions on the final model after multiple rounds of training and testing.

[0013] The present invention has the following advantages: a method and device for predicting conversion therapy using preoperative CT images, which can combine the computed tomography (CT) signs of primary tumors with artificial intelligence to predict the response of patients with advanced gastric cancer to conversion therapy, and produce good diagnostic results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural schematic diagram of the progressive distillation training of the present invention; Figure 2 The following are CT images and heat maps of four types of clinical responses in four patients with advanced gastric cancer. Figure a is an image of a 69-year-old male with complete remission as the pathological response, Figure b is an image of a 69-year-old male patient with partial remission as the pathological response, Figure c is an image of a 59-year-old man with stable disease as the pathological response, and Figure d is an image of a 58-year-old man with progressive disease as the pathological response. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided below in conjunction with the drawings is not intended to limit the scope of protection of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application. The present invention is further described below in conjunction with the drawings.

[0016] The present invention specifically relates to a method for predicting conversion therapy using preoperative CT images, which specifically includes the following contents: 1. Data preprocessing: There are 169 patients in total, each of whom has an indefinite number of original abdominal CT images. The images of each patient are preprocessed to obtain images for training; specifically, the following sub-steps are included:.

[0017] 1.1 Check the original abdominal CT slices. If the patient's gastric cancer tumor has metastasized outside the abdominal cavity, delete the patient's image and do not use the patient's data for model training; 1.2 Check the original abdominal CT slices, delete the patient images with low image quality, and do not use the patient data for model training; 1.3 Check the original abdominal CT slices, delete all images of the patient except the portal venous phase, and only keep the portal venous phase images for model training; 1.4 Convert all images to a window width of 35 and a window position of 250; 1.5 The image obtained after the above steps is scaled to 256×256 pixels, randomly cropped to 224×224 pixels, and subjected to data enhancement processing such as random flipping and rotation to obtain an image for training the deep learning model.

[0018] 2. Model training method: Based on knowledge distillation, the present invention adds multiple model iterations and introduces functions such as label smoothing, cosine dynamic learning rate adjustment, and model noise, thereby obtaining a method named progressive distillation; the method specifically includes the following contents: 2.1 Knowledge distillation: The main goal of knowledge distillation is to minimize the difference between the student network and the teacher network through loss function evaluation. In this way, the student network can learn the knowledge imparted by the teacher network and then obtain comparable or better performance than the teacher network. During the training process, the present invention adds distillation loss to suppress the deviation of the normalized exponential function (softmax function) between the teacher model and the student model. Given an input, assume that p is the true probability distribution, z and r represent the outputs of the last fully connected layer of the student model and the teacher model, respectively. In the equation, (p, softmax(z)) represents the cross entropy loss, which measures the difference between the true probability distribution p and the softmax output z. Therefore, the loss function (p, softmax(z)) becomes: ; in, is a temperature hyperparameter used to smooth the softmax output, thereby extracting knowledge of label distribution from the teacher’s predictions; α is a coefficient between 0 and 1.

[0019] 2.2 Using traditional knowledge distillation to extract knowledge to the student model may lead to premature overfitting, which limits the potential of model performance and causes significant performance degradation after several training rounds. In order to alleviate this problem and enhance the robustness of the model, the present invention introduces a difficult learning environment in terms of model noise for the student model. During the student model training process, Dropout and random depth are used as model noise.

[0020] 2.3 The introduction of model noise often hinders the student model from achieving the maximum learning ability in a single distillation process. Therefore, a multiple iteration method is adopted to gradually refine the teacher knowledge. In addition, the present invention also hopes that the student model can better adapt to the model noise by using an equal or larger student model so that the student model has sufficient capacity to acquire knowledge. The iteration of the invention mainly consists of three steps: (1) training the teacher model in conventional training; (2) using the teacher model to generate soft labels for distillation loss; (3) a loss function composed of Kullback-Leibler divergence loss (KLDivLoss) and student loss (i.e., cross entropy loss), while adding the model noise mentioned in 2.2 to train the student model. The student is regarded as a teacher to generate soft labels, and is used to train a new student, thereby iterating the algorithm multiple times.

[0021] 2.4 Adjusting the learning rate is crucial in training because it helps the model converge more stably and avoids sudden jumps or oscillations during training. A simplified version of the cosine annealing strategy is to reduce the learning rate from the initial value to 0 through the cosine function. Assuming that the total number of batches is T (the warm-up phase is negligible), then at batch t, ​​the learning rate ηt is calculated as: ; Where η is the initial learning rate. This study used cosine learning rate decay during model training.

[0022] 2.5 Label smoothing is a regularization technique used during training to prevent the model from becoming overconfident and overly dependent on the training data. It replaces hard labels with soft labels, distributes some probabilities to other categories, and changes the construction of the true probability to: ; Where β is a constant and K is the number of labels. In this study, label smoothing is applied to the cross entropy loss.

[0023] 2.6 Progressive distillation: 2.1~2.5 are combined to form the progressive distillation method, which is the combination of knowledge distillation and multiple iterations.

[0024] Furthermore, if Figure 1 As shown in the figure, the first step is to initialize and train the teacher model. After obtaining the teacher model, the teacher model outputs the soft label prediction for the image and passes the soft label to the student model for knowledge transfer. The teacher model freezes itself in the process of transferring knowledge, that is, the soft label prediction will not change. The second step is the first round of progressive distillation iteration. The student model 1 adds model noise during the training process; after the training is completed, the student model 1 will output a soft label for the KLD loss function to calculate the loss, and output a hard label for the student loss function (i.e., the cross entropy loss function) to calculate the loss. The two losses are multiplied by their respective coefficients (α parameter in 2.1) to obtain the final loss of the student model 1. Then the student model 1 is used as the teacher model again, and the soft label is output for passing to the student model 2 for knowledge transfer. The next step is the same iteration, that is, the student model 2 receives the knowledge transfer from the student model 1, learns by itself, and finally serves as a teacher to teach the student model 3 again.

[0025] 3. Model training and testing: Finally, 140 patients were included in the data set, and the training set and test set were divided into 8:2 ratios, with a total of 112 patients in the training set and 28 patients in the test set. At the same time, according to the postoperative pathological response grading of the patients, the patients were divided into 0 categories of good response and 1 categories of poor response. The training and testing details are as follows: 3.1 Model training: The training model uses EfficientNet and loads the ImageNet pre-trained model. During the training process, we use stochastic gradient descent (SGD) as the optimizer, with an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005. The training rounds are 15 to 30 rounds. During this period, the cosine annealing dynamic learning rate adjustment strategy is used, gradually decaying from 0.001, decaying according to the cosine function, with the lowest point of 0.00001, reaching the lowest point in 30 rounds. The training environment is Python 3.9.16 version, using the Pytorch framework, version 2.0.1.

[0026] 3.2 Model testing: The trained model outputs the predicted probability that all images belonging to each patient in the test set are classified into class 1 based on the CT images, and then calculates the average of the 10 highest probabilities as the prediction result. If the value is greater than the threshold, it is classified into class 1, otherwise it is classified into class 0.

[0027] After multiple rounds of training and testing, the final model is obtained for prediction.

[0028] like Figure 2 As shown, the heat map generated using the gradient weighted activation mapping method shows that the red and yellow areas represent areas activated by the deep neural convolutional network and have the highest predictive significance. In contrast, the green and blue backgrounds reflect areas with weaker predictive value. The darker the feature color, the greater the response and contribution of the corresponding area in the original image to the network. This shows that the deep learning network focuses on the image features that are most predictive of clinical response.

[0029] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used for various other combinations, modifications and improvements, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not deviate from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for predicting conversion therapy using preoperative CT images, characterized in that: The method comprises: Step 1: Preprocess the data to obtain images for training the deep learning model; Step 2: Add multiple model iterations based on knowledge distillation, introduce label smoothing, cosine dynamic learning rate adjustment and model noise, and obtain a progressive distillation method for training the model; Step 3: Perform multiple rounds of training and testing on the EfficientNet network pre-trained on the ImageNet dataset according to the training set and test set divided into the set ratio, and obtain the prediction results by predicting the final model after multiple rounds of training and testing.

2. The method for predicting conversion therapy using preoperative CT images according to claim 1, characterized in that: The data preprocessing in step 1 includes: S11, check the original abdominal CT slices. If the patient's gastric cancer tumor has metastasized outside the abdominal cavity, delete the patient's image and do not use the patient's image for model training; S12, review the original abdominal CT slices, delete the patient images with low image quality, and do not use the patient images for model training; S13, check the original abdominal CT slices, delete all images of the patient except the portal venous phase, and only keep the portal venous phase images for model training; S14, convert all images to a window width of 35 and a window position of 250; S15. Scale the obtained image to 256×256 pixels, randomly crop it to 224×224 pixels, and then perform data augmentation processing of random flipping and rotation.

3. The method for predicting conversion therapy using preoperative CT images according to claim 1, characterized in that: The step 2 specifically includes the following contents: S21, initialize and train the teacher model. After obtaining the teacher model, the teacher model outputs the soft label prediction of the image and passes the soft label to the student model for imparting knowledge. The teacher model freezes itself in the process of transferring knowledge, that is, the soft label prediction does not change; S22, the first round of progressive distillation iteration, model noise is added to student model 1 during the training process, and after the training is completed, student model 1 outputs a soft label for the KLD loss function to calculate the loss, and the two losses are multiplied by their respective coefficients to obtain the final loss of student model 1; S23, re-using the student model 1 as the teacher model, outputting soft labels to be passed to the student model 2 for imparting knowledge; S24, student model 2 receives the knowledge transfer from student model 1, learns, and finally serves as the student model after the teacher’s teaching.

4. The method for predicting conversion therapy using preoperative CT images according to claim 1, characterized in that: The step three specifically includes the following contents: S31. The training model uses EffcientNet and loads the ImageNet pre-trained model. In the training process summary, the stochastic gradient descent method is used as the optimizer, the initial learning rate is 0.001, the momentum is 0.9, the weight decay is 0.0005, and the training rounds are NM rounds. During this period, the cosine annealing dynamic learning rate adjustment strategy is used, gradually decaying from 0.001, decaying according to the cosine function, the lowest point is 0.00001, and the lowest point is reached in the M rounds; S32, the trained model outputs the predicted probability that all images of each patient in the test set are classified into class 1 according to the CT images, and calculates the average of the 10 highest probabilities as the prediction result. If the average is greater than the threshold, it is classified into class 1, otherwise it is classified into class 0, where class 1 represents the poor response class and class 0 represents the good response class; S33. Prediction is performed using the final model after multiple rounds of training and testing to obtain prediction results.

5. A device for predicting conversion therapy using preoperative CT images, characterized in that: The device comprises: a data preprocessing module, a progressive distillation module and a training and testing module; The data preprocessing module is used to preprocess the data to obtain images for training the deep learning model; The progressive distillation module is used to add multiple model iterations on the basis of knowledge distillation, and introduce label smoothing, cosine dynamic learning rate adjustment and model noise to obtain a progressive distillation method for training the model; The training and testing module is used to perform multiple rounds of training and testing on the EfficientNet network pre-trained on the ImageNet dataset according to the training set and test set divided into a set ratio, and obtain the prediction result by predicting the final model after multiple rounds of training and testing.

6. The device for predicting conversion therapy using preoperative CT images according to claim 5, characterized in that: The data preprocessing module specifically includes the following contents: Check the original abdominal CT slices. If the patient's gastric cancer has metastasized outside the abdominal cavity, delete the patient's image and do not use it for model training. Review the original abdominal CT slices, delete patient images with low image quality, and do not use patient images for model training; Check the original abdominal CT slices, delete all images of the patient except the portal venous phase, and only keep the portal venous phase images for model training; Convert all images to a window width of 35 and a window level of 250; The resulting image is scaled to 256×256 pixels, randomly cropped to 224×224 pixels, and then randomly flipped and rotated for data augmentation.

7. The device for predicting conversion therapy using preoperative CT images according to claim 5, characterized in that: The progressive distillation module specifically includes the following contents: Initialize and train the teacher model. After obtaining the teacher model, the teacher model outputs the soft label prediction for the image and passes the soft label to the student model for knowledge transfer. The teacher model freezes itself during the process of knowledge transfer, that is, the soft label prediction does not change. In the first round of progressive distillation iteration, model noise is added to student model 1 during the training process. After training, student model 1 outputs a soft label for KLD loss function to calculate the loss. The two losses are multiplied by their respective coefficients to obtain the final loss of student model 1. The student model 1 is used as the teacher model again, and the output soft labels are passed to the student model 2 for knowledge transfer; Student model 2 receives the knowledge transfer from student model 1, learns, and finally serves as the student model after teacher teaching again.

8. The device for predicting conversion therapy using preoperative CT images according to claim 5, characterized in that: The training and testing module specifically includes the following contents: The training model uses EffcientNet and loads the ImageNet pre-training model. In the training process, the stochastic gradient descent method is used as the optimizer, the initial learning rate is 0.001, the momentum is 0.9, the weight decay is 0.0005, and the training rounds are NM rounds. During this period, the cosine annealing dynamic learning rate adjustment strategy is used to gradually decay from 0.001, and decay according to the cosine function. The lowest point is 0.00001, and the lowest point is reached in the M rounds; The trained model outputs the predicted probability that all images of each patient in the test set are classified into class 1 based on the CT image, and calculates the average of the 10 highest probabilities as the prediction result. If the average is greater than the threshold, it is classified into class 1, otherwise it is classified into class 0, where class 1 represents the poor response class and class 0 represents the good response class. The prediction results are obtained by performing predictions on the final model after multiple rounds of training and testing.

Images