Neoadjuvant chemoradiotherapy curative effect prediction method based on multi-task learning

Abstract

The invention provides a neoadjuvant chemoradiotherapy curative effect prediction method based on multi-task learning, and belongs to the technical field of cancer curative effect prediction, and the method comprises the following steps: training a convolutional neural network model A by using an MSI image data set; marking the NCRT image data set by using the convolutional neural network model A; training a convolutional neural network model B by using the NCRT image data set; marking the MSI image data set by using the convolutional neural network model B; training the convolutional neural network model B by using the MSI image data, and updating the convolutional neural network model B; marking the NCRT image data set by using the convolutional neural network model B; training a convolutional neural network model B by using NCRT image data, calculating classification loss and adjusting loss weight, and updating the convolutional neural network model B; and repeatedly training the convolutional neural network model B, and outputting a neoadjuvant chemoradiotherapy curative effect prediction result. According to the invention, the interpretability and accuracy of neoadjuvant chemoradiotherapy curative effect prediction can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cancer treatment efficacy prediction, and particularly relates to a neoadjuvant chemoradiotherapy efficacy prediction method based on multi-task learning. Background Art

[0002] Cancer continues to be a major challenge to global public health and poses a serious threat to human health. According to the World Health Organization, cancer is one of the leading causes of death globally, with approximately 10 million people dying from various cancers in 2020, accounting for nearly one-sixth of the total global deaths.

[0003] Cancers at different stages have their corresponding treatment regimens. Early-stage cancers have low invasiveness, shallow invasion depth, small scope, and low lymph node metastasis rate. By early detection and combined with surgical resection of tumor cells in the body, and cooperating with subsequent drug and chemoradiotherapy, the cure rate of early-stage cancers can reach over 80%-90%. Due to the high invasiveness of advanced-stage cancers and the frequent occurrence of lymph node metastasis or distant organ metastasis, the treatment of advanced-stage cancer patients mainly uses neoadjuvant chemoradiotherapy (NCRT) as the treatment method. Through systemic chemoradiotherapy before surgery, the tumor mass is reduced and tumor metastatic cells are eliminated, which is of great significance for the subsequent tumor downstaging rate and pathological remission rate. The best effect of surgery is pathologic complete response (PCR). However, statistical data shows that although the cancer downstaging rate is high after neoadjuvant chemoradiotherapy, the PCR rate is very low. This means that neoadjuvant chemoradiotherapy is not suitable for all cancer patients. Some patients are insensitive or even resistant to neoadjuvant chemoradiotherapy, and in a few patients, cancer cell metastasis occurs during treatment, missing the best surgical time. Coupled with the harm caused by the side effects of chemotherapy to the patient's body, it indirectly accelerates the deterioration of the condition. Therefore, predicting the efficacy of preoperative neoadjuvant chemoradiotherapy for advanced-stage cancer patients and thus formulating personalized treatment plans for patients is of great significance for improving the cure rate of cancer.

[0004] Microsatellite instability (MSI) is a phenomenon caused by base replication errors. This phenomenon was first discovered in colorectal cancer and was considered a characteristic of hereditary non-polyposis colorectal cancer. Subsequently, it was also found in various sporadic tumors, such as gastric cancer, lung cancer, and endometrial cancer. Microsatellite instability is considered a good prognostic factor and is of great significance for the diagnosis of cancer. Tumors with high microsatellite instability have poor effects in traditional neoadjuvant chemotherapy or radiotherapy, and microsatellite instability can be detected in digital pathology images.

[0005] Digital pathological images contain rich information on cell tissue structure, which can reflect potential cell expression characteristics and the disease development process, and are regarded as the gold standard for cancer diagnosis. Researchers have explored the field of efficacy prediction by combining digital pathological images. Existing research mostly combines manual screening with machine learning. The manual screening method includes fine-grained annotation of images by experienced physicians and selection of regions of interest (ROIs) in the images. However, the manual screening method requires extremely strong professional knowledge, which to a certain extent limits the possibility of non-medical research teams replicating the work. At the same time, existing research mostly only analyzes digital pathological images, rarely combines knowledge in fields such as biomarkers, and lacks a certain degree of interpretability. For example, in a pathological image of cancer, there are normal regions, diseased regions, and intermediate-state regions. Just looking at the diseased regions, they can be divided into multiple subtypes according to the tumor differentiation situation. Directly putting the image into the model, extracting features and then classifying. Due to the lack of introduction of relevant information, even if the final performance is ideal, it is impossible to determine whether it is due to the model extracting reasonable features or mining pseudo-patterns or pseudo-features.

[0006] There is an urgent need for a cancer prognosis method that does not require additional annotation and comprehensively considers microsatellite instability to improve the interpretability and accuracy of predicting the efficacy of neoadjuvant chemoradiotherapy. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning, which further improves the interpretability of efficacy prediction and the accuracy of classification by introducing microsatellite instability judgment to assist in efficacy prediction.

[0008] To solve the above technical problems, the technical solution adopted by the present invention is a method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning, including the following steps:

[0009] S1. Establish an MSI image dataset and use the MSI image dataset to train a convolutional neural network model A;

[0010] S2. Establish an NCRT image dataset and use the convolutional neural network model A to annotate the NCRT image dataset;

[0011] Construct a convolutional neural network model B including an MSI classification head and an NCRT classification head, use the NCRT image dataset to train the convolutional neural network model B, and calculate the classification loss to backpropagate and update the convolutional neural network model B;

[0012] S3. Use the convolutional neural network model B to annotate the MSI image dataset, generate NCRT pseudo-labels for the MSI image dataset and calculate the reliability;

[0013] S4. Train the convolutional neural network model B using the MSI image data, calculate the classification loss and adjust the loss weights, and update the convolutional neural network model B by backpropagation;

[0014] S5. Use the convolutional neural network model B to label the NCRT image dataset, generate MSI pseudo-labels for the NCRT image dataset and calculate the reliability;

[0015] S6. Train the convolutional neural network model B using the NCRT image data, calculate the classification loss and adjust the loss weights, and update the convolutional neural network model B by backpropagation;

[0016] S7. Repeat training the convolutional neural network model B and output the prediction result of the efficacy of neoadjuvant chemoradiotherapy.

[0017] Further, in step S1, the training of the convolutional neural network model A using the MSI image dataset is as follows:

[0018] Randomly select j groups of data from the MSI image dataset and input them into the convolutional neural network model A. Calculate the overall loss between the output result and the label of the MSI image data, and update the convolutional neural network model A by backpropagation, which is expressed as:

[0019] X = {x m1 , x m2 , …, x mj}

[0020] Y = {y m1 , y m2 , …, y mj}

[0021]

[0022] In the formula, X represents the MSI image dataset, x mj represents the j-th data in the MSI image dataset, f 1 (x mj ) represents the representation obtained after x mj is input into the intermediate layer of the convolutional neural network model A. Y represents the label of the MSI image dataset, y mj represents the label of the j-th data in the MSI image dataset, L cls represents the overall loss, i represents the total number of image data in the MSI image dataset, and l represents the classification loss function.

[0023] Further, in step S2, the labeling of the NCRT image dataset using the convolutional neural network model A is as follows:

[0024] Randomly select k groups of data from the NCRT image dataset and input them into the convolutional neural network model A. Use the output of the convolutional neural network model A as the initial MSI pseudo-label of the NCRT image data, which is expressed as:

[0025] Z = {z n1 , z n2 , …, z nk}

[0026] W = {w n1 , w n2 , …, w nk}

[0027]

[0028] In the formula, Z represents the NCRT image dataset, z nk represents the k-th data in the NCRT image dataset, W represents the label set of the NCRT image dataset, and w nk represents the label of the k-th data in the NCRT image dataset. W' (0) represents the initial MSI pseudo-label set of the NCRT image dataset, represents the initial MSI pseudo-label of the k-th data in the NCRT image dataset.

[0029] Furthermore, in step S3, the use of the convolutional neural network model B to label the MSI image dataset is as follows:

[0030] Input the MSI image dataset into the convolutional neural network model B, and use the output of the NCRT classification head as the initial NCRT pseudo-label of the MSI image dataset, which is expressed as:

[0031]

[0032] In the formula, Y' (0) represents the initial NCRT pseudo-label set of the MSI image dataset, represents the initial NCRT pseudo-label of the j-th data in the MSI image dataset;

[0033] The calculation of the reliability is as follows:

[0034] Calculate the confidence of the initial NCRT pseudo-label of the MSI image dataset, perform data augmentation on the MSI image dataset, calculate the consistency before and after the augmentation of the MSI image dataset, and calculate the reliability, which is expressed as:

[0035]

[0036] In the formula, Represents the prediction confidence of the initial NCRT pseudo-label for the j-th data in the MSI image dataset, conf j Represents the output confidence of the convolutional neural network model B for the initial NCRT pseudo-label of the j-th data in the MSI image dataset. Avg(·) represents the averaging operation, Std(·) represents the standard deviation calculation operation, and i represents the total number of data in the MSI image dataset. Represents the prediction consistency of the j-th data in the MSI image dataset before and after data augmentation, consis j Represents the output consistency of the convolutional neural network model B for the j-th data in the MSI image dataset before and after data augmentation. Represents the reliability of the initial NCRT pseudo-label for the j-th data in the MSI image dataset.

[0037] Furthermore, in step S4, the calculation of the classification loss is expressed as:

[0038]

[0039] Represents the classification loss calculated between the output of the MSI classification head and the MSI image data label. i represents the total number of data in the MSI image dataset, x mj Represents the j-th data in the MSI image dataset, f 2 (x mj ) represents the representation obtained after x mj is input to the intermediate layer of the convolutional neural network model B, y mj Represents the label of the j-th data in the MSI image dataset, and l represents the classification loss function. Represents the classification loss calculated between the output of the NCRT classification head and the initial NCRT pseudo-label of the MSI image data. Represents the initial NCRT pseudo-label of the j-th data in the MSI image dataset. Represents the reliability of the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

[0040] Furthermore, in step S4, the adjusted loss weight is:

[0041] Calculate the variance of the prediction results of the convolutional neural network model B for the MSI image data, which is:

[0042]

[0043] In the formula, μ 2 Represents the mean of the prediction results of the convolutional neural network model B for the MSI image data. Represents the prediction result of the convolutional neural network model B for the j-th data in the MSI image data. Denotes the variance of the prediction result of the convolutional neural network model B for the MSI image data;

[0044] According to Adjust And The weights of, and calculate the overall loss, expressed as:

[0045]

[0046] In the formula Denotes the adjusted Denotes the adjusted σ 2 Is The square root of has no practical significance, Denotes And The overall loss of.

[0047] Furthermore, in step S5, the use of the convolutional neural network model B to label the NCRT image dataset is:

[0048] Input the NCRT image dataset into the convolutional neural network model B, and use the output of the MSI classification head as the iterative MSI pseudo-label of the NCRT image dataset, expressed as:

[0049]

[0050] In the formula, W' (1) Denotes the iterative MSI pseudo-label set of the NCRT image dataset, Denotes the iterative MSI pseudo-label of the k-th data in the NCRT image dataset;

[0051] The calculation of the reliability is:

[0052] Calculate the confidence of the iterative MSI pseudo-labels of the NCRT image dataset, perform data augmentation on the NCRT image dataset, calculate the consistency before and after the augmentation of the NCRT image dataset, and calculate the reliability, expressed as:

[0053]

[0054]

[0055] In the formula, Denotes the prediction confidence of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, conf kRepresents the output confidence of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset by the convolutional neural network model B. Avg(·) represents the averaging operation, Std(·) represents the standard deviation calculation operation, and I represents the total number of data in the NCRT image dataset. Represents the prediction consistency of the k-th data in the NCRT image dataset before and after data augmentation, consis k Represents the output consistency of the convolutional neural network model B for the k-th data in the NCRT image dataset before and after data augmentation. Represents the reliability of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset.

[0056] Furthermore, in step S6, the calculation of the classification loss is as follows:

[0057]

[0058] In the formula, L MSI Represents the classification loss calculated between the output of the MSI classification head and the iterative MSI pseudo-label of the NCRT image data. I represents the total number of data in the NCRT image dataset. Represents the reliability of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset. l represents the classification loss function, z nk Represents the k-th data in the NCRT image dataset, f 2 (z nk ) represents the representation obtained after z nk is input into the intermediate layer of the convolutional neural network model B. Represents the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, L NCRT Represents the classification loss calculated between the output of the NCRT classification head and the label of the NCRT image data. w nk Represents the label of the k-th data in the NCRT image dataset.

[0059] Furthermore, in step S6, the adjustment of the loss weight is as follows:

[0060] Calculate the variance of the prediction results of the convolutional neural network model B for the NCRT image data, which is:

[0061]

[0062] In the formula, μ 1 Represents the mean of the prediction results of the convolutional neural network model B for the NCRT image data. Represents the prediction result of the convolutional neural network model B for the k-th data in the NCRT image data. Represents the variance of the prediction results of the convolutional neural network model B for the NCRT image data.

[0063] According to Adjust L MSI With L NCRT weights, and calculate the overall loss, expressed as:

[0064]

[0065] L all = L' NCRT + L' MSI

[0066] Where L' MSI represents the adjusted L MSI , L' NCRT represents the adjusted L NCRT , σ 1 is the square root of which has no practical significance, and L all represents the overall loss of L' MSI and L' NCRT .

[0067] Furthermore, in step S7, the convolutional neural network model B is repeatedly trained until the loss function value converges or reaches a preset number of iterations, and the output of the NCRT classification head is used as the new auxiliary chemo-radiotherapy efficacy prediction result.

[0068] The beneficial effects of the present invention are as follows:

[0069] (1) The present invention labels the NCRT image data and MSI image data to generate pseudo-labels, and reduces the influence of potential errors of the pseudo-labels on the model classification performance through iterative loop training. By judging whether the digital pathology image shows microsatellite instability, the auxiliary efficacy prediction is realized, and finally the accuracy and interpretability of the efficacy prediction are improved.

[0070] (2) The present invention proposes to use the microsatellite instability information on the pathological image for efficacy prediction. By generating highly reliable pseudo-labels to identify the microsatellite instability information in the image, a new type of supervision information is provided for the existing efficacy dataset, and the addition of this supervision information reduces the difficulty of predicting the efficacy of neoadjuvant chemo-radiotherapy.

[0071] (3) The present invention proposes a training method for judging the reliability of pseudo-labels, which can reduce the negative impact of potential errors of pseudo-labels on the model. And it is proposed to dynamically adjust the corresponding loss weights by calculating the variance of the model classification loss, so as to realize the reasonable update of the convolutional neural network B. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0073] Figure 1 is the cyclic training flowchart of the present invention.

[0074] Figure 2 is the process of judging the reliability degree of pseudo-labels of the present invention. Detailed implementation manners

[0075] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0076] See Figure 1 , the embodiments of the present invention provide a new method for predicting the efficacy of adjuvant chemoradiotherapy based on multi-task learning, including the following steps:

[0077] S1. Establish an MSI image dataset, construct a convolutional neural network model A. The convolutional neural network model A includes an input layer, an intermediate layer, and a classification head. Initialize the parameters of the convolutional neural network model A, randomly extract j groups of data from the MSI image dataset and input them into the convolutional neural network model A for training, calculate the overall loss between the output result and the MSI image data label, and backpropagate to update the convolutional neural network model A, which is expressed as:

[0078] X = {x m1 , x m2 , …, x mj}

[0079] Y = {y m1 , y m2 , …, y mj}

[0080]

[0081] In the formula, X represents the MSI image dataset, x mj represents the jth data in the MSI image dataset, Y represents the label of the MSI image dataset, y mj represents the label of the jth data in the MSI image dataset, and L clsDenote the overall loss, i denote the total number of image data in the MSI image dataset, l denote the classification loss function, and f 1 (x mj ) represents the representation obtained after x mj is input into the intermediate layer of the convolutional neural network model A.

[0082] In the present invention, the MSI image dataset consists of two types of data, namely tumor images and labels showing microsatellite instability, and tumor images and labels not showing microsatellite instability.

[0083] S2. Establish an NCRT image dataset, randomly select k groups of data from the NCRT image dataset and input them into the convolutional neural network model A for annotation, and use the output of the convolutional neural network model A as the initial MSI pseudo-label of the NCRT image data, denoted as:

[0084] Z = {z n1 , z n2 , …, z nk}

[0085] W = {w n1 , w n2 , …, w nk}

[0086]

[0087] In the formula, Z represents the NCRT image dataset, z nk represents the k-th data in the NCRT image dataset, W represents the label set of the NCRT image dataset, w nk represents the label of the k-th data in the NCRT image dataset, and W' (0) represents the initial MSI pseudo-label set of the NCRT image dataset, represents the initial MSI pseudo-label of the k-th data in the NCRT image dataset.

[0088] In the present invention, the NCRT image dataset includes preoperative digital pathology images and postoperative tumor regression grade scores (Tumor Regression Grade, TRG). TRG is divided into four grades. Grade zero represents no viable cancer cells, that is, the neoadjuvant treatment effect is very good; grade one represents single or small clusters of cancer cells remaining, that is, the neoadjuvant treatment effect is medium; grade two represents residual cancer foci with stromal fibrosis, that is, the neoadjuvant treatment effect is poor; grade three represents little or no tumor regression change and a large number of cancer cells remaining, that is, the neoadjuvant treatment effect is poor. In the present invention, patients with grade zero and grade one are classified into one category, and patients with grade two and grade three are classified into one category, and their TRG scores are used to classify the NCRT image dataset.

[0089] Construct a convolutional neural network model B, where the convolutional neural network model B includes an input layer, an intermediate layer, an MSI classification head, and an NCRT classification head; initialize the parameters of the convolutional neural network model B, use the NCRT image dataset to train the convolutional neural network model B, calculate the classification loss and backpropagate to update the convolutional neural network model B, which is expressed as:

[0090]

[0091] In the formula, represents the initial classification loss calculated between the output of the MSI classification head and the initial MSI pseudo-label of the NCRT image data, represents the initial classification loss calculated between the output of the NCRT classification head and the label of the NCRT image data.

[0092] In the convolutional neural network model B of the present invention, an MSI classification head and an NCRT classification head are set. The MSI classification head outputs the presence or absence of microsatellite instability, and the NCRT classification head outputs good or poor efficacy of neoadjuvant chemoradiotherapy.

[0093] S3. Use the convolutional neural network model B to label the MSI image dataset to generate the NCRT pseudo-label of the MSI image dataset, which is expressed as:

[0094] Input the MSI image dataset into the convolutional neural network model B, and use the output of the NCRT classification head as the initial NCRT pseudo-label of the MSI image dataset, which is expressed as:

[0095]

[0096] In the formula, Y' (0) represents the set of initial NCRT pseudo-labels of the MSI image dataset, represents the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

[0097] Calculate the confidence of the initial NCRT pseudo-label of the MSI image dataset, perform data augmentation on the MSI image dataset, and calculate the consistency of the MSI image dataset before and after augmentation. As Figure 2 shown, calculate the reliability, which is expressed as:

[0098]

[0099]

[0100] In the formula, represents the predicted confidence of the initial NCRT pseudo-label of the j-th data in the MSI image dataset, conf jrepresents the output confidence of the initial NCRT pseudo-label of the j-th data in the MSI image dataset by the convolutional neural network model B. Avg(·) represents the averaging operation, and Std(·) represents the standard deviation calculation operation. represents the prediction consistency of the j-th data in the MSI image dataset before and after data augmentation, consis j represents the output consistency of the convolutional neural network model B for the j-th data in the MSI image dataset before and after data augmentation. represents the reliability of the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

[0101] In the present invention, the data augmentation methods include reducing or magnifying the magnification of the original image, cropping any part of the image, and changing the gray level of the original image.

[0102] S4. Train the convolutional neural network model B using the MSI image data, and calculate the classification loss, expressed as:

[0103]

[0104] represents the classification loss calculated between the output of the MSI classification head and the label of the MSI image data, f 2 (x mj ) represents the representation obtained after x mj is input to the intermediate layer of the convolutional neural network model B. represents the classification loss calculated between the output of the NCRT classification head and the initial NCRT pseudo-label of the MSI image data. represents the initial NCRT pseudo-label of the j-th data in the MSI image dataset. represents the reliability of the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

[0105] And adjust the loss weight, and update the convolutional neural network model B by backpropagation, expressed as:

[0106] Calculate the variance of the prediction result of the convolutional neural network model B for the MSI image data, which is:

[0107]

[0108] In the formula, μ 2 represents the mean of the prediction result of the convolutional neural network model B for the MSI image data. represents the prediction result of the convolutional neural network model B for the j-th data in the MSI image data. represents the variance of the prediction result of the convolutional neural network model B for the MSI image data;

[0109] According to Adjust the weights of and calculate the overall loss, expressed as:

[0110]

[0111] where represents the adjusted represents the adjusted σ 2 is the square root of which has no practical significance, represents the overall loss between and

[0112] S5. Use the convolutional neural network model B to label the NCRT image dataset, and generate the MSI pseudo-labels of the NCRT image dataset, expressed as:

[0113] Input the NCRT image dataset into the convolutional neural network model B, and use the output of the MSI classification head as the iterative MSI pseudo-labels of the NCRT image dataset, expressed as:

[0114]

[0115] where W' (1) represents the set of iterative MSI pseudo-labels of the NCRT image dataset, represents the iterative MSI pseudo-label of the k-th data in the NCRT image dataset;

[0116] Calculate the confidence of the iterative MSI pseudo-labels of the NCRT image dataset, perform data augmentation on the NCRT image dataset, calculate the consistency of the NCRT image dataset before and after augmentation, and calculate the reliability, expressed as:

[0117]

[0118] where represents the predicted confidence of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, conf k represents the output confidence of the convolutional neural network model B for the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, I represents the total number of data in the NCRT image dataset, represents the predicted consistency of the k-th data in the NCRT image dataset before and after augmentation, consis k represents the output consistency of the convolutional neural network model B for the k-th data in the NCRT image dataset before and after augmentation, represents the reliability of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset.

[0119] S6. Train the convolutional neural network model B using the NCRT image data and calculate the classification loss, expressed as:

[0120]

[0121] In the formula, L MSI represents the classification loss calculated between the output of the MSI classification head and the iterative MSI pseudo-labels of the NCRT image data, represents the reliability of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, f 2 (z nk ) represents the representation obtained after z nk is input to the intermediate layer of the convolutional neural network model B, represents the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, L NCRT represents the classification loss calculated between the output of the NCRT classification head and the NCRT image data labels.

[0122] And adjust the loss weights, and update the convolutional neural network model B by backpropagation, expressed as:

[0123] Calculate the variance of the prediction results of the convolutional neural network model B for the NCRT image data, which is:

[0124]

[0125] In the formula, μ 1 represents the mean of the prediction results of the convolutional neural network model B for the NCRT image data, represents the prediction result of the convolutional neural network model B for the k-th data in the NCRT image data, represents the variance of the prediction results of the convolutional neural network model B for the NCRT image data;

[0126] According to Adjust the weights of L MSI and L NCRT and calculate the overall loss, expressed as:

[0127]

[0128] L all = L’ NCRT + L’ MSI

[0129] In the formula, L' MSI represents the adjusted L MSI , L' NCRT represents the adjusted L NCRT , σ 1For the square root of has no practical meaning, and L all represents L' MSI and L' NCRT total loss.

[0130] S7. Repeat steps S3 - S6 to continue training the convolutional neural network model B until the value of the loss function converges or reaches the preset number of iterations. When any index is completed, the step ends, and the output of the NCRT classification head is used as the new auxiliary chemo - radiotherapy efficacy prediction result.

[0131] The method of the present invention and other methods in the same field are used to predict the same efficacy prediction data set, and the prediction results are shown in Table 1:

[0132] Table 1 Efficacy Prediction Data

[0133]

[0134] As can be seen from Table 1, the efficacy prediction method proposed by the present invention is significantly superior to other methods in the same field in terms of performance. Among them, the accuracy (ACC) of the method of the present invention reaches 0.75, which means that it can correctly classify in the prediction task and can provide judgment information for diagnosis. In addition, the AUC of the present invention reaches 0.77, which indicates that the method of the present invention has strong discrimination ability when processing data and can still maintain stable performance under different categories. Finally, the F1 value of the present invention is 0.68, which reflects the advantage of the method of the present invention in balancing precision and recall, ensuring that the model can not only accurately identify positive class samples, but also cover all positive class instances as much as possible, thus avoiding missed detections or misjudgments.

[0135] The effects of each method in the present invention are evaluated through ablation experiments, and the results are shown in Table 2:

[0136] Table 2 Ablation Experiment Results

[0137]

[0138] As can be seen from Table 2, both the proposed adjustment of damage weight and the cyclic training model in the present invention can further improve the prediction performance.

[0139] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0140] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning, characterized in that: The following steps are involved: S1. Establish an MSI image dataset and use the MSI image dataset to train a convolutional neural network model A; S2, establish NCRT image dataset, and use convolutional neural network model A to annotate the NCRT image dataset; Construct a convolutional neural network model B including an MSI classification head and an NCRT classification head, use the NCRT image dataset to train the convolutional neural network model B, calculate the classification loss and back-propagate to update the convolutional neural network model B; S3, use the convolutional neural network model B to annotate the MSI image dataset, generate the NCRT pseudo-label of the MSI image dataset and calculate the reliability; S4, use the MSI image data to train the convolutional neural network model B, calculate the classification loss and adjust the loss weight, and back-propagate to update the convolutional neural network model B; S5. Use the convolutional neural network model B to annotate the NCRT image dataset, generate the MSI pseudo-label of the NCRT image dataset and calculate the reliability; S6. Use NCRT image data to train convolutional neural network model B, calculate classification loss and adjust loss weight, and back-propagate to update convolutional neural network model B; S7. Repeat the training of convolutional neural network model B to output the prediction results of the efficacy of neoadjuvant chemoradiotherapy.

2. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S1, the convolutional neural network model A is trained using the MSI image dataset as follows: Randomly extract j groups of data from the MSI image dataset and input them into the convolutional neural network model A. Calculate the overall loss with the output result and the label of the MSI image data, and back-propagate to update the convolutional neural network model A, which can be expressed as: X={x m1 ,x m2 ,…,x mj } And={and m1 ,and m2 ,…,and mj } Where X represents the MSI image dataset, x mj represents the jth data in the MSI image dataset, f1(x mj ) represents x mj The representation obtained after inputting into the middle layer of the convolutional neural network model A, Y represents the label of the MSI image dataset, y mj represents the label of the jth data in the MSI image dataset, L cls represents the overall loss, i represents the total number of image data in the MSI image dataset, and l represents the classification loss function.

3. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S2, the NCRT image dataset is annotated using the convolutional neural network model A as follows: K groups of data are randomly extracted from the NCRT image dataset and input into the convolutional neural network model A. The output of the convolutional neural network model A is used as the initial MSI pseudo label of the NCRT image data, which is expressed as: Z={z n1 ,With n2 ,…,With nk } In={in n1 ,In n2 ,…,In nk } Where Z represents the NCRT image dataset, z nk represents the kth data in the NCRT image dataset, W represents the label set of the NCRT image dataset, and w nk Represents the label of the kth data in the NCRT image dataset, W' (0) represents the initial MSI pseudo-label set of the NCRT image dataset, Represents the initial MSI pseudo label of the k-th data in the NCRT image dataset.

4. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S3, the MSI image dataset is annotated using the convolutional neural network model B as follows: The MSI image dataset is input into the convolutional neural network model B, and the output of the NCRT classification head is used as the initial NCRT pseudo label of the MSI image dataset, which is expressed as: In the formula, Y' (0) represents the initial NCRT pseudo-label set for the MSI image dataset, Represents the initial NCRT pseudo label of the jth data in the MSI image dataset; The reliability of the calculation is: Calculate the confidence of the initial NCRT pseudo-label of the MSI image dataset, perform data enhancement on the MSI image dataset, calculate the consistency of the MSI image dataset before and after enhancement, and calculate the reliability, which is expressed as: In the formula, Represents the prediction confidence of the initial NCRT pseudo label of the jth data in the MSI image dataset, conf j represents the output confidence of the initial NCRT pseudo-label of the jth data in the MSI image dataset by the convolutional neural network model B, Avg(·) represents the averaging operation, Std(·) represents the standard deviation operation, i represents the total number of data in the MSI image dataset, represents the prediction consistency before and after the jth data enhancement in the MSI image dataset, j It represents the output consistency of the convolutional neural network model B before and after the j-th data enhancement in the MSI image dataset. Indicates the reliability of the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

5. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S4, the calculation of classification loss is expressed as: represents the classification loss calculated between the MSI classification head output and the MSI image data label, i represents the total number of data in the MSI image dataset, and x mj represents the jth data in the MSI image dataset, f2(x mj ) represents x mj The representation obtained after inputting into the middle layer of the convolutional neural network model B, y mj represents the label of the jth data in the MSI image dataset, l represents the classification loss function, represents the classification loss calculated between the NCRT classification head output and the initial NCRT pseudo-labels of the MSI image data, represents the initial NCRT pseudo label of the jth data in the MSI image dataset, Indicates the reliability of the initial NCRT pseudo-label of the j-th data in the MSI image dataset.

6. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 5, characterized in that: In step S4, the loss weight is adjusted as follows: Calculate the variance of the prediction results of the convolutional neural network model B for the MSI image data: Where μ2 represents the mean of the prediction results of the convolutional neural network model B for MSI image data, It represents the prediction result of the convolutional neural network model B for the jth data in the MSI image data. Represents the variance of the prediction results of the convolutional neural network model B for MSI image data; according to Adjustment and The weights of , and calculate the overall loss, expressed as: In the formula Indicates the adjusted Indicates the adjusted σ2 is The square root of has no practical meaning. express and of overall losses.

7. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S5, the NCRT image dataset is annotated using the convolutional neural network model B as follows: The NCRT image dataset is input into the convolutional neural network model B, and the output of the MSI classification head is used as the iterative MSI pseudo-label of the NCRT image dataset, which is expressed as: Where W' (1) represents the iterative MSI pseudo-label set of the NCRT image dataset, Represents the iterative MSI pseudo-label of the k-th data in the NCRT image dataset; The reliability of the calculation is: Calculate the confidence of the iterative MSI pseudo-label of the NCRT image dataset, perform data enhancement on the NCRT image dataset, calculate the consistency of the NCRT image dataset before and after enhancement, and calculate the reliability, which is expressed as: In the formula, Represents the prediction confidence of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, conf k represents the output confidence of the iterative MSI pseudo-label of the kth data in the NCRT image dataset by the convolutional neural network model B, Avg(·) represents the averaging operation, Std(·) represents the standard deviation operation, and I represents the total number of data in the NCRT image dataset. represents the prediction consistency before and after the kth data enhancement in the NCRT image dataset, k It represents the output consistency of the convolutional neural network model B before and after the k-th data enhancement in the NCRT image dataset. Indicates the reliability of the iterative MSI pseudo-label of the k-th data in the NCRT image dataset.

8. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S6, the classification loss is calculated as: Where, L MSI represents the classification loss calculated between the MSI classification head output and the iterative MSI pseudo-label of the NCRT image data, I represents the total number of data in the NCRT image dataset, represents the reliability of the iterative MSI pseudo-label of the kth data in the NCRT image dataset, l represents the classification loss function, z nk represents the kth data in the NCRT image dataset, f2(z nk ) represents z nk The representation obtained after inputting into the middle layer of the convolutional neural network model B, represents the iterative MSI pseudo-label of the k-th data in the NCRT image dataset, L NCRT represents the classification loss calculated between the NCRT classification head output and the NCRT image data label, w nk Represents the label of the k-th data in the NCRT image dataset.

9. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 8, characterized in that: In step S6, the adjusted loss weight is: Calculate the variance of the prediction results of the convolutional neural network model B for NCRT image data: In the formula, μ1 represents the mean value of the prediction results of the convolutional neural network model B for NCRT image data, It represents the prediction result of the convolutional neural network model B for the kth data in the NCRT image data. Represents the variance of the prediction results of the convolutional neural network model B for NCRT image data; according to Adjust L MSI With L NCRT The weights of , and calculate the overall loss, expressed as: L all =L′ NCR T+L′ MSI Where L' MSI Represents the adjusted L MSI , L' NCRT Represents the adjusted L NCRT ,σ1 is The square root of L has no practical meaning. all Indicates L' MSI With L' NCRT of overall losses.

10. The method for predicting the efficacy of neoadjuvant chemoradiotherapy based on multi-task learning according to claim 1, characterized in that: In step S7, the convolutional neural network model B is repeatedly trained until the loss function value converges or reaches a preset number of iterations, and the output of the NCRT classification head is used as the prediction result of the efficacy of neoadjuvant chemoradiotherapy.

Images