A tumor progression assessment method based on two-stage multi-task learning

Abstract

The present invention relates to a tumor progression assessment method based on two-stage multi-task learning. The method comprises: CT image acquisition; coarse segmentation prediction; and establishment of a two-stage multi-task learning framework. The method integrates the correlation between longitudinal images and clinical prior information to generate a feature map. Based on the multi-task learning framework, a disease progression classification sub-model based on total tumor volume and a progression classification sub-model based on new lesions are constructed to respectively assess the progression of total tumor volume changes and new lesions. The disease progression classification sub-model based on total tumor volume adopts a two-branch structure of segmentation and classification. In the classification branch, progression judgment labels are continuously and orderly encoded to optimize the coarse segmentation results of forward registration and return them through inverse transformation. The results of the total tumor volume change progression assessment and the new lesion assessment are integrated to obtain a tumor progression assessment result. Compared with the existing technology, the present invention has the advantage of high tumor progression assessment accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of tumor image analysis, and in particular to a tumor progression assessment method based on two-stage multi-task learning. Background Art

[0002] Objective response rate (ORR) and progression-free survival (PFS) are the most common primary endpoints in Phase I / II / III clinical trials. These indicators rely on accurate assessment of drug efficacy. Radiological follow-up in oncology is indispensable for evaluating treatment response. In clinical practice, oncologists must accurately identify lesions in serial CT sections according to the Response Evaluation Criteria in Solid Tumors (RECIST), assess drug efficacy based on changes in tumor burden, and make optimal treatment decisions.

[0003] Different readers often have inconsistent judgments about tumor size and location in images. This inconsistency is more pronounced in imaging assessments of the efficacy of tumor immunotherapy, as the definition of imaging data is more difficult. Inaccurate efficacy assessments may lead to incorrect medical decisions, thereby affecting treatment outcomes and shortening patient survival. In addition, observing and identifying lesions from CT images and determining progression through comparative assessment of patient longitudinal data is often time-consuming and labor-intensive for clinicians.

[0004] Currently available segmentation models can achieve a certain degree of accurate lesion identification. However, for liver tumors, due to the complex abdominal anatomical structure, low image contrast between organs, unclear boundaries of primary liver tumors, and the heterogeneity of liver tumor subtypes (nodular, massive, and diffuse liver cancers have different radiological characteristics), it is more difficult to automatically identify and segment tumors from CT images compared to other tumors. Even the best segmentation models currently available make completely different predictions for the same lesion in longitudinal scans, and this phenomenon is more pronounced in ambiguous lesions. Specifically, inaccurate grasp of image acquisition time or unclear boundaries of lesions can cause the model to incorrectly evaluate a lesion that actually has the same trend in longitudinal images at different follow-ups, resulting in overestimation or underestimation of tumor burden and incorrect drug efficacy evaluation results.

[0005] In addition, there is currently no evaluation method for judging the progress of drug efficacy that fully complies with the clinical solid tumor efficacy evaluation standards and takes into account changes in total tumor burden and new lesions. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide a tumor progression assessment method based on two-stage multi-task learning with high tumor progression assessment accuracy. The method focuses on the lesion differences between longitudinal CT sequence scans to capture the image lesion correlation, that is, the change pattern or growth trend of the lesion, and proposes a multi-task learning framework that integrates medical prior information. It can optimize the total tumor load progression assessment results after coarse segmentation of the existing segmentation model, and combine the graph matching strategy to give new lesion judgments, and finally obtain the disease progression judgment that complies with the clinical solid tumor efficacy evaluation standards, thereby improving the accuracy of drug efficacy in evaluating tumor progression.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, a tumor progression assessment method based on two-stage multi-task learning is provided, the method comprising:

[0009] Step S1, CT image acquisition: acquiring a venous phase CT image of a tumor patient;

[0010] Step S2, coarse segmentation prediction: input the CT image into the lesion segmentation model to obtain the coarse segmentation lesion result;

[0011] Step S3, multi-task learning framework construction: Integrate the correlation between longitudinal images and clinical prior information to generate a feature map. Based on the multi-task learning framework, construct a disease progression classification sub-model based on total tumor volume and a disease progression classification sub-model based on new lesions;

[0012] Step S4, total tumor volume change progression assessment: The total tumor volume disease progression classification sub-model adopts a two-branch structure of segmentation and classification. In the classification branch, the progression judgment labels are encoded in a continuous and orderly manner to optimize the coarse segmentation results of forward registration and return them through inverse transformation;

[0013] Step S5, new lesion assessment: using a new lesion-based progression classification sub-model, constructing a longitudinal lesion association map based on the feature map to assess new lesions;

[0014] Step S6, tumor progression assessment: integrating the tumor total volume change progress assessment results and the new lesion assessment results to obtain the tumor progression assessment results.

[0015] Preferably, the step S1 further includes preprocessing the CT image, specifically:

[0016] Format conversion: convert DICOM format venous phase CT images into NII format;

[0017] Resampling: resampling the venous phase CT images at set slice intervals;

[0018] Grayscale normalization;

[0019] Irrelevant background removal: The segmentation mask is obtained using a pre-trained segmentation tool, and the pixels of the venous phase CT image are adjusted by linear interpolation;

[0020] Image augmentation: Random flipping and random rotation are performed with set probabilities, and the corresponding labels are also synchronously flipped and rotated.

[0021] Preferably, the lesion segmentation model is a lesion segmentation model based on convolutional neural network and transformation neural network.

[0022] Preferably, the convolutional neural network is nnU-Net, and the transformation neural network is Swin-Unetr.

[0023] Preferably, step S3 includes:

[0024] Perform deformable registration of baseline and follow-up images, and generate difference maps based on the coarse segmentation results.

[0025] The baseline lesion prediction image, difference map, and registered follow-up lesion prediction image are spliced ​​in the channel dimension and input into the compression and excitation network Se-Unet. A mapping from the continuous and ordered sum of tumor volumes to the three-category results of disease progression judgment is designed, and multi-task learning with dual-task branches of classification and segmentation is used to optimize the coarse segmentation results.

[0026] Preferably, the difference map is generated by performing a three-dimensional connected domain analysis after subtracting the aligned follow-up lesion prediction image from the baseline lesion prediction image, removing lesions with drastic changes in standard deviation during longitudinal follow-up, and using a liver mask to remove extrahepatic lesions to obtain a difference map.

[0027] Preferably, the loss function of the compression and excitation network Se-Unet is the sum of the segmentation loss and the weighted classification loss; wherein the segmentation loss includes the baseline segmentation loss L seg_baseline and the follow-up segmentation loss L seg_followup ; The classification loss is a three-category loss for progress judgment, and adopts the form of continuous cross entropy loss.

[0028] Preferably, the compression and excitation network Se-Unet comprises three encoding and decoding blocks and four bottleneck blocks.

[0029] Preferably, in step S5, a progression classification sub-model based on new lesions is used to construct a longitudinal lesion association map based on the characteristic map to evaluate new lesions, specifically:

[0030] The ratio of the Euclidean distance between the lesion center points in the baseline lesion prediction image and the registered follow-up tumor prediction image to the sum of the two lesion radii was calculated. If the ratio was greater than 1, the lesion was considered to be a new lesion. Otherwise, the lesion in the registered follow-up tumor prediction image was considered to be associated with the corresponding lesion in the baseline lesion prediction image.

[0031] Preferably, step S6 is specifically as follows: if step S5 determines that there are new lesions, the tumor progression assessment result is disease progression; if there are no new lesions, the tumor progression assessment result is consistent with the determination result of step S4.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1) Compared with the existing deep learning segmentation model, which does not consider the correlation between image samples and medical prior information, it is easy to produce inconsistent predictions between different longitudinal time nodes of the same lesion, leading to misjudgment of progression. The present invention integrates a two-stage multi-task learning framework on the tumor coarse segmentation results, uses the correlation information between the longitudinal data of progression judgment and clinical prior information to generate feature maps, and continuously and orderly encodes the progression judgment labels, which can reduce the problems of inconsistent and inaccurate predictions and improve the accuracy of drug efficacy progression judgment.

[0034] 2) The present invention obtains coarse tumor segmentation results using the convolutional neural network nnU-Net and the transformation neural network Swin-Unetr, with higher accuracy.

[0035] 3) The present invention follows the clinical efficacy evaluation standards for solid tumors. The drug efficacy progression assessment results take into account both the change in total tumor volume and the judgment of new lesions. Compared with traditional evaluation methods that only consider total volume changes or only evaluate new lesions, the present invention provides a more standardized judgment of progression assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the RECORD model architecture for tumor progression assessment based on two-stage multi-task learning in the embodiment, where Part A is a schematic diagram of tumor coarse segmentation based on CT images, and Part B is a schematic diagram of the RECORD model principle. It uses the coarse segmentation result as input, the upper half is the tumor total volume change trend progression judgment prediction model, and the lower half is the new lesion judgment model. After integrating the two branches, the total progression judgment result is obtained.

[0037] Figure 2 Schematic diagram of image preprocessing in the embodiment; wherein, method 1 and method 2 are two different existing liver mask segmentation models.

[0038] Figure 3 Schematic diagram of the specific process of generating the RECORD model difference map in the embodiment.

[0039] Figure 4 Schematic diagram of the data screening process in the embodiment.

[0040] Figure 5 : This is a comparison chart of the AUC results of RECORD and the existing model in the embodiment, wherein part A is the AUC comparison under disease remission, part B is the AUC comparison under disease stability, and part C is the AUC comparison under disease progression; the P value is calculated using the Delong test method.

[0041] Figure 6 This is a comparison chart of the confusion matrices of RECORD and existing models in the embodiment, where parts A and B respectively compare the improvement of the effect of RECORD on the two existing models.

[0042] Figure 7 The figure is a comparison chart of the accuracy of RECORD and the existing model in the embodiment, and the P value is calculated using the Mc Nemar mid-p test method. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0044] Example

[0045] This embodiment relates to a method for assessing tumor progression based on venous CT images. The method aims to provide a fully automated two-stage deep learning convolutional neural network, employing a multi-task learning framework and optimization strategies related to progression assessment. Following clinical solid tumor efficacy evaluation criteria, the method directly assesses drug efficacy in patients from tumor CT images (RECORD model). The method comprises the following steps:

[0046] Step S1, CT image acquisition: acquiring a venous phase CT image of a tumor patient;

[0047] Step S2, coarse segmentation prediction: input the CT image into the lesion segmentation model to obtain the coarse segmentation lesion result;

[0048] Step S3, multi-task learning framework construction: Integrate the correlation between longitudinal images and clinical prior information to generate a feature map. Based on the multi-task learning framework, construct a disease progression classification sub-model based on total tumor volume and a disease progression classification sub-model based on new lesions;

[0049] Step S4, total tumor volume change progression assessment: The total tumor volume disease progression classification sub-model adopts a two-branch structure of segmentation and classification. In the classification branch, the progression judgment labels are encoded in a continuous and orderly manner to optimize the coarse segmentation results of forward registration and return them through inverse transformation;

[0050] Step S5, new lesion assessment: using a new lesion-based progression classification sub-model, constructing a longitudinal lesion association map based on the feature map to assess new lesions;

[0051] Step S6, tumor progression assessment: integrating the tumor total volume change progress assessment results and the new lesion assessment results to obtain the tumor progression assessment results.

[0052] Next, the method of this embodiment is described in detail using a liver tumor dataset as an example.

[0053] 1. CT image acquisition

[0054] Venous phase CT images of patients with liver tumors were collected and processed by professional doctors to annotate the tumor's 3D contours. The venous phase CT images in this example included patients with advanced, unresectable hepatocellular carcinoma undergoing a specific immunotherapy (Cohort A), patients with metastatic liver cancer (Cohort B), and patients with liver cancer undergoing combination therapy (Cohort C).

[0055] 2. CT image preprocessing

[0056] 1) Format conversion: Convert venous phase CT images from the clinically acquired DICOM sequence format to the NII format, which is more convenient for storage and reading, with a slice interval of less than or equal to 5 mm.

[0057] 2) Resampling: The venous phase images of liver cancer patients were resampled from 5mm to 2mm slice interval. Specifically, the XYZ resolution of CT images of different patients is different. In order to directly reflect the tumor volume with the number of pixels, in this example, all patient images were linearly interpolated to an isotropic 2×2×2mm. 3 .

[0058] 3) Grayscale normalization: adjust the window width and window position of the original CT image to remove noise points, and the normalized image value ranges from -200 to 200.

[0059] 4) Irrelevant background removal: To reduce the computational complexity of the subsequent 3D segmentation model, only the liver surrounding slices are retained ( Figure 1 ). Using existing liver mask prediction tools ( https: / / github.com / andreped / livermask and https: / / zenodo.o rg / record / 3734294#.ZAGgnHZBw2z) and the predicted masks were unioned. Based on the liver mask of the union, the upper and lower bounds were expanded outward by 4 slices to ensure image integrity. The image was resized from the original 512×512 pixels to 205×205 pixels through linear interpolation.

[0060] 5) Data augmentation: The CT images are randomly flipped and rotated with a probability of 0.5, and the corresponding labels are also flipped and rotated synchronously.

[0061] 3. Rough segmentation of tumor area

[0062] The pre-processed CT image is input into the lesion segmentation model to obtain the coarse segmentation lesion result ( Figure 2 Part A); The lesion segmentation model in this embodiment adopts a lesion segmentation model based on convolutional neural network and transformation neural network, wherein; the convolutional neural network is preferably nnU-Net, and the transformation neural network is preferably Swin-Unetr.

[0063] Due to the high computational complexity of 3D image processing, actual model optimization requires selecting patches for model training. nnU-Net uses the dataset fingerprint to select the optimal patch size of 112×160×128, while Swin-Unet uses a patch size of 96×96×96. Both models use 1000 training rounds.

[0064] 4. Design of continuous and orderly coding of progress judgment labels

[0065] The disease process is a continuous process. Discrete encoding of it will result in information loss. This embodiment designs a continuous ordered label encoding method targeting the continuity of the disease process. The sum of the baseline and follow-up tumor volumes is mapped to the three-category probability of disease progression, as shown in Equation (1).

[0066]

[0067]

[0068] in,

[0069] 1) w can be written as formula (2). To comply with the comprehensive consideration of the absolute and relative volume change rates of RECIST (Responsiveness Evaluation Criteria in Solid Tumors), w approaches 0 only when the baseline and follow-up lesions are very small, and approaches 1 in all other scenarios.

[0070] 2) V1 is the sum of baseline lesion volumes, and V2 is the sum of follow-up lesion volumes;

[0071] 3) C iis a series of set hyperparameters, where C1=15, C2=5, C3=0.1;

[0072] 4)r PR , r PD The cutoff between disease remission and stable disease was obtained by extrapolation from the RECIST unidimensional criteria and matched according to the actual clinical trial standards. PR =0.65, the boundary between disease progression and stable disease is r PD =1.44;

[0073] 5) σ is the Sigmoid function, mapping [-∞, +∞] to [0, 1];

[0074] 6) V is the minimum volume change considered disease progression, which is set to V = 400 and is obtained by extrapolation from the RECIST one-dimensional criteria.

[0075] 5. RECORD model architecture

[0076] Figure 2 The RECORD model architecture allows accepting the tumor region prediction results of any existing segmentation model as input and performing optimization under the multi-task learning framework. It can be divided into Figure 2 The disease progression classification sub-model based on total tumor volume in the upper middle branch and the progression classification sub-model based on new lesions in the lower middle branch are integrated to obtain the overall progression judgment result.

[0077] The upper branch of the disease progression classification sub-model based on total tumor volume specifically includes:

[0078] 1) “Baseline-Follow-up” Image Registration

[0079] We use the advanced normalization tool ANTs to generate forward and inverse image deformation fields. The original ANTs was implemented in C++. To ensure gradient differentiability within a deep learning framework, we reimplemented the deformation fields generated by ANTs using the MONAI framework.

[0080] 2) Generation of difference maps guided by longitudinal correlation information and medical prior information under weak supervision ( Figure 3 )

[0081] After subtracting the baseline and registered follow-up lesions, the difference information at the single lesion level was obtained through 3D connected domain analysis, and the number of pixels in the difference part was counted. If the difference was greater than 25,000 pixels (i.e., 200 cm 3) then the lesion is classified as a suspected false positive lesion. The longitudinal standard deviation of the suspected false positive lesions is calculated. If the standard deviation is large, it indicates that the lesion has changed dramatically in a short period of time and does not conform to conventional medical laws. Using the weak supervision information of the longitudinal image correlation, the lesions with a standard deviation greater than 4000 pixels (i.e., 8cm 3 ) lesions and identified them as having a high probability of false positive lesions, which were removed from the difference map. Finally, the liver mask was applied to the processed feature map, and the prediction of extrahepatic false positive lesions was removed to obtain the final "baseline-follow-up" difference map.

[0082] 3) Optimization of coarse segmentation results

[0083] The "baseline lesion prediction-difference map-follow-up lesion prediction after registration" is spliced ​​in the channel dimension and input into the compression and excitation network Se-Unet consisting of three encoding and decoding blocks and four bottleneck blocks. A mapping from the continuous and ordered sum of tumor volumes to the three-category results of disease progression judgment is designed, and multi-task learning with dual-task branches of classification and segmentation is used to optimize the coarse segmentation results.

[0084] By using multi-task learning, the classification head optimizes the three-category prediction accuracy of progression judgment, and the segmentation head optimizes the lesion recognition accuracy, which can effectively improve the accuracy of progression judgment.

[0085] The loss function of the compression and excitation network Se-Unet is the sum of the segmentation loss and the weighted classification loss; among them, the segmentation loss includes the baseline segmentation loss L seg_baseline and the follow-up segmentation loss L seg_followup ; The classification loss is a three-category loss for progress judgment, and adopts the form of continuous cross entropy loss.

[0086] The loss function of the compression and excitation network Se-Unet in this embodiment can be written as follows:

[0087] L=L seg +w·L cls ,w=0.55 (3)

[0088] L seg =L seg_baseline +L seg_followup (4)

[0089] L seg_i =L DICE +L CE (5)

[0090]

[0091] Among them, L seg is the segmentation loss, which is DICE(L DICE ) and cross entropy (LCE ), weighted by the baseline segmentation loss L seg_baseline and the follow-up segmentation loss L seg_followup Two items: L cls In order to make progress in judging the three-category loss, the continuous cross entropy loss is adopted, y and All are continuous three-class probability under the continuous ordered coding design of progress judgment labels, y is the true value, is the predicted value.

[0092] The progression classification sub-model based on new lesions in the lower half is divided into the following steps:

[0093] 1) Three-dimensional connected domain analysis was performed on baseline and follow-up lesions to obtain results at the single lesion level.

[0094] 2) Constructing the longitudinal lesion map network G = (L B , L F , E). For all single lesion nodes at baseline, To follow up all single lesion nodes. If the calculated relative distance of the lesion is If it is less than 1, add a and All failed follow-up nodes are marked as new lesions. ij is the Euclidean distance between the center points of the baseline i lesion and the follow-up j lesion, is the radius of the baseline i lesion, is the radius of the lesion at follow-up j.

[0095] 3) After obtaining the change trend of the total tumor volume and the judgment results of new lesions, the final drug efficacy judgment results are obtained according to the solid tumor efficacy evaluation criteria.

[0096] 5. RECORD model training

[0097] The training data was divided into five folds based on the patient level, and the model was cross-trained and validated using the five-fold cross-training. To address the computational complexity of 3D images, data parallelism in distributed computing was used to divide the image into several patches, distribute them to each computing node, and then restore them to their full resolution.

[0098] The training was performed using the PyTorch (torch==1.11.0, torchvision==0.12.0) framework on the Siyuan high-performance supercomputer at Shanghai Jiao Tong University, using 4×NVIDIA A100s with 40GB of video memory per GPU. https: / / github.com / Proiect-MONAI / ) to implement the model.

[0099] Model training and verification uses subset A1 in queue A, and subset A2, queue B, and queue C are used to verify model generalization.

[0100] 6. Model Evaluation

[0101] The AUC and F1 score on the validation set were used to evaluate the accuracy of predicting total tumor volume trends. Accuracy, precision, and recall were used to evaluate the results of identifying new lesions. Accuracy was used to assess the accuracy of overall drug efficacy assessment. AUCs were compared using the Delong test, and accuracy was compared using the McNemar mid-p test to correct for cases where diagonal elements significantly outnumbered off-diagonal elements.

[0102] To verify the accuracy of the present invention, the example verification content of this embodiment is as follows:

[0103] 1) Dataset

[0104] The data came from 891 venous phase CT scan images of 206 patients with liver cancer between 2017 and 2022.

[0105] Inclusion criteria were: (1) presence of at least one RECIST measurable lesion; and (2) at least one follow-up visit.

[0106] Exclusion criteria were: (1) no venous phase abdominal CT images; (2) the interval between two consecutive scans was greater than 18 weeks; and (3) low image quality (snowflake artifacts appeared on the CT scan).

[0107] The screening process is as follows Figure 4 Cohort A consists of patients with hepatocellular carcinoma receiving immunotherapy, and is divided into the following groups according to the center: Cohort A1 is used for five-fold cross-training validation of the model, and Cohort A2 is used for external generalization validation; Cohort B consists of patients with metastatic liver cancer receiving immunotherapy, and Cohort C consists of patients with liver cancer receiving combination therapy.

[0108] Table 1 gives the detailed statistical description of the dataset.

[0109] Table 1 Statistical description of the data set

[0110]

[0111]

[0112] 2) The RECORD model is applied to the existing segmentation model based on the total tumor volume and the effect of improving

[0113] Internal five-fold cross-validation: nnU-Net and Swin-Unetr models were trained on cohort A1 to obtain coarse segmentation results. The coarse segmentation tumor pixels were summed to obtain the total tumor volume (SOV) to determine the coarse segmentation progression. This coarse segmentation progression judgment was compared with the progression judgment obtained through the RECORD two-stage multi-task learning. The results were compared using the area under the curve (AUC) for disease remission, stable disease, and progression, as well as the macro F1 score, micro F1 score, and weighted F1 score.

[0114] At the level of AUC of three-class progress, the RECORD proposed in this paper improves the nnU-Net coarse segmentation results from AUC PR-nnUnet =0.961 improved to AUC PR-RECORD =0.976, AUC SD-nnUnet =0.859 improved to AUC SD-RECORD =0.904, AUC PD-nnU-Net =0.917 improved to AUC PD-RECORD =0.933. At the level of Swin-Unetr coarse segmentation results, RECORD also has a similar improvement trend. Figure 5 The two curves of medium depth and shallow depth, and the two AUC curves were compared using the Delong test. It was found that RECORD has a significant improvement in AUC compared to the current SOTA segmentation model.

[0115] In terms of F1 scores, Table 2 (first row, cohort A1) compares the SOTA model and the RECORD model. RECORD improves on the macro F1 score, micro F1 score, and weighted F1 score compared to the SOTA model. In terms of the accuracy of progression assessment based on total tumor volume and , the nnU-Net-based RECORD optimization improves by 3.0%, and the Swin-Unetr-based RECORD optimization improves by 3.2% ( Figure 7 ), according to the McNemar mid-p test, both improvements were significant. In the three-category judgment, the optimization of RECORD significantly reduced the confusion between stable disease and remission and progression ( Figure 6 ).

[0116] Table 2 Comparison of F1 scores between SOTA model and RECORD model

[0117]

[0118]

[0119] External Validation:

[0120] Table 3 AUC indicator statistics in the external validation set

[0121]

[0122]

[0123] Table 3 shows the AUC comparison results between the state-of-the-art model and RECORD on the external validation set, namely cohorts A2, B, and C. Prediction and optimization were performed on the five-fold model trained on cohort A1. The results are presented as the five-fold average and its 95% confidence interval. Table 2 shows the F1 score metric on the external validation set. This comparison reveals that RECORD still performs better than the state-of-the-art model on this set, demonstrating its good generalization performance.

[0124] 3) Improved effectiveness of the RECORD model in assessing new lesions using existing segmentation models

[0125] A longitudinal lesion map matching method was used to identify new lesions. Comparative assessment of new lesions was conducted in cohorts A1, A2, and B. In cohort C, most patients had liver damage exceeding 50%, making new lesions difficult to define in this cohort, so this cohort was excluded from the new lesion assessment. RECORD superimposed a difference map on the coarse segmentation results of the SOTA model to remove false-positive lesions outside the liver and unreasonable lesions. The difference map can convert some false positives into true negatives. Specifically, the original SOTA model achieved high recall but relatively low precision, indicating that it is prone to false positive predictions outside the liver. Incorporating the difference map with anatomical location information can improve precision while maintaining recall (Table 4). For example, after adding the difference map, the nnU-Net prediction accuracy for new lesions in cohort A1 increased from 0.730 to 0.880, while the recall remained at 0.844. The accuracy increased from 0.904 to 0.945. A similar improvement was observed in the SOTA model predictions for cohort A2. Cohort B presented tumors with clear boundaries and high contrast. Therefore, RECORD is non-inferior to the SOTA segmentation model, and in most cohorts with unclear lesion boundaries, it significantly improves the effect of the SOTA segmentation model.

[0126] Table 4 Results of new lesions

[0127]

[0128] 4) The RECORD model is applied to the overall progress evaluation of existing segmentation models to improve the effect

[0129] By integrating the judgment results based on total tumor volume and new lesions, according to the corresponding criteria in Table 5 below and following the RECIST standard for evaluating the efficacy of solid tumors, an overall drug efficacy evaluation result can be given.

[0130] Table 5 Corresponding criteria for total tumor volume and new lesions to obtain overall progression assessment

[0131]

[0132]

[0133] Table 6 compares the performance of the optimized RECORD model with the SOTA model in overall drug efficacy evaluation. Cohorts A1, A2, and B all use a combined evaluation of total tumor volume and new lesions. Due to the difficulty identifying new lesions in cohort C, the change in total tumor volume was used as the overall drug efficacy evaluation result. For nnU-Net, applying RECORD optimization improved the overall drug efficacy evaluation accuracy by an average of 3.3±2.0%. For Swin-Unetr, applying RECORD optimization improved the overall drug efficacy evaluation accuracy by an average of 3.0±2.1%. This demonstrates that by integrating longitudinal image correlations and medical prior information within a multi-task learning framework, more accurate progression assessment can be performed on cohort images with unclear or difficult-to-read tumor boundaries.

[0134] Table 6 Overall drug efficacy evaluation accuracy results

[0135]

[0136] In summary, the present invention integrates a multi-task learning module based on the automatic tumor segmentation from CT images using convolutional neural networks and transformation neural networks, effectively utilizes the correlation and medical prior knowledge in the longitudinal data of progression judgment, and can significantly improve the assessment accuracy of the three types of progression judgment results: disease remission (Response), disease stability (Stable Disease), and disease progression (Progressive Disease).

[0137] The foregoing description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art will readily conceive of various equivalent modifications or substitutions within the technical scope disclosed herein, and such modifications or substitutions are intended to be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A tumor progression assessment method based on two-stage multi-task learning, characterized in that: The method includes: Step S1, CT image acquisition: acquiring a venous phase CT image of a tumor patient; Step S2, coarse segmentation prediction: input the CT image into the lesion segmentation model to obtain the coarse segmentation lesion result; Step S3, multi-task learning framework construction: Integrate the correlation between longitudinal images and clinical prior information to generate a feature map. Based on the multi-task learning framework, construct a disease progression classification sub-model based on total tumor volume and a disease progression classification sub-model based on new lesions; Step S4, total tumor volume change progression assessment: The total tumor volume disease progression classification sub-model adopts a two-branch structure of segmentation and classification. In the classification branch, the progression judgment labels are encoded in a continuous and orderly manner to optimize the coarse segmentation results of forward registration and return them through inverse transformation; Step S5, new lesion assessment: using a new lesion-based progression classification sub-model, constructing a longitudinal lesion association map based on the feature map to assess new lesions; Step S6, tumor progression assessment: integrating the tumor total volume change progress assessment results and the new lesion assessment results to obtain the tumor progression assessment results.

2. A tumor progression assessment method based on two-stage multi-task learning according to claim 1, characterized in that: The step S1 also includes preprocessing the CT image, specifically: Format conversion: convert DICOM format venous phase CT images into NII format; Resampling: resampling the venous phase CT images at set slice intervals; Grayscale normalization; Irrelevant background removal: The segmentation mask is obtained using a pre-trained segmentation tool, and the pixels of the venous phase CT image are adjusted by linear interpolation; Image augmentation: Random flipping and random rotation are performed with set probabilities, and the corresponding labels are also synchronously flipped and rotated.

3. The tumor progression assessment method based on two-stage multi-task learning according to claim 1, characterized in that: The lesion segmentation model is a lesion segmentation model based on convolutional neural network and transformation neural network.

4. A tumor progression assessment method based on two-stage multi-task learning according to claim 3, characterized in that: The convolutional neural network is nnU-Net, and the transformation neural network is Swin-Unetr.

5. The tumor progression assessment method based on two-stage multi-task learning according to claim 1, characterized in that: The step S3 comprises: Perform deformable registration of baseline and follow-up images, and generate difference maps based on the coarse segmentation results. The baseline lesion prediction image, difference map, and registered follow-up lesion prediction image are spliced ​​in the channel dimension and input into the compression and excitation network Se-Unet. A mapping from the continuous and ordered sum of tumor volumes to the three-category results of disease progression judgment is designed, and multi-task learning with dual-task branches of classification and segmentation is used to optimize the coarse segmentation results.

6. A tumor progression assessment method based on two-stage multi-task learning according to claim 5, characterized in that: The difference map is generated specifically by performing a three-dimensional connected domain analysis on the difference between the registered follow-up lesion prediction image and the baseline lesion prediction image, removing lesions with drastic changes in standard deviation during longitudinal follow-up, and removing extrahepatic lesions using a liver mask to obtain a difference map.

7. The tumor progression assessment method based on two-stage multi-task learning according to claim 5, characterized in that: The loss function of the compression and excitation network Se-Unet is the sum of the segmentation loss and the weighted classification loss; wherein the segmentation loss includes the baseline segmentation loss L seg_baseline and the follow-up segmentation loss L seg_followup ; The classification loss is a three-category loss for progress judgment, and adopts the form of continuous cross entropy loss.

8. The tumor progression assessment method based on two-stage multi-task learning according to claim 5, characterized in that: The compression and excitation network Se-Unet includes three encoding and decoding blocks and four bottleneck blocks.

9. The tumor progression assessment method based on two-stage multi-task learning according to claim 5, characterized in that: In step S5, a progression classification sub-model based on new lesions is used to construct a longitudinal lesion association map based on the characteristic map to evaluate new lesions, specifically: The ratio of the Euclidean distance between the lesion center points in the baseline lesion prediction image and the registered follow-up tumor prediction image to the sum of the two lesion radii was calculated. If the ratio was greater than 1, the lesion was considered to be a new lesion. Otherwise, the lesion in the registered follow-up tumor prediction image was considered to be associated with the corresponding lesion in the baseline lesion prediction image.

10. The tumor progression assessment method based on two-stage multi-task learning according to claim 1, characterized in that: The step S6 is specifically as follows: if the step S5 determines that there is a new lesion, the tumor progression assessment result is disease progression; if there is no new lesion, the tumor progression assessment result is consistent with the determination result of the step S4.

Images