Prognosis prediction method for non-small cell lung cancer based on multimodal fusion

Abstract

The present disclosure provides a method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion, including obtaining a pre-treatment CT image and a radiotherapy three-dimensional dose distribution image of a target patient; obtaining a region of interest based on the pre-treatment CT image; obtaining a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest; obtaining a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest; and realizing the prediction of the patient's PFS based on the multi-omics prediction label, the deep survival label and the prognosis-related clinical features. The non-small cell lung cancer prognosis prediction method based on multimodal fusion disclosed in the present disclosure, combined with multimodal data such as CT images and dose distribution images, can fully combine the multimodal complementary feature information contained in the primary tumor and the affected lymph nodes, and can process the truncated data in the prognosis label, thereby realizing the accurate prediction of the patient's progression-free survival after radiotherapy.

Description

Technical Field

[0001] The present disclosure relates to the field of prognosis prediction models, and in particular to a prognosis prediction method for non-small cell lung cancer based on multimodal fusion. Background Art

[0002] Non-small cell lung cancer (NSCLC) is a common type of lung cancer, accounting for about 85% of lung cancers; about 20%-25% of patients are diagnosed with locally advanced non-small cell lung cancer. For inoperable locally advanced non-small cell lung cancer, synchronous chemoradiotherapy is the current standard treatment. However, due to the heterogeneity of tumors, there are still certain differences in the response of patients in the same stage after standard treatment. Therefore, accurately predicting the prognosis of patients with locally advanced non-small cell lung cancer at the individual level is of great significance for clinical precision treatment.

[0003] For patients with locally advanced non-small cell lung cancer who have undergone radiotherapy, in addition to the inherent anatomical information displayed on computed tomography (CT), the three-dimensional dose distribution of radiotherapy also has an important impact on the patient's prognosis. Most existing studies only focus on the extraction and mining of features in CT single-modality images, ignoring the rich information contained in the three-dimensional dose distribution, which limits the improvement of the effectiveness of the prognosis prediction model. Therefore, developing an effective multimodal fusion method to fully mine and integrate the complementary representations in CT images and dose distribution maps is crucial to achieve accurate prognosis prediction. Summary of the invention

[0004] The technical problem to be solved by the present disclosure is to overcome the defects in the prior art and provide a multimodal prognosis prediction method for NSCLC that integrates the information of primary tumor and involved lymph nodes.

[0005] The present disclosure solves the above technical problems through the following technical solutions: a method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion, comprising the following steps:

[0006] Obtain the target patient's pre-treatment CT image and radiotherapy three-dimensional dose distribution image;

[0007] Acquiring a region of interest according to the pre-treatment CT image;

[0008] Obtaining a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image, and the region of interest;

[0009] Obtaining a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image, and the region of interest;

[0010] Based on the multi-omics prediction signature, deep survival signature and prognosis-related clinical characteristics, the patient's progression-free survival (PFS) is predicted.

[0011] Preferably, before the step of obtaining a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest, the step further includes:

[0012] Preprocessing the pre-treatment CT image and the radiotherapy three-dimensional dose distribution image, wherein the preprocessing includes matching the pre-treatment CT image and the radiotherapy three-dimensional dose distribution image and interpolating them to the same resolution;

[0013] The region of interest includes the primary tumor region (ROI T ) and affected lymph node region (ROI N ).

[0014] Preferably, the multi-omics prediction label is obtained based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest; including:

[0015] Based on the pre-processed CT image and the three-dimensional dose distribution image before treatment, respectively extracting high-dimensional radiomics features and dosiomics features in the region of interest;

[0016] Screening a subset of multi-omics features with prognostic prediction potential and low redundancy from the high-dimensional imaging omics features and the dosimetry features;

[0017] Based on the multi-omics feature subset, a pre-constructed first model is input, and the prediction probability output by the first model is used as the multi-omics prediction label;

[0018] The first model is a multi-factor Cox regression model, which is used to output corresponding PFS predictions based on the input multi-omics feature subsets.

[0019] Preferably, the deep survival label is obtained based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest, including:

[0020] According to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region, a mask image corresponding to the lymph node region, and a spatial distance map between the lymph node and the tumor are obtained;

[0021] Obtaining a first channel input spliced ​​image based on the pre-processed CT image before treatment, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor;

[0022] Obtaining a second channel input spliced ​​image based on the preprocessed radiotherapy three-dimensional dose distribution image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor;

[0023] Input the first channel stitched image and the second channel stitched image into a pre-built second model, and use the predicted probability output by the second model as a deep survival prediction label;

[0024] The pre-built second model includes two single-modal branch networks, which are a three-dimensional densely connected network obtained by end-to-end training using the first channel stitched image and the second channel stitched image obtained by training samples as inputs of the two branch networks and the PFS label as supervision information.

[0025] Preferably, the step of training the three-dimensional densely connected network comprises:

[0026] Taking the first channel stitched image and the second channel stitched image as input, respectively, two single-modal branch networks are trained separately;

[0027] Using a gating mechanism to integrate the feature representations extracted by the unimodal network branch;

[0028] Features are further extracted through the cascaded fully connected layers to output the predicted probability.

[0029] Preferably, the prediction of patient PFS based on the multi-omics prediction signature, deep survival signature and prognosis-related clinical features includes:

[0030] The multi-omics prediction label, the deep survival label, and the prognosis-related clinical features are input into a pre-built third model, and the final prediction probability is obtained as output;

[0031] The third model is a multi-factor Cox regression model with elastic network constraints (Elastic-Cox), in which LASSO and Ridge penalty terms are introduced into the Cox regression.

[0032] Preferably, according to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region, a mask image corresponding to the lymph node region and a spatial distance map between the lymph node and the tumor are obtained, including:

[0033] According to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region is obtained. T) and the mask image corresponding to the lymph node area (MASK N );

[0034] The pixel value of each pixel in the mask image corresponding to the lymph node region is converted into the shortest distance between each voxel of the lymph node and the tumor, so as to obtain a spatial distance map (NT map) between the lymph node and the tumor.

[0035] Another aspect of the present disclosure provides a non-small cell lung cancer prognosis prediction device based on multimodal fusion, comprising:

[0036] A first acquisition module is used to acquire a pre-treatment CT image and a radiotherapy three-dimensional dose distribution image of a target patient;

[0037] A second acquisition module, used for acquiring a region of interest according to the pre-treatment CT image;

[0038] A multi-omics prediction label module, used to obtain a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest;

[0039] A deep survival label module, used for obtaining a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest;

[0040] The final prediction module is used to predict the patient's PFS based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features.

[0041] Another aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor implements the above-mentioned non-small cell lung cancer prognosis prediction method based on multimodal fusion when executing the computer program.

[0042] Another aspect of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for predicting the prognosis of non-small cell lung cancer based on multimodal fusion.

[0043] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0044] The positive progressive effect of the present disclosure lies in: the non-small cell lung cancer prognosis prediction method based on multimodal fusion of the present disclosure, combined with the multimodal prognosis prediction of CT images and dose distribution images, can fully combine the multimodal complementary feature information contained in the primary tumor and the affected lymph nodes, and can process the truncated data in the prognosis label, thereby achieving accurate prediction of the patient's progression-free survival after radiotherapy, helping clinical screening of high-risk groups for progression after radiotherapy, and then guiding the formulation of individualized treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the process of the non-small cell lung cancer prognosis prediction method based on multimodal fusion provided in Example 1 of the present disclosure;

[0046] Figure 2a is a schematic diagram of the CT image before treatment. Figure 2b It is a schematic diagram of a three-dimensional dose distribution image of radiotherapy;

[0047] Figure 3 A schematic flow chart of step S30 of the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion provided in Example 1 of the present disclosure;

[0048] Figure 4 A schematic flow chart of step S32 of the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion provided in Example 1 of the present disclosure;

[0049] Figure 5 A schematic flow chart of step S40 of the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion provided in Example 1 of the present disclosure;

[0050] Figure 6a The CT image before treatment and the delineated region of interest provided in Example 1 of the present disclosure, wherein the red color represents the primary tumor region and the green color represents the affected lymph node region; Figure 6b is the mask image corresponding to the tumor area (MASK T ), Figure 6c The mask image corresponding to the lymph node area (MASK N ), Figure 6d is the obtained spatial distance map between lymph nodes and tumors (N-Tmap);

[0051] Figure 7 A schematic diagram of the framework of a non-small cell lung cancer prognosis prediction device based on multimodal fusion provided in Example 2 of the present disclosure;

[0052] Figure 8 A schematic diagram of the framework of a multi-omics prediction label module of a non-small cell lung cancer prognosis prediction device based on multimodal fusion provided in Example 2 of the present disclosure;

[0053] Fig. 9 A schematic diagram of the framework of a deep survival label module of a non-small cell lung cancer prognosis prediction device based on multimodal fusion provided in Example 2 of the present disclosure;

[0054] Fig.10 This is a schematic diagram of the structure of an electronic device shown in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0055] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.

[0056] The prefixes such as "first" and "second" used in the embodiments of the present disclosure are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the objects being described. The use of prefixes such as ordinal numbers used to distinguish description objects in the embodiments of the present disclosure does not limit the objects being described. For the description of the objects being described, please refer to the context of the claims or embodiments.

[0057] The description of the present embodiment should not constitute an unnecessary restriction due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0058] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0059] Example 1

[0060] Figure 1 A flowchart of a method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion provided in Example 1 of the present disclosure. The method comprises the following steps:

[0061] S10, obtaining a pre-treatment CT image and a three-dimensional radiotherapy dose distribution image of the target patient;

[0062] S20, acquiring a region of interest according to the pre-treatment CT image;

[0063] S30, obtaining a multi-omics prediction label (P dual-omics );

[0064] S40, obtaining a deep survival label (P) based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image, and the region of interest. deepsurv );

[0065] S50, predicting the patient's PFS based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features.

[0066] Before step S10, the case information of the target patient can be obtained. The target patient is a locally advanced NSCLC patient receiving radiotherapy; a NSCLC patient with pathological stage III; and a total dose of no less than 50 Gy of radiotherapy. Exclusion criteria are: the patient has received adjuvant or neoadjuvant therapy such as targeted therapy; suffers from autoimmune diseases; died within one month after radiotherapy; image or clinical information is missing. The case information includes clinical feature information, including: age, gender, smoking history, whether consolidation treatment, stage, radiotherapy dose, radiotherapy mode, etc.

[0067] In the training process of each model involved in the method, in order to obtain training samples and test samples, the case information of multiple historical patients can be collected. The historical patients are also patients with locally advanced NSCLC who received radiotherapy. The case information of the historical patients includes clinical, prognostic and other characteristic information, including: age, gender, smoking history, whether consolidation treatment is given, staging, radiotherapy dose, radiotherapy mode, etc., and the patient's prognosis, that is, progression-free survival (PFS) is obtained through examination records and follow-up, which is defined as the time from the start of radiotherapy to the occurrence of progression or the last follow-up.

[0068] The pre-treatment CT image and the radiotherapy three-dimensional dose distribution image obtained in step S10 are respectively as follows: Figure 2a , 2b shown.

[0069] The step S20 is to obtain a region of interest based on the pre-treatment CT image. In this step, the region of interest includes the primary tumor region ROI T and affected lymph node regions ROI N Optionally, an experienced physician can outline the primary tumor region ROI on the MIM software. T and affected lymph node regions ROI N silhouette.

[0070] Before step S30, the method further includes preprocessing the acquired pre-treatment CT image and the radiotherapy three-dimensional dose distribution image. The preprocessing step includes matching the pre-treatment CT image and the radiotherapy three-dimensional dose distribution image and interpolating them to the same resolution. Specifically, the method includes:

[0071] S301, converting the original pixel value of the CT image before treatment into a projection value (HU value); wherein,

[0072] HU=Rescale slope*CT pixelvalue+Rescale intercept

[0073] CT pixelvalue Represents the grayscale value of the CT image before treatment, Rescale slope is (0028, 1053) read from the dicom file, and Rescale intercept is (0028, 1052) read from the dicom file.

[0074] S302, matching the pre-treatment CT image converted into projection values ​​and the three-dimensional dose distribution image with each other and resampling them to the same resolution, preferably, for example, resampling to a resolution of 1 mm×1 mm×4 mm (corresponding to x, y, z axes).

[0075] The step S30 includes: Figure 3 As shown,

[0076] S31, based on the pre-processed pre-treatment CT image and the three-dimensional dose distribution image, extract high-dimensional radiomics features and dosiomics features in the region of interest respectively; the high-dimensional radiomics features are extracted from the pre-processed pre-treatment CT image, including shape-based features, first-order statistical features, texture features, first-order statistics and texture features based on wavelet transform; the dosiomics features are extracted from the processed three-dimensional dose distribution image, including first-order statistical features, texture features, first-order statistics and texture features based on wavelet transform.

[0077] Specifically, the Pyradiomics package (a toolkit specifically used to extract medical imaging features) of Python (a computer programming language) can be used to extract 14 shape-based features, 18 first-order statistical features, 73 texture features, and 728 first-order statistical and texture features based on wavelet transform. Finally, a total of 833 imaging features and 819 dosimetric features were extracted in each region of interest.

[0078] Before this step, in order to facilitate the extraction of imaging omics and dosimetry features, the pre-processed pre-treatment CT image and radiotherapy three-dimensional dose distribution image can also be discretized, where the discretization interval width of the pre-treatment CT image is 25HU, and the discretization interval width of the radiotherapy three-dimensional dose distribution image is 1Gy.

[0079] After this step, the extracted high-dimensional imaging features and dosimetric features can also be normalized. Specifically, all features are normalized using the maximum and minimum method, and all feature values ​​are normalized to between 0 and 1:

[0080]

[0081] Among them, x * is the normalized eigenvalue, x is the original eigenvalue extracted, and x min is the minimum value of the feature, x max is the maximum value of the feature.

[0082] S32, screening out a subset of multi-omics features with prognostic prediction potential and low redundancy from the high-dimensional imaging omics features and the dosimetric features;

[0083] In step S32, a three-step feature screening algorithm is used to screen the high-dimensional imaging omics features and the dosimetry features. The specific steps are as follows: Figure 4 As shown,

[0084] S321, using a univariate Cox regression algorithm to preliminarily select the extracted high-dimensional imaging genomics features and the dose genomics features, and screen out a first feature with prognosis prediction potential; specifically, the selection is performed based on the P value, and features with corresponding P values ​​less than 0.01 are selected.

[0085] S322, further screening the first feature using a maximum correlation minimum redundancy algorithm to obtain a second feature; specifically, screening the first feature to obtain the second feature by maximizing the correlation between the feature and the prognosis and minimizing the redundancy between the features;

[0086] The specific formula is as follows:

[0087]

[0088]

[0089] Where I(x,y) represents the mutual information between features x and y, x represents the extracted features, y represents the features in the selected feature set, or the prognostic label. ρ(x,y) represents the correlation coefficient between x and y, S represents the selected feature subset, x k Represents the features in S by continuously maximizing mRMR(x j ) Select feature subsets that are highly correlated with prognosis and have low redundancy with each other.

[0090] S323, inputting the second feature into a pre-established LASSO-Cox model, and compressing unimportant feature coefficients to 0 by adding an L1 regularization constraint term in the Cox regression, thereby achieving feature selection to obtain a multi-omics feature subset.

[0091] Specifically, the calculation formula for this step is:

[0092]

[0093]

[0094] Where L(β) is the partial likelihood function of the Cox regression model; λ is the LASSO penalty parameter. T represents the transpose of the β vector, R(t i ) represents the time t i The set of individuals that have not experienced the event before, X j The characteristics of these individuals are represented. By minimizing the above objective function, the coefficient values ​​corresponding to each feature are obtained, among which the radiomics and dosimetric features with coefficients not equal to 0 are used for subsequent predictive modeling.

[0095] S33, based on the multi-omics feature subset screened out in the above step, inputting a pre-constructed first model, and using the prediction probability output by the first model as the multi-omics prediction label;

[0096] The first model is a multi-factor Cox regression model, which is used to output the corresponding PFS prediction according to the input multi-omics feature subset. The multi-factor Cox regression model is a pre-constructed model, whose input is the multi-omics feature subset obtained by the training sample according to the same processing method as above, and the output is the progression-free survival corresponding to the training sample. After the pre-constructed training, the coefficients of each feature in the multi-omics feature subset are obtained.

[0097] Specifically, in some embodiments, the formula for obtaining the multi-omics prediction label is:

[0098]

[0099] Among them, Dosiomics represents dosimetric features, and Radiomics represents radiomics features; T represents features extracted from the primary tumor area, and N represents features extracted from the affected lymph node area; W represents features extracted after wavelet transform of the image. Specifically, the image is high-pass or low-pass filtered in the x, y, and z directions, where H represents the high-frequency filter kernel and L represents the low-frequency filter kernel. glcm represents the gray-level co-occurrence matrix, glszm represents the gray-level region size matrix, glrlm represents the gray-level run matrix, ngtdm represents the adjacent gray-level difference matrix, and gldm represents the gray-level dependence matrix; ClsuterTendency represents the clustering trend feature, GLNU represents the gray-level inhomogeneity feature, LRLGLE represents the long-run low-gray-level emphasis feature, LGLE represents the low-gray-level emphasis feature, Busyness represents the busyness feature, LAHGLE represents the large-region high-gray-level emphasis feature, RunPercentage represents the run percentage feature, SAHGLE represents the small-region high-gray-level emphasis feature, SALGLE represents the small-region low-gray-level emphasis feature, and SZNU represents the region size inhomogeneity feature; firstorder represents the first-order statistical feature, Skewness represents the skewness feature, Median represents the median feature, TotalEnergy represents the total energy feature, InterquartileRange represents the interquartile range feature, and Variance represents the variance feature.

[0100] S40, obtaining a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image, and the region of interest; Figure 5 As shown, including

[0101] S41, obtaining a mask image corresponding to the tumor region, a mask image corresponding to the lymph node region, and a spatial distance map between the lymph node and the tumor based on the pre-processed CT image before treatment and the region of interest; this step is specifically,

[0102] S411, obtaining a mask image (MASK) corresponding to the tumor region according to the pre-processed CT image before treatment and the region of interest T ) and the mask image corresponding to the lymph node area (MASK N ). The pre-processed CT images before treatment are binarized based on the region of interest, that is, on the pre-processed CT images before treatment, the pixel values ​​in the delineated tumor region are set to 1, and the pixel values ​​outside the region are set to 0. Similarly, the pixel values ​​in the delineated affected lymph node region are set to 1, and the pixel values ​​outside the region are set to 0. Figure 6a Shown is a CT image before treatment, in which red is the primary tumor area and green is the affected lymph node area; Figure 6bThe mask image (MASK T ), Figure 6c The mask image corresponding to the lymph node area (MASK N ).

[0103] S412, converting the pixel value of each pixel in the mask image corresponding to the lymph node region into the shortest distance between each voxel of the lymph node and the tumor, to obtain a spatial distance map (NT map) between the lymph node and the tumor; Figure 6d For the corresponding Figure 6a The resulting spatial distance map between lymph nodes and tumors.

[0104] The spatial distance map between lymph nodes and tumors can fully explore the interaction between primary tumors and affected lymph nodes, and further help achieve more accurate prognosis prediction.

[0105] S42, obtaining a first channel input spliced ​​image based on the pre-processed CT image before treatment, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor;

[0106] Specifically, the step is to firstly crop image blocks of fixed size, for example, 192×192×48, on the pre-treatment CT image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor based on the center of the region of interest; and then perform channel splicing on the pre-treatment CT image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor after pre-processing, to obtain the first channel input spliced ​​image [pre-treatment CT image, MASK T ,N-Tmap], for example, the first channel input stitching image is a 192×192×48×3 image.

[0107] S43, obtaining a second channel input stitched image based on the preprocessed radiotherapy three-dimensional dose distribution image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor;

[0108] Similarly, this step firstly crops an image block of a fixed size, for example, an image block of 192×192×48, based on the center of the region of interest on the preprocessed three-dimensional radiation dose distribution image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor; and then performs channel stitching on the preprocessed three-dimensional radiation dose distribution image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor to obtain a second channel input stitched image [three-dimensional dose distribution map, MASK T ,NT map], for example, the second channel input stitching image is a 192×192×48×3 image.

[0109] The first channel input stitched image and the second channel stitched image have the same size and serve as inputs of two branches of the subsequent model.

[0110] S44, using the first channel stitched image and the second channel stitched image as inputs of two channels, inputting a pre-built second model, and using the predicted probability output by the second model as a deep survival prediction label;

[0111] The pre-built second model includes two single-modal branch networks, a three-dimensional densely connected network model obtained by end-to-end training using the first channel spliced ​​image and the second channel spliced ​​image obtained by the training sample as the input of the two branch networks, and the PFS label as the supervision information, and the survival probability of the fourth time interval output by the model is used as the deep survival label P deepsurv .

[0112] Specifically, the steps of training the three-dimensional densely connected network model include:

[0113] S441, using the first channel stitched image and the second channel stitched image as input to train two single-modal branch networks separately;

[0114] S442, integrating the feature representations extracted by the single-modal branch network using a gating mechanism;

[0115] S443, further extract features through the cascaded fully connected layer and output the predicted probability.

[0116] Specifically, each of the two single-mode branch networks contains three dense blocks, and each dense block contains four bottleneck units. The bottleneck unit includes a batch normalization layer (Batch Normalization, BN), a Relu activation layer, a 1×1×1 convolution layer, and a 3×3×3 convolution layer.

[0117] In each dense block, the output of each layer is connected to the output of all previous layers to form a dense connection. Different dense blocks are compressed and downsampled through 1×1×1 convolutional layers and average pooling layers. Global pooling is used to convert the output of the last dense block into a one-dimensional feature vector (120×1).

[0118] The feature vectors output by the global pooling layers of the two branches are added and fused at the element level; then the cascaded fully connected layers are used to further extract features (the number of neurons is: 256, 64, 32, 6); the output of the last fully connected layer is processed with sigmoid activation to limit the output between 0 and 1, indicating the conditional survival probability of the patient in different time intervals. Six time intervals are defined: 0-3, 3-6, 6-9, 9-12, 12-24, 24-72 months. The following discrete time survival loss function is used to optimize the model training:

[0119] L surv =∑ M -loglik;

[0120]

[0121] Where loglik is the individual log-likelihood probability of the patient; M is the total number of patients, and the total loss is the sum of the negative log-likelihoods of all patients; surv pred An n-dimensional vector representing the output of the model, where n is the number of time intervals. s is an n-dimensional vector, where the vector value of the time period before the final event occurs is 1, and the vector value of the time period after the event occurs is 0. f It is also an n-dimensional vector, in which the vector value of the time period where the ending event occurs is 1, and the vector value of the other time periods is 0.

[0122] According to the performance after verification, the survival probability of the fourth time interval output by the model is used as the deep survival label P deepsurv .

[0123] The step S50 predicts the patient's PFS based on the multi-omics prediction label, the deep survival label and the prognosis-related clinical characteristics, specifically including taking the multi-omics prediction label, the deep survival label and the prognosis-related clinical characteristics as input, inputting a pre-constructed third model, and outputting a final prediction probability; the third model is a multi-factor Cox regression model (Elastic-Cox) constrained by an elastic network, introducing LASSO and Ridge penalty terms.

[0124] The prognosis-related clinical characteristics include age, consolidation therapy, stage, radiotherapy dose and radiotherapy mode. Similarly, training verification is performed based on the training samples and the test samples to obtain the corresponding coefficients of the multi-omics prediction label, deep survival label and prognosis-related clinical characteristics.

[0125] For example, the following formula can be used:

[0126] P fusion =Pf1×Pdeepsurv +Pf2×P dual-omics +Pf3×radiotherapy mode+Pf4×consolidation therapy+Pf5×stage+Pf6×radiotherapy dose+Pf7×age;

[0127] Where Pf1-Pf7 are the coefficients corresponding to each feature, preferably,

[0128] P fusion = -1.787 × P deepsurv +0.826×P dual-omics -0.373×radiotherapy mode-0.152×consolidation therapy+0.048×stage+0.041×radiotherapy dose-0.002×age;

[0129] Among them, P deepsurv is the constructed deep survival label, P dual-omics Predicted signatures for the established multi-omics.

[0130] The prediction method of this embodiment is equivalent to constructing a fusion model using the above-mentioned first model, second model and third model, and the performance evaluation method of the fusion model can be used to determine the accuracy of the prediction method. The above-mentioned first model, second model and third model are trained and evaluated using the same data processing steps and model training methods as in the above-mentioned prediction method. For example, case information of 329 patients with locally advanced NSCLC who received radiotherapy can be collected, and the patients can be randomly divided into a training set and an independent test set at a ratio of 2:1. Both the training samples and the test samples have the above-mentioned case information, and the case information includes clinical feature information, which also includes: age, gender, smoking history, whether consolidation treatment, staging, radiotherapy dose, radiotherapy mode, etc., and the patient's prognosis, that is, PFS, is obtained through inspection records and follow-up, which is defined as the time from the start of radiotherapy to the occurrence of progression or the last follow-up. Based on these training sets, the above-mentioned first, second and third models can be trained.

[0131] When evaluating the performance of this method, the predictive effectiveness of the multimodal fusion model is evaluated on the test set. The consistency index (C-index) and time-dependent AUC value are used to measure the discrimination ability of the model. According to the predicted probability of the final output, the patients are divided into high-risk and low-risk groups through threshold division (median value) to guide the clinical formulation of individualized treatment strategies.

[0132] The C-index of the established fusion model in the training set and test set reached 0.737 and 0.691 respectively, which significantly improved the prediction efficiency compared with the traditional prognosis prediction model based only on clinical characteristics (C-index: 0.624, 0.591). The AUC of the first, second and third years in the training set reached 0.874, 0.841 and 0.830, and the AUC of the first, second and third years in the test set reached 0.718, 0.738 and 0.727.

[0133] According to this prediction method, patients with locally advanced NSCLC can be significantly divided into high and low progression risk subgroups (P<0.001). Among them, the high-risk group needs further optimization of radiotherapy regimen or the use of other intensive treatment methods to improve patient survival.

[0134] The non-small cell lung cancer prognosis prediction method based on multimodal fusion of this embodiment, combined with the multimodal prognosis prediction of CT images and dose distribution images, can fully combine the multimodal complementary feature information contained in the primary tumor and the affected lymph nodes, and can process the truncated data in the prognosis label, thereby achieving accurate prediction of the patient's progression-free survival after radiotherapy, helping clinical screening of high-risk groups for progression after radiotherapy, and then guiding the formulation of individualized treatment plans.

[0135] Example 2

[0136] Corresponding to the aforementioned method embodiment, the present disclosure also provides a non-small cell lung cancer prognosis prediction device based on multimodal fusion, such as Figure 7 shown.

[0137] The device comprises,

[0138] A first acquisition module 10 is used to acquire a pre-treatment CT image and a radiotherapy three-dimensional dose distribution image of a target patient;

[0139] A second acquisition module 20, used for acquiring a region of interest according to the pre-treatment CT image;

[0140] A multi-omics prediction label module 30, used to obtain a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest;

[0141] A deep survival label module 40, used to obtain a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest;

[0142] The final prediction module 50 is used to predict the patient's PFS based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features.

[0143] The device of this embodiment may also include a case acquisition module 60 for acquiring case information of a target patient, where the target patient is a locally advanced NSCLC patient receiving radiotherapy, wherein the case information includes clinical characteristic information, including: age, gender, smoking history, whether consolidation treatment is performed, staging, radiotherapy dose, radiotherapy mode, etc.

[0144] The second acquisition module 20 includes a tumor acquisition module and a lymph node acquisition module, which are respectively used to acquire the primary tumor region ROI. T and affected lymph node regions ROI N .

[0145] The device of this embodiment further includes a preprocessing module 70 for preprocessing the acquired pre-treatment CT image and radiotherapy three-dimensional dose distribution image. The preprocessing steps of the preprocessing module 70 are the same as those in the method of embodiment 1 and will not be repeated here.

[0146] The multi-omics prediction label module 30, such as Figure 8 As shown, it can specifically include:

[0147] The feature extraction module 31 is used to extract high-dimensional imaging omics features and dosimetric features in the region of interest based on the pre-processed pre-treatment CT image and the three-dimensional dose distribution image. The high-dimensional imaging omics features are extracted from the pre-processed pre-treatment CT image, including shape-based features, first-order statistical features, texture features, first-order statistics and texture features based on wavelet transform, and the dosimetric features are extracted from the processed three-dimensional dose distribution image, including first-order statistical features, texture features, first-order statistics and texture features based on wavelet transform.

[0148] A feature screening module 32, for screening a multi-omics feature subset with prognostic prediction potential and low redundancy from the high-dimensional imaging omics features and the dosimetry features;

[0149] The screening step of the feature screening module 32 is to use a three-step feature screening algorithm to screen the high-dimensional imaging omics features and the dose omics features. The specific method has been described in the above embodiment 1 and will not be repeated here.

[0150] The multi-omics prediction label module 30 also includes a first model prediction module 33, which is used to input the multi-omics feature subset into a pre-constructed first model, and use the prediction probability output by the first model as the multi-omics prediction label. The specific method of the first model prediction module 33 for prediction has been described in the above embodiment 1, and will not be repeated here.

[0151] Preferably, in order to facilitate the extraction of imaging omics and dosimetry features, a discretization processing module 34 can also be included to discretize the pre-processed pre-treatment CT image and the radiotherapy three-dimensional dose distribution image, wherein the discretization interval width of the pre-treatment CT image is 25HU, and the discretization interval width of the radiotherapy three-dimensional dose distribution image is 1Gy.

[0152] The multi-omics prediction label module 30 may further include a normalization module 35 for normalizing the extracted high-dimensional imaging omics features and dosimetry features. Specifically, all features are normalized using the maximum-minimum method to normalize all feature values ​​to between 0 and 1.

[0153] The deep survival tag module 40 is as follows: Fig. 9 Shown, including

[0154] The image processing module 41 is used to obtain a mask image corresponding to the tumor area, a mask image corresponding to the lymph node area, and a spatial distance map between the lymph node and the tumor based on the pre-processed CT image before treatment and the region of interest; this step is specifically,

[0155] According to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region is obtained. T ) and the mask image corresponding to the lymph node area (MASK N ). The pre-processed CT images before treatment are binarized based on the region of interest, that is, on the pre-processed CT images before treatment, the pixel value in the delineated tumor region is set to 1, and the pixel value outside the region is set to 0. Similarly, the pixel value in the delineated affected lymph node region is set to 1, and the pixel value outside the region is set to 0. Figure 6a Shown is a CT image before treatment, in which red is the primary tumor area and green is the affected lymph node area; Figure 6b is the mask image corresponding to the tumor area (MASK T ), Figure 6c The mask image corresponding to the lymph node area (MASK N ).

[0156] The pixel value of each pixel in the mask image corresponding to the lymph node region is converted into the shortest distance between each voxel of the lymph node and the tumor, so as to obtain a spatial distance map (NT map) between the lymph node and the tumor.

[0157] The first channel image acquisition module 42 obtains a first channel input spliced ​​image based on the pre-processed CT image before treatment, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor.

[0158] Specifically, based on the center of the region of interest, a fixed-size image block, such as a 192×192×48 image block, is cropped on the pre-treatment CT image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor. Then, the pre-processed pre-treatment CT image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor are channel-joined to obtain the first channel input stitched image [pre-treatment CT image, MASK T ,NT map], for example, the first channel input spliced ​​image is a 192×192×48×3 image.

[0159] A second channel image acquisition module 43, configured to obtain a second channel input spliced ​​image based on the preprocessed radiotherapy three-dimensional dose distribution image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor;

[0160] Similarly, this step firstly crops an image block of a fixed size, for example, an image block of 192×192×48, based on the center of the region of interest on the preprocessed three-dimensional radiation dose distribution image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor; and then performs channel stitching on the preprocessed three-dimensional radiation dose distribution image, the mask image corresponding to the tumor region, and the spatial distance map between the lymph node and the tumor to obtain a second channel input stitched image [three-dimensional dose distribution map, MASK T ,NT map], for example, the second channel input stitching image is a 192×192×48×3 image.

[0161] A second model prediction module 44 is used to construct a second model in advance by taking the first channel stitched image and the second channel stitched image as inputs of two channels, and taking the predicted probability output by the second model as a deep survival prediction label;

[0162] The pre-built second model includes two single-modal branch networks, a three-dimensional densely connected network obtained by end-to-end training using the first channel spliced ​​image and the second channel spliced ​​image obtained by the training sample as the input of the two branch networks, and the PFS label as the supervision information, and the survival probability of the fourth time interval output by the model is used as the deep survival label P deepsurv .

[0163] The second model prediction module 44 includes a model training module 441, which is used to train the three-dimensional densely connected network model. The specific training steps have been described in the method of embodiment 1 and will not be repeated here.

[0164] The final prediction module 50 is used to predict the patient's PFS based on the multi-omics prediction label, the deep survival label and the prognosis-related clinical characteristics, specifically including taking the multi-omics prediction label, the deep survival label and the prognosis-related clinical characteristics as input, inputting a pre-constructed third model, and outputting the final prediction probability; the third model is a multi-factor Cox regression model constrained by an elastic network, introducing LASSO and Ridge penalty terms.

[0165] The prognosis-related clinical characteristics include age, consolidation therapy, stage, radiotherapy dose and radiotherapy mode. Similarly, training verification is performed based on the training samples and the test samples to obtain the corresponding coefficients of the multi-omics prediction label, deep survival label and prognosis-related clinical characteristics.

[0166] For example, the following formula can be used:

[0167] P fusion =Pf1×P deepsurv +Pf2×P dual-omics +Pf3×radiotherapy mode+Pf4×consolidation therapy+Pf5×stage+Pf6×radiotherapy dose+Pf7×age;

[0168] Where Pf1-Pf7 are the coefficients corresponding to each parameter, preferably,

[0169] P fusion = -1.787 × P deepsurv +0.826×P dual-omics -0.373×radiotherapy mode-0.152×consolidation therapy+0.048×stage+0.041×radiotherapy dose-0.002×age;

[0170] Among them, P deepsurv is the constructed deep survival label, P dual-omics Predicted signatures for the established multi-omics.

[0171] The non-small cell lung cancer prognosis prediction device based on multimodal fusion of this embodiment, combined with the multimodal prognosis prediction model of CT images and dose distribution images, can fully combine the multimodal complementary feature information contained in the primary tumor and the affected lymph nodes, and can process the truncated data in the prognosis label, so as to achieve accurate prediction of the patient's progression-free survival after radiotherapy, help clinical screening of high-risk groups for progression after radiotherapy, and then guide the formulation of individualized treatment plans.

[0172] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.

[0173] Example 3

[0174] Fig.10 This is a structural schematic diagram of an electronic device shown in an example embodiment of the present disclosure, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, the non-small cell lung cancer prognosis prediction method based on multimodal fusion described in any of the above embodiments is implemented. Fig.10 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0175] like Fig.10 As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different device components (including the memory 92 and the processor 91).

[0176] The bus 93 includes a data bus, an address bus, and a control bus.

[0177] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .

[0178] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) of program modules 924, such program modules 924 including but not limited to: operating means, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0179] The processor 91 executes various functional applications and data processing by running the computer program stored in the memory 92, such as the non-small cell lung cancer prognosis prediction method based on multimodal fusion provided in any of the above embodiments.

[0180] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) devices, tape drives, and data backup storage devices, etc.

[0181] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0182] Example 4

[0183] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion provided in any of the above embodiments is implemented.

[0184] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0185] Example 5

[0186] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for predicting the prognosis of non-small cell lung cancer based on multimodal fusion.

[0187] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0188] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion, characterized in that: The following steps are involved: Obtain the target patient's pre-treatment CT image and radiotherapy three-dimensional dose distribution image; Acquiring a region of interest according to the pre-treatment CT image; Based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest, a multi-omics prediction label is obtained, including: Based on the pre-processed CT image and the three-dimensional dose distribution image before treatment, respectively extracting high-dimensional imaging features and dosimetric features in the region of interest; Screening a subset of multi-omics features with prognostic prediction potential and low redundancy from the high-dimensional imaging omics features and the dosimetry features; Based on the multi-omics feature subset, a pre-constructed first model is input, and the prediction probability output by the first model is used as the multi-omics prediction label; The first model is a multi-factor Cox regression model, which is used to output corresponding PFS predictions based on the input multi-omics feature subsets; Based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest, a deep survival label is obtained, including: According to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region, a mask image corresponding to the lymph node region, and a spatial distance map between the lymph node and the tumor are obtained; Obtaining a first channel input spliced ​​image based on the pre-processed CT image before treatment, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor; Obtaining a second channel input spliced ​​image based on the preprocessed radiotherapy three-dimensional dose distribution image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor; Input the first channel stitched image and the second channel stitched image into a pre-built second model, and use the predicted probability output by the second model as a deep survival prediction label; The pre-built second model includes two single-modal branch networks, which are a three-dimensional densely connected network obtained by end-to-end training using the first channel stitching image and the second channel stitching image obtained by the training sample as inputs of the two branch networks and the PFS label as supervision information; The patient's progression-free survival is predicted based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features.

2. The method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to claim 1, characterized in that: Before the step of obtaining a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest, the step further includes: Preprocessing the pre-treatment CT image and the radiotherapy three-dimensional dose distribution image, wherein the preprocessing includes matching the pre-treatment CT image and the radiotherapy three-dimensional dose distribution image and interpolating them to the same resolution; The region of interest includes the primary tumor area and the affected lymph node area.

3. The method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to claim 1, characterized in that: The steps of training the three-dimensional densely connected network include: Taking the first channel stitched image and the second channel stitched image as input, respectively, two single-modal branch networks are trained separately; Using a gating mechanism to integrate the feature representations extracted by the unimodal network branch; Features are further extracted through the cascaded fully connected layers to output the predicted probability.

4. The method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to claim 1, characterized in that: The patient's PFS prediction is achieved based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features. include, The multi-omics prediction label, the deep survival label, and the prognosis-related clinical features are input into a pre-built third model, and the final prediction probability is obtained as output; The third model is a multi-factor Cox regression model with elastic network constraints, and LASSO and Ridge penalty terms are introduced into the Cox regression.

5. The method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to claim 1, characterized in that: According to the pre-processed CT image before treatment and the region of interest, a mask image corresponding to the tumor region, a mask image corresponding to the lymph node region and a spatial distance map between the lymph node and the tumor are obtained, including: Obtaining a mask image corresponding to the tumor region and a mask image corresponding to the lymph node region according to the pre-processed CT image before treatment and the region of interest; The pixel value of each pixel in the mask image corresponding to the lymph node region is converted into the shortest distance between each voxel of the lymph node and the tumor, so as to obtain a spatial distance map between the lymph node and the tumor.

6. A non-small cell lung cancer prognosis prediction device based on multimodal fusion, characterized in that: include, A first acquisition module is used to acquire a pre-treatment CT image and a radiotherapy three-dimensional dose distribution image of a target patient; A second acquisition module, used for acquiring a region of interest according to the pre-treatment CT image; A multi-omics prediction label module, used to obtain a multi-omics prediction label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest; The prediction label module includes: A feature extraction module is used to extract high-dimensional imaging features and dosimetric features in the region of interest based on the pre-processed pre-treatment CT images and three-dimensional dose distribution images; A feature screening module, used to screen out a multi-omics feature subset with prognostic prediction potential and low redundancy from the high-dimensional imaging omics features and the dosimetry features; A first model prediction module, configured to input the multi-omics feature subset into a pre-built first model, and use the prediction probability output by the first model as the multi-omics prediction label; The first model is a multi-factor Cox regression model, which is used to output corresponding PFS predictions based on the input multi-omics feature subsets; A deep survival label module, used for obtaining a deep survival label based on the pre-treatment CT image, the radiotherapy three-dimensional dose distribution image and the region of interest; The deep survival tag module includes: An image processing module, used to obtain a mask image corresponding to the tumor area, a mask image corresponding to the lymph node area, and a spatial distance map between the lymph node and the tumor according to the pre-processed CT image before treatment and the region of interest; A first channel image acquisition module, which obtains a first channel input spliced ​​image based on the pre-processed CT image before treatment, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor; A second channel image acquisition module, used for obtaining a second channel input spliced ​​image based on the preprocessed radiotherapy three-dimensional dose distribution image, the mask image corresponding to the tumor area, and the spatial distance map between the lymph node and the tumor; A second model prediction module is used to use the first channel stitched image and the second channel stitched image as the second model pre-built as the input of the two channels, and use the predicted probability output by the second model as the deep survival prediction label; The pre-built second model includes two single-modal branch networks, which are a three-dimensional densely connected network obtained by end-to-end training using the first channel stitching image and the second channel stitching image obtained by the training sample as inputs of the two branch networks and the PFS label as supervision information; The final prediction module is used to predict the patient's PFS based on the multi-omics prediction signature, deep survival signature, and prognosis-related clinical features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to any one of claims 1 to 5 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for predicting the prognosis of non-small cell lung cancer based on multimodal fusion according to any one of claims 1 to 5 is implemented.

Images