Colorectal cancer liver-lung metastasis prediction method and system based on multi-modal large model

Abstract

The invention discloses a colorectal cancer liver-lung metastasis prediction method and system based on a multi-modal large model, and the method comprises the steps: S1, collecting the multi-modal data of a colorectal cancer patient, carrying out the preprocessing, and carrying out the feature extraction and feature fusion, and obtaining the multi-modal input data; s2, constructing a multi-modal large model, processing multi-modal input data, and outputting a prediction probability; s3, training the multi-modal large model, and then evaluating the performance of the multi-modal large model; s4, performing risk assessment on the patient and generating a dynamic follow-up visit plan according to the prediction probability; s5, generating a corresponding treatment suggestion and an explanatory report based on the feature contribution weight and the risk assessment result; and S6, performing performance verification on the model on the independent test set. According to the invention, a major breakthrough is made in the aspects of accurate prediction and patient management, and a powerful technical support is provided for early monitoring and intervention of colorectal cancer liver-lung metastasis. The follow-up management process is optimized, and the medical resource utilization efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a method and system for predicting liver-pulmonary metastasis of colorectal cancer based on a multimodal large model. Background Art

[0002] Colorectal cancer is the third most common cancer in the world, and its metachronous liver and lung metastasis is one of the main causes of poor prognosis and death in patients. According to the "2021 Global Colorectal Cancer Report", there are approximately 1.93 million new cases of colorectal cancer each year worldwide, of which more than 50% of patients will develop metachronous liver and lung metastases after diagnosis, and the five-year survival rate of these patients is less than 10%. Further studies have shown that for patients with metastatic colorectal cancer, if effective monitoring and intervention can be carried out in the early stages of metastasis, the patient's survival rate can be increased by 20%-30%. However, in actual clinical practice, misdiagnosis or missed diagnosis caused by delayed diagnosis causes many patients to miss the best time for early treatment.

[0003] Existing follow-up management systems have significant limitations in clinical applications. First, current methods mainly rely on single-modality data, such as risk assessment based on radiomics features or clinical features alone, which leads to insufficient data utilization. Radiomics features (such as tumor texture, shape, and density) can reflect the microscopic heterogeneity of tumors, but lack global background information, while clinical features (such as patient age, pathological stage, CEA, and CA19-9) provide an overall overview of the disease, but cannot capture the microscopic characteristics of the tumor, resulting in one-sided prediction results. Studies have shown that the predictive performance of single-modality models is limited, and its AUC value usually hovers between 0.65-0.78, with a missed diagnosis rate of up to 30% for high-risk patients. Secondly, existing follow-up management systems usually adopt fixed plans and cannot adjust the frequency and content of follow-up in a timely manner according to the dynamic changes of the patient's condition. For example, for patients with a rapidly increasing risk of liver and lung metastasis, a fixed follow-up every 6 months obviously cannot meet clinical needs and delay the timing of intervention. Finally, existing methods lack support for the interpretability of model prediction results, making it difficult for doctors to understand the key basis of risk assessment results, resulting in a lack of sufficient reference information when formulating personalized treatment plans. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method and system for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model.

[0005] The objective of the present invention is achieved through the following technical solutions: In a first aspect, the present invention discloses a method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model, comprising the following steps: S1. Multimodal data collection and preprocessing: collect multimodal data of colorectal cancer patients, including imaging data, clinical data and genetic data, extract features after preprocessing the imaging data to obtain imaging features, encode clinical data and genetic data respectively to obtain clinical features and genetic features, and then fuse the imaging features, clinical features and genetic features to obtain multimodal input data; S2. Construction of a multimodal risk prediction model: construct a multimodal large model, process the imaging features, clinical features and genetic features in the multimodal input data through the multimodal large model, and output the patient's metastasis risk prediction probability; S3, model training and evaluation, training the multimodal large model and then evaluating the performance of the multimodal large model; S4. Dynamic follow-up management: According to the predicted probability of metastasis risk output by the multimodal large model, patients are risk assessed and divided into low, medium and high risk groups, and a dynamic follow-up plan is generated; S5. Calculate the feature contribution weights, generate corresponding treatment recommendations based on the risk assessment results and feature contribution weights, and generate an explanatory report, including feature importance analysis and risk prediction basis; S6. Model verification: verify the performance of the model on an independent test set.

[0006] Based on the first aspect, step S1 specifically includes: the preprocessing includes normalizing the size and grayscale of the image data by normalization. ,in is the original image data, is the mean pixel value of the original image data, is the standard deviation of the pixel values ​​of the original image data, and then the preprocessed image data Perform feature extraction , get image features ,in Represent the omics features in the imaging data; encode the clinical data, which includes the patient's age, gender, pathological stage and biochemical indicators, and encode the discrete clinical data into continuous numerical features to obtain clinical features ,in Indicates i The features corresponding to the clinical data are encoded, and the gene data includes the KRAS gene mutation status and the BRAF gene mutation status to obtain the gene features. ,in Indicates i Gene mutation characteristics; Clinical features Normalize , mapping it to the range [0,1], where Represents the normalized i The features corresponding to the clinical data are maxx represents the maximum value of the feature corresponding to the clinical data, minx The minimum value of the feature corresponding to the clinical data; for the gene feature Normalize , mapping it to the range [0,1], where Represents the normalized i Gene mutation characteristics, Indicates the minimum value of gene mutation characteristics, Represents the maximum value of the gene mutation feature; then the imaging features, clinical features and gene features are fused to obtain multimodal input data .

[0007] Based on the first aspect, step S2 specifically includes: constructing a multimodal large model, the multimodal large model includes an image modality encoder, a clinical modality encoder, a genetic modality encoder, a modality fusion module and a fully connected classification layer, the image modality encoder uses a visual Transformer to process image features and generate an embedded representation ; Clinical modality encoder uses multi-layer perceptron to process clinical features ; Genetic modality encoder models genetic features through deep neural networks ; The modality fusion module uses the self-attention mechanism Integrate to obtain multimodal features, where the attention weight calculation formula is: , where Q represents the query matrix, K represents the key matrix, and V represents the value matrix. represents the transpose of K, Represents the dimension of the key matrix; multimodal features Input to the fully connected classification layer and output the patient's metastasis risk prediction probability , where W is the weight matrix and b is the bias.

[0008] Based on the first aspect, step S3 specifically includes: by minimizing the cross entropy loss function Train the multimodal large model, where N is the total number of samples, is the real transfer label, is the predicted probability; the performance of the multimodal large model is evaluated by the AUC value , where TPR and FPR are the true positive rate and false positive rate, respectively.

[0009] Based on the first aspect, step S4 specifically includes: according to the transfer risk probability predicted by the model , patients are divided into low, medium and high risk groups, and a dynamic follow-up plan is generated; the follow-up plan is dynamically optimized based on time series data ,in represents the change in the patient's condition at time t, represents the follow-up plan at time t+1, represents the follow-up plan at time t.

[0010] Based on the first aspect, step S5 specifically includes: by formula Calculate the feature contribution weight, where Factors are calculated for gradient attribution, and corresponding treatment recommendations are generated based on risk assessment results and feature contribution weights. An explanatory report is also generated, including feature importance analysis and risk prediction basis.

[0011] In a second aspect, the present invention discloses a colorectal cancer liver and lung metastasis prediction system based on a multimodal large model, which is used in the above-mentioned colorectal cancer liver and lung metastasis prediction method based on a multimodal large model, comprising: A data acquisition module, which is used to collect multimodal data of colorectal cancer patients, including imaging data, clinical data, and genetic data; A data processing module, used for preprocessing multimodal data collected from patients with colorectal cancer; A multimodal large model building module, used to build a multimodal large model; Model training and evaluation module, used to train the model and evaluate the model performance; Dynamic follow-up module, used to perform risk stratification and generate dynamic follow-up plans; Functional module for generating corresponding treatment recommendations and corresponding explanatory reports; The model verification module is used to verify the performance of the constructed multimodal large model.

[0012] Based on the second aspect, the multimodal large model includes: Image modality encoder, used to process image features; clinical modality coders, which process clinical features; Gene modality encoder, used to process genetic features; The modality fusion module is used to fuse imaging features, clinical features, and genetic features through a self-attention mechanism to obtain multimodal features; Classification layer: used to process the input multimodal features and output the patient's colorectal cancer metastasis risk probability.

[0013] The beneficial effects of the present invention are: 1) Multimodal fusion improves prediction performance. By integrating imaging features, clinical features and genetic data, the present invention overcomes the limitation of insufficient information utilization of single-modality methods and comprehensively captures the multidimensional characteristics of liver and lung metastases: imaging features: reflect the microscopic characteristics of tumors (such as texture, shape and density), and can provide rich disease phenotype information; clinical features: provide the overall disease background of patients (such as age, pathological stage, biochemical indicators, etc.); genetic data: reveal the molecular biological characteristics of tumors (such as KRAS and BRAF mutation status); through deep fusion between modalities, the prediction model of the present invention has an AUC value of 0.94 on an independent test set, which is significantly higher than the single-modality model (imaging AUC 0.78, clinical feature AUC 0.72). Especially in the identification of high-risk patients, the present invention significantly reduces the missed diagnosis rate, and the recall rate reaches 90%.

[0014] 2) Modeling of complex interactions between modalities. This paper uses the self-attention mechanism to model the complex interactions between radiomics, clinical and genetic features, dynamically assigns feature weights, and highlights the most important modality features for risk prediction. Compared with traditional weighted fusion methods, this paper can capture the potential correlation between different modalities, improve the model's ability to handle heterogeneous data, and make the prediction results more comprehensive and accurate.

[0015] 3) Dynamic follow-up management to improve management efficiency. Traditional follow-up systems are mostly based on fixed time plans and cannot adjust the follow-up frequency according to the dynamic changes in the patient's condition. The present invention uses time series modeling to achieve dynamic adjustment of the follow-up plan: low-risk patients: follow-up is recommended every 6 months; medium-risk patients: follow-up is recommended every 3 months, combined with imaging and biochemical examinations; high-risk patients: monthly follow-up is recommended, combined with targeted gene testing and dynamic imaging evaluation; it can monitor the patient's risk changes in real time, optimize the allocation of medical resources by dynamically updating the follow-up plan, and provide high-risk patients with the opportunity for timely intervention.

[0016] 4) Personalized treatment recommendations to improve treatment accuracy. By analyzing the contribution weights of multimodal features to risk prediction, the present invention can generate personalized treatment recommendations for patients at different risk levels. For example: low-risk patients: routine monitoring, no special intervention required; medium-risk patients: adjuvant therapy (such as chemotherapy) and intensive monitoring of tumor changes are recommended; high-risk patients: targeted therapy or immunotherapy is recommended, and further genetic testing is performed; feature contribution weights are calculated using a gradient attribution method, for example, tumor texture heterogeneity (contribution 35%) and CEA levels (contribution 28%) are marked as major risk factors. This personalized treatment support greatly improves the scientificity and accuracy of clinical decision-making.

[0017] 5) Strong adaptability and a wide range of clinical application scenarios. The present invention is highly adaptable and compatible with imaging data from different sources (such as CT, MRI) and clinical data in various formats. It is suitable for a variety of clinical scenarios: Postoperative follow-up: helps doctors dynamically monitor the metastasis risk of postoperative patients; Early intervention: provides timely treatment recommendations for high-risk patients to prolong survival time; Personalized treatment: tailor-made treatment and monitoring plans for patients to improve treatment outcomes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the steps of a method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model according to an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a colorectal cancer liver and lung metastasis prediction system based on a multimodal large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] This paper proposes a method and system for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model. By integrating the patient's imaging features (such as tumor texture, shape, density, etc.), clinical features (such as pathological stage, biochemical indicators, age) and genetic data (such as mutation status of KRAS, BRAF, etc.), a multimodal large model (MLLM) is used to achieve accurate assessment and dynamic management of liver and lung metastasis risk. Visual Transformer (ViT) and multi-layer perceptron (MLP) are used to deeply encode images and clinical data, and the fusion of inter-modal features is achieved through the self-attention mechanism (Self-Attention), so as to capture the complex interactive relationship between images and clinical features and comprehensively improve the accuracy of risk prediction. On the test data set, the constructed prediction model has an AUC value of 0.94 and a recall rate of 90%, which is significantly better than the traditional single-modal model. In addition, time series modeling supports dynamic monitoring of patients' conditions and real-time adjustment of follow-up plans. For example, for high-risk patients, the platform can adjust the follow-up frequency from every 6 months to every 1 month, and generate personalized treatment recommendations in real time, including targeted therapy, imaging monitoring frequency adjustment, etc. At the same time, an interpretable report is generated to show the contribution weights of key predictive features (such as tumor texture heterogeneity or CEA levels) to help doctors formulate personalized treatment plans more efficiently. Through the integration and dynamic management of multimodal data, the present invention has made major breakthroughs in accurate prediction and patient management, and provides strong technical support for early monitoring and intervention of colorectal cancer liver and lung metastasis. It not only optimizes the follow-up management process, but also greatly improves the efficiency of medical resource utilization, significantly improves patient survival rate and quality of life, and has broad promotion potential in clinical applications.

[0021] Specifically, the present invention discloses a method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model, and the schematic diagram of the steps is as follows: Figure 1 As shown in the figure, by integrating radiomics features, clinical features and genetic data, a multimodal large model is used to achieve accurate prediction of metastasis risk and dynamic follow-up management, while providing personalized treatment recommendations. The following steps are included: S1. Multimodal data acquisition and preprocessing: multimodal data of colorectal cancer patients, including imaging data, clinical data and genetic data, are collected. The imaging data is derived from CT or MRI images. After preprocessing the imaging data, image features such as texture, shape and density of the tumor are extracted to obtain image features. Clinical data and genetic data are encoded respectively to obtain clinical features and genetic features. Then, the imaging features, clinical features and genetic features are fused to obtain multimodal input data. S2. Construction of a multimodal risk prediction model: construct a multimodal large model, process the imaging features, clinical features and genetic features in the multimodal input data through the multimodal large model, and output the patient's metastasis risk prediction probability; S3, model training and evaluation, training the multimodal large model and then evaluating the performance of the multimodal large model; S4. Dynamic follow-up management: According to the predicted probability of metastasis risk output by the multimodal large model, patients are risk assessed and divided into low, medium and high risk groups, and a dynamic follow-up plan is generated; S5. Calculate the feature contribution weights, generate corresponding treatment recommendations based on the risk assessment results and feature contribution weights, and generate an explanatory report, including feature importance analysis and risk prediction basis; S6. Model verification: The performance of the model was verified on an independent test set. The test results showed that the prediction model had an AUC value of 0.94 and a recall rate of 90%, which was significantly better than the existing single-modal method (AUC value of 0.78).

[0022] Specifically, step S1 specifically includes: the preprocessing includes normalizing the size and grayscale of the image data by normalization. ,in is the original image data, is the mean pixel value of the original image data, is the standard deviation of the pixel values ​​of the original image data, and then the preprocessed image data Perform feature extraction , get image features ,in Represent the omics features in the imaging data; encode the clinical data, which includes the patient's age, gender, pathological stage and biochemical indicators (such as CEA and CA19-9), etc., and encode the discrete clinical data into continuous numerical features to obtain clinical features ,in Indicates i The features corresponding to the clinical data are encoded, and the gene data includes the KRAS gene mutation status and the BRAF gene mutation status, etc., to obtain the gene features. ,in Indicates i Gene mutation characteristics; Clinical features Normalize , mapping it to the range [0,1], where Represents the normalized i The features corresponding to the clinical data are maxx represents the maximum value of the feature corresponding to the clinical data, minx The minimum value of the feature corresponding to the clinical data; for the gene feature Normalize , mapping it to the range [0,1], where Represents the normalized i Gene mutation characteristics, Indicates the minimum value of gene mutation characteristics, Represents the maximum value of the gene mutation feature; then the imaging features, clinical features and gene features are fused to obtain multimodal input data .

[0023] Specifically, step S2 specifically includes: constructing a multimodal large model, the multimodal large model includes an image modality encoder, a clinical modality encoder, a genetic modality encoder, a modality fusion module and a fully connected classification layer, the image modality encoder uses a visual Transformer (ViT) to process image features and generate an embedded representation ; Clinical modality encoder uses a multi-layer perceptron (MLP) to process clinical features ; Genetic modality encoder models genetic features through deep neural network (DNN) ; The modality fusion module uses the self-attention mechanism Integrate to obtain multimodal features, where the attention weight calculation formula is: , where Q represents the query matrix, K represents the key matrix, and V represents the value matrix. represents the transpose of K, Represents the dimension of the key matrix; multimodal features Input to the fully connected classification layer and output the patient's metastasis risk prediction probability , where W is the weight matrix and b is the bias.

[0024] Specifically, step S3 specifically includes: by minimizing the cross entropy loss function Train the multimodal large model, where N is the total number of samples, is the real transfer label, is the predicted probability; the performance of the multimodal large model is evaluated by the AUC value , where TPR and FPR are the true positive rate and false positive rate, respectively.

[0025] Specifically, step S4 includes: according to the transfer risk probability predicted by the model , patients are divided into low, medium and high risk groups, and a dynamic follow-up plan is generated. Low-risk patients are recommended to be followed up every 6 months, medium-risk patients every 3 months, and high-risk patients every 1 month; the follow-up plan is dynamically optimized based on time series data ,in represents the change in the patient's condition at time t, represents the follow-up plan at time t+1, Shows the follow-up plan at time t.

[0026] Specifically, step S5 specifically includes: by formula Calculate the feature contribution weight, where The system calculates factors for gradient attribution, generates corresponding treatment recommendations based on risk assessment results and feature contribution weights, and generates explanatory reports, including feature importance analysis and risk prediction basis; routine monitoring is recommended for low-risk groups, adjuvant therapy is recommended for medium-risk groups, and further targeted therapy or immunotherapy is recommended for high-risk groups. The system also generates detailed explanatory reports, including feature importance analysis and risk prediction basis, to help doctors develop accurate treatment plans.

[0027] Specifically, the structural schematic diagram of a colorectal cancer liver and lung metastasis prediction system based on a multimodal large model disclosed in the present invention is as follows: Figure 2 As shown, the system used for the above-mentioned prediction method of colorectal cancer liver and lung metastasis based on a multimodal large model is integrated into the clinical information platform, supporting real-time data input and risk assessment, and providing doctors with immediate follow-up management and treatment support. Including: A data acquisition module, which is used to collect multimodal data of colorectal cancer patients, including imaging data, clinical data, and genetic data; A data processing module, used for preprocessing multimodal data collected from patients with colorectal cancer; A multimodal large model building module, used to build a multimodal large model; Model training and evaluation module, used to train the model and evaluate the model performance; Dynamic follow-up module, used to perform risk stratification and generate dynamic follow-up plans; Functional module for generating corresponding treatment recommendations and corresponding explanatory reports; The model verification module is used to verify the performance of the constructed multimodal large model.

[0028] Specifically, the multimodal large model includes: Image modality encoder, used to process image features; clinical modality coders, which process clinical features; Gene modality encoder, used to process genetic features; The modality fusion module is used to fuse imaging features, clinical features, and genetic features through a self-attention mechanism to obtain multimodal features; Classification layer: used to process the input multimodal features and output the patient's colorectal cancer metastasis risk probability.

[0029] The present invention realizes the deep fusion of radiomics, clinical characteristics and genetic data through the design and application of a multimodal large model, accurately predicts the patient's risk of liver and lung metastasis, and adjusts the follow-up plan and treatment recommendations according to the patient's dynamic condition. The system significantly improves the prediction accuracy (the AUC value reaches 0.94), optimizes the follow-up process and the efficiency of medical resource utilization, provides intelligent and personalized diagnosis and treatment support for colorectal cancer patients, and has a wide range of clinical application value. The core lies in the innovative application of multimodal data fusion, using the deep integration of radiomics features, clinical characteristics and genetic data to construct a multimodal large model (Multimodal Large Model, MLLM) to achieve accurate prediction, dynamic follow-up and personalized treatment support for the risk of liver and lung metastasis of colorectal cancer. Specifically including: multimodal data fusion, the present invention realizes the deep fusion of radiomics features, clinical characteristics and genetic data through a multimodal large model, overcoming the problem of insufficient information utilization in the single-modal data method. Radiomic features (such as tumor texture, shape, and density) provide microscopic tumor characteristics, clinical features (such as age, pathological stage, CEA, CA19-9, etc.) provide the patient's global disease background, and genetic data (such as KRAS and BRAF mutation status) reveal the molecular biological characteristics of the tumor. By integrating these features, the patient's metastasis risk factors are fully captured, significantly improving the accuracy of risk prediction. Modeling of intermodal interaction relationships, using the self-attention mechanism to model the complex interaction between radiomics, clinical, and genetic features, thereby mining the potential correlation between multimodal data. This mechanism can dynamically assign feature weights, highlight the most important modal features for risk prediction, and improve the modeling ability of the model for heterogeneous data. Dynamic follow-up management, the system realizes dynamic monitoring of patient risks through time series modeling, and can adjust risk assessment results and follow-up plans in real time according to follow-up data. It is recommended that low-risk patients be followed up every 6 months, medium-risk patients every 3 months, and high-risk patients be followed up once a month, and the management process is optimized by combining dynamic imaging and biochemical examination frequency. Personalized treatment support: Based on the risk prediction results and feature contribution analysis generated by the multimodal large model, the system can provide personalized treatment recommendations for patients at different risk levels. For example, routine monitoring is recommended for low-risk patients, adjuvant therapy is recommended for medium-risk patients, and targeted therapy or immunotherapy is recommended for high-risk patients. The feature contribution weight is calculated using the gradient attribution method, so that the treatment recommendations have sufficient scientific basis.

[0030] Specifically, the design and application of multimodal large models, using a multimodal Transformer architecture to deeply integrate radiomics features, clinical features and genetic features; the specific implementation of modality encoders (imaging modality, clinical modality, genetic modality) and modality fusion modules. The application of self-attention mechanism for modeling the interactive relationship between modalities. The deep integration of radiomics and clinical features provides a feature fusion method for radiomics, clinical and genetic data, including the specific process of feature extraction, standardization and data integration. The joint analysis of radiomics features (such as texture, shape, etc.) and patient clinical background information is used to improve model performance. Dynamic follow-up management, to achieve dynamic risk assessment based on time series, follow-up plan generation and optimization. Support dynamic update of risk over time, build dynamic risk curves for real-time adjustment of follow-up plans. Personalized treatment recommendation generation, based on multimodal data, generates personalized treatment recommendations for patients with different risk levels, including targeted therapy, immunotherapy and adjuvant therapy. Calculation method of feature contribution weights, and explanatory treatment decision support based on contribution weights.

[0031] The core technology of this invention lies in the deep fusion of multimodal data and dynamic risk management, which uses a large multimodal model to comprehensively capture the multidimensional information of patients' metastatic risk and provide accurate risk prediction and personalized treatment recommendations. Through the joint modeling of imaging, clinical and genetic features, the introduction of self-attention mechanism and the realization of dynamic follow-up management.

[0032] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model, characterized in that: The following steps are involved: S1. Multimodal data collection and preprocessing: collect multimodal data of colorectal cancer patients, including imaging data, clinical data and genetic data, extract features after preprocessing the imaging data to obtain imaging features, encode clinical data and genetic data respectively to obtain clinical features and genetic features, and then fuse the imaging features, clinical features and genetic features to obtain multimodal input data; S2. Construction of a multimodal risk prediction model: construct a multimodal large model, process the imaging features, clinical features and genetic features in the multimodal input data through the multimodal large model, and output the patient's metastasis risk prediction probability; S3, model training and evaluation, training the multimodal large model and then evaluating the performance of the multimodal large model; S4. Dynamic follow-up management: According to the predicted probability of metastasis risk output by the multimodal large model, patients are risk assessed and divided into low, medium and high risk groups, and a dynamic follow-up plan is generated; S5. Generate corresponding treatment recommendations and explanatory reports. Calculate feature contribution weights, generate corresponding treatment recommendations based on risk assessment results and feature contribution weights, and also generate explanatory reports, including feature importance analysis and risk prediction basis; S6. Model verification: verify the performance of the model on an independent test set.

2. The method for predicting colorectal cancer liver and lung metastasis based on a multimodal large model according to claim 1, characterized in that: Step S1 specifically includes: the preprocessing includes normalizing the size and grayscale of the image data by normalization. ,in is the original image data, is the mean pixel value of the original image data, is the standard deviation of the pixel values ​​of the original image data, and then the preprocessed image data Perform feature extraction , get image features ,in Represent the omics features in the imaging data; encode the clinical data, which includes the patient's age, gender, pathological stage and biochemical indicators, and encode the discrete clinical data into continuous numerical features to obtain clinical features ,in Indicates i The features corresponding to the clinical data are encoded, and the gene data includes the KRAS gene mutation status and the BRAF gene mutation status to obtain the gene features. ,in Indicates i Gene mutation characteristics; Clinical features Normalize , mapping it to the range [0,1], where Represents the normalized i The features corresponding to the clinical data are maxx represents the maximum value of the feature corresponding to the clinical data, minx The minimum value of the feature corresponding to the clinical data; for the gene feature Normalize , mapping it to the range [0,1], where Represents the normalized i Gene mutation characteristics, Indicates the minimum value of gene mutation characteristics, Represents the maximum value of the gene mutation feature; then the imaging features, clinical features and gene features are fused to obtain multimodal input data .

3. The method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model according to claim 2, characterized in that: Step S2 specifically includes: constructing a multimodal large model, the multimodal large model includes an image modality encoder, a clinical modality encoder, a genetic modality encoder, a modality fusion module and a fully connected classification layer, the image modality encoder uses a visual Transformer to process image features and generate an embedded representation ; Clinical modality encoder uses multi-layer perceptron to process clinical features ; Genetic modality encoder models genetic features through deep neural networks ; The modality fusion module uses the self-attention mechanism Integrate to obtain multimodal features, where the attention weight calculation formula is: , where Q represents the query matrix, K represents the key matrix, and V represents the value matrix. represents the transpose of K, Represents the dimension of the key matrix; multimodal features Input to the fully connected classification layer and output the patient's metastasis risk prediction probability , where W is the weight matrix and b is the bias.

4. The method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model according to claim 3, characterized in that: Step S3 specifically includes: by minimizing the cross entropy loss function Train the multimodal large model, where N is the total number of samples, is the real transfer label, is the predicted probability; the performance of the multimodal large model is evaluated by the AUC value , where TPR and FPR are the true positive rate and false positive rate, respectively.

5. The method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model according to claim 4, characterized in that: Step S4 specifically includes: , patients are divided into low, medium and high risk groups, and a dynamic follow-up plan is generated; the follow-up plan is dynamically optimized based on time series data ,in represents the change in the patient's condition at time t, represents the follow-up plan at time t+1, represents the follow-up plan at time t.

6. The method for predicting liver and lung metastasis of colorectal cancer based on a multimodal large model according to claim 5, characterized in that: Step S5 specifically includes: Calculate the feature contribution weight, where Factors are calculated for gradient attribution, and corresponding treatment recommendations are generated based on risk assessment results and feature contribution weights. An explanatory report is also generated, including feature importance analysis and risk prediction basis.

7. A colorectal cancer liver and lung metastasis prediction system based on a multimodal large model, used in a colorectal cancer liver and lung metastasis prediction method based on a multimodal large model according to any one of claims 1 to 6, characterized in that: include: A data acquisition module, which is used to collect multimodal data of colorectal cancer patients, including imaging data, clinical data, and genetic data; A data processing module, used for preprocessing multimodal data collected from patients with colorectal cancer; A multimodal large model building module, used to build a multimodal large model; Model training and evaluation module, used to train the model and evaluate the model performance; Dynamic follow-up module, used to perform risk stratification and generate dynamic follow-up plans; Functional module for generating corresponding treatment recommendations and corresponding explanatory reports; The model verification module is used to verify the performance of the constructed multimodal large model.

8. The colorectal cancer liver and lung metastasis prediction system based on a multimodal large model according to claim 7, characterized in that: The multimodal large model includes: Image modality encoder, used to process image features; clinical modality coders, which process clinical features; Gene modality encoder, used to process gene features; The modality fusion module is used to fuse imaging features, clinical features, and genetic features through a self-attention mechanism to obtain multimodal features; Classification layer: used to process the input multimodal features and output the patient's colorectal cancer metastasis risk probability.

Images