Colorectal cancer metastasis prediction method and system and storage medium

Abstract

The invention discloses a colorectal cancer metastasis prediction method and system and a storage medium, and belongs to the technical field of medical data analysis. The method comprises the following steps: S1, preprocessing multi-modal data; s2, performing training and fine tuning on the prediction model by using the preprocessed multi-modal data; s3, performing modal fusion and external knowledge query; s4, inputting a to-be-predicted sample into the classifier to obtain a transfer prediction result, and outputting description information of the transfer prediction result by using a language encoder; s5, performing evaluation through an expert system according to professional knowledge to obtain a feedback value; when a new feedback value arrives, the prediction model carries out parameter updating through a reinforcement learning algorithm, and a prediction strategy is adjusted; model parameters are updated through a feedback value and a Bellman equation, and are gradually adjusted through multiple iterations. According to the invention, by fusing the medical image data and the electronic health record, the image information, the text information and the clinical data of the patient are effectively integrated, and the prediction precision and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data analysis, and in particular to a method, system and storage medium for predicting colorectal cancer metastasis. Background Art

[0002] Colorectal cancer (CRC) is one of the malignant tumors with high morbidity and mortality rates worldwide. With the advancement of medical research and diagnosis and treatment technology, the early diagnosis and treatment of colorectal cancer have been significantly improved, but its metastasis is still an important cause of high mortality. The metastasis of colorectal cancer usually first appears in the liver, lungs and other distant organs, and the prediction and early detection of the metastasis process are crucial to improving treatment efficacy and survival rate.

[0003] Imaging examination is one of the most common methods for diagnosing and predicting metastasis of colorectal cancer. With the help of CT, MRI, etc., it can provide imaging data about the size, location and surrounding tissues of the tumor. However, relying solely on imaging examinations for metastasis prediction faces some challenges. Although imaging data can reveal the spatial distribution information of colorectal cancer, it is relatively limited in evaluating the tumor microenvironment, molecular characteristics, etc. In addition, imaging data is often affected by factors such as examination technology and operator experience, resulting in certain limitations in its accuracy in metastasis prediction.

[0004] Electronic Health Records (EHR), as an important part of digital medical data, contains a large amount of clinical data of patients, including historical medical records, examination results, treatment records, drug use, etc. EHR plays an important role in providing doctors with comprehensive health information of patients and supporting clinical decision-making. In recent years, with the development of machine learning and data mining technology, EHR has shown great potential in disease prediction and treatment effect evaluation.

[0005] Most existing colorectal cancer metastasis prediction schemes rely only on medical images, such as CT and MRI, but rarely effectively integrate EHR data with imaging groups to improve the accuracy of colorectal cancer metastasis prediction. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a method, system and storage medium for predicting colorectal cancer metastasis.

[0007] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: a method for predicting colorectal cancer metastasis, comprising the following steps: S1: Preprocess multimodal data; S2: Use the preprocessed multimodal data to train and fine-tune the prediction model; S3: Perform modality fusion and external knowledge query; S4: inputting the sample to be predicted into the classifier to obtain a transfer prediction result, and using a language encoder to output description information of the transfer prediction result; S5: For each prediction sample and its corresponding transfer prediction result, an expert system evaluates it based on professional knowledge to obtain a feedback value; whenever a new feedback value arrives, the prediction model updates its parameters through a reinforcement learning algorithm and adjusts the prediction strategy; the model parameters are updated through the feedback value and the Bellman equation, and are gradually adjusted through multiple iterations.

[0008] Preferably, the multimodal data includes medical imaging data , electronic health record data and colorectal cancer knowledge graph data , the S1 further comprises the following steps: For medical imaging data , remove unclear or abnormal image samples, and perform uniform resizing and normalization on the image samples. x i For stored medical images, For medical image description, p i The value is 0 or 1, indicating whether the patient has cancer metastasis. info i is the image description information, EHR i Index information for electronic health records corresponding to medical images; Electronic health record data Includes patient information and corresponding medical information R ={ r 1 , r 2 , …, r |R|}, each visit r i contains various types of medical codes, which connect patient information to EHR i Establish a one-to-one index relationship; Colorectal cancer knowledge graph data ,Through the construction of the colorectal cancer medical literature database and the ,medical encyclopedia information database, the entities or concepts related to colorectal cancer and ,their relationships are saved as an external knowledge base, in which V is a set of nodes in the knowledge graph, E is the set of edges between nodes.

[0009] Preferably, the prediction model is a large multimodal model based on the Transformer architecture, including a language modality encoder, an image encoder, a knowledge graph encoder and a modality fusion module; S2 includes the following steps: For the language modality encoder and image encoder, the pre-training-fine-tuning paradigm is adopted. The pre-trained model is used to perform fine-tuning training on the pre-processed multi-modal data. The fine-tuning loss function adopts the cross entropy loss function: ,in Predict probabilities for the model.

[0010] Preferably, the S3 comprises the following steps: During training and querying, for samples , the electronic health record data corresponding to the text information index is used to form the text modality, and the text modality encoding is obtained through the language modality encoder emb s , medical images x i Input the image encoder to get the image code emb I , a unified encoding representation is obtained through the modal fusion module: ; For external knowledge graph libraries, for each node , calculate the query code emb pre With Node v j The embedding vector emb vj The similarity measure between: ; Select the set of nodes most relevant to the query using a similarity measure ,in k is the number of related nodes; using the graph neural network GNN, the adjacency information of the nodes in the knowledge graph is aggregated through the message passing mechanism, and for each query-related node, the information of the node is transmitted through the graph neural network GNN; suppose the nodes in the graph neural network GNN v j The embedding vector of , after t iterations, it is: , in Is a node v j The set of adjacent nodes of is the learnable weight matrix for the tth iteration, It is a slave node vj To adjacent nodes v k The transfer coefficient, σ is a nonlinear activation function; through multiple iterations of nodes v j Representation Gradually include more neighbor node information, and finally add it to the original query code emb pre Perform secondary fusion to generate the final retrieval results: .

[0011] The second aspect of the present invention provides: a colorectal cancer metastasis prediction system, used to implement any of the above-mentioned colorectal cancer metastasis prediction methods, comprising: A multimodal data preprocessing module, used for preprocessing multimodal data; Model training and fine-tuning module, used to train and fine-tune the prediction model using preprocessed multimodal data; Modality fusion and external knowledge query module, used for modality fusion and external knowledge query; A transfer prediction module is used to input the sample to be predicted into the classifier to obtain a transfer prediction result, and at the same time use a language encoder to output description information of the transfer prediction result; The feedback reinforcement learning module is used to evaluate each prediction sample and its corresponding transfer prediction result according to professional knowledge through the expert system to obtain a feedback value; whenever a new feedback value arrives, the prediction model updates the parameters through the reinforcement learning algorithm and adjusts the prediction strategy; the model parameters are updated through the feedback value and the Bellman equation, and are gradually adjusted through multiple iterations.

[0012] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned colorectal cancer metastasis prediction methods is implemented.

[0013] The beneficial effects of the present invention are: 1) Multimodal data fusion improves metastasis prediction accuracy: This invention effectively integrates image information, text information and patient clinical data by fusing medical imaging data with electronic health records (EHR). Medical imaging data provides the morphological characteristics of tumors, while EHR data includes key information such as the patient's medical history and treatment records. This cross-modal data fusion greatly enhances the model's understanding of colorectal cancer metastasis and improves the accuracy and robustness of prediction.

[0014] 2) External knowledge graph enhances reasoning ability: The present invention uses the external knowledge graph in the field of colorectal cancer to dynamically aggregate colorectal cancer-related knowledge through Graph Transformer Networks (GTN). Through the embedding vector similarity measurement, the present invention can effectively identify and retrieve medical knowledge related to the current case, and assist the model in making more accurate metastasis predictions. The introduction of this external knowledge breaks through the limitations of the traditional single data source and significantly enhances the medical reasoning ability of the model.

[0015] 3) Expert feedback and reinforcement learning mechanism to optimize model performance: This invention introduces an expert feedback mechanism, which allows medical experts to evaluate the model's prediction results and provide feedback, and the system uses reinforcement learning algorithms to dynamically adjust the model. This mechanism enables the model to gradually learn the knowledge and experience of experts, thereby continuously optimizing the prediction strategy and adapting to the characteristics of different patients, ultimately improving the accuracy and personalization of the prediction results; in the self-collected data set of colorectal cancer metastasis, compared with traditional solutions, the accuracy rate increased from 63.82% to 81.93%. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart of the method for predicting colorectal cancer metastasis. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 The first aspect of the present invention provides: a method for predicting colorectal cancer metastasis, comprising the following steps: S1: Preprocess multimodal data; S2: Use the preprocessed multimodal data to train and fine-tune the prediction model; S3: Perform modality fusion and external knowledge query; S4: inputting the sample to be predicted into the classifier to obtain a transfer prediction result, and using a language encoder to output description information of the transfer prediction result; S5: For each prediction sample and its corresponding transfer prediction result, an expert system evaluates it based on professional knowledge to obtain a feedback value; whenever a new feedback value arrives, the prediction model updates its parameters through a reinforcement learning algorithm and adjusts the prediction strategy; the model parameters are updated through the feedback value and the Bellman equation, and are gradually adjusted through multiple iterations.

[0019] In this embodiment, the accuracy of colorectal cancer metastasis prediction is improved by integrating and applying electronic health records and medical imaging group information. Model parameters are updated through the Bellman equation using feedback values. Through multiple iterations, the model can gradually adjust its parameters so that when faced with new patient cases, the prediction results can be as accurate as possible. The present invention has a dynamic external knowledge retrieval mechanism to improve the efficiency of knowledge application. The system automatically determines whether external knowledge needs to be retrieved based on the output of the current model and the credibility of the prediction results, and adjusts the call to the most relevant knowledge base based on the accuracy of the feedback. This mechanism avoids the inefficiency of static knowledge base calls in traditional methods, can improve the accuracy and response speed of knowledge retrieval in real time, and ensures that the most relevant auxiliary information can be obtained at each prediction stage.

[0020] The system architecture design of the present invention is extremely flexible and scalable. In addition to combining EHR and imaging data, the system can also flexibly introduce other modality data (such as genomic data, real-time monitoring data, etc.) according to needs, and supports optimizing model performance through fine-tuning strategies. The system can adapt to different medical scenarios and needs, and can be expanded and maintained in the long term with the development of data and technology.

[0021] In some embodiments, the multimodal data includes medical imaging data , electronic health record data and colorectal cancer knowledge graph data , the S1 further comprises the following steps: For medical imaging data , remove unclear or abnormal image samples, and perform uniform resizing and normalization on the image samples. x i For stored medical images, For medical image description, p i The value is 0 or 1, indicating whether the patient has cancer metastasis. info i is the image description information, EHR i Index information for electronic health records corresponding to medical images; Electronic health record data Includes patient information and corresponding medical information R ={ r 1 , r 2 , …, r |R|}, each visit r i contains various types of medical codes, which connect patient information toEHR i Establish a one-to-one index relationship; Colorectal cancer knowledge graph data ,Through the construction of the colorectal cancer medical literature database and the ,medical encyclopedia information database, the entities or concepts related to colorectal cancer and ,their relationships are saved as an external knowledge base, in which V is a set of nodes in the knowledge graph, E is the set of edges between nodes.

[0022] In this embodiment, the medical code includes the patient's medical information such as diseases, drugs and procedures. In order to protect the privacy of patient information, the index information is in a one-to-one correspondence with the patient information. The database system does not store the patient information, but only retains the retrieval function.

[0023] In some embodiments, the prediction model is a large multimodal model based on the Transformer architecture, including a language modality encoder, an image encoder, a knowledge graph encoder, and a modality fusion module; S2 includes the following steps: For the language modality encoder and image encoder, the pre-training-fine-tuning paradigm is adopted. The pre-trained model is used to perform fine-tuning training on the pre-processed multi-modal data. The fine-tuning loss function adopts the cross entropy loss function: ,in Predict probabilities for the model.

[0024] In some embodiments, the S3 comprises the following steps: During training and querying, for samples , the electronic health record data corresponding to the text information index is used to form the text modality, and the text modality encoding is obtained through the language modality encoder emb s , medical images x i Input the image encoder to get the image encoding emb I , a unified encoding representation is obtained through the modal fusion module: ; For external knowledge graph libraries, for each node , calculate the query code emb pre With Node v j The embedding vector emb vj The similarity measure between: ; Select the set of nodes most relevant to the query using a similarity measure ,ink is the number of related nodes; using the graph neural network GNN, the adjacency information of the nodes in the knowledge graph is aggregated through the message passing mechanism, and for each query-related node, the information of the node is transmitted through the graph neural network GNN; suppose the nodes in the graph neural network GNN v j The embedding vector of , after t iterations, it is: , in Is a node v j The set of adjacent nodes of is the learnable weight matrix for the tth iteration, It is a slave node v j To adjacent nodes v k The transfer coefficient, σ is a nonlinear activation function; through multiple iterations of nodes v j Representation Gradually include more neighbor node information, and finally add it to the original query code emb pre Perform secondary fusion to generate the final retrieval results: .

[0025] In this embodiment, in order to obtain richer information related to the query, the graph neural network GNN is used to aggregate the adjacency information of nodes in the knowledge graph through a message passing mechanism.

[0026] The second aspect of the present invention provides: a colorectal cancer metastasis prediction system, used to implement any of the above-mentioned colorectal cancer metastasis prediction methods, comprising: A multimodal data preprocessing module, used for preprocessing multimodal data; Model training and fine-tuning module, used to train and fine-tune the prediction model using preprocessed multimodal data; Modality fusion and external knowledge query module, used for modality fusion and external knowledge query; A transfer prediction module is used to input the sample to be predicted into the classifier to obtain a transfer prediction result, and at the same time use a language encoder to output description information of the transfer prediction result; The feedback reinforcement learning module is used to evaluate each prediction sample and its corresponding transfer prediction result according to professional knowledge through the expert system to obtain a feedback value; whenever a new feedback value arrives, the prediction model updates the parameters through the reinforcement learning algorithm and adjusts the prediction strategy; the model parameters are updated through the feedback value and the Bellman equation, and are gradually adjusted through multiple iterations.

[0027] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned colorectal cancer metastasis prediction methods is implemented.

[0028] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for predicting colorectal cancer metastasis, characterized in that: The following steps are involved: S1: Preprocess multimodal data; S2: Use the preprocessed multimodal data to train and fine-tune the prediction model; S3: Perform modality fusion and external knowledge query; S4: inputting the sample to be predicted into the classifier to obtain a transfer prediction result, and using a language encoder to output description information of the transfer prediction result; S5: For each prediction sample and its corresponding transfer prediction result, an expert system evaluates it based on professional knowledge to obtain a feedback value. Whenever a new feedback value arrives, the prediction model updates its parameters through a reinforcement learning algorithm and adjusts the prediction strategy. The model parameters are updated through feedback values ​​and the Bellman equation and are gradually adjusted through multiple iterations.

2. The method for predicting colorectal cancer metastasis according to claim 1, characterized in that: The multimodal data includes medical imaging data , electronic health record data and colorectal cancer knowledge graph data , the S1 further comprises the following steps: For medical imaging data , remove unclear or abnormal image samples, and perform uniform resizing and normalization on the image samples. x i For stored medical images, For medical image description, p i The value is 0 or 1, indicating whether the patient has cancer metastasis. info i is the image description information, EHR i Index information for electronic health records corresponding to medical images; Electronic health record data Includes patient information and corresponding medical information R ={ r 1 , r 2 , …, r |R| }, each visit r i contains various types of medical codes, which connect patient information to EHR i Establish a one-to-one index relationship; Colorectal cancer knowledge graph data ,Through the construction of the colorectal cancer medical literature database and the ,medical encyclopedia information database, the entities or concepts related to colorectal ,cancer and their mutual relationships are saved as an external knowledge ,base, where V is a set of nodes in the knowledge graph, E is the set of edges between nodes.

3. The method for predicting colorectal cancer metastasis according to claim 2, characterized in that: The prediction model is a large multimodal model based on the Transformer architecture, including a language modality encoder, an image encoder, a knowledge graph encoder and a modality fusion module; S2 includes the following steps: For the language modality encoder and image encoder, the pre-training-fine-tuning paradigm is adopted. The pre-trained model is used to perform fine-tuning training on the pre-processed multi-modal data. The fine-tuning loss function adopts the cross entropy loss function: ,in Predict probabilities for the model.

4. The method for predicting colorectal cancer metastasis according to claim 3, characterized in that: The S3 comprises the following steps: During training and querying, for samples , the electronic health record data corresponding to the text information index is used to form the text modality, and the text modality encoding is obtained through the language modality encoder emb s , medical images x i Input the image encoder to get the image encoding emb I , a unified encoding representation is obtained through the modal fusion module: ; For external knowledge graph libraries, for each node , calculate the query code emb pre With Node v j The embedding vector emb vj The similarity measure between: ; Select the set of nodes most relevant to the query using a similarity measure ,in k is the number of related nodes; using the graph neural network GNN, the adjacency information of the nodes in the knowledge graph is aggregated through the message passing mechanism, and for each query-related node, the information of the node is transmitted through the graph neural network GNN; suppose the nodes in the graph neural network GNN v j The embedding vector of , after t iterations, it is: ， in Is a node v j The set of adjacent nodes of is the learnable weight matrix for the tth iteration, It is a slave node v j To adjacent nodes v k The transfer coefficient, σ is a nonlinear activation function; through multiple iterations of nodes v j Representation Gradually include more neighbor node information, and finally add it to the original query code emb pre Perform secondary fusion to generate the final retrieval results: .

5. A colorectal cancer metastasis prediction system, characterized in that: Used to implement the method for predicting colorectal cancer metastasis according to any one of claims 1 to 4, comprising: A multimodal data preprocessing module, used for preprocessing multimodal data; Model training and fine-tuning module, used to train and fine-tune the prediction model using preprocessed multimodal data; Modality fusion and external knowledge query module, used for modality fusion and external knowledge query; A transfer prediction module is used to input the sample to be predicted into the classifier to obtain a transfer prediction result, and at the same time use a language encoder to output description information of the transfer prediction result; The feedback reinforcement learning module is used to evaluate each prediction sample and its corresponding transfer prediction result according to professional knowledge through the expert system to obtain a feedback value; whenever a new feedback value arrives, the prediction model updates the parameters through the reinforcement learning algorithm and adjusts the prediction strategy; the model parameters are updated through the feedback value and the Bellman equation, and are gradually adjusted through multiple iterations.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the colorectal cancer metastasis prediction method as described in any one of claims 1-4 is implemented.

Images