Intelligent decision-making method and device based on knowledge retrieval multi-modal large model

Abstract

The invention discloses an intelligent decision-making method and device based on a knowledge retrieval multi-modal large model, and belongs to the technical field of artificial intelligence and medical data processing.The method comprises the steps that multi-modal diagnosis and treatment data of a patient are obtained and preprocessed, and corresponding feature vectors are generated; performing enhancement and cross-modal feature vector fusion on the feature vector of each modal based on an enhancement fusion model to obtain a patient personalized fusion feature vector; and performing knowledge retrieval in a tumor treatment whole-process electronic record database based on the personalized fusion feature vector, obtaining a plurality of treatment schemes under similar clinical manifestations, inputting the retrieved treatment schemes and the personalized fusion feature vector of the patient into a large language model for decision making, obtaining a personalized treatment scheme of the patient, and evaluating the risk of the personalized treatment scheme. A doctor is assisted to give an actual treatment scheme; and if the multi-mode diagnosis and treatment data of the patient is updated along with the actual treatment process, updating the personalized treatment scheme and the risk thereof so as to provide real-time dynamic assistance for a doctor.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and medical data processing, and in particular relates to an intelligent decision-making method and device based on a knowledge retrieval multimodal large model. Background Art

[0002] When making decisions on tumor treatment, doctors usually consider a variety of diagnostic and treatment data such as the patient's clinical manifestations, pathology reports, and imaging data, and refer to the latest clinical guidelines and expert consensus to develop treatment plans. However, in clinical scenarios, the same indication is often recommended by clinical guidelines for multiple standard treatment options. The lack of precise targeting indicators is a serious problem facing tumor treatment decisions. The unpredictable mutation patterns and variable gene expression of tumor cells make tumors highly complex and heterogeneous in the human body. The treatment effects of different patients with the same type of tumor under the same treatment method also vary greatly. The patient's drug metabolism ability, the tumor microenvironment in the body, and the immune system response will affect the treatment effect. These have brought huge challenges to precision treatment. Therefore, how to integrate multimodal clinical data on the basis of following the guidelines, accurately predict the patient's tumor treatment response under different treatment options, and flexibly adjust to study the most suitable personalized tumor treatment model for patients is crucial to improving treatment effects and improving patient prognosis.

[0003] The diagnosis and treatment of tumors is a complex and multi-dimensional decision-making task. Single-modal data is difficult to fully reflect the characteristics of tumors, especially in the case of early tumors or small lesions, the model effect is not ideal or difficult to predict. Multimodal models have the ability to process heterogeneous and unstructured data, and are suitable for discovering nonlinear and high-dimensional relationships in multimodal data. Using multimodal models to integrate and analyze various types of diagnosis and treatment data such as clinical medical images, genomic data, and inspections and tests for joint decision-making can obtain more accurate and reliable auxiliary decision support. With the introduction of language big models, multimodal big model technology is becoming more and more mature. Multimodal big models automatically capture the deep correlation between each modality through self-attention mechanism and Transformer structure in large-scale pre-training data, and discover the potential correspondence between modalities. The current medical big model is based on the natural language big model and the natural image big model, and uses a large number of public diagnosis and treatment databases for full fine-tuning, aiming to build a general medical question-answering knowledge system or medical image interpretation architecture. Integrating multimodal large models into the tumor treatment decision-making process, imitating the decision-making ideas of clinicians, integrating multi-modal data including medical images, imaging reports, test results, electronic medical records, etc., and analyzing the complex patterns of tumor progression can effectively improve the accuracy of tumor treatment plans.

[0004] However, existing technologies based on multimodal diagnostic and treatment data to assist tumor diagnosis and treatment still have certain problems.

[0005] For tumor treatment response prediction models based on traditional multimodal learning, the correlation between modalities is ignored when making multimodal fusion decisions, the complementarity between clinical data and the complex development environment of tumors are not fully explored, and the information utilization rate is low.

[0006] When applying multimodal large models to medical practice, we face the following problems:

[0007] (1) Insufficient accuracy in specific tasks. Medical big models can use semantic information interpretation and image understanding technology to learn a large number of medical terms and image representation constructions, and are widely applicable to multiple target tasks, such as disease prediction, medication recommendations, image analysis, etc. based on patient complaints and examination results. However, the results given are often text answers formed by comprehensive analysis of the terminology library, which only has a popular science effect and does not introduce active decision-making. In specific tasks such as tumor treatment decision-making, the accuracy is far from enough and cannot provide effective decision-making guidance to clinicians. The side effects of tumor treatment will have a serious impact on the patient's life. How to balance the effect of tumor treatment and the prognosis of quality of life is also an issue that needs to be considered in tumor treatment decision-making.

[0008] (2) Low personalization of decision making. Large models learn general feature expressions based on large-scale pre-training data to ensure the strong generalization ability of the model. However, tumors have highly heterogeneous expressions both within and between individuals. Patients with similar images and similar examination results may also have inconsistent adaptation plans due to differences in immune response or genome. Large models tend to ignore the individualized expression of patients and find it difficult to provide personalized decision plans.

[0009] (3) Insufficient capture of temporal progression features. Limited by training data, existing large models are not able to perceive clinical temporal data effectively, and often analyze static data based on given data to produce inference results. However, tumors are in a state of development and evolution in the patient's body, and their characteristics at different time points may be quite different. For example, the early imaging features of tumors are obvious, while the late imaging features of tumors are obvious. In addition, the dynamic changes in the tumor microenvironment, the generation of drug resistance, and the changes in immune response are difficult to capture with static data models, which will affect the final decision-making results. Summary of the invention

[0010] In view of the problems existing in the prior art, the purpose of the embodiments of the present application is to provide an intelligent decision-making method and device based on a knowledge retrieval multimodal large model.

[0011] According to a first aspect of an embodiment of the present application, there is provided an intelligent decision-making method based on a knowledge retrieval multimodal large model, comprising:

[0012] Acquiring multimodal diagnosis and treatment data of the patient, preprocessing the multimodal diagnosis and treatment data and generating corresponding feature vectors;

[0013] Based on the enhanced fusion model, the feature vectors of each modality are enhanced and the cross-modality feature vectors are fused to obtain the patient's personalized fusion feature vector;

[0014] Based on the patient's personalized fusion feature vector, knowledge retrieval is performed in the electronic record database of the entire tumor treatment process to obtain several treatment plans under similar clinical manifestations, and the retrieved treatment plans are used as prompt information and input into the large language model together with the patient's personalized fusion feature vector for decision-making to obtain the patient's personalized treatment plan, and the risk of the personalized treatment plan for the patient is evaluated, wherein the personalized treatment plan and its risk are used to assist doctors in giving actual treatment plans;

[0015] If the patient's multimodal diagnosis and treatment data is updated with the actual treatment progress, the personalized treatment plan and its risks are updated, thereby providing real-time dynamic assistance to doctors.

[0016] Furthermore, the multimodal diagnosis and treatment data includes basic patient information, consultation records, test results, laboratory test results, nursing records, genomic data, imaging data and corresponding imaging reports, wherein the consultation records and imaging reports are text data, and the basic patient information, test results, laboratory test results and nursing records are structured data; preprocessing the multimodal diagnosis and treatment data and generating feature vectors include:

[0017] Remove noise from text data by removing low-frequency words and term mapping, remove noise from structured data by detecting outliers and smoothing data; remove noise from image data by using image enhancement or filtering operations;

[0018] By unifying the scale units of text data and structured data and standardizing the grayscale values ​​of image data, the scale of multimodal diagnosis and treatment data can be normalized;

[0019] For the denoised and scale-normalized multimodal diagnosis and treatment data, the text feature vector is obtained through the pre-trained language model, the structured feature vector is generated by the fully connected layer or MLP, the genomic feature vector is generated by the graph neural network or the bioinformatics large model, the image data is segmented and feature extracted to obtain the image graph embedding vector, and the image word embedding vector is obtained based on the image segmentation result.

[0020] Furthermore, the enhanced fusion model includes a single-modal feature enhancement module and a cross-modal feature fusion module;

[0021] In the unimodal feature enhancement module, the text feature vector and the structured feature vector are enhanced by using the self-attention mechanism, and the image embedding vector is enhanced by using the deformable self-attention mechanism;

[0022] The cross-modal feature fusion module enhances the enhanced feature vectors of each modality through a self-attention mechanism. and the mode B eigenvector , cross-modal fusion is performed through the following formula:

[0023]

[0024] in is the fusion feature vector of mode A-mode B, is the length of each mode eigenvector.

[0025] Furthermore, the cross-modal feature fusion module is trained by the following comprehensive loss function:

[0026]

[0027]

[0028]

[0029] in and is the weight coefficient, Express expectations, is a set of data pairs, is the similarity of data pairs, is the true label of whether the data pair is similar, is the data pair matching degree, is the true label of whether the data pair matches.

[0030] Furthermore, the electronic record database of the entire process of tumor treatment is a vector database, in which the stored data is updated regularly, including tumor-related medical knowledge, treatment guidelines, clinical cases, follow-up records, and medical literature. Various types of data are generated through feature extraction to generate embedded high-dimensional vector representations, which are stored in the electronic record database of the entire process of tumor treatment in the form of dense vectors.

[0031] Furthermore, the patient personalized fusion feature vector is used as a query vector for the database Each patient diagnosis and treatment data vector in , query vector Each patient diagnosis and treatment data vector in the database Calculate the similarity and return the treatment plans corresponding to the most relevant patient diagnosis and treatment data vectors, where the similarity .

[0032] Furthermore, a retrieval-enhanced generative model is used as a large language model for generating personalized treatment plans, wherein the retrieval-enhanced generative model has pre-learned tumor-related medical knowledge, treatment guidelines, follow-up records, and medical literature in the electronic record database of the entire tumor treatment process.

[0033] According to a second aspect of an embodiment of the present application, there is provided an intelligent decision-making device based on a knowledge retrieval multimodal large model, comprising:

[0034] A feature extraction module, used to obtain multimodal diagnosis and treatment data of a patient, preprocess the multimodal diagnosis and treatment data and generate corresponding feature vectors;

[0035] The feature enhancement and fusion module is used to enhance the feature vectors of each modality and fuse the feature vectors across modalities based on the enhanced fusion model to obtain the patient's personalized fusion feature vector;

[0036] A personalized plan generation module is used to perform knowledge retrieval in the electronic record database of the entire process of tumor treatment based on the patient's personalized fusion feature vector, obtain several treatment plans under similar clinical manifestations, use the retrieved treatment plans as prompt information, and input them together with the patient's personalized fusion feature vector into a large language model for decision-making, obtain the patient's personalized treatment plan, and evaluate the risk of the personalized treatment plan for the patient, wherein the personalized treatment plan and its risk are used to assist doctors in giving actual treatment plans;

[0037] The plan update module is used to update the personalized treatment plan and its risks if the patient's multimodal diagnosis and treatment data is updated with the actual treatment progress, thereby providing real-time dynamic assistance to doctors.

[0038] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0039] one or more processors;

[0040] A memory for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0042] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0043] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0044] It can be seen from the above embodiments that the present application simulates the doctor's thinking when making clinical tumor treatment decisions, incorporates the multi-modal diagnosis and treatment data generated during the diagnosis and treatment process into the intelligent decision-making process, and provides clinicians with more accurate treatment decision assistance: based on a pre-constructed electronic record database of the entire process of tumor treatment, by retrieving the most similar cases from the database, providing evidence-based clinical decision support in the model of evidence-based medicine, and increasing the interpretability of the results; using the retrieved evidence support as enhanced prompt information, combined with patient-specific examination results to make data-driven decisions for the patient, can effectively improve the personalization and accuracy of the treatment plan, and feed back the newly generated data of the patient's clinical response under the treatment plan to the system, so as to dynamically and continuously optimize the treatment plan.

[0045] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of an intelligent decision-making method based on a knowledge retrieval multimodal large model according to the present invention.

[0048] Figure 2 Flowchart for the generation of personalized fusion features for patients of the present invention.

[0049] Figure 3 The flowchart of the cross-modal feature enhancement fusion of the present invention.

[0050] Figure 4 The present invention is a block diagram of an intelligent decision-making device based on a knowledge retrieval multimodal large model.

[0051] Figure 5 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] This application provides an intelligent decision-making method based on a knowledge retrieval multimodal large model, such as Figure 1 As shown, the main purpose of this method is to use the multimodal diagnosis and treatment data of the entire clinical diagnosis and treatment process of tumor patients, including but not limited to basic information of patients, consultation records, test results, laboratory results, nursing records, genomic data, imaging data and corresponding imaging reports, etc., to generate personalized treatment plans through large model analysis and retrieval, and to prompt the prognostic risk assessment results of patients under different plans, so as to assist doctors in making efficient and accurate treatment decisions. The method includes the following steps:

[0056] (1) obtaining multimodal diagnosis and treatment data of a patient, preprocessing the multimodal diagnosis and treatment data and generating corresponding feature vectors;

[0057] Specifically, this step preprocesses the diagnosis and treatment data of patients from different sources and of different types, such as Figure 2 As shown in the figure, the features are extracted by the language model and the image model to obtain the standardized feature vectors of each modality. The diagnosis and treatment data involved include but are not limited to basic patient information (height, weight, medical history, family history, etc.), consultation records (patient self-reported symptoms, clinical manifestations, preliminary diagnosis results, etc.), test results (electrocardiogram, blood pressure, etc.), laboratory results (blood indicators, urine indicators, tumor markers, pathological results, etc.), nursing records (medication records, medical records, doctor's orders, etc.), genomic data (gene mutation information of tumor tissue, gene expression spectrum, etc.), imaging data (CT, ultrasound, MRI, pathology, etc. original images) and their corresponding imaging reports.

[0058] (1.1) Data preprocessing.

[0059] In order to avoid the lack of accuracy in subsequent matching due to differences in dimension scale, data quality, etc. between data from different sources (different hospitals, different recorders, different scanning instruments, different electronic medical record systems, etc.), medical data needs to be preprocessed to ensure data compatibility. The preprocessing mainly includes denoising and scale normalization.

[0060] Medical data can generate noise due to recording errors and operational errors, such as irrelevant words in medical reports, a large number of artifacts in images, abnormal single values ​​in blood tests, etc. The disturbances caused by these noises can sometimes have a catastrophic impact on the model, so all types of data need to be denoised. Specifically, the noise in text data (including medical records and imaging reports) can be avoided by removing low-frequency words and mapping terms; the noise in structured data (including basic patient information, test results, laboratory results, and nursing records) can be corrected through outlier detection, data smoothing and other technologies; the noise in image data can be removed using image enhancement technology or filtering operations.

[0061] The data record scales of different hospitals may be inconsistent. For example, blood draw results may have two measurement units, ng / ml and nmol / L. CT image values ​​range from [-1000HU, 1000HU] depending on the shooting site, while the effective range of tissues is only within a very small window, such as [-140HU, 100HU] for abdominal tissues. In order to avoid the deviation of learning information due to insufficient medical knowledge of the large model, the scale of the diagnosis and treatment data is normalized before feature extraction. Specifically, text data and structured data can be unified in scale units through term mapping, unit mapping, etc.; image data is uniformly standardized in grayscale values ​​and mapped to the range of (0-255) using the Min-Max normalization method.

[0062] The genomic data is obtained through accurate sequencing, and its error problem is not considered in this embodiment, and no preprocessing is performed.

[0063] (1.2) Feature vector generation

[0064] Specifically, feature extraction is performed on the multimodal diagnosis and treatment data preprocessed in step (1.1), so as to facilitate subsequent feature integration analysis through a multimodal large model, thereby obtaining a patient-specific feature vector.

[0065] For text data, the system calls pre-trained language models (BERT, GPT and related variant models, etc.) to learn rich contextual information such as medical records and imaging reports, and converts them into high-dimensional word embedding encoding, which is recorded as a text feature vector. .

[0066] For structured data, use the fully connected layer or MLP to extract features and generate structured feature vectors .

[0067] Genomic data uses graph neural networks (GNN) or bioinformatics models (such as BioBERT, etc.) to mine patient tumor-related gene mutations and generate embedded vector representations carrying mutant gene node information, recorded as genomic feature vectors , which facilitates subsequent analysis of information such as tumor characteristics and patient drug resistance.

[0068] The image data uses the Vision Transformer structure to obtain the tumor's substantial segmentation results and image embedding representation, denoted as , extract information including tumor volume, morphology, boundary density, etc. from the mask results of the substantial segmentation using SimpleITK, PyRadiomics algorithm library or 3D Slicer medical image processing software, and convert it into word embedding form using word embedding models such as Word2Vec, BERT or FastText, recorded as image word embedding vector Since the image embedding representation cannot be directly understood by the language model, it is necessary to align the image embedding representation with the word embedding representation. A linear layer is added to the model to obtain the vectorized representation of the image embedding, which is recorded as the image embedding vector .

[0069] The information from medical data of different modalities is mapped to a high-dimensional feature space to generate a unified feature vector after independent feature extraction, which strengthens the features of each modality to ensure the extraction of lesion-related information. .

[0070]

[0071] (2) Based on the enhanced fusion model, the feature vectors of each modality are enhanced and the cross-modal feature vectors are fused to obtain the patient's personalized fused feature vector;

[0072] Specifically, for the medical data feature vectors of different modalities obtained in step (1.2), an enhanced fusion model is established to achieve effective fusion of cross-modal features to mine high-quality decision information. Figure 3As shown, in the enhanced fusion model, each modal data is first input into the single-modal feature enhancement module, and for each modal data, the self-attention mechanism is used to model its internal features to capture the local and global relationships in the modal data, thereby achieving information enhancement of single-modal medical features; on this basis, a cross-modal feature fusion module is used to model the correlation between different modal features, focusing on the complementary and interactive information between modalities, thereby achieving effective fusion of multi-modal medical features.

[0073] The self-attention part can ensure that the effective features in each modality are fully focused on and enhanced, and more accurately extract the detailed information of each modality, thereby improving the expressiveness of each modality feature and avoiding the loss and omission of information when multi-modal features are processed together. For text feature vectors, structured feature vectors, genomic feature vectors and image word embedding vectors, the self-attention mechanism is used in the unimodal feature enhancement module, hoping that the patient patterns and statistical information therein can be fully mined; for image embedding vectors, the unimodal feature enhancement module uses a deformable self-attention mechanism to dynamically adjust the sampling points or the size of the attention window, so that the system notices the changes in certain local features, such as the fine contours of the tumor.

[0074] For enhanced modal features, for example, the image embedding vector shows more morphological and texture features of the tumor, while the genomic features can provide molecular information related to the tumor phenotype. The cross-modal feature fusion module can calculate the correlation and interaction between different modalities, dynamically fuse their features according to the contextual information of each modality, and automatically align the complementary information between the modalities.

[0075] For modal The eigenvector corresponding to the data and modal The eigenvector corresponding to the data , and the corresponding feature vectors after self-attention mechanism enhancement are recorded as:

[0076]

[0077]

[0078] in is the vector to be fused and Length.

[0079] First, the feature vectors of the two modes are multiplied and then the interaction feature group is obtained. , that is, the fusion feature vector of mode A-mode B, recorded as:

[0080]

[0081] Then, for each feature pair in the interactive feature group, two loss functions are calculated, namely contrast loss and matching loss. Contrast loss is used to optimize the similarity and discrimination between samples, while matching loss is used to ensure that the representation of data in different modalities is as close as possible in the embedding space. Finally, a weighted combination of the two losses is calculated to balance the set alignment and semantic consistency between modalities, so that the cross-modal data representation is more consistent and robust, and effective cross-modal data fusion is achieved.

[0082] Contrastive loss is used to measure the similarity of cross-modal data and ensure that different modal representations of the same type of samples are close. For example, if a patient has a benign gallbladder mass and a malignant liver tumor, the image data corresponding to the benign mass and its corresponding text data will be mapped to a shared feature space as much as possible, while the image data corresponding to the benign mass and the text data corresponding to the malignant tumor will be mapped to feature representations that are farther away. Contrastive loss ensures the spatial similarity between different modal information and ensures that the fused features can more comprehensively reflect the patient's various information. It lays the foundation for feature matching in subsequent matching loss. Contrastive loss can be expressed as:

[0083]

[0084] in express expectations; is the similarity of data pairs, usually using Euclidean distance or cosine distance; is the true label of whether the data pair is similar. , when not similar .

[0085] Matching loss is used to measure the Whether there is a relevant matching relationship or a mismatch relationship between them, for example, in the matching of text data and image data, a diagnosis of "there is a solid shadow in the upper part of the liver" shows a matching relationship with the texture and grayscale features of this part in the CT image, while "the patient has a long history of smoking" has a lower matching relationship with the spleen cyst in the patient's abdominal B-ultrasound image. It is expressed as:

[0086]

[0087] in is the matching degree of the data pair, obtained through the matching module. The matching module can be composed of a simple binary classification network or a fully connected layer with a sigmoid function. is the true label of whether the data pair matches. , when it does not match .

[0088] Comprehensive loss function for cross-modal attention It is expressed as:

[0089]

[0090] in and is the weight coefficient, which is used to balance the importance of the two losses in the model.

[0091] Based on the comprehensive loss function The cross-modal feature fusion module is trained so that it can explore the correlation between the features of each modality, more flexibly allocate the information weights of different modalities, effectively reduce redundant information and strengthen the transmission of relevant information, and provide more refined support for decision-making tasks.

[0092] For the vector group merged in (1.2) In this step, the cross-modal enhanced fusion features are calculated in pairs, and the calculated features are spliced ​​to obtain the patient's personalized fusion feature vector, which is recorded as .

[0093] In one embodiment, the most important modal feature vector (such as the image embedding vector ), and then calculate the cross-modal enhanced fusion features with other vectors to obtain the patient's personalized fusion feature vector .

[0094] The structural design of cross-modal feature enhancement fusion ensures that the system can process multimodal data more flexibly and efficiently, enhance the expression of information of each modality through the self-attention mechanism, and explore the potential correlation between different modalities through the cross-modal attention mechanism. Dynamic adjustment and weighting ensure the effective fusion of multimodal features, thereby achieving more accurate cross-modal analysis and decision support.

[0095] (3) Based on the patient's personalized fusion feature vector, perform knowledge retrieval in the electronic record database of the entire tumor treatment process to obtain several treatment plans under similar clinical manifestations, use the retrieved treatment plans as prompt information, and input them together with the patient's personalized fusion feature vector into the large language model for decision-making to obtain the patient's personalized treatment plan, and evaluate the risk of the personalized treatment plan for the patient, wherein the personalized treatment plan and its risks are used to assist doctors in giving actual treatment plans;

[0096] The electronic record database of the entire tumor treatment process is the core knowledge base of the system. By integrating tumor-related medical knowledge, treatment cases, clinical trial data, etc., it provides a rich knowledge base for the multimodal large model and provides reliable evidence support for the recommendation of tumor treatment plans. Specifically, the content stored in the database includes basic medical knowledge related to tumors, treatment guidelines, clinical cases, follow-up records, medical literature, etc.

[0097] Basic medical knowledge: including tumor classification, staging, pathological subtypes, histological characteristics, molecular biological characteristics, clinical manifestations, diagnostic criteria, morbidity, mortality, etc.;

[0098] Treatment guidelines: National guidelines for clinical treatment of tumors, various treatments and their indications, risks, complications, etc., various chemotherapy drugs and their characteristics, side effects, etc.;

[0099] Clinical cases: They consist of two parts. The first part is the diagnosis and treatment data, which includes basic patient information, test results, laboratory test results, genomic data, nursing records, medication records, imaging data and corresponding imaging reports, etc., which are collected throughout the patient's visit and are used for retrieval and matching. The second part is the treatment plan, which includes the patient's treatment methods and changes in clinical symptoms, tumor changes, side effects, etc. after treatment, which are used to recommend treatment plans.

[0100] Follow-up records: long-term follow-up records of tumor recurrence, nursing plan, quality of life score, complications, sequelae, etc. after the patient completes treatment;

[0101] Medical literature: medical journals, clinical trial reports and other related literature content.

[0102] The database structure designed in this application is a vector database. Multimodal clinical information is extracted through feature extraction to generate an embedded high-dimensional vector representation, which is stored in the database in the form of a dense vector. Each vector corresponds to a piece of data, and all vectors are stored in fixed dimensions, including but not limited to 256, 512, 1024, length, and each dimension consists of a 32-bit floating point number. The database is integrated in a dynamic and scalable manner, and new tumor types, clinical cases, and drug data are regularly updated to the database to ensure the accuracy and timeliness of treatment recommendations.

[0103] It should be noted that the diagnosis and treatment data of patients in clinical cases are generated using the same method as steps (1)-(2).

[0104] In the specific implementation, the specific operation of step (3) is as follows:

[0105] (3.1) Query vector generation

[0106] The patient personalized fusion feature vector is used as the query vector

[0107] (3.1) Matching retrieval

[0108] The query vector is sent to the vector database storing clinical cases to retrieve the most relevant treatment options. The matching of the vector database is based on similarity retrieval, and the similarity between the query vector and the database vector is calculated by cosine similarity. Each patient diagnosis vector in , query vector Each patient diagnosis and treatment vector in the database Calculate similarity and return the most relevant The treatment plan corresponding to the patient's diagnosis and treatment data vector. The similarity calculation method is:

[0109] .

[0110] The most similar diagnosis and treatment data vector retrieved is output, and the corresponding treatment plan is used as the retrieval result to provide an evidence-based basis for doctors to formulate treatment plans for patients.

[0111] (3.3) Generation of personalized treatment plans

[0112] The retrieved treatment plan is input into the RAG model as prompt information together with the patient's multimodal feature vector. The model uses the retrieved treatment plan and combines it with the patient's specific situation (such as drug resistance, allergy history, surgical experience, gene mutation site, etc.) for personalized adjustment. For example, if the patient has a specific gene mutation or has a history of allergy to certain drugs, the system will automatically exclude these unsuitable treatment options in the recommended plan, or adjust the treatment intensity according to the patient's tolerance. The multimodal large model mines and strengthens the information correlation between different data, handles the heterogeneity and orthogonality of features of multiple modalities, learns the potential relationship between various patient data, and uses tumor-related professional medical knowledge in the database to generate personalized treatment plans. The generated personalized treatment plan includes detailed treatment including treatment methods (surgery / chemotherapy / neoadjuvant therapy), dosage and frequency, expected effects, personalized precautions, etc., to provide doctors with detailed and referenceable clinical treatment decision guidance.

[0113] (3.4) Prognostic risk assessment

[0114] Combined with the generated personalized treatment plan, factors affecting survival prognosis and treatment effect such as the patient's psychological state, age, economic status, etc. are further introduced into the RAG model to comprehensively evaluate the patient's survival period, recurrence risk, treatment side effects, economic pressure and quality of life under the treatment plan, so as to better foresee potential risks, reduce the risks of the treatment process, and help doctors make decisions that best meet the needs of patients.

[0115] (4) If the patient's multimodal diagnosis and treatment data is updated as the actual treatment progresses, the personalized treatment plan and its risks are updated, thereby providing real-time dynamic assistance to the doctor;

[0116] Considering that clinical treatment and disease development are a dynamic process, clinicians integrate the personalized treatment plan and prognostic risk warnings generated by the system, review the plan in combination with the patient's condition, and finally give an actual treatment plan and implement the treatment. With further treatment, the patient will generate new clinical data, such as new blood test results, medical images of tumor changes after radiotherapy, etc. These clinical data will be used to update the patient's multimodal diagnosis and treatment data. By repeating steps (1)-(3), a second or multiple decision-making process is made based on the latest diagnosis and treatment data. This can follow the patient's clinical manifestations, update and optimize the treatment plan in real time, adjust the treatment details, and ensure intelligent, real-time and refined decision-making.

[0117] Therefore, the technical solution of this application can be summarized as follows:

[0118] 1) Knowledge enhancement mechanism: By supplementing external databases, external knowledge is integrated into the large model generation process to avoid the uncertainty of results caused by catastrophic forgetting, get rid of knowledge limitations, and increase the knowledge priors of the large model in the medical field;

[0119] 2) Incremental learning mechanism: Database updates do not require system shutdown. New medical literature and clinical cases can be added to the database in real time to ensure that decision-making results are consistent with medical progress.

[0120] 3) Knowledge retrieval mode: by retrieving the most similar cases from professional databases, it provides evidence-based clinical decision support in the model of evidence-based medicine and increases the interpretability of the results;

[0121] 4) Personalized decision-making: Using the retrieved evidence as enhanced prompt information and combining it with patient-specific examination results to make data-driven decisions for the patient can effectively improve the personalization and accuracy of the treatment plan;

[0122] 5) Dynamic update mechanism: After the system makes a decision, the newly generated data on the patient’s clinical response under the treatment plan will be fed back to the system, which can dynamically and continuously optimize the treatment plan.

[0123] This application first uses feature extraction to extract distinguishing enhanced features from different modalities of data such as patients' electronic medical records, medical images, test data, genetic information, etc., and then uses the high-dimensional information processing and data comprehensive analysis capabilities of large models to integrate the feature complementarity and correlation between different modalities to more accurately understand and depict the patient's health portrait. This invention effectively overcomes the shortcomings of traditional single-modality or multi-modality methods, such as single data source and relatively independent decision-making process, and uses a multi-modality large model to mine and analyze the potential connections between various clinical data generated during the tumor treatment process, assisting doctors in making more accurate treatment decisions.

[0124] The present invention builds a database containing rich medical knowledge and clinical cases, and uses the RAG mechanism to match the cases in the database that are most similar to the patient's condition. This effectively overcomes the shortcomings of traditional large models, such as insufficient medical knowledge and low accuracy of medical tasks, and provides doctors with evidence-based, data-driven diagnosis and treatment support.

[0125] By utilizing the generation mechanism of the RAG architecture, combined with the patient's clinical data and similar case information, the most suitable treatment plan for the patient is generated, which improves the treatment effect while reducing possible risks. This overcomes the shortcomings of the low degree of personalization of traditional large models and provides doctors with more accurate and personalized diagnosis and treatment support.

[0126] The database carried by the present invention can be regularly maintained, dynamically updated with medical progress and reference cases, and ensure the timeliness of matching solutions. In addition, the clinical data and side effects generated by the patient according to the treatment method recommended by the system can also be fed back to the system for multiple decisions, and the treatment plan can be dynamically adjusted and continuously optimized. This overcomes the deficiency that the inference results of static data cannot adapt to the rapid progression of tumors, and provides doctors with dynamic and sophisticated clinical decision support.

[0127] Corresponding to the aforementioned embodiment of the intelligent decision-making method based on the knowledge retrieval multimodal large model, the present application also provides an embodiment of the intelligent decision-making device based on the knowledge retrieval multimodal large model.

[0128] Figure 4 is a block diagram of an intelligent decision-making device based on a knowledge retrieval multimodal large model according to an exemplary embodiment. Figure 4 , the device may include:

[0129] A feature extraction module 21 is used to obtain multimodal diagnosis and treatment data of a patient, preprocess the multimodal diagnosis and treatment data and generate corresponding feature vectors;

[0130] The feature enhancement and fusion module 22 is used to enhance the feature vectors of each modality and fuse the feature vectors across modalities based on the enhancement and fusion model to obtain a patient-personalized fusion feature vector;

[0131] A personalized treatment plan generation module 23 is used to perform knowledge retrieval in the electronic record database of the entire tumor treatment process based on the personalized fusion feature vector of the patient, obtain treatment plans under several similar clinical manifestations, use the retrieved treatment plans as prompt information, and input them together with the personalized fusion feature vector of the patient into the large language model for decision-making, obtain the personalized treatment plan of the patient, and evaluate the risk of the personalized treatment plan for the patient, wherein the personalized treatment plan and its risk are used to assist doctors in giving actual treatment plans;

[0132] The plan updating module 24 is used to update the personalized treatment plan and its risks if the multimodal diagnosis and treatment data of the patient is updated with the actual treatment progress, so as to provide real-time dynamic assistance to the doctor.

[0133] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0134] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0135] Correspondingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned intelligent decision-making method based on the knowledge retrieval multimodal large model.

[0136] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned intelligent decision-making method based on the knowledge retrieval multimodal large model. Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability where an intelligent decision-making device based on a knowledge retrieval multimodal large model provided by an embodiment of the present invention is located, except Figure 5In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0137] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the intelligent decision-making method based on the knowledge retrieval multimodal large model as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0138] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. An intelligent decision-making method based on a knowledge retrieval multimodal large model, characterized in that: include: Acquiring multimodal diagnosis and treatment data of the patient, preprocessing the multimodal diagnosis and treatment data and generating corresponding feature vectors; Based on the enhanced fusion model, the feature vectors of each modality are enhanced and the cross-modality feature vectors are fused to obtain the patient's personalized fusion feature vector; Based on the patient's personalized fusion feature vector, knowledge retrieval is performed in the electronic record database of the entire tumor treatment process to obtain several treatment plans under similar clinical manifestations, and the retrieved treatment plans are used as prompt information and input into the large language model together with the patient's personalized fusion feature vector for decision-making to obtain the patient's personalized treatment plan, and the risk of the personalized treatment plan for the patient is evaluated, wherein the personalized treatment plan and its risk are used to assist doctors in giving actual treatment plans; If the patient's multimodal diagnosis and treatment data is updated with the actual treatment progress, the personalized treatment plan and its risks are updated, thereby providing real-time dynamic assistance to doctors.

2. The method according to claim 1, characterized in that The multimodal diagnosis and treatment data includes basic patient information, consultation records, test results, laboratory test results, nursing records, genomic data, imaging data and corresponding imaging reports, wherein the consultation records and imaging reports are text data, and the basic patient information, test results, laboratory test results and nursing records are structured data; Preprocessing the multimodal diagnosis and treatment data and generating feature vectors include: Remove noise from text data by removing low-frequency words and term mapping, remove noise from structured data by detecting outliers and smoothing data; remove noise from image data by using image enhancement or filtering operations; By unifying the scale units of text data and structured data and standardizing the grayscale values ​​of image data, the scale of multimodal diagnosis and treatment data can be normalized; For the denoised and scale-normalized multimodal diagnosis and treatment data, the text feature vector is obtained through the pre-trained language model, the structured feature vector is generated by the fully connected layer or MLP, the genomic feature vector is generated by the graph neural network or the bioinformatics large model, the image data is segmented and feature extracted to obtain the image graph embedding vector, and the image word embedding vector is obtained based on the image segmentation result.

3. The method according to claim 2, characterized in that The enhanced fusion model includes a single-modal feature enhancement module and a cross-modal feature fusion module; In the unimodal feature enhancement module, the text feature vector and the structured feature vector are enhanced by using the self-attention mechanism, and the image embedding vector is enhanced by using the deformable self-attention mechanism; The cross-modal feature fusion module enhances the enhanced feature vectors of each modality through a self-attention mechanism. and the mode B eigenvector , cross-modal fusion is performed through the following formula: ， in is the fusion feature vector of mode A-mode B, is the length of each mode eigenvector.

4. The method according to claim 3, characterized in that The cross-modal feature fusion module is trained by the following comprehensive loss function: ， ， ， in and is the weight coefficient, Express expectations, is a set of data pairs, is the similarity of data pairs, is the true label of whether the data pair is similar, is the data pair matching degree, is the true label of whether the data pair matches.

5. The method according to claim 1, characterized in that The electronic record database of the entire process of tumor treatment is a vector database, in which the data stored is updated regularly, including tumor-related medical knowledge, treatment guidelines, treatment plans, follow-up records, and medical literature. Various types of data are generated through feature extraction to generate embedded high-dimensional vector representations, which are stored in the electronic record database of the entire process of tumor treatment in the form of dense vectors.

6. The method according to claim 1, characterized in that The patient personalized fusion feature vector is used as the query vector. Each patient diagnosis and treatment data vector in , query vector Each patient diagnosis and treatment data vector in the database Calculate the similarity and return the treatment plans corresponding to the most relevant patient diagnosis and treatment data vectors, where the similarity .

7. The method according to claim 1, characterized in that A retrieval-enhanced generative model is used as a large language model for generating personalized treatment plans, wherein the retrieval-enhanced generative model pre-learns tumor-related medical knowledge, treatment guidelines, follow-up records, and medical literature in the electronic record database of the entire tumor treatment process.

8. An intelligent decision-making device based on a knowledge retrieval multimodal large model, characterized in that: include: A feature extraction module, used to obtain multimodal diagnosis and treatment data of a patient, preprocess the multimodal diagnosis and treatment data and generate corresponding feature vectors; The feature enhancement and fusion module is used to enhance the feature vectors of each modality and fuse the feature vectors across modalities based on the enhanced fusion model to obtain the patient's personalized fusion feature vector; A personalized plan generation module is used to perform knowledge retrieval in the electronic record database of the entire process of tumor treatment based on the patient's personalized fusion feature vector, obtain several treatment plans under similar clinical manifestations, use the retrieved treatment plans as prompt information, and input them together with the patient's personalized fusion feature vector into a large language model for decision-making, obtain the patient's personalized treatment plan, and evaluate the risk of the personalized treatment plan for the patient, wherein the personalized treatment plan and its risk are used to assist doctors in giving actual treatment plans; The plan update module is used to update the personalized treatment plan and its risks if the patient's multimodal diagnosis and treatment data is updated with the actual treatment progress, thereby providing real-time dynamic assistance to doctors.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images