Medical aid decision-making system based on multi-modal large model

Abstract

The invention discloses a medical aid decision-making system based on a multi-modal large model, and relates to the technical field of medical artificial intelligence, and the system comprises a multi-modal data fusion module, a treatment scheme coding and management module, a prognosis prediction model module, a scheme simulation and deduction module, and a visual comparison module. The comprehensive state vector of a patient and the action vector of a candidate treatment scheme are jointly calculated, multiple long-term prognosis indexes after the scheme is executed are directly simulated, the model converts the treatment scheme into a computable variable by learning rules in historical treatment data, probabilistic prediction is carried out on a future result, and the prediction accuracy is improved. This enables a doctor to clearly see the risk brought by different selections and the long-term influence of treatment changes before making a decision, thereby converting the decision mode from experience-based inference to future simulation-based anticipation.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a medical auxiliary decision-making system based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, medical decision support systems have become a core component of smart healthcare. Most existing decision support systems focus on the disease diagnosis process. For example, they use convolutional neural networks to detect and segment lesions in medical images, or use natural language processing technology to extract key information from electronic medical records to help generate preliminary diagnostic reports. Their application scope is obviously limited. The current decision support capabilities end after diagnosis, leaving the more complex and critical decision-making process of selecting and formulating treatment plans to doctors based on their personal experience, memorized clinical guidelines, and limited evidence-based medicine.

[0003] However, existing technologies, when dealing with the complexity of treatment decisions, suffer from fragmented information integration and fail to achieve multimodal data fusion. They often analyze images, texts, or data in isolation, lacking a unified framework to construct a comprehensive health profile of the patient. This results in incomplete decision-making basis, diagnoses based on current static data, an inability to extrapolate and simulate dynamic processes, and difficulty in making intuitive and quantitative comparisons of the long-term benefits of different treatment options. Doctors cannot efficiently weigh the pros and cons, making it difficult to conduct high-quality doctor-patient shared decision-making. These shortcomings are particularly prominent in the treatment of complex chronic diseases such as cancer.

[0004] Therefore, clinical practice urgently calls for a new generation of decision support tools that can overcome the limitations of current decision-making systems, deeply integrate multi-source heterogeneous medical data, and on this basis, perform dynamic simulation, quantitative deduction, and optimal treatment plan selection, thereby combining doctors' experience with data to select the most suitable and personalized treatment path for patients. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a medical auxiliary decision-making system based on a multimodal large model. This system can jointly calculate the patient's comprehensive state vector and the action vector of candidate treatment plans through a prognostic prediction model, directly simulating multiple long-term prognostic indicators after the implementation of the plan. By learning the patterns in historical treatment data, the model transforms treatment plans into calculable variables and makes probabilistic predictions of future outcomes. This allows doctors to clearly see the risks of different choices and the long-term impact of treatment changes before making a decision, thereby transforming the decision-making mode from experience-based inference to prediction based on future simulation.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a medical auxiliary decision-making system based on a multimodal large model, the system comprising: a multimodal data fusion module, a treatment plan coding and management module, a prognosis prediction model module, a plan simulation and deduction module, and a visualization comparison module; The multimodal data fusion module is used to receive and process various types of medical data from patients, and generate a comprehensive state vector that fully represents the patient's current state through a pre-trained multimodal large model. ; The treatment protocol encoding and management module is used to build and maintain a standard treatment protocol library, and to encode each treatment protocol into a machine-readable action vector. ; The prognostic prediction model module receives the comprehensive state vector through the prognostic prediction model. Action vectors of candidate treatment options As a combined input, the output includes multiple expected prognostic indicators for the patient after implementing this treatment plan. ; The scheme simulation and deduction module is used to receive one or more candidate treatment schemes specified by the user, call the prognostic prediction model to perform deduction, and obtain a set of prognostic indicators corresponding to each scheme deduction. It also has a preset reward function that automatically calculates the comprehensive score of all feasible solutions. ; The visualization comparison module obtains the simulation results of different candidate treatment plans output by the plan simulation and deduction module through the data receiving interface, and displays the comparison results through the visualization interface and the plan management interface.

[0007] Furthermore, the various types of medical data in the multimodal data fusion module include electronic medical record text data, medical image data, laboratory test data, and physiological signal data; The medical data is processed by feature extraction and fusion using a pre-trained multimodal large model. For different types of data, a dedicated sub-model is used to extract features. Specifically, semantic features are extracted for text data, lesion and organ structure features are extracted for image data, quantitative features are extracted for laboratory test data using numerical standardization, and temporal features are extracted for physiological signal data. By allocating weights to each modal feature through an attention mechanism, a comprehensive state vector is generated that fully represents the patient's current health status. , This refers to the time point for data collection.

[0008] Furthermore, the treatment plan coding and management module includes a standardized plan knowledge base and a dynamic coding unit; The standardized protocol knowledge base stores treatment protocol templates maintained by historical clinical treatment guidelines. The templates include treatment protocol name, drug ingredients, administration method, dosage, treatment cycle, and combination therapy. The dynamic encoding unit is used to encode the treatment plans in the knowledge base into unified numerical action vectors based on their treatment areas and element composition. , For the first The candidate schemes are described in the encoding process, which combines one-hot encoding and embedding encoding. One-hot encoding is used for discrete categorical variables, while continuous numerical variables are normalized and then directly embedded to form a complete vector.

[0009] Furthermore, the specific steps for the dynamic coding unit to encode the treatment plan are as follows: It receives treatment plans from a standardized protocol knowledge base and user input, and extracts key information using natural language processing technology; The core elements of the treatment plan are identified and extracted using a named entity recognition model based on medical ontology. These core elements include treatment modality, drug name, route of administration, dosage, frequency of administration, number of treatment cycles, and the interrelationships among the components in the combination drug regimen. The extracted unstructured text information is transformed into structured key-value pair data. Based on the obtained structured data, an action vector is constructed using a hybrid encoding strategy. ,Right now: For categorical variables: they are converted into sparse vectors using one-hot encoding. The categorical variables include treatment modality, drug name, and route of administration. For numerical variables: the max-min normalization method is used to map them to the interval [0, 1], and they are directly embedded as one dimension of the vector. The numerical variables include drug dosage, number of treatment cycles, etc. For relational variables: For combination drug regimens, generate a corresponding sub-vector for each drug in the regimen independently, and concatenate all sub-vectors in a preset order to form a long vector representing the entire combination regimen. For single-drug regimens, the sub-vectors are directly used as the final vector.

[0010] Furthermore, the prognostic prediction model is constructed based on a deep neural network architecture and employs multi-task deep learning network technology. Its construction process specifically includes: Historical de-identified electronic medical record data was used as the training set. Each training sample contained three parts: input data, ground truth labels, and a mask matrix. The input data included a patient comprehensive state vector generated by the multimodal data fusion module. Action vectors corresponding to historical treatment plans The true label contains the true values ​​of multiple prognostic indicators, including short-term and long-term prognostic indicators, and the mask matrix is ​​used to identify the prognostic labels available in each sample. A multi-task deep learning network is employed, comprising an input layer, three hidden layers, and multiple task-specific output layers. The input layer receives the concatenated comprehensive state vector. and action vectors Each of the three hidden layers contains 512 neurons, and each of the multiple task-specific output layers corresponds to a prediction task of a prognostic indicator. Define the overall loss function as the weighted sum of the losses from each task: ,in, For the number of prognostic indicators, For the first Loss function for each prognostic indicator, These are the weighting coefficients for the corresponding tasks; The model is trained by iteratively updating all weight parameters using forward propagation and gradient descent optimizers, setting an initial learning rate, and using a learning rate decay strategy until the model converges.

[0011] Furthermore, the execution process of the prognostic prediction model is as follows: Receive a simulation request initiated by the scheme simulation and deduction module, the request containing the current patient's comprehensive state vector. An action vector of a candidate treatment plan ; The processed [ ; The input is fed into the prognostic prediction model to perform a forward propagation calculation, which sequentially passes through the linear transformation and nonlinear activation function calculation of each hidden layer, and reaches multiple task-specific output layers. Each sub-network in the multiple task-specific output layers performs synchronous calculations, outputting the predicted values ​​of various prognostic indicators for the patient after implementing the treatment plan, which together constitute the prognostic indicator vector. ; Will Return to the simulation and deduction module of the proposed scheme to complete one deduction calculation.

[0012] Furthermore, the reward function of the scheme simulation and deduction module is used to receive one or more candidate treatment schemes from the user interface, call the service of the treatment scheme encoding and management module, and convert the treatment schemes into corresponding action vectors in real time. Iterate through each action vector. For each Combine it with the current patient's comprehensive state vector Combine and call the prognostic prediction model to synchronously obtain the prediction results. All results are aggregated to generate a set of inference results. Multiple prognostic indicators Mapped to a scalar comprehensive score The ,in, Let be the weight coefficients for each dimension, and satisfy . , The value represents the degree of therapeutic effect and ranges from (0, 1). To quantify standardized treatment costs, Let be the risk utility value, ranging from (0, 1), where 1 represents almost no risk and 0 represents extremely high risk. Calculate the risk utility for each option. The values ​​are then sorted in descending order to generate a sorted list of schemes.

[0013] Furthermore, the scheme management interface in the visualization comparison module allows authorized users to perform add, delete, modify, and query operations on the standardized scheme knowledge base. When a scheme is added or modified, the system automatically calls the dynamic encoding unit to generate and update the corresponding action vector. Furthermore, a mapping index relationship between the scheme and the vector is established, and scheme deduction and comprehensive score calculation are carried out simultaneously.

[0014] Compared with existing technologies, this medical auxiliary decision-making system based on a multimodal large model has the following advantages: This invention uses a prognostic prediction model to jointly calculate the patient's comprehensive state vector and the action vector of candidate treatment plans, directly simulating multiple long-term prognostic indicators after the implementation of the plan. By learning the patterns in historical treatment data, the model transforms treatment plans into calculable variables and makes probabilistic predictions of future outcomes. This allows doctors to clearly see the risks of different choices and the long-term impact of treatment changes before making decisions, thereby transforming the decision-making model from experience-based inference to prediction based on future simulation.

[0015] The optimal solution mechanism adopted in this invention provides a clear quantitative basis for complex decision-making. By quickly eliminating infeasible solutions, it ensures the bottom line of decision safety and projects the remaining solutions into a multi-dimensional space with benefits and treatment risks as the core. By identifying the optimal treatment solution that achieves the best balance in all aspects, the system can accurately capture the matching rules between individual patient characteristics and solutions, present the doctor with the correlation logic between the solution and the patient's condition, and help him understand the core basis for solution selection.

[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is an operation flowchart for a medical auxiliary decision-making system based on a multimodal large model. Figure 2 This is a modular composition diagram of a medical auxiliary decision-making system based on a multimodal large model. Figure 3 This is a logical framework diagram of a medical auxiliary decision-making system based on a multimodal large model. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 This embodiment uses tumor treatment as a specific application scenario to construct a medical auxiliary decision-making system based on a multimodal large model. It aims to solve problems in tumor treatment such as the difficulty in integrating multi-source heterogeneous medical data, the lack of quantitative basis for treatment plan selection, and the difficulty in predicting prognoses. Figure 2 As shown, the system integrates patient electronic medical record text, medical images, laboratory tests, and physiological signal data through a multimodal data fusion module to generate a comprehensive state vector. The treatment plan coding and management module transforms standardized tumor treatment plans into machine-readable action vectors. The prognostic prediction model module, based on a deep neural network architecture, takes the comprehensive state vector and action vector as input and outputs multiple prognostic indicators. The plan simulation and deduction module calculates and ranks the comprehensive scores of each plan using a reward function. The visualization and comparison module provides an intuitive display of plan results and manages the plan library. Through this system, doctors can obtain quantitative prognostic information for different plans during the tumor treatment decision-making stage, enabling a shift from experience-driven decision-making to data simulation-driven decision-making, and improving the scientific rigor and personalization of treatment plan selection.

[0020] The multimodal data fusion module acquires the patient's comprehensive health status, receives and processes various types of medical data from the patient, and generates a comprehensive state vector that fully represents the patient's current state through a pre-trained multimodal large model. (Where t is the data acquisition time node, used to distinguish the patient's status at different time points and provide a time dimension reference for dynamic treatment decisions). The various types of medical data specifically include electronic medical record text data, medical imaging data, laboratory test data, and physiological signal data. For electronic medical record text data, it is obtained through the standardized data interface of the hospital information system. The data format includes structured text (such as tabular data such as patient basic information, past medical history, surgical history, etc.) and unstructured text (such as free text such as doctor's progress notes, discharge summaries, etc.). Text cleaning methods in natural language processing are used to remove special symbols, redundant spaces, and irrelevant information from the text to prepare for subsequent semantic feature extraction. For medical imaging data, which includes CT images, MRI images, pathological slide images, etc. of tumor patients, it is obtained through the medical imaging archive and communication system. The image data is format converted and then image preprocessing operations are performed, including grayscale normalization (eliminating grayscale value fluctuations caused by differences in imaging from different devices), image denoising (using Gaussian filtering to remove noise interference in the image), and image segmentation (segmenting tumor areas and normal organ tissue areas) to facilitate subsequent lesion feature extraction. For laboratory test data... The data includes numerical data such as complete blood count, biochemical indicators, and tumor marker tests, in structured tabular format. Due to significant differences in the units and ranges of values ​​for different tests, data integrity verification is necessary. Data samples with excessively high missing values ​​are removed. For a small number of missing values, mean or median imputation methods are used to fill in the missing values, laying the foundation for subsequent quantitative feature extraction. The physiological signal data includes time-series data such as electrocardiograms (ECG), heart rate, and blood pressure, acquired through vital sign monitoring equipment. The data format is time-series sequence data. Time-series alignment of the physiological signal data is performed. The system unifies signals collected from different devices to the same time sampling frequency. Signal filtering techniques are used to remove noise such as baseline drift and power frequency interference. Time-domain and frequency-domain features are extracted to capture the patterns of physiological signal changes over time. A pre-trained multimodal large model is used to extract and fuse features from the pre-processed medical data. A strategy of dedicated sub-model extraction and attention-based weighting is employed to ensure effective extraction and reasonable fusion of features from each modality. Specifically, dedicated sub-model feature extraction extracts unique features tailored to the characteristics of different data types. Semantic feature extraction from text data: The preprocessed electronic medical record text is input into the large model. The encoder layer of the model performs semantic encoding on the text sequence to obtain the contextual semantic feature vector of the text. This vector is then transformed into a vector with semantic representation capabilities, which can reflect information such as the patient's condition description, the severity of symptoms, and the tendency of disease diagnosis.

[0021] Image data lesion and organ structure feature extraction: A pre-trained convolutional neural network is used as the image sub-model. The pre-processed medical images are input into the model, and the convolutional and pooling layers of the model are used to extract features from the images layer by layer, gradually transitioning from low-level features to high-level features. Finally, the image feature vector is output. This vector can quantitatively represent the lesion features and organ structure status of the patient, providing an image-level basis for the formulation of tumor treatment plans.

[0022] Quantitative feature extraction of laboratory test data: Numerical standardization is used as a sub-model for laboratory test data. For preprocessed numerical data of laboratory tests, the reference range of each test indicator is first determined. Standardization method or maximum-minimum normalization method (maximum-minimum normalization is used in this embodiment, and the numerical variable processing in subsequent treatment plan coding is also used to maintain the consistency of standardization method) is used to map the values ​​to a specific interval, eliminate the influence of different indicator units and value ranges, and extract quantitative feature vectors. This vector can intuitively reflect the degree of abnormality of various physiological indicators of patients.

[0023] Physiological signal data temporal feature extraction: Capturing long-term dependencies in temporal data, the preprocessed physiological signal temporal data is input into an LSTM model, and the signal features of historical moments are stored through the model's memory unit. The temporal change features of the signals are extracted step by step, and a temporal feature vector is output. This vector can reflect the dynamic change trend of the patient's physiological state and provide a reference for judging the patient's tolerance to treatment.

[0024] Attention-based weighted fusion: After obtaining the feature vectors of each modality, an attention mechanism is used to assign weights to the features of each modality, generating a comprehensive state vector. The attention mechanism assigns different weights to different modal features based on their importance in representing the patient's current health status, thus increasing the weight of important modal features in the comprehensive vector. It unifies the dimensions of each modal feature vector (textual semantic feature vector, image feature vector, laboratory quantitative feature vector, and physiological signal temporal feature vector), maps each vector to the same dimensional space through a fully connected layer, calculates the attention weight for each modal feature vector, multiplies each modal feature vector by its corresponding attention weight, and then sums the vectors to obtain the comprehensive state vector. .

[0025] The treatment protocol coding and management module includes a standard treatment protocol library and a dynamic coding unit, which encodes each treatment protocol into a machine-readable action vector. (Where i represents the i-th candidate treatment plan, used to distinguish different treatment plans), achieving structured storage and machine computability of treatment plans, providing standardized plan inputs for prognostic prediction models. The standardized plan knowledge base stores tumor treatment plan templates maintained by historical clinical guidelines and clinical practice experience. To ensure the timeliness and accuracy of the plans, the knowledge base has a regular update mechanism, adding, modifying, or deleting plan templates based on the latest clinical guidelines and evidence-based medicine. The knowledge base storage structure uses a relational database, establishing associations between drug component IDs and drug information tables (storing drug names, mechanisms of action, adverse reactions, etc.), facilitating quick queries of the relationship between plans and drugs, suitable for storing complex relationships in combination drug regimens; the dynamic coding unit is a practical... The core component for transforming treatment plans from "natural language descriptions" to "machine-readable vectors" employs a hybrid coding strategy combining one-hot encoding and embedding encoding. The specific steps are as follows: The dynamic coding unit receives two types of treatment plan data through the system interface: one is a predefined standard treatment plan from a standardized treatment plan knowledge base, and the other is a personalized treatment plan input by the doctor based on the patient's specific condition. For the received treatment plan, natural language processing technology is first used to extract key information, including core entity categories such as treatment modality, drug name, route of administration, dosage, frequency of administration, number of treatment cycles, and combination drug relationships. The extracted unstructured text information is then converted into structured key-value pair data for subsequent encoding processing. The hybrid coding strategy constructs action vectors. Based on the obtained structured key-value pair data, an action vector is constructed using a hybrid encoding strategy. Different encoding methods are used for different types of variables: Categorical variables in tumor treatment protocols mainly include treatment modality, drug name, and route of administration. For these variables, one-hot encoding is used to convert them into sparse vectors. The principle of one-hot encoding is to assign a unique binary bit to each possible value of each variable. When the variable takes that value, the corresponding binary bit is 1, and the rest are 0. In this embodiment, taking the treatment modality as an example, if "targeted therapy" is the third value of the variable, then its one-hot encoded vector is [0, 0, 1, 0, ..., 0] (vector length is K, only the third bit is 1); if "oral administration" is the second value in the route of administration, then its one-hot encoded vector is [0, 1, 0, ..., 0] (vector length is N, only the second bit is 1). For numerical variables in tumor treatment protocols, mainly including drug dosage and number of treatment cycles, a maximum-minimum encoding method is used. The minimum normalization method maps the values ​​of numerical variables to the [0, 1] interval through linear transformation, eliminating the difference in value ranges with other variables (such as the number of treatment cycles). For relational variables, specifically referring to the relationships between drug components in the combination therapy, including drug combination order, dosage adjustment correlation, and treatment cycle synchronization, a sub-vector is independently generated for each drug in the combination therapy. These sub-vectors are then concatenated in a preset order to form a long vector representing the entire combination therapy. Specifically, for drug sub-vector generation: for each drug in the combination therapy, an independent sub-vector for that drug is generated using the method of "one-hot encoding of categorical variables + normalized embedding of numerical variables". For the preset sub-vector order: to ensure the consistency of the combination therapy coding, the system presets the sub-vector concatenation order based on the clinical logic of tumor treatment, concatenating all drug sub-vectors in the preset order to form the action vector of the combination therapy. ,Right now ,in, The number of drug types in a combination therapy regimen is given. For a single-drug regimen (without combination therapy), the sub-vector of that drug is directly used as the final action vector. Without the need for splicing operations, this method can transform both single-drug and multi-drug combination regimens into action vectors in a unified format, ensuring that subsequent prognostic prediction models can process them uniformly.

[0026] The prognostic prediction model module receives the comprehensive state vector through the prognostic prediction model. Action vectors of candidate treatment options As a combined input, the output includes multiple expected prognostic indicators for the patient after implementing this treatment plan. The prognostic prediction model is constructed based on a deep neural network architecture and employs multi-task deep learning network technology. Historical de-identified electronic medical record data is used as the training set. Each training sample contains three parts: input data, true labels, and a mask matrix. The input data includes a comprehensive patient state vector generated by the multimodal data fusion module. Action vectors corresponding to historical treatment plans The true labels contain the true values ​​of multiple prognostic indicators, including short-term and long-term prognostic indicators. The mask matrix is ​​used to identify the available prognostic labels for each sample. A multi-task deep learning network is employed, which includes an input layer, three hidden layers, and multiple task-specific output layers. The input layer receives the concatenated comprehensive state vector. and action vectors Each of the three hidden layers contains 512 neurons, and each of the multiple task-specific output layers corresponds to a prediction task for a prognostic metric. The comprehensive loss function is defined as the weighted sum of the losses for each task. ,in, For the number of prognostic indicators, For the first Loss function for each prognostic indicator, The weights are assigned to the corresponding tasks. The model is trained by iteratively updating all weight parameters using forward propagation and gradient descent optimizers, setting an initial learning rate, and employing a learning rate decay strategy until the model converges. The model receives a simulation and deduction request initiated by the scheme simulation and deduction module, which includes the current patient's comprehensive state vector. An action vector of a candidate treatment plan The processed [ ; The input is fed into the prognostic prediction model for a forward propagation calculation, sequentially passing through linear transformations and nonlinear activation function calculations in each hidden layer, reaching multiple task-specific output layers. Each sub-network in these task-specific output layers performs synchronous calculations, outputting the predicted values ​​of various prognostic indicators for the patient after implementing the treatment plan, collectively forming a prognostic indicator vector. ,Will Return to the simulation and deduction module of the proposed scheme to complete one deduction calculation.

[0027] The treatment plan simulation and extrapolation module receives one or more candidate treatment plans specified by the user, calls the prognostic prediction model to perform extrapolation, and obtains a set of prognostic indicators corresponding to each extrapolation plan. It also has a preset reward function that automatically calculates the comprehensive score of all feasible solutions. By receiving one or more candidate treatment plans from the user interface, the system invokes the service of the treatment plan encoding and management module to convert the treatment plan into corresponding action vectors in real time. Iterate through each action vector. For each Combine it with the current patient's comprehensive state vector Combine and call the prognostic prediction model to synchronously obtain the prediction results. , used to indicate the first The prognostic index vectors of each candidate scheme are aggregated to generate a set of inference results. Multiple prognostic indicators Mapped to a scalar comprehensive score The ,in, Let be the weight coefficients for each dimension, and satisfy . , The value represents the degree of therapeutic effect and ranges from (0, 1). To quantify standardized treatment costs, Let be the risk utility value, ranging from (0, 1), where 1 represents almost no risk and 0 represents extremely high risk. Calculate the risk utility for each option. The values ​​are then used to sort the treatment plans in descending order, generating a list of sorted plans for doctors' reference.

[0028] The visualization and comparison module acquires the simulation results through the data receiving interface and displays the prognostic indicators and comprehensive scores of each scheme side-by-side in the visualization interface. Simultaneously, it provides a scheme management interface, allowing authorized users to perform CRUD operations on the scheme library. When a scheme is added or modified, the system automatically calls the dynamic coding unit to generate and update the corresponding action vector. Furthermore, a mapping index relationship between the scheme and the vector is established, and scheme deduction and comprehensive score calculation are carried out simultaneously.

[0029] This embodiment details the working principle of a medical decision support system based on a multimodal large model. Through the collaborative work of five modules—multimodal data fusion, treatment plan coding, prognosis prediction, plan deduction, and visualization comparison—the system achieves intelligent decision support from patient data integration to treatment plan optimization. By quantitative deduction and multi-dimensional comparison, it significantly improves the scientific nature, personalization, and interpretability of treatment decisions. This system is particularly suitable for the long-term treatment management of complex diseases such as tumors and chronic diseases.

[0030] Example 2 This embodiment provides an operational flow for a medical auxiliary decision-making system based on a multimodal large model in performing medical auxiliary decisions, such as... Figure 1 As shown, the specific steps of this process are as follows: (1) Data preparation and patient status modeling Collect multimodal medical data: Collect patients' electronic medical records, medical images, laboratory test data, physiological signals, etc.

[0031] Feature extraction and fusion: Pre-trained sub-models are used to extract features for each modality (such as text semantics, image structure, test values, and physiological time series).

[0032] By fusing features from various modalities through an attention mechanism, a comprehensive state vector of the patient is generated.

[0033] (2) Treatment plan coding and management Build a standardized protocol library: store structured treatment protocols from clinical guidelines and treatment templates.

[0034] Treatment plan code: Use natural language processing technology to extract key elements (such as drugs, dosages, cycles, etc.) from treatment plans.

[0035] Encode features into machine-readable action vectors: Categorical variables (such as drug names) use one-hot encoding; Normalize numerical variables (such as dosage); Combination therapy regimens then concatenate the sub-vectors of each drug.

[0036] (3) Prognostic prediction and scheme deduction Input joint vector: The patient state vector is concatenated with the action vector of the candidate treatment plan and input into the prognostic prediction model.

[0037] Multi-task prediction: The model uses a multi-layer neural network for forward propagation; The predicted values ​​of multiple prognostic indicators are output to form a prognostic indicator vector.

[0038] (4) Evaluation and ranking of the schemes Calculate the overall score: Multiple prognostic indicators are mapped to a scalar score R(O) using a pre-defined reward function; The total score is calculated by considering factors such as treatment effectiveness, cost, and risk, and then weighted accordingly.

[0039] Sorting and Recommendation Scheme: Sort all candidate schemes in descending order of comprehensive score and generate a recommendation list.

[0040] (5) Visualization and Interaction Results visualization: Compare the prognostic indicators and scores of different options using charts; Provides a solution management interface, supporting users to add, modify, or delete solutions.

[0041] Shared decision-making between doctors and patients: Doctors combine system recommendations with the patient's actual situation to make the final choice of treatment plan.

[0042] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A medical decision support system based on a multimodal large model, characterized in that, The system consists of: a multimodal data fusion module, a treatment plan coding and management module, a prognosis prediction model module, a treatment plan simulation and deduction module, and a visualization and comparison module; The multimodal data fusion module is used to receive and process various types of medical data from patients, and generate a comprehensive state vector that fully represents the patient's current state through a pre-trained multimodal large model. ; The treatment protocol encoding and management module is used to build and maintain a standard treatment protocol library, and to encode each treatment protocol into a machine-readable action vector. ; The prognostic prediction model module receives the comprehensive state vector through the prognostic prediction model. Action vectors of candidate treatment options As a combined input, the output includes multiple expected prognostic indicators for the patient after implementing this treatment plan. ; The scheme simulation and deduction module is used to receive one or more candidate treatment schemes specified by the user, call the prognostic prediction model to perform deduction, and obtain a set of prognostic indicators corresponding to each scheme deduction. It also has a preset reward function that automatically calculates the comprehensive score of all feasible solutions. ; The visualization comparison module obtains the simulation results of different candidate treatment plans output by the plan simulation and deduction module through the data receiving interface, and displays the comparison results through the visualization interface and the plan management interface.

2. The medical auxiliary decision-making system based on a multimodal large model according to claim 1, characterized in that, The multimodal data fusion module contains various types of medical data, including electronic medical record text data, medical image data, laboratory test data, and physiological signal data. The medical data is processed by feature extraction and fusion using a pre-trained multimodal large model. For different types of data, a dedicated sub-model is used to extract features. Specifically, semantic features are extracted for text data, lesion and organ structure features are extracted for image data, quantitative features are extracted for laboratory test data using numerical standardization, and temporal features are extracted for physiological signal data. By allocating weights to each modal feature through an attention mechanism, a comprehensive state vector is generated that fully represents the patient's current health status. , This refers to the time point for data collection.

3. A medical auxiliary decision-making system based on a multimodal large model according to claim 1, characterized in that, The treatment plan coding and management module includes a standardized plan knowledge base and dynamic coding units; The standardized protocol knowledge base stores treatment protocol templates maintained by historical clinical treatment guidelines. The templates include treatment protocol name, drug ingredients, administration method, dosage, treatment cycle, and combination therapy. The dynamic encoding unit is used to encode the treatment plans in the knowledge base into unified numerical action vectors based on their treatment areas and element composition. , For the first The candidate schemes are described in the encoding process, which combines one-hot encoding and embedding encoding. One-hot encoding is used for discrete categorical variables, while continuous numerical variables are normalized and then directly embedded to form a complete vector.

4. A medical auxiliary decision-making system based on a multimodal large model according to claim 3, characterized in that, The specific steps for the dynamic coding unit to encode the treatment plan are as follows: It receives treatment plans from a standardized protocol knowledge base and user input, and extracts key information using natural language processing technology; The core elements of the treatment plan are identified and extracted using a named entity recognition model based on medical ontology. These core elements include treatment modality, drug name, route of administration, dosage, frequency of administration, number of treatment cycles, and the interrelationships among the components in the combination drug regimen. The extracted unstructured text information is transformed into structured key-value pair data. Based on the obtained structured data, an action vector is constructed using a hybrid encoding strategy. ,Right now: For categorical variables: they are converted into sparse vectors using one-hot encoding. The categorical variables include treatment modality, drug name, and route of administration. For numerical variables: the maximum-minimum normalization method is used to map them to the interval [0, 1] and embed them directly as one dimension of the vector. The numerical variables include drug dosage and number of treatment cycles. For relational variables: For combination drug regimens, generate a corresponding sub-vector for each drug in the regimen independently, and concatenate all sub-vectors in a preset order to form a long vector representing the entire combination regimen. For single-drug regimens, the sub-vectors are directly used as the final vector.

5. A medical auxiliary decision-making system based on a multimodal large model according to claim 1, characterized in that, The prognostic prediction model is constructed based on a deep neural network architecture and employs multi-task deep learning network technology. Its construction process specifically includes: Historical de-identified electronic medical record data was used as the training set. Each training sample contained three parts: input data, ground truth labels, and a mask matrix. The input data included a patient comprehensive state vector generated by the multimodal data fusion module. Action vectors corresponding to historical treatment plans The true label contains the true values ​​of multiple prognostic indicators, including short-term and long-term prognostic indicators, and the mask matrix is ​​used to identify the prognostic labels available in each sample. A multi-task deep learning network is employed, comprising an input layer, three hidden layers, and multiple task-specific output layers. The input layer receives the concatenated comprehensive state vector. and action vectors Each of the three hidden layers contains 512 neurons, and each of the multiple task-specific output layers corresponds to a prediction task of a prognostic indicator. Define the overall loss function as the weighted sum of the losses from each task: ,in, For the number of prognostic indicators, For the first Loss function for each prognostic indicator, These are the weighting coefficients for the corresponding tasks; The model is trained by iteratively updating all weight parameters using forward propagation and gradient descent optimizers, setting an initial learning rate, and using a learning rate decay strategy until the model converges.

6. A medical auxiliary decision-making system based on a multimodal large model according to claim 5, characterized in that, The execution process of the prognostic prediction model is as follows: Receive a simulation request initiated by the scheme simulation and deduction module, the request containing the current patient's comprehensive state vector. An action vector of a candidate treatment plan ; The processed [ ; The input is fed into the prognostic prediction model to perform a forward propagation calculation, which sequentially passes through the linear transformation and nonlinear activation function calculation of each hidden layer, and reaches multiple task-specific output layers. Each sub-network in the multiple task-specific output layers performs synchronous calculations, outputting the predicted values ​​of various prognostic indicators for the patient after implementing the treatment plan, which together constitute the prognostic indicator vector. ; Will Return to the simulation and deduction module of the proposed scheme to complete one deduction calculation.

7. A medical auxiliary decision-making system based on a multimodal large model according to claim 1, characterized in that, The reward function of the treatment scheme simulation and deduction module is used to receive one or more candidate treatment schemes from the user interface, call the service of the treatment scheme encoding and management module, and convert the treatment schemes into corresponding action vectors in real time. Iterate through each action vector. For each Combine it with the current patient's comprehensive state vector Combine and call the prognostic prediction model to synchronously obtain the prediction results. , used to indicate the first The prognostic index vectors of each candidate scheme are aggregated to generate a set of inference results. Multiple prognostic indicators Mapped to a scalar comprehensive score The ,in, Let be the weight coefficients for each dimension, and satisfy . , The value represents the degree of therapeutic effect and ranges from (0, 1). To quantify standardized treatment costs, Given a risk utility value ranging from (0, 1), calculate the risk utility for each option. The values ​​are then sorted in descending order to generate a sorted list of schemes.

8. A medical auxiliary decision-making system based on a multimodal large model according to claim 1 or 3, characterized in that, The scheme management interface in the visualization comparison module allows authorized users to add, delete, modify, and query the standardized scheme knowledge base. When adding or modifying a scheme, the system automatically calls the dynamic coding unit to generate and update the corresponding action vector. Furthermore, a mapping index relationship between the scheme and the vector is established, and scheme deduction and comprehensive score calculation are carried out simultaneously.

Images