Medical aid decision-making system based on multi-modal large model

Abstract

The invention, which relates to the technical field of the medical platform, discloses a medical aid decision-making system based on a multi-modal large model, comprising an electronic medical record, a medical image and a laboratory examination result. Carrying out the preprocessing of the collected multi-modal medical data, and obtaining the preprocessed multi-modal medical data; the data processing module is further used for preprocessing the preprocessed multi-modal medical data, comparing the preprocessed multi-modal medical data with an existing medical knowledge base, verifying the reasonability of the data, evaluating the data, and deleting the non-conforming multi-modal medical data to obtain the evaluated multi-modal medical data; the problem that the generalization ability of a multi-modal large model in the medical field is limited is solved through the decision reasoning module, rich background knowledge is provided for the model by means of construction and distillation of a medical knowledge graph, the understanding and processing ability of the model for different tasks is improved, and the system has good application prospects. And the problem of insufficient reliability of the multi-modal large model in actual clinical application is solved through the verification module.

Description

Technical Field

[0001] The present invention relates to the field of medical platform technology, and more specifically, to a medical auxiliary decision-making system based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence technology, its application in the medical field is becoming more and more extensive, especially in the medical decision support system (CDSS). The medical decision support system uses machine learning, natural language processing and image analysis technologies to integrate various medical data to provide doctors with decision support for diagnosis and treatment; Multimodal AI technology integrates medical data from different sources, such as text cases, clinical reports, medical images, and pathological slide data, to achieve comprehensive understanding and accurate diagnosis of diseases; However, in the actual use process, the multimodal large model has the problem of "hallucination" or "fabrication of facts", that is, it may output wrong, inaccurate, and untrue information. The basic principle is derived from the "lossy compression" of Internet information. In the medical field, this "hallucination" problem may lead to serious consequences, causing the system to make decisions based on wrong information, which may directly endanger the health of patients and weaken the professional judgment of doctors; In addition, the generalization ability of large multimodal models is limited, and they are prone to fall into local optimal solutions during training, resulting in poor performance on unseen cases. The diversity and complexity of medical data make it difficult for the model to maintain high accuracy and robustness in all cases. When dealing with new medical knowledge and concepts, it may not be able to accurately diagnose or provide advice, and may produce misleading outputs. This lack of generalization ability affects the reliability of the model in clinical applications, reduces doctors' trust in the model, and reduces its actual application value in medical decision-making. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a medical decision-making support system based on a multimodal large model.

[0004] The present invention provides a medical decision-making assistance system based on a multimodal large model, comprising a data processing module, a fusion extraction module, a decision-making reasoning module and a verification module; The data processing module is used to collect multimodal medical data, specifically including electronic medical records, medical images and laboratory test results; and preprocess the collected multimodal medical data to obtain preprocessed multimodal medical data; The data processing module is also used to compare the pre-processed multimodal medical data with the existing medical knowledge base, verify the rationality of the data, and evaluate, delete the non-compliant multimodal medical data, obtain the evaluated multimodal medical data, and transmit the evaluated multimodal medical data to the fusion extraction module; The fusion extraction module includes a modality fusion unit and a feature extraction unit, wherein the modality fusion unit is used to receive the multimodal medical data evaluated by the data processing module and fuse the evaluated multimodal medical data to obtain fused medical data; The feature extraction unit is used to extract features from the fused medical data, obtain key features, and transmit them to the decision reasoning module; The decision reasoning module includes a knowledge graph unit and a generation unit. The knowledge graph unit is used to construct a medical knowledge graph, structure the representation and association of scattered medical knowledge, and distill it into the generation unit; The generation unit has a built-in model, generates a medical decision based on the key features received from the feature extraction unit, obtains medical decision information, and transmits it to the verification module; the verification module is used to evaluate the medical decision information of the generation unit, calculate its reliability, and output the calculation results to medical personnel.

[0005] Preferably, the specific working steps of the data processing module are as follows: First, multimodal medical data is collected; Duplicate records were then identified by examining unique identifiers in the multimodal medical data; Keep one unique record and delete other duplicates, while identifying and removing fields that are not relevant to medical decision making; Normalize all numerical data in multimodal medical data, and use the median of the same type of data to fill in missing values ​​in numerical data; For all text data in multimodal medical data, the text data is segmented, long texts are split into words or phrases, stop words in the text are removed, and words are restored to their stem forms; for all imaging data in multimodal medical data, the values ​​of missing data points are calculated by linear interpolation based on known data points, or the values ​​of the nearest data points are used to fill in the values ​​of missing data points; The preprocessing is completed to obtain preprocessed multimodal medical data.

[0006] Preferably, the data processing module verifies the rationality of the data and evaluates the specific working steps as follows: For each electronic medical record data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the electronic medical record data , where n is the number of records in the electronic medical record data, is the matching degree between the ith understanding in the electronic medical record data and the corresponding entry in the medical knowledge base, which is obtained by calculating the cosine similarity; For the preprocessed multimodal medical data, obtain the rationality value of each medical imaging data 2; For the preprocessed multimodal medical data, obtain the rationality value of each laboratory test result data 3; The threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical image data, and the threshold W3 of the rationality value of the laboratory test result data are set in advance. , the rationality value of each medical imaging data 2 and the reasonableness value of each laboratory test result data 3. Compare the threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical imaging data, and the threshold W3 of the rationality value of the laboratory test result data respectively, mark all data whose comparison results are less than the corresponding thresholds as non-compliant multimodal medical data, delete them, and obtain the evaluated multimodal medical data.

[0007] Preferably, for the pre-processed multimodal medical data, the rationality value of each medical image data is obtained. The specific steps of 2 are as follows: For each piece of medical imaging data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the medical imaging data , where m is the number of images in the medical imaging data, It is the similarity between the i-th image in the medical image data and the standard image in the medical knowledge base, which is obtained by SURF feature matching calculation.

[0008] Preferably, for the pre-processed multimodal medical data, the rationality value of each laboratory test result data is obtained. The specific steps of 3 are as follows: For each laboratory test result data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the laboratory test result data , where k is the number of items in the laboratory test result data, is the result value of the ith item in the laboratory test result data, The mean value of the result of the ith item in the medical knowledge base, is the standard deviation of the result value of the i-th item in the medical knowledge base.

[0009] Preferably, the specific working steps of the modality fusion unit are as follows: First, the text data, image data, and numerical data in the evaluated multimodal medical data are spliced ​​together; The importance of each modality is given the same weight and weighted average is performed to complete the fusion of modalities and obtain the fused medical data.

[0010] Preferably, the specific working steps of the feature extraction unit are as follows: Obtain fused medical data; For the text data in the fused medical data, the LSTM network is used to extract the semantic features of the electronic medical record text data. For the image data in the fused medical data, the pre-trained convolutional neural network is used to extract the high-dimensional feature vectors of the medical imaging data. For the laboratory test result data in the fused medical data, the statistical feature extraction is used to obtain the features of the laboratory test result data. All extracted features are transmitted to the decision reasoning module as key features.

[0011] Preferably, the specific working steps of the knowledge graph unit are as follows: Firstly, the multimodal medical data collected by the data processing unit is obtained, and medical-related entities are identified using natural language processing technology; and normalize these entities to standard terminology; Then, the relationship between entities is determined through relationship extraction technology; In addition, attribute values ​​are calculated for entities in the graph; After adding new data, new concepts are obtained and automatically added to the concept layer of the knowledge base to complete the construction of the knowledge graph; Then, key knowledge is extracted from the constructed medical knowledge graph, including entities, relationships, and attributes, and this knowledge is encoded into a structured representation. The entities and relationships are converted into vector representations using embedding technology, and then trained using a graph neural network model to generate a lightweight model, and the knowledge in the lightweight model is passed to the generation unit.

[0012] Preferably, the specific working steps of the generating unit are as follows: Firstly, receiving key features from the feature extraction unit; Initialize the built-in decision generation model; Input the received key features into the model, use the model to analyze and process the input key features, and generate medical decision information; Medical decision information is obtained and transmitted to the verification module.

[0013] Preferably, the specific working steps of the verification module are as follows: Obtaining the medical decision information output by the generation unit, determining evaluation indicators for evaluating the reliability of the medical decision information, specifically including accuracy, precision, recall rate, F1 score, and setting a threshold for each evaluation indicator; Collect the expert diagnosis results of the same case as the gold standard, compare the decision results of the generation unit with the expert diagnosis results, and calculate the value G of each evaluation indicator; Collect historical medical records as a reference, compare the decision results of the generation unit with the diagnosis and treatment results of historical cases, and calculate the value H of each evaluation indicator; According to the formula , calculate and obtain the reliability value V of the medical decision information of the generating unit, and output the calculation result to the medical personnel.

[0014] Beneficial effects: The data processing module solves the "hallucination" problem that may be caused by data quality problems in the medical field of multimodal large models, ensures the quality and accuracy of input data, lays a solid foundation for subsequent links, and reduces the risk of the model generating erroneous information; the fusion extraction module solves the "hallucination" phenomenon that may be caused by single modality data or inaccurate feature extraction when multimodal large models process medical data, and comprehensively utilizes information from different modalities to improve the robustness of features, so that the model can still maintain a high accuracy under different data distributions, further reducing the risk of "hallucination"; The decision-making reasoning module solves the problem of limited generalization ability of large multimodal models in the medical field. With the help of the construction and distillation of medical knowledge graphs, rich background knowledge is provided for the model, which improves the model's understanding and processing capabilities of different tasks, enabling it to more accurately diagnose or provide advice when faced with new medical knowledge and concepts, and enhances the model's ability to maintain high accuracy and robustness in all situations. The verification module solves the problem of insufficient reliability of large multimodal models in actual clinical applications. A variety of verification methods are used to comprehensively evaluate the reliability of the generated medical decision-making information, timely discover and solve the problem of insufficient generalization ability, and continuously optimize the model based on the verification results, which improves the model's performance under new data and new tasks, enhances medical staff's trust in the system, and improves its actual application value in medical decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0016] Application scenarios: In the specific use process, the multimodal large model has the problem of "hallucination" or "fabrication of facts", that is, it may output wrong, inaccurate, and untrue information. The basic principle is derived from the "lossy compression" of Internet information. In the medical field, this "hallucination" problem may lead to serious consequences, causing the system to make decisions based on wrong information, which may directly endanger the health of patients and weaken the professional judgment of doctors; In addition, the generalization ability of large multimodal models is limited, and they are prone to fall into local optimal solutions during training, resulting in poor performance on unseen cases. The diversity and complexity of medical data make it difficult for the model to maintain high accuracy and robustness in all cases. When dealing with new medical knowledge and concepts, it may not be able to accurately diagnose or provide advice, and may produce misleading outputs. This lack of generalization ability affects the reliability of the model in clinical applications, reduces doctors' trust in the model, and reduces its actual application value in medical decision-making.

[0017] like Figure 1 As shown: including data processing module, fusion extraction module, decision reasoning module and verification module; The data processing module is used to collect multimodal medical data, specifically including electronic medical records, medical images and laboratory test results; and preprocess the collected multimodal medical data to obtain preprocessed multimodal medical data; it should be noted that the data processing module is responsible for collecting multimodal medical data, covering multi-source information such as electronic medical records, medical images, laboratory test results, etc. After the collection is completed, the data is preprocessed, including cleaning, standardization, missing value filling and other operations to ensure the integrity and consistency of the data. Subsequently, the preprocessed data is compared and verified with the medical knowledge base to evaluate the rationality of the data, delete or correct the data that does not conform to medical knowledge, and finally obtain the evaluated multimodal medical data, and transmit it to the fusion extraction module. The purpose of this module is to ensure the quality and accuracy of the input data; The data processing module is also used to compare the pre-processed multimodal medical data with the existing medical knowledge base, verify the rationality of the data, and evaluate, delete the non-compliant multimodal medical data, obtain the evaluated multimodal medical data, and transmit the evaluated multimodal medical data to the fusion extraction module; The fusion extraction module includes a modality fusion unit and a feature extraction unit, wherein the modality fusion unit is used to receive the multimodal medical data evaluated by the data processing module and fuse the evaluated multimodal medical data to obtain fused medical data; The feature extraction unit is used to extract features from the fused medical data, obtain key features, and transmit them to the decision reasoning module; it should be noted that the modality fusion unit receives the evaluated multimodal medical data, effectively fuses the data of different modalities, and generates fused medical data. The feature extraction unit accurately extracts the key features that are most valuable for medical decision-making, and transmits these key features to the decision reasoning module. This module aims to provide high-quality input information for decision reasoning through the fusion and feature extraction of multimodal data; The decision reasoning module includes a knowledge graph unit and a generation unit. The knowledge graph unit is used to construct a medical knowledge graph, structure the representation and association of scattered medical knowledge, and distill it into the generation unit; The generation unit has a built-in model, and generates medical decisions based on the key features received by the feature extraction unit, obtains medical decision information, and transmits it to the verification module; it should be noted that the knowledge graph unit is responsible for constructing a medical knowledge graph, and structuredly represents and associates scattered medical knowledge, such as medical literature, clinical guidelines, expert experience, etc., and regularly updates it to ensure the latest and accuracy of knowledge, and organizes structured medical knowledge into the generation unit. The generation unit has built-in multi-task learning and reinforcement learning algorithms, and generates medical decision information, such as disease diagnosis results, treatment plan recommendations, etc., based on the received key features and structured medical knowledge, and transmits this information to the verification module, which can generate accurate and reliable medical decision information based on the extracted key features and rich medical knowledge; The verification module is used to evaluate the medical decision information of the generation unit, calculate its reliability, and output the calculation results to the medical staff. It should be noted that the verification module evaluates the medical decision information output by the generation unit, uses a variety of verification methods such as comparison with expert diagnosis results and comparison with historical cases, comprehensively calculates its reliability, and outputs the evaluation results to the medical staff to assist them in making the final decision; It should also be noted that the data processing module solves the "hallucination" problem that may be caused by data quality problems in the medical field of multimodal large models, ensures the quality and accuracy of input data, lays a solid foundation for subsequent links, and reduces the risk of the model generating erroneous information; the fusion extraction module solves the "hallucination" phenomenon that may be caused by single modality data or inaccurate feature extraction when multimodal large models process medical data, and comprehensively utilizes information from different modalities to improve the robustness of features, so that the model can still maintain a high accuracy under different data distributions, further reducing the risk of "hallucination"; The problem of limited generalization ability of multimodal large models in the medical field is solved by the decision-making and reasoning module. With the construction and distillation of the medical knowledge graph, rich background knowledge is provided for the model, improving the model's understanding and processing ability for different tasks, enabling it to make more accurate diagnoses or provide suggestions when facing new medical knowledge and concepts, and enhancing the model's ability to maintain high accuracy and robustness in all situations; the problem of insufficient reliability of multimodal large models in actual clinical applications is solved by the verification module. A variety of verification methods are used to comprehensively evaluate the reliability of the generated medical decision-making information, timely discover and solve the problem of insufficient generalization ability, continuously optimize the model according to the verification results, improve the model's performance under new data and new tasks, enhance medical staff's trust in the system, and提升 its practical application value in medical decision-making.

[0018] As an optional embodiment: The specific working steps of the data processing module are as follows: First, collect multimodal medical data; it should be noted that multimodal medical data is specifically collected from multi-source systems such as hospital information systems (HIS), electronic medical record systems (EMR), and picture archiving and communication systems (PACS). Multimodal medical data specifically includes electronic medical records: including text information such as the patient's medical history, symptoms, diagnosis, treatment records, etc., medical images: including image data such as X-rays, CTs, MRIs, ultrasounds, etc., laboratory test results: including laboratory data such as blood tests, urine tests, biochemical tests, etc., and other data: such as other medical data like electrocardiograms (ECGs), electroencephalograms (EEGs). After that, identify duplicate records by checking the unique identifiers in the multimodal medical data; in this embodiment, the unique identification symbols are patient ID and examination date. Retain one unique record, delete other duplicate records, and at the same time identify and delete fields irrelevant to medical decision-making; such as non-medical related parts in the patient's personal information (such as home address, contact phone number, etc.). Normalize all numerical data in the multimodal medical data. For missing values in the numerical data, fill them with the median of the same type of data; it should be noted that for numerical data such as age, blood pressure, blood sugar, etc., the standardization method is Z-score standardization. It should also be noted that numerical data appears in electronic medical records, medical images, and laboratory test results. For all text data in the multimodal medical data, perform word segmentation on the text data, split long texts into words or phrases, remove stop words in the text, and restore the words to their stem forms; such as common but meaningless words like "的", "是", "和", etc.; text data appears in electronic medical records, medical images, and laboratory test results. For all imaging data in multimodal medical data, the values ​​of missing data points are calculated by linear interpolation based on known data points, or the values ​​of the nearest data points are used to fill in the values ​​of missing data points; imaging data generally appear in medical images; The preprocessing is completed to obtain the preprocessed multimodal medical data. It should be noted that through the above detailed steps, the multimodal medical data can be effectively preprocessed, including data cleaning, data standardization and missing value filling, which ensures the quality and accuracy of the input data, provides a reliable data basis for subsequent feature extraction and decision reasoning, effectively reduces the model "hallucination" phenomenon caused by data quality problems, and improves the overall performance and reliability of the system.

[0019] As an optional embodiment: the data processing module verifies the rationality of the data and evaluates the specific working steps as follows; For each electronic medical record data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the electronic medical record data , where n is the number of records in the electronic medical record data, It is the matching degree between the i-th understanding in the electronic medical record data and the corresponding entry in the medical knowledge base, which is obtained by cosine similarity calculation; it should be noted that the electronic medical record contains a large number of records, and the rationality of each record may affect the final diagnosis result. By calculating the average rationality of all records, the reliability of the entire electronic medical record can be comprehensively evaluated; For the preprocessed multimodal medical data, obtain the rationality value of each medical imaging data 2; For the preprocessed multimodal medical data, obtain the rationality value of each laboratory test result data 3; The threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical image data, and the threshold W3 of the rationality value of the laboratory test result data are set in advance. , the rationality value of each medical imaging data 2 and the reasonableness value of each laboratory test result data 3, respectively compare the threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical imaging data, and the threshold W3 of the rationality value of the laboratory test result data, mark all data whose comparison results are less than the corresponding thresholds as non-compliant multimodal medical data, delete them, and obtain the evaluated multimodal medical data. It should be noted that in this embodiment, the specific values ​​of the threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical imaging data, and the threshold W3 of the rationality value of the laboratory test result data are all 0.9, and the value selection method refers to the standards and specifications of medical examinations.

[0020] As an optional embodiment: for the pre-processed multimodal medical data, obtaining the rationality value of each medical image data The specific steps of 2 are as follows: For each piece of medical imaging data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the medical imaging data , where m is the number of images in the medical imaging data, It is the similarity between the i-th image in the medical image data and the standard image in the medical knowledge base, which is obtained by SURF feature matching calculation.

[0021] As an optional embodiment: for the pre-processed multimodal medical data, the rationality value of each laboratory test result data is obtained. The specific steps of 3 are as follows: For each laboratory test result data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the laboratory test result data , where k is the number of items in the laboratory test result data, is the result value of the ith item in the laboratory test result data, The mean value of the result of the ith item in the medical knowledge base, is the standard deviation of the result value of the i-th item in the medical knowledge base.

[0022] As an optional embodiment: the specific working steps of the modality fusion unit are as follows: First, the text data, image data, and numerical data in the evaluated multimodal medical data are spliced ​​together; The importance of each modality is given the same weight and weighted average is performed to complete the fusion of modalities and obtain the fused medical data.

[0023] As an optional embodiment: the specific working steps of the feature extraction unit are as follows: Obtain fused medical data; For the text data in the fused medical data, the LSTM network is used to extract the semantic features of the electronic medical record text data. For the image data in the fused medical data, the pre-trained convolutional neural network is used to extract the high-dimensional feature vector of the medical imaging data. For the laboratory test result data in the fused medical data, the statistical feature extraction is used to obtain the features of the laboratory test result data. It should be noted that the convolutional neural networks used for training include ResNet and VGG. All extracted features are transmitted to the decision reasoning module as key features.

[0024] As an optional embodiment: the specific working steps of the knowledge graph unit are as follows: First, the multimodal medical data collected by the data processing unit is obtained, and the medical-related entities are identified using natural language processing technology; it should be noted that, for example, symptoms, test items, diseases, drugs and surgeries; And normalize these entities to standard terminology; such as ICD-10 or SNOMED-CT to ensure consistency in the knowledge graph; Then, the relationship between entities is determined through relation extraction technology (such as dependency parsing or pre-trained language model BERT), such as the relationship between disease and symptoms, the relationship between drugs and diseases, etc. In addition, attribute values ​​are calculated for entities in the graph, such as frequency, intensity, etc., to enrich the information of the knowledge graph; After adding new data, new concepts are obtained and automatically added to the concept layer of the knowledge base to complete the construction of the knowledge graph; Then, key knowledge is extracted from the constructed medical knowledge graph, including entities, relationships, and attributes, and this knowledge is encoded into a structured representation. The entities and relationships are converted into vector representations using embedding technology, and then trained using a graph neural network model to generate a lightweight model, and the knowledge in the lightweight model is passed to the generation unit.

[0025] It should be noted that through these steps, the knowledge graph unit not only constructs an accurate and reliable medical knowledge graph, but also effectively transfers key knowledge to the generation unit through knowledge distillation, providing strong support for medical decision-making and improving the scientificity and accuracy of medical decision-making.

[0026] As an optional embodiment: the specific working steps of the generation unit are as follows: First, receiving key features from the feature extraction unit; it should be noted that these features are obtained through multimodal data fusion and feature extraction processes, are highly representative and relevant, and can provide important information support for medical decision-making; Initialize the built-in decision-making model; the model can be a machine learning-based classifier, regression model, or a deep learning model such as a neural network. The choice of model should be determined based on the specific medical task and data characteristics to ensure that the model can effectively process the input features and generate accurate decisions; The received key features are input into the model, and the model is used to analyze and process the input key features to generate medical decision information; this process involves the forward propagation of the model, and the features are comprehensively evaluated through the internal mechanism of the model, and finally the decision result is output. The decision result can be the diagnosis of the disease, the recommendation of the treatment plan, the calculation of the drug dosage, etc., depending on the needs of the medical task; Medical decision information is obtained and transmitted to the verification module.

[0027] As an optional embodiment: the specific working steps of the verification module are as follows: The medical decision information output by the generation unit is obtained, and evaluation indicators for evaluating the reliability of the medical decision information are determined, including accuracy, precision, recall, and F1 score, and a threshold value for each evaluation indicator is set; it should be noted that in this embodiment, the accuracy is the number of correct predictions divided by the total number of predictions, the precision is the number of true positive examples divided by the sum of true positive examples and false positive examples, the recall is the number of true positive examples divided by the sum of true positive examples and false negative examples, and the F1 score is 2 multiplied by , the threshold is the average value of historical data; Collect the expert diagnosis results of the same case as the gold standard, compare the decision results of the generation unit with the expert diagnosis results, and calculate the value G of each evaluation indicator; it should be noted that the specific steps are as follows: data alignment: ensure that the decision results of the generation unit and the expert diagnosis results are compared at the same time point and on the same case, calculate the evaluation indicators: use the above-set evaluation indicators (accuracy, precision, recall, F1 score) to calculate the consistency between the decision results of the generation unit and the expert diagnosis results, accuracy: calculate the proportion of the decision results of the generation unit that are consistent with the expert diagnosis results, precision: calculate the proportion of the decision results predicted by the generation unit to be positive that are actually positive, recall: calculate the proportion of the decision results predicted by the generation unit to be positive that are actually positive, F1 score: calculate the harmonic mean of precision and recall; Collect historical medical record data as a reference, compare the decision results of the generation unit with the diagnosis and treatment results of historical cases, and calculate the value H of each evaluation indicator; it should be noted that the historical medical record data may include: diagnosis results: diagnosis conclusions of similar cases in the past, treatment recommendations: treatment plans of similar cases in the past, drug dosage: drug dosage of similar cases in the past; The specific steps for calculating the value H of each evaluation indicator are as follows: Data alignment: ensure that the decision results of the generation unit and the diagnosis results of historical cases are compared at the same time point and on similar cases. Calculate evaluation indicators: use the above-set evaluation indicators (accuracy, precision, recall, F1 score) to calculate the consistency between the decision results of the generation unit and the diagnosis and treatment results of historical cases. Accuracy: calculate the proportion of the decision results of the generation unit that are consistent with the diagnosis results of historical cases. Precision: calculate the proportion of decision results predicted by the generation unit to be positive that are actually positive. Recall: calculate the proportion of decision results predicted by the generation unit to be positive that are actually positive. F1 score: calculate the harmonic mean of precision and recall. According to the formula , calculate and obtain the reliability value V of the medical decision information of the generating unit, and output the calculation result to the medical personnel.

[0028] How it works The data processing module is used to collect multimodal medical data and preprocess the collected multimodal medical data to obtain preprocessed multimodal medical data; it should be noted that the data processing module is responsible for collecting multimodal medical data, covering multi-source information such as electronic medical records, medical images, and laboratory test results. After the collection is completed, the data is preprocessed, including cleaning, standardization, missing value filling and other operations to ensure the integrity and consistency of the data. Subsequently, the preprocessed data is compared and verified with the medical knowledge base to evaluate the rationality of the data, and the data that does not conform to medical knowledge is deleted or corrected. Finally, the evaluated multimodal medical data is obtained and transmitted to the fusion extraction module. The purpose of this module is to ensure the quality and accuracy of the input data; The data processing module is also used to compare the pre-processed multimodal medical data with the existing medical knowledge base, verify the rationality of the data, and evaluate, delete the non-compliant multimodal medical data, obtain the evaluated multimodal medical data, and transmit the evaluated multimodal medical data to the fusion extraction module; The fusion extraction module includes a modality fusion unit and a feature extraction unit, wherein the modality fusion unit is used to receive the multimodal medical data evaluated by the data processing module and fuse the evaluated multimodal medical data to obtain fused medical data; The feature extraction unit is used to extract features from the fused medical data, obtain key features, and transmit them to the decision reasoning module; it should be noted that the modality fusion unit receives the evaluated multimodal medical data, and adopts a combination of early fusion, mid-term fusion and late fusion to effectively fuse the data of different modalities to generate fused medical data. The feature extraction unit uses advanced feature extraction algorithms based on the fused data, such as convolutional neural network (CNN) to extract medical image features, long short-term memory network (LSTM) to extract text features, etc., to accurately extract the most valuable key features for medical decision-making, and transmit these key features to the decision reasoning module. This module aims to provide high-quality input information for decision reasoning through the fusion and feature extraction of multimodal data; The decision reasoning module includes a knowledge graph unit and a generation unit. The knowledge graph unit is used to construct a medical knowledge graph, structure the representation and association of scattered medical knowledge, and distill it into the generation unit; The generation unit has a built-in model, and generates medical decisions based on the key features received by the feature extraction unit, obtains medical decision information, and transmits it to the verification module; it should be noted that the knowledge graph unit is responsible for constructing a medical knowledge graph, and structuredly represents and associates scattered medical knowledge, such as medical literature, clinical guidelines, expert experience, etc., and regularly updates it to ensure the latest and accuracy of knowledge, and organizes structured medical knowledge into the generation unit. The generation unit has built-in multi-task learning and reinforcement learning algorithms, and generates medical decision information, such as disease diagnosis results, treatment plan recommendations, etc., based on the received key features and structured medical knowledge, and transmits this information to the verification module, which can generate accurate and reliable medical decision information based on the extracted key features and rich medical knowledge; The verification module is used to evaluate the medical decision information of the generation unit, calculate its reliability, and output the calculation results to medical personnel. It should be noted that the verification module evaluates the medical decision information output by the generation unit, uses a variety of verification methods such as comparison with expert diagnosis results and comparison with historical cases, comprehensively calculates its reliability, and outputs the evaluation results to medical personnel to assist them in making the final decision. According to the verification results, the model is fed back and adjusted to continuously optimize the decision-making performance of the model, in order to ensure the rationality and reliability of the generated medical decision information and enhance the medical staff's trust in the system.

[0029] The above are only preferred implementations of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical staff in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of this template.

Claims

1. A medical decision-making support system based on a multimodal large model, characterized in that: It includes data processing module, fusion extraction module, decision reasoning module and verification module; The data processing module is used to collect multimodal medical data, specifically including electronic medical records, medical images and laboratory test results; and preprocess the collected multimodal medical data to obtain preprocessed multimodal medical data; The data processing module is also used to compare the pre-processed multimodal medical data with the existing medical knowledge base, verify the rationality of the data, and evaluate, delete the non-compliant multimodal medical data, obtain the evaluated multimodal medical data, and transmit the evaluated multimodal medical data to the fusion extraction module; The fusion extraction module includes a modality fusion unit and a feature extraction unit, wherein the modality fusion unit is used to receive the multimodal medical data evaluated by the data processing module and fuse the evaluated multimodal medical data to obtain fused medical data; The feature extraction unit is used to extract features from the fused medical data, obtain key features, and transmit them to the decision reasoning module; The decision reasoning module includes a knowledge graph unit and a generation unit. The knowledge graph unit is used to construct a medical knowledge graph, structure the representation and association of scattered medical knowledge, and distill it into the generation unit; The generation unit has a built-in model, generates a medical decision based on the key features received from the feature extraction unit, obtains medical decision information, and transmits it to the verification module; the verification module is used to evaluate the medical decision information of the generation unit, calculate its reliability, and output the calculation results to medical personnel.

2. According to claim 1, a medical decision-making support system based on a multimodal large model is characterized in that: The specific working steps of the data processing module are as follows: First, multimodal medical data is collected; Duplicate records were then identified by examining unique identifiers in the multimodal medical data; Keep one unique record and delete other duplicates, while identifying and removing fields that are not relevant to medical decision making; Normalize all numerical data in multimodal medical data, and use the median of the same type of data to fill in missing values ​​in numerical data; For all text data in multimodal medical data, the text data is segmented, long texts are split into words or phrases, stop words in the text are removed, and words are restored to their stem forms; for all imaging data in multimodal medical data, the values ​​of missing data points are calculated by linear interpolation based on known data points, or the values ​​of the nearest data points are used to fill in the values ​​of missing data points; The preprocessing is completed to obtain preprocessed multimodal medical data.

3. The medical decision-making assistance system based on a multimodal large model according to claim 2, characterized in that: The data processing module verifies the rationality of the data and evaluates the specific working steps as follows: For each electronic medical record data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the electronic medical record data , where n is the number of records in the electronic medical record data, is the matching degree between the ith understanding in the electronic medical record data and the corresponding entry in the medical knowledge base, which is obtained by calculating the cosine similarity; For the preprocessed multimodal medical data, obtain the rationality value of each medical image data 2; For the preprocessed multimodal medical data, obtain the rationality value of each laboratory test result data 3; The threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical image data, and the threshold W3 of the rationality value of the laboratory test result data are set in advance. , the rationality value of each medical imaging data 2 and the reasonableness value of each laboratory test result data 3. Compare the threshold W1 of the rationality value of the electronic medical record data, the threshold W2 of the rationality value of the medical imaging data, and the threshold W3 of the rationality value of the laboratory test result data respectively, mark all data whose comparison results are less than the corresponding thresholds as non-compliant multimodal medical data, delete them, and obtain the evaluated multimodal medical data.

4. The medical decision-making support system based on a multimodal large model according to claim 3 is characterized in that: For the pre-processed multimodal medical data, the rationality value of each medical image data is obtained. The specific steps of 2 are as follows: For each piece of medical imaging data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the medical imaging data , where m is the number of images in the medical imaging data, It is the similarity between the i-th image in the medical image data and the standard image in the medical knowledge base, which is obtained by SURF feature matching calculation.

5. The medical decision-making assistance system based on a multimodal large model according to claim 3 is characterized in that: For the pre-processed multimodal medical data, the rationality value of each laboratory test result data is obtained. The specific steps of 3 are as follows: For each laboratory test result data in the preprocessed multimodal medical data, according to the formula; , calculate and obtain the rationality value of the laboratory test result data , where k is the number of items in the laboratory test result data, is the result value of the ith item in the laboratory test result data, The mean value of the result of the i-th item in the medical knowledge base, is the standard deviation of the result value of the i-th item in the medical knowledge base.

6. The medical decision-making support system based on a multimodal large model according to claim 1, characterized in that: The specific working steps of the modality fusion unit are as follows: First, the text data, image data, and numerical data in the evaluated multimodal medical data are spliced ​​together; The importance of each modality is given the same weight and weighted average is performed to complete the fusion of modalities and obtain the fused medical data.

7. The medical decision-making assistance system based on a multimodal large model according to claim 6, characterized in that: The specific working steps of the feature extraction unit are as follows: Obtain fused medical data; For the text data in the fused medical data, the LSTM network is used to extract the semantic features of the electronic medical record text data. For the image data in the fused medical data, the pre-trained convolutional neural network is used to extract the high-dimensional feature vectors of the medical imaging data. For the laboratory test result data in the fused medical data, the statistical feature extraction is used to obtain the features of the laboratory test result data. All extracted features are transmitted to the decision reasoning module as key features.

8. The medical decision-making assistance system based on a multimodal large model according to claim 1, characterized in that: The specific working steps of the knowledge graph unit are as follows: Firstly, the multimodal medical data collected by the data processing unit is obtained, and medical-related entities are identified using natural language processing technology; and normalize these entities to standard terminology; Then, the relationship between entities is determined through relationship extraction technology; In addition, attribute values ​​are calculated for entities in the graph; After adding new data, new concepts are obtained and automatically added to the concept layer of the knowledge base to complete the construction of the knowledge graph; Then, key knowledge is extracted from the constructed medical knowledge graph, including entities, relationships, and attributes, and this knowledge is encoded into a structured representation. The entities and relationships are converted into vector representations using embedding technology, and then trained using a graph neural network model to generate a lightweight model, and the knowledge in the lightweight model is passed to the generation unit.

9. The medical decision-making assistance system based on a multimodal large model according to claim 8, characterized in that: The specific working steps of the generation unit are as follows: Firstly, receiving key features from the feature extraction unit; Initialize the built-in decision generation model; Input the received key features into the model, use the model to analyze and process the input key features, and generate medical decision information; Medical decision information is obtained and transmitted to the verification module.

10. The medical decision-making support system based on a multimodal large model according to claim 1, characterized in that: The specific working steps of the verification module are as follows: Obtaining the medical decision information output by the generation unit, determining evaluation indicators for evaluating the reliability of the medical decision information, specifically including accuracy, precision, recall rate, F1 score, and setting a threshold for each evaluation indicator; Collect the expert diagnosis results of the same case as the gold standard, compare the decision results of the generation unit with the expert diagnosis results, and calculate the value G of each evaluation indicator; Collect historical medical record data as a reference, compare the decision results of the generation unit with the diagnosis and treatment results of historical cases, and calculate the value H of each evaluation indicator; According to the formula , calculate and obtain the reliability value V of the medical decision information of the generating unit, and output the calculation result to the medical personnel.

Images