Medical diagnosis system based on chain reasoning and multi-modal fusion and application thereof

Abstract

The invention provides a medical diagnosis system based on chain reasoning and multi-modal fusion and application thereof, and belongs to the technical field of medical care informatics. The system comprises a data preprocessing module, a data fusion module, a model reasoning module and an output module, and can perform deep processing and intelligent analysis on multi-modal medical data (including text data, medical image data and structured data) of a patient. And the output module provides a diagnosis result and a detailed reasoning process, so that the interpretability and the credibility of the diagnosis result are improved. Through multi-modal data fusion, a chained reasoning mechanism and optimized loss function design, the accuracy, generalization ability and interpretability of disease diagnosis are remarkably improved, the limitation of the traditional technology in the aspects of single-modal analysis and black box reasoning is broken through, transparent, efficient and intelligent diagnosis decision support is provided for clinicians, and the clinical application prospect is wide. And intelligent and precise development of medical diagnosis is promoted.

Description

Technical Field

[0001] The present invention relates to a medical diagnosis system based on chain reasoning and multimodal fusion and application thereof, belonging to the technical field of medical care informatics. Background Art

[0002] With the rapid development of information technology and medical technology, the medical industry has generated a massive amount of multimodal data, including electronic medical records (EMR), medical images (such as X-rays, CT, MRI), laboratory test results, and patients' subjective symptom descriptions. These data contain rich patient information and are of great value for disease diagnosis, treatment, and prognosis assessment. However, how to efficiently integrate and analyze these multimodal medical data has become a major challenge facing the current medical field.

[0003] Traditional medical diagnosis mainly relies on the doctor's experience and single-modality data, such as making a diagnosis based only on the patient's symptoms or a single test result. This method makes it difficult to make full use of all available information in a timely and comprehensive manner, resulting in limited accuracy and efficiency of diagnosis.

[0004] In recent years, breakthroughs in artificial intelligence technology, especially deep learning and natural language processing (NLP) technology, have provided new possibilities for solving the above problems. Large language models (LLMs) have demonstrated excellent capabilities in text understanding and generation, and can handle complex language structures and semantic relationships.

[0005] Although there have been attempts to apply AI to medical diagnosis, there are still some major drawbacks: (1) Difficulty in fusing multimodal data: Existing diagnostic models usually target data of a single modality and lack effective fusion of multimodal data. This results in the model being unable to fully utilize the correlation between different types of data, limiting the accuracy of diagnosis. Specifically, there is a lack of a unified multimodal representation method. Data of different modalities (such as text, images, and structured data) have different feature spaces and data structures, and direct fusion is technically difficult. In addition, each modality of data may contain repeated, similar, or conflicting information. How to effectively remove redundancy and resolve conflicts remains an urgent problem to be solved.

[0006] (2) Lack of deep semantic understanding and reasoning capabilities: Most existing models rely on shallow feature extraction and statistical analysis and are unable to deeply understand the complex semantics and causal relationships in medical texts and images. Traditional NLP methods perform poorly when dealing with professional medical terminology, complex sentence structures, and implicit semantic relationships, and have difficulty accurately extracting key information. In addition, existing models lack modeling of the causal relationship between symptoms, signs, and diseases, and are unable to simulate the doctor's diagnostic thinking process, resulting in a lack of credibility and interpretability in diagnostic recommendations.

[0007] (3) The model has weak generalization ability and insufficient diagnostic accuracy: Due to individual differences among patients and the complexity of the disease, the model needs to have good generalization ability. However, existing models often perform unstably under different data sets and scenarios, and their diagnostic accuracy and reliability are insufficient. Specifically, the model may overfit the training data and fail to adapt to new patient data or unseen case types; at the same time, the model lacks sufficient training data when dealing with rare diseases and special cases, resulting in low diagnostic accuracy.

[0008] (4) Lack of explainability and credibility: Medical diagnosis is related to the life safety of patients, and the decision-making process of the model needs to be transparent and explainable. However, many deep learning-based models are "black box" models that cannot provide understandable diagnostic reasons, limiting their application in clinical practice. Doctors and patients have a low level of understanding and trust in diagnostic results, which hinders the widespread application of artificial intelligence in the medical field.

[0009] (5) Lack of sufficiently effective large-scale medical diagnosis models, especially large-scale models with multimodal data fusion and chain reasoning capabilities: Although existing large-scale language models (such as GPT-4 and other open source LLMs) perform well in general natural language understanding and generation tasks, they face significant limitations in the field of medical diagnosis. On the one hand, these models have not been fine-tuned specifically for the medical field, and it is difficult to accurately identify and understand complex professional medical terms, clinical descriptions, and diagnosis and treatment logic, resulting in their inability to meet the needs of efficient and accurate diagnosis in actual clinical scenarios. On the other hand, traditional medical diagnosis models usually rely on single-modal data (such as text, medical images, or structured data), lack effective fusion of multimodal data, and cannot fully integrate multi-dimensional information such as the patient's medical history, symptoms, imaging examinations, and laboratory results, thereby limiting the accuracy and comprehensiveness of the diagnosis. In addition, existing models generally lack chain reasoning capabilities and cannot simulate the doctor's diagnostic thinking of step-by-step analysis and deduction. In the process of diagnosis, doctors often form a set of logically clear and causal reasoning chains by step-by-step analyzing information such as medical history, symptoms, and examination results to reach a diagnosis conclusion. However, current diagnostic models mainly rely on shallow feature extraction and single-step prediction, lack the ability to model the causal relationship between disease symptoms, and are unable to generate a diagnostic reasoning chain that conforms to clinical logic. This leads to a lack of transparency and credibility in diagnostic results, making it difficult to provide effective decision support for clinicians and gain the trust of patients.

[0010] In summary, existing technologies have significant deficiencies in multimodal data fusion, deep semantic understanding, reasoning ability, generalization ability, chain reasoning medical diagnosis and interpretability. Traditional diagnostic models lack effective integration of multimodal data and cannot fully capture patient information; although existing large language models have language understanding capabilities, they lack effective fine-tuning for medical diagnosis tasks and are difficult to meet clinical high-precision requirements. In addition, the lack of support for chain reasoning mechanisms makes it impossible for existing models to simulate the doctor's step-by-step deduction logic, and the reasoning process lacks transparency and credibility. Therefore, a new technical solution is urgently needed to break through the current bottleneck, achieve efficient, accurate, and explainable automated disease diagnosis, and promote the development of medical intelligence to a new level. Summary of the invention

[0011] The present invention provides a medical diagnosis system based on chain reasoning and multimodal fusion and its application, which can effectively solve the above problems.

[0012] The present invention is achieved in that: A medical diagnosis system based on chain reasoning and multimodal fusion, comprising: A data preprocessing module, used to preprocess the multimodal medical data of the patient and generate feature representations of each multimodal medical data; Data fusion module, used to fuse the feature representations of each multimodal medical data to form a unified multimodal feature vector ; Model inference module for multimodal feature vectors using a trained large language model Perform inference and generate diagnostic results and chains of reasoning; Output module, used to output diagnostic results and chains of reasoning; The multimodal medical data includes text data , image data I and structured data S; For text data The preprocessing is as follows: first, the text data is cleaned to remove noise and irrelevant information, then the text is segmented into words or subword sequences using the tokenizer of the pre-trained language model, and the embedding function is used to Map it into word vector representation , ,in, It captures the semantic and syntactic information of the text; The preprocessing of image data I is as follows: first, the image data is standardized, resized and denoised, and then a pre-trained convolutional neural network is used. Extract features from the image and obtain image feature representation , , Contains the spatial structure and texture features of the image; The preprocessing of structured data S is as follows: structured data S is processed by data standardization, and the numerical data is normalized to make its mean 0 and standard deviation 1; the categorical data is encoded by One-Hot Encoding to obtain the standardized structured data representation. , ; In some embodiments, the multimodal feature vector +w2 +w3 ,in, represents the feature concatenation operation, represents the attention mechanism, , w2, w3 are weight coefficients of weighted summation, and the multimodal feature vector M contains comprehensive information of the patient.

[0013] A chain thinking reasoning mechanism is introduced into the large language model, and its structure is designed to be an input layer, a hidden layer and an output layer; The input layer receives the fused multimodal feature vector , initialize the hidden state , where the initial implicit state Contains all the patient's information; The hidden layer uses the large language model Layer, the implicit state is updated multiple times iteratively to form a chain reasoning process. For the t-th step reasoning, the implicit state is updated to ,in, is the hidden state at step t, It is the tth Layer, including multi-head self-attention mechanism and feed-forward neural network; The output layer is the last step n, through linear transformation and Function, generate diagnostic results , ,in, is the weight matrix, is the bias vector, is the hidden state of the nth step, The function converts the output of the linear transformation into a probability distribution and obtains the predicted probability of each disease category.

[0014] In some embodiments, the text data It includes one or more of the patient's past medical history text information, chief complaint text information, symptom description text information and doctor's medical record text information; the image data I is medical imaging information; the structured data S includes numerical or categorical data of medical examinations.

[0015] In some embodiments, the large language model is one or more of GPT-4, Llama, Bio_ClinicalBERT, RoBERTa, XLNet, MedX_v1, and Medicine-LLM-13B-GGUF.

[0016] In some embodiments, the training of the large language model introduces transfer learning or federated learning.

[0017] In some embodiments, the training of the large language model further defines a loss function, including a loss function for the diagnosis result. diagnosis And the consistency loss function of the reasoning process COT ; The loss function of the diagnosis result diagnosis for ,in, is the total number of disease categories, is the true label components, whose values ​​are 0 or 1, The model predicts The probability of each category; The consistency loss function of the inference process COT for ,in, represents the square norm of a vector, is an ideal inference function, which transforms the implicit state of the t-1th step to obtain the ideal state of the tth step; Total loss function ,in, and is a weight hyperparameter used to balance the impact of the two parts of the loss.

[0018] In some embodiments, during the training of the large language model, an Adam optimizer is used to update parameters.

[0019] In some embodiments, the ideal inference function Assume it is an identity mapping, that is , indicating that the implicit states of adjacent reasoning steps are expected to transition smoothly.

[0020] In some embodiments, the multimodal medical data further includes genomic data and metabolomics data.

[0021] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the functions of the medical diagnosis system based on chain reasoning and multimodal fusion are realized.

[0022] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the functions of the medical diagnosis system based on chain reasoning and multimodal fusion.

[0023] The beneficial effects of the present invention are: The present invention adopts different preprocessing methods for different modal medical data, including text, image and structured data, to obtain word vector representations respectively. , Image feature representation and structured data representation The multimodal feature vector is obtained by combining feature concatenation, attention mechanism and weighted summation, which enables a comprehensive analysis of patient information and significantly improves the efficiency and accuracy of disease screening. This multimodal data fusion technology enables the system to comprehensively analyze patient information from different angles and dimensions, improving the comprehensiveness and accuracy of diagnosis.

[0024] The present invention proposes a large language model system that integrates multimodal data fusion and chain reasoning capabilities, breaking through the bottleneck of current technology. First, the system efficiently fine-tunes the large language model for special data sets in the medical field, enabling it to understand and process complex medical semantics, professional terms, and clinical texts, thereby improving the accuracy and adaptability of diagnosis. Secondly, the system innovatively introduces multimodal data fusion technology to deeply fuse the patient's text data, medical imaging data, and structured data to generate a unified multimodal feature vector, comprehensively integrating information under different modalities, and effectively improving the comprehensiveness and accuracy of diagnosis.

[0025] More importantly, the present invention simulates the doctor's step-by-step reasoning of diagnostic thinking by introducing a chain thinking reasoning mechanism (COT), realizes the step-by-step deduction and decision-making from symptoms to diseases, and generates a clear and explainable diagnostic reasoning chain. This reasoning chain not only improves the logic and transparency of the diagnostic results, but also provides doctors with a detailed reasoning process, which helps to understand and verify the diagnostic basis of the model, and enhances the credibility and clinical applicability of the diagnostic results.

[0026] In summary, the present invention solves the problems of insufficient fine-tuning of existing models in diagnostic tasks, lack of multimodal processing and chain reasoning capabilities, and provides an efficient, accurate and explainable intelligent medical diagnosis solution by constructing a large language model system with multimodal fusion and chain reasoning capabilities. This innovative solution will greatly improve the accuracy and reliability of medical diagnosis, provide strong decision-making support for clinicians, and promote the implementation and popularization of intelligent diagnosis technology in the medical field, which has important clinical application value and social influence.

[0027] The present invention combines the Large Language Model (LLM) and Chained Inference Technology (COT) to enhance the model's ability to deeply understand the semantics of medical texts and images, and simulate the doctor's diagnostic thinking process to perform complex causal reasoning. This not only improves the logic and scientificity of diagnostic recommendations, but also enhances the credibility and reliability of diagnostic results.

[0028] The present invention enhances the generalization ability and robustness of the model by adapting the model training method and loss function design to the medical field. The model can maintain a high level of diagnostic performance under different data sets and clinical scenarios, reduces dependence on specific data sets, and improves the versatility and applicability of the system.

[0029] The present invention utilizes the COT reasoning mechanism, and provides detailed reasoning process and reasons while generating the diagnosis results. This transparent decision-making process increases the doctor's trust in the model, and helps to discover and correct possible errors in the model, ensuring the accuracy and safety of the diagnosis.

[0030] The present invention realizes an automated disease diagnosis process, significantly improving the speed and efficiency of diagnosis. The model can quickly process a large amount of patient data, reduce the workload of doctors, and improve the efficiency of medical resource utilization. Through automated data analysis and diagnostic suggestions, the system can complete complex diagnostic tasks in a short time, meeting the dual needs of medical services for efficiency and accuracy.

[0031] The present invention comprehensively improves the speed and accuracy of medical diagnosis, reduces the workload of doctors, and improves the utilization efficiency of medical resources, thereby providing patients with better-quality and more personalized medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 Schematic diagram of the structure of a medical diagnosis system based on chain reasoning and multimodal fusion according to an embodiment of the present invention.

[0034] Figure 2 This is the pseudo code of the algorithm of an embodiment of the present invention, showing the complete process from data preprocessing to model inference. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0037] Embodiment 1 The embodiment of the present invention provides a medical diagnosis system based on chain reasoning and multimodal fusion, such as Figure 1 As shown in the figure, the medical diagnosis system based on chain reasoning and multimodal fusion includes: The data preprocessing module is used to preprocess the multimodal medical data of the patient and generate feature representations of each multimodal medical data; the multimodal medical data includes text data, image data I and structured data S. The purpose of preprocessing is to generate feature representations of each multimodal medical data so that subsequent modules can process these data more effectively.

[0038] Data fusion module, used to fuse the feature representations of each multimodal medical data to form a unified multimodal feature vector ; This vector It contains information about all relevant medical data and provides a basis for subsequent model reasoning.

[0039] Model inference module for multimodal feature vectors using a trained large language model Perform inference and generate diagnostic results and chains of reasoning.

[0040] Output module, used to output diagnostic results and chains of reasoning.

[0041] The large language model (LLM) is a pre-trained large language model (Med-LLM) suitable for the medical field, such as GPT-4, Llama, Bio_ClinicalBERT, RoBERTa, XLNet, MedX_v1, Medicine-LLM-13B-GGUF and other models that have been fine-tuned in the medical field. Different models have their own characteristics in performance and resource consumption. Selecting the most suitable model can optimize the overall performance of the system and meet the needs of specific scenarios. For example, BERT performs well in sentence-level understanding, while the GPT series is more prominent in generation capabilities. According to the specific medical application scenarios, selecting the most suitable model can further improve the accuracy and efficiency of diagnosis. These models perform well in processing complex language structures and understanding professional terms. The selected LLM is further fine-tuned to enable it to better understand medical texts and professional terms. During the fine-tuning process, a large amount of medical text data is used for training to enhance the model's ability to capture medical semantics.

[0042] In some embodiments, the text data It includes one or more of the patient's past medical history text information, chief complaint text information, symptom description text information and doctor's medical record text information; the image data I is medical imaging information, such as the patient's X-ray, CT, MRI, etc.; the structured data S includes numerical or categorical data of medical examinations, such as the patient's physiological indicators, laboratory test results and vital signs parameters.

[0043] In some embodiments, the text data The preprocessing is as follows: first, the text data is cleaned to remove noise and irrelevant information, such as special characters (such as \t, \b, \n), stop words and repeated content, and then the text is segmented into words or subword sequences using the tokenizer of the pre-trained language model, and the embedding function is used to extract the text. Map it into word vector representation , ,in, It captures the semantic and syntactic information of the text; The preprocessing of image data I is as follows: first, the image data is standardized, resized and denoised, and then a pre-trained convolutional neural network is used. Extract features from the image and obtain image feature representation , , Contains the spatial structure and texture features of the image; The preprocessing of structured data S is as follows: structured data S is processed by data standardization, and the numerical data is normalized to make its mean 0 and standard deviation 1; the categorical data is encoded by One-Hot Encoding to obtain the standardized structured data representation. , ; In some embodiments, the multimodal feature vector , where it concats the feature vectors of different modalities to form a comprehensive feature representation. Attention represents the attention mechanism, which can dynamically adjust the weights of each feature vector according to different tasks and contexts to highlight important feature information. w1, w2, and w3 are the corresponding weighted summation weight coefficients, which are used to adjust the contribution of each modality feature vector in the final multimodal feature vector M. In this way, the multimodal feature vector M can effectively integrate a variety of patient information, including text information, image information, and sound information, thereby providing a comprehensive and rich feature representation for subsequent analysis and processing.

[0044] A chain thinking reasoning mechanism is introduced into the large language model, and its structure is designed to be an input layer, a hidden layer and an output layer.

[0045] The input layer receives the fused multimodal feature vector , initialize the hidden state , where the initial implicit state Contains all the patient's information; The hidden layer uses the large language model Layer, the implicit state is updated multiple times iteratively to form a chain reasoning process. For the t-th step reasoning, the implicit state is updated to ,in, is the hidden state at step t, It is the tth Layer, including multi-head self-attention mechanism and feed-forward neural network; The output layer is the last step n, through linear transformation and Function, generate diagnostic results , ,in, is the weight matrix, is the bias vector, is the hidden state of the nth step, The function converts the output of the linear transformation into a probability distribution and obtains the predicted probability of each disease category.

[0046] In order to simulate the diagnostic thinking process of doctors, the embodiment of the present invention introduces a chain thinking (COT) reasoning mechanism. The disease diagnosis process is regarded as a multi-step reasoning process, and each step of reasoning is based on the result of the previous step, gradually approaching the final diagnosis. This mechanism enables the system to be closer to the diagnostic logic of human doctors, thereby improving the accuracy and reliability of the diagnosis results.

[0047] In some embodiments, the training of the large language model further defines a loss function, including a loss function for the diagnosis result. diagnosis And the consistency loss function of the reasoning process COT ; The loss function of the diagnosis result diagnosis for ,in, is the total number of disease categories, is the true label components, whose values ​​are 0 or 1, The model predicts The loss function of the diagnosis result (classification loss) is to use the cross entropy loss function to measure the difference between the diagnosis result predicted by the model and the true label.

[0048] The consistency loss function of the inference process COT for ,in, represents the square norm of a vector, is an ideal inference function, which transforms the implicit state of the t-1th step to obtain the ideal state of the tth step. The consistency loss function of the inference process ensures the rationality of the model's inference process.

[0049] Total loss function ,in, and is a weight hyperparameter used to balance the impact of the two parts of the loss.

[0050] In some specific embodiments, for the training process of the large language model, we use the Adam optimizer to update the model parameters. The entire training step can be described in detail as follows: First, we conduct the forward propagation phase: In this phase, we feed the input data into the model and calculate the output of the model. , and at the same time record the implicit state information of each step. This implicit state information is crucial for the subsequent training process because it contains the intermediate representation of the model when processing input data.

[0051] Next is the loss calculation phase: In this phase, we convert the model output Compared with the true label, a total loss value or error value is calculated through a loss function (such as the cross entropy loss function). We use the symbol To represent this total loss function. The value of this loss function reflects the gap between the current model output and the true label, which is the object we need to minimize in the subsequent optimization process.

[0052] Then comes the back propagation phase: In this phase, we calculate the total loss function sign To calculate the gradient of loss to model parameters. This gradient information tells us the direction and magnitude of the change of loss function with model parameters, which is an important basis for us to update parameters.

[0053] The last is the parameter update phase: In this phase, we update the model parameters based on the gradient information calculated previously. Specifically, we use the Adam optimizer to adjust these parameters so that the loss function sign is The value of can change in the direction of decrease. Adam optimizer is an optimization algorithm with adaptive learning rate. It combines the advantages of momentum and RMSprop optimization algorithms and can effectively handle the training problems of large-scale deep learning models.

[0054] Through the iterative cycles of the above four stages, we can gradually optimize the model parameters so that the output of the model is closer and closer to the true label, thereby achieving the purpose of training a large language model.

[0055] By adjusting the hyperparameters (such as learning rate, , etc.), optimize model performance and prevent overfitting or underfitting.

[0056] In addition, model training can also introduce training methods such as transfer learning and federated learning to utilize more data resources and protect data privacy. While improving model performance, it can also meet the requirements of data security and privacy protection and adapt to the particularity of medical data. Transfer learning can use existing pre-trained models to quickly adapt to new tasks, reduce training time and data requirements; federated learning protects patient data privacy through distributed training and complies with data security regulations in the medical industry. In addition, these methods can make full use of cross-institutional data resources, improve the generalization ability and robustness of the model, and enhance the practicality and credibility of the system in practical applications.

[0057] In some specific embodiments, we set the ideal inference function is an identity map, which means , such a setting is intended to ensure that the implicit state between adjacent reasoning steps can achieve a smooth transition. In other words, we hope that the implicit state can maintain a certain degree of coherence and stability in consecutive reasoning steps.

[0058] In addition, we can also obtain the ideal inference function in the following ways: : Predefined rules: First, we set a series of inference rules based on medical knowledge and diagnostic logic. These rules are summarized based on professional knowledge and experience and can reflect the logical relationship in medical diagnosis. Then, we encode these rules into a function form for use in the inference process. In this way, we can ensure that the inference process meets the standards and requirements of the medical field.

[0059] Teacher Model: We can also use a good teacher model to generate ideal reasoning steps. The teacher model is usually a well-trained and well-performing model that can provide high-quality reasoning results. By imitating the reasoning steps of the teacher model, the student model can learn how to make effective reasoning. The advantage of this method is that the teacher model can provide a high-quality reference standard to help the student model quickly improve its reasoning ability.

[0060] Labeled data: We can also use manually labeled intermediate reasoning results to train the model. In this case, experts or labelers manually label the key steps and intermediate results in the reasoning process to form a set of high-quality labeled data. By using this labeled data to train the model, the model can learn the ideal reasoning process, thereby better simulating the reasoning ability of human experts in practical applications. The advantage of this method is that the labeled data can provide detailed reasoning steps and logical relationships, which helps the model better understand complex reasoning processes.

[0061] In some embodiments, data modalities are increased or decreased according to specific diagnostic tasks. For example, for genetic diseases, genomic data can be introduced; for metabolic diseases, metabolomics data can be introduced. Flexible modality combinations can provide more sophisticated and accurate diagnosis for specific diseases. Different disease types may depend on different data types. By flexibly adjusting the data modality, the system can better capture disease characteristics and improve the pertinence and effectiveness of diagnosis. For example, the diagnosis of genetic diseases requires the support of genetic data, while metabolic diseases require the analysis of metabolite data.

[0062] Among them, genomic data includes the following data: DNA sequencing data: Whole genome sequencing (WGS) or whole exome sequencing (WES) of an individual's genome can provide information about gene mutations, structural variations, single nucleotide polymorphisms (SNPs), insertions / deletions, etc. Gene expression data: RNA sequencing (RNA-Seq) or microarray technology can be used to assess changes in gene expression levels and identify genes that may be associated with disease; Mutation analysis: For example, analyzing mutations in specific genes can help identify mutation sites associated with genetic diseases.

[0063] The preprocessing of genomic data before multimodal data fusion includes data quality control, sequence alignment, variant detection and filtering, variant annotation, data standardization and format conversion, etc. Through these preprocessing steps, it can be ensured that genomic data can be effectively combined with other data (such as clinical data, metabolomics data, etc.) in multimodal fusion, thus laying a solid foundation for subsequent analysis and modeling.

[0064] Among them, metabolomics data include the following data: Metabolite concentration data: Mass spectrometry (MS) or nuclear magnetic resonance (NMR) techniques are used to measure the concentrations of thousands of metabolites in blood, urine, or other body fluids, such as amino acids, fatty acids, sugars, etc. Metabolic network analysis: Through the construction of metabolic pathways and networks, we can understand how metabolites interact with each other and which metabolic pathway abnormalities are related to diseases; Metabolic fingerprint analysis: By comparing the metabolic fingerprints of different samples, the characteristics of metabolic diseases can be revealed, which is especially useful for early detection.

[0065] The preprocessing of metabolomics data before multimodal data fusion includes data quality control, noise removal, batch effect correction, peak alignment and standardization, feature selection and dimensionality reduction, data standardization and normalization, format conversion, etc. Through these preprocessing methods, metabolomics data can be effectively integrated with data from other modalities (such as genomic data and clinical data), providing a solid foundation for subsequent multimodal analysis.

[0066] The fusion of the genomic data and metabolomics data is performed by using the above-mentioned Concat splicing, Attention mechanism and weighted summation method to form a composite hybrid fusion method, which can overcome the difficulty of fusing multimodal data in the prior art.

[0067] The inference process of the model inference module is as follows: after the model training is completed, new patient multimodal data X is input, and the fusion feature M is obtained after preprocessing. Then, the inference is performed according to the following steps: Initial state: Set the initial implicit state to: .

[0068] Iterative reasoning: For each reasoning step , update the implicit state: ,At each step, the model can generate intermediate reasoning reasons and conclusions to form a reasoning chain.

[0069] Generate diagnostic results: In the final step, the output layer is used to generate diagnostic results , by selecting the disease category with the highest probability, the final disease diagnosis is obtained.

[0070] Output reasoning chain: Provide the intermediate results and reasoning reasons of the entire reasoning process to form a complete reasoning chain for doctors' reference and improve the interpretability of diagnostic results.

[0071] The following is a specific example, in which the following patient information is obtained and the medical diagnosis system based on chain reasoning and multimodal fusion according to an embodiment of the present invention is used to diagnose the patient's disease, including input, generated reasoning chain and final output.

[0072] Patient information includes text data (T), image data (I) and structured data (S). Text data (T) includes the following: Medical history: The patient has a history of hypertension and there are cases of diabetes in the family.

[0073] Chief complaint: Frequent headaches and blurred vision over the past month.

[0074] Symptom description: The headache is mainly located in the forehead, the pain is moderate, and there is no obvious nausea or vomiting.

[0075] Doctor's medical record: The patient's blood pressure is poorly controlled and further examination is recommended.

[0076] Image data (I) is: MRI scan image of the brain, showing mild ventriculomegaly.

[0077] Structured data (S) include physiological indicators: blood pressure: 150 / 95mmHg, heart rate: 80bpm, blood sugar: 6.5mmol / L; laboratory test results: blood lipids: total cholesterol 6.2mmol / L, hemoglobin: 13.5g / dL.

[0078] The diagnostic process is as follows: Step 1: Data Preprocessing Text processing: Clean text data and remove irrelevant information. Use word segmentation and embedding functions to generate text embeddings. T .

[0079] Image processing: Normalize and denoise the brain MRI images. Use pre-trained CNN to extract image features F 1 .

[0080] Structured data processing: Normalize numerical data such as blood pressure, heart rate, and blood sugar. Perform unique hot encoding on categorical data to obtain standardized structured data Ns.

[0081] Data Fusion: Multimodal Feature Vectors .

[0082] Step 2: Model Inference initialization:

[0083] Iterative reasoning (assuming the number of reasoning steps n=3): Step 1:

[0084] Generate reasoning 1: Reasoning 1 = "The patient has a history of hypertension and poor blood pressure control, which may lead to abnormal blood flow to the brain." Step 2:

[0085] Generate reasoning 2: Reasoning 2 = "Brain MRI shows mild ventriculomegaly. Combined with the history of hypertension, chronic hypertensive encephalopathy is considered." Step 3:

[0086] Generate Reasoning 3: Reasoning 3 = "The patient's symptoms, such as frequent headaches and blurred vision, further support the diagnosis of chronic hypertensive encephalopathy." Generate diagnostic results: .

[0087] Assume that the diagnostic probabilities output by the model are as follows: .

[0088] Corresponding disease categories: 1. Migraine; 2. Chronic hypertensive encephalopathy: 80%; 3. Retinopathy: 5%; 4. Others: 5%.

[0089] Generate reasoning chain: Reasoning chain = "reasoning reason 1, reasoning reason 2, reasoning reason 3" The chain of reasoning is: 1. "The patient has a history of hypertension and poorly controlled blood pressure, which may lead to abnormal blood flow to the brain." 2. "Brain MRI showed mild ventriculomegaly, and combined with the history of hypertension, chronic hypertensive encephalopathy was considered." 3. "The patient's symptoms, such as frequent headaches and blurred vision, further support the diagnosis of chronic hypertensive encephalopathy." Step 3: Output the results Diagnosis result: Final diagnosis: chronic hypertensive encephalopathy (80% probability).

[0090] The chain of reasoning is as above.

[0091] This reasoning chain provides a detailed reasoning process to help doctors understand the diagnostic basis of the model and enhance the credibility and interpretability of the diagnostic results.

[0092] Through the above diagnostic process, the method of the present invention successfully integrates multimodal medical data, uses the language understanding ability of the medical diagnosis big model and the reasoning mechanism of chain reasoning based on the big model, generates a detailed reasoning chain, and finally outputs a high-probability diagnosis result. This not only improves the accuracy and efficiency of diagnosis, but also provides doctors with an interpretable basis for diagnosis, and enhances the practicality and credibility of the model in clinical applications.

[0093] Embodiment 2 An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the functions of the medical diagnosis system based on chain reasoning and multimodal fusion of one of the embodiments are implemented.

[0094] The computer program described in the embodiments of the present invention may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments that can implement specific functions.

[0095] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0096] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0097] Embodiment 3 An embodiment of the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the functions of the medical diagnosis system based on chain reasoning and multimodal fusion according to one of the embodiments are implemented.

[0098] Any entity or device of the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0099] In summary, the embodiment of the present invention proposes a multimodal medical large model automated disease diagnosis method based on large language model (LLM) and chain of thought (COT) reasoning technology. The method is designed to process various types of medical data, including but not limited to text, images, structured data, etc., to achieve efficient and accurate automated analysis and auxiliary decision support.

[0100] First, the present invention can be applied to the field of disease screening and early diagnosis. By comprehensively analyzing the patient's symptom description, medical images and laboratory test results, the system can identify potential disease risks at an early stage and significantly improve the efficiency and accuracy of disease screening. This is of great significance for early intervention and treatment, and helps reduce the morbidity and mortality of the disease.

[0101] Secondly, in terms of diagnostic decision support, the present invention provides clinicians with diagnostic suggestions based on multimodal data. By integrating and analyzing complex case information, the system can assist doctors in handling complex cases, reduce the occurrence of misdiagnosis and missed diagnosis, and improve the objectivity and consistency of diagnosis. This not only improves the quality of diagnosis, but also enhances the work efficiency of doctors.

[0102] In addition, the present invention also has broad application prospects in health risk prediction. By monitoring and analyzing the patient's long-term data, the system can predict the individual's health risks and disease development trends, and support personalized health management and the formulation of preventive measures. This helps to achieve precision medicine and improve the quality of life of patients.

[0103] Finally, the present invention provides a powerful tool for medical data analysis and research. Medical institutions and researchers can use the system to conduct large-scale medical data mining and clinical research, and accelerate the development of new diagnostic and treatment methods and drugs. This will promote the advancement of medical technology and promote the intelligent development of the medical industry.

[0104] The present invention aims to comprehensively improve the speed and accuracy of medical diagnosis, reduce the workload of doctors, and improve the utilization efficiency of medical resources, thereby providing patients with better quality and more personalized medical services.

[0105] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A medical diagnosis system based on chain reasoning and multimodal fusion, characterized in that: include: A data preprocessing module, used to preprocess the multimodal medical data of the patient and generate feature representations of each multimodal medical data; Data fusion module, used to fuse the feature representations of each multimodal medical data to form a unified multimodal feature vector ; Model inference module for multimodal feature vectors using a trained large language model Perform inference and generate diagnostic results and chains of reasoning; Output module, used to output diagnostic results and chains of reasoning; The multimodal medical data includes text data , image data I and structured data S; For text data The preprocessing is as follows: first, the text data is cleaned to remove noise and irrelevant information, then the text is segmented into words or subword sequences using the tokenizer of the pre-trained language model, and the embedding function is used to Map it into word vector representation , ,in, It captures the semantic and syntactic information of the text; The preprocessing of image data I is as follows: first, the image data is standardized, resized and denoised, and then a pre-trained convolutional neural network is used. Extract features from the image and obtain image feature representation , , Contains the spatial structure and texture features of the image; The preprocessing of structured data S is as follows: structured data S is processed by data standardization, and the numerical data is normalized to make its mean 0 and standard deviation 1; the categorical data is encoded by One-Hot Encoding to obtain the standardized structured data representation. , ; The multimodal feature vector ,in, represents the feature concatenation operation, represents the attention mechanism, , w2, w3 are weight coefficients of weighted summation, and the multimodal feature vector M contains the comprehensive information of the patient; A chain thinking reasoning mechanism is introduced into the large language model, and its structure is designed to be an input layer, a hidden layer and an output layer; The input layer receives the fused multimodal feature vector , initialize the hidden state , where the initial implicit state Contains all the patient's information; The hidden layer uses the large language model Layer, the implicit state is updated multiple times iteratively to form a chain reasoning process. For the t-th step reasoning, the implicit state is updated to ,in, is the hidden state at step t, It is the tth Layer, including multi-head self-attention mechanism and feed-forward neural network; The output layer is the last step n, through linear transformation and Function, generate diagnostic results , ,in, is the weight matrix, is the bias vector, is the hidden state of the nth step, The function converts the output of the linear transformation into a probability distribution and obtains the predicted probability of each disease category.

2. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 1 is characterized in that: The text data It includes one or more of the patient's past medical history text information, chief complaint text information, symptom description text information and doctor's medical record text information; the image data I is medical imaging information; the structured data S includes numerical or categorical data of medical examinations.

3. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 1, characterized in that: The large language model is one or more of GPT-4, Llama, Bio_ClinicalBERT, RoBERTa, XLNet, MedX_v1, and Medicine-LLM-13B-GGUF.

4. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 3 is characterized in that: The training of the large language model introduces transfer learning or federated learning.

5. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 1, characterized in that: The training of the large language model also defines a loss function, including a loss function for the diagnosis result diagnosis And the consistency loss function of the reasoning process COT ; The loss function of the diagnosis result diagnosis for ,in, is the total number of disease categories, is the true label components, whose values ​​are 0 or 1, The model predicts The probability of each category; The consistency loss function of the inference process COT for ,in, represents the square norm of a vector, is an ideal inference function, which transforms the implicit state of the t-1th step to obtain the ideal state of the tth step; Total loss function ,in, and is a weight hyperparameter used to balance the impact of the two parts of the loss.

6. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 1, characterized in that: During the training of the large language model, the Adam optimizer is used to update parameters.

7. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 5, characterized in that: The ideal inference function Assume it is an identity mapping, that is , indicating that the implicit states of adjacent reasoning steps are expected to transition smoothly.

8. The medical diagnosis system based on chain reasoning and multimodal fusion according to claim 5, characterized in that: The multimodal medical data also includes genomic data and metabolomics data.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, which, when executed by the processor, implement the functions of the medical diagnosis system based on chain reasoning and multimodal fusion according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the functions of the medical diagnosis system based on chain reasoning and multimodal fusion according to any one of claims 1 to 8.

Images