A step-by-step explainable multi-modal decision method, system and device for general diagnosis of traditional Chinese medicine

Abstract

The application discloses a step-by-step explainable multi-modal decision method, system and equipment for general diagnosis of traditional Chinese medicine, and relates to the field of traditional Chinese medicine. The method comprises feature extraction and fusion of inspection, auscultation and interrogation, and cutting, and construction of a multi-modal diagnosis database; a first deep learning model is trained using the multi-modal diagnosis database to obtain a multi-modal syndrome differentiation integrated diagnosis model; a credibility evaluation system of the multi-modal syndrome differentiation integrated diagnosis model and a traditional Chinese medicine diagnosis rule base are established; the fused inspection, auscultation and interrogation and cutting data of a current patient are input into the multi-modal syndrome differentiation integrated diagnosis model to obtain the clinical decision and the confidence of the current patient; the credibility is obtained according to the confidence and the credibility evaluation system; the clinical decision is subjected to rule reasoning using the traditional Chinese medicine diagnosis rule base; if successful, the clinical decision and the credibility of the current patient are output; and if failed, the credibility is degraded and then output. The application can make the traditional Chinese medicine diagnosis have explainability and transparency, and improve the accuracy and universality of the traditional Chinese medicine syndrome differentiation diagnosis.

Description

Technical Field

[0001] This application relates to the field of traditional Chinese medicine, and in particular to a step-by-step interpretable multimodal decision-making method, system and device for general diagnosis in traditional Chinese medicine. Background Technology

[0002] Traditional Chinese medicine (TCM) diagnosis relies on information obtained through the four diagnostic methods of observation, auscultation and olfaction, inquiry, and palpation. Doctors make a diagnosis based on detailed observation and analysis. However, in the current TCM diagnostic method, patients cannot know how the diagnosis was determined step by step, leading to problems of inexplicability and lack of transparency, and making it impossible to judge the accuracy of the TCM diagnostic results. Summary of the Invention

[0003] The purpose of this application is to provide a step-by-step interpretable multimodal decision-making method, system, and device for general TCM diagnosis, which can make TCM syndrome differentiation diagnosis interpretable and transparent, and improve the accuracy and universality of TCM syndrome differentiation diagnosis.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a step-by-step interpretable multimodal decision-making method for general diagnosis in Traditional Chinese Medicine (TCM), comprising: acquiring observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and extracting features from the observation, auscultation, inquiry, and palpation data of each historical patient; constructing an observation database, an auscultation database, an inquiry database, and a palpation database together with the corresponding observation, auscultation, inquiry, and palpation data; respectively fusing the observation, auscultation, inquiry, and palpation data of each historical patient in the observation database, the auscultation database, the inquiry database, and the palpation database, and combining the fused observation, auscultation, inquiry, and palpation data of each historical patient with the clinical decisions of each historical patient to form a multimodal diagnostic database; training a first deep learning model using the multimodal diagnostic database to obtain a multimodal syndrome differentiation integrated diagnostic model; and establishing a multimodal... The document outlines a multimodal integrated diagnostic model for clinical diagnosis, including: a credibility evaluation system; a TCM diagnostic rule base; acquisition of the patient's observation, auscultation, inquiry, and palpation data; feature extraction and fusion of these data to obtain fused observation, auscultation, inquiry, and palpation data; input of this fused data into the multimodal integrated diagnostic model to obtain the patient's clinical decision and confidence level; credibility of the patient based on their confidence level using the credibility evaluation system; rule-based reasoning for the patient's clinical decision using the TCM diagnostic rule base; output of the patient's clinical decision and credibility level if the reasoning is successful; and output of the patient's clinical decision and credibility level if the reasoning fails.

[0006] Secondly, this application provides a step-by-step interpretable multimodal decision-making system for general diagnosis in traditional Chinese medicine, including: a historical data acquisition module, a database construction module, a feature fusion module, a training module, an evaluation system establishment module, a rule base construction module, a feature fusion module, a decision-making module, a credibility calculation module, a rule reasoning module, a reasoning success module, and a reasoning failure module.

[0007] The historical data acquisition module is used to acquire the observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and to extract features from the observation, auscultation, inquiry, and palpation data of each historical patient.

[0008] The database construction module is used to combine the extracted features with the corresponding observation, auscultation, inquiry, and palpation data to form the observation database, auscultation database, inquiry database, and palpation database.

[0009] The feature fusion module is used to fuse the observation, auscultation, inquiry, and palpation data of each historical patient from the inspection database, auscultation, inquiry, and palpation database, respectively, and to combine the fused observation, auscultation, inquiry, and palpation data of each historical patient with the clinical decisions of each historical patient to form a multimodal diagnostic database.

[0010] The training module is used to train a first deep learning model using the multimodal diagnostic database to obtain a multimodal dialectical integrated diagnostic model.

[0011] The evaluation system establishment module is used to establish a credibility evaluation system for the multimodal dialectical integrated diagnostic model.

[0012] The rule base construction module is used to build a rule base for TCM diagnosis.

[0013] The feature fusion module is used to acquire the current patient's observation, auscultation, inquiry, and palpation data, and to extract and fuse features from the current patient's observation, auscultation, inquiry, and palpation data to obtain the current patient's fused observation, auscultation, inquiry, and palpation data.

[0014] The decision module is used to input the integrated diagnostic data of the current patient (observation, auscultation, inquiry, and palpation) into the multimodal syndrome differentiation integrated diagnostic model to obtain the current patient's clinical decision and confidence level.

[0015] The credibility calculation module is used to obtain the credibility of the current patient based on the current patient's confidence level and using the credibility evaluation system of the multimodal dialectical integrated diagnostic model.

[0016] The rule-based reasoning module is used to perform rule-based reasoning on the current patient's clinical decisions using the TCM diagnostic rule base.

[0017] The reasoning success module is used to output the current patient's clinical decision and confidence level if the rule reasoning is successful.

[0018] The reasoning failure module is used to output the current patient's clinical decision if the rule reasoning fails, and outputs it after downgrading the confidence level.

[0019] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the step-by-step interpretable multimodal decision-making method for general diagnosis of traditional Chinese medicine as described above.

[0020] According to the specific embodiments provided in this application, this application has the following technical effects:

[0021] This application provides a step-by-step interpretable multimodal decision-making method, system, and device for general TCM diagnosis. It extracts and fuses features from the observation, auscultation, inquiry, and palpation data of each historical patient, achieving interpretability of feature recognition and fusion. The fused observation, auscultation, inquiry, and palpation data of the current patient is input into the multimodal syndrome differentiation integrated diagnostic model to obtain the current patient's clinical decision and confidence level, achieving interpretability of the clinical decision output. A TCM diagnostic rule base is used to perform rule-based reasoning on the current patient's clinical decision, forming interpretability of the syndrome differentiation diagnostic rules. Through step-by-step interpretable decision-making involving feature extraction, feature fusion, model decision-making, and prior rules, a step-by-step, transparent, and interpretable TCM clinical diagnosis is formed, making TCM diagnosis universal. Furthermore, compared to existing TCM syndrome differentiation diagnostic methods, rule-based reasoning can output different levels of confidence for the current patient's clinical decision, improving the accuracy of TCM syndrome differentiation diagnosis. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a step-by-step interpretable multimodal decision-making method for general diagnosis using traditional Chinese medicine, as described in one embodiment of this application.

[0024] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] In one exemplary embodiment, such as Figure 1 As shown, a step-by-step, interpretable, multimodal decision-making method for general diagnosis in Traditional Chinese Medicine is provided, including the following steps 101 to 112. Wherein:

[0028] Step 101: Obtain the observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and extract features from the observation, auscultation, inquiry, and palpation data of each historical patient.

[0029] Step 102: Combine the extracted features with the corresponding observation, auscultation, inquiry, and palpation data to form the observation database, auscultation database, inquiry database, and palpation database.

[0030] Step 103: Integrate the observation, auscultation, inquiry, and palpation data of each historical patient from the inspection database, auscultation database, inquiry database, and palpation database respectively, and combine the integrated observation, auscultation, inquiry, and palpation data of each historical patient with the clinical decisions of each historical patient to form a multimodal diagnostic database.

[0031] Step 104: Train the first deep learning model using the multimodal diagnostic database to obtain the multimodal dialectical integrated diagnostic model.

[0032] Step 105: Establish a credibility evaluation system for the multimodal dialectical integrated diagnostic model.

[0033] Step 106: Construct a rule base for TCM diagnosis.

[0034] Step 107: Obtain the current patient's observation, auscultation, inquiry, and palpation data, and perform feature extraction and fusion on the current patient's observation, auscultation, inquiry, and palpation data to obtain the current patient's fused observation, auscultation, inquiry, and palpation data.

[0035] Step 108: Input the integrated observation, auscultation, inquiry and palpation data of the current patient into the multimodal syndrome differentiation integrated diagnostic model to obtain the current patient's clinical decision and confidence level.

[0036] Step 109: Based on the current patient's confidence level, obtain the current patient's confidence level using the confidence evaluation system of the multimodal dialectical integrated diagnostic model.

[0037] Step 110: Use the TCM diagnostic rule base to perform rule-based reasoning on the current patient's clinical decision.

[0038] Step 111: If the rule reasoning is successful, output the current patient's clinical decision and credibility.

[0039] Step 112: If the rule reasoning fails, output the current patient's clinical decision and output it with a downgraded confidence level.

[0040] Implementing steps 101 to 112 above can make TCM syndrome differentiation and diagnosis more interpretable and transparent, and improve the accuracy and efficiency of TCM syndrome differentiation and diagnosis.

[0041] In another exemplary embodiment of this application, the step 101 above, in which multiple historical patients' visual examination data are obtained and features are extracted from the visual examination data of each historical patient, can be replaced by the following steps 201 to 206:

[0042] Step 201: Obtain facial images of multiple historical patients.

[0043] High-definition cameras are used to capture images of the patient's face.

[0044] Step 202: Extract facial features from the facial images of each historical patient and reduce the dimensionality to obtain the facial features.

[0045] Facial image recognition technology analyzes a patient's facial features, including appearance, complexion, and eye expression, to help determine the patient's symptoms, syndromes, and patterns in Traditional Chinese Medicine (TCM). Machine learning algorithms are used to analyze these features and construct a facial diagnosis database that conforms to TCM diagnostic methods to provide diagnostic data for facial diagnosis.

[0046] Step 203: Collect tongue images from multiple historical patients.

[0047] Step 204: Use image segmentation technology to separate the tongue body and tongue coating from the tongue image of each historical patient.

[0048] Step 205: Identify tongue features from the tongue image after separating the tongue body and tongue coating using a second deep learning model.

[0049] By obtaining the patient's tongue diagnosis information and using image recognition technology to observe the tongue appearance, we can analyze the characteristics of the tongue, such as its color, shape, and coating, and identify changes in the sublingual veins, thus providing a basis for diagnosis.

[0050] Step 206: Standardize the facial features and tongue features, and use the standardized facial features and tongue features as diagnostic features.

[0051] In another exemplary embodiment of this application, auscultation is used to understand a patient's health status by auditory identification of sounds such as speech, breathing, and coughing. This includes identifying characteristics such as pitch, intensity, and clarity of sound. Therefore, the step 101 above, which involves acquiring auscultation data from multiple historical patients and extracting features from the auscultation data of each historical patient, can be replaced by the following steps 301-302:

[0052] Step 301: Collect the voices of multiple historical patients; the voices include speech, breathing and coughing.

[0053] Use audio capture equipment to record the patient's voice.

[0054] Step 302: Extract features from the voice of each historical patient using a third deep learning model to obtain voice features, which are then used as auscultation features; the voice features include timbre and tone.

[0055] Acoustic analysis techniques are used to extract sound features; combined with traditional Chinese medicine theory, the sound features are then subjected to pathological analysis.

[0056] In another exemplary embodiment of this application, the step 101 above, in which multiple historical patients' consultation data are obtained and features are extracted from the consultation data of each historical patient, can be replaced by the following steps 401 to 403:

[0057] Step 401: Collect the patient's chief complaint and symptom description from history.

[0058] Step 402: Using the fourth deep learning model, the chief complaints and symptom descriptions of historical patients are transformed into historical text.

[0059] Step 403: Analyze the symptom characteristics in historical texts using natural language processing techniques and use them as diagnostic features.

[0060] By using natural language processing technology to analyze key information in the text, such as symptoms, signs, diet, duration of illness, and other disease information, the content of the consultation is completed to determine the basic information for syndrome differentiation and diagnosis.

[0061] In another exemplary embodiment of this application, the step 101 above, in which multiple historical patients' palpation data are obtained and features are extracted from the palpation data of each historical patient, can be replaced by the following steps 501 to 507:

[0062] Step 501: Collect pulse signals from multiple historical patients.

[0063] A pulse sensor is used to collect the patient's pulse signal.

[0064] Step 502: Extract pulse features from the pulse signals of each historical patient using the fifth deep learning model.

[0065] Pulse characteristics are extracted using signal processing techniques. For example, pulse wave processing technology obtains pulse information by analyzing the wave characteristics of the pulse, such as pulse position, pulse rate, and pulse strength.

[0066] Step 503: Use artificial intelligence algorithms to classify pulse characteristics and obtain pulse classification features; the pulse classification features include deep pulse and no pulse.

[0067] Step 504: Obtain body temperature data from multiple historical patients.

[0068] A non-contact infrared thermometer is used to obtain the patient's body temperature data.

[0069] Step 505: Using the sixth deep learning model, divide the temperature range of the body temperature pattern; the body temperature pattern includes abnormal body temperature and normal body temperature.

[0070] Using intelligent algorithms to analyze the trends of body temperature data and identify the ranges of abnormal and normal body temperatures helps in identifying thermal information.

[0071] Step 506: Based on the temperature range of the divided body temperature patterns, determine the body temperature pattern corresponding to the body temperature data of each historical patient.

[0072] This application employs intelligent body temperature processing technology, which accurately measures and analyzes changes in the patient's body temperature.

[0073] Step 507: Use pulse classification features and body temperature patterns as palpation features.

[0074] In another exemplary embodiment of this application, steps 201 to 507 employ a multimodal TCM diagnostic information extraction technology based on integrated artificial intelligence. This technology encompasses the four diagnostic methods of TCM: observation, auscultation, inquiry, and palpation. It includes multi-task data extraction processes such as natural language processing, facial image recognition, tongue diagnosis image recognition, pulse wave processing, and intelligent body temperature processing, as well as the construction of multiple corresponding credibility results. The involved artificial intelligence model can be called a diagnostic data feature extraction model. Using deep learning technology, the diagnostic data feature extraction model is trained to transform diagnostic information into feature vectors, thus achieving interpretability of data transformation and feature recognition.

[0075] Furthermore, for the feature extraction model of syndrome differentiation and diagnosis data, a credibility system for multimodal data information extraction of traditional Chinese medicine's four diagnostic methods (inspection, auscultation, inquiry, and palpation) is constructed: Deep learning is used to represent the model's accuracy performance index (confidence level) in multimodal "inspection, auscultation, inquiry, and palpation" data as a multimodal credibility grading standard. The multimodal credibility grading standard is defined as classifying the credibility of the inspection, auscultation, inquiry, and palpation models into high, medium, low, and extremely low credibility levels. A new comprehensive credibility standard is constructed by integrating the credibility of the inspection, auscultation, inquiry, and palpation models: the credibility data standard for the four diagnostic methods includes a combination of high, medium, low, and extremely low credibility data from inspection, auscultation, inquiry, and palpation. The credibility standards for the four diagnostic methods (inspection, auscultation, inquiry, and palpation) are defined as follows: High credibility data has an average credibility of 90% or higher; Medium credibility data has an average credibility of 80% to 90%; Low credibility data has an average credibility of 70% to 80%; and Extremely low credibility data has an average credibility below 70%. This establishes a credibility data evaluation standard system for intelligent identification of the four diagnostic methods.

[0076] In another exemplary embodiment of this application, after extracting features from multiple modalities such as images and text, these features need to be effectively fused, involving multimodal attention mechanisms and graph neural network technology to achieve deep interaction and fusion of cross-modal information. Since different modal data reside in different feature spaces, modal alignment techniques are needed to map them to the same space. Deep learning models, such as the multimodal Transformer deep learning network, are used for feature fusion to integrate information from different modalities. Using a pre-trained language model, based on the BERT model, the diagnostic features of the four diagnostic methods can be fully fused, and the fused features can be further processed to enable the model to understand and generate tasks related to the four diagnostic methods of traditional Chinese medicine. Therefore, step 103 can be replaced by steps 601-602:

[0077] Step 601: Use an ensemble deep learning model to modally align the observation, auscultation, inquiry, and palpation data of each historical patient in the inspection database, auscultation database, inquiry database, and palpation database to map them to the same space.

[0078] Step 602: Use a pre-trained language model to fuse the mapped observation, auscultation, inquiry, and palpation data.

[0079] The first to sixth deep learning models mentioned in this application, as well as the ensemble deep learning model, are not limited to specific models, as long as they can achieve the corresponding functions. The ensemble deep learning model contains multiple deep learning models, and each deep learning model can process data of one modality.

[0080] In another exemplary embodiment of this application, a labeled dataset of the four diagnostic methods of traditional Chinese medicine (a multimodal diagnostic database) is used to train the model, optimize model parameters, and improve the model's accuracy and robustness. Through incremental learning and expert feedback mechanisms, the model's professional level is continuously improved.

[0081] In another exemplary embodiment of this application, different modalities of data (such as text, images, and sounds) from the four diagnostic methods are fused to reveal the intrinsic relationships between the data, forming a credibility evaluation standard system for the ensemble model. The credibility evaluation system of the multimodal dialectical integrated diagnostic model needs to be transformed based on the confidence level corresponding to the performance results of the model's own evaluation indicators. Step 105 above, the credibility evaluation system of the multimodal dialectical integrated diagnostic model, includes:

[0082] If the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 90%, it is defined as high confidence; if the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 70% and less than 90%, it is defined as medium confidence; if the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 50% and less than 70%, it is defined as low confidence; if the confidence level of the multimodal dialectical integrated diagnostic model output is less than 50%, it is defined as very low confidence.

[0083] By introducing artificial intelligence technology based on prior rules, a credibility system for a multimodal fusion model of TCM syndromes is provided, which solves the black box problem of deep learning models.

[0084] In one example, the processed data, model, and model credibility evaluation system are integrated into a multimodal model system to complete the model-stage diagnostic task for the first TCM diagnosis. The system can provide syndrome differentiation results and credibility stratification results based on the input multimodal data.

[0085] In another exemplary embodiment of this application, in order to improve the clinical decision-making effect of clinical multimodal diagnostic results, this application constructs a priori rule-based verification module based on a clinical professional diagnostic and treatment knowledge system to supplement the clinical decision-making accuracy of the integrated model and further explain the uninterpretability of deep learning, forming a system based on the combination of rules and deep learning. The construction of the TCM diagnostic rule base in step 106 above can be replaced by the following steps 701 to 703:

[0086] Step 701: Based on the clinical professional diagnosis and treatment knowledge system, establish rule logic using an association rule learning algorithm.

[0087] Step 702: Construct an initial TCM diagnostic rule base based on the rule logic.

[0088] The rules in the initial TCM diagnostic rule base are all derived from authoritative professional data such as expert experience and clinical guidelines that conform to evidence-based medicine.

[0089] Step 703: Based on prior rules, use a genetic algorithm to optimize the initial TCM diagnostic rule base to obtain the final TCM diagnostic rule base.

[0090] Genetic algorithms are used to optimize the rule base in a fuzzy inference system, and the rules are continuously optimized through iteration.

[0091] Then, a second auxiliary rule enhancement is performed by constructing a commonly used rule-based logic algorithm for pre- and post-inference rule reasoning.

[0092] For example, a rule correction module is developed that uses a genetic algorithm to optimize the rule base in the fuzzy inference system. The rules are continuously optimized through iteration to form rule correction based on prior rules.

[0093] In steps 107 to 112, this application combines the output of the deep learning model with the rule base of the proofreading module for comprehensive reasoning. During this process, techniques such as multimodal attention mechanisms can be employed to achieve deep interaction and fusion of information from different modalities, thus completing the combination and inference of the model and rules. The deep learning model's output is combined with the rule base. This can be achieved by designing a specific loss function, which includes not only the model's prediction loss but also the loss based on consistency with prior rules.

[0094] When rule-based reasoning succeeds, the credibility of the rule logic is used as the standard, and the credibility is calculated by averaging the results of the comprehensive data. When rule-based reasoning fails, the credibility result of the multimodal model is used as the standard, and the credibility is downgraded by one level.

[0095] The system construction of a rule-based multimodal fusion model for TCM syndrome differentiation was completed, providing step-by-step results and interpretability and credibility analysis information for clinical TCM clinical decision-making.

[0096] This application's method is based on evidence-based clinical decision-making. It achieves intelligent step-by-step decision-making through "intelligent extraction of multimodal disease and syndrome information - model feature processing - model fusion - prior rule-based verification - clinical decision diagnosis." By constructing a hierarchy of "credibility of intelligently extracted data + credibility of model results + credibility of the decision fusion model," it enhances the credibility and interpretability of TCM clinical decision-making by providing a layered interpretation of the step-by-step multimodal results. The main technologies designed include intelligent multimodal data extraction, feature fusion based on large-scale language models, prior rule-based construction, and verification based on evidence-based TCM diagnosis. This enables a universal multimodal diagnostic technique for multiple diseases using TCM, as well as an interpretable hierarchical decision-making technique.

[0097] The process of the multimodal syndrome differentiation system in traditional Chinese medicine:

[0098] The construction of a multimodal diagnostic and interpretable system in Traditional Chinese Medicine (TCM) not only includes information on the results of each step, but also involves the step-by-step results on the credibility of each result, in order to improve the interpretability of clinical decisions: Constructing a multimodal diagnostic data extraction system based on TCM's four diagnostic methods (inspection, auscultation, inquiry, and palpation) → Forming a multimodal diagnostic database → Constructing a multimodal general language model diagnostic system → Constructing a diagnostic verification system based on prior rule logic → A TCM diagnostic decision fusion system based on model and rule integration.

[0099] User clinical diagnostic application process:

[0100] User data acquisition (including information acquisition from the four diagnostic methods of observation, auscultation, inquiry, and palpation) → Identification of whether the data meets the standards of the multimodal model → Feature transformation is completed using the multimodal model after identification → Information diagnosis based on model and rule logic is completed → Model results and rule logic results are given → Step-by-step TCM syndrome differentiation and diagnosis results and credibility information are given, based on the "data-model-rule" approach.

[0101] The standard for identifying whether data conforms to the multimodal model refers to comparing real-time acquired patient data with the database to determine whether the patient's data conforms to the model input.

[0102] This application relates to a medical diagnostic system combining traditional Chinese medicine (TCM) and artificial intelligence (AI) technologies. The technology and system construction encompasses step-by-step interpretable decision-making for TCM syndrome differentiation and diagnosis, including but not limited to diagnostic information such as etiology, pathogenesis, symptoms, signs, and treatment methods. A syndrome differentiation and diagnosis recognition module based on AI algorithms is developed. This module can automatically extract syndrome differentiation and diagnosis information from patient descriptions using techniques such as natural language processing, image recognition, pulse diagnosis, and body temperature detection, completing the intelligent data recognition technology and interpretable decision-making of the data acquisition module. Deep learning technology is used to train a syndrome differentiation and diagnosis data feature extraction model, transforming syndrome differentiation and diagnosis information into feature vectors, completing the interpretable module for data transformation and feature recognition. The TCM syndrome differentiation rule logic for diseases, syndromes, and symptoms is constructed. Combined with the syndrome differentiation and diagnosis feature vectors, matching and reasoning are performed using knowledge rule expression logic to form an interpretable module for syndrome differentiation and diagnosis rules. A step-by-step interpretable rule and model fusion rule logic and framework system is constructed. Based on the results of prior rules and models, an interpretability credibility system for each part is formed. Through step-by-step interpretable decision-making involving data extraction, data feature fusion, prior rules, and model algorithms, doctors can easily input patient information and access a fully transparent process for querying diagnostic results, ensuring interpretability of the results. The diagnostic process is transparently identified and its credibility is displayed. By combining diagnostic results with clinical experience and patient feedback, iterative optimization is performed, and the results are validated using an evidence-based TCM diagnostic knowledge base, resulting in a final step-by-step, transparent, and interpretable TCM clinical diagnosis. This application presents a step-by-step interpretable multimodal decision-making method for general TCM diagnosis, aiming to address the lack of interpretability and transparency in existing TCM syndrome differentiation diagnoses, while improving the accuracy and efficiency of TCM syndrome differentiation diagnoses, providing technical support for the modernization and internationalization of TCM.

[0103] Based on the same inventive concept, this application also provides a step-by-step interpretable multimodal decision-making system for general TCM diagnosis, used to implement the step-by-step interpretable multimodal decision-making method for general TCM diagnosis described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the step-by-step interpretable multimodal decision-making system for general TCM diagnosis provided below can be found in the limitations of the step-by-step interpretable multimodal decision-making method for general TCM diagnosis described above, and will not be repeated here.

[0104] In one exemplary embodiment, a step-by-step interpretable multimodal decision-making system for general diagnosis in traditional Chinese medicine is provided, comprising: a historical data acquisition module, a database construction module, a feature fusion module, a training module, an evaluation system establishment module, a rule base construction module, a feature fusion module, a decision-making module, a credibility calculation module, a rule reasoning module, a reasoning success module, and a reasoning failure module.

[0105] The historical data acquisition module is used to acquire the observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and to extract features from the observation, auscultation, inquiry, and palpation data of each historical patient.

[0106] The database construction module is used to combine the extracted features with the corresponding observation, auscultation, inquiry, and palpation data to form the observation database, auscultation database, inquiry database, and palpation database.

[0107] The feature fusion module is used to fuse the observation, auscultation, inquiry, and palpation data of each historical patient from the inspection database, auscultation database, inquiry database, and palpation database respectively, and to combine the fused observation, auscultation, inquiry, and palpation data of each historical patient with the clinical decisions of each historical patient to form a multimodal diagnostic database;

[0108] The training module is used to train a first deep learning model using the multimodal diagnostic database to obtain a multimodal dialectical integrated diagnostic model.

[0109] The evaluation system establishment module is used to establish a credibility evaluation system for the multimodal dialectical integrated diagnostic model;

[0110] The rule base construction module is used to build a rule base for TCM diagnosis.

[0111] The feature fusion module is used to acquire the current patient's observation, auscultation, inquiry, and palpation data, and to extract and fuse the features of the current patient's observation, auscultation, inquiry, and palpation data to obtain the current patient's fused observation, auscultation, inquiry, and palpation data;

[0112] The decision module is used to input the integrated diagnostic data of the current patient (observation, auscultation, inquiry, and palpation) into the multimodal diagnostic model to obtain the current patient's clinical decision and confidence level.

[0113] The credibility calculation module is used to obtain the credibility of the current patient based on the current patient's confidence level using the credibility evaluation system of the multimodal dialectical integrated diagnostic model.

[0114] The rule reasoning module is used to perform rule reasoning on the current patient's clinical decisions using the TCM diagnostic rule base;

[0115] The reasoning success module is used to output the current patient's clinical decision and confidence level if the rule reasoning is successful.

[0116] The reasoning failure module is used to output the current patient's clinical decision if the rule reasoning fails, and outputs it after downgrading the confidence level.

[0117] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 2As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores current patient clinical decisions and their reliability. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a step-by-step, interpretable, multimodal decision-making method for general diagnosis in Traditional Chinese Medicine.

[0118] Those skilled in the art will understand that Figure 2 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A step-by-step interpretable multimodal decision-making method for general diagnosis in Traditional Chinese Medicine, characterized in that, include: Acquire the observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and extract features from the observation, auscultation, inquiry, and palpation data of each historical patient; The extracted features, together with the corresponding observation, auscultation, inquiry, and palpation data, constitute the observation diagnosis database, auscultation diagnosis database, inquiry diagnosis database, and palpation diagnosis database; The observation, auscultation, inquiry, and palpation data of each historical patient are respectively integrated from the inspection database, the auscultation database, the inquiry database, and the palpation database. The integrated observation, auscultation, inquiry, and palpation data of each historical patient are combined with the clinical decisions of each historical patient to form a multimodal diagnostic database. The first deep learning model is trained using the multimodal diagnostic database to obtain a multimodal dialectical integrated diagnostic model; Establish a credibility evaluation system for multimodal dialectical integrated diagnostic models; Construct a rule base for TCM diagnosis; Acquire the current patient's observation, auscultation, inquiry, and palpation data, and extract and fuse the features of the current patient's observation, auscultation, inquiry, and palpation data to obtain the current patient's fused observation, auscultation, inquiry, and palpation data; The integrated diagnostic data from the four diagnostic methods (inspection, auscultation, inquiry, and palpation) of the current patient are input into the multimodal diagnostic model to obtain the current patient's clinical decision and confidence level. Based on the current patient's confidence level, the confidence level of the current patient is obtained using the confidence evaluation system of the multimodal dialectical integrated diagnostic model; Use a traditional Chinese medicine diagnostic rule base to perform rule-based reasoning for the current patient's clinical decisions; If the rule-based reasoning is successful, the current patient's clinical decision and credibility will be output. If the rule-based reasoning fails, the current patient's clinical decision will be output, and the confidence level will be downgraded before outputting.

2. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, Acquire visual diagnosis data from multiple historical patients and extract features from the visual diagnosis data of each historical patient, specifically including: Acquire facial images from multiple historical patients; Facial features are extracted from facial images of each historical patient and then dimensionality reduced to obtain facial features; Collect tongue images from multiple historical patients; Image segmentation techniques were used to separate the tongue body and tongue coating from the tongue images of each historical patient. The second deep learning model is used to identify tongue features from the tongue image after separating the tongue body and tongue coating; Facial and tongue features were standardized, and the standardized facial and tongue features were used as diagnostic features for observation.

3. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, Acquire auscultation data from multiple historical patients and extract features from the auscultation data of each historical patient, specifically including: The audio was collected from multiple historical patients; the audio included speech, breathing, and coughing. The third deep learning model extracts features from the voice of each historical patient to obtain voice features, which are then used as auscultation features; the voice features include timbre and tone.

4. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, Acquire consultation data from multiple historical patients and extract features from the consultation data of each historical patient, specifically including: Collect patients' chief complaints and symptom descriptions from their medical history; The fourth deep learning model is used to transform the chief complaints and symptom descriptions of historical patients into historical text. Natural language processing techniques are used to analyze the characteristics of symptoms in historical texts and to serve as diagnostic features.

5. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, Acquire palpation data from multiple historical patients and extract features from the palpation data of each historical patient, specifically including: Collect pulse signals from multiple historical patients; The fifth deep learning model is used to extract pulse features from the pulse signals of each historical patient. Artificial intelligence algorithms are used to classify pulse characteristics to obtain pulse classification features; the pulse classification features include deep pulse and no pulse; Obtain body temperature data from multiple historical patients; The sixth deep learning model is used to divide the temperature range of the body temperature pattern; the body temperature pattern includes abnormal body temperature and normal body temperature. Based on the temperature range of the defined temperature patterns, determine the temperature pattern corresponding to the temperature data of each historical patient. Pulse classification features and body temperature patterns were used as palpation features.

6. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, This involves integrating the observation, auscultation, inquiry, and palpation data of each historical patient from the visual diagnosis database, the olfactory diagnosis database, the inquiry database, and the palpation database, specifically including: An ensemble deep learning model is used to modally align the observation, auscultation, inquiry, and palpation data of each historical patient in the inspection, auscultation, inquiry, and palpation databases to map them to the same space; We use a pre-trained language model to fuse the mapped observation, auscultation, inquiry, and palpation data.

7. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, The credibility evaluation system of the multimodal dialectical integrated diagnostic model includes: If the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 90%, it is defined as high confidence. If the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 70% and less than 90%, it is defined as medium confidence. If the confidence level of the multimodal dialectical integrated diagnostic model output is greater than or equal to 50% and less than 70%, it is defined as low confidence. If the confidence level of the multimodal dialectical integrated diagnostic model output is less than 50%, it is defined as extremely low confidence.

8. The step-by-step interpretable multimodal decision-making method for general diagnosis in traditional Chinese medicine according to claim 1, characterized in that, The construction of a traditional Chinese medicine diagnostic rule base includes: Based on the clinical professional diagnosis and treatment knowledge system, rule logic is established using an association rule learning algorithm; Based on the rule logic, an initial TCM diagnostic rule base is constructed; Based on prior rules, a genetic algorithm is used to optimize the initial TCM diagnostic rule base to obtain the final TCM diagnostic rule base.

9. A step-by-step interpretable multimodal decision-making system for general diagnosis in traditional Chinese medicine, characterized in that, include: The historical data acquisition module is used to acquire the observation, auscultation, inquiry, and palpation data of multiple historical patients and the clinical decisions of each historical patient, and to extract features from the observation, auscultation, inquiry, and palpation data of each historical patient. The database construction module is used to combine the extracted features with the corresponding observation, auscultation, inquiry, and palpation data to form the observation database, auscultation database, inquiry database, and palpation database. The feature fusion module is used to fuse the observation, auscultation, inquiry, and palpation data of each historical patient from the inspection database, auscultation database, inquiry database, and palpation database respectively, and to combine the fused observation, auscultation, inquiry, and palpation data of each historical patient with the clinical decisions of each historical patient to form a multimodal diagnostic database; The training module is used to train a first deep learning model using the multimodal diagnostic database to obtain a multimodal dialectical integrated diagnostic model. The evaluation system establishment module is used to establish a credibility evaluation system for the multimodal dialectical integrated diagnostic model; The rule base construction module is used to build a rule base for TCM diagnosis. The feature fusion module is used to acquire the current patient's observation, auscultation, inquiry, and palpation data, and to extract and fuse the features of the current patient's observation, auscultation, inquiry, and palpation data to obtain the current patient's fused observation, auscultation, inquiry, and palpation data; The decision module is used to input the integrated diagnostic data of the current patient (observation, auscultation, inquiry, and palpation) into the multimodal diagnostic model to obtain the current patient's clinical decision and confidence level. The credibility calculation module is used to obtain the credibility of the current patient based on the current patient's confidence level using the credibility evaluation system of the multimodal dialectical integrated diagnostic model. The rule reasoning module is used to perform rule reasoning on the current patient's clinical decisions using the TCM diagnostic rule base; The reasoning success module is used to output the current patient's clinical decision and confidence level if the rule reasoning is successful. The reasoning failure module is used to output the current patient's clinical decision if the rule reasoning fails, and outputs it after downgrading the confidence level.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the step-by-step interpretable multimodal decision-making method for general diagnosis of traditional Chinese medicine as described in any one of claims 1-8.

Images