Intelligent online diagnosis and treatment automation method and system based on multi-modal large model

Abstract

The invention discloses an intelligent online diagnosis and treatment automation method and system based on a multi-modal large model. The system adopts a'four-layer double-engine 'framework, solves the problems of low pre-inquiry efficiency, poor medical record generation quality, insufficient decision support, poor system collaboration and the like in the existing online diagnosis and treatment through a dynamic pre-inquiry engine, a multi-mode medical record pre-generation engine and a decision support mechanism, realizes automation and high efficiency of the whole online diagnosis and treatment process, and improves the diagnosis and treatment efficiency. The diagnosis and treatment efficiency and the medical quality are obviously improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent medical treatment and artificial intelligence, and particularly relates to an online diagnosis and treatment full-process automation system and method fusing a multi-modal large model and a medical knowledge graph, which is particularly suitable for intelligent pre-diagnosis, automatic pre-generation of electronic medical records, and integrated scenarios of diagnosis and treatment decision support in an Internet hospital. BACKGROUND

[0002] The emergence and maturity of Internet of Things technology provide technical support for online diagnosis and treatment. Patients can detect their own basic signs through various electronic devices, and medical staff can obtain the sign data through Internet of Things technology and realize online preliminary diagnosis through inquiry. For patients with mild symptoms, medical staff can quickly diagnose and issue corresponding medical advice, and for patients with severe symptoms, they can guide patients to hospital for treatment.

[0003] Low pre-diagnosis efficiency: Traditional pre-diagnosis adopts a static questionnaire form, the questions are fixed and lack of personalization, resulting in long time consumption for patients to fill in, an average of 15-20 minutes, and insufficient completeness of information collection, only 60-70% of key information is recorded.

[0004] Poor quality of medical record generation: Existing automatic medical record generation systems are mostly based on rule templates and cannot effectively process unstructured doctor-patient conversations. The generated medical records have problems such as logical discontinuity and non-standard terminology, and doctors spend more than 30% of the total time on modification.

[0005] Insufficient diagnosis and treatment decision support: Existing systems lack deep correlation analysis of pre-diagnosis data and historical medical records, and doctors need to manually integrate multiple information, resulting in low prescription issuance efficiency, an average of 8-10 minutes per case, and the risk of missing drug interactions.

[0006] Poor system collaboration: Pre-diagnosis, medical record generation, and prescription systems are mostly independent modules, and data flow depends on manual intervention, resulting in broken diagnosis and treatment process, and an average online inquiry time of 25-30 minutes. SUMMARY

[0007] The present application aims to solve the problems of low pre-diagnosis efficiency, poor quality of medical record generation, insufficient decision support, and poor system collaboration in existing online diagnosis and treatment, and to realize the automation and intelligentization of the full process of online diagnosis and treatment, and to improve the efficiency and quality of diagnosis and treatment.

[0008] To solve the above problems, the technical solution of the present application is: an intelligent online diagnosis and treatment automation system based on a multi-modal large model, characterized by comprising:

[0009] A data access layer for integrating patient historical medical record data, patient complaint information, and medical knowledge graph of HIS / EMR system;

[0010] The intelligent processing layer includes a dynamic pre-diagnosis engine and a multimodal medical record pre-generation engine. The dynamic pre-diagnosis engine is based on Fine-tuned LLM to realize personalized pre-diagnosis, and the multimodal medical record pre-generation engine integrates templates and generative AI to realize automatic pre-generation of electronic medical records.

[0011] The decision support layer provides diagnostic suggestions and prescription optimization support based on a knowledge base, which is used to correlate and analyze pre-consultation data with historical medical records.

[0012] The application service layer provides API services to connect to doctor workstations, patient terminals, and management backends, for pushing pre-consultation results, electronic medical records, treatment suggestions, and monitoring system operation status.

[0013] Furthermore, the dynamic pre-diagnosis engine adopts a two-stage large model architecture: the first stage uses a Bi-LSTM+Attention model to achieve coarse-grained symptom classification, and the second stage uses Fine-tunedLLM to dynamically generate fine-grained follow-up questions.

[0014] Furthermore, the multimodal medical record pre-generation engine adopts a three-stage processing flow: the information structuring stage extracts clinical entities through the BERT+CRF model, the logic verification stage verifies entity relationships through the medical rule engine, and the natural language generation stage integrates template filling and generative AI to generate standardized medical records.

[0015] Furthermore, the knowledge base of the decision support layer includes disease knowledge, medical order templates, and drug interaction rules, and dynamically optimizes the prescription recommendation strategy through reinforcement learning.

[0016] The intelligent online diagnosis and treatment automation method based on a multimodal large model is characterized by the following steps:

[0017] Step 1: Data access and preprocessing. Obtain patient's historical medical records from the HIS / EMR system, receive patient chief complaint information, and integrate medical knowledge graph to construct an entity relationship network;

[0018] Step 2: Dynamic pre-diagnosis. The Bi-LSTM+Attention model is used to classify the patient's chief complaint into coarse-grained symptoms to generate an initial question set. Then, based on Fine-tunedLLM, personalized follow-up questions are dynamically generated according to the patient's real-time answers until the information is complete.

[0019] Step 3: Multimodal medical record pre-generation. The BERT+CRF model is used to extract clinical entities from doctor-patient dialogues. The logical relationships between entities are verified through a medical rule engine. Then, template filling and generative AI are combined to output the first draft of the electronic medical record.

[0020] Step 4: Diagnosis decision support, generate diagnosis suggestions based on knowledge graph analysis of patient data, optimize prescription recommendation through reinforcement learning to avoid drug interaction risks;

[0021] Step 5: System output and service, push pre-consultation results, electronic medical records and diagnosis suggestions to the doctor workstation, update the consultation progress to the patient terminal, and monitor the system performance through the management background.

[0022] Further, the "coarse-grained symptom classification" in step 2 is specifically: semantic encoding of patient complaint text, focusing on key symptom words through Attention mechanism, outputting corresponding disease category probability distribution to generate an initial question set.

[0023] Further, the "logic verification" in step 3 specifically includes: verifying the time sequence consistency of symptom occurrence time and disease progression, the causal relationship between medication history and current diagnosis, and the matching degree of surgical records and postoperative symptoms.

[0024] Further, the "reinforcement learning optimization of prescription recommendation" in step 4 is specifically: maximizing drug efficacy, minimizing side effects, and minimizing interaction risks as reward functions, and optimizing prescription combination strategies through multiple rounds of iterative training.

[0025] Further, the application service layer supports concurrent consultations ≥1000 times / second, end-to-end response delay <1 second, and system availability ≥99.99%.

[0026] Further, the performance indicators of "management background monitoring" in step 5 include: pre-consultation information completeness, medical record specification compliance rate, prescription rationality rate, and response time of each module.

[0027] The advantages of the present application compared with the existing technology are: through the dynamic pre-consultation system with context awareness, the patient pre-consultation time is shortened from the traditional 15-20 minutes to an average of 5 minutes, reducing by 67%; the multi-modal medical record pre-generation engine reduces the time-consuming of medical record generation from 8 minutes to 1 minute, saving 87.5%, and the complete consultation process is controlled within 12 minutes, which is 60% faster than the industry average, while supporting 1000+ / second of concurrent consultations, end-to-end response delay <1 second, significantly improving the overall efficiency of online diagnosis and treatment, pre-consultation information completeness reaches 92%, which is 35% higher than the existing technology; medical record specification compliance rate is 96%, exceeding the third level of Cybersecurity requirements; with the help of knowledge graph and reinforcement learning decision support, the prescription rationality rate is improved to 99.2%, effectively reducing the risk of drug interaction detection, and comprehensively ensuring the quality and safety of online diagnosis and treatment. BRIEF DESCRIPTION OF DRAWINGS

[0028] Fig. 1 is a four-layer double-engine architecture structure diagram of the present application.

[0029] Fig. 2 is a multi-modal electronic medical record generation flowchart structure of the present application. DETAILED DESCRIPTION

[0030] The specific embodiments of the present application will be further described below with reference to the accompanying drawings. The same parts are denoted by the same reference numerals throughout the drawings.

[0031] It should be noted that the words "front", "back", "left", "right", "up" and "down" used in the following description refer to the directions in the drawings, and the words "inner" and "outer" refer to the directions towards or away from the geometric center of a particular part.

[0032] In order to make the content of the present application more easily understood, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0033] As shown in Figs. 1-2 The intelligent online diagnosis and treatment automation system based on a multi-modal large model adopts a "four-layer double-engine" architecture, and the specific composition and interaction logic of each layer are as follows:

[0034] Data access layer

[0035] Data source and processing: interface with hospital HIS / EMR, synchronize patient historical medical record data in real time through standardized interface, including structured data such as past diagnosis, medication record, operation history, allergy history, etc., receive patient complaint information: support text input or voice input, integrate medical knowledge graph: build a knowledge network containing 100,000+ entities and 500,000+ relationships, store in Neo4j graph database, support millisecond-level entity query.

[0036] Intelligent processing layer

[0037] Dynamic pre-diagnosis engine, two-stage large model architecture: the first stage is coarse-grained symptom classification, using Bi-LSTM+Attention model to process the patient's complaint text: through Bi-LSTM, the text is bidirectionally semantic encoded to generate context vectors; using the Attention mechanism to focus on key symptom words such as "fever" and "cough", output the probability distribution of corresponding disease categories, such as respiratory diseases, digestive system diseases, etc., based on the classification results to generate an initial question set: for example, if it is determined to be a respiratory disease, the initial questions include "fever body temperature range?" "Cough with sputum?" and other 5-8 core questions; the second stage is fine-grained dynamic follow-up, based on Fine-tunedLLM to realize personalized follow-up: according to the patient's answers to the initial questions, real-time generation of targeted questions, such as the patient's answer "cough with blood", then ask "blood sputum color and frequency?", termination condition: when the model determines that the completeness of the key information is ≥92% through semantic understanding, the pre-diagnosis is automatically ended.

[0038] Multi-modal case pre-generation engine

[0039] The first stage is information structuring, using BERT+CRF model to extract clinical entities from doctor-patient dialogue text: BERT is responsible for deep semantic encoding of the text, and the CRF layer learns entity boundary features based on medical field annotation data to extract entity types including symptoms, signs, medications, etc., with an entity recognition accuracy of ≥93%; the second stage is logical verification, deploying a medical rule engine with pre-set 3000+ verification rules, for example: time sequence logical verification: if the patient describes "fever 3 days after surgery", it will automatically match the surgery date in the surgery record to verify the time consistency, cause-effect relationship verification: if the diagnosis is "gastric ulcer", it will verify whether there is "non-steroidal anti-inflammatory drugs" in the medication record to determine whether there is a cause-effect relationship; the third stage is natural language generation, combining template filling and generative AI: for structured information, use pre-set templates to fill in; for unstructured descriptions, call Fine-tunedLLM to generate standard medical expressions, and finally output the initial draft of the medical record in accordance with the "Basic Specification for Electronic Medical Records".

[0040] Decision layer support

[0041] Knowledge base composition, knowledge graph supports "patient current symptoms to possible diseases" reasoning, reasoning path is explainable, reinforcement learning module: taking "prescription rationality" as the goal, building a reward function, training the prescription recommendation model through PPO algorithm;

[0042] Diagnosis and treatment decision output

[0043] Diagnosis suggestion: based on the matching degree of patient symptoms and knowledge graph, generate Top3 possible diagnoses and supporting evidence, prescription optimization: detect the drug interactions of the doctor's preliminary prescription and recommend alternative solutions.

[0044] Application service layer

[0045] AIP service interface

[0046] A JSON format data interface is provided to the doctor workstation, including pre-consultation summary, electronic medical record draft, diagnosis suggestion and prescription verification result, supporting online editing and confirmation of doctors, pushing real-time progress to patient terminals, and showing simplified consultation results, providing monitoring interfaces to the management background, outputting key indicators, supporting abnormal alarms; performance guarantee, distributed deployment is adopted, supporting concurrent consultation volume of 1000+ / second, reducing data transmission delay through edge computing nodes, ensuring end-to-end response time <1 second, adopting active-active dual-machine hot backup mechanism, and the system availability reaches 99.99%.

[0047] The intelligent online diagnosis and treatment automatic method of the application comprises the following steps, and the specific execution logic of each step is as follows:

[0048] Step 1: data access and preprocessing

[0049] 1.1 Pull the patient's historical data from the HIS / EMR system, clean it through the ETL tool, and store it in the MySQL database;

[0050] 1.2 Receive the patient's chief complaint: if it is a voice input, convert it to text by calling the ASR model, and correct the recognition error by the text correction model;

[0051] 1.3 Load the medical knowledge graph, align the key words in the patient's chief complaint with the standard entities in the graph through entity linking technology, and provide a basis for subsequent reasoning.

[0052] Step 2: dynamic pre-consultation

[0053] 2.1 Coarse-grained classification: input the patient's chief complaint text into the Bi-LSTM+Attention model, output the disease category probability, and generate the corresponding initial question set;

[0054] 2.2 Fine-grained follow-up questions: fine-tunedLLM dynamically generates follow-up questions based on the context of the patient's answers, and the loop is executed until the information completeness is >92%, and the structured pre-consultation result is output.

[0055] Step 3: multi-modal medical record pre-generation

[0056] 3.1 Information structuring: input the pre-consultation dialogue text into the BERT+CRF model, extract clinical entities and label types;

[0057] 3.2 Logic verification: the medical rule engine verifies the extracted entities, and if a contradiction is found, it is automatically marked and triggers supplementary follow-up questions;

[0058] 3.3 Natural language generation: Call pre-trained medical text generation model, combine department-specific templates, generate electronic medical record draft containing "chief complaint, history of present illness, past medical history, preliminary examination" modules, format consistent with "Electronic Medical Record Application Management Specification".

[0059] Step 4: Diagnosis and treatment decision support

[0060] 4.1 Diagnosis suggestion generation: Knowledge graph combines patient history through "symptom-disease" path reasoning to output Top3 diagnoses and confidence;

[0061] 4.2 Prescription optimization: Reinforcement learning model optimizes the doctor's proposed prescription with "efficacy-risk ratio" as the target and labels the basis.

[0062] Step 5: System output and service

[0063] 5.1 Doctor workstation: Receive pre-consultation results, draft medical records and decision suggestions through API, doctors can directly edit and modify, and finally confirm to generate formal medical records and prescriptions;

[0064] 5.2 Patient terminal: Real-time display of consultation progress, push prescription QR code and medication guidance after consultation is completed;

[0065] 5.3 Management background: Real-time monitoring of system performance indicators, supporting abnormal data tracing and module fault alarm.

[0066] The above describes the present application and its embodiments, which are not limited, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solution can be designed, which should belong to the protection scope of the present application.

Claims

1. An intelligent online diagnosis and treatment automation system based on a multimodal large model, characterized by: include: The data access layer is used to integrate patient historical medical record data, patient chief complaint information, and medical knowledge graphs from the HIS / EMR system. The intelligent processing layer includes a dynamic pre-diagnosis engine and a multimodal medical record pre-generation engine. The dynamic pre-diagnosis engine is based on Fine-tuned LLM to realize personalized pre-diagnosis, and the multimodal medical record pre-generation engine integrates templates and generative AI to realize automatic pre-generation of electronic medical records. The decision support layer provides diagnostic suggestions and prescription optimization support based on a knowledge base, which is used to correlate and analyze pre-consultation data with historical medical records. The application service layer provides API services to connect to doctor workstations, patient terminals, and management backends, for pushing pre-consultation results, electronic medical records, treatment suggestions, and monitoring system operation status.

2. The intelligent online diagnosis and treatment automation system based on a multimodal large model according to claim 1, characterized in that, The dynamic pre-diagnosis engine adopts a two-stage large model architecture: the first stage uses a Bi-LSTM+Attention model to achieve coarse-grained symptom classification, and the second stage uses Fine-tunedLLM to dynamically generate fine-grained follow-up questions.

3. The intelligent online diagnosis and treatment automation system based on a multimodal large model according to claim 1, characterized in that, The multimodal medical record pre-generation engine adopts a three-stage processing flow: the information structuring stage extracts clinical entities through the BERT+CRF model, the logic verification stage verifies entity relationships through the medical rule engine, and the natural language generation stage integrates template filling and generative AI to generate standardized medical records.

4. The intelligent online diagnosis and treatment automation system based on a multimodal large model according to claim 1, characterized in that, The knowledge base of the decision support layer includes disease knowledge, medical order templates, and drug interaction rules, and dynamically optimizes the prescription recommendation strategy through reinforcement learning.

5. A method for automated online diagnosis and treatment based on a multimodal large model, characterized in that: Includes the following steps: Step 1: Data access and preprocessing. Obtain patient's historical medical records from the HIS / EMR system, receive patient chief complaint information, and integrate medical knowledge graph to construct an entity relationship network; Step 2: Dynamic pre-diagnosis. The Bi-LSTM+Attention model is used to classify the patient's chief complaint into coarse-grained symptoms to generate an initial question set. Then, based on Fine-tunedLLM, personalized follow-up questions are dynamically generated according to the patient's real-time answers until the information is complete. Step 3: Multimodal medical record pre-generation. The BERT+CRF model is used to extract clinical entities from doctor-patient dialogues. The logical relationships between entities are verified through a medical rule engine. Then, template filling and generative AI are combined to output the first draft of the electronic medical record. Step 4: Treatment decision support, generating diagnostic suggestions based on knowledge graph analysis of patient data, and optimizing prescription recommendations through reinforcement learning to avoid the risk of drug interactions; Step 5: System output and services. Push the pre-consultation results, electronic medical records and treatment suggestions to the doctor's workstation, update the consultation progress to the patient's terminal, and monitor system performance through the management backend.

6. The intelligent online diagnosis and treatment automation method based on a multimodal large model according to claim 5, characterized in that, Step 2, "coarse-grained symptom classification," specifically involves semantically encoding the patient's complaint text, focusing on key symptom words through an attention mechanism, and outputting the probability distribution of the corresponding disease category to generate an initial question set.

7. The intelligent online diagnosis and treatment automation method based on a multimodal large model according to claim 5, characterized in that, Step 3, "Logical Verification," specifically includes: verifying the consistency between the timing of symptom onset and disease progression, the causal relationship between medication history and current diagnosis, and the matching degree between surgical records and postoperative symptoms.

8. The intelligent online diagnosis and treatment automation method based on a multimodal large model according to claim 5, characterized in that, Step 4, "Reinforcement learning to optimize prescription recommendation," specifically involves using the maximization of drug efficacy, minimization of side effects, and interaction risk as reward functions, and optimizing the prescription combination strategy through multiple rounds of iterative training.

9. The intelligent online diagnosis and treatment automation method based on a multimodal large model according to claim 5, characterized in that, The application service layer supports concurrent consultations of ≥1000 times / second, end-to-end response latency of <1 second, and system availability of ≥99.99%.

10. The intelligent online diagnosis and treatment automation method based on a multimodal large model according to claim 5, characterized in that, The performance metrics for "Management Backend Monitoring" in step 5 include: completeness of pre-consultation information, compliance rate of medical record standards, rationality rate of prescriptions, and response time of each module.

Images