Multi-mode artificial intelligence virtual patient inquiry teaching system

Abstract

The invention discloses a multi-modal artificial intelligence virtual patient inquiry teaching system. The system comprises a case arrangement agent, a dialogue planning agent, a portrait and emotion agent, an image acquisition module, a double-flow emotion representation fusion network, a medical verification agent and a teaching feedback agent. According to the invention, through multi-agent cooperative work of the case arrangement agent, the dialogue planning agent, the portrait and emotion agent, the medical verification agent and the teaching feedback agent, accurate presentation of medical information and true expression of a natural language of a virtual patient in an inquiry process are realized. The system can automatically identify questioning intentions of students, call a case manuscript fact unit and generate answers conforming to medical logic and humanistic features in combination with patient portraits and emotional states, thereby greatly improving authenticity of inquiry interaction and medical rationality.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence application and medical education informatization technology, specifically relating to a multimodal artificial intelligence virtual patient consultation teaching system. Background Technology

[0002] Clinical teaching holds a crucial position in the medical education system, with history taking, doctor-patient communication, and humanistic care being core skills that young physicians must master. Traditional teaching methods typically rely on standardized patients and real clinical environments. However, recruiting and training standardized patients is costly and limited in number, making it difficult to meet the needs of large-scale teaching. Furthermore, real patients exhibit inherent differences in symptom presentation, disease progression, and communication styles, resulting in a lack of uniformity in teaching cases and hindering students from gaining repeatable and comparable learning experiences during training. In addition, teaching directly with real patients carries potential medical risks and ethical issues, limiting the breadth and depth of teaching.

[0003] Under current conditions, teachers often have to juggle heavy clinical duties and teaching responsibilities, making it difficult to provide sufficient guidance and feedback to each student. This situation leads to uneven training outcomes for students in areas such as history taking, logical thinking, and communication skills, further exacerbating the contradiction between insufficient teaching resources and learning needs.

[0004] In recent years, the development of artificial intelligence technology has provided new ideas for solving these problems. Large language models have demonstrated powerful capabilities in natural language understanding and generation, making it possible to construct virtual patients with highly anthropomorphic features. In this way, students can practice repeatedly in a risk-free environment and obtain a standardized learning experience. However, relying solely on a single model often fails to simultaneously meet the demands for clinical accuracy, contextual realism, and teaching feedback. To address this, researchers have proposed introducing multi-agent technology into virtual patient systems. Agents with different functions can collaborate, each responsible for symptom description, medical history narration, emotional responses, medical knowledge verification, and teaching feedback, thereby achieving higher levels of interactivity and medical reliability in the system.

[0005] In summary, existing technologies in medical education still face significant challenges, including high teaching costs, limited case resources, and a lack of standardized and personalized feedback. A new technological solution is urgently needed to support clinical teaching in a low-cost, scalable, and repeatable manner, while ensuring the accuracy of medical content and the authenticity of the interactive experience, thereby comprehensively improving the quality and efficiency of medical education. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal artificial intelligence virtual patient consultation teaching system.

[0007] A multimodal AI-powered virtual patient consultation teaching system includes a case arrangement agent, a dialogue planning agent, a profiling and emotion agent, an image acquisition module, a dual-stream emotion representation fusion network, a medical verification agent, and a teaching feedback agent.

[0008] The case arrangement agent is used to load and initialize case files. Loading refers to the case arrangement agent reading all fields of the case file from the case database and converting them into internal data that can be used later. Initialization refers to the case arrangement agent parsing each element in the case file, extracting callable fact units through named entity recognition technology, and storing them in an internal buffer.

[0009] The dialogue planning agent is used to parse the statements in the student's input text and identify the key information points therein. Through semantic mapping, it establishes an association between the statements and the corresponding fact units in the medical record and generates a dialogue plan.

[0010] The image acquisition module is used to collect real-time images of the facial expressions and body postures of the patients during consultations.

[0011] The profiling and emotional intelligence agent is used to process the information acquired by the image acquisition module based on the dual-stream emotion representation fusion network after obtaining the dialogue plan and fact unit to generate the patient's emotion vector. Finally, it determines the appropriate expression method based on the patient's emotion vector and generates a natural language response.

[0012] The dual-stream emotion representation fusion network is used to identify the emotional expressions of trainees through facial expressions and body language, and adjusts the patient's emotion in real time based on the trainee's emotions. The dual-stream emotion representation fusion network includes a facial expression recognition submodule, a body posture emotion submodule, and a dual-stream fusion submodule. The facial expression recognition submodule and the body posture emotion submodule receive facial expression and body posture image information from trainees, respectively, and generate facial modality emotion representation and body modality emotion representation. Finally, the dual-stream fusion submodule generates a joint emotion feature vector of the trainee. This joint emotion feature vector is further input to a linear projection layer, which adapts the dimensions of the profile to the input feature vector dimensions required by the emotional intelligence agent. The feature vector after linear projection is called the patient's emotion vector.

[0013] The medical verification agent comprehensively reviews the natural language responses generated by the profiling and emotional agents to ensure that the output conforms to the factual units in the medical record and remains consistent in logical expression.

[0014] The teaching feedback agent is used to comprehensively compare and evaluate the interaction records and the initial medical records written by the students.

[0015] The case arrangement agent, dialogue planning agent, profiling and emotion agent, medical verification agent, and teaching feedback agent are all implemented based on a large language model.

[0016] In one possible implementation, the case files refer to a collection of specific clinical patient case information gathered and organized through multiple channels within the scope permitted for medical teaching and research. Sources of patient case information include: publicly published medical cases and academic papers, clinical treatment guidelines published by authoritative publishers, medical textbooks, and authentic medical records approved by the ethics committee for teaching purposes. By organizing the patient case information and treatment plan information from the above sources, and referring to the "Basic Norms for Medical Record Writing" to write the corresponding patient's initial medical record file, this serves as the standard storage and retrieval format for the case files. The diverse case files are then tagged and organized to form a case database.

[0017] In one possible implementation, when creating the medical record, the original patient medical information needs to be anonymized and standardized. The anonymization process employs field-level de-identification, including deleting or replacing directly identifying information and obfuscating indirectly identifying information.

[0018] During data standardization, patient case information needs to be reorganized according to a pre-designed data structure. This data structure is the initial medical record structure specified in the "Basic Standards for Medical Record Writing," and includes, but is not limited to, the following elements: disease diagnosis, patient's chief complaint, accompanying symptoms, physical examination results, past medical history, medication history, laboratory tests, imaging examinations, and auxiliary examinations. Each element is stored in a structured field format for easy retrieval by the intelligent agent.

[0019] After data anonymization and structuring, the case files are stored in a case database, with each case assigned a unique identifier. The case files in the database are grouped and managed according to the patient's primary diagnosis, the organ system involved in the patient's disease, or specific teaching objectives to support different teaching needs.

[0020] In one possible implementation, the dialogue planning agent uses a large language model to achieve semantic mapping:

[0021] After identifying key information points, the dialogue planning agent generates a dialogue plan. This dialogue plan refers to determining the necessary medical information to be retrieved and the order and method of information return based on the content of the current student's question, combined with factual units already loaded in the medical record, semantic relevance, clinical logic, and dialogue context information, through reasoning using a large language model.

[0022] After the dialogue plan is generated, the dialogue planning agent calls the case orchestration agent to extract the factual units from the case file that correspond to the trainee's questions. The dialogue planning agent has an exception control mechanism when processing trainee input. When the trainee's question exceeds the scope of the case file, the system will output "The patient failed to provide relevant information"; when the trainee's question has the probability of leading to content that does not comply with medical ethics or teaching standards, the system will automatically block it to avoid outputting inappropriate answers.

[0023] In one possible implementation, the facial expression recognition submodule employs a lightweight convolutional neural network, MobileNetV3, to perform multi-layer convolutional operations on the input facial image, extracting local texture features and muscle movement features of the facial region. After global average pooling, a high-dimensional feature representation, i.e., the overall feature vector, is formed. This feature representation is processed through two parallel output branches. One branch is a facial action unit recognition branch, which uses a multi-label classification structure and a sigmoid activation function to output the probability of occurrence of each action unit, and calculates the corresponding intensity value through a linear regression layer. The other branch is an emotion dimension regression branch, which performs a two-channel linear mapping on the overall feature vector to obtain continuous values ​​of emotion positivity (Valence) and activation level (Arousal), i.e., emotion dimension values. Subsequently, the outputs of the two branches are fused at the feature level, that is, the action unit intensity features and the emotion dimension features are concatenated to form a comprehensive facial feature vector. After normalization, the comprehensive facial feature vector obtains a uniform-scale facial modality emotion representation, which is then input to the two-stream fusion submodule.

[0024] The body posture emotion submodule, based on the OpenPose pose estimation algorithm, extracts the skeletal keypoint coordinate sequence from the upper body image and inputs the keypoint coordinates of each frame into a temporal convolutional neural network in chronological order to model the dynamic features of body posture changes over time. During convolution calculation, the submodule uses the spatial coordinate difference of keypoints in adjacent frames as input features to calculate the relative displacement vector, joint angle change, and displacement amplitude between keypoints. A one-dimensional temporal convolution kernel slides along the time dimension, performing weighted summation and nonlinear transformation on the difference features to extract intermediate feature maps representing upper limb movement, trunk posture changes, and body stability. After multiple convolutions and feature aggregation, a low-dimensional emotion feature vector reflecting the dynamic changes in the trainee's body posture is generated and, after normalization, used as the body modality emotion representation input to the dual-stream fusion submodule.

[0025] The dual-stream fusion submodule is used to fuse facial modal emotion representations output from the facial expression recognition submodule and body modal emotion representations output from the body posture emotion submodule. Specifically, the facial modal emotion representations output from the facial expression recognition submodule and the body modal emotion representations output from the body posture emotion submodule are first weighted and summed to obtain a preliminary joint feature vector. During the weighted summation fusion process, a confidence gating mechanism is introduced to dynamically adjust the weights of the facial modal emotion representations and the body modal emotion representations based on the confidence of face detection and skeleton keypoint detection. In the temporal dimension, the dual-stream fusion submodule uses a sliding time window mechanism to aggregate the joint feature vectors of several consecutive frames to form a feature sequence. Subsequently, a one-dimensional temporal convolution operation is performed on this feature sequence to obtain the final trainee joint emotion feature vector.

[0026] In one possible implementation, after the dialogue planning agent generates a dialogue plan and obtains factual units from the medical record, the profiling and emotional intelligence agent determines an appropriate expression method based on the patient's emotional vector. This ensures that the output accurately reflects both the medical facts and the patient's profile information and emotional state. The profile information refers to the patient's gender, age, occupation, education level, personal history, marital and reproductive history, menstrual history, and family history characteristics from the retrieved medical record.

[0027] The system combines factual units provided by the case arrangement agent with patient emotion vectors to transform them into natural language expressions that closely resemble those of real patients. Simultaneously, the profiling and emotional agent employs constraint mechanisms to avoid generating diagnostic statements or expressions beyond the patient's identity, ensuring that responses always align with the patient's role.

[0028] In one possible implementation, the medical verification agent first compares the answers with the medical record item by item to check whether the profile and the content output by the emotional agent accurately reflect the related information in the medical record.

[0029] After comparing the data with the medical records, the medical verification agent will perform a logical consistency check. This check is based on the general rules of the medical records and the medical knowledge base to ensure that the output content does not contain contradictions or conflict with basic medical common sense.

[0030] When the medical verification agent confirms that the answer is consistent with the medical record and logical rules, the system outputs the answer to the student as a valid result of the consultation interaction, allowing the student to continue with the next round of questions. If there are problems, the system returns to the dialogue planning agent, requiring regeneration. The entire verification process is recorded in the system log for teachers and administrators to review and evaluate in subsequent teaching.

[0031] In one possible implementation, after the trainee completes one or more rounds of consultations and writes the initial medical record, the system will collect and analyze the entire process. First, all patient responses verified by the medical verification agent are saved, forming a complete interaction record. The trainee's initial medical record is also archived by the system, serving as an important basis for evaluating their history taking and information organization abilities. The initial medical record refers to the first medical record information for the corresponding patient as defined in the "Basic Standards for Medical Record Writing."

[0032] In one possible implementation, the teaching feedback agent analyzes whether the trainee covered key information points in the medical record during the consultation, whether the order of questions conformed to clinical logic, and whether the expression conformed to communication norms; and whether the initial medical record submitted by the trainee accurately and completely reflects the patient's medical information. Based on the set evaluation indicators, the teaching feedback agent generates personalized feedback reports, which clearly point out the deficiencies.

[0033] The specific steps for using a multimodal artificial intelligence virtual patient consultation teaching system are as follows:

[0034] Step 1: Construction and initialization of medical records

[0035] After the patient's medical records are standardized and organized into medical record drafts, the diverse medical record drafts are tagged and organized to form a medical record database, and a unique identifier is assigned to each case.

[0036] Step 2: Student Input Analysis and Dialogue Plan Generation

[0037] Trainees interact with the system through natural language. The text input by the trainee first enters the dialogue planning agent, which parses the statements, identifies key information points, and uses semantic mapping to associate them with corresponding factual units in the medical record. After identifying the key information points, the dialogue planning agent generates a dialogue plan. Following the generation of the dialogue plan, the dialogue planning agent calls the medical record orchestration agent to extract the factual units in the medical record that correspond to the trainee's questions.

[0038] Step 3: Overlaying the portrait with emotion

[0039] After the dialogue planning agent generates a dialogue plan and obtains factual units from the case file, the appropriate expression method is determined through profiling and emotional agents.

[0040] The image acquisition module collects real-time images of patients' facial expressions and body postures. A dual-stream emotion representation fusion network identifies the emotional expression in the patients' facial expressions and body language, adjusting the patient's emotion in real time based on the patient's mood. The profiling and emotion intelligence agent combines factual units provided by the case arrangement intelligence agent with the patient's emotion vector, transforming them into natural language expressions that closely resemble those of real patients.

[0041] Step 4: Medical Verification and Logical Review

[0042] After generating a natural language response that includes profile and emotional features, it is thoroughly reviewed by a medical verification agent to ensure that the output conforms to the factual units in the case file and is consistent in logical expression.

[0043] Step 5: Case Evaluation and Teaching Feedback Generation

[0044] After trainees complete one or more rounds of consultations and write their initial medical records, the system will collect and analyze the entire process. The teaching feedback agent will then comprehensively compare and evaluate the interaction records and the initial medical records.

[0045] The beneficial effects of this invention are as follows:

[0046] 1. Improving the medical accuracy and interactive realism of virtual patients. This invention achieves accurate presentation of medical information and authentic natural language expression in virtual patients during consultations through the collaborative work of multiple intelligent agents, including a case arrangement agent, a dialogue planning agent, a profiling and emotion agent, a medical verification agent, and a teaching feedback agent. The system can automatically identify the trainee's questioning intent, retrieve factual units from the case draft, and generate answers that conform to medical logic and humanistic characteristics by combining the patient's profile and emotional state, thereby significantly improving the realism and medical rationality of the consultation interaction.

[0047] 2. Achieving Emotional Interaction and Adaptive Patient Profile Generation. The system utilizes an image acquisition module and a dual-stream emotion representation fusion network to identify trainees' emotional characteristics in real time and dynamically influence the emotional state of virtual patients. The patient profile and emotional intelligence agent combine the patient's profile characteristics with their emotional state to generate natural language responses that fit the actual clinical communication context, simulating "two-way emotional feedback between doctors and patients," greatly enhancing the interactive immersion and teaching experience.

[0048] 3. Ensuring logical consistency and factual reliability in medical expression. The medical verification agent reviews each generated content to ensure that the output is consistent with the medical record and conforms to medical logic and common sense. This avoids medical errors or contradictions caused by model reasoning in traditional virtual patient systems, significantly improving the reliability of the system output and the safety of teaching.

[0049] 4. Achieve a closed-loop teaching assessment and personalized feedback. The teaching feedback intelligent agent compares the consultation interaction with the student's medical records and generates personalized feedback reports based on standardized scoring indicators. This not only points out the student's shortcomings but also provides targeted improvement suggestions, promoting the student's comprehensive improvement in areas such as history taking, clinical reasoning, and standardized medical record writing, thus forming a complete teaching closed loop.

[0050] 5. Scalable and low-risk teaching environment: The multi-agent collaborative system constructed in this invention can be flexibly expanded according to different teaching tasks, disease types, and teaching stages, supporting simulation training in multiple scenarios and with multiple cases. The system does not rely on the participation of real patients, reducing teaching risks and costs, while ensuring the stability and reusability of teaching resources. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is a flowchart illustrating the overall system architecture of an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall structure of the portrait and emotional intelligence agent in an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] This application provides a multimodal AI-powered virtual patient consultation teaching system, including a case arrangement intelligent agent, a dialogue planning intelligent agent, a profiling and emotion intelligent agent, an image acquisition module, a medical verification intelligent agent, and a teaching feedback intelligent agent. This system improves the medical accuracy and interactive realism of virtual patients, addressing issues such as insufficient standardized patient resources, high case heterogeneity, and limited personalized feedback in existing clinical consultation teaching. The case arrangement intelligent agent is used to load and initialize the medical record. Loading refers to the case arrangement intelligent agent reading all fields of the medical record from the medical record database and converting them into internal data that can be subsequently accessed. Initialization refers to the case arrangement intelligent agent parsing each element in the medical record, extracting callable fact units through named entity recognition technology, and storing them in an internal buffer. Figure 1 This is a flowchart illustrating the overall system architecture of an embodiment of the present invention.

[0056] The dialogue planning agent is used to parse the statements in the student's input text and identify the key information points therein. Through semantic mapping, it establishes an association between the statements and the corresponding fact units in the medical record and generates a dialogue plan.

[0057] The profile and emotional intelligence agent are used to determine the appropriate expression based on the patient's emotional vector after obtaining the dialogue plan and fact unit, and generate a natural language response.

[0058] The image acquisition module is used to collect real-time images of the facial expressions and body postures of the patients during consultations.

[0059] The dual-stream emotion representation fusion network is used to identify the emotional expressions of trainees through facial expressions and body language, and adjusts the patient's emotion in real time based on the trainee's emotions. The dual-stream emotion representation fusion network includes a facial expression recognition submodule, a body posture emotion submodule, and a dual-stream fusion submodule. The facial expression recognition submodule and the body posture emotion submodule receive facial expression and body posture image information from trainees, respectively, and generate facial modality emotion representation and body modality emotion representation. Finally, the dual-stream fusion submodule generates a joint emotion feature vector of the trainee. This joint emotion feature vector is further input to a linear projection layer, which adapts the dimensions of the profile to the input feature vector dimensions required by the emotional intelligence agent. The feature vector after linear projection is called the patient's emotion vector.

[0060] The medical verification agent comprehensively reviews the natural language responses generated by the profiling and emotional agents to ensure that the output conforms to the factual units in the medical record and remains consistent in logical expression.

[0061] The teaching feedback agent is used to comprehensively compare and evaluate the interaction records and the initial medical records written by the students.

[0062] The case arrangement agent, dialogue planning agent, profiling and emotion agent, medical verification agent, and teaching feedback agent are all implemented based on a large language model.

[0063] The usage method of a multimodal artificial intelligence virtual patient consultation teaching system is as follows:

[0064] Step 1: Construction and initialization of medical records

[0065] To implement the standardized patient simulation of this invention, it is first necessary to establish case files. These case files refer to a collection of specific clinical patient case information gathered and organized through multiple channels within the scope permitted for medical teaching and research. Sources of this patient case information include: publicly published medical cases and academic papers, clinical treatment guidelines published by authoritative publishers, medical textbooks, and authentic medical records approved by the ethics committee for teaching purposes. By organizing the patient case information and treatment plan information from the above sources, and referring to the "Basic Norms for Medical Record Writing" (Document No. 11 of 2010 issued by the Ministry of Health and Medical Administration) to write the corresponding patient's initial medical record (where the patient's past medical history includes information on "personal history, marital and reproductive history, menstrual history, and family history" in the prescribed current medical history document; the previously prescribed past medical history is now uniformly referred to as "past medical history"), this document serves as the standard storage and retrieval format for the case files. Based on a large number of case files, a case database covering multiple disease types, different disease stages, and diverse manifestations is constructed. In short, after standardizing and organizing patient case information to form case drafts, the diverse case drafts are then tagged and organized to form a case database.

[0066] When creating medical record drafts, the original patient medical information needs to be anonymized and standardized. Anonymization is performed at the field level, including deleting or replacing direct identifying information such as name, ID number, hospital number, and phone number, as well as obscuring indirect identifying information such as geographical location, date of birth, and workplace. This process ensures that the patient medical information can be used to construct simulated patients without infringing on the privacy of real patients.

[0067] During data standardization, patient case information needs to be reorganized according to a pre-designed data structure. This data structure is the initial medical record structure stipulated in the Basic Norms for Medical Record Writing (Document No. 11 of 2010, issued by the Ministry of Health and Medical Administration), which is complete and comprehensive in information. It includes, but is not limited to, the following elements: disease diagnosis, patient's chief complaint, accompanying symptoms, physical examination results, past medical history, medication history, laboratory tests, imaging examinations, and auxiliary examinations. Each element is stored in a structured field format for easy retrieval by the intelligent agent. For example, "chief complaint" can be set as a string field to record the patient's main symptoms at the time of consultation; "accompanying symptoms" can be set as a list field to record multiple related manifestations that may occur; and "past medical history" includes subfields such as surgical history, medication history, and family history to ensure comprehensiveness and scalability.

[0068] After data anonymization and structuring, the case files are stored in a case database, with each case assigned a unique identifier. This identifier is used for quick retrieval and recall during subsequent consultation simulations, avoiding case confusion. The case files in the database are grouped and managed according to the patient's primary diagnosis, the organ systems involved in the patient's disease, or specific teaching objectives to support different teaching needs.

[0069] The case orchestration agent is used to load and initialize case files. Loading refers to the agent reading all fields of the case file from the case database and converting them into internal data that can be used later. Initialization refers to the agent parsing each element in the case file and extracting callable fact units using named entity recognition technology. For example, for the patient's chief complaint: recurrent hematuria for three weeks, the agent breaks it down into the fact unit {symptom: hematuria, duration: three weeks, nature: recurrent} and stores it in an internal buffer for on-demand retrieval during the dialogue.

[0070] Through the aforementioned steps of constructing and initializing medical records, this invention establishes a privacy-preserving, structurally clear, and accessible medical record database. This database provides factual evidence for subsequent multi-agent collaboration, ensuring the medical reliability and consistency of responses generated during the consultation process.

[0071] Step 2: Student Input Analysis and Dialogue Plan Generation

[0072] After loading and initializing the medical records, trainees interact with the system through natural language. The text input by the trainee first enters the dialogue planning agent, which parses the statements and identifies key information points. For example, when a trainee asks, "Is the patient's pain continuous or intermittent?", the system will recognize that the question involves "the nature of the pain" and, through semantic mapping, establish an association between it and the corresponding factual units in the medical records.

[0073] Semantic mapping is achieved using a large language model:

[0074] After identifying key information points, the dialogue planning agent generates a dialogue plan. This dialogue plan refers to determining the necessary medical information to be retrieved and the order and manner of information return based on the content of the student's current question, combined with factual units already loaded in the medical record, semantic relevance, clinical logic, and dialogue context information, through reasoning using a large language model. In this way, the system ensures that the student's questions receive targeted answers while maintaining the logical flow and consistency of the consultation process.

[0075] After the dialogue plan is generated, the dialogue planning agent invokes the case orchestration agent to extract factual units from the case file that correspond to the trainee's questions. To ensure the security and standardization of the interaction, the dialogue planning agent has an exception control mechanism when processing trainee input. When the trainee's question exceeds the scope of the case file, the system will output "The patient failed to provide relevant information"; when the trainee's question has the probability of leading to content that does not comply with medical ethics or teaching standards, the system will automatically block it to avoid outputting inappropriate answers.

[0076] Through the above process, the dialogue planning agent realizes a complete processing chain from student input to dialogue plan generation: first, it parses the student input and identifies key information points; then, it generates a dialogue plan; and finally, it calls upon the case orchestration agent to provide the corresponding fact units. This mechanism ensures the logical structure and medical rationality of the consultation interaction, laying the foundation for subsequent emotion expression generation, medical verification, and teaching feedback.

[0077] Step 3: Overlaying the portrait with emotion

[0078] After the dialogue planning agent generates a dialogue plan and obtains factual units from the medical record, in order to ensure that the output accurately reflects medical facts while also embodying the patient's profile information and emotional state, a profile and emotional agent ( Figure 2 The appropriate expression method is determined. The profile information refers to the patient's gender, age, occupation, education level, personal history, marital and reproductive history, menstrual history, and family history characteristics in the retrieved medical records; the emotional state refers to the emotions that the patient may exhibit, such as anxiety, tension, pain, or calmness. For example, when the fact unit in the medical records is {symptom: hematuria, duration: three weeks, nature: recurrent}, the dialogue planning agent will request the medical record arrangement agent to provide this information, and then the profile and emotion agent will convert it into a natural language expression that conforms to the patient's identity and emotions, such as "For the past three weeks, I seem to often find some red, blood-like stuff in my urine."

[0079] This system initializes the patient's emotion as "calm," then collects real-time images of the patient's facial expressions and body postures using an image acquisition module. A dual-stream emotion representation fusion network, combining a profile and an emotional intelligence agent, identifies the patient's facial expressions and body language emotional expressions. The system adjusts the patient's emotion in real-time based on the patient's emotions; for example, if the patient expresses "contempt," the patient's emotion changes to "anger." The dual-stream emotion representation fusion network includes a facial expression recognition submodule, a body posture emotion submodule, and a dual-stream fusion submodule. The facial expression recognition and body posture emotion submodules receive the patient's facial expression and body posture images, respectively, generating facial modality and body modality emotion representations. Finally, the dual-stream fusion submodule generates a joint emotion feature vector for the patient.

[0080] The facial expression recognition submodule employs the lightweight convolutional neural network MobileNetV3 to perform multi-layer convolution operations on the input facial image, extracting local texture features and muscle movement features of the facial region. After global average pooling, a high-dimensional feature representation, i.e., the overall feature vector, is formed. This feature representation is processed through two parallel output branches. One branch is the facial action unit recognition branch, which uses a multi-label classification structure and a sigmoid activation function to output the probability of occurrence of each action unit, and calculates the corresponding intensity value through a linear regression layer. The other branch is the emotion dimension regression branch, which performs a two-channel linear mapping on the overall feature vector to obtain continuous values ​​of emotion positivity (Valence) and activation level (Arousal), i.e., the emotion dimension value. Subsequently, the outputs of the two branches are fused at the feature level, that is, the action unit intensity feature and the emotion dimension feature are concatenated to form a comprehensive facial feature vector. After normalization, the comprehensive facial feature vector obtains a uniform-scale facial modality emotion representation, which is then input to the two-stream fusion submodule.

[0081] The body posture emotion submodule, based on the existing posture estimation algorithm OpenPose, extracts the skeletal keypoint coordinate sequence from the upper body image and inputs the keypoint coordinates of each frame into a temporal convolutional neural network in chronological order to model the dynamic features of body posture changes over time. During convolution calculation, the submodule uses the spatial coordinate difference of keypoints in adjacent frames as input features to calculate the relative displacement vector, joint angle change, and displacement amplitude between keypoints. A one-dimensional temporal convolution kernel slides along the time dimension to perform weighted summation and nonlinear transformation on the difference features, thereby extracting intermediate feature maps representing upper limb movement, trunk posture changes, and body stability. After multiple convolutions and feature aggregation, a low-dimensional emotion feature vector reflecting the dynamic changes in the trainee's body posture is generated and, after normalization, used as a body modality emotion representation input to the dual-stream fusion submodule.

[0082] The dual-stream fusion submodule is used to fuse facial modal emotion representations output from the facial expression recognition submodule and body modal emotion representations output from the body posture emotion submodule. Specifically, the facial modal emotion representations output from the facial expression recognition submodule and the body modal emotion representations output from the body posture emotion submodule are first weighted and summed to obtain a preliminary joint feature vector. During the weighted summation fusion process, a confidence gating mechanism is introduced to dynamically adjust the weights of the facial and body modal emotion representations based on the confidence scores of face detection and skeleton keypoint detection. The preliminary joint feature vector represents the comprehensive emotional features of the face and body within a single frame. In the temporal dimension, the dual-stream fusion submodule uses a sliding time window mechanism to aggregate the joint feature vectors from several consecutive frames to form a feature sequence. Subsequently, a one-dimensional temporal convolution operation is performed on this feature sequence to capture the coordinated changes in facial expressions and body postures over short time scales, thereby enhancing the temporal modeling capability of dynamic emotional changes. This mechanism effectively suppresses noise caused by factors such as occlusion, lighting changes, or detection instability, improving the reliability of the fusion results. After processing with one-dimensional temporal convolution, the final joint emotion feature vector of the trainees is obtained. This vector integrates spatial modality and temporal information at the feature level, and can comprehensively represent the overall emotional state of the trainees in the current time period.

[0083] The trainee's joint emotional feature vector is further input into a linear projection layer to adapt the dimensions of the input feature vectors required by the profile and the emotional agent. The feature vector after linear projection is called the patient's emotional vector. The profile and the emotional agent combine the factual units provided by the case arrangement agent with the patient's emotional vector, transforming them into natural language expressions that closely resemble those of real patients. For example, when the factual unit is {symptom: hematuria, duration: three weeks, nature: recurrent}, if the patient profile is a "middle-aged male" and the trainee's joint emotional feature vector leans towards "perfunctory" emotions, then the patient's emotional vector projected from it will lean towards "anxiety," and the output might be: "I've had blood in my urine every time for the past three weeks, I'm so worried." If the patient profile is an "elderly woman" and the trainee's joint emotional feature vector leans towards "patience" emotions, then the patient's emotional vector will lean towards "relaxation," and the output might be: "I've had blood in my urine frequently for the past three weeks, but I haven't felt particularly uncomfortable."

[0084] Meanwhile, the profiling and emotional intelligence agents have limiting mechanisms to avoid generating diagnostic statements or statements that exceed the patient's identity, ensuring that responses always align with the patient's role. Through this process, the system enhances the realism of interaction and the effectiveness of teaching while maintaining the accuracy of medical facts, enabling better simulation of consultations in emergency rooms, special treatment scenarios, and complex doctor-patient communication environments.

[0085] Step 4: Medical Verification and Logical Review

[0086] After generating a natural language response that includes profile and emotional features, it is thoroughly reviewed by a medical verification agent to ensure that the output conforms to the factual units in the case file and is consistent in logical expression.

[0087] The medical verification agent first compares the responses item by item with the medical record, checking whether the content output by the profile and the emotional agent accurately reflects the relevant information in the medical record. For example, if the medical record states "recurrent hematuria for three weeks," while the output only states "hematuria was only occasionally found during urination," it is judged to be inconsistent with the record and needs to be regenerated. This step ensures that the output is consistent with the medical record and has traceability.

[0088] After comparing the results with the medical records, the medical verification agent performs a logical consistency check. This check, based on common rules in the medical records and the medical knowledge base, ensures that the output does not contain contradictions or conflict with basic medical common sense. For example, if the answer contains both "persistent high fever for several weeks" and "no fever," it is judged as a logical contradiction and needs to be corrected. In this way, the system's answers are guaranteed to be medically reasonable while remaining within the scope of the patient's identity.

[0089] When the medical verification agent confirms that the answer is consistent with the medical record and logical rules, the system outputs the answer to the trainee as a valid result of the consultation interaction, allowing the trainee to continue to the next round of questions. If there are problems, the system returns to the dialogue planning agent, requiring regeneration. The entire verification process is recorded in the system log for teachers and administrators to review and evaluate in subsequent teaching. Through this verification and logical review process, this invention ensures that the virtual patient's answers are reliable in terms of both medical facts and logic, thereby improving the stability of consultation training and the effectiveness of teaching.

[0090] Step 5: Case Evaluation and Teaching Feedback Generation

[0091] After trainees complete one or more rounds of consultations and write their initial medical records, the system will collect and analyze the entire process. First, all patient responses verified by the medical verification agent are saved, forming a complete interaction record. The trainees' initial medical records are also archived by the system, serving as an important basis for evaluating their history taking and information organization abilities. The initial medical record refers to the patient's first medical record information as stipulated in the Basic Norms for Medical Record Writing (Document No. 11 of 2010 issued by the Ministry of Health and Medical Administration) (where the patient's past medical history should include information on "personal history, marital and reproductive history, menstrual history, and family history" as specified in the current medical history document).

[0092] Based on this, the teaching feedback agent comprehensively compares and evaluates the interaction records and the initial medical record. The analysis includes whether the trainee covered key information points in the case draft during the consultation, whether the order of questions conformed to clinical logic, and whether the expression conformed to communication norms; and whether the initial medical record submitted by the trainee accurately and completely reflects the patient's case information. The specific standard indicators (hereinafter referred to as "indicators") referenced in the above analysis in this embodiment are detailed in Appendix 1, "SOAP Initial Medical Record Scoring Standard." Based on the set evaluation indicators, the teaching feedback agent generates a personalized feedback report, clearly pointing out deficiencies and providing improvement suggestions, such as "failure to follow up on accompanying symptoms" or "lack of family history records in the initial medical record."

[0093] The generated feedback reports can be provided to trainees for self-reflection immediately, and can also be compiled for instructors to review, thus assisting in teaching management and assessment. Through this process, the invention ensures a closed loop between medical history taking and medical record writing training, effectively improving trainees' medical knowledge, clinical reasoning, and writing standards.

[0094] Through the above steps, this invention enables standardized patient simulation training based on multi-agent collaboration, providing a low-risk, repeatable, and scalable teaching environment while ensuring the accuracy of medical facts and the realism of interaction. This system effectively solves the problems of limited resources, high case heterogeneity, and insufficient feedback in traditional teaching models, offering a novel intelligent solution for medical education.

[0095] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multimodal artificial intelligence virtual patient consultation teaching system, characterized in that, This includes a case arrangement agent, a dialogue planning agent, a profiling and emotion agent, an image acquisition module, a two-stream emotion representation fusion network, a medical verification agent, and a teaching feedback agent; The case arrangement agent is used to load and initialize the case files; loading refers to the case arrangement agent reading all fields of the case files from the case database and converting them into internal data that can be called later; initialization refers to the case arrangement agent parsing each element in the case files, extracting callable fact units through named entity recognition technology, and storing them in an internal buffer. The dialogue planning agent is used to parse the sentences in the student's input text and identify the key information points therein, and through semantic mapping, establish an association between them and the corresponding fact units in the medical record to generate a dialogue plan; The image acquisition module is used to collect real-time image information of the facial expressions and body postures of the patients during consultations. The profiling and emotional intelligence agent is used to process the information acquired by the image acquisition module based on the dual-stream emotion representation fusion network after obtaining the dialogue plan and fact unit to generate the patient's emotion vector. Finally, it determines the appropriate expression method based on the patient's emotion vector and generates a natural language response. The dual-stream emotion representation fusion network is used to identify the emotional expression of trainees through facial expressions and body language, and adjusts the patient's emotion in real time according to the trainee's emotion. The dual-stream emotion representation fusion network includes a facial expression recognition submodule, a body posture emotion submodule, and a dual-stream fusion submodule. The facial expression recognition submodule and the body posture emotion submodule receive the trainee's facial expression and body posture image information, respectively, and generate facial modal emotion representation and body modal emotion representation. Finally, the dual-stream fusion submodule generates a joint emotion feature vector of the trainee. The joint emotion feature vector of the trainee is further input into a linear projection layer, which adapts the dimensions of the profile and the input feature vector dimensions required by the emotional intelligence agent. The feature vector after linear projection is called the patient's emotion vector. The medical verification agent comprehensively reviews the natural language responses generated by the profiling and emotional agents to ensure that the output conforms to the factual units in the medical record and remains consistent in logical expression. The teaching feedback agent is used to comprehensively compare and evaluate the interaction records and the first medical records written by the students. The case arrangement agent, dialogue planning agent, profiling and emotion agent, medical verification agent, and teaching feedback agent are all implemented based on a large language model.

2. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, The aforementioned case files refer to a collection of specific clinical patient case information gathered and organized through multiple channels within the scope permitted for medical teaching and research. Sources of patient case information include: publicly published medical cases and academic papers, clinical treatment guidelines published by authoritative publishers, medical textbooks, and authentic medical records approved by the ethics committee for use in teaching purposes. By organizing the patient case information and treatment plans from the above sources, and referring to the "Basic Norms for Medical Record Writing," the initial medical record file for the corresponding patient is written as the standard storage and retrieval format for the case files. The diverse case files are then tagged and organized to form a case database.

3. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 2, characterized in that, When creating the medical record drafts, the original patient medical record information needs to be anonymized and standardized. The de-identification process employs a field-level de-identification method, including deleting or replacing direct identification information and obfuscating indirect identification information; During the data standardization process, patient case information needs to be reorganized according to a pre-designed data structure. The data structure is the initial medical record structure specified in the "Basic Norms for Medical Record Writing", which includes, but is not limited to, the following elements: disease diagnosis, patient chief complaint, accompanying symptoms, physical examination results, past medical history, medication history, laboratory tests, imaging examinations, and auxiliary examinations. Each element is stored in the form of structured fields to facilitate subsequent intelligent agent calls. After data anonymization and structuring, the case files are stored in the case database, and each case is assigned a unique identifier. The case files in the case database are grouped and managed according to the patient's primary diagnosis, the organ system involved in the patient's disease, or specific teaching objectives to support the needs of different teaching tasks.

4. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, The dialogue planning agent uses a large language model to achieve semantic mapping: After identifying key information points, the dialogue planning agent generates a dialogue plan; the dialogue plan refers to determining the medical information to be called and the order and method of information return based on the content of the current student's question, combined with the fact units loaded in the medical record, and by integrating semantic relevance, clinical logic and dialogue context information, through reasoning of a large language model. After the dialogue plan is generated, the dialogue planning agent calls the case arrangement agent to extract the factual units in the case file that correspond to the trainee's questions. The dialogue planning agent has an exception control mechanism when processing trainee input. When the trainee's questions exceed the scope that the case file can cover, the system will output "The patient failed to provide relevant information". When the trainee's questions have the probability of leading to content that does not comply with medical ethics or teaching norms, the system will automatically block them to avoid outputting inappropriate answers.

5. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, The facial expression recognition submodule uses a lightweight convolutional neural network MobileNetV3 to perform multi-layer convolution operations on the input facial image, extract local texture features and muscle movement features of the facial region, and form a high-dimensional feature representation, i.e., the overall feature vector, after global average pooling. This feature representation is processed through two parallel output branches. One branch is the facial action unit recognition branch, which uses a multi-label classification structure and a Sigmoid activation function to output the occurrence probability of each action unit and calculates the corresponding intensity value through a linear regression layer. The other branch is the emotion dimension regression branch, which performs a two-channel linear mapping on the overall feature vector to obtain continuous values ​​of emotion positivity and activation level, i.e., emotion dimension values. Subsequently, the outputs of the two branches are fused at the feature level, that is, the action unit intensity feature and the emotion dimension feature are concatenated to form a comprehensive facial feature vector. After normalization, the comprehensive facial feature vector obtains a uniform-scale facial modality emotion representation, which is used as input to the two-stream fusion submodule. The body posture emotion submodule is based on the posture estimation algorithm OpenPose. It extracts the skeleton key point coordinate sequence from the upper body image and inputs the key point coordinates of each frame into the temporal convolutional neural network in chronological order to model the dynamic features of body posture changes over time. During the convolution calculation, the body posture emotion submodule uses the spatial coordinate difference of key points in adjacent frames as input features to calculate the relative displacement vector, joint angle change, and displacement amplitude between key points. Through a one-dimensional temporal convolution kernel sliding in the time dimension, the difference features are weighted and summed and nonlinearly transformed to extract intermediate feature maps representing upper limb movement, trunk posture changes, and body stability. After multiple convolutions and feature aggregation, a low-dimensional emotion feature vector reflecting the dynamic changes of the trainee's body posture is generated and normalized before being input as the body modality emotion representation to the two-stream fusion submodule. The dual-stream fusion submodule is used to fuse facial modal emotion representations output from the facial expression recognition submodule and body modal emotion representations output from the body posture emotion submodule. Specifically, the facial modal emotion representations output from the facial expression recognition submodule and the body modal emotion representations output from the body posture emotion submodule are first weighted and summed according to their weights to obtain a preliminary joint feature vector. During the weighted summation fusion process, a confidence gating mechanism is introduced to dynamically adjust the weights of the facial modal emotion representations and the body modal emotion representations based on the confidence of face detection and skeleton key point detection. In the time dimension, the dual-stream fusion submodule adopts a sliding time window mechanism to aggregate the joint feature vectors of several consecutive frames to form a feature sequence. Subsequently, a one-dimensional temporal convolution operation is performed on this feature sequence to obtain the final trainee joint emotion feature vector.

6. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, After the dialogue planning agent generates a dialogue plan and obtains factual units from the medical record, the profiling and emotional intelligence agent determines an appropriate expression method based on the patient's emotional vector, so that the output content can accurately reflect medical facts and also reflect the patient's profile information and emotional state; the profile information refers to the patient's gender, age, occupation, education level, personal history, marital and reproductive history, menstrual history and family history characteristics in the medical record; The case arrangement agent combines factual units provided by the case arrangement agent with patient emotion vectors to transform them into natural language expressions that closely resemble those of real patients. At the same time, the profiling and emotional agent have a constraint mechanism to avoid generating statements that are diagnostic or beyond the patient's identity, ensuring that the answers always conform to the patient's role.

7. A multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, The medical verification agent first compares the answers with the medical record document item by item to check whether the profile and the content output by the emotional agent accurately reflect the relevant information in the medical record document. After comparing the data with the medical records, the medical verification agent will perform a logical consistency check. This check is based on the general rules of the medical records and the medical knowledge base to ensure that the output content does not contradict itself or conflict with basic medical common sense. When the medical verification agent confirms that the answer is consistent with the medical record and logical rules, the system outputs the answer to the trainee as a valid result of the consultation interaction and allows the trainee to continue to the next round of questions; If a problem is found, the dialogue planning agent is returned and regenerated; the entire verification process is recorded in the system log for teachers and administrators to trace and assess the quality of subsequent teaching.

8. The multimodal artificial intelligence virtual patient consultation teaching system according to claim 1, characterized in that, After trainees complete one or more rounds of consultations and write their initial medical records, the system will collect and analyze the entire process. First, all patient responses confirmed as qualified by the medical verification agent are saved, forming a complete interaction record. The initial medical records written by trainees are also archived by the system as an important basis for evaluating their ability to collect medical history and organize information. The initial medical records refer to the first medical record information of the corresponding patient as stipulated in the "Basic Norms for Medical Record Writing".

9. A multimodal artificial intelligence virtual patient consultation teaching system according to claim 1 or 8, characterized in that, The teaching feedback agent analyzes whether trainees cover key information points in the case draft during consultations, whether the order of questions conforms to clinical logic, and whether the expression conforms to communication norms; whether the initial medical record submitted by trainees accurately and completely reflects the patient's case information; and based on the set evaluation indicators, the teaching feedback agent generates personalized feedback reports that clearly point out deficiencies.

10. A method for using a multimodal artificial intelligence virtual patient consultation teaching system, characterized in that, The specific steps are as follows: Step 1: Construction and initialization of medical records After the patient's medical records are standardized and organized into medical record drafts, the diverse medical record drafts are tagged and organized to form a medical record database, and a unique identifier is assigned to each case. Step 2: Student Input Analysis and Dialogue Plan Generation Trainees interact with the system through natural language for consultations; the text input by trainees first enters the dialogue planning agent, which parses the sentences and identifies key information points, and establishes an association between them and the corresponding fact units in the medical record through semantic mapping; after identifying key information points, the dialogue planning agent generates a dialogue plan; After the dialogue plan is generated, the dialogue planning agent calls the case orchestration agent to extract the factual units in the case draft that correspond to the trainee's questions. Step 3: Overlaying the portrait with emotion After the dialogue planning agent generates a dialogue plan and obtains factual units from the medical record, the appropriate expression method is determined through profiling and emotional agent. Based on the image acquisition module, the system collects facial expression and body posture images of the patients in real time. The system uses a dual-stream emotion representation fusion network to identify the emotional expression of the patients' facial expressions and body language. The system then adjusts the patient's emotions in real time according to the patient's emotions. The profiling and emotional intelligence agent combines factual units provided by the case arrangement intelligence agent with patient emotion vectors, and transforms them into natural language expressions that closely resemble real patients; Step 4: Medical Verification and Logical Review After generating a natural language response that includes profile and emotional features, it is thoroughly reviewed by a medical verification agent to ensure that the output conforms to the factual units in the medical record and is consistent in logical expression. Step 5: Case Evaluation and Teaching Feedback Generation After trainees complete one or more rounds of consultations and write their initial medical records, the system will collect and analyze the entire process; through the teaching feedback agent, the interaction records and initial medical records will be comprehensively compared and evaluated.

Images