A method and system for virtual standardized patient avatar generation and dialogue
By generating realistic virtual patient portraits using large language models and latent diffusion models, the problems of fragmented multimodal features and lack of interactive logic in virtual standardized patient systems are solved, enabling the automation of high-fidelity medical simulation teaching and the construction of a low-cost case database.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE SECOND XIANGYA HOSPITAL OF CENT SOUTH UNIV
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-12
AI Technical Summary
Existing virtual standardized patient systems suffer from technical bottlenecks such as fragmented multimodal features, lack of patient persona constraints in interaction logic, and reliance on manual customization for scenario construction, making it difficult to meet the needs of high-fidelity medical simulation teaching.
The system employs a large language model to extract multi-dimensional information from internet medical data, generates realistic virtual patient portraits through a text encoder and a potential diffusion model, and combines emotional factors to construct interactive logic, thereby achieving automated mapping from text to visual and speech and automatically building a high-fidelity virtual medical record library.
It achieves multimodal consistency between virtual patients and case descriptions, possesses complex interaction logic and low-cost, high-efficiency case database construction, and significantly enhances the immersion and realism of simulation teaching.
Smart Images
Figure CN121545653B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and medical education technology, specifically relating to a method and system for generating and communicating with virtual standardized patient images. Background Technology
[0002] In modern medical education and clinical skills assessment, standardized patients (SPs) are widely used to simulate real doctor-patient communication scenarios and are considered the "gold standard" for evaluating medical students' consultation skills, clinical reasoning, and humanistic care. However, training real SPs is costly, script standardization is difficult, and resources are extremely scarce, making it difficult to support large-scale, high-frequency teaching needs.
[0003] With the evolution of artificial intelligence technology, Virtual Standardized Patients (VSPs) have emerged. Early VSP systems were mostly based on pre-defined decision trees or finite state machines, with simple interaction paths and a lack of natural language understanding capabilities, which was referred to by the academic community as "menu-style consultation." Recently, the emergence of Large Language Models (LLMs), represented by ChatGPT, has provided a new technical path for constructing generative virtual patients.
[0004] Despite the strong potential of large language models in medical text processing, a review of existing technical literature reveals the following major technical bottlenecks when directly applying them to construct high-fidelity virtual standardized patients:
[0005] (1) Multimodal features are fragmented and lack visual mapping based on pathological logic (“visual mismatch”).
[0006] Most existing medical simulation systems focus on optimizing the text-based dialogue layer, while neglecting the logical consistency between visual phenotypes and medical text.
[0007] Current general-purpose generative AI models lack deep medical domain knowledge constraints, making it difficult to establish an accurate mapping from "clinical text descriptions" to "patient visual features." For example, a system might generate case text describing "severe liver failure," but the corresponding virtual image might not exhibit key signs such as "jaundice (yellowing of the sclera and skin)" or "palmar erythema." Authoritative literature points out that while LLM has made significant progress in processing clinical text, it still faces enormous challenges in integrating multimodal data (such as converting text into images with medical precision). Existing general-purpose models often produce "illusions" or generic, mediocre images that fail to meet the precision requirements of clinical visual diagnosis teaching.
[0008] (2) The interaction logic lacks the constraint of "patient persona", and the authenticity of emotions and behaviors is insufficient.
[0009] Dialogue systems based directly on general LLM often appear overly rational, compliant, or omniscient, lacking the concealment, hesitation, non-linear expression, and emotional fluctuations commonly seen in real patients during consultations.
[0010] Studies have shown that while general-purpose large models (such as GPT-4) can provide accurate medical information in medical scenarios, they are prone to "out-of-character" behavior when acting as "simulated patients." This means they either step out of the patient's role to offer professional diagnostic advice or fail to maintain emotional consistency in long-term conversations. Furthermore, research in the field of medical education in China also indicates that existing generative AI remains rigid in simulating patients' psychosocial responses and emotional empathy, making it difficult to effectively train doctors in the soft skills of uncovering hidden medical histories and comforting patients.
[0011] (3) Scene construction relies on manual customization and cannot utilize massive amounts of unstructured data.
[0012] Building high-fidelity virtual medical records typically requires clinical experts to spend considerable time writing structured scripts and technical teams to custom-develop 3D assets. Current technology lacks an automated pipeline capable of automatically extracting features, reconstructing logic, and directly generating interactive scenarios from massive amounts of unstructured medical Q&A data on the internet (such as online consultation records). This results in the virtual medical record database lagging far behind changes in the clinical disease spectrum. Summary of the Invention
[0013] This invention provides a method and system for generating and interacting with a virtual standardized patient image, which can generate a visual, audible, and intelligently interactive virtual patient entity based on static Internet medical text.
[0014] To achieve the above technical objectives, the present invention adopts the following technical solution:
[0015] A method for generating and engaging in dialogue with a virtual standardized patient avatar includes:
[0016] The large language model is used to extract multi-dimensional information about patients from raw medical data and construct structured case data, as well as to infer the visual characteristics of patients and integrate the visual characteristics of patients into portrait description prompts.
[0017] A text encoder is used to convert portrait description prompts into high-dimensional semantic vectors, which then guide a pre-trained latent diffusion model to generate patient portrait images that conform to medical data.
[0018] The system acquires the user's voice signal and converts it into natural language text as role prompts; it constructs system prompts based on the patient's structured medical records and preset emotion factors; and it generates response text for the user's voice signal using a large language model based on role prompts, system prompts, and historical dialogues.
[0019] The speech engine is activated to convert the reply text into a reply speech, which is then output through the speaker.
[0020] Furthermore, the extracted multi-dimensional information, including department affiliation, clinical diagnosis results, chief complaint, present illness history, past medical history, and allergy history, is mapped to obtain case data with a standard key-value pair structure.
[0021] Furthermore, infer the patient's visual characteristics, including biological features, pathological facial features, hair features, and facial expression features.
[0022] Furthermore, a text encoder based on the Transformer architecture is used to convert portrait description prompts into high-dimensional semantic vectors.
[0023] Furthermore, when the latent diffusion model generates a portrait image, it first initializes a random Gaussian noise matrix; then, using the high-dimensional semantic vector obtained by converting prompt words as a guide, it gradually predicts and subtracts noise through a multi-step reverse diffusion process to reconstruct the latent feature map; then, the latent image features are mapped to the pixel space through a decoder to obtain the patient's portrait image.
[0024] Furthermore, the speech signal is converted into natural language text. Using a model based on the Vosk architecture, the acoustic features are first extracted and decoded, the continuous audio waveform is converted into a phoneme sequence, and then the phoneme sequence is decoded into natural language text.
[0025] Furthermore, using a text-to-speech engine, based on the biological characteristics in the patient's visual features obtained through reasoning, the corresponding timbre model is automatically matched, and by analyzing the prosodic features of the text, a digital speech signal with intonation fluctuations is synthesized.
[0026] Furthermore, the emotional factors include anxiety, irritability, fear, sadness, and / or trust.
[0027] A virtual standardized patient avatar generation and dialogue system includes:
[0028] The feature extraction and reasoning module is used to: extract multi-dimensional information of patients from raw medical data using a large language model and construct structured case data, and to reason about the visual features of patients and integrate the visual features of patients into portrait description prompts;
[0029] The visual generation module is used to: convert portrait description prompts into high-dimensional semantic vectors using a text encoder, thereby guiding a pre-trained latent diffusion model to generate patient portrait images that conform to medical data;
[0030] The voice interaction module is used to: acquire the user's voice signal and convert it into natural language text as role prompts; construct system prompts based on the patient's structured medical data and preset emotion factors; generate reply text for the user's voice signal using a large language model and based on role prompts, system prompts, and historical dialogues; and finally convert the reply text into reply speech and output it through a speaker.
[0031] Compared with the prior art, the present invention has the following technical effects:
[0032] (1) High multimodal consistency: This invention utilizes large language model reasoning to achieve automated logical mapping from text case to visual image, ensuring that the generated virtual patients are highly consistent in appearance characteristics (such as age, gender, and illness appearance) and internal condition (such as disease type and severity), significantly improving the immersion and realism of simulated teaching.
[0033] (2) The interactive logic has clinical depth: By integrating emotional factors and nonlinear expression logic, this invention can simulate the complex dynamics of doctor-patient communication in real medical scenarios. It not only tests the doctor's diagnostic ability, but also effectively trains the doctor's clinical soft skills in uncovering hidden information and soothing the patient's emotions.
[0034] (3) Large data throughput and low generation cost: The present invention fully automates the entire process from raw data cleaning to final interactive scene generation. There is no need to manually write scripts or draw 3D models. It can quickly build a huge case database covering rare and common diseases based on existing Internet medical big data, and has extremely high scalability and economic benefits. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the method for generating and interacting with a virtual standardized patient image as described in the embodiments of this application.
[0036] Figure 2 This is a schematic diagram of the system for generating and interacting with a virtual standardized patient image as described in the embodiments of this application. Detailed Implementation
[0037] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0038] Example 1
[0039] This embodiment provides a method for generating and engaging in dialogue with a virtual standardized patient avatar, referencing... Figure 1 , Figure 2 As shown, it includes:
[0040] Step 1: Use a large language model to extract multi-dimensional information about patients from the raw medical data and construct structured case data, as well as infer the visual characteristics of patients and integrate the visual characteristics of patients into portrait description prompts.
[0041] The raw medical data in this embodiment is obtained by random sampling from a massive unstructured raw medical dataset. The unstructured raw medical data text (containing discrete dialogue fragments, medical orders, and descriptions) is input into the large language model as role prompt words. By constructing specific prompt engineering instructions, the large language model is instructed to perform multi-dimensional information extraction and logical reasoning tasks. Specific prompts are implemented through system prompts, such as "Integrate and extract content, correct inconsistencies, and strictly output according to the corresponding items: department, hospital name, medical history, symptoms, chief complaint, diagnosis, dialogue, and portrait." Another example is "Generate the patient's portrait features based on the provided case description. The following points should be noted: Gender estimation: Estimate gender based on the characteristics of the disease to ensure accuracy. For example, breast cancer is female, prostate cancer is male. Facial features: Even without a clear description, estimate common facial features such as eyes, nose, and mouth. Skin color: Select skin color based on health status or common preferences. Hair: Estimate hairstyle (e.g., short hair, long hair) and color. Expression and background: Estimate expression (e.g., fatigue, anxiety) and simple background based on the case. If a feature is missing from the case, a reasonable guess can be made. The portrait is ultimately compiled into a complete English sentence describing a realistic human portrait."
[0042] The specific processing details are as follows: First, the large language model cleans the original medical data text, removing irrelevant noise. Then, based on the medical knowledge graph integrated within the large language model, it identifies key entities in the text, including department affiliation, clinical diagnosis results, chief complaint, present illness history, past medical history, and allergy history, mapping them into a standardized key-value pair structure (JSON format). Simultaneously, the large language model performs a visual reasoning task based on pathological features. Specifically, based on the diagnostic results and symptom descriptions in the key-value pair structure obtained by the large language model, it logically infers the patient's visual characteristics: biological characteristics (gender, age group), pathological facial features (such as skin color, mental state), hair features, and facial expression features. Subsequently, the large language model translates and integrates these inferred visual features into a portrait description prompt that conforms to the semantic understanding specifications of image generation models, thus completing the cross-modal feature mapping from "medical text" to "visual semantics."
[0043] In this embodiment, the large language model is based on a probabilistic prediction model. Given the context of a case text, the goal is to maximize the probability of predicting a certain visual feature description and infer various visual features. Here, P represents the conditional probability distribution function; Context represents the original unstructured case text input to the large language model (including chief complaint, present medical history, etc.); and Feature represents the specific visual feature description inferred by the model (such as "pale complexion," "sunken eyes," etc.).
[0044] This formula expresses the meaning of known case texts. Under certain conditions, infer a certain visual feature The maximum likelihood probability.
[0045] Step 2 involves using a text encoder to convert portrait description prompts into high-dimensional semantic vectors, which then guide a pre-trained latent diffusion model to generate patient portrait images that match the medical data. This step utilizes a pre-loaded SDXL (StableDiffusion XL, a mainstream AI model for text-to-image generation) pipeline to achieve a high-fidelity visual representation of the virtual patient.
[0046] First, the portrait description prompts obtained in step 1 are input into a Transformer-based text encoder to transform the text description into a high-dimensional semantic vector. Then, this high-dimensional semantic vector is injected into the denoising network of a pre-trained latent diffusion model.
[0047] When generating a portrait image, the latent diffusion model first initializes a random Gaussian noise matrix. Then, using a high-dimensional semantic vector obtained from prompt word transformation as a guide, it progressively predicts and subtracts noise through a multi-step reverse diffusion process to reconstruct a latent feature map that conforms to the semantic description. A decoder then maps the latent image features to pixel space, generating a patient portrait image with realistic lighting details and conforming to medical reasoning characteristics (such as a haggard appearance due to a specific disease). This image is then fixed as the visual front end of the virtual case file, displayed in the interactive interface, and serves as the visual object for doctors during consultations.
[0048] The core of the latent diffusion model in this embodiment is learning the data distribution p(x). The generation process is the inverse of a Markov chain, and the iterative logic is as follows:
[0049] ;
[0050] in: This indicates the time step during the reverse diffusion process. The latent feature map at that time is the feature vector that is closer to the target image after denoising calculation; Indicates the current time step Noisy latent feature map at that time; This represents the preset noise variance scheduling parameters, used to control the step size and magnitude of each denoising step; express The cumulative product of the parameters reflects the original signal strength retained from the initial state to the current time. This refers to the trained denoising neural network (i.e., U-Net) in this invention. These represent the model parameters of the network; This indicates that the neural network is based on input features and time step The predicted noise components; represents the variance coefficient that controls the randomness of the generation process; represents the introduced random Gaussian noise used to increase the texture detail and diversity of the generated image.
[0051] Step 3: Acquire the user's voice signal and convert it into natural language text as role prompts; construct system prompts based on the patient's structured case data and preset emotion factors; use a large language model and based on the role prompts and system prompts to generate the response text for the user's voice signal.
[0052] Step 3.1, Auditory Perception and Transcription: Acquire the user's speech signal through an audio acquisition device, extract and decode the acoustic features using an existing Vosk-based model, convert the continuous audio waveform into a phoneme sequence, and then decode it into natural language text as role prompts.
[0053] Step 3.2, Semantic Understanding and Response Generation: In the initialization phase, the structured case data extracted in Step 1 (including concealed diagnoses and specific symptom descriptions) and preset emotional factors are injected into the context window of the large language model to construct system prompts. In addition to the structured case data and preset emotional factors, the system prompts also strictly limit the behavior pattern of the large language model by setting prompts, requiring it to simulate the non-professional expression logic of real patients and to set dynamic emotional states according to the patient's condition. It strictly prohibits the active disclosure of core diagnostic information, for example, "You are conducting a doctor-patient communication. Describe your symptoms step by step, do not actively reveal the diagnosis name, leave it to the doctor to judge." The role prompts, system prompts, and historical dialogue context are input into the large language model. The large language model analyzes the current dialogue context based on its internally integrated attention mechanism, combining the preset patient persona and disease logic (which belong to the structured case data extracted by the large language model from the original medical data) to generate a natural language response that matches the current emotional state.
[0054] In this embodiment, the preset emotional factors and current emotional state are both derived from a preset emotional factor library, obtained through random sampling or conditional probability sampling based on the user's speech rate. The emotional factor library can include common negative emotions of patients, such as: "Anxiety"—"You feel anxious, worried about the severity of your condition or the risks of surgery."; "Irritability"—"You feel irritable about your physical condition, impatient with your condition that hasn't improved over a long period."; "Fear"—"You are somewhat fearful, worried about your condition worsening or the need for surgery."; "Sadness"—"You feel down about your condition and lack confidence."; "Trust"—"You trust your doctor, are willing to cooperate with treatment, and express patience and confidence." etc.
[0055] This embodiment uses a large language model and outputs natural language response text based on system prompts and role prompts. Specifically, the large language model employs a Transformer-based autoregressive model to calculate conditional probabilities:
[0056] ;
[0057] in: This indicates the current state of the large language model. Lexical units to be generated; This represents the sequence of historical response terms generated before the current moment; The model input consists of system cue words composed of historical dialogue context, structured case data, and emotions from a potential emotion factor library, as well as user speech-transcribed text (i.e., role cue words); the formula indicates that the model is based on an attention mechanism, adjusting according to context. Predicting the next word from historical sequences The probability of.
[0058] Step 4, Speech Synthesis and Feedback: Activate the speech engine to convert the reply text into reply speech, and output the reply speech through the speaker.
[0059] Specifically, the generated natural language response text is transmitted to a text-to-speech engine. In this embodiment, the existing Edge-TTS engine is used. Based on the gender and age characteristics inferred in step 1, the engine automatically matches the corresponding timbre model. By analyzing the prosodic features of the text, it synthesizes a digital speech signal with intonation fluctuations, which is then played through an audio output device, thus completing a full doctor-patient interaction cycle. For example, based on gender and age characteristics, the engine synthesizes tones such as weakness, fear, and anxiety, as well as pure tones such as coughing and laughter, and synthesizes timbre for elderly women, middle-aged women, and middle-aged men.
[0060] This invention establishes a logical bridge between unstructured medical text and high-dimensional visual features through the deep reasoning capabilities of a large language model. It automatically completes the missing visual descriptions in the original data based on disease principles, making the generated portrait logically sound (e.g., inferring facial expressions based on disease severity, and inferring gender based on epidemiology), thus solving the problem of the disconnect between virtual avatars and case descriptions. Simultaneously, by embedding structured case information and emotional factors into the underlying logic (system prompts) of the dialogue system, this invention endows virtual patients with a dynamic thought process. Unlike traditional keyword matching, the responses generated by this invention rely on an understanding of the entire medical history context and a pre-set emotional state, simulating the hesitation, anxiety, or unclear expression of real patients, forcing doctors to use diagnostic techniques in the dialogue, thereby solving the problem of rigid interaction logic. Furthermore, the fully automated data processing flow of this invention makes it possible to generate millions of unique, standardized case records using existing massive amounts of internet medical data, greatly reducing technical barriers and costs.
[0061] Example 2
[0062] This embodiment provides a virtual standardized patient avatar generation and dialogue system, such as Figure 2 As shown, it includes:
[0063] The feature extraction and reasoning module is used to: extract multi-dimensional information of patients from raw medical data using a large language model and construct structured case data, and to reason about the visual features of patients and integrate the visual features of patients into portrait description prompts;
[0064] The visual generation module is used to: convert portrait description prompts into high-dimensional semantic vectors using a text encoder, thereby guiding a pre-trained latent diffusion model to generate patient portrait images that conform to medical data;
[0065] The voice interaction module is used to: acquire the user's voice signal and convert it into natural language text as role prompts; construct system prompts based on the patient's structured medical data and preset emotion factors; generate response text for the user's voice signal using a large language model and based on the role prompts and system prompts; and finally convert the response text into response speech and output it through a speaker.
[0066] The specific implementation methods of each module in the system described in this embodiment are the same as those described in Embodiment 1.
[0067] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A method for generating and engaging in dialogue with a virtual standardized patient avatar, characterized in that, include: The large language model is used to extract multi-dimensional information about patients from raw medical data and construct structured case data, as well as to infer the visual characteristics of patients and integrate the visual characteristics of patients into portrait description prompts. A text encoder is used to convert portrait description prompts into high-dimensional semantic vectors, which then guide a pre-trained latent diffusion model to generate patient portrait images that conform to medical data. Acquire users' voice signals and convert them into natural language text, which serves as role prompts; construct system prompts based on patients' structured medical records and preset emotion factors; The system uses a large language model and generates response text based on role prompts, system prompts, and historical dialogues to generate user voice signals. The speech engine is activated to convert the reply text into a reply speech, which is then output through the speaker.
2. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, The extracted multi-dimensional information, including department affiliation, clinical diagnosis results, chief complaint, present illness history, past medical history and allergy history, is mapped to obtain case data with a standard key-value pair structure.
3. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, Inferring the patient's visual characteristics, including biological features, pathological facial features, hair features, and facial expression features.
4. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, A text encoder based on the Transformer architecture is used to convert portrait description prompts into high-dimensional semantic vectors.
5. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, When the latent diffusion model generates a portrait image, it first initializes a random Gaussian noise matrix; then, using the high-dimensional semantic vector obtained by converting prompt words as a guide, it gradually predicts and subtracts noise through a multi-step reverse diffusion process to reconstruct the latent feature map; then, the latent image features are mapped to the pixel space through a decoder to obtain the patient's portrait image.
6. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, To convert speech signals into natural language text, a model based on the Vosk architecture is used. First, acoustic features are extracted and decoded, and continuous audio waveforms are converted into phoneme sequences. Then, the phoneme sequences are decoded into natural language text.
7. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, Using a text-to-speech engine, the system automatically matches the corresponding timbre model based on the biological characteristics in the patient's visual features obtained through reasoning. By analyzing the prosodic features of the text, it synthesizes a digital speech signal with intonation fluctuations.
8. The method for generating and conversing with a virtual standardized patient avatar according to claim 1, characterized in that, The emotional factors include anxiety, irritability, fear, sadness, and / or trust.
9. A virtual standardized patient avatar generation and dialogue system, characterized in that, include: The feature extraction and reasoning module is used to: extract multi-dimensional information of patients from raw medical data using a large language model and construct structured case data, and to reason about the visual features of patients and integrate the visual features of patients into portrait description prompts; The visual generation module is used to: convert portrait description prompts into high-dimensional semantic vectors using a text encoder, thereby guiding a pre-trained latent diffusion model to generate patient portrait images that conform to medical data; The voice interaction module is used to: acquire the user's voice signal and convert it into natural language text as role prompts; and construct system prompts based on the patient's structured medical data and preset emotion factors. Using a large language model and based on role prompts, system prompts, and historical dialogues, the system generates response text for the user's voice signal; finally, the response text is converted into response speech and output through a speaker.