Digital doctor expression, action and emotion interactive simulation system based on GAN

Abstract

The invention provides a GAN-based digital doctor expression, action and emotion interaction simulation system. The system comprises a data set construction module, an audio encoder, an action mask encoder, a Transform converter, a training module and a digital doctor generation module. The data set construction module is used for constructing a high-quality medical scene data set; the audio encoder extracts advanced feature representation from the audio by adopting a pre-training model; the motion mask encoder is used for extracting motion feature information in a binary image with the same size as a video frame; the Transform converter is used for realizing the fusion of the multi-modal data; the training module is used for training the GAN network; and the digital doctor generation module generates a vivid whole body action and expression sequence through a trained generator. According to the system, expressions and actions of the digital doctor can be more accurately generated according to voice dialogues in the current system, and the problems that in a traditional online diagnosis and treatment system, users are not satisfactory, and the expressions of the digital doctor are not generated well are solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a technology for generating digital doctor expressions and movements, and specifically to a system and method for simulating the interaction of digital doctor expressions, movements and emotions based on a generative adversarial network (GAN) and emotion recognition. Background Art

[0002] With the advancement of artificial intelligence (AI) technology, digital doctors are increasingly being used in telemedicine, online consultations, and healthcare. These virtual assistants not only provide health advice, disease diagnosis, and treatment recommendations, but also enhance the user experience through realistic expressions and gestures, improving communication effectiveness and patient satisfaction. Existing technologies still face numerous challenges in achieving this goal. While existing digital doctor diagnostic systems have improved the accessibility and efficiency of healthcare services to a certain extent, they still have some significant limitations: 1) It is difficult to ensure that the generated expressions and gestures are consistent with the conversation content and the emotions it implies. The systems may generate overly exaggerated or unnatural expressions, resulting in a poor user experience. 2) High-quality, well-annotated, and diverse training datasets are extremely scarce, especially in medical settings. This limits the learning ability and generalization performance of the models. 3) High-quality, well-annotated, and diverse training datasets are extremely scarce, especially in medical settings. This limits the learning ability and generalization performance of the models. Summary of the Invention

[0003] In response to the shortcomings of existing digital doctor diagnosis systems, such as unnatural expression and movement generation and difficulty in combining them with emotional states, the present invention provides a GAN-based digital doctor expression, movement and emotion interaction simulation system.

[0004] To achieve the above object, the present invention adopts the following technical solutions: A GAN-based digital doctor expression, action, and emotional interaction simulation system, including a dataset construction module, an audio feature encoder, an action mask encoder, a Transformer converter, a training module, and a digital doctor generation module: The dataset construction module is used to construct a high-quality medical scenario dataset.

[0005] The audio feature encoder uses a pre-trained model to extract high-level feature representations from audio, adds additional fully connected layers on this basis to further process the high-dimensional features generated by the pre-trained model, and uses a small amount of labeled audio data obtained from the dataset construction module for fine-tuning.

[0006] The motion mask encoder is used to extract motion feature information from a binary image with the same size as the video frame.

[0007] The Transformer converter is used to achieve the fusion of multimodal data.

[0008] The training module is used to train the GAN network; the output features of the Transformer converter are used as the input of the GAN network to train the GAN network.

[0009] The digital doctor generation module: generates realistic whole-body movements and expression sequences through the generator trained by the training module, while maintaining a high degree of consistency and naturalness with the diagnosis and treatment dialogue in details.

[0010] In one possible implementation, the dataset construction module uses professional equipment to record the movements and expressions of real doctors in various medical scenarios from multiple angles, and simultaneously records clear audio. A 2D pose estimation tool is then used to extract action key point information from each second of video, including facial and limb key point coordinates, locations, and confidence scores. The audio is then segmented into 1-second intervals and labeled with emotion labels (happy, sad, angry, normal). A bandpass filter is applied to retain the frequency range between 300Hz and 3400Hz, and spectral subtraction is used to remove background noise. Finally, an action mask is created for the video frame, marking known cue frames and blank areas that need to be predicted, generating a binary image of the same size as the video frame to support subsequent model training and action generation.

[0011] In one possible implementation, a multi-channel convolutional neural network (CNN) is designed as an action mask encoder, which uses convolution kernels of different sizes to capture multi-scale spatial information and represent actions at different levels of abstraction. By performing element-wise multiplication of the action mask with the output feature map of each layer, only the features of the known action area are retained, while the features of the area to be predicted are learned during the subsequent GAN network training process. In addition, a spatial attention mechanism is further introduced to enable the action mask encoder to focus on key areas and improve the accuracy of understanding the area to be predicted. Ultimately, the model outputs a probability distribution map of the same size as the input image, where each pixel represents the possibility that the location belongs to a specific action or expression category. The probability distribution map is the action mask feature extracted by the action mask encoder.

[0012] In one possible embodiment, the Transformer converter is composed of a multi-layer Transformer architecture including a cross-modal fusion layer, position encoding and mask processing. First, the audio features and action mask features are mapped to the same dimension. In the Transformer architecture, a multi-head self-attention mechanism is introduced to enable the model to focus on different parts of the input sequence at the same time, effectively capturing the complex relationship between audio and action. Then, the mask matrix is used to clearly distinguish between known action prompt frames and parts that need to be predicted, helping the model understand the order in which the actions occur, and outputting multimodal features after the fusion of audio and action masks.

[0013] In one possible implementation, the GAN network includes a generator and a discriminator; through a designed multi-task learning architecture, the generator is able to simultaneously optimize the consistency of full-body movements and facial expressions as well as the spatiotemporal consistency.

[0014] The generator is based on an RNN network and adds a temporal modeling component to improve the model's expressiveness, predicting expressions and actions based on audio features. The discriminator is used to evaluate the authenticity of generated samples.

[0015] The training module uses supervised loss, adversarial loss, and cycle consistency loss to train the GAN network to ensure that the mapping relationship between different facial expressions and audio modalities is stable and reliable.

[0016] In one possible implementation, the digital doctor generation module first segments the real-time audio data, extracts features using an audio feature encoder, and then feeds it into a trained generator, generating a series of contextually accurate digital doctor action and expression frames. This approach ensures that the generated content is not only authentic but also accurately reflects the emotions and interactive details of the diagnosis and treatment process.

[0017] The present invention can effectively improve the user experience and the practicality of the remote online medical diagnosis system through the above technical means.

[0018] The beneficial effects of the present invention are as follows: This paper proposes a new framework based on a GAN network and combined with audio and motion mask converters, aiming to overcome the limitations of existing technologies and provide users with a more natural and personalized interactive experience. This paper has the following significant advantages: 1) The system incorporates a multimodal fusion mechanism, integrating an audio feature encoder and an action mask encoder to effectively process multiple input signals (including speech and video). This design not only enhances the model's ability to capture cross-modal information but also ensures that the generated expressions and actions are highly consistent with the conversation content and emotional state. 2) A hierarchical decoding structure allows the system to gradually refine the action generation process from overall posture to local details. This approach results in more refined results, making it particularly suitable for scenarios requiring precise control of facial expressions and hand movements. Furthermore, this structure supports flexible adjustment of action priorities for different parts of the body to suit diverse application scenarios. 3) To address data scarcity, automated data processing tools are utilized to significantly improve data preparation efficiency and assist manual correction, ensuring data quality while saving significant time. Furthermore, data augmentation techniques are employed to generate new training examples by transforming existing samples, further enriching the diversity of the dataset.

[0019] This invention effectively addresses existing challenges through technological innovation, not only improving the performance of the digital doctor's interactive simulation system for facial expressions, movements, and emotions, but also providing a new development direction for future intelligent medical assistants. This advancement will significantly improve the quality of doctor-patient communication and promote the intelligent and humane development of medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0021] Figure 1 This is a schematic diagram of the operation flow of the GAN-based digital doctor expression, action and emotion interaction simulation system in an embodiment of the present invention.

[0022] Figure 2 Schematic diagram of an audio feature encoder according to an embodiment of the present invention.

[0023] Figure 3 Schematic diagram of an action mask encoder according to an embodiment of the present invention.

[0024] Figure 4 Schematic diagram of multimodal data fusion according to an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of GAN network training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0027] The system of the present invention uses a GAN network to integrate motion coding and audio speech emotion recognition technology, aiming to improve the personalized and natural interaction between patients and digital doctors during remote medical diagnosis.

[0028] A method for simulating the interactive expressions, gestures, and emotions of digital doctors based on generative adversarial networks (GANs) is a key technology in the fields of telemedicine question-answering and online medical diagnosis. The goal of this invention is to enhance user experience and satisfaction in online medical systems. By integrating GANs with audio and speech emotion recognition, this method can more accurately generate digital doctors' expressions and gestures based on existing voice conversations, thus addressing issues such as user dissatisfaction and poorly generated digital doctors' expressions in traditional online diagnosis and treatment systems.

[0029] like Figure 1 FIG. 1 is a schematic diagram of the operation flow of a GAN-based digital doctor expression, action, and emotion interaction simulation system according to an embodiment of the present invention, including: Step (1): Build a high-quality medical scenario dataset through the dataset construction module.

[0030] Step (2): The audio feature encoder uses a pre-trained model to extract high-level feature representations from the audio. On this basis, additional fully connected layers are added to further process the high-dimensional features generated by the pre-trained model, and fine-tuned using a small amount of labeled audio data obtained from the dataset building module.

[0031] Step (3): Extract the action feature information in the binary image with the same size as the video frame through the action mask encoder.

[0032] Step (4): Use Transformer converter to achieve multimodal data fusion.

[0033] Step (5): Train the GAN network through the training module: The output features of the Transformer converter are used as the input of the GAN network to train the GAN network.

[0034] Step (6): The digital doctor generation module generates realistic full-body movements and expression sequences through the generator trained by the training module, while maintaining high consistency and naturalness with the diagnosis and treatment dialogue in details.

[0035] In one possible implementation, the specific operations of the diagnosis and treatment process data collection of the data set construction module are as follows: To build a high-quality medical scenario dataset, we need to systematically record, process, and annotate the movements and expressions of real doctors in various medical scenarios, and simultaneously process the audio data. The following are detailed implementation steps and formulas.

[0036] Collection of facial expression and action data sets: Professional camera equipment is used to record the doctor's movements and expressions during diagnosis and treatment from multiple angles to ensure that the movements and expressions are clearly visible. The OpenPose tool is used to extract the key point information of the action in the video frame, and each second frame is processed to extract the key point set. , where each key point Contains coordinates and confidence scores Professionals annotate known action frames and blank areas requiring prediction based on actual conditions to create an action mask. The action mask is a binary image the same size as each video frame, where a 1 indicates that the pixel location belongs to a known action frame, and a 0 indicates an unknown area requiring model prediction.

[0037] Audio Dataset Collection: Professional recording equipment was used to synchronously record audio from the diagnosis and treatment process, ensuring clear sound quality and synchronization with the video. The audio was segmented into one-second intervals. Each segment was labeled with the emotion {happy, sad, angry, normal}. A bandpass filter was applied to remove low- and high-frequency noise, retaining the frequency range between 300Hz and 3400Hz. Further analysis and spectrum correction were performed to remove background noise and reduce the impact of non-speech components. in Represents the result after bandpass filtering. is the estimated noise spectrum. is the adjustment factor.

[0038] In a possible implementation, the audio feature encoder is specifically implemented as follows: like Figure 2 As shown in Figure 1, the purpose of the audio feature encoder is to extract high-level feature representations from audio. It uses the pre-trained model Wav2Vec and performs supervised fine-tuning by adding additional fully connected network layers. The following are the detailed implementation steps and formulas: Input the audio into the Wav2Vec 2.0 model to extract high-dimensional feature representations. Let the input audio be A, and the feature representation of the Wav2Vec 2.0 model output be F = Wav2Vec(A). Based on the Wav2Vec 2.0 model, add an additional fully connected network layer to transform the high-dimensional features. Let the weight matrix of the fully connected layer be W, and the bias vector be b. The transformed feature representation is: Where W is the weight vector, b is the bias parameter, is the ReLu activation function. A small amount of labeled audio data is used for supervised fine-tuning. The labeled audio data set is ,in Represents an audio clip, is its corresponding sentiment label. The goal of fine-tuning is to minimize the loss function: in represents the model parameters, is the probability distribution predicted by the model. The feature representation of the audio data output by the fine-tuned model is ,These features not only retain the representation ability of the pre-trained model, but also accurately meet the specific requirements of downstream tasks.

[0039] In a possible implementation, the action mask encoder is specifically implemented as follows: like Figure 3 As shown in Figure 1, in order to understand the known and predicted action regions in the video frame, the action mask encoder uses a multi-channel convolutional neural network (CNN) and combines it with a spatial attention mechanism to improve the accuracy of action understanding. The following are the detailed implementation steps and formulas: Assume that the size of each frame after video segmentation is W×H, and the action mask M is the same as the size of the video frame. Use convolution kernels of different sizes to process the input frame to capture spatial information at different scales. Let the input frame be I, after the first The feature map after the layer convolution operation is : in, It is The convolution kernel of the layer.

[0040] The action mask M is directly combined with the output feature map of each layer Perform element-wise multiplication to retain features corresponding only to known action regions. Calculate the masked feature map : in Indicates the corresponding multiplication of elements. The attention mechanism is further introduced to convert the extracted key point information into a heat map (Heatmap), and it is used as part of the attention weight to generate the key point heat map H. The elements of H have the same dimension as the feature map same: in Control the range of the heat map, and is the coordinate of the key point. Calculate the spatial attention weight, combining the key point heat map H and the feature map , get the fused features : The Attention implementation mechanism is as follows: Is an adjustable parameter to balance the contribution of feature maps and heat maps: Apply attention weights to the fused features Get action feature data : After three layers of convolution and attention mechanism processing, the final output is a probability distribution map with the same size as the input image. The probability distribution map is the action mask feature extracted by the action mask encoder, where each pixel corresponds to the probability that the position belongs to a specific action and expression category. The specific formula is as follows: in, and They are the fully connected layer weight matrix and bias vector when the attention mechanism is fused.

[0041] In a possible implementation, the multimodal data fusion of the Transformer converter is specifically performed as follows: The Transformer converter consists of a multi-layer Transformer architecture, such as Figure 4 As shown, the action and audio feature data are fused, and the processed audio feature data is , the action mask feature data after fusion with the mask attention mechanism is First, the audio features and action mask features Map to the same dimension: in 、 are the weight matrices of audio features and action mask features respectively, 、 are the bias terms for audio features and action mask features, respectively. In the Transformer architecture, the use of a multi-head self-attention mechanism allows the model to focus on different parts of the input sequence simultaneously, thereby capturing the relationship between different modalities.

[0042] Assume that the input of the Transformer converter is , which is the concatenated audio and action mask features. In order to better learn continuous action features, position encoding is added to represent action sequences at different time sequences to retain the position attribute of sequence information. Let the position encoding be PE, then the final input of the Transformer converter is =X+PE. In order to distinguish the known action prompt frame from the part that needs to be predicted, a mask matrix Mask is constructed. For the part that needs to be predicted, the corresponding position in the mask matrix is set to negative infinity so that these values are close to 0 during the Softmax process and do not affect the output result. After the Transformer converter is processed by the multi-layer Transformer architecture, the final probability distribution map is output. , It is the multimodal feature after the fusion of audio and action mask, which indicates the possibility of belonging to a specific action or expression category at each moment: in Refers to the last layer output of the Transformer converter, and They are the weight matrix and bias vector of the fully connected layer when predicting the probability distribution graph.

[0043] The training module trains the GAN network based on the facial expression data set and the audio data set obtained by the data set construction module; In one possible implementation, the GAN network includes a generator and a discriminator: like Figure 5 As shown in the figure, in order to design a multi-task learning architecture that enables the generator to simultaneously optimize the consistency of full-body movements, facial expression targets, and spatiotemporal consistency, and predict expressions and movements based on audio features, a RNN network, a temporal modeling component, a discriminator, and a cycle consistency loss are combined. The following are the detailed implementation steps and formulas: Generator base: Use RNN (such as LSTM or GRU) as the generator base, input audio features to predict facial expressions and body movements G( ),in Multimodal features obtained by fusion of Transformer converter: in, is the hidden state sequence of the RNN. Furthermore, a temporal convolutional network (TCN) is added to the generator base to capture dependencies in the temporal dimension and improve the expressiveness of the model: Finally, the key point sequence is generated: Discriminator construction: Build a discriminator to evaluate the authenticity of the generated samples. The input of the discriminator is the real sample Or generate samples , the output is the authenticity score .

[0044] The training module trains the GAN network based on the multimodal features fused by the Transformer converter, and the loss function used is as follows: Set a supervised loss term: ensure that the generated key point sequence is consistent with the true key point sequence: Setting the adversarial loss term: The generator tries to “fool” the discriminator so that it cannot distinguish between real samples and generated samples: Set cycle consistency loss: ensure that the mapping relationship between different expressions and audio modalities is stable and reliable, based on the generated key point sequence Generate audio features again , the cycle consistency loss is: The total loss function combines supervised loss, adversarial loss, and cycle consistency loss: in 、 、 is a hyperparameter used to balance the contributions of each loss component. Through the above steps and formulation, a multi-task learning architecture can be designed that enables the generator to simultaneously optimize full-body motion, facial expression consistency, and spatiotemporal consistency, while also predicting expressions and motions based on audio features. This approach not only enhances the model's robustness and generalization capabilities, but also ensures the authenticity and consistency of the generated results.

[0045] In a possible implementation, the digital doctor generation module specifically operates as follows: Through the training of the training module, a trained GAN network generator is obtained, which can not only generate realistic whole-body movements and expression sequences, but also maintain a high degree of consistency and naturalness with the diagnosis and treatment dialogue in details. Specifically, the real-time audio data is also segmented and processed at one-second time intervals, and then the audio feature encoder is used to extract the features of the audio data. After being fed into the trained generator, a series of frames of digital doctor movements and expressions that match the context are generated. This approach not only ensures the authenticity of the generated content, but also accurately reflects the emotions and interactive details of the diagnosis and treatment process, improving the overall user experience and the practicality of the system. In addition, by combining a multi-task learning architecture, adversarial training, and cycle consistency loss, the generator is able to establish a stable mapping relationship between different modalities, further enhancing the quality and consistency of the generated results. Ultimately, this comprehensive approach provides virtual doctors in medical scenarios with more realistic and natural performance, significantly improving the reliability of the system and user satisfaction.

[0046] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0047] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A GAN-based digital doctor expression, action and emotion interaction simulation system, characterized by: Including dataset construction module, audio feature encoder, action mask encoder, Transformer converter, training module and digital doctor generation module: The dataset construction module is used to construct a high-quality medical scenario dataset; The audio feature encoder uses a pre-trained model to extract high-level feature representations from the audio, adds additional fully connected layers to further process the high-dimensional features generated by the pre-trained model, and is fine-tuned using a small amount of labeled audio data obtained from the dataset construction module; The action mask encoder is used to extract action feature information from a binary image with the same size as the video frame; The Transformer converter is used to achieve the fusion of multimodal data; The training module is used to train the GAN network; the output features of the Transformer converter are used as the input of the GAN network to train the GAN network; The digital doctor generation module: generates realistic whole-body movements and expression sequences through the generator trained by the training module, while maintaining a high degree of consistency and naturalness with the diagnosis and treatment dialogue in details.

2. The GAN-based digital doctor expression, action and emotion interactive simulation system according to claim 1 is characterized in that: The dataset construction module uses professional equipment to record the movements and expressions of real doctors in various medical scenarios from multiple angles, and simultaneously records clear audio; then uses a 2D pose estimation tool to extract action key point information in each second of video, including facial and limb key point coordinates, location and confidence score; then the audio is segmented into 1-second intervals and labeled with emotion labels, a bandpass filter is applied to retain the frequency range between 300Hz and 3400Hz, and spectral subtraction is used to remove background noise; finally, an action mask is created for the video frame, marking the known prompt frames and the blank areas that need to be predicted, and generating a binary image with the same size as the video frame.

3. The GAN-based digital doctor expression, action and emotion interactive simulation system according to claim 1 is characterized in that: A multi-channel convolutional neural network is designed as an action mask encoder, which uses convolution kernels of different sizes to capture multi-scale spatial information and represent actions at different levels of abstraction. By performing element-wise multiplication of the action mask with the output feature map of each layer, only the features of the known action area are retained. In addition, a spatial attention mechanism is further introduced to enable the action mask encoder to focus on key areas and improve the accuracy of understanding the areas to be predicted. Finally, the model outputs a probability distribution map of the same size as the input image, where each pixel represents the probability that the location belongs to a specific action or expression category. The probability distribution map is the action mask feature extracted by the action mask encoder.

4. The GAN-based digital doctor expression, action and emotion interactive simulation system according to claim 1 is characterized in that: The Transformer converter is composed of a multi-layer Transformer architecture including cross-modal fusion layers, position encoding and mask processing. First, the audio features and action mask features are mapped to the same dimension. In the Transformer architecture, a multi-head self-attention mechanism is introduced to enable the model to simultaneously focus on different parts of the input sequence, effectively capturing the complex relationship between audio and action. Then, a mask matrix is used to clearly distinguish between known action prompt frames and parts that need to be predicted, helping the model understand the order in which actions occur, and outputting multimodal features after the audio and action mask are fused.

5. The GAN-based digital doctor expression, action and emotion interactive simulation system according to claim 1 is characterized in that: The GAN network consists of a generator and a discriminator. Through the designed multi-task learning architecture, the generator can simultaneously optimize the consistency of full-body movements and facial expressions as well as the spatiotemporal consistency. The generator is based on an RNN network and adds a temporal modeling component to improve the model's expressiveness, predicting expressions and actions based on audio features; the discriminator is used to evaluate the authenticity of the generated samples; The training module uses supervised loss, adversarial loss, and cycle consistency loss to train the GAN network to ensure that the mapping relationship between different facial expressions and audio modalities is stable and reliable.

6. The GAN-based digital doctor expression, action and emotion interactive simulation system according to claim 1 is characterized in that: The digital doctor generation module described above: first, the audio data collected in real time is segmented and processed, and features are extracted through an audio feature encoder, and then input into a trained generator to generate a series of digital doctor action and expression frames that match the situation; this method ensures that the generated content is not only authentic and credible, but also can accurately reflect the emotions and interactive details during the diagnosis and treatment process.

Images