A Real-Time Interactive Response Method for Digital Humans Based on Multimodal Emotion Perception

By synchronously collecting and fusing multimodal signals, and combining a large language model and feedback adjustment mechanism, the problem of multimodal perception and response in digital human emotional interaction is solved, achieving a high-precision, natural and personalized interactive experience.

CN123114263APending Publication Date: 2026-11-17HUNAN INST OF INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611412635.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, digital humans suffer from limited multimodal emotion perception capabilities, rigid emotional responses, poor accuracy of synchronous expression, and a lack of closed-loop feedback and adaptive adjustment capabilities in emotional interaction, resulting in a mechanical and unnatural interactive experience.

Method used

By simultaneously acquiring speech, facial video, and text signals, a multimodal emotion fusion encoder based on a cross-attention mechanism is used for feature alignment and fusion. Natural responses are generated by combining a large language model decision module, and adaptive adjustment is performed through emotion feedback closed-loop verification.

Benefits of technology

It improves the accuracy and robustness of emotion recognition, enables multimodal synchronous expression and natural emotional response, enhances the realism and immersion of digital humans, and has personalized adaptation capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114263A_ABST
    Figure CN123114263A_ABST
Patent Text Reader

Abstract

This disclosure provides a real-time interactive response method for digital humans based on multimodal emotion perception, belonging to the field of data processing technology. Specifically, it includes: synchronously acquiring the user's voice audio, facial video, and text input signals and aligning them with timestamps; extracting acoustic features, facial motion features, and semantic features respectively, aligning and fusing the three-modal features, and outputting a user emotional state vector; concatenating the emotional state vector with the dialogue context and inputting it into a large language model decision module to generate dialogue response text, emotional expression parameters, and action expression parameters; synchronously driving the digital human's speech synthesis, facial animation, and body movement generation based on the emotional expression parameters, enabling coordinated expression of voice, facial expressions, and actions; and acquiring the user's emotional feedback again after the digital human outputs a response, determining whether to trigger an emotion adjustment and correction process based on its consistency with the expected emotional goal. Through this disclosed solution, high-precision and highly natural digital human emotional interaction is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a real-time interactive response method for digital humans based on multimodal emotion perception. Background Technology

[0002] Currently, with the development of human-computer interaction technology, digital humans with emotional interaction capabilities are increasingly widely used in fields such as intelligent customer service, online education, and virtual assistants. However, existing methods for emotional interaction in digital humans typically suffer from the following shortcomings: First, multimodal emotion perception capabilities are limited, mostly relying on a single modality (such as voice or facial expression). Even when attempting to fuse multimodal features, simple feature splicing or weighted summation is often used, lacking in-depth modeling of temporal alignment and cross-modal correlation, resulting in low accuracy and robustness of emotion recognition.

[0003] Secondly, the emotional response strategies are rigid, often relying on preset rules or finite state transition diagrams to drive the digital human. This makes it difficult to dynamically generate natural and diverse responses and emotional expressions based on the context of the conversation, resulting in a mechanical and rigid interactive experience.

[0004] Third, the synchronization accuracy of voice, facial expressions and actions is poor. Each module operates independently and lacks a unified timeline scheduling, which easily leads to problems such as lip movements being out of sync with voice, delayed facial expressions, and incoordination between actions and emotional context.

[0005] Fourth, the system lacks interactive closed-loop feedback and adaptive adjustment capabilities. It cannot dynamically optimize response strategies based on subsequent emotional feedback from users, making it difficult to meet personalized and scenario-adaptive emotional interaction needs.

[0006] It is evident that there is an urgent need to propose a real-time interactive response for digital humans based on multimodal emotion perception that can achieve accurate multimodal emotion perception, intelligent emotion response, multimodal synchronous expression, and closed-loop feedback regulation. Summary of the Invention

[0007] In view of this, the present disclosure provides a real-time interactive response method for digital humans based on multimodal emotion perception, which at least partially solves some of the problems existing in the prior art.

[0008] This disclosure provides a real-time interactive response method for digital humans based on multimodal emotion perception, including: Step 1, Multimodal Emotion Signal Acquisition: Simultaneously acquire voice audio signals, facial video signals, and text input signals during user interaction, and align the timestamps of each modality signal to obtain a multimodal synchronous data stream; Step 2, Multimodal Emotion Feature Encoding: Extract acoustic features from speech audio signals, facial motion features from facial video signals, and semantic features from text input signals. Then, input the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism. Align and fuse the three modal features in the time dimension to output the user's emotional state vector. Step 3, Digital Human Emotional Response Decision: The user's emotional state vector is concatenated with the dialogue context information and input into the large language model decision module, which then generates the digital human's dialogue response text, emotional expression parameters, and action expression parameters. Step 4, Digital Human Multimodal Synchronous Expression: Based on the emotional expression parameters, the digital human's speech synthesis, facial animation and body movement generation are synchronously driven to make the digital human's speech, facial expressions and movements aligned in time and coordinated in expression; Step 5, Emotional Feedback Closed-Loop Verification: After the digital human outputs a response, Step 1 and Step 2 are executed again to collect the user's emotional feedback on the new round of interaction, and the consistency between the emotional feedback and the expected emotional goal is used to determine whether the emotional adjustment and correction process is triggered.

[0009] According to a specific implementation of this disclosure, the step of inputting the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism, aligning and fusing the three modal features in the time dimension, and outputting a user emotion state vector includes: Using a sequence of acoustic features as the query and a sequence of facial motion features as the key and value, the speech-face cross-modal attention weights are calculated through the first cross-attention layer to obtain a speech-guided facial emotion representation. Using voice-guided facial emotion representations as queries and sequences of semantic features as keys and values, cross-modal fusion features are computed through a second cross-attention layer. Cross-modal fusion features are input into a temporal Transformer encoder for temporal modeling, and the output, after passing through a fully connected layer, is a user sentiment state vector containing sentiment category labels and their confidence scores.

[0010] According to a specific implementation of this disclosure, the step of extracting facial motion features from a facial video signal includes: A three-dimensional convolutional neural network is used to extract spatiotemporal feature maps from a sequence of consecutive frames of facial video. Local features of the eye region, mouth region, and eyebrow region are extracted from the spatiotemporal feature map using the region of interest alignment method. By fusing local features with facial action unit intensity values ​​obtained from facial keypoint detection along the channel dimension, multi-scale facial emotion features are obtained.

[0011] According to a specific implementation of this disclosure, before step 3, the method further includes: The user's sentiment state vector is encoded into natural language description text, wherein the natural language description text includes sentiment category, confidence level, and suggested response sentiment style; The natural language description text is used as a prefix for the system prompts, and is concatenated with the user's dialogue history and current user input to form a complete large language model input prompt; The output of the constraint large language model decision module is structured data, which includes dialogue response text, speech tone parameters, facial expression coefficients, and body movement style labels.

[0012] According to a specific implementation of an embodiment of this disclosure, step 4 specifically includes the following process: A unified timeline scheduler is built in the digital human rendering engine, using the phoneme timestamps output by the speech synthesis module as the synchronization master clock reference. The facial animation module pre-generates lip-sync and facial expression animation curves based on phoneme timestamps to ensure that lip movements are synchronized with pronunciation. The motion generation module selects motion segments from the pre-trained motion library based on the motion style labels and emotional intensity values ​​in the motion performance parameters. After transition processing, the motion segments are aligned with the speech under the control of the timeline scheduler.

[0013] According to a specific implementation of this disclosure, the step of determining whether to trigger the emotion adjustment and correction process based on the consistency between the emotional feedback and the expected emotional goal includes: Calculate the similarity between the currently collected user emotion feedback vector and the expected emotion target vector, and trigger correction when the similarity is lower than a preset threshold; Calculate the difference vector between the user's emotional feedback vector and the expected emotional target vector; The difference vector is injected as a correction signal into the soft cue of the decision module of the large language model to adjust the emotional response intensity coefficient of the large language model. The correction process is recorded and stored in the emotion regulation log to form an emotion response preference database for specific users or scenarios.

[0014] The real-time interactive response scheme for digital humans based on multimodal emotion perception in this embodiment includes: Step 1, multimodal emotion signal acquisition: synchronously acquiring voice audio signals, facial video signals, and text input signals during user interaction, and aligning the timestamps of each modality signal to obtain a multimodal synchronous data stream; Step 2, multimodal emotion feature encoding: extracting acoustic features from the voice audio signals, facial motion features from the facial video signals, and semantic features from the text input signals, and then inputting the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism, aligning and fusing the three modal features in the time dimension, and outputting a user emotion state vector; Step 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ... Step 3, Digital Human Emotional Response Decision: The user's emotional state vector is concatenated with the dialogue context information and input into the large language model decision module, which generates the digital human's dialogue response text, emotional expression parameters, and action expression parameters. Step 4, Digital Human Multimodal Synchronous Expression: Based on the emotional expression parameters, the digital human's speech synthesis, facial animation, and body movement generation are synchronously driven, so that the digital human's speech, facial expressions, and actions are aligned in time and coordinated in expression. Step 5, Emotional Feedback Closed-Loop Verification: After the digital human outputs a response, Steps 1 and 2 are executed again to collect the user's emotional feedback on the new round of interaction, and the consistency between the emotional feedback and the expected emotional goal is used to determine whether to trigger the emotional adjustment and correction process.

[0015] The beneficial effects of the embodiments disclosed herein are as follows: 1. High accuracy and robustness of multimodal emotion perception: By simultaneously acquiring speech, facial video and text signals, and using a multimodal emotion fusion encoder based on cross-attention mechanism, the three-modal features are deeply aligned and fused in the time dimension, which significantly improves the accuracy of emotion recognition and the robustness to noise interference. 2. Natural and intelligent emotional response: The fused emotional state vector and the dialogue context are input into the decision module of the large language model, which can generate dialogue responses, voice tone, facial expressions and body movement parameters according to the user's emotional dynamics, avoiding templated responses and enabling digital humans to have more natural and empathetic interaction capabilities. 3. Precise synchronization of multimodal expression: Based on the phoneme timestamps of speech synthesis, a unified timeline scheduler coordinates the generation of facial animation and body movements, ensuring the precise alignment and coordinated expression of speech, facial expressions, and movements, significantly enhancing the realism and immersion of digital humans; 4. Possesses closed-loop adaptive capability: After the digital human outputs a response, it collects the user's emotional feedback again, calculates its similarity to the expected emotional target, and automatically triggers an emotional adjustment and correction process when the deviation is too large, forming a continuous optimization mechanism. At the same time, the correction records are stored in a preference database, which can be used for personalized adaptation for different users or scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a real-time interactive response method for digital humans based on multimodal emotion perception, provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram illustrating the specific implementation process of a real-time interactive response method for digital humans based on multimodal emotion perception, provided in an embodiment of this disclosure. Detailed Implementation

[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0023] This disclosure provides a real-time interactive response method for digital humans based on multimodal emotion perception. The method can be applied to human-computer interaction processes in scenarios such as intelligent customer service, online education, and virtual assistants.

[0024] See Figure 1 This is a flowchart illustrating a real-time interactive response method for digital humans based on multimodal emotion perception, provided in an embodiment of this disclosure. Figure 1 As shown, the method mainly includes the following steps: Step 1, Multimodal Emotion Signal Acquisition: Simultaneously acquire voice audio signals, facial video signals, and text input signals during user interaction, and align the timestamps of each modality signal to obtain a multimodal synchronous data stream; In practice, voice audio signals, facial video signals, and text input signals during user interaction can be collected simultaneously, and the timestamps of each modal signal can be aligned to obtain a multimodal synchronous data stream.

[0025] Step 2, Multimodal Emotion Feature Encoding: Extract acoustic features from speech audio signals, facial motion features from facial video signals, and semantic features from text input signals. Then, input the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism. Align and fuse the three modal features in the time dimension to output the user's emotional state vector. In specific implementation, Mel spectrum features and prosodic features are extracted from speech audio signals as acoustic features, facial action unit (AU) intensity sequences and micro-expression spatiotemporal features are extracted from facial video signals as facial motion features, and semantic embedding and sentiment polarity vectors are extracted from text input signals as semantic features. The features of the three modalities are input into a multimodal sentiment fusion encoder. The encoder aligns the three modal features in the time dimension based on a cross-attention mechanism and outputs a user sentiment state vector, which includes sentiment category labels and their confidence scores.

[0026] Furthermore, the multimodal emotion fusion encoder employs a temporal cross-attention fusion architecture, specifically including: Using speech feature sequences as queries and facial feature sequences as keys and values, a speech-face cross-modal attention weight is calculated through a cross-attention layer to obtain a speech-guided facial emotion representation. Using voice-guided facial emotion representations as queries and textual semantic features as keys and values, cross-modal fusion features are computed through a second cross-attention layer; The cross-modal fusion features are input into a temporal Transformer encoder, and after multi-layer self-attention processing, they are output through a fully connected layer to show the probability distribution of seven basic sentiment categories and their corresponding confidence scores.

[0027] Further methods for extracting spatiotemporal features of facial micro-expressions include: A lightweight 3D convolutional neural network was used to extract spatiotemporal feature maps from a sequence of 16 consecutive frames of facial video. Local features of the eye, mouth, and eyebrow regions are extracted using the RoIAlign (Region of Interest Alignment) method on the spatiotemporal feature map; Local features are spliced ​​and fused with facial action unit (AU) intensity values ​​extracted based on MediaPipe along the channel dimension to obtain multi-scale facial emotion features.

[0028] Step 3, Digital Human Emotional Response Decision: The user's emotional state vector is concatenated with the dialogue context information and input into the large language model decision module, which then generates the digital human's dialogue response text, emotional expression parameters, and action expression parameters. In practice, the emotional state vector can be concatenated with the user's dialogue text and then input into the Large Language Model (LLM) decision module. The LLM decision module outputs the digital human's dialogue response text, emotional expression parameters, and action expression parameters. Among them, the emotional expression parameters include voice tone parameters, facial expression blendshape coefficients, and body movement style labels.

[0029] Furthermore, the input format for the decision module of the large language model is Structured Sentiment Context Cue (ECP), which is constructed as follows: The sentiment state vector is encoded into natural language descriptive text in the format: "The user's current sentiment state is [sentiment category], confidence level [score], and it is suggested that the digital human respond in the style of [sentiment style]". The emotional state description text is used as a prefix for system prompts and concatenated with the user's dialogue history and current user input to construct a complete LLM input prompt; The LLM output format is constrained to be structured JSON, containing four fields: "Dialogue Response", "Voice Tone", "Face Parameters", and "Action Style".

[0030] Step 4, Digital Human Multimodal Synchronous Expression: Based on the emotional expression parameters, the digital human's speech synthesis, facial animation and body movement generation are synchronously driven to make the digital human's speech, facial expressions and movements aligned in time and coordinated in expression; In practice, the speech synthesis module (TTS) of the digital human can be driven to adjust the speech rate and tone according to the emotional expression parameters, the facial animation module can generate corresponding facial expression animations, and the action generation module can generate corresponding body movements, so that the digital human's speech, facial expressions and actions are precisely aligned in time and expressed in a coordinated manner.

[0031] Furthermore, the method for synchronizing and aligning the voice, facial expressions, and movements of a digital human is as follows: A unified timeline scheduler is built in the digital human rendering engine, using the timestamp of each phoneme output by the speech synthesis module as the master clock reference. The facial animation module pre-generates lip-sync and facial expression animation curves based on phoneme timestamps to ensure precise synchronization between lip movements and pronunciation. The motion generation module selects matching motion segments from the pre-trained motion library based on the motion style labels and emotion intensity values ​​output by the LLM, and aligns them with the speech on the timeline after linear interpolation and fade-in / fade-out transitions.

[0032] Step 5, Emotional Feedback Closed-Loop Verification: After the digital human outputs a response, Step 1 and Step 2 are executed again to collect the user's emotional feedback on the new round of interaction, and the consistency between the emotional feedback and the expected emotional goal is used to determine whether the emotional adjustment and correction process is triggered.

[0033] In practice, after the digital human outputs a response, steps 1 and 2 are executed again to collect the user's emotional feedback on the new round of interaction. The cosine similarity between the emotional feedback vector and the expected emotional target in step 3 is calculated. If the similarity is lower than a preset threshold, the emotional adjustment and correction process is triggered to adjust the LLM decision strategy parameters in step 3.

[0034] Furthermore, the emotional regulation and correction process includes: When the feedback cosine similarity is lower than a preset threshold, calculate the difference vector between the current emotion feedback vector and the expected emotion target vector; The difference vector is used as a correction signal and input into the soft prompt of the LLM decision module to adjust the emotional response intensity coefficient of the LLM. The correction process is recorded in the emotion regulation log to form a database of emotion response preferences for specific user scenarios, which can be used for initial parameter optimization in subsequent scenarios with the same user.

[0035] This embodiment provides a real-time interactive response method for digital humans based on multimodal emotion perception. By simultaneously acquiring speech, facial video, and text signals, and employing a multimodal emotion fusion encoder based on a cross-attention mechanism, it deeply aligns and fuses trimodal features in the temporal dimension, significantly improving the accuracy of emotion recognition and robustness to noise interference. The fused emotion state vector is input into the large language model decision module along with the dialogue context, enabling the generation of dialogue responses, voice tone, facial expressions, and body movement parameters based on user emotions. This avoids templated responses and gives the digital human a more natural and empathetic interactive ability. Using the phoneme timestamps of speech synthesis as a benchmark, a unified timeline scheduler coordinates the generation of facial animation and body movements, ensuring precise alignment and coordinated expression of speech, expressions, and actions, significantly enhancing the realism and immersion of the digital human. After the digital human outputs a response, user emotional feedback is collected again, and its similarity to the expected emotional target is calculated. When the deviation is too large, an emotion adjustment and correction process is automatically triggered, forming a continuous optimization mechanism. Simultaneously, correction records are stored in a preference database for personalized adaptation to different users or scenarios.

[0036] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0037] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A real-time interactive response method for digital humans based on multimodal emotion perception, characterized in that, include: Step 1, Multimodal Emotion Signal Acquisition: Simultaneously acquire voice audio signals, facial video signals, and text input signals during user interaction, and align the timestamps of each modality signal to obtain a multimodal synchronous data stream; Step 2, Multimodal Emotion Feature Encoding: Extract acoustic features from speech audio signals, facial motion features from facial video signals, and semantic features from text input signals. Then, input the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism. Align and fuse the three modal features in the time dimension to output the user's emotional state vector. Step 3, Digital Human Emotional Response Decision: The user's emotional state vector is concatenated with the dialogue context information and input into the large language model decision module, which then generates the digital human's dialogue response text, emotional expression parameters, and action expression parameters. Step 4, Digital Human Multimodal Synchronous Expression: Based on the emotional expression parameters, the digital human's speech synthesis, facial animation and body movement generation are synchronously driven to make the digital human's speech, facial expressions and movements aligned in time and coordinated in expression; Step 5, Emotional Feedback Closed-Loop Verification: After the digital human outputs a response, Step 1 and Step 2 are executed again to collect the user's emotional feedback on the new round of interaction, and the consistency between the emotional feedback and the expected emotional goal is used to determine whether the emotional adjustment and correction process is triggered.

2. The method according to claim 1, characterized in that, The step of inputting the extracted acoustic features, facial motion features, and semantic features into a multimodal emotion fusion encoder based on a cross-attention mechanism, aligning and fusing the three modal features in the temporal dimension, and outputting a user emotion state vector includes: Using a sequence of acoustic features as the query and a sequence of facial motion features as the key and value, the speech-face cross-modal attention weights are calculated through the first cross-attention layer to obtain a speech-guided facial emotion representation. Using voice-guided facial emotion representations as queries and sequences of semantic features as keys and values, cross-modal fusion features are computed through a second cross-attention layer. Cross-modal fusion features are input into a temporal Transformer encoder for temporal modeling, and the output, after passing through a fully connected layer, is a user sentiment state vector containing sentiment category labels and their confidence scores.

3. The method according to claim 1, characterized in that, The step of extracting facial motion features from facial video signals includes: A three-dimensional convolutional neural network is used to extract spatiotemporal feature maps from a sequence of multiple consecutive frames of facial video. Local features of the eye region, mouth region, and eyebrow region are extracted from the spatiotemporal feature map using the region of interest alignment method. By fusing local features with facial action unit intensity values ​​obtained from facial keypoint detection along the channel dimension, multi-scale facial emotion features are obtained.

4. The method according to claim 1, characterized in that, Before step 3, the method further includes: The user's sentiment state vector is encoded into natural language description text, wherein the natural language description text includes sentiment category, confidence level, and suggested response sentiment style; The natural language description text is used as a prefix for the system prompts, and then concatenated with the user's dialogue history and current user input to form a complete large language model input prompt; The output of the large language model decision module is structured data, which includes dialogue response text, speech tone parameters, facial expression coefficients, and body movement style labels.

5. The method according to claim 1, characterized in that, The specific process of step 4 includes: A unified timeline scheduler is built in the digital human rendering engine, using the phoneme timestamps output by the speech synthesis module as the synchronization master clock reference. The facial animation module pre-generates lip-sync and facial expression animation curves based on phoneme timestamps to ensure that lip movements are synchronized with pronunciation. The motion generation module selects motion segments from the pre-trained motion library based on the motion style labels and emotional intensity values ​​in the motion performance parameters. After transition processing, the motion segments are aligned with the speech under the control of the timeline scheduler.

6. The method according to claim 1, characterized in that, The step of determining whether the emotional adjustment and correction process is triggered based on the consistency between the emotional feedback and the expected emotional goal includes: Calculate the similarity between the currently collected user emotion feedback vector and the expected emotion target vector, and trigger correction when the similarity is lower than a preset threshold; Calculate the difference vector between the user's emotional feedback vector and the expected emotional target vector; The difference vector is injected as a correction signal into the soft cue of the decision module of the large language model to adjust the emotional response intensity coefficient of the large language model. The correction process is recorded and stored in the emotion regulation log to form an emotion response preference database for specific users or scenarios.