A method and system for driving digital human actions using a large language model

Abstract

The present invention discloses a method and system for driving digital human actions using a large language model, the method comprising calling a large model to identify the current and previous conversation content, predicting the character's emotions and actions, and outputting coherent emotion and action labels; according to the mapping relationship between different emotions and actions, combining deep learning with a rule base construction method, establishing a digital human rule base covering various behavior patterns; according to the identified emotion and action labels of the current conversation, calling the bound expression and action rendering logic in the digital human rule base to generate a real-time conversation video stream; playing the generated digital human conversation video stream frame by frame in real time. The advantages are: it allows the digital human to combine the conversation content more naturally, generate a video stream with expressions and actions, and achieve the effect of playing the digital human conversation video stream in real time through loop execution, reducing the possibility of stiff expressions and actions or mismatch with the conversation content in traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and virtual human interaction technology, and in particular to a method and system for driving digital human actions using a large language model. Background Art

[0002] In the existing digital human interaction technology, rule-based behavior control methods and simple emotion recognition models are mainly used to generate digital human facial expressions and body movements. These technologies usually rely on preset rule libraries and limited emotion tags for action matching and generation, and have the following main problems:

[0003] (1) Insufficient accuracy in semantic and emotional analysis: In the existing technology, most systems use emotional analysis methods based on keyword matching or simple emotional classification. Such methods are often unable to accurately understand the semantics and infer emotions in complex dialogue situations. For example, when a user expresses implicit or multi-layered emotions, the existing system may misjudge the intensity of the emotion or fail to fully capture the potential emotional changes, causing the digital human's response to appear stiff and unnatural.

[0004] (2) Emotion-behavior mapping is not flexible and natural enough: In the existing technology, the mapping relationship between emotion and action is relatively fixed, lacking sufficient flexibility to cope with dynamic emotional changes in multiple rounds of dialogue. For example, the system cannot effectively adjust the movement frequency or expression details of the digital human according to different emotional fluctuations, resulting in the digital human showing overly stereotyped behavior during the interaction and unable to achieve natural transition.

[0005] (3) Lack of smooth processing of emotional fluctuations: In the existing technology, emotional fluctuations usually directly affect the behavioral labels of digital humans, causing expressions or actions to switch frequently in a short period of time, resulting in inconsistent performance. Many systems fail to consider the smooth transition of emotional fluctuations, resulting in the inability of digital humans to maintain stable and coherent expressions and actions in rapidly changing emotional states.

[0006] (4) Real-time rendering and response delay: In the existing technology, although multimodal rendering technology can produce relatively realistic digital human movements, it often has obvious rendering delays in high-frequency interactions, affecting the user experience. In addition, the existing technology still has certain problems of freezes and delays in the transmission and display of video streams. Especially in complex scenes, the system's response speed cannot match the user's interaction needs in a timely manner.

[0007] (5) Insufficient optimization of emotional behavior: In the existing technology, the rule base is updated in a relatively fixed manner and lacks an adaptive optimization mechanism. The update of emotion and action labels usually relies on manual adjustment or a relatively static rule base, which cannot be optimized in real time based on actual interaction feedback. Therefore, it is impossible to adjust emotional behavior in a personalized way according to the needs of different users and scenarios. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for driving digital human actions using a large language model, thereby solving the above-mentioned problems existing in the prior art.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] A method for driving digital human actions using a large language model comprises the following steps:

[0011] S1. Identify conversation content: Call the big model to identify the current and previous conversation content, predict the character's emotions and actions, and output coherent emotion and action labels;

[0012] S2. Establish a digital human rule library: Based on the mapping relationship between different emotions and actions, a digital human rule library covering various behavior patterns is established by combining deep learning with rule library construction methods. In the digital human rule library, the expressions and actions of the digital human are bound to the corresponding rendering logic.

[0013] S3, inference to generate video stream: Based on the recognized emotion and action tags of the current session, call the bound expression and action rendering logic in the digital human rule library to generate a real-time session video stream;

[0014] S4. Real-time video stream playback: Play the generated digital human conversation video stream frame by frame in real time.

[0015] Preferably, step S1 specifically includes the following contents:

[0016] S11, calling the emotion recognition and semantic processing large model, carrying out multi-level emotion analysis and semantic analysis on the current conversation content and historical context, and predicting the emotion and action information in the conversation content;

[0017] S12. Generate emotion and action labels based on the analysis results; and ensure the consistency of the currently generated labels by considering the labels of the previous conversation content.

[0018] Preferably, the emotion recognition and semantic processing large model is based on the Transformer network structure, processes the embedding vector of each word in the conversation text, and uses the time flow and language dependency of the dialogue sequence to achieve deep semantic understanding and reasoning; through the vectorized encoding method, the conversation emotion and action prediction results are converted into a label set.

[0019] Preferably, step S1 further comprises:

[0020] S13. To address the frequent action switching caused by emotional fluctuations in multiple rounds of conversations, a multi-layer emotion smoothing algorithm is used to optimize the stability and consistency of behavior labels.

[0021] Preferably, step S13 specifically comprises combining the emotion regression model, utilizing the mean filtering and exponential weighted smoothing method of the previous conversation labels, and dynamically adjusting the change weights of the Qing Guangxu and behavior labels; using the moving average window and exponential weighted technology to filter the action and emotion change curves, and ensuring that the label output is more in line with real human conversation habits.

[0022] Preferably, step S2 specifically includes the following contents:

[0023] S21. Based on the knowledge graph, the relationship between emotion categories and action expressions is logically modeled to support the analysis of complex emotion combinations;

[0024] S22. Use LSTM network to model the time series of emotions and behaviors to ensure the naturalness and smooth transition of digital human movements.

[0025] Preferably, step S2 further includes,

[0026] S23. Combining the recognized emotions and action labels, dynamically adjust the digital human's expression and action sequence through the temporal convolutional network, and generate the character's natural behavior labels suitable for the current session based on user feedback to optimize the digital human rule library.

[0027] Preferably, step S23 specifically includes adjusting the action frequency and expression details of the digital human in real time according to the emotional changes of multiple rounds of dialogues through the weight rules of the pre-action data stream; using a deep Q learning algorithm, combined with user ratings and feedback, to generate natural behavior labels of the characters suitable for the current conversation, continuously optimizing the digital human rule library, and improving the naturalness of performance in complex interactive scenarios.

[0028] Preferably, video streaming playback supports 1080p high-definition resolution and 60 frames refresh rate, and combined with a dynamic delay compensation mechanism, instantly updates the digital human movements during user interaction to ensure seamless connection during each session; a live-while-caching strategy is adopted during playback to reduce delays in large-scale rendering and video playback, ensuring high-quality real-time interactive effects.

[0029] The present invention also aims to provide a system for driving digital human actions using a large language model, the system being capable of implementing the above-mentioned method, the system comprising:

[0030] Conversation content recognition module: calls the large model to recognize the current and previous conversation content, predicts the character's emotions and actions, and outputs coherent emotion and action labels;

[0031] Digital human rule library building module: Based on the mapping relationship between different emotions and actions, a digital human rule library covering various behavior patterns is established by combining deep learning and rule library building methods. In the digital human rule library, the expressions and actions of the digital human are bound to the corresponding rendering logic.

[0032] Video stream reasoning generation module: Based on the recognized emotion and action tags of the current session, the bound expression and action rendering logic is called in the digital human rule library to generate a real-time session video stream;

[0033] Video stream real-time playback module: plays the generated digital human conversation video stream frame by frame in real time.

[0034] The beneficial effects of the present invention are as follows: 1. The high-precision semantic parsing and emotion inference provided by the present invention can accurately identify complex emotions in multiple rounds of dialogue, and accurately analyze the intensity of emotions and potential intentions, thereby generating more natural and contextual digital human actions. 2. The flexible emotion behavior mapping provided by the present invention dynamically adjusts the mapping relationship between emotions and actions through a deep learning model, so that digital humans can naturally transition according to real-time emotion changes in multiple rounds of dialogue. It can be applied to 2D digital humans based on wav2l ip reasoning, and can also be applied to the expression and action control of 3D digital humans based on unity or ue engines. 3. The adaptive emotion behavior optimization mechanism provided by the present invention optimizes the rule base in real time through user feedback and deep Q learning algorithms to ensure that the behavior and emotional response of digital humans can be continuously improved according to actual conditions. 4. The present invention uses the emotion and action label generation technology driven by a large language model to enable digital humans to dynamically adjust expressions and actions according to the conversation context and improve the continuity of emotions and actions. 5. The present invention optimizes the emotion label smoothing and video rendering process, shortens the response time of action generation and rendering, and enables the system to show a reaction speed close to that of humans in the conversation, achieving real-time and low-latency interaction. 6. The present invention can adapt to different interactive scene requirements, from scenes with high-intensity emotional changes (such as emotional companionship) to information-intensive scenes (such as virtual guided tours), and can ensure the smoothness and naturalness of digital human movements, reduce the phenomenon of abrupt movements or stiff expressions, and make digital humans more widely applicable. 7. Under the premise of ensuring real-time performance and interactive experience, the present invention adopts multi-threaded processing and streaming media optimization technology to effectively reduce system resource consumption. Tests show that under standard hardware configuration, the system memory occupied by the present invention is reduced by more than 8%, and the rendering speed is increased by 5%, thereby reducing the system operation cost and improving the overall system efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0037] like Figure 1As shown, in this embodiment, a method for driving digital human actions using a large language model is provided. Through high-precision semantic parsing and emotion inference, flexible emotion and behavior mapping, adaptive emotion label optimization mechanism and smoothing algorithm, the problems of unnatural digital human behavior generation, frequent emotion fluctuations, and inflexible emotion and action mapping in the prior art are solved, thereby achieving a more natural and coherent digital human interaction experience. This method includes the following four parts:

[0038] 1. Identify the content of the conversation

[0039] Call the big model to identify the current and previous conversation content, predict the character's emotions and actions, and output coherent emotion and action labels. Specifically, it includes the following:

[0040] 1.1. Multi-round conversation analysis and contextual semantic understanding: Call the large model of emotion recognition and semantic processing to carry out multi-level emotion analysis and semantic analysis (including semantic decomposition, emotion intensity analysis and potential intention mining) on ​​the current conversation content and historical context, and predict the emotion and action information in the conversation content.

[0041] In this embodiment, the core of the model is: based on the Transformer network structure, the embedding vector of each word in the conversation text is processed, and the time flow and language dependency of the conversation sequence are used to achieve deep semantic understanding and reasoning.

[0042] Emotion label generation: Through vectorized encoding methods, the conversation emotions and action prediction results are converted into a set of labels. The labels contain emotion intensity, action frequency, and performance rhythm, forming emotionally coherent behavioral instructions.

[0043] For example, if we input the dialogue "I am very happy today!", the generated emotion tags include:

[0044] Sentiment: "Positive";

[0045] Strength: 0.9;

[0046] Expression: "Smile";

[0047] Adjusted output: After smoothing by the emotion coherence parameter, the generated label is {emotion: "positive", intensity: 0.8, action frequency: 3Hz}, which enhances the system's context adaptation ability.

[0048] 1.2. Tag embedding structure: Generate emotion and action tags based on the analysis results; and ensure the coherence of the currently generated tags by considering the tags of the previous conversation content.

[0049] The model supports high-dimensional semantic embedding and emotion prediction, with an embedding vector dimension of 512. The output results include emotion vectors (Emotion Embedding), action vectors (Action Embedding), and time series vectors (Temporal Embedding).

[0050] Label generation logic: generate action and emotion mapping after parsing the input text.

[0051] For example, "happy" is mapped to "smile + nod quickly"; "sad" is mapped to "lower your head + blink slowly".

[0052] 1.3. Smooth optimization of emotional state changes: In view of the frequent action switching caused by emotional fluctuations in multiple rounds of conversations, a multi-layer emotional smoothing algorithm is used to optimize the stability and consistency of behavior labels.

[0053] Specifically: Combined with the emotion regression model, the mean filtering and exponential weighted smoothing method of the previous conversation labels are used to dynamically adjust the change weights of Qing Guangxu and behavior labels; the moving average window and exponential weighted technology are used to filter the action and emotion change curves to ensure that the label output is more in line with real human conversation habits.

[0054] 2. Establishing a Digital Human Rule Base

[0055] According to the mapping relationship between different emotions and actions, combined with deep learning and rule base construction methods, a digital human rule base covering various behavior patterns is established; in the digital human rule base, the expressions and actions of the digital human are bound to the corresponding rendering logic. Specifically, it includes the following contents:

[0056] 2.1. Construction of a high-precision emotion behavior label rule library (digital human rule library): Based on the mapping relationship between different emotion combinations and action labels, combined with deep learning and rule library construction methods, a high-precision rule library covering various behavior patterns is established.

[0057] (1) Emotion mapping rules: Based on the knowledge graph, the relationship between emotion categories and action expressions is logically modeled to support the analysis of complex emotion combinations;

[0058] (2) Time series modeling: The LSTM network is used to model the time series of emotions and behaviors to ensure the naturalness of the digital human’s movements and smooth transitions.

[0059] 2.2. Adaptive label update and behavior optimization: Combining the recognized emotion and action labels, dynamically adjust the digital human expression and action sequence through the temporal convolutional network, and generate the natural behavior label of the character suitable for the current session based on user feedback to optimize the digital human rule library.

[0060] (1) Dynamic label update: Based on the emotional changes in multiple rounds of conversations, the digital human’s action frequency and expression details are adjusted in real time through the weight rules of the previous action data stream.

[0061] (2) Reinforcement learning optimization: Using a deep Q-learning algorithm, combined with user ratings and feedback, we generate natural behavior labels for characters that are suitable for the current session, continuously optimize the digital human rule library, and improve the naturalness of performance in complex interactive scenarios.

[0062] In this embodiment, user feedback is collected after each session, and the behavior parameters in the label rule base are adjusted based on indicators such as user ratings and interaction time. The deep Q learning algorithm (DQN) is used to automatically optimize the rule base, and the coherence and naturalness of the digital human in interaction are continuously enhanced through reinforcement learning. The optimization of user ratings and behavior labels makes the system gradually mature and perfect in multi-round conversation scenarios to meet high-frequency interaction needs.

[0063] 3. Inference to generate video stream

[0064] Based on the identified emotion and action tags of the current conversation, the bound expression and action rendering logic is called in the digital human rule library to generate a real-time conversation video stream.

[0065] Digital humans can be 2D digital humans based on wav2 l ip, or 3D digital humans based on unity or ue.

[0066] 4. Real-time video streaming

[0067] The generated digital human conversation video stream is played frame by frame in real time.

[0068] Video streaming playback supports 1080p high-definition resolution and 60 frames refresh rate. Combined with a dynamic delay compensation mechanism, it instantly updates the digital human's movements during user interaction, ensuring seamless connection during each session. During playback, a buffering strategy is adopted to reduce delays in large-scale rendering and video playback, ensuring high-quality real-time interactive effects.

[0069] In this embodiment, a system for driving digital human actions using a large language model is provided. The system can implement the above-mentioned method. The system includes:

[0070] (1) Conversation content recognition module: calls the large model to recognize the current and previous conversation content, predicts the character's emotions and actions, and outputs coherent emotion and action labels;

[0071] (2) Digital human rule library building module: Based on the mapping relationship between different emotions and actions, a digital human rule library covering various behavior patterns is established by combining deep learning and rule library building methods. In the digital human rule library, the expressions and actions of the digital human are bound to the corresponding rendering logic.

[0072] (3) Video stream reasoning generation module: Based on the identified emotion and action tags of the current conversation, the bound expression and action rendering logic is called in the digital human rule library to generate a real-time conversation video stream;

[0073] (4) Video stream real-time playback module: plays the generated digital human conversation video stream frame by frame in real time.

[0074] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0075] The present invention provides a method and system for driving digital humans using a large language model. The high-precision semantic parsing and emotion inference provided by the present invention can accurately identify complex emotions in multiple rounds of dialogue, and accurately analyze the intensity of emotions and potential intentions, thereby generating more natural and contextual digital human actions. The flexible emotion behavior mapping provided by the present invention dynamically adjusts the mapping relationship between emotions and actions through a deep learning model, so that digital humans can naturally transition according to real-time emotional changes in multiple rounds of dialogue. It can be applied to 2D digital humans based on wav2 lip reasoning, and can also be applied to the expression and action control of 3D digital humans based on unity or ue engines. The adaptive emotion behavior optimization mechanism provided by the present invention optimizes the rule base in real time through user feedback and deep Q learning algorithms to ensure that the behavior and emotional response of digital humans can be continuously improved according to actual conditions. The present invention uses emotion and action label generation technology driven by a large language model to enable digital humans to dynamically adjust expressions and actions according to the conversation context and improve the continuity of emotions and actions. The present invention optimizes the emotion label smoothing and video rendering process, shortens the response time of action generation and rendering, and enables the system to show a reaction speed close to that of humans in the conversation, achieving real-time and low-latency interaction. The present invention can adapt to different interactive scene requirements, from scenes with high-intensity emotional changes (such as emotional companionship) to information-intensive scenes (such as virtual guided tours), and can ensure the smoothness and naturalness of digital human movements, reduce the phenomenon of abrupt movements or stiff expressions, and make digital humans more widely applicable. Under the premise of ensuring real-time performance and interactive experience, the present invention adopts multi-threaded processing and streaming media optimization technology to effectively reduce system resource consumption. Tests show that under standard hardware configuration, the system memory occupied by the present invention is reduced by more than 8%, and the rendering speed is increased by 5%, thereby reducing the system operation cost and improving the overall system efficiency.

[0076] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for driving digital human actions using a large language model, characterized in that: The following steps are included: S1. Identify conversation content: Call the big model to identify the current and previous conversation content, predict the character's emotions and actions, and output coherent emotion and action labels; S2. Establish a digital human rule library: Based on the mapping relationship between different emotions and actions, a digital human rule library covering various behavior patterns is established by combining deep learning with rule library construction methods. In the digital human rule library, the expressions and actions of the digital human are bound to the corresponding rendering logic. S3, inference to generate video stream: Based on the recognized emotion and action tags of the current session, call the bound expression and action rendering logic in the digital human rule library to generate a real-time session video stream; S4. Real-time video stream playback: Play the generated digital human conversation video stream frame by frame in real time.

2. The method for driving digital human motion using a large language model according to claim 1, characterized in that: Step S1 specifically includes the following contents: S11, calling the emotion recognition and semantic processing large model, carrying out multi-level emotion analysis and semantic analysis on the current conversation content and historical context, and predicting the emotion and action information in the conversation content; S12. Generate emotion and action labels based on the analysis results; and ensure the consistency of the currently generated labels by considering the labels of the previous conversation content.

3. The method for driving digital human motion using a large language model according to claim 2, characterized in that: The emotion recognition and semantic processing large model is based on the Transformer network structure, processes the embedding vector of each word in the conversation text, and uses the time flow and language dependency of the conversation sequence to achieve deep semantic understanding and reasoning; through the vectorized encoding method, the conversation emotion and action prediction results are converted into a label set.

4. The method of using a large language model to drive digital human motion according to claim 2, characterized in that: Step S1 also includes, S13. To address the frequent action switching caused by emotional fluctuations in multiple rounds of conversations, a multi-layer emotion smoothing algorithm is used to optimize the stability and consistency of behavior labels.

5. The method of using a large language model to drive digital human motion according to claim 4, characterized in that: Specifically, step S13 combines the emotion regression model and uses the mean filtering and exponential weighted smoothing method of the previous conversation labels to dynamically adjust the change weights of the Qing Guangxu and behavior labels; uses the moving average window and exponential weighted technology to filter the action and emotion change curves to ensure that the label output is more in line with real human conversation habits.

6. The method of using a large language model to drive digital human motion according to claim 1, characterized in that: Step S2 specifically includes the following contents: S21. Based on the knowledge graph, the relationship between emotion categories and action expressions is logically modeled to support the analysis of complex emotion combinations; S22. Use LSTM network to model the time series of emotions and behaviors to ensure the naturalness and smooth transition of digital human movements.

7. The method of using a large language model to drive digital human motion according to claim 6, characterized in that: Step S2 also includes, S23. Combining the recognized emotions and action labels, dynamically adjust the digital human's expression and action sequence through the temporal convolutional network, and generate the character's natural behavior labels suitable for the current session based on user feedback to optimize the digital human rule library.

8. The method of using a large language model to drive digital human motion according to claim 7, characterized in that: Step S23 specifically involves adjusting the digital human's action frequency and expression details in real time according to the emotional changes in multiple rounds of dialogues through the pre-action data stream weight rules; using a deep Q learning algorithm, combined with user ratings and feedback, to generate natural character behavior labels suitable for the current conversation, continuously optimizing the digital human rule library, and improving the naturalness of performance in complex interactive scenarios.

9. The method of using a large language model to drive digital human motion according to claim 1, characterized in that: Video streaming playback supports 1080p high-definition resolution and 60 frames refresh rate. Combined with a dynamic delay compensation mechanism, it instantly updates the digital human's movements during user interaction, ensuring seamless connection during each session. During playback, a buffering strategy is adopted to reduce delays in large-scale rendering and video playback, ensuring high-quality real-time interactive effects.

10. A system for driving digital human actions using a large language model, characterized in that: The system can implement the method described in any one of claims 1 to 9, and the system includes: Conversation content recognition module: calls the large model to recognize the current and previous conversation content, predicts the character's emotions and actions, and outputs coherent emotion and action labels; Digital human rule library building module: Based on the mapping relationship between different emotions and actions, a digital human rule library covering various behavior patterns is established by combining deep learning and rule library building methods. In the digital human rule library, the expressions and actions of the digital human are bound to the corresponding rendering logic. Video stream reasoning generation module: Based on the recognized emotion and action tags of the current session, the bound expression and action rendering logic is called in the digital human rule library to generate a real-time session video stream; Video stream real-time playback module: plays the generated digital human conversation video stream frame by frame in real time.

Images