A multi-stage semantic emotion-driven digital human motion generation method

Abstract

The application provides a multi-stage semantic emotion-driven digital human action generation method, and relates to the technical field of digital human action generation. The application extracts audio features and text semantic features and completes time alignment and fusion through multi-modal preprocessing of input audio and text to obtain a multi-modal feature sequence. Then, the sequence is used to identify semantic intent and analyze emotion to obtain a semantic intent sequence and an emotion curve containing categories, intensity and change trend. Subsequently, a global action script containing action type, intensity, duration and priority is generated according to a preset action mapping rule. The script is encoded into a feature vector and input into a lightweight neural network to obtain an action parameter sequence containing facial expressions, mouth shapes, skeletal actions and gaze directions. After time sequence smoothing and physical constraint checking, the final digital human driving parameters are output, which can realize low-power real-time operation, smooth action, accurate mouth shape synchronization and emotion and semantic matching.

Description

Technical Field

[0001] This application relates to the field of digital human motion generation technology, and in particular to a multi-stage semantic emotion-driven digital human motion generation method. Background Technology

[0002] Digital human motion generation is a core component of digital human technology, enabling virtual characters to exhibit body and facial expressions that fit the interactive scenarios. This technology is widely used in fields such as virtual live streaming, online education, and digital entertainment, and market demand continues to grow.

[0003] Current methods for generating digital human motion typically employ end-to-end deep learning models. The core of these models involves directly mapping input audio or text data into motion parameters for the digital human using network architectures such as CNNs and Transformers. This approach relies heavily on training with large amounts of data. Figure One The transformation from input to action output is completed in one go.

[0004] However, these methods are computationally intensive, making real-time response difficult on low-computing-power devices like smartphones. Furthermore, the generated actions are prone to inter-frame jitter, lip-sync accuracy is insufficient, and emotional expression fails to align with semantic content. These issues result in a poor digital human interaction experience. Therefore, existing technologies suffer from insufficient naturalness, synchronization, and adaptability in digital human motion generation. Summary of the Invention

[0005] The purpose of this application is to provide a multi-stage semantic emotion-driven digital human action generation method to solve the problems of insufficient naturalness, synchronization and adaptability in the generation of digital human actions in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a multi-stage semantic emotion-driven digital human action generation method, comprising: Multimodal preprocessing is performed on the input audio and text to extract audio features and text semantic features, and time alignment and feature fusion are performed to obtain a multimodal feature sequence; Based on the multimodal feature sequence, semantic intent recognition and sentiment analysis are performed to obtain a semantic intent sequence and a sentiment curve. The sentiment curve includes a sentiment category sequence, a sentiment intensity sequence, and a sentiment change trend. Based on the semantic intent sequence and emotion curve, a global action script is generated through preset action mapping rules. The global action script includes action type, action intensity, duration, and priority. The global action script is encoded into a feature vector and input into a lightweight neural network for action parameter mapping to obtain an action parameter sequence, which includes facial expression parameters, lip shape parameters, skeletal action parameters, and gaze direction parameters. The sequence of motion parameters is subjected to temporal smoothing and physical constraint checks, and the final digital human driving parameters are output.

[0007] Optionally, the step of performing semantic intent recognition and sentiment analysis based on the multimodal feature sequence to obtain a semantic intent sequence and a sentiment curve includes: The multimodal feature sequence is input into the semantic intent recognition model to obtain the probability distribution of the intent category for each time frame, and the semantic intent sequence is determined based on the probability distribution. The semantic intent recognition model is a bidirectional long short-term memory network. The multimodal feature sequence is input into the sentiment analysis model to obtain the probability distribution of sentiment category and sentiment intensity value of each time frame, forming sentiment category sequence and sentiment intensity sequence. The sentiment analysis model is a hybrid network of convolutional neural network and long short-term memory network. The emotion category sequence and the emotion intensity sequence are smoothed and optimized to generate an emotion curve, which includes the emotion category sequence, the emotion intensity sequence, and the emotion change trend.

[0008] Optionally, the step of generating a global action script based on the semantic intent sequence and emotion curve using preset action mapping rules includes: Based on the intent category in the semantic intent sequence and the emotion category in the emotion curve, a predefined action mapping rule table is queried to determine the corresponding action type; The action intensity of the action type is calculated based on the emotion intensity in the emotion curve and the intention weight corresponding to the intention category; The duration of the action type is calculated based on the duration characteristics of the emotion in the emotion curve and the length of the current semantic unit. Based on the action type, action intensity, action duration, and preset priority parameters, a preliminary action script is generated; The initial action script is subjected to action conflict detection, and the conflicts are eliminated according to a preset resolution strategy to generate the final global action script.

[0009] Optionally, the step of performing action conflict detection on the preliminary action script and eliminating conflicts according to a preset resolution strategy to generate the final global action script includes: Detect whether there are conflicting combinations of action types in the preliminary action script; When a conflicting combination of action types is detected, it is processed according to a preset resolution strategy. The resolution strategy includes: prioritizing the retention of actions with higher priority; if the priorities are the same, retaining the action with greater intensity; if both the priority and the intensity are the same, retaining the action with longer duration.

[0010] Optionally, the step of performing multimodal preprocessing on the input audio and text, extracting audio features and text semantic features, and performing time alignment and feature fusion to obtain a multimodal feature sequence includes: Time-frequency analysis and feature extraction are performed on the input audio to obtain an audio feature vector sequence containing Mel spectrum features, pitch features, and energy features; The input text is encoded using a pre-trained language model to obtain a text semantic vector; The text semantic vector is expanded into a text feature sequence aligned with the audio feature vector sequence in the time dimension, and the audio feature vector sequence and the text feature sequence are weighted and concatenated to generate the multimodal feature sequence.

[0011] Optionally, the step of encoding the global action script into a feature vector and inputting it into a lightweight neural network for action parameter mapping to obtain an action parameter sequence includes: The action type, action intensity, action duration and priority parameters contained in each time frame of the global action script are uniformly encoded to obtain the corresponding action script feature vector. The action script feature vector sequence is input into a lightweight neural network, which is used to map high-level action features to low-level driving parameters. The lightweight neural network maps and outputs a sequence of motion parameters for each time frame, including facial expression parameters, lip shape parameters, skeletal motion parameters, and gaze direction parameters.

[0012] Optionally, the lightweight neural network is a lightweight structure containing multiple fully connected layers with fewer than 1 million parameters, and it employs optimization strategies to accelerate inference, including INT8 quantization, operator fusion, and batch processing.

[0013] Optionally, the step of performing temporal smoothing and physical constraint checks on the motion parameter sequence to output the final digital human driving parameters includes: The motion parameter sequence is smoothed dimension by dimension using a temporal smoothing filter to obtain a smoothed motion parameter sequence. The smoothed motion parameter sequence is subjected to physical constraint checks, which include: joint angle limitation checks, facial expression parameter range checks, lip shape key point position checks, and gaze direction angle checks. Based on the inspection results, parameter values ​​that violate physical constraints in the smoothed motion parameter sequence are trimmed to the preset legal range, and the digital human driving parameters are output.

[0014] Optionally, it also includes: During smoothing, a sliding window caching mechanism is used to cache the processed action parameters of the most recent frames to avoid repeated smoothing calculations for overlapping frame segments. To reduce computational load, actions with a priority lower than a preset threshold in the action parameter sequence are updated at a reduced frequency.

[0015] Optionally, it also includes: The temporal smoothing and physical constraint checking process of the current frame is asynchronously and parallelized with the multimodal feature extraction process of the next frame.

[0016] The multi-stage semantic emotion-driven digital human action generation method provided in this application obtains comprehensive and temporally consistent input information by performing multimodal preprocessing on input audio and text, laying a solid foundation for subsequent analysis; semantic intent recognition and emotion analysis based on multimodal feature sequences can accurately capture content intent and emotion changes, providing core basis for action generation; generating a global action script based on intent sequences and emotion curves can clarify key action information and ensure accurate matching of actions with intent and emotion; mapping action parameters through lightweight neural networks can efficiently convert them into specific action parameters, reducing computational consumption; temporal smoothing and physical constraint checks on the action parameter sequence can improve the smoothness of actions and ensure that actions conform to physical laws.

[0017] Furthermore, the multimodal feature sequences are input into a semantic intent recognition model composed of a bidirectional long short-term memory network and a hybrid sentiment analysis model composed of a convolutional neural network and a long short-term memory network, respectively, to obtain the semantic intent sequence, sentiment category sequence, and sentiment intensity value for each time frame. The sentiment category sequence and sentiment intensity sequence are then smoothed and optimized to generate a sentiment curve containing sentiment category, intensity, and trend. This step, by selectively choosing an appropriate model, accurately captures the temporal and local features of semantic intent and sentiment. After smoothing and optimization, frequent sentiment switching is effectively avoided, outputting accurate and stable semantic intent sequences and high-quality sentiment curves, providing reliable support for the subsequent generation of global action scripts. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a multi-stage semantic emotion-driven digital human action generation method provided in this application embodiment; Figure 2This is a flowchart illustrating another multi-stage semantic emotion-driven digital human action generation method provided in an embodiment of this application. Detailed Implementation

[0020] In the field of digital human motion generation, existing technologies often employ end-to-end deep learning models. This approach attempts... Figure One While directly converting audio or text into digital human motion parameters in one go has significant shortcomings: on the one hand, the model requires a large amount of computation and is difficult to respond in real time on low-computing-power devices such as mobile phones, resulting in delays in motion generation; on the other hand, the lack of deep understanding of semantics and emotions not only makes the generated actions prone to inter-frame jitter, but also makes it difficult to accurately synchronize lip movements with speech, and emotional expression is also disconnected from the content of speech, seriously affecting the naturalness of digital human interaction.

[0021] To address the aforementioned issues, this invention proposes a multi-stage semantic emotion-driven digital human action generation method. The core of this method is to break down the complex action generation process into several concise steps: first, integrating audio and text information; second, accurately identifying content intent and emotional changes; and third, generating a suitable global action plan. This plan is then transformed into specific action parameters using a lightweight model, and finally, the smoothness and plausibility of the actions are optimized. This method reduces the computational burden through multi-stage decomposition, allowing even low-computing-power devices to run in real-time. Simultaneously, by leveraging intent and emotion analysis, it ensures precise matching of actions with content and emotion. Furthermore, subsequent optimizations address issues such as motion jitter and lip-sync asynchrony, fundamentally improving upon the shortcomings of existing technologies and enhancing the naturalness and adaptability of digital human interaction.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The core of this application is to provide a multi-stage semantic emotion-driven digital human action generation method, the flowchart of one specific implementation of which is shown below. Figure 1 As shown, the method includes: S101. Perform multimodal preprocessing on the input audio and text, extract audio features and text semantic features, and perform time alignment and feature fusion to obtain a multimodal feature sequence.

[0024] Among them, the multimodal feature sequence is a feature set that integrates key audio information and text semantic information and remains consistent in the time dimension, which is used to provide comprehensive and synchronous input data for subsequent semantic intent recognition and sentiment analysis.

[0025] S101 specifically includes: S1011. Perform time-frequency analysis and feature extraction on the input audio to obtain an audio feature vector sequence containing Mel spectrum features, pitch features, and energy features.

[0026] Among them, Mel-frequency characteristics simulate the human ear's perception of sounds at different frequencies. Frequency-related data extracted from audio reflects the timbre and spectral distribution of the sound. Pitch characteristics are the high and low attributes of audio, corresponding to the pitch variations in human speech. Energy characteristics are the strength data of the audio signal, reflecting the volume of speech.

[0027] S1012. Encode the input text using a pre-trained language model to obtain a text semantic vector.

[0028] Among them, the pre-trained language model is an intelligent model trained in advance on massive amounts of text data, which has the ability to understand the meaning of words and capture semantic logic. The text semantic vector is a numerical vector that transforms text content and can intuitively carry the semantic information of the text.

[0029] S1013. Expand the text semantic vector into a text feature sequence aligned with the audio feature vector sequence in the time dimension, and then perform weighted concatenation and fusion of the audio feature vector sequence and the text feature sequence to generate the multimodal feature sequence.

[0030] Time alignment ensures a one-to-one correspondence between text features and audio features on the timeline, guaranteeing that the semantics of the text at a given moment matches the audio information from the same period. Weighted splicing fusion combines the two types of features into a unified sequence according to preset weights, allowing the acoustic information of the audio and the semantic information of the text to complement each other.

[0031] In one specific implementation, this step first extracts the core features of audio and text separately, then eliminates the temporal discrepancy between the two types of data through time alignment, and finally forms a multimodal feature sequence that combines acoustic and semantic information through weighted fusion. Specifically: First, in step S1011, a short-time Fourier transform algorithm is used to perform time-frequency analysis on the input audio. The window size is set to 512 sampling points and the step size to 256 sampling points. A Hanning window is selected to reduce spectral leakage. The algorithm converts the continuous audio signal into a discrete time-frequency matrix. Based on this matrix, key features are extracted sequentially: Mel spectral features are obtained by processing with 80 Mel filter banks that simulate human hearing characteristics; with the help of the PyWorld audio processing library, pitch features in the range of 50 to 500 Hz are extracted with parameters of 512 frame length and 256 frame shift. These features can reflect the pitch changes of speech; by calculating the root mean square value of each audio frame, the energy features that dynamically change over time are obtained. These features can reflect the volume strength. These three types of features are concatenated according to dimensions to form an audio feature vector sequence with a time step of 10 milliseconds and a total dimension of 128.

[0032] Secondly, in step S1012, the text that completely corresponds to the audio content is encoded using a BERT pre-trained language model. This model first performs word segmentation on the text, splitting the sentence into independent words, and then converting the words into fixed-dimensional word vectors. Through a 12-layer Transformer network, it captures the contextual semantic relationships between words. The model has been pre-trained on a massive amount of Chinese and English text with 110M parameters, and can accurately identify the semantic logic and potential sentiment of the text. Finally, it outputs a 768-dimensional text semantic vector, which condenses the core meaning of the entire text.

[0033] Finally, time alignment is achieved through the forced alignment algorithm in step S1013. First, the start and end times of each word in the text are analyzed in the audio. Then, the 768-dimensional text semantic vector is expanded according to the time frames of the audio to make the time length of the text feature sequence completely consistent with the audio feature vector sequence, ensuring that the audio features of each time frame can match the corresponding text semantics. After that, the two types of sequences are concatenated according to the preset weights, with the audio feature weight being 0.6 and the text feature weight being 0.4, generating a multimodal feature sequence with a dimension of 896.

[0034] As an example, we input a 30-second audio clip expressing the anticipation of a weekend outing in 16kHz mono 16-bit WAV format, along with corresponding text. First, we perform time-frequency analysis on the audio using the STFT algorithm, setting the window size to 512, the step size to 256, and using a Hanning window. This converts the audio into 3000 time frames, calculated by dividing 30 seconds by 0.01 seconds, forming a time-frequency matrix. Based on this matrix, we extract 80-dimensional Mel-spectral features, which reflect the timbre of words like "outing" and "anticipation." We also extract pitch features in the 50-500 Hz range, where the pitch of "anticipation" is higher than "weekend." Finally, we extract energy features that change over time, with the energy of the "anticipation" portion being higher than the first half. These three types of features are then concatenated to form a 3000-frame, 128-dimensional audio feature vector sequence.

[0035] Next, the text is input into the BERT-base pre-trained model, which has 110M parameters. After being pre-trained on a massive amount of Chinese text, the model first segments the text into eight word units: weekend, want, go, outing, comma, too, expect, and finished. After converting the word units into word vectors, the Transformer network captures the semantic relationships between words. It can identify that outing is the core event and expectation is the emotional tendency, and finally outputs a 768-dimensional text semantic vector.

[0036] Finally, the forced alignment algorithm is used to determine the audio time position corresponding to each word. For example, "outing" corresponds to the 10th to 15th second, which is 100 frames to 150 frames. The 768-dimensional text semantic vector is expanded into a 3000-frame multiplied 768-dimensional text feature sequence. Then, the two types of sequences are concatenated according to the ratio of audio feature weight 0.6 and text feature weight 0.4 to form a 3000-frame multiplied 896-dimensional multimodal feature sequence.

[0037] This application, through this step, fully integrates the acoustic information of audio and the semantic information of text, ensuring that the two types of features are synchronously matched in the time dimension. This effectively avoids the limitations of single-modal information and provides high-quality and comprehensive input data for subsequent semantic intent recognition and sentiment analysis, ensuring the accuracy of subsequent processing steps.

[0038] S102. Based on the multimodal feature sequence, perform semantic intent recognition and sentiment analysis to obtain semantic intent sequence and sentiment curve.

[0039] The semantic intent sequence is a set of speaking intentions corresponding to each time frame, covering types such as statements, questions, and exclamations, used to clarify the intentional direction of the digital human's actions. The emotion curve is a dynamic sequence integrating emotion categories, intensity, and trends, which can intuitively reflect the fluctuations of emotions over time. The emotion curve includes an emotion category sequence, an emotion intensity sequence, and an emotion change trend. The emotion category sequence is a set of specific emotion types corresponding to each time frame, including happiness, sadness, and neutrality; the emotion intensity sequence is a set of numerical values ​​representing the intensity of the emotion corresponding to each time frame, ranging from 0 to 1; and the emotion change trend is the change in the difference in emotion intensity between adjacent time frames, used to reflect the gradual or abrupt changes in emotion.

[0040] S102 specifically includes: S1021. Input the multimodal feature sequence into the semantic intent recognition model to obtain the probability distribution of the intent category for each time frame, and determine the semantic intent sequence based on the probability distribution.

[0041] Among them, the semantic intent recognition model uses a bidirectional long short-term memory network, which is a temporal model that can simultaneously capture information about the preceding and following directions of a sequence. It is used to mine semantic associations in the temporal dimension from multimodal feature sequences and accurately identify the speaking intent of each time frame.

[0042] S1022. Input the multimodal feature sequence into the emotion analysis model to obtain the probability distribution of emotion category and emotion intensity value of each time frame, forming an emotion category sequence and an emotion intensity sequence.

[0043] The emotion analysis model is a combination of convolutional neural networks and long short-term memory networks: the convolutional neural network part is used to extract local key information from multimodal features, such as acoustic or semantic fragments related to specific emotions; the long short-term memory network part is used to capture the continuous change pattern of emotions over time, and the two work together to achieve accurate identification of emotion categories and intensities.

[0044] S1023. Smooth and optimize the emotion category sequence and the emotion intensity sequence to generate an emotion curve.

[0045] Among them, smoothing and optimization processing uses specific algorithms to reduce meaningless mutations and noise in the emotional sequence, avoid frequent switching of emotions, and make emotional changes more in line with the laws of natural human expression.

[0046] In one specific implementation, this step first identifies semantic intent and emotion-related sequences using a dedicated model, then smooths and optimizes the emotion sequence to finally obtain the semantic intent sequence and the complete emotion curve. Specifically: First, step S1021 addresses semantic intent recognition: a multimodal feature sequence dataset labeled with five intent categories—category 0 (statement), category 1 (question), category 2 (exclamation), category 3 (command), and category 4 (other)—is constructed. This dataset is used to train a bidirectional long short-term memory network. This network captures the temporal dependency of the feature sequence through information transmission in both directions. During training, the network parameters are continuously adjusted to optimize the intent recognition accuracy. The multimodal feature sequence is then input into the trained model, which outputs the probability distribution of the five intent categories for each time frame. The intent with the highest probability is selected as the final intent for that time frame, and the intents of all time frames are combined to form a semantic intent sequence.

[0047] Secondly, emotion analysis is performed in step S1022: a multimodal feature sequence dataset is constructed, which is labeled with seven emotion categories: Category 0 (happiness), Category 1 (sadness), Category 2 (anger), Category 3 (surprise), Category 4 (fear), Category 5 (disgust), and Category 6 (neutrality), along with their corresponding intensity values. This dataset is used to train a hybrid model of convolutional neural network and long short-term memory network. Specifically, the convolutional neural network first extracts local key information from the features, and then the long short-term memory network captures the temporal changes of emotions. During training, the model parameters are optimized to reduce recognition errors. The multimodal feature sequence is input into the trained hybrid model, which outputs the probability distribution of the seven emotion categories and their corresponding intensity values ​​for each time frame. These are then combined to form emotion category sequences and emotion intensity sequences, respectively.

[0048] Finally, the emotion sequence is smoothed and optimized in step S1023: a sliding window averaging algorithm is used, a window size of 5 frames is set, the average value of the emotion intensity in each window is calculated, and the intensity value of the center frame of the window is replaced; through a three-frame confirmation mechanism, the emotion switch is confirmed only when the same emotion category change occurs in three consecutive frames; the difference in emotion intensity between adjacent time frames is calculated to obtain the emotion change trend, and the processed emotion category sequence, intensity sequence and change trend are integrated to generate an emotion curve.

[0049] As an example, processing is performed based on a 3000-frame × 896-dimensional multimodal feature sequence corresponding to the previous 30 seconds of audio. First, semantic intent recognition is performed. The multimodal feature sequence is input into a trained bidirectional long short-term memory network. The model captures the semantic statement "going on a weekend outing" in the first 2000 frames, with the output category 0 statement intent probability all above 0.8. The exclamation semantic statement "I'm so looking forward to it" in the last 1000 frames is captured, with the output category 2 exclamation intent probability all above 0.9. The intent with the highest probability in each frame is selected to form a 3000-frame semantic intent sequence, with the first 2000 frames being category 0 statements and the last 1000 frames being category 2 exclamations.

[0050] Next, sentiment analysis is performed. The multimodal feature sequence is input into the CNN-LSTM hybrid model. The convolutional layer extracts the local features of positive emotions corresponding to "expectation", and the LSTM layer captures the change of emotions from calm to excitement. In the first 1500 frames, the probability of neutral emotion category 6 is higher than 0.7 and the intensity value is between 0.2 and 0.3. In the next 1500 frames, the probability of happy emotion category 0 gradually increases from 0.4 to 0.9 and the intensity value gradually increases from 0.3 to 0.8, forming the emotion category sequence and the emotion intensity sequence, respectively.

[0051] Finally, a smoothing optimization is performed using a 5-frame sliding window averaging algorithm to process the intensity sequence. The intensity values ​​for category 0 (happy emotion) in the last 1500 frames transition more smoothly. A three-frame confirmation mechanism confirms the official switch to category 0 happy emotion starting from frame 1500. The intensity difference between adjacent frames is calculated, revealing a positive and gradually increasing trend in the intensity difference over the last 1500 frames, ultimately generating an emotion curve containing all three categories of information. The above example is merely one illustration of this application. In practical applications, model training parameters, window size, number of confirmation frames, etc., can be adjusted according to requirements; this application does not impose any limitations on these adjustments.

[0052] This application, through this step, accurately captures the type of semantic intent and the dynamic changes of emotion, effectively avoiding abrupt changes in emotion sequences and noise interference, making the output semantic intent sequence and emotion curve more in line with the actual expression scenario, and laying a reliable foundation for the subsequent generation of digital human actions that conform to intent and emotion.

[0053] S103. Generate a global action script based on the semantic intent sequence and emotion curve using preset action mapping rules.

[0054] The global action script includes action type, action intensity, duration, and priority. The global action script is a set of instructions that integrates the core information of the actions the digital human needs to perform. It clarifies what action the digital human should take in each time frame, the magnitude of the action, the duration of execution, and which action should be prioritized. Action type refers to specific behavioral instructions, such as smiling, nodding, or raising eyebrows, conforming to human expression habits. Action intensity is the magnitude of the action, reflecting its prominence. Duration is the length of time a single action lasts from start to finish. Priority is the rule governing the order of action execution, ensuring that critical actions are performed first.

[0055] S103 specifically includes: S1031. Based on the intent category in the semantic intent sequence and the emotion category in the emotion curve, query the predefined action mapping rule table to determine the corresponding action type.

[0056] The action mapping rule table is a pre-defined combination of intent categories and emotion categories, and a corresponding action type association table. It covers the action matching logic in common expression scenarios, ensuring that the action type and semantics and emotions are accurately matched.

[0057] S1032. Calculate the action intensity of the action type based on the emotion intensity in the emotion curve and the intention weight corresponding to the intention category.

[0058] Among them, the intention weight is a weight value set according to different intention types, used to adjust the intensity of the action so that the action is more in line with the focus of the intention expression. The calculation of action intensity combines the intensity of emotion and the weight of intention to dynamically adjust the amplitude of the action.

[0059] S1033. Calculate the action duration of the action type based on the emotion duration characteristics in the emotion curve and the length of the current semantic unit.

[0060] Among them, the emotion duration feature is the duration information of the emotion category remaining unchanged. The semantic unit length is the length of words or phrases in the current expression. Together, they determine the reasonable duration of the action execution, avoiding actions that are too short or too long, which would appear unnatural.

[0061] S1034. Based on the action type, action intensity, action duration, and preset priority parameters, generate a preliminary action script.

[0062] The preset priority parameters are fixed priority levels set for different types of actions, clearly defining the execution priority of each type of action, with core actions being given priority for execution. The initial action script is a set of initial instructions that integrates core action information but does not handle conflicts.

[0063] S1035. Perform action conflict detection on the preliminary action script and eliminate the conflict according to the preset resolution strategy to generate the final global action script.

[0064] Action conflicts refer to combinations of actions that cannot be executed simultaneously within the same time frame. Conflict detection is the process of identifying such contradictory combinations in the initial action script. The resolution strategy is to select and retain the more suitable action according to predetermined rules, ensuring that the action execution is conflict-free.

[0065] S1035 specifically includes: The system detects whether there are conflicting action type combinations in the preliminary action script. When a conflicting action type combination is detected, it is processed according to a preset resolution strategy. The resolution strategy includes: prioritizing the retention of actions with higher priority; if the priorities are the same, retaining the action with greater intensity; if both the priority and the intensity are the same, retaining the action with longer duration.

[0066] Common combinations of conflicting actions include nodding and shaking the head, turning left and turning right, and smiling and frowning. These actions cannot be performed simultaneously physiologically and require filtering through problem-solving strategies.

[0067] like Figure 2 As shown, in one specific implementation, this step first matches the action type with the intention and emotion, then calculates the action intensity and duration sequentially, sets priorities, handles conflicts, and finally generates a complete global action script. Specifically: First, the action type is determined through step S1031, and a predefined action mapping rule table is established, covering six core combination rules. The contents of the mapping rule table are as follows: Rule 1: Happy + Statement corresponds to the action type [smiling, slight nodding, increased blinking frequency]; Rule 2: Question + Neutral correspondence to action type [raising eyebrows, tilting head, pointing gesture]; Rule 3: Surprise + Exclamation correspond to the following action types: [wide eyes, open mouth, lean back]; Rule 4: Sad + Statement corresponds to the following action types: [drooping corners of the mouth, head down, reduced blinking frequency]; Rule 5: Anger + Command corresponds to the following action types: [frowning, emphasizing gestures, leaning forward]; Rule 6: Default rule (other combinations) corresponds to the action type [neutral expression, slight nod].

[0068] For each time frame, the intent category and emotion category are queried to determine the corresponding list of action types. At the same time, the number of actions is adjusted according to the emotion intensity. When the emotion intensity is low, only the main actions are retained, and when the intensity is high, all actions are retained.

[0069] Secondly, the action intensity is calculated in step S1032: A base intensity is set for each action type, such as 0.6 for smiling, 0.5 for nodding, and 0.7 for raising eyebrows; intention weights are set: Category 1: Question 1.2, Category 2: Exclamation 1.1, Category 0: Statement 0.8, Category 3: Command 1.0, and Category 4: Other 1.0; using the formula: (1) The calculation is performed, and the result is limited to between 0 and 1.

[0070] Then, the action duration is calculated in step S1033: A base duration is set for each action type, such as 300ms for nodding and 200ms for raising an eyebrow; the emotion duration factor is calculated, which is the current emotion duration frame count divided by the average emotion duration frame count, normalized to between 0.8 and 1.2; the semantic unit length factor is calculated, which is the current word or phrase length divided by the average word length, normalized to between 0.9 and 1.1; using the formula: (2) Calculate to ensure the duration is within a reasonable range.

[0071] Next, step S1034 sets priorities: priority levels are set according to the importance of the action, with priority 5 for lip movements, priority 4 for facial expressions, priority 3 for eye movements, priority 2 for head movements, and priority 1 for body movements; the action type, intensity, duration, and corresponding priority are integrated to generate a preliminary action script.

[0072] Finally, conflict is handled in step S1035: check if there are conflicting combinations such as nodding and shaking, turning left and turning right, smiling and frowning in the preliminary action script; if there is a conflict, first compare the action priorities and keep the action with higher priority; if the priorities are the same, keep the action with greater intensity; if the intensity is also the same, keep the action with longer duration, and finally generate a conflict-free global action script.

[0073] As an example, processing is performed based on the semantic intent sequence and emotion curve of 3000 frames corresponding to the previous 30 seconds of audio. First, the action type is determined: the first 2000 frames are category 0 statement, the emotion of the first 1500 frames is category 6 neutral, the rule table is queried to match the default rule, the action type is neutral expression, slight nodding, because the emotion intensity of 0.2 to 0.3 is low, only slight nodding is retained; the emotion of the next 500 frames changes to category 0 happy, the rule table is queried to match category 0 happy plus category 0 statement, the action type is smiling, slight nodding, increased blinking frequency, the emotion intensity gradually increases from 0.3 to 0.8, after the intensity exceeds 0.7, all three actions are retained; the next 1000 frames are category 2 exclamation plus category 0 happy, the corresponding action type in the rule table is wide-eyed, mouth open, body leaning back.

[0074] Next, calculate the action intensity: taking the smiling action as an example, the base intensity is 0.6, the emotional intensity of a certain time frame in the last 500 frames is 0.8, and the intention weight is 0.8. Substituting into formula (1), the action intensity is calculated to be 0.6×0.8×0.8=0.384; in the last 1000 frames, the base intensity of the wide-eyed action is 0.9, the emotional intensity is 0.8, and the intention weight is 1.1. Substituting into formula (1), the action intensity is calculated to be 0.9×0.8×1.1=0.792.

[0075] Then calculate the duration of the action: taking a slight nod as an example, the base duration is 300ms, the emotion duration is 1500 frames in the first 1500 frames, the average emotion duration is 1000 frames, and the emotion duration factor is... After normalization, the length is 1.2; the current semantic unit "will go on the weekend" has a length of 4, the average word length is 3, and the semantic unit length factor is... After normalization, 1.1; Substituting into equation (2), the duration of the action is calculated to be 300×1.2×1.1=396ms, which is valid in the range of 100 to 2000ms.

[0076] Next, priorities are set: a slight nod is a head movement priority 2, a smile is an facial expression priority 4, widening the eyes is an facial expression priority 4, opening the mouth is a mouth shape priority 5, and leaning back is a body movement priority 1. Finally, conflicts are handled: widening the eyes and opening the mouth are not conflicting in the last 1000 frames and are both retained; if a hypothetical conflict arises between smiling and frowning, since both have a priority of 4, the stronger movement is retained after comparing their strengths. The above example is only one example of this application. In practical applications, the rule table content, base strength, priority settings, etc., can be adjusted according to requirements, and this application does not limit this.

[0077] This application transforms semantic intent and emotion into specific, reasonable, and conflict-free action instructions through this step, ensuring that the action type, intensity, and duration are highly consistent with the expression scenario, and providing a clear and feasible execution plan for digital humans to generate natural and fitting actions.

[0078] S104. The global action script is encoded into a feature vector and input into a lightweight neural network for action parameter mapping to obtain an action parameter sequence.

[0079] The motion parameter sequence is a set of specific values ​​that directly drive the digital human to perform corresponding actions. It includes facial expression parameters, lip shape parameters, skeletal motion parameters, and gaze direction parameters, serving as a crucial bridge connecting abstract motion instructions with the actual actions of the digital human. Feature vectors transform different types of information from the global motion script into numerical vectors in a unified format, facilitating neural network recognition and processing. Lightweight neural networks are network structures with few parameters and high computational efficiency, designed specifically for low-computing-power devices, and can quickly convert motion instructions into specific parameters.

[0080] S104 specifically includes: S1041. The action type, action intensity, action duration and priority parameters contained in each time frame of the global action script are uniformly encoded to obtain the corresponding action script feature vector.

[0081] Uniform encoding is the process of transforming action information with different attributes and formats into numerical vectors of fixed dimensions and uniform format, ensuring that the neural network can stably receive and process input information. The action script feature vector is the final product after encoding, condensing all the core attributes of a single frame's action and providing standardized input for subsequent network mapping.

[0082] S1042. Input the action script feature vector sequence into a lightweight neural network, which is used to map high-level action features to low-level driving parameters.

[0083] Here, high-level action features refer to the abstract action information contained in the action script feature vector; low-level driving parameters are the specific values ​​that directly control the joints, facial muscles, etc. of the digital human model. The lightweight neural network is a lightweight structure containing multiple fully connected layers, with fewer than 1 million parameters, and it employs optimization strategies to accelerate inference, including INT8 quantization, operator fusion, and batch processing.

[0084] S1043. Through the mapping of the lightweight neural network, output the sequence of motion parameters for each time frame.

[0085] Among these parameters, facial expression parameters are the weighted values ​​that control the facial muscle movements of the digital human, such as the degree of muscle contraction corresponding to opening the eyes and smiling. Lip shape parameters are the two-dimensional keypoint coordinates that control the shape of the digital human's lips, corresponding to the lip shape during different pronunciations. Skeletal movement parameters are the three-dimensional coordinates that control the position of the digital human's skeletal joints, such as the movement trajectory of the head and limbs. Gaze direction parameters are the values ​​that control the rotation angle of the digital human's eyes, such as pitch and yaw angles, determining the direction of the gaze.

[0086] In one specific implementation, this step first uniformly encodes the global action script to obtain a standardized feature vector, then inputs it into a trained lightweight neural network, and outputs a complete sequence of action parameters through network mapping. Specifically: First, unified encoding is performed through step S1041: For action type, 20 common action types are preset, and one-hot encoding is used. Each action corresponds to a 20-dimensional vector, with the dimension value corresponding to the action being 1 and the rest being 0; for action intensity, the original value between 0 and 1 is directly used, occupying 1 dimension; for action duration, the original duration (in milliseconds) is divided by the maximum duration of 2000 milliseconds and normalized to between 0 and 1, occupying 1 dimension; for priority, the original priority (1 to 5) is divided by 5 and normalized to between 0 and 1, occupying 1 dimension; these four encoding results are concatenated according to dimension to form a 23-dimensional action script feature vector, with each time frame corresponding to a feature vector, forming a feature vector sequence.

[0087] Secondly, a lightweight neural network is constructed and trained in step S1042: the network adopts a three-layer fully connected layer structure, with an input layer dimension of 23, a first hidden layer dimension of 128, a second hidden layer dimension of 64, and an output layer dimension of 169; a dataset labeled with action script feature vectors and corresponding action parameter sequences is constructed, and the network is trained using this dataset. During training, the ReLU activation function is used to enhance the nonlinear expression capability of the network, a Dropout mechanism is added to prevent overfitting, and the network weight parameters are continuously adjusted to minimize the error between the network output action parameters and the labeled values; after training, the ONNXRuntime framework is used for inference optimization, INT8 quantization is enabled to reduce data storage and computation, multiple computation steps are merged by operator fusion, and the feature vectors of 8 time frames are processed at once in batch processing to improve inference speed.

[0088] Finally, motion parameter mapping is performed in step S1043: the feature vector sequence is input into the optimized lightweight neural network, and the network maps the 23-dimensional high-level feature vector into 169-dimensional low-level driving parameters through the calculation of multiple fully connected layers. These parameters include 52-dimensional facial expression parameters, 40-dimensional lip shape parameters, 75-dimensional skeletal motion parameters, and 2-dimensional gaze direction parameters. The motion parameters for each time frame are output to form a motion parameter sequence.

[0089] As an example, processing is performed based on the global action script of 3000 frames corresponding to the previous 30 seconds of audio. First, uniform encoding is performed: the action type of the first 1500 frames is slight nodding, and the dimension value corresponding to "slight nodding" in one-hot encoding is 1, while the other 19 dimensions are 0; the action intensity is 0.3, the duration is 396 milliseconds, and after normalization, it is 0.198; the priority is 2, and after normalization, it is 0.4. These are concatenated to form a 23-dimensional feature vector. The next 1500 frames contain actions such as smiling, widening eyes, and opening the mouth, each corresponding to its own one-hot encoding. The action intensity is 0.384 to 0.792, the duration is 300 to 500 milliseconds, and after normalization, it is 0.15 to 0.25; the priority is 4 or 5, and after normalization, it is 0.8 or 1.0. These are concatenated to form a 23-dimensional feature vector.

[0090] Next, a lightweight neural network is input: the 3000-frame feature vector sequence is input into the trained network in batches of size 8. The INT8 quantization and operator fusion mechanism reduce the computational overhead, and the network quickly completes the mapping calculation.

[0091] The final output sequence of motion parameters is as follows: the first 1500 frames output the corresponding 75-dimensional skeletal motion parameters (3D coordinates of the neck joint, corresponding to a slight nodding motion) and 52-dimensional facial expression parameters (neutral expression weights, all around 0.2); the next 1500 frames output the corresponding 52-dimensional facial expression parameters (smiling corresponds to a weight of 0.384, widening eyes corresponds to a weight of 0.792), 40-dimensional mouth shape parameters (2D coordinates of lip shape keypoints corresponding to opening the mouth), 75-dimensional skeletal motion parameters (3D coordinates of the torso joints, corresponding to a backward leaning motion), and 2-dimensional gaze direction parameters (eye pitch angle of 0 degrees, maintaining a straight gaze). The above example is only one example of this application. In practical applications, the encoding method, network layer dimensions, optimization strategy parameters, etc., can be adjusted according to requirements, and this application does not limit this.

[0092] The lightweight neural network can be a one-dimensional adaptation of MobileNetV2, which further reduces the amount of computation through depthwise separable convolutions. The inference optimization strategy can add constant folding to complete the constant calculations in the network in advance, thereby reducing the amount of real-time computation during inference. The core process is the same as described above.

[0093] This application achieves efficient conversion from abstract action scripts to specific driving parameters through this step. The lightweight design and inference optimization ensure compatibility with low-computing-power devices. The parameter dimensions comprehensively cover the core needs of digital human actions, providing a direct basis for the precise execution of actions by digital humans.

[0094] S105. Perform temporal smoothing and physical constraint checks on the motion parameter sequence, and output the final digital human driving parameters.

[0095] Among them, the digital human driving parameters are the final set of parameters that have been optimized and verified and can be directly used to control the digital human's actions, ensuring that the digital human's movements are both smooth and natural and conform to physiological laws. Temporal smoothing is the elimination of meaningless abrupt changes in the sequence of action parameters through specific algorithms, making the transition of actions more coherent. Physical constraint checking verifies whether the parameters conform to the range of human physiological activity and the laws of facial muscle movement, avoiding unnatural postures.

[0096] S105 specifically includes: S1051. The motion parameter sequence is smoothed dimension by dimension using a temporal smoothing filter to obtain a smoothed motion parameter sequence.

[0097] Among them, the time-series smoothing filter is a tool specifically designed for processing sequence data. It can identify and filter high-frequency noise and sudden fluctuations in parameter sequences, retain the core trend of motion changes, and make motion parameters change more smoothly over time.

[0098] S1052. Perform physical constraint checks on the smoothed motion parameter sequence. The physical constraint checks include: joint angle limitation checks, facial expression parameter range checks, lip shape key point position checks, and gaze direction angle checks.

[0099] Among these checks, the joint angle limitation check verifies whether the rotation angles of the digital human's skeletal joints are within the physiologically permissible range of motion, such as the limits of movement in the neck, shoulders, and elbows. The facial expression parameter range check ensures that the parameters controlling facial muscles are within a reasonable and effective range, avoiding exaggerated or distorted expressions. The lip shape key point position check confirms that the lip shape key points are within a reasonable area of ​​the face and do not exceed the facial boundaries. The gaze direction angle check verifies whether the eyeball rotation angle conforms to the range of human eye movement, avoiding excessively exaggerated gaze deviations.

[0100] S1053. Based on the inspection results, trim the parameter values ​​that violate physical constraints in the smoothed motion parameter sequence to the preset legal range, and output the digital human driving parameters.

[0101] Among them, trimming refers to adjusting parameter values ​​that exceed the preset legal range to the maximum or minimum value of the range boundary, ensuring that all parameters meet the physical constraint requirements, while not changing the overall trend of parameter change.

[0102] In one specific implementation, this step first smooths the motion parameter sequence dimension by dimension to eliminate jitter and abrupt changes, then verifies the validity of the parameters through multiple physical constraints, and finally prunes illegal parameters to output the final driving parameters. Specifically: First, temporal smoothing is performed in step S1051: a Savitzky-Golay filter is used, and the window size is dynamically adjusted according to the action type. A 5-frame window is set for fast actions and a 7-frame window is set for slow actions. The polynomial order is set to 3, and each dimension of the action parameter sequence is filtered separately. At the action type switching point (such as switching from nodding to leaning back), the cubic spline interpolation method is used to perform interpolation calculations within a range of 3 frames before and after the switching point to ensure that the action transition is smooth without any stutters or abrupt changes, thus obtaining a smoothed action parameter sequence.

[0103] Secondly, physical constraint checks are performed in step S1052: For joint angle limitation checks, the range of neck pitch angle is set to -45 degrees to 45 degrees, yaw angle to -60 degrees to 60 degrees, roll angle to -30 degrees to 30 degrees, shoulder flexion to 0 degrees to 180 degrees, abduction to 0 degrees to 90 degrees, and elbow flexion to 0 degrees to 150 degrees, and the joint angles in the skeletal motion parameters are checked one by one; for facial expression parameter range checks, it is confirmed that all facial expression parameters are between 0 and 1; for lip shape key point position checks, it is verified whether the coordinates of each lip shape key point are within the predefined facial bounding box; for gaze direction angle checks, it is ensured that the eyeball pitch angle and yaw angle are both between -30 degrees and 30 degrees.

[0104] Finally, parameter trimming is performed in step S1053: for parameters that violate constraints found during the inspection, their values ​​are adjusted to the legal boundary values ​​of the corresponding constraints. For example, if the neck yaw angle exceeds 60 degrees, it is adjusted to 60 degrees; if the facial expression parameter is greater than 1, it is adjusted to 1; if the lip key point exceeds the bounding box, it is adjusted to the edge of the bounding box. After trimming, the final digital human driving parameters are output.

[0105] As an example, processing was performed based on a sequence of 3000 frames of motion parameters corresponding to the previous 30 seconds of audio. First, temporal smoothing was performed: the first 1500 frames corresponded to a slight nodding motion, and there were slight fluctuations in the neck pitch angle in the skeletal motion parameters. A 5-frame window Savitzky-Golay filter was used for smoothing, which eliminated the fluctuations and made the motion transition more coherent. The next 1500 frames contained actions such as smiling and leaning back. Before and after the 1500th frame, which switched from smiling to widening the eyes, cubic spline interpolation was used to avoid abrupt changes in facial expression parameters.

[0106] Next, physical constraint checks were performed: the neck pitch angle of the first 1500 frames was checked, and it was found that some frames reached 50 degrees, exceeding the constraint range of -45 degrees to 45 degrees; the torso bone angle corresponding to the body tilting backward was checked in the last 1500 frames, and the shoulder abduction reached 95 degrees in some frames, exceeding the range of 0 degrees to 90 degrees; at the same time, it was confirmed that the weight of the smile in the facial expression parameters was 0.384 to 0.792, all between 0 and 1, the key points of the mouth shape were all within the facial bounding box, and the angle of the gaze direction was kept at about 0 degrees, which met the constraint requirements.

[0107] Finally, parameter trimming is performed: the neck pitch angle is adjusted from 50 degrees to 45 degrees, the shoulder abduction angle is adjusted from 95 degrees to 90 degrees, and the remaining parameters remain unchanged. The final output is 3000 frames of digital human driving parameters, which can be directly used to control the digital human to perform smooth and natural movements. The above example is only one example of this application. In practical applications, the filter type, window size, constraint range, etc., can be adjusted according to requirements, and this application does not limit this.

[0108] This application effectively eliminates jitter and abrupt changes in motion parameters through this step, ensuring smooth and continuous motion. At the same time, through physical constraint verification and parameter adjustment, the digital human's motion conforms to the laws of human physiological activity. The final output digital human driving parameters can be directly used for actual driving, ensuring the naturalness and rationality of the digital human's motion.

[0109] Optionally, step S105 further includes: During smoothing, a sliding window caching mechanism is used to cache the processed action parameters of the most recent frames to avoid repeated smoothing calculations for overlapping frame segments. To reduce computational load, actions with a priority lower than a preset threshold in the action parameter sequence are updated at a reduced frequency.

[0110] In one specific implementation, step S105 adds two optimization strategies during the timing smoothing process. The core objective is to reduce redundant calculations, lower the overall computational load, and improve the efficiency of motion parameter processing without compromising the smoothing effect and the accuracy of core actions. Specifically, a sliding window caching mechanism is first used to avoid redundant smoothing calculations in overlapping frame segments, and then a frequency reduction update is performed on low-priority motion parameters. These two strategies work together to optimize the computation process. One approach is the sliding window caching mechanism, designed to address the issue of redundant calculations for overlapping frames during temporal smoothing. Its core principle is to cache the smoothing results of already processed frames, performing calculations only on newly added frames. First, a cache pool with a capacity matching the smoothing window is preset. This cache pool stores motion parameters that have been smoothed in the most recent few frames. When processing the sequence of motion parameters, the smoothing window is advanced frame by frame. Each time it advances, the cache pool is checked to see if it contains the smoothing results for overlapping frames within the current window. If it does, the result is directly used, and smoothing calculations are performed only on unprocessed parameters in the newly entered window. If not, the initial smoothing calculation is performed on the entire window frame, and the result is stored in the cache pool. After processing each window, the cache pool is updated synchronously, removing the oldest frame results that exceed the window's range and storing the results of newly calculated frames, ensuring that the cache pool always retains processed parameters within the size of the most recent window. This mechanism covers motion parameters across all dimensions, including facial expressions, lip movements, skeletal motion, and gaze direction, calculating only newly added frames and avoiding redundant calculations on overlapping frames.

[0111] Another approach is to reduce the frequency of updates, designed for low-priority actions that do not require frame-by-frame updates. The core principle is to reduce the computational load of low-priority actions while ensuring the accuracy of core actions. First, a preset action priority threshold is established, and the priority of each action type is defined. The action parameter sequence is analyzed frame-by-frame to identify action parameters with a priority lower than the preset threshold in each frame. For these low-priority actions, a fixed frequency reduction update interval is set, and the parameters for these actions are smoothly calculated and updated, with the parameter results from the previous update frame being reused in the interval frames. During processing, if the intensity change of a low-priority action exceeds the preset threshold, the frequency reduction is temporarily canceled, and frame-by-frame updates are resumed to ensure that action changes are accurately captured. Core high-priority actions continue to be smoothly updated frame-by-frame, without affecting the presentation of core actions.

[0112] This optional solution reduces the amount of redundant calculations for overlapping frames through a sliding window caching mechanism and reduces the overall computational load by updating low-priority actions at lower frequencies. While ensuring the accuracy of core actions and the smoothness of actions, it significantly improves the efficiency of action parameter processing and is suitable for the real-time processing needs of low-computing-power devices.

[0113] Optionally, the method further includes: The temporal smoothing and physical constraint checking process of the current frame is asynchronously and parallelized with the multimodal feature extraction process of the next frame.

[0114] The core objective of this alternative solution is to shorten the overall processing time by executing tasks in parallel, thereby further improving the real-time performance of digital human motion generation, especially to meet the high-efficiency operation requirements of low-computing-power devices. Its core logic is to break the serial mode of "processing all steps in a single frame before starting the next frame," allowing the post-processing steps of the current frame to overlap with the pre-feature extraction steps of the next frame in time, thus proceeding in parallel.

[0115] In one specific implementation, this solution achieves parallel processing by splitting tasks into independent ones, allocating dedicated resources, and establishing an asynchronous mechanism. First, two independent tasks are separated: temporal smoothing and physical constraint checking of the current frame, and multimodal feature extraction of the next frame. Different processing cores on the device are allocated to each task to avoid resource contention. The original audio and text are pre-split and cached frame by frame. Once the motion parameter mapping for the current frame is completed, its subsequent processing tasks are immediately initiated, simultaneously triggering the feature extraction task for the next frame, without waiting for the previous task to finish. Cached data and results are marked with frame sequence numbers to ensure proper frame order. If a task times out, subsequent triggering is paused to avoid data accumulation and ensure orderly and efficient parallel processing.

[0116] The foregoing has provided a detailed description of a multi-stage semantic emotion-driven digital human action generation method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A multi-stage semantic emotion-driven digital human action generation method, characterized in that, include: Multimodal preprocessing is performed on the input audio and text to extract audio features and text semantic features, and time alignment and feature fusion are performed to obtain a multimodal feature sequence; Based on the multimodal feature sequence, semantic intent recognition and sentiment analysis are performed to obtain a semantic intent sequence and a sentiment curve. The sentiment curve includes a sentiment category sequence, a sentiment intensity sequence, and a sentiment change trend. Based on the semantic intent sequence and emotion curve, a global action script is generated through preset action mapping rules. The global action script includes action type, action intensity, duration, and priority. The global action script is encoded into a feature vector and input into a lightweight neural network for action parameter mapping to obtain an action parameter sequence, which includes facial expression parameters, lip shape parameters, skeletal action parameters, and gaze direction parameters. The sequence of motion parameters is subjected to temporal smoothing and physical constraint checks, and the final digital human driving parameters are output.

2. The method according to claim 1, characterized in that, The semantic intent recognition and sentiment analysis based on the multimodal feature sequence, resulting in a semantic intent sequence and a sentiment curve, includes: The multimodal feature sequence is input into the semantic intent recognition model to obtain the probability distribution of the intent category for each time frame, and the semantic intent sequence is determined based on the probability distribution. The semantic intent recognition model is a bidirectional long short-term memory network. The multimodal feature sequence is input into the sentiment analysis model to obtain the probability distribution of sentiment category and sentiment intensity value of each time frame, forming sentiment category sequence and sentiment intensity sequence. The sentiment analysis model is a hybrid network of convolutional neural network and long short-term memory network. The emotion category sequence and the emotion intensity sequence are smoothed and optimized to generate an emotion curve, which includes the emotion category sequence, the emotion intensity sequence, and the emotion change trend.

3. The method according to claim 1, characterized in that, The step of generating a global action script based on the semantic intent sequence and emotion curve using preset action mapping rules includes: Based on the intent category in the semantic intent sequence and the emotion category in the emotion curve, a predefined action mapping rule table is queried to determine the corresponding action type; The action intensity of the action type is calculated based on the emotion intensity in the emotion curve and the intention weight corresponding to the intention category; The duration of the action type is calculated based on the duration characteristics of the emotion in the emotion curve and the length of the current semantic unit. Based on the action type, action intensity, action duration, and preset priority parameters, a preliminary action script is generated; The initial action script is subjected to action conflict detection, and the conflicts are eliminated according to a preset resolution strategy to generate the final global action script.

4. The method according to claim 3, characterized in that, The step of detecting action conflicts in the preliminary action script and eliminating conflicts according to a preset resolution strategy to generate the final global action script includes: Detect whether there are conflicting combinations of action types in the preliminary action script; When a conflicting combination of action types is detected, it is processed according to a preset resolution strategy. The resolution strategy includes: prioritizing the retention of actions with higher priority; if the priorities are the same, retaining the action with greater intensity; if both the priority and the intensity are the same, retaining the action with longer duration.

5. The method according to claim 1, characterized in that, The process involves multimodal preprocessing of the input audio and text, extracting audio features and text semantic features, and performing time alignment and feature fusion to obtain a multimodal feature sequence, including: Time-frequency analysis and feature extraction are performed on the input audio to obtain an audio feature vector sequence containing Mel spectrum features, pitch features, and energy features; The input text is encoded using a pre-trained language model to obtain a text semantic vector; The text semantic vector is expanded into a text feature sequence aligned with the audio feature vector sequence in the time dimension, and the audio feature vector sequence and the text feature sequence are weighted and concatenated to generate the multimodal feature sequence.

6. The method according to claim 1, characterized in that, The step of encoding the global action script into a feature vector and inputting it into a lightweight neural network for action parameter mapping to obtain an action parameter sequence includes: The action type, action intensity, action duration and priority parameters contained in each time frame of the global action script are uniformly encoded to obtain the corresponding action script feature vector. The action script feature vector sequence is input into a lightweight neural network, which is used to map high-level action features to low-level driving parameters. The lightweight neural network maps and outputs a sequence of motion parameters for each time frame, including facial expression parameters, lip shape parameters, skeletal motion parameters, and gaze direction parameters.

7. The method according to claim 6, characterized in that, The lightweight neural network is a lightweight structure containing multiple fully connected layers with fewer than 1 million parameters. It employs optimization strategies to accelerate inference, including INT8 quantization, operator fusion, and batch processing.

8. The method according to claim 1, characterized in that, The step of performing temporal smoothing and physical constraint checks on the motion parameter sequence to output the final digital human driving parameters includes: The motion parameter sequence is smoothed dimension by dimension using a temporal smoothing filter to obtain a smoothed motion parameter sequence. The smoothed motion parameter sequence is subjected to physical constraint checks, which include: joint angle limitation checks, facial expression parameter range checks, lip shape key point position checks, and gaze direction angle checks. Based on the inspection results, parameter values ​​that violate physical constraints in the smoothed motion parameter sequence are trimmed to the preset legal range, and the digital human driving parameters are output.

9. The method according to claim 8, characterized in that, Also includes: During smoothing, a sliding window caching mechanism is used to cache the processed action parameters of the most recent frames to avoid repeated smoothing calculations for overlapping frame segments. To reduce computational load, actions with a priority lower than a preset threshold in the action parameter sequence are updated at a reduced frequency.

10. The method according to claim 1, characterized in that, Also includes: The temporal smoothing and physical constraint checking process of the current frame is asynchronously and parallelized with the multimodal feature extraction process of the next frame.

Images