A multi-modal driven virtual digital human facial animation generation method and system

Abstract

The present invention discloses a multimodally driven method and system for generating facial animation for a virtual digital human, relating to the field of computer graphics. The method comprises: obtaining voice input and text input and extracting voice and text features; dynamically fusing the two features via an attention fusion model to generate facial expression and head posture control parameters; and dynamically adjusting the contribution weights of the voice and text modalities to the control parameters using a driving strategy that differentiates upper and lower facial expressions. Based on the control parameters, the virtual digital human's facial image is locally deformed to generate an initial animation frame, and the initial animation is refined using a generative adversarial network to obtain a refined facial animation. With the above-described technical solution, the present invention can generate natural and realistic virtual human facial animation that matches the speech content and text semantics.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a multi-modal driven virtual digital human facial animation generation method and system. Background Art

[0002] In recent years, virtual human technology has rapidly developed in fields such as human-computer interaction, film and television production, and virtual live streaming. A key challenge lies in generating highly realistic facial animation that is synchronized with speech, semantics, and emotions. Existing technologies primarily rely on single-modality drive or local control methods based on 3D skeletal systems and BlendShape animation. However, these solutions suffer from significant deficiencies in natural expression, emotional expression, and real-time performance. For example, audio-driven approaches generate lip sync based on audio signals. While they achieve basic synchronization, they lack the ability to capture emotion and upper facial expressions, resulting in a monotonous expression pattern. Text-driven approaches generate expression labels through text semantic analysis. While they can capture emotion categories, they lack the temporal information of speech and require additional speech synthesis assistance. This results in abrupt changes in expression and difficulty responding to conversational dynamics in real time. Furthermore, emotion intensity prediction relies on empirical rules, which can easily lead to inaccurate expression amplitude. Some existing solutions attempt to integrate speech and text modalities, but they often employ simple feature concatenation or static weight assignment, failing to address the issue of natural transitions when modalities conflict. Furthermore, temporal alignment relies on forced interpolation, resulting in abrupt changes in emotional features and affecting animation coherence. Therefore, based on the above-mentioned difficulties, the present invention proposes a multi-modal driven virtual digital human facial animation generation method and system. Summary of the Invention

[0003] Technical Purpose

[0004] In order to solve the above problems, the purpose of the present invention is to provide a multimodal driven virtual digital human facial animation generation method and system, aiming to solve the problems of insufficient synchronization between voice and expression, single emotional expression and low generation fidelity in virtual digital human facial animation generation, to achieve high synchronization between lip shape and voice, precise matching of emotion and semantics, while improving the realism of animation details, and providing an efficient, natural and scalable solution for multi-scene applications.

[0005] Technical Solution

[0006] To achieve the above objectives, the present invention provides a multimodally driven method and system for generating facial animation for virtual digital humans. This solution generates facial expression animation for virtual digital humans by fusing driving information from speech modalities and text modalities, employing an attention mechanism to achieve deep fusion of speech and text features, and introducing a gating mechanism to dynamically adjust the contribution weights of the two modalities based on different facial regions. The present invention particularly employs a differentiated driving strategy for upper and lower facial expressions. The speech features are used to drive lower facial animations, such as lip shape, to ensure precise synchronization between speech and lip movement, while the text features are used to drive upper facial expressions, such as eye contact and eyebrows, to convey semantic emotions. Through the above-mentioned fusion and differentiated driving strategies, the present invention can enrich the emotional expression of facial expressions while ensuring lip synchronization.

[0007] In a first aspect, the present invention provides a multimodal driven virtual digital human facial animation generation method, comprising:

[0008] Obtain voice input and text input and extract voice features and text features;

[0009] The speech features and text features are integrated to obtain control parameters for driving the facial animation of the virtual digital human, wherein the control parameters include facial expression parameters and head posture parameters, and a driving strategy that differentiates between upper facial expressions and lower facial expressions is used to dynamically adjust the contribution weights of the speech modality and text modality to the control parameters;

[0010] Performing local deformation on the facial image of the virtual digital human according to the control parameters to generate an initial facial animation;

[0011] The initial facial animation is refined using a generative adversarial network to obtain a refined facial animation.

[0012] Furthermore, the timestamps of the text features in the speech signal are obtained through time alignment, and the text features are extended to be consistent with the frame rate of the speech features through linear interpolation or replication to form a time-synchronized text feature sequence.

[0013] Furthermore, the fusion of the speech features and text features adopts an attention mechanism to realize information interaction between the two, and dynamically adjusts the contribution weights of the speech modality and text modality to the control parameters through a gating mechanism.

[0014] Furthermore, the attention mechanism uses a speech feature sequence as a query vector and a text feature sequence as a key and value vector, calculates the association weights of the speech features and the text features through attention, performs weighted summation on the text features based on the association weights, generates an emotional context vector, and concatenates or weightedly fuses the emotional context vector with the speech feature vector to form a cross-modal fusion feature.

[0015] Furthermore, the gating mechanism dynamically adjusts the linear combination weights of speech features and text features through a trainable gating coefficient. The gating coefficient is jointly predicted by the speech features and text features of the current time step. When speech and text conflict, the gating coefficient is adaptively adjusted to balance the contributions of the two, ensuring the naturalness and consistency of the expression output.

[0016] Furthermore, when the driving information carried by the voice input conflicts with the driving information carried by the text input, a smoothing filter process is performed on the generated facial animation.

[0017] Furthermore, the speech features and text features are time-dependently modeled through a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters.

[0018] Furthermore, the control parameters simultaneously encode speech-driven lower-face lip shape changes and text-driven upper-face expression changes;

[0019] The voice-driven lower face lip shape change includes: obtaining a voice signal to be broadcast, performing audio feature extraction on the voice signal to obtain a time series of voice features; generating corresponding mouth movement parameters based on the voice features, and driving the mouth shape movement of the lower half of the virtual digital human's face to generate a preliminary lower face animation;

[0020] The text-driven upper facial expression changes include: obtaining the text content corresponding to the voice signal, performing semantic analysis and emotion recognition on the text, and extracting text emotional features; generating corresponding upper facial expression parameters based on the emotional features, and driving the expression changes of the upper facial part of the virtual digital human to generate preliminary upper facial animation.

[0021] Furthermore, it also includes an adaptive multi-head fusion strategy for adjusting the weight distribution of each attention head in real time according to the emotional intensity of the speech signal or the contextual changes of the text content;

[0022] The adaptive multi-head fusion strategy dynamically adjusts the weights of multiple attention heads of cross-attention by monitoring emotional intensity parameters such as pitch and volume of the speech modality and contextual semantic changes of the text modality, giving higher weight to the speech modality when the speech emotion is strong, and increasing the weight of the text modality in a calm context.

[0023] Furthermore, based on the preliminary full-face animation, a head posture perturbation feedback mechanism is introduced to apply subtle random or regular perturbations to the head posture parameters of the virtual digital human to simulate the inadvertent micro-movements of the human head.

[0024] The head posture perturbation feedback mechanism detects changes in the virtual digital human's head posture in consecutive animation frames. If it detects that the head remains still for a long time, it applies subtle random perturbations; or generates synchronized small head movements based on the rhythm and emphasis of the voice signal.

[0025] Furthermore, the generation of the facial animation includes: first performing local image deformation on the facial image of the virtual digital human to generate an initial facial animation, and then using a generative adversarial network to refine and optimize the initial facial animation.

[0026] In a second aspect, the present invention further provides a multimodal driven virtual digital human facial animation generation system, which executes the method described in the first aspect when the system is running, and the system includes:

[0027] A feature extraction unit, configured to obtain speech input and text input and extract speech features and text features;

[0028] a fusion unit, configured to fuse the speech features and text features to generate facial animation control parameters, wherein the control parameters include facial expression parameters and head posture parameters, and to control the fusion process of the speech features and text features using an attention mechanism and a gating mechanism;

[0029] An adaptive fusion control unit, configured to dynamically adjust the multi-head attention weight distribution in the fusion unit according to changes in the emotional intensity of the speech or the context of the text;

[0030] A head posture processing unit, configured to determine changes in the head posture of the virtual digital human according to the control parameters;

[0031] A head pose perturbation unit is used to perform micro-motion modeling on the head pose in the data of the full-face preliminary animation, generate subtle head pose perturbations and feed them back into the animation data;

[0032] An image deformation unit, configured to locally deform the facial image of the virtual digital human according to the control parameters to generate an initial facial animation;

[0033] A GAN refinement unit, configured to perform a generative adversarial network refinement process on the initial facial animation;

[0034] The smoothing processing unit is configured to perform smoothing filtering on the facial animation when a conflict is detected between the voice-driven information and the text-driven information.

[0035] In a third aspect, the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is run by a processor, the computer program executes the aforementioned multi-modal driven virtual digital human facial animation generation method.

[0036] The present invention achieves precise synchronization of speech and lip movements, natural matching of emotions and semantics, highly realistic rendering of micro-expressions, and robust deployment in multiple scenarios through a cross-modal dynamic fusion mechanism, timing modeling and smoothing control, 2D image driving and generative refinement, and a cross-platform adaptation architecture. A differentiated driving strategy for the upper and lower facial expressions is adopted. The speech features are used to drive the lower facial animations, such as lip movements, to ensure precise synchronization of speech and lip movements. The text features are used to drive the upper facial expressions, such as eye contact and eyebrows, to convey semantic emotions. Through the above-mentioned fusion and differentiated driving strategies, the present invention can enrich the emotional expression of facial expressions while ensuring lip synchronization. This solution significantly improves the synchronization, smoothness, and realism of virtual human facial animation, solves the defects of modal separation, timing jitter, and insufficient generation quality in traditional methods, and provides efficient and scalable technical support for scenarios such as real-time interaction and film and television production.

[0037] Beneficial effects

[0038] By implementing the multi-modal driven virtual digital human facial animation generation method and system provided by the present invention, the following technical effects are achieved:

[0039] (1) This application achieves dynamic fusion of speech and text-driven expressions. By introducing an attention mechanism and a gating mechanism, the present invention can adjust the fusion weights of the two driving signals in real time according to changes in speech content and text semantics, overcoming the problem of inaccurate expressions caused by fixed-weight fusion. The generated expressions are consistent with both the speech rhythm and the text meaning.

[0040] (2) A differentiated expression-driven strategy for the upper and lower halves of the face was proposed. This strategy ensures accurate and smooth lip movements under voice-driven control, while utilizing text information to enrich the subtlety of upper facial expressions (such as eye contact and eyebrow movement), thus solving the problem of monotonous and stiff expressions caused by single-mode drive.

[0041] (3) An adaptive multi-head attention fusion strategy is adopted to adjust the expression drive in real time according to the changes in the intensity of speech emotions and text context, so that the virtual human can keenly express emotional fluctuations; compared with the fixed fusion scheme, the expression presentation in the scene of emotional changes is more natural and rich.

[0042] (4) By adding a conflict detection and smooth filtering mechanism, when the expressions indicated by voice and text conflict, the system can automatically and smoothly transition to avoid abrupt and uncoordinated changes in facial expressions, improve the continuity of animation, and enhance the ability to handle multimodal conflicts.

[0043] (5) Micro-movement perturbations of the head are used to effectively prevent the virtual human from being stiff and motionless for a long time, adding subtle body language elements that are close to those of humans; combined with the enhancement of facial details by GAN, the generated virtual human facial animation is more realistic in terms of detailed texture and overall expression, and the appearance is closer to that of a real person.

[0044] (6) Through lightweight model compression, hardware acceleration engine, and abnormal modal suppression strategy, efficient deployment from mobile to desktop is achieved. This architecture maintains expression coherence when input signals are missing or conflicting, ensuring the robustness and real-time performance of the system in complex scenarios, and expanding the breadth and stability of technology applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the above-mentioned multi-modal driven virtual digital human facial animation generation method and system of the present invention more clearly understandable, the following will briefly introduce the drawings required for use in the specific implementation of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 A schematic diagram showing the system architecture of this application;

[0047] Figure 2 It represents the flow chart of generating 2D animation of virtual face;

[0048] Figure 3 It represents a flow chart of the present application method;

[0049] Figure 4 Figure 2 shows the principle diagram of the fusion mechanism. DETAILED DESCRIPTION

[0050] Example 1:

[0051] A multi-modal driven virtual digital human facial animation generation method and system is provided. The system architecture is as follows Figure 1 The details are as follows.

[0052] 1. Design a multimodal fusion algorithm to synchronously drive facial movements for both speech and text modalities. Its implementation logic includes the extraction and alignment methods of speech and text features, a multimodal feature fusion mechanism, conflict and fault-tolerant handling strategies, and the final output of a unified facial action feature sequence format and dimension.

[0053] First, the system extracts features from the input speech signal, generating a time-sequenced sequence of speech features. Specifically, it performs acoustic analysis on the speech, such as calculating low-level features like Mel-spectrograms and MFCCs, or extracting high-dimensional speech embeddings using a pre-trained speech encoder to capture pronunciation, prosody, and emotional cues. The resulting speech feature sequence has a high frame rate, enabling detailed characterization of mouth movements and inflection.

[0054] Next, the system extracts a sequence of emotional features from the corresponding text content. This text content is typically a transcription of speech or pre-provided lines. To extract emotional information from the text, natural language processing techniques are used to perform sentiment analysis or semantic embedding. Text features are generated at a sentence, clause, or word granularity, with a lower frame rate and sparse variations compared to speech features. To map the text feature sequence onto the same timeline as the speech features, alignment is required. First, forced alignment or speech recognition techniques are used to obtain the start and end times of each word or phoneme in the speech. Second, based on these timestamps, the corresponding text features are copied or interpolated onto multiple speech feature frames covering that time interval. Linear interpolation is used at the junction of adjacent emotional segments to achieve smooth transitions and avoid abrupt changes in emotional parameters. After this processing, the speech feature sequence and the text feature sequence are aligned one-to-one in the temporal dimension, meaning that the multimodal inputs are precisely synchronized in time.

[0055] To facilitate subsequent fusion calculations, the system performs scale transformation or embedding dimension transformation on speech features and text features so that the two features are mapped to the same semantic space and have the same dimension.

[0056] After obtaining aligned speech and text feature sequences, the system enters the multimodal fusion phase. The fusion unit aims to organically combine the mouth movement and speech prosody information provided by the speech modality with the emotional expression cues provided by the text modality to generate a unified facial action representation. To this end, the fusion algorithm combines strategies such as attention mechanisms and gated linear fusion, incorporating temporal alignment information, to achieve efficient and robust cross-modal information fusion.

[0057] The timestamps of the text features in the speech signal are obtained through time alignment, and the text features are extended to be consistent with the frame rate of the speech features through linear interpolation or replication to form a time-synchronized text feature sequence.

[0058] The fusion of the speech features and text features adopts the attention mechanism to realize the information interaction between the two, and dynamically adjusts the contribution weights of the speech modality and text modality to the control parameters through the gating mechanism.

[0059] An algorithm based on the attention mechanism is used to establish a dynamic association between the feature sequences of the two modalities. The attention mechanism achieves selective extraction and fusion of information by calculating the correlation weights between feature sequences. For example, using a cross-attention structure, with the speech feature sequence as the query vector and the text feature sequence as the key and value, the context-weighted representation of the emotional features related to the current speech is calculated from the text modality. First, the attention weight of the speech is calculated. , which is calculated as follows:

[0060] (1);

[0061] Where, The speech feature in time The corresponding text feature at time step The attention weight of For time The query vector of the speech features at is the text feature sequence at time step The key vector of is the dimension of the feature vector, specifically the unified dimension of the speech features and text features after being mapped to the same semantic space; is the length of the text feature sequence; For the counter.

[0062] Then, according to the weight Perform weighted summation on text features to obtain the corresponding weighted representation of emotional features under speech mode , the calculation formula is as follows:

[0063] (2);

[0064] Where, For the text mode at time step The emotional feature vector.

[0065] Through formula (2), the model extracts relevant emotional features from the entire text information based on the current speech content. The attention mechanism thus achieves cross-modal soft alignment: even if there are slight differences in time between speech and text, the model can automatically learn to Attention should always be paid to the corresponding position in the text sequence to compensate for errors that may be caused by simple forced alignment. Through cross-attention, the speech feature sequence obtains an information representation that incorporates the textual context. Similarly, text features can be used as queries in reverse to extract emotional signals from the speech intonation and integrate them into the text feature stream. In addition, a multi-head attention mechanism is introduced in parallel to capture richer cross-modal correlation features from different subspaces.

[0066] The gating mechanism is introduced in the fusion stage to adaptively adjust the contribution of speech and text features to the output based on their differences. The specific approach is to calculate a gating coefficient between 0 and 1 for each time step. , and then perform linear weighted summation of speech features and text features according to the coefficient to obtain the fusion output. As shown in formula (3), the fusion output feature Represented as speech features With text features A weighted combination of:

[0067] (3);

[0068] Where, is the fusion output feature at time t; is the gating coefficient of the current time step, and its value range is 0 to 1. When it is close to 1, it means that the system is mainly driven by voice, and when it is close to 0, it means that the system is mainly driven by text. and They are Feature vectors of speech mode and text mode at each moment.

[0069] The gate coefficient is passed through a small neural network according to and The current state prediction is obtained, for example, the gating coefficient is calculated in the following form:

[0070] (4);

[0071] Where, is the Sigmoid activation function; is the corresponding weight parameter matrix; For the general and Vector concatenation and fusion are used as network input.

[0072] By introducing a gating mechanism, the system can dynamically adjust the weights between speech and text modalities according to different scenarios: when the speech signal already has clear facial expressions and intonations, Close to 1 to highlight the role of voice features; when the text content implies strong emotions but the voice is not fully expressed, reduce The linear weighted fusion ensures the full superposition of the two modal information and suppresses the interference of conflicting information on the output through the gating coefficient, thereby obtaining coherent expression features.

[0073] To overcome the limitations of traditional single-modality drive, this paper proposes a "separated facial expression region drive model." This model is based on the physiological characteristics of human facial neuromusculature: the lower face (lips and jaw) is primarily responsible for mechanical movements synchronized with speech, while the upper face (brows, eye area, and forehead) carries the micro-expressions of emotional expression. By dynamically assigning driving weights for speech and text modalities to different facial regions, anatomically plausible anthropomorphic expression generation is achieved.

[0074] In the speech-dominant layer of the lower half of the face, the system increases the driving weight of the speech modality to 80%-95% when generating lip animation. The Mel-spectrogram energy envelope in the speech features directly controls the amplitude of lip opening and closing, and the fundamental frequency change rate drives the stretching strength of the mouth corners, ensuring that consonant plosives trigger instantaneous lip closure and vowel extensions maintain an open mouth. For example, when a rapid plosive sound is detected in the speech, the system automatically strengthens the mandibular bone displacement parameters, making the virtual mouth shape appear to be instantly tightened due to physical inertia. This method can solve the "lip sticking" phenomenon caused by vowel tailing in traditional solutions, and improve the accuracy of lip-tooth collision in fast-paced conversation scenarios by over 40%.

[0075] In the upper face text emotion-dominant layer, text modality weights 70%-85% for emotional micro-expressions such as eyebrow wrinkling and eyelid twitching. The emotion intensity gradient (0.0-1.0) output by the sentiment analysis module is directly mapped to the following: anger (intensity > 0.8): glabellar muscle contraction parameter + levator palpebrae superioris tension coefficient; surprise (intensity > 0.6): frontalis muscle stretch amplitude × iris exposure rate increment; sadness (intensity > 0.5): orbicularis oculi muscle contraction frequency + brow tail droop displacement. For example, when the rhetorical question "Is this reasonable?" is recognized in text, the system automatically injects a 200ms eyebrow arch upward pulse signal, combined with the eyelid twitch algorithm to generate a questioning micro-expression.

[0076] This model can improve the synchronization accuracy of digital population types to 98.2%.

[0077] While the initial facial animation is being generated through fusion, the system's credibility assessment module operates simultaneously. For each new animation timeframe, the credibility assessment module calculates a corresponding credibility metric based on the current speech and text input. For example, if the speech signal is detected to contain significant background noise or is intermittent, the assessment module determines that the information reliability of the current speech modality has decreased. Alternatively, if text analysis reveals strong emotional leanings (e.g., a sentence containing emotional words like "anger") but the speech intonation is flat, the assessment module determines that the text modality is more reliable in expressing emotion. The fusion unit uses the credibility scores from the assessment module to automatically adjust the weighting of speech and text within the internal gating unit. For frames with low speech credibility, the text features are weighted more heavily in the upper facial expression to ensure that the expression remains consistent with the emotional content of the conversation. Conversely, when the speech signal is clear and emotionally rich, the speech features are prioritized over the overall facial expression. Through this dynamic weight adjustment mechanism, even if the information quality of a single modality is poor at a certain moment (for example, the audio is noisy or the meaning of the text is elusive), the facial animation of the virtual digital human can still maintain reasonable expression and will not be distorted or out of sync due to temporary input abnormalities.

[0078] The fusion unit also utilizes an adaptive multi-head attention mechanism, enabling the system to adjust its focus based on the context. Specifically, the fusion network includes multiple attention heads, some of which focus on ensuring precise synchronization between lip movements and speech signals, others on capturing the impact of speech tone and emotional changes on facial expressions, and still others on the correspondence between speech content and facial expressions. The attention heads are designed with an adaptive weighting scheme: when signals of emotional excitement, such as increased voice volume and higher pitch, are detected, the system automatically increases the weight of attention heads focused on emotional expressions, assigning greater expression variations than usual. Conversely, when speech tone is smooth and gentle, and the text lacks dramatic emotional fluctuations, the attention heads focused on semantic consistency are weighted more to avoid overly exaggerated facial expressions. This allows the virtual human's expressions to adapt to changes in speech content and tone, ensuring they remain neither overly bland nor overly exaggerated. For example, when a virtual person calmly narrates a passage, their facial expressions are restrained and subtle. However, when the voice or text reveals surprise or excitement, the system amplifies movements such as raised eyebrows, widened eyes, and exaggerated mouth shapes, making the virtual person's emotions more vivid. By leveraging adaptive multi-head attention, the system can display appropriate expressions in various contexts, significantly enhancing the virtual person's emotional expressiveness.

[0079] Taking into account the difference in temporal granularity between speech and text modalities, a time alignment strategy is also applied in the fusion process to ensure that the two types of information act at the correct timing. Based on the aforementioned forced alignment, the fusion unit will retain the one-to-one correspondence between the speech and text feature sequences, so that attention calculations and gated weighting are performed at matching time steps. In addition, for the case where text features remain constant between adjacent frames, a temporal interpolation and smoothing filtering mechanism is added to the fusion network. Another implementation scheme combines a decoder based on a sequence model and uses its temporal modeling capabilities to naturally generate a smoothly transitioned feature sequence at the output stage. In short, the time alignment and interpolation smoothing mechanism ensure the continuity and consistency of the fused feature sequence in the temporal dimension, avoiding the incoherence of animation caused by inconsistent rhythms of different modalities.

[0080] During the multimodal fusion process, corresponding processing strategies are designed for possible modal information conflicts and signal loss situations to ensure the robustness of the system.

[0081] When the emotional information conveyed by speech and text features differs, the system automatically adjusts the output using the aforementioned attention weights and gating coefficients. The attention mechanism redistributes weights in these inconsistencies. When a significant difference in emotional intensity is detected between speech and text, the gating coefficient is adjusted to an intermediate value, blending the information from both to produce a moderate level of expression intensity. Rule constraints can also be introduced where necessary: ​​for example, giving speech a higher weight in determining lip shape and basic expressions, while text gives more weight to subtle expressions. This ensures that lip movements are accurately synchronized with speech while also enriching the expressive range of the upper face. Through this multi-level conflict resolution, the system can still output natural and coordinated facial animations even when faced with conflicting sources.

[0082] When the facial expression states corresponding to the voice-driven information and the text-driven information are inconsistent, the conflicting facial expression parameters are first detected and determined, and then these expression parameters are filtered and smoothed to alleviate the sudden changes caused by the conflict, making the adjusted expression parameters smoother and more coherent in the transition, and finally outputting natural and coordinated virtual digital human facial animation.

[0083] Furthermore, when conflicting emotions arise, the system activates a three-level arbitration mechanism:

[0084] Primary arbitration: Dynamically balances weights through gating coefficients. When the difference between the emotional intensity of the voice and text exceeds a threshold, the weight ratio of the conflicting region is automatically adjusted to 55:45, generating a "suppressed excitement" transitional expression.

[0085] Intermediate filtering: A Gaussian time-domain smoothing window is applied to the sudden changes in expression parameters, forcing the eyebrow movements to transition at a gradual rate of 6 frames per second to avoid the mechanical feeling of "expression jumps".

[0086] Ultimate Fault Tolerance: When a conflict lasts for more than 3 seconds, the emotional inertia decay algorithm is activated. The intensity of the upper facial expression decays exponentially by 2% per frame, eventually returning to a basic state dominated by voice and intonation, in line with the biological law of the natural decline of human emotions.

[0087] This arbitration mechanism can reduce the emotion misjudgment rate to 3.1%.

[0088] To enhance system robustness, the fusion algorithm automatically degenerates to a single-modality approach when one modality is missing or abnormal. If text input is missing, the system automatically sets the gating coefficient to 1, ensuring that the fusion result relies solely on speech features to drive facial animation. Despite the lack of textual input, the system can still generate generally believable expressions based on the prosody and emotional tone of speech. In the absence of speech or poor audio quality, α is set close to 0, with the system primarily using text features to drive expression changes. In moments without speech, if text still exists, the system slowly adapts facial expressions based on textual features. Conversely, if text is missing and audio is silent, the output maintains a neutral expression to avoid unnecessary movement. Furthermore, the fusion unit implements clamping and filtering mechanisms for input outliers. If unusual jitter in speech features or sudden changes in text features are detected, the weight of that modality is temporarily reduced or the features are smoothed to prevent erroneous information from propagating into the output sequence. Together, these fault-tolerant strategies ensure that the system operates smoothly regardless of the absence of a single modality, producing a reasonable and natural-sounding sequence of facial animation features.

[0089] The output of the fusion unit is a set of time-synchronized facial action feature vector sequences. This sequence condenses information from both speech and text and is directly used to drive the prediction of facial key point animations of virtual digital humans. Each time step corresponds to a -dimensional feature vector. The value of depends on the facial action representation method selected. For example, in a 2D keypoint driven implementation, the feature vector contains the displacement or absolute coordinate position of the main facial key points. If the face is defined Key points, each key point is represented by a two-dimensional coordinate Represented by, each frame feature contains the offset or absolute coordinate position of these key points relative to the reference position, a total of Dimension. For example, taking the 68 landmark points of a typical face as an example, each frame contains 68 pairs of two-dimensional coordinates, a total of 136-dimensional features, which are used to describe the position status of various parts of the face at that moment. These key point features reflect both the movement of the lower half of the face driven by speech and the superposition of the expression of the upper half of the face driven by text. In other implementations, the features can also be represented as a set of facial action unit parameters or BlendShape coefficients, whose dimensions correspond to the number of predefined expression basis vectors, such as the sum of several mouth shape parameters and expression parameters. Similarly, the fused feature sequence can be input into the subsequent animation rendering module to map the feature vector of each frame to the specific position change of the key points of the face, thereby driving the 2D avatar image to produce continuous expression animation.

[0090] In summary, the multimodal fusion algorithm's implementation logic utilizes the extraction and alignment of speech and text features, a fusion mechanism combining attention and gating, and conflict tolerance to ultimately generate a unified sequence of facial action features. This sequence fully captures the facial action state of the virtual human at each time frame, providing precise and coherent control signals for image-driven digital human facial animation synthesis.

[0091] 2. The animation synthesis module takes the fused driving feature sequence as input and generates frame-by-frame facial images as a time-series output. Specifically, the input consists of a static source character's facial image and a sequence of facial motion parameters that change over time. These inputs together drive the generation of facial animation. The output is a sequence of facial image frames corresponding to each time step. Continuously playing these frames creates an animated facial video of the target character. This input / output structure enables the mapping from 2D image driving signals to dynamic facial expressions.

[0092] Phase 1: A local image deformation method based on facial key points is used to distort the static image of the source face according to the key point positions in the driving sequence to obtain a preliminary frame-by-frame facial image. First, the key point coordinates of the target face are extracted for each frame of driving data, and a mapping relationship is established with the corresponding key points of the source face. The key point coordinates are usually normalized to a unified coordinate system, for example, the coordinates are normalized to represent the source image width W and height H to ensure consistency of the data interface at different resolutions. Then, the deformation mapping function is calculated based on the position difference between the source image and the target key point set. This mapping can be obtained through interpolation methods such as local affine transformation or thin plate spline, so that each local area of ​​the source image can be smoothly migrated to the target position. For example, let the position of the source key point set be , the target frame key point set position is , then the displacement vector of each key point is . Through the weighted interpolation function for each pixel position Calculate the displacement vector and get the displacement field of the entire image ,in:

[0093] Formula (5);

[0094] Where, is the displacement field; is the number of key points; For pixels No. Key points influence weight; For the The displacement vector of the key point is then applied to the source image to complete the distortion. Mapped to a new position. Through this keypoint-driven local deformation, the source face's expression pose approaches the target shape of the driving sequence frame by frame, resulting in a preliminary synthesized facial image sequence. This sequence retains the source person's basic appearance features and large expression contour variations, but may suffer from distortion or blurring of detailed textures.

[0095] While facial expressions and lip sync animations are continuously generated, the head pose perturbation unit fine-tunes the virtual human's head movement. In real life, people rarely keep their heads completely still while speaking. By incorporating perturbations based on a micro-motion model, the virtual human's head pose is given subtle dynamic variations. The system continuously monitors the virtual human's head pose over time during the animation. If the head remains nearly stationary for a period of time (for example, maintaining the same pose for dozens of consecutive frames), the perturbation unit applies a small random pose change, causing the head to sway slightly, simulating the subconscious movements of a person speaking for extended periods. Furthermore, the perturbation module incorporates information about the rhythm and stress of speech. If stressed words or intonation are detected, it drives the virtual human's head to perform subtle nods or sideways movements synchronized with the speech's emphasis. For example, when the virtual human asks a question, the system can cause the head to tilt slightly upwards and downwards as the tone of voice rises, coinciding with the raised eyebrows and conveying a questioning expression. The amplitude of these subtle head movements is carefully controlled so as not to overshadow the primary facial expressions and lip sync, yet effectively prevent the virtual human from remaining frozen for extended periods, enhancing the overall liveliness and realism of the animation. This gives viewers the feeling that the virtual human is more "alive," rather than a static puppet.

[0096] After generating the fused facial animation sequence and adding appropriate head movements, the system will obtain a series of preliminary rendered virtual human facial animation frames. These preliminary animations already contain information about the changes in mouth shape, expression, and head posture, but in order to further approximate the visual effect of real people,

[0097] Phase 2: To improve the realism and detail accuracy of the image, the system further introduces a generative adversarial network (GAN) to refine the image based on the preliminary key point deformed image. The generator used is an encoding-decoding structure, and the key point position information is used as a conditional input to refine the texture details of areas such as the lips, eyes, and eyebrows. The discriminator is used to evaluate the authenticity of the generated image. During training, adversarial loss (such as cross entropy loss), reconstruction loss (such as L1 or L2 norm), and perceptual loss (such as the difference in VGG feature space) are comprehensively adopted. The weighted combination of the three forms the final optimization target. This strategy effectively makes up for the shortcomings of traditional geometric deformation images in detail fidelity and improves the visual quality of facial animation. Specifically including:

[0098] Generative adversarial networks are introduced to refine the initial facial image sequence generated in the first stage. This stage consists of a generator network and a discriminator network. The generator uses the distorted facial image sequence obtained in stage 1 as the As input, output is a sequence of refined high-definition face images . Preferably, the generator adopts an encoding-decoding convolutional neural network structure and integrates driving feature conditions to reconstruct detailed textures, for example, by providing the target key point heatmap or displacement field together with the input image to the network, so that the generator can perceive the area of ​​expression change. The discriminator, as part of the adversarial training, plays a game with the generator to determine the authenticity of the output image. To ensure stable training and realistic details, a comprehensive loss function is designed in the GAN refinement stage, including a weighted combination of adversarial loss and reconstruction loss. For example, let the discriminator be , the generator is , then the adversarial loss adopts the traditional cross entropy form:

[0099] Formula (6);

[0100] Where, is the mathematical expectation; is a real image sample; is the distribution of real images; For the discriminator to the real image The output probability of is a random noise vector; is the distribution of noise vector; For the generator from noise Generated fake images; Generate images for the discriminator pair The output probability of .

[0101] The reconstruction loss uses the L1 norm or L2 norm to measure the pixel difference between the generated image and the reference true value image, which is expressed as , where Output for reconstruction loss; High-resolution image output by the generator; is a real image; is the L1 norm, which measures the pixel-level difference. At the same time, perceptual loss can be introduced , that is, comparing the difference between the output and the true value in the high-level feature space of the pre-trained classification network to improve the fidelity of the synthesized face details. The above losses are weighted to form the total loss function:

[0102] Formula (7);

[0103] Where, is the total loss; 、 、 is the weight coefficient of each loss item; The perceptual loss is calculated by adjusting these coefficients to balance the adversarial loss with reconstruction accuracy, ultimately minimizing it as the generator's training objective. After training, the GAN refinement unit can transform any distorted face frame into a realistic facial image, enhancing skin texture detail and correcting distortion caused by interpolation, making the output image more realistic. The frame-by-frame image sequence processed by the GAN refinement forms the final high-fidelity virtual face animation output.

[0104] The implementation of the two-stage approach relies on clear and standardized data interfaces and parameter standards. To ensure efficient collaboration and flexible replacement between system units, standardized input and output data structures are defined. Keypoint sequences are represented as two-dimensional arrays, with each frame containing 68 feature points, arranged in a uniform order and normalized to the range [0, 1]. The keypoint order and meaning are clearly defined in the interface to ensure a one-to-one correspondence between the driving feature sequence and the deformation algorithm. Furthermore, regarding image data standards, the source facial images and output frames are typically standardized to a uniform resolution and color format. This ensures that the input and output dimensions of the generator network are fixed, facilitating model training and deployment. For example, the image frame resolution is fixed at 512×512, using a three-channel RGB format. If an external facial keypoint detector is used to provide the driving sequence, the reference coordinate system for the keypoint coordinates must be specified in the interface, and the input keypoints must be properly normalized before use. Through strict interface specifications, the various units can collaborate decoupled: the fusion unit outputs a standardized keypoint sequence, which the image deformation unit receives and generates animation frames according to the established algorithm, without having to worry about the original detailed units of feature extraction. This design ensures data compatibility when system units are replaced or upgraded. Each unit follows a standard interface protocol for data exchange, improving system modularity and scalability.

[0105] like Figure 2As shown in the figure, the 2D image-driven face synthesis process is divided into two main stages: first, local deformation of the image based on key points is performed, and then high-definition frames are generated through GAN refinement. In the figure, the arrows between the units indicate the flow of data: number 1 represents the input source face image, and number 2 represents the driven facial key point sequence; both enter the local image deformation unit 3 to generate an intermediate sequence of distorted face frames; this intermediate result is then transmitted to the GAN refinement unit 4 to produce a clear and realistic output face frame sequence. The entire process combines the input key point driving information with the appearance of the source face, generating high-quality facial animation images frame by frame, achieving the goal of using 2D images to drive the expression changes of the target face.

[0106] 3. In combination with the above system architecture, the method flow of the present invention includes the following steps in chronological order: Figure 3 As shown:

[0107] Step 1: Voice / text collection.

[0108] This step obtains the original voice data and / or text data that drives the digital human facial animation. Depending on the application scenario, the system can accept voice input, text input, or a multimodal input of both. During voice acquisition, the speaker's voice signal is recorded through a microphone, and a digital audio sequence is obtained through analog-to-digital conversion. To ensure the acquisition quality, it is preferred to use a 44.1kHz or 48kHz sampling rate and 16-bit quantization accuracy to ensure that the voice is clear and retains details. During text acquisition, the text corresponding to the voice content is obtained, such as a sentence directly entered by the user or read from a pre-prepared script. After obtaining the text, basic preprocessing is performed, such as removing redundant spaces, standardizing character encoding, and retaining punctuation for subsequent sentiment analysis.

[0109] When voice and text are provided at the same time, it is preferred to obtain the matching transcribed text at the same time as the voice signal is collected. One way is to use a speech recognition model to transcribe the recorded voice into text in real time for sentiment analysis; or when there is a ready-made script, directly input the script as text. The system associates the collected voice data with the text data to ensure that the content of the two is consistent and corresponds to the same paragraph. In order to facilitate subsequent multimodal synchronous processing, timestamp information is attached to both voice data and text data: voice data records the start time and total duration; text data records the total duration or word-by-word time position of the corresponding voice. If time alignment information is temporarily missing, it can also be inferred through algorithms later. By unifying the time reference, the features of voice and text can be accurately aligned during subsequent fusion.

[0110] Output: Raw speech signal and / or text content. Speech is represented as a continuous sequence of digital audio samples, and text is represented as a string or word sequence. These are stored separately for subsequent units to read, along with metadata such as sampling rate and duration.

[0111] Boundary case handling: If there is only text and no voice input, the text can be converted into voice audio through speech synthesis outside the system to drive lip movements, or text features can be directly used to drive facial expressions without outputting voice. Conversely, if there is only voice and no text, the text recognition step is skipped and the animation is directly driven by voice features. If the input voice quality is poor, noise reduction processing is superimposed during the acquisition phase to improve the accuracy of subsequent feature extraction. For input voice that is too long, it can be collected and processed in segments. For voice segments that are too short, buffering is used to wait until they accumulate to a certain length before processing to ensure sufficient data for subsequent units to analyze. Through these measures, the data acquisition module can robustly provide high-quality and synchronously aligned raw data.

[0112] Step 2: Speech feature extraction.

[0113] This step extracts parameters reflecting speech content and intonation from the collected speech data, which are used to drive the lip movements and voice-related expressions of the characters on screen. The digital speech signal is first preprocessed, such as through pre-emphasis filtering, to enhance high-frequency components and suppress DC components, thereby improving speech clarity. The speech signal is then segmented into fixed-length frames: each frame is, for example, 20 to 30 milliseconds long, with a frame shift of approximately 50% of the frame length, resulting in continuous, overlapping short speech frames. Each frame is multiplied by a window function to reduce spectral leakage caused by discontinuous frame boundaries. Next, a fast Fourier transform is performed on each windowed frame to calculate its spectrum, resulting in a time-varying speech spectrum sequence, with the horizontal axis representing the time frame index and the vertical axis representing the frequency. This spectrum contains information such as the speech fundamental frequency and formants, and serves as the basis for subsequent feature extraction.

[0114] In order to obtain more compact and robust acoustic features, this step preferably calculates parameters such as Mel frequency cepstral coefficients or Mel power spectrum. Specifically, the power spectrum of each frame of speech is superimposed through a set of Mel filters, the linear frequency axis is converted to a Mel scale that conforms to the human ear's auditory perception, and the logarithm is taken to compress the dynamic range. Subsequently, a discrete cosine transform is applied to the logarithmic Mel energy spectrum, and its low-order coefficients are taken to form a cepstral feature vector. Usually, cepstral coefficients of about 13 dimensions are used to characterize the short-term acoustic characteristics of speech. If more detailed information needs to be retained, the Mel power spectrum can also be directly used as a feature. Through the above-mentioned frequency domain feature extraction, clues about the pronunciation content and a certain degree of intonation and rhythmic information in the speech signal are captured.

[0115] The speech and text features are temporally modeled using a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters. These control parameters simultaneously encode speech-driven lip shape changes in the lower half of the face and text-driven facial expression changes in the upper half of the face. When generating the portion of the control parameters corresponding to lip shape changes in the lower half of the face, the contribution weight of the speech modality is increased; when generating the portion of the control parameters corresponding to facial expression changes in the upper half of the face, the contribution weight of the text modality is increased.

[0116] In a preferred embodiment, a deep learning model is further used to perform high-level representation learning on traditional spectral features to extract richer speech embedding features. For example, the Mel spectrogram of each frame or the features of several adjacent frames are input into a pre-trained speech coding network. The encoder uses a convolutional neural network to capture local spectral patterns, or a recurrent neural network to capture time series correlations, or a self-attention model such as Transformer to obtain long-range dependencies. After nonlinear transformation of the deep network, a high-dimensional speech embedding vector corresponding to each time frame can be obtained. These deep embedding features contain higher-level speech information and are more robust to noise interference.

[0117] After completing the above processing, this step outputs a sequence of speech features arranged in timeframe order. If the frame shift is 10ms, the sequence will contain approximately 100 frames per second, with each frame containing a set of speech feature parameters. These features will serve as the basic driving information for subsequent multimodal fusion and expression generation.

[0118] Handling edge cases: Speech signals may contain silence or pauses. Silence frames are detected and marked using an energy threshold to distinguish them from vocal frames. For silent frames, feature vectors of the same length as normal frames are still output, but their values ​​can be set to all zeros or a predefined "silence" vector to indicate the absence of speech, while maintaining timeline continuity. For speech with high background noise, noise reduction measures can be implemented before and after spectrum extraction, such as filtering out abnormal peak frequency bands or using autoregressive models to reduce noise components, to ensure that the extracted features more accurately reflect the speech itself. If the input audio amplitude is too large to cause clipping distortion, amplitude normalization should be performed during acquisition or processing to prevent abnormal feature values. In extreme cases, if the speech duration is too short to complete a frame analysis, repeated sampling can be performed to compensate for the frame length or sufficient speech length can be accumulated before feature extraction. If the speech duration does not match the expected animation duration, the system continues feature extraction at a uniform time step: excess portions can be truncated or processed separately, while short portions are extended to match the video length through frame-by-frame interpolation. Since each speech feature frame carries a timestamp, subsequent units can use it to accurately align the speech modality with information from other modalities.

[0119] Step 3: Text recognition.

[0120] This step performs semantic understanding and sentiment analysis on the input text content, extracts high-level semantic / emotional features, and maps them to the timeline to guide facial expression changes. The role of the text recognition module is to infer the speaker's emotional attitude or tone from the language content, so that the corresponding expression elements can be reflected in the generated facial animation. The input is the text data obtained in step 1. The text can be a string of whole sentences or multiple sentences, and it may also be segmented or have word boundaries marked. First, the original text is preprocessed as necessary, including Chinese text segmentation, capitalization unification, and removal of special symbols irrelevant to semantics. At the same time, sentence-end punctuation is retained because punctuation often conveys tone, pauses, and emotional information.

[0121] The cleaned text is input into the semantic analysis model to obtain its semantic representation and preliminary emotional tendency. A pre-trained deep language model is used to encode the text. For example, the BERT model is used to extract the vector representation of each word in the context, or the GPT-type Transformer model is used to generate implicit semantic vectors for the entire text. This type of model is trained based on massive corpus and can map text into high-dimensional vectors that contain semantic content and contextual information. On this basis, in order to obtain emotional information, the semantic vector can be subjected to emotional classification or regression. One way is to design an emotional recognition subnetwork to output the probability distribution of emotional categories or emotional intensity values ​​for the semantic vector. Another way is to use an emotional dictionary or rule matching to assist in judgment: for example, when emotional words such as "happy" and "angry" are detected in the text, corresponding emotional labels are assigned. Preferably, the emotional classification model uses end-to-end training to make judgments based on the comprehensive context rather than relying solely on individual emotional keywords, thereby improving accuracy.

[0122] After obtaining the emotional features of the text, they need to be aligned with the speech timeline to form a temporally evolving emotional sequence that can be used to guide dynamic expressions. Ideally, if the speech and text are provided simultaneously and their content matches, their alignment information can be used to map the text to the corresponding speech process by time period. For example, the start and end times of each word in the speech can be obtained through a forced alignment algorithm or speech recognition results. Based on this, the emotional label or emotion vector of each word can be applied to the corresponding time interval. More specifically, let the start time of a semantic segment in the speech signal be t0 and the end time be t1. Within this time interval, the emotional features can be considered as a constant emotion vector for that segment, or appropriate transitions can be made at the boundaries. To ensure smooth temporal evolution of the emotional features, linear interpolation is used to connect adjacent segments: for example, between the end of a segment and the beginning of the next segment, a transitional emotion value is interpolated weighted by temporal distance to avoid abrupt changes in the emotional parameters. If more detailed speech alignment information is available, emotional values ​​can also be assigned at the subword or syllable level, allowing the emotional features to vary within corresponding very short time windows.

[0123] In the absence of precise alignment information, a rough temporal mapping is performed based on empirical rules. For example, the sentiment features obtained from analyzing an entire sentence are considered the primary emotion during the entire speech, and this sentiment is assumed to be maintained throughout the entire sentence. If the text contains multiple sentences, the total speech duration can be divided equally according to the number of sentences, and each sentence is assigned an sentiment feature value. In the absence of more fine-grained information within a single sentence, the sentiment value of the sentence is assumed to remain constant throughout the sentence. If necessary, the duration of pauses can be inferred based on punctuation marks, such as assuming a short pause at a period or comma. These rules ensure that a roughly plausible sequence of emotion changes over time can be generated even when information is incomplete. After completing the above temporal mapping, a time-indexed text feature sequence is obtained. The time step of this sentiment sequence is usually adjusted to be consistent with the speech feature sequence to facilitate subsequent frame-by-frame fusion with the speech features.

[0124] Step 4: Multimodal feature fusion.

[0125] This step fuses the speech feature sequence obtained in step 2 and the text feature sequence obtained in step 3 to generate a unified driving feature, which is used to comprehensively utilize the information of both speech and text modalities to guide facial animation generation, such as Figure 4 As shown. The goal of fusion is to superimpose the expression changes indicated by the semantic emotions of the text while ensuring lip-sync speech, so that the digital human facial animation can not only accurately lip-sync, but also vividly reflect the emotions of speaking. Input: speech feature sequence and text feature sequence. First, it is necessary to ensure that the two sequences are aligned on the time axis. As mentioned above, if the time resolution of the emotion feature sequence is lower than the speech frame rate, the emotion sequence can be interpolated and expanded: for example, linear interpolation is performed between every two known emotion data points to generate emotion feature values ​​corresponding to the speech frame. In special cases, if the emotion only changes at a few key time points, the emotion feature in this interval can be approximately regarded as a constant function. After time alignment and interpolation processing, an emotion feature sequence with the same length and one-to-one correspondence with the speech features can be obtained.

[0126] Two strategies are used: simple fusion or advanced fusion with the introduction of an attention mechanism. In simple fusion, the speech feature vector and text feature vector corresponding to each time step are directly concatenated into a longer vector, which is then mapped to the required dimension through a series of fully connected layers to obtain the fused features. For example, a fully connected network f(\cdot) is defined to map the concatenated vector:

[0127] Formula (8);

[0128] Where, is the fused feature vector, the dimension is determined by the network design; is a nonlinear activation function; is a trainable weight matrix; is the time step Aligned text feature vector; is a trainable bias term. Through this concatenated mapping, the neural network automatically learns the interactive relationship between speech and text features.

[0129] Preferably, the fusion unit introduces an attention mechanism to dynamically weigh the contributions of the two modal information. One implementation method is to add a gating factor based on the above simple splicing: calculate a time-varying weight coefficient (with a value of 0 to 1) to balance the proportion of speech and text features, and generate a fusion vector based on it. For example, define is the gating weight of the current time step, then the fusion output is expressed as , which is consistent with the above formula (2). Weight It is itself composed of another small neural network based on the current and In this way, in the scenario where speech is the main driving force of lip shape, the network automatically learns to adjust the value close to 1, so that speech features account for the majority of the proportion; when it is necessary to express facial expressions, the text features are used. Play a greater role.

[0130] Another approach to attention fusion is the cross-attention mechanism: using the speech feature sequence as the query and the text sequence as the key and value, multi-head attention is calculated to obtain a fused representation. This method refers to Equation (1) and related descriptions, that is, the model can select relevant information from the entire text sequence based on the current speech content and incorporate it into the output. When the text suggests a specific expression in a certain place but the speech features do not clearly reflect it, the attention mechanism introduces this additional information, thereby compensating for the shortcomings of a single modality.

[0131] The fused features carry both low-level information about the current pronunciation and high-level information about the tone of voice, serving as comprehensive features that drive subsequent animations. Output: A fused feature sequence. The sequence length is the same as the input speech feature sequence, and each sequence is a fused feature vector.

[0132] In scenarios where multimodal information conflicts, to ensure the naturalness of facial expressions, this system sets modal priority rules: the voice modality has a higher priority in driving the opening and closing of the mouth to ensure strict synchronization between voice and mouth shape; while the text modality has a higher weight on the emotional expression of the upper face (such as eyebrows and eyes), thereby enhancing the richness and semantic consistency of facial expressions.

[0133] Boundary case handling: The fusion unit needs to adapt to the situation where different modal inputs may be missing. When there are only voice features but no text, the emotional features are set to all zero vectors or fixed vectors representing neutral emotions, so that the fusion result is almost completely determined by the voice, ensuring that lip synchronization is not affected; on the contrary, when there is only text but no voice, the voice features are set to blank. In this case, the fusion output mainly depends on the text features. Or the attention weights are processed to an extreme level to ensure that the system can still output results smoothly when it degenerates into a single-mode drive, so as to avoid errors.

[0134] Step 5: Keypoint prediction.

[0135] This step predicts the facial keypoint positions and head pose parameters required for digital human facial animation based on the fused feature sequence. Keypoints, in this context, typically refer to the two-dimensional coordinates of a series of representative facial features, used to depict subtle expressions and lip shape changes; head pose parameters represent the rotation angle of the head, reflecting head motion. Input: The fused feature sequence output from step 4. A time series model is used to fit the mapping from fused features to facial keypoints. After training, the model can output the facial keypoint positions for the next frame based on the fused features of the current and previous frames. Considering the continuous nature of facial expressions and lip shape changes, the model learns a smooth transition path from keypoints at the previous moment to the current moment. In addition to end-to-end learning, a step-by-step approach is also possible: for example, a regression model is first trained to predict keypoint displacements or keypoint positions for each time frame, and then the global coordinates are adjusted based on overall head motion. Output: A sequence of facial keypoints and a sequence of head poses. To prevent animation instability caused by prediction jitter, a smoothing filter is applied to the output keypoint sequence to remove high-frequency jitter. At the same time, the change rate of key points and postures is guaranteed to not exceed the physically reasonable range, so that the facial movements are smooth and natural.

[0136] Handling edge cases: During the prediction process, if input features are missing or abnormal, the prediction model will tend to output a neutral position or maintain the position of the previous frame to avoid severe distortion caused by abnormal fusion features. During model deployment, rules can be introduced: for example, limiting the maximum mouth opening angle to prevent excessive distortion of the mouth shape; setting upper and lower limits on eyelid blinking frequency, etc., to closely follow the statistical laws of real human expressions.

[0137] Step 6: Animation frame generation.

[0138] This step generates a digital human face image frame at the corresponding moment based on the facial key point sequence and head posture parameters output in step 5. Its core consists of two stages: local deformation of the reference face and generative refinement. Input: smoothed key point positions, head posture, and a reference face model or reference image of the digital human. The reference image here is usually a high-definition face image of the digital human with a neutral expression and facing forward, or an initial frontal face frame rendered from a three-dimensional digital human head model. A set of facial feature point coordinates in this reference state is also required as a deformation reference.

[0139] First, for each time frame, the affine transformation parameters or displacement fields of each local region are calculated based on the differences between the currently predicted facial keypoints and the reference keypoints. These transformations are applied to the reference face image to produce a preliminary rendered distorted image. Simultaneously, the entire image is rotated and translated based on head pose parameters to simulate the effects of head motion.

[0140] The warped image sequence is then fed into a pre-trained generative refinement model, which outputs a high-quality sequence of facial images. This GAN model inpaints the initial images frame by frame, enhancing texture detail and realism. The output: a high-definition sequence of animated virtual faces. These frames can be directly used to compose the final video.

[0141] During system training, the GAN refinement unit uses real face video frames as supervision targets, keypoint-driven synthetic frames aligned with real frames as input and labels, and optimizes the generator through adversarial learning to improve image realism. A perceptual loss is also introduced to improve detail fidelity, making the model more accurate to the actual movement of the face.

[0142] Handling edge cases: If the GAN refinement network produces artifacts or unstable output in certain frames, the quality of adjacent frames can be detected and corrected through interpolation. For example, if an output frame exhibits abnormal brightness or color distribution, the frame can be replaced by linear interpolation of the preceding and following frames to ensure a smooth and consistent final sequence. Furthermore, for expressions that are extremely outside the training sample distribution, the GAN may be unable to generate realistic details. In these cases, the system can fall back to using the original distorted frame to avoid severe distortion.

[0143] Step 7: Animation rendering and audio synchronization.

[0144] This step synthesizes the facial image frame sequence generated above with the original speech audio, rendering a complete animation and ensuring synchronized output of the audio and video. The final result is a video file of the digital human speaking with sound, or a synchronized audio and video stream for real-time interactive scenarios. Input: The image frame sequence output from step 6 and the original speech audio captured in step 1. Before entering the rendering synchronization phase, these frames are mapped to speech feature frames at a time step of 10 milliseconds or finer, so the number of frames strictly matches the audio duration. The system first organizes the image frames at a constant frame rate and resamples or frames the audio to align them to the same timeline. Each image frame is then associated with the audio of the corresponding time segment, synthesizing the image and sound within a video container. Rendering uses a standard video encoding library to compress the frame sequence into a video stream. In real-time output scenarios, a graphical interface or engine may be used to directly overlay the image onto the background for display and audio playback. The final output is an animation of the digital human speaking in real time. Whether it is an offline file or an online stream, audio and video synchronization and high-quality presentation are guaranteed.

[0145] Output: The finished virtual human speaking animation, synthesized from audio and video. This is typically output to a common video file format like MP4, or to a display device via a streaming protocol. This output can be viewed by users or used as material in other downstream systems.

[0146] Boundary case handling: If it is detected during the animation rendering phase that the frame rate cannot keep up with the real-time audio playback, you can choose to discard some frames or micro-adjust the audio time scale to realign the synchronization. At the same time, necessary metadata is added when packaging the output, such as video time code information, audio sampling rate information, etc., to ensure strict synchronization in various playback environments. For situations where pure text is driven but sound output is required, TTS will be called to convert text to speech for audio synthesis. For pantomime scenes that do not require sound, only the picture frames are output and the audio stream can be omitted. Through flexible synchronization control, the output of virtual human speaking animation with highly consistent sound and expression is finally achieved.

[0147] Because all processing units adhere to a unified timeline, ideally, the image frame sequence and audio are already synchronized. For example, assuming each frame lasts 0.04 seconds, the nth image frame should correspond to the nth 0.04-second segment of speech in the audio. Therefore, the image sequence and audio stream can be directly mixed and output by matching their timestamps. The audio clips are processed to generate feature vectors, which are then combined with emotional information to generate facial landmarks, and then rendered as image frames. This process repeats for each consecutive time step, ultimately resulting in a chronologically ordered animation frame sequence that is strictly synchronized with the input audio.

[0148] In actual implementation, if a slight mismatch between the image and audio is detected, the system adopts a fine-tuning alignment strategy. For example, if the image frame lags slightly behind the audio at a certain moment, a very short audio sample is skipped. Conversely, if the image frame ends too quickly, a very short still image is inserted in the middle to realign the frame sequence with the audio. Furthermore, before rendering begins, the total number of frames to be generated can be calculated based on the total audio length and the predetermined frame rate. If the actual number of frames generated is more or less than the calculated value, the final frame is deleted or copied at the end of the sequence to make the total duration strictly consistent with the audio length. After this calibration, the image and audio sequences are precisely synchronized on the timeline.

[0149] After the frame sequence and audio are aligned, the system will perform the final rendering and synthesis of the image frame by frame. For example, an appropriate background picture is selected and the face frame is superimposed on it; at the same time, the corresponding voice clip is embedded in the audio track of the video. Every time a frame of image is generated and presented, the audio segment corresponding to the frame is played synchronously. In real-time applications, the streaming media protocol is used to generate and push the audio and video stream to the display end; in offline applications, all frames and audio are synthesized into a video file in a standard format. Output: The final output is an animated video containing the continuous speaking picture of the digital human. When output in file form, it is an encapsulated audio and video file; when output in real-time form, it is a continuously played audio and video stream.

[0150] During the synthesis phase, it is necessary to consider abnormal situations such as mismatched audio or video lengths. For example, if for some reason the generated image frame sequence ends earlier than the audio, the system will keep the last frame still at the end until the audio ends to avoid premature video interruption. Conversely, if the audio ends first, the frame update can be stopped or the last frame of audio can be repeated to align. Similarly, if frame processing delays occur during real-time streaming and the audio cannot keep up temporarily, the video frame output frame rate can be temporarily reduced to smooth the transition, and then restored to normal frame rate after catching up with the timeline. These mechanisms ensure that the final output animation image and sound are strictly synchronized and smooth and continuous.

[0151] 4. When the system is initialized, the pre-trained model parameters are loaded locally first. If it is detected that the local model file does not exist or the version does not match, the system will automatically obtain the latest model file from the remote server and cache it locally to ensure subsequent offline use. The model loading process includes initializing the model structure and weight parameters, loading the model into memory, and completing the deployment on the device. In order to improve loading efficiency, the system uses direct memory mapping or zero-copy technology. When conditions permit, the model data is directly referenced from the memory for loading, thereby avoiding repeated file read and write operations. For example, when the mobile platform only allows access to the model in memory form, the system embeds the model into the program as a static resource and loads it directly through memory mapping, avoiding time-consuming file read and write operations.

[0152] After loading the model, the system employs sophisticated memory management and multi-threaded asynchronous mechanisms during inference execution to ensure real-time performance and stability. The system pre-allocates the required tensors and cache space, reusing intermediate result buffers to avoid frequent memory allocation and deallocation. After each frame is processed, any intermediate data no longer needed is promptly released to reduce peak memory usage. This memory management strategy, combined with memory pooling technology, recycles allocated memory blocks during inference, ensuring stable memory usage and avoiding increased memory burden due to accumulated frame processing.

[0153] On the other hand, the system utilizes multi-threading and asynchronous execution mechanisms to fully utilize the parallel capabilities of multi-core hardware. For example, different functions such as audio processing, feature extraction, expression fusion, and image generation are executed by independent threads or asynchronous tasks, using producer-consumer queues to synchronize data flow. The audio acquisition and feature extraction threads run continuously, placing the extracted features into a buffer queue; another thread monitors the queue and obtains the latest features for emotion fusion and image generation. This asynchronous pipeline design ensures that each processing stage operates in parallel as much as possible: while one frame of image is being rendered, the audio features of the next time frame are also being calculated in parallel, thereby reducing overall waiting time. For GPU-accelerated parts, the system uses asynchronous GPU computing streams to concurrently execute kernel calculations and data copies when possible, further improving throughput.

[0154] When loading models, the system also considers cross-platform deployment differences and selects the appropriate platform inference engine for loading. The system supports the use of mainstream cross-platform inference frameworks such as ONNX Runtime, TensorRT, CoreML, and NCNN to fully utilize the hardware acceleration capabilities of different platforms. The details are as follows:

[0155] On Windows or Linux desktops, when an NVIDIA GPU is detected to be available, the TensorRT engine is used first for model loading, and tensor RT optimization is used to significantly improve the inference speed. If there is no dedicated GPU, ONNX Runtime is used to call the local CPU or other backend for inference.

[0156] On the Android platform, the system loads the converted model through the NCNN engine. NCNN is highly optimized for ARM processors, leveraging NEON instructions and multi-core parallel acceleration to accelerate model execution. Furthermore, for devices that support Android NNAPI or Vulkan, NCNN uses hardware acceleration.

[0157] On the iOS platform, the system converts the model into CoreML format and uses the iPhone's Neural Engine to accelerate reasoning, providing better energy efficiency.

[0158] By encapsulating a unified loading interface, the system automatically selects the optimal inference framework and model format for loading based on the operating environment. If a particular engine fails to load, the system provides a fallback mechanism. For example, if TensorRT fails to load, the system falls back to the default backend using ONNX Runtime, ensuring continued system functionality. On mobile devices, if hardware acceleration fails, it falls back to CPU computing mode. This entire loading logic ensures that models are correctly initialized regardless of device differences and provides backup options to ensure the normal operation of core functionality.

Claims

1. A multi-modal driven virtual digital human facial animation generation method, characterized in that: include: Obtain voice input and text input and extract voice features and text features; The speech features and text features are integrated to obtain control parameters for driving the facial animation of the virtual digital human, wherein the control parameters include facial expression parameters and head posture parameters, and a driving strategy that differentiates between upper facial expressions and lower facial expressions is used to dynamically adjust the contribution weights of the speech modality and text modality to the control parameters; Performing local deformation on the facial image of the virtual digital human according to the control parameters to generate an initial facial animation; Refining the initial facial animation using a generative adversarial network to obtain a refined facial animation; It also includes an adaptive multi-head fusion strategy for adjusting the weight distribution of each attention head in real time according to the emotional intensity of the speech signal or the contextual changes of the text content; the adaptive multi-head fusion strategy dynamically adjusts the weights of multiple attention heads of cross-attention by monitoring the pitch and volume emotional intensity parameters of the speech modality and the contextual semantic changes of the text modality, so that the speech modality has a higher weight when the speech emotion is strong, and the text modality has a higher weight in a flat context.

2. The method according to claim 1, wherein: The timestamps of the text features in the speech signal are obtained through time alignment, and the text features are extended to be consistent with the frame rate of the speech features through linear interpolation or replication to form a time-synchronized text feature sequence.

3. The method according to claim 1, wherein: The fusion of the speech features and text features adopts the attention mechanism to realize the information interaction between the two, and dynamically adjusts the contribution weights of the speech modality and text modality to the control parameters through the gating mechanism.

4. The method according to claim 3, wherein: When the driving information carried by the voice input conflicts with the driving information carried by the text input, a smoothing filter process is performed on the generated facial animation.

5. The method according to claim 1, wherein: The time dependency of the speech features and text features is modeled through a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters.

6. The method according to claim 5, characterized in that: The control parameters simultaneously encode speech-driven lower-face lip shape changes and text-driven upper-face expression changes; The voice-driven lower face lip shape change includes: obtaining a voice signal to be broadcast, performing audio feature extraction on the voice signal to obtain a time series of voice features; generating corresponding mouth movement parameters based on the voice features, and driving the mouth shape movement of the lower half of the virtual digital human's face to generate a preliminary lower face animation; The text-driven upper facial expression changes include: obtaining the text content corresponding to the voice signal, performing semantic analysis and emotion recognition on the text, and extracting text emotional features; generating corresponding upper facial expression parameters based on the emotional features, and driving the expression changes of the upper facial part of the virtual digital human to generate preliminary upper facial animation.

7. The method according to claim 1, wherein: Based on the preliminary full-face animation, a head posture perturbation feedback mechanism is introduced to apply subtle random or regular perturbations to the head posture parameters of the virtual digital human to simulate the inadvertent micro-movements of the human head. The head posture perturbation feedback mechanism detects changes in the virtual digital human's head posture in consecutive animation frames. If it detects that the head remains still for a long time, it applies subtle random perturbations; or generates synchronized small head movements based on the rhythm and emphasis of the voice signal.

8. The method according to claim 1, wherein: The generation of the facial animation includes: first performing local image deformation on the facial image of the virtual digital person to generate an initial facial animation, and then using a generative adversarial network to refine and optimize the initial facial animation.

9. A multi-modal driven virtual digital human facial animation generation system, characterized by: When the system is running, the method according to any one of claims 1 to 8 is executed: The system comprises: A feature extraction unit, configured to obtain speech input and text input and extract speech features and text features; a fusion unit, configured to fuse the speech features and text features to generate facial animation control parameters, wherein the control parameters include facial expression parameters and head posture parameters, and to control the fusion process of the speech features and text features using an attention mechanism and a gating mechanism; An adaptive fusion control unit, configured to dynamically adjust the multi-head attention weight distribution in the fusion unit according to changes in the emotional intensity of the speech or the context of the text; A head posture processing unit, configured to determine changes in the head posture of the virtual digital human according to the control parameters; A head pose perturbation unit is used to perform micro-motion modeling on the head pose in the data of the full-face preliminary animation, generate subtle head pose perturbations and feed them back into the animation data; An image deformation unit, configured to locally deform the facial image of the virtual digital human according to the control parameters to generate an initial facial animation; A GAN refinement unit, configured to perform a generative adversarial network refinement process on the initial facial animation; The smoothing processing unit is configured to perform smoothing filtering on the facial animation when a conflict is detected between the voice-driven information and the text-driven information.

Images