Multi-modal driven virtual digital human face animation generation method and system

Abstract

The invention discloses a multi-mode driven virtual digital human face animation generation method and system, and relates to the field of computer graphics, and the method comprises the steps: obtaining voice input and text input, and extracting voice features and text features; dynamically fusing the two features through an attention fusion model to generate facial expression and head posture control parameters, and dynamically adjusting contribution weights of a voice mode and a text mode to the control parameters by adopting a driving strategy of differentiation of facial upper expression and facial lower expression; and performing local deformation on the virtual digital human face image based on the control parameter to generate an initial animation frame, and performing refining processing on the initial animation by using a generative adversarial network to obtain a refined face animation. By means of the technical scheme, the natural and vivid virtual human face animation matched with the voice content and the text semantics can be generated.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and particularly to a method and system for generating virtual digital human facial animations driven by multiple modalities. Background Art

[0002] In recent years, virtual digital human technology has developed rapidly in the fields of human-computer interaction, film and television production, virtual live broadcast, etc. The core challenge lies in how to generate highly realistic facial animations that are synchronized with speech, semantics, and emotions. Existing technologies mainly rely on single-modal driving or local control methods based on 3D bone systems and BlendShape animations. However, these solutions have significant deficiencies in terms of expression naturalness, emotion expression, and real-time performance. For example, in the audio-driven method, mouth shape animations are generated based on audio signals. Although basic synchronization can be achieved, the capture of emotions and upper face expressions is lacking, resulting in a single expression. In the text-driven method, expression tags are generated through text semantic analysis. Although the emotion category can be reflected, the temporal information of speech is lacking, and additional speech synthesis assistance is required, resulting in a rigid expression change and difficulty in responding to dialogue dynamics in real time. In addition, emotion intensity prediction relies on empirical rules, and it is easy to have an inaccurate expression amplitude. Some existing solutions attempt to fuse the speech and text modalities, but mostly use simple feature splicing or static weight allocation, and do not solve the problem of natural transition when there is a modality conflict. In addition, temporal alignment relies on forced interpolation, resulting in sudden changes in emotion features and affecting the animation coherence. Therefore, based on the above problems, the present invention proposes a method and system for generating virtual digital human facial animations driven by multiple modalities. Summary of the Invention

[0003] Technical Objectives In order to solve the above problems, the objective of the present invention is to provide a method and system for generating virtual digital human facial animations driven by multiple modalities, aiming to solve the problems of insufficient synchronization between speech and expressions, single emotion expression, and low generation fidelity in virtual digital human facial animation generation, realizing high synchronization between mouth shapes and speech, accurate matching between emotions and semantics, and at the same time improving the authenticity of animation details, providing an efficient, natural, and scalable solution for multi-scenario applications.

[0004] Technical Solutions To achieve the above object, the present invention provides a method and system for generating facial animations of virtual digital humans driven by multiple modalities. This solution generates facial expression animations of virtual digital humans by fusing driving information from the speech modality and the text modality, uses an attention mechanism to achieve deep fusion of speech and text features, and introduces a gating mechanism to dynamically adjust the contribution weights of the two modalities according to different facial regions. The present invention particularly adopts a differential driving strategy for the upper half and lower half facial expressions. The speech features are used to drive animations of the lower half of the face such as lip movements, so as to ensure accurate synchronization of speech and lip movements. The text features are used to drive expressions of the upper half of the face such as eye expressions and eyebrows, to convey semantic emotions. Through the above fusion and differential driving strategies, the present invention can enrich the emotional expression of facial expressions while ensuring lip synchronization.

[0005] In a first aspect, the present invention provides a method for generating facial animations of virtual digital humans driven by multiple modalities, including: Obtaining speech input and text input and extracting speech features and text features; Fusing the speech features and text features to obtain control parameters for driving the facial animations of the virtual digital human, the control parameters including facial expression parameters and head pose parameters, and dynamically adjusting the contribution weights of the speech modality and the text modality to the control parameters by using a driving strategy that differentiates between upper facial expressions and lower facial expressions; Performing local deformation on the facial image of the virtual digital human according to the control parameters to generate an initial facial animation; Using a generative adversarial network to refine the initial facial animation to obtain a refined facial animation.

[0006] Further, obtaining the time stamps of the text features in the speech signal through time alignment, and extending the text features to be consistent with the frame rate of the speech features through linear interpolation or replication to form a temporally synchronized text feature sequence.

[0007] Further, the fusion of the speech features and text features uses an attention mechanism to achieve information interaction between the two, and dynamically adjusts the contribution weights of the speech modality and the text modality to the control parameters through a gating mechanism.

[0008] Further, the attention mechanism uses the speech feature sequence as the query vector, the text feature sequence as the key and value vectors, calculates the correlation weight between the speech features and the text features through attention, performs weighted summation on the text features based on the correlation weight to generate an emotional context vector, and splices or weights and fuses the emotional context vector with the speech feature vector to form a cross-modal fusion feature.

[0009] Furthermore, the gating mechanism dynamically adjusts the linear combination weights of speech features and text features through trainable gating coefficients, which are jointly predicted by the speech features and text features at the current time step. When there is a conflict between speech and text, the gating coefficients are adaptively adjusted to balance the contributions of both, ensuring the naturalness and consistency of the expression output.

[0010] Furthermore, when there is a conflict between the driving information carried by the speech input and the driving information carried by the text input, smoothing filtering is performed on the generated facial animation.

[0011] Furthermore, time-dependent modeling of the speech features and text features is performed through a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters.

[0012] Furthermore, the control parameters simultaneously encode the lower face mouth shape changes driven by speech and the upper face expression changes driven by text; The speech-driven lower face mouth shape changes include: obtaining the speech signal to be broadcast, extracting audio features from the speech signal to obtain time-series speech features; generating corresponding mouth movement parameters based on the speech features, and driving the mouth shape movement of the lower part of the virtual digital human face to generate a preliminary lower face animation; The text-driven upper face expression changes include: obtaining the text content corresponding to the speech signal, performing semantic parsing and emotion recognition on the text, and extracting text emotion features; generating corresponding upper face expression parameters based on the emotion features, and driving the expression changes of the upper part of the virtual digital human face to generate a preliminary upper face animation.

[0013] Furthermore, an adaptive multi-head fusion strategy is also included, which is used to adjust the weight distribution of each attention head of the attention in real time according to the emotion intensity of the speech signal or the context change of the text content; The adaptive multi-head fusion strategy dynamically adjusts the weights of multiple attention heads of cross-attention by monitoring emotion intensity parameters such as pitch and volume of the speech modality and context semantic changes of the text modality, assigning higher weights to the speech modality when the speech emotion is strong, and increasing the weight of the text modality in a flat context.

[0014] Furthermore, on the basis of the full-face preliminary animation, a head pose perturbation feedback mechanism is introduced to apply slight random or regular perturbations to the head pose parameters of the virtual digital human to simulate the inadvertent micro-movements of the human head; The head pose perturbation feedback mechanism detects the changes in the head pose of the virtual digital human in consecutive animation frames. If it detects that the head remains stationary for a long time, slight random perturbations are applied; or small-scale synchronous head movements are generated according to the rhythm and stress of the speech signal.

[0015] Furthermore, the generation of the facial animation includes: first performing local image deformation on the facial image of the virtual digital human to generate an initial facial animation, and then using a generative adversarial network to refine and optimize the initial facial animation.

[0016] In a second aspect, the present invention further provides a multi-modal driven virtual digital human facial animation generation system. When the system runs, it executes the method described in the first aspect above. The system includes: A feature extraction unit, configured to obtain voice input and text input and extract voice features and text features; A fusion unit, configured to fuse the voice features and text features to generate facial animation control parameters. The control parameters include facial expression parameters and head pose parameters, and an attention mechanism and a gating mechanism are used to control the fusion process of the voice features and text features; An adaptive fusion control unit, configured to dynamically adjust the multi-head attention weight allocation in the fusion unit according to the change in the emotional intensity of the voice or the change in the context of the text; A head pose processing unit, configured to determine the head pose change of the virtual digital human according to the control parameters; A head pose perturbation unit, configured to perform micro-motion modeling on the head pose in the data of the preliminary animation of the whole face, generate a subtle head pose perturbation and feedback it to the animation data; An image deformation unit, configured to perform local deformation on the facial image of the virtual digital human according to the control parameters to generate an initial facial animation; A GAN refinement unit, configured to perform refinement processing of the generative adversarial network on the initial facial animation; A smoothing processing unit, configured to perform smoothing filtering processing on the facial animation when it is detected that there is a conflict between the voice driving information and the text driving information.

[0017] In a third aspect, the present invention further provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when the computer program is run by a processor, it executes the multi-modal driven virtual digital human facial animation generation method described above.

[0018] The present invention realizes the precise synchronization of speech and lip shape, the natural matching of emotion and semantics, the high-fidelity rendering of micro-expressions and the robust deployment of multiple scenes through cross-modal dynamic fusion mechanism, timing modeling and smooth control, 2D image driving and generative refinement and cross-platform adaptation architecture. A differentiated driving strategy for the upper and lower facial expressions is adopted. The speech features are used to drive the lower facial animations such as lip shape, so as to ensure the precise synchronization of speech and lip movement. The text features are used to drive the upper facial expressions such as eyes and eyebrows to convey semantic emotions. Through the above-mentioned fusion and differential driving strategies, the present invention can enrich the emotional expression of facial expressions while ensuring the synchronization of lip shape. This scheme significantly improves the synchronization, fluency and realism of virtual human facial animation, solves the defects of modal splitting, timing jitter and insufficient generation quality in traditional methods, and provides efficient and scalable technical support for scenes such as real-time interaction and film and television production.

[0019] Beneficial Effects By implementing the multi-modal driven virtual digital human facial animation generation method and system provided by the present invention, the following technical effects are achieved: (1) This application realizes the dynamic fusion of speech and text drive. By introducing the attention mechanism and the gating mechanism, the present invention can adjust the fusion weight of the two driving signals in real time according to the changes in the speech content and text semantics, overcoming the problem of inaccurate expressions caused by fixed weight fusion, so that the generated expressions are consistent with both the speech rhythm and the text meaning.

[0020] (2) A differentiated expression driving strategy for the upper and lower faces is proposed. This strategy ensures that the lip movements driven by speech are accurate and smooth, while using text information to enrich the delicacy of the upper face expressions (such as eye contact and eyebrow movements), solving the problem of monotonous and stiff expressions caused by single-mode driving.

[0021] (3) An adaptive multi-head attention fusion strategy is adopted to adjust the expression drive in real time according to the changes in the intensity of speech emotions and text context, so that the virtual human can sensitively express emotional ups and downs; compared with the fixed fusion scheme, the expression presentation in the emotional change scene is more natural and rich.

[0022] (4) By adding a conflict detection and smooth filtering mechanism, when the expressions indicated by voice and text conflict, the system can automatically and smoothly transition to avoid abrupt and uncoordinated changes in facial expressions, improve animation continuity, and enhance the ability to handle multimodal conflicts.

[0023] (5) Micro-movement perturbations of the head can effectively prevent the virtual human from being stiff and motionless for a long time, adding subtle body language elements that are close to humans; combined with GAN's enhancement of facial details, the generated virtual human facial animation is more realistic in terms of detail texture and overall expression, and the look and feel is closer to that of a real person.

[0024] (6) Through lightweight model compression, hardware acceleration engines, and abnormal modality suppression strategies, efficient deployment from mobile devices to desktop devices is achieved. This architecture maintains facial expression coherence when the input signal is missing or conflicting, ensuring the robustness and real-time performance of the system in complex scenarios, and expanding the breadth and stability of technology applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To make the above-mentioned multi-modal driven virtual digital human facial animation generation method and system of the present invention more clearly understandable, the drawings required for the specific implementation manners of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0026] Figure 1 Represents the schematic diagram of the system architecture of the present application; Figure 2 Represents the flowchart of virtual human face 2D animation generation; Figure 3 Represents the flowchart of the method of the present application; Figure 4 Represents the schematic diagram of the fusion mechanism. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] Embodiment 1: A multi-modal driven virtual digital human facial animation generation method and system are provided, and the system architecture is as Figure 1 shown. Specifically, it is described as follows.

[0028] 1. For the two modalities of voice and text, a multi-modal fusion algorithm is designed to synchronously drive facial movements. Its implementation logic includes the extraction and alignment method of voice features and text features, the multi-modal feature fusion mechanism, the conflict and fault tolerance processing strategy, and the unified facial movement feature sequence format and dimension of the final output.

[0029] First, the system extracts features from the input voice signal to obtain a time-sequential voice feature sequence. Specifically, acoustic analysis can be performed on the voice, such as calculating low-level features such as Mel spectrogram and MFCC, or using a pre-trained voice encoder to extract high-dimensional voice embeddings to capture pronunciation, prosody, and emotional cues in the voice. The obtained voice feature sequence has a high frame rate and can finely depict mouth movements and intonation changes.

[0030] Secondly, the system extracts the emotional feature sequence from the corresponding text content. The text content is usually the transcription of speech or pre-provided lines. To extract the emotional information of the text, natural language processing techniques are used to perform emotional analysis or semantic embedding on the text. The text features are generated at the granularity of sentences, clauses or words, with a relatively low frame rate and sparse changes compared to the speech features. To map the text feature sequence to the same time axis as the speech features, alignment processing is required: on the one hand, forced alignment or speech recognition technology is used to obtain the start and end times of each word or phoneme in the speech; on the other hand, according to these timestamps, the corresponding text features are copied or interpolated onto multiple speech feature frames covering that time interval. At the junction of adjacent emotional segments, linear interpolation is used to achieve a smooth transition and avoid sudden changes in emotional parameters. After the above processing, the speech feature sequence and the text feature sequence are aligned one-to-one in the time dimension, that is, the multi-modal input is precisely synchronized in time sequence.

[0031] For the convenience of subsequent fusion calculation, the system performs scale transformation or embedding dimension transformation on the speech features and text features, so that the two features are mapped to the same semantic space and have the same dimension.

[0032] After obtaining the aligned speech and text feature sequences, the system enters the multi-modal fusion stage. The fusion unit aims to organically combine the lip movement and speech prosody information provided by the speech modality with the emotional expression clues provided by the text modality to generate a unified facial action representation. To this end, the fusion algorithm comprehensively adopts strategies such as attention mechanism and gated linear fusion, and combines the time alignment information to achieve efficient and robust cross-modal information fusion.

[0033] The timestamps of the text features in the speech signal are obtained through time alignment, and the text features are extended to be consistent with the frame rate of the speech features by linear interpolation or replication to form a text feature sequence synchronized in time sequence.

[0034] The fusion of the speech features and text features uses the attention mechanism to realize the information interaction between the two, and dynamically adjusts the contribution weights of the speech modality and the text modality to the control parameters through the gating mechanism.

[0035] An algorithm based on the attention mechanism is used to establish a dynamic association between the feature sequences of the two modalities. The attention mechanism realizes the selective extraction and fusion of information by calculating the correlation weights between the feature sequences. For example, using a cross-attention structure, with the speech feature sequence as the query vector, the text feature sequence as the key and value, a context-weighted representation of the emotional features related to the current speech is calculated from the text modality. First, calculate the attention weights of the speech , and its calculation formula is as follows: (1); In the formula, is the attention weight of the speech feature at time corresponding to the text feature at the time step ; is the query vector of the speech feature at time ; is the key vector of the text feature sequence at the time step ; is the dimension of the feature vector, specifically the unified dimension after the speech feature and the text feature are mapped to the same semantic space; is the length of the text feature sequence; is the counter.

[0036] Next, according to the weight weighted sum of the text features is performed to obtain the weighted representation of the corresponding emotional feature in the speech modality , and the calculation formula is as follows: (2); In the formula, is the emotional feature vector of the text modality at the time step .

[0037] Through formula (2), the model extracts relevant emotional features from the entire text information according to the current speech content. The attention mechanism thus realizes cross-modal soft alignment: even if there are slight differences in time between the speech and the text, the model can automatically learn the corresponding positions in the text sequence that should be focused on at moment, so as to make up for the errors that may be generated by simple forced alignment. Through cross-attention, the speech feature sequence obtains an information representation that fuses the text context. Similarly, the text feature can also be used as a query in reverse to extract the emotional signal in the speaking intonation from the speech feature and fuse it into the text feature stream. In addition, a multi-head attention mechanism is also introduced in parallel to capture richer cross-modal correlation features from different subspaces.

[0038] In the fusion stage, a gating mechanism is introduced to adaptively adjust the contributions of the speech and text features to the output according to the differences in their contents. The specific approach is to calculate a gating coefficient between 0 and 1 for each time step, and then perform a linear weighted sum of the speech feature and the text feature according to this coefficient to obtain the fusion output. As shown in formula (3), the fusion output feature is expressed as the weighted combination of the speech feature and the text feature : (3); In the formula, is the fusion output feature at time t; is the gating coefficient for the current time step, and its value ranges from 0 to 1. When it is close to 1, it indicates that the speech drive is dominant, and when it is close to 0, it indicates that the text drive is dominant; and are respectively the feature vectors of the speech modality and the text modality at the moment.

[0039] The gating coefficient is predicted by a small neural network layer based on and the current states of. For example, the gating coefficient is calculated in the following form: (4); In the formula, is the Sigmoid activation function; is the corresponding weight parameter matrix; is to and vector concatenation fusion as the network input.

[0040] By introducing the gating mechanism, the system can dynamically adjust the weights between the speech and text modalities according to different scenarios: when the speech signal already shows clear facial expressions and intonations, can be made to approach 1 to highlight the role of speech features; when the text content implies strong emotions but the speech does not fully express them, then is reduced to enhance the role of text features. Linear weighted fusion ensures the full superposition of information from the two modalities, and suppresses the interference of conflicting information on the output through the gating coefficient, so as to obtain coherent and consistent facial expression features.

[0041] To break through the limitations of traditional single-modal driving, the present invention proposes a "facial expression region separation driving model". This model is based on the physiological characteristics of the human facial neuromuscles - the lower half of the face (lips, mandible) is mainly responsible for the mechanical movements of speech synchronization, while the upper half of the face (eyebrows, periorbital area, forehead) bears the micro-expression changes of emotional expression. By dynamically allocating the driving weights of the speech and text modalities in different regions of the face, anatomically reasonable anthropomorphic expression generation is achieved.

[0042] In the lower half of the face speech-dominant layer, when generating mouth shape animations, the system raises the driving weight of the speech modality to 80%-95%. The Mel spectral energy envelope in the speech features directly controls the opening and closing amplitude of the lips, and the fundamental frequency change rate drives the stretching intensity of the corners of the mouth, ensuring that consonant plosives trigger instantaneous lip closure and vowel prolongation maintains the open state of the oral cavity. For example, when detecting a rapid plosive sound in the speech, the system automatically strengthens the mandibular bone displacement parameter, making the virtual human mouth shape show an instantaneous tightening physical inertia effect. This method can solve the "mouth shape stickiness" phenomenon caused by vowel trailing in traditional solutions, and improve the lip and tooth collision accuracy in fast dialogue scenarios by more than 40%.

[0043] In the upper half of the face text emotion dominant layer, for emotional micro-expressions such as eyebrow wrinkles and eyelid opening and closing, the weight of the text modality accounts for 70%-85%. The emotional intensity gradient value (0.0-1.0) output by the emotion analysis module is directly mapped to: Anger (intensity > 0.8): Corrugator supercilii muscle contraction parameter + Levator palpebrae superioris muscle tension coefficient; Surprise (intensity > 0.6): Frontalis muscle stretching amplitude × Iris exposure rate increment; Sadness (intensity > 0.5): Orbicularis oculi muscle contraction frequency + Tail of eyebrow downward displacement amount. For example, when the rhetorical question "Is this reasonable?" is recognized in the text, the system automatically injects a pulse signal of eyebrow arch raising (duration 200ms), and generates a skeptical micro-expression in cooperation with the eyelid micro-tremor algorithm.

[0044] This model can improve the lip synchronization accuracy of the digital human to 98.2%.

[0045] During the process of fusing and generating the preliminary facial animation, the credibility assessment module of the system is also working synchronously. Whenever a new animation time frame is processed, the credibility assessment module calculates the corresponding credibility index according to the current voice and text input situation. For example, when obvious background noise or intermittency and instability are detected in the voice signal, the assessment module determines that the information reliability of the current voice modality has decreased; another example is when the text analysis shows a strong emotional tendency (such as the sentence contains emotional words like "anger") but the intonation of the voice is flat, the assessment module determines that the text modality is more reliable in expressing emotions. The fusion unit will obtain the credibility score from the assessment module and automatically adjust the weight distribution of voice and text in the internal gate control unit: on the frames with low voice credibility, increase the influence weight of text features on the upper half face expression to ensure that the expression still conforms to the emotion of the conversation content; on the contrary, when the voice signal is clear and full of emotions, increase the contribution of voice features to the overall facial expression. Through this dynamic weight adjustment mechanism, even if the information quality of a single modality is poor at a certain moment (such as audio noise or unclear text meaning), the facial animation of the virtual digital human can still maintain a reasonable expression performance and will not be distorted or out of sync due to temporary input abnormalities.

[0046] The fusion unit also uses an adaptive multi-head attention mechanism to enable the system to adjust its focus according to the needs of the scene. Specifically, there are multiple attention heads in the fusion network, some of which focus on ensuring accurate synchronization between lip movements and voice signals, others focus on capturing the impact of speaking tone and emotional changes on facial expressions, and some may focus on the correspondence between voice content and facial expression details. The attention head design has an adaptive weighting scheme: when the volume of the voice increases, the tone becomes higher, and other signals indicating emotional excitement are detected, the system will automatically increase the weight of the attention heads that focus on emotional expressions, giving them a larger range of expression changes than usual; on the contrary, when the voice tone is smooth and soft, and the text content does not have drastic emotional fluctuations, the weight of the attention head that focuses on semantic consistency is increased to avoid excessive exaggeration of facial expressions. In this way, the expression of the virtual digital person can be adjusted accordingly with the changes in language content and tone, neither too bland nor exaggerated and distorted. For example, when the virtual person calmly narrates a passage, his or her facial expressions will be more restrained and delicate; however, when the voice and text reveal surprise or excitement, the system will promptly amplify the range of movements such as eyebrows raising, eyes widening, and mouth shape exaggeration, making the virtual person's emotional expression more vivid. Through the coordination of adaptive multi-head attention, the system of the present invention can show appropriate expressions in various contexts, significantly enhancing the emotional expression of the virtual person.

[0047] Considering the difference in temporal granularity between speech and text modalities, a time alignment strategy is also applied in the fusion process to ensure that the two types of information act at the correct timing. On the basis of the aforementioned forced alignment, the fusion unit retains the one-to-one correspondence between speech and text feature sequences, so that attention calculations and gated weightings are performed at matching time steps. In addition, for the case where text features remain constant between adjacent frames, time interpolation and smoothing filtering mechanisms are added to the fusion network. Another implementation scheme combines a decoder based on a sequence model and uses its temporal modeling capabilities to naturally generate a smoothly transitioned feature sequence at the output stage. In short, the time alignment and interpolation smoothing mechanisms ensure the continuity and consistency of the fused feature sequence in the temporal dimension, avoiding animation incoordination caused by inconsistent rhythms of different modalities.

[0048] In the process of multimodal fusion, corresponding processing strategies are designed for possible modal information conflicts and signal missing situations to ensure the robustness of the system.

[0049] When the emotional information conveyed by the voice features and text features is inconsistent, the system automatically adjusts the output through the above-mentioned attention weights and gating coefficients. The attention mechanism will reallocate the weights in such inconsistent situations. When the gating coefficient detects a large difference in the emotional intensity between the voice and the text, it can be adjusted to an intermediate value to mix the information of both and generate an appropriate expression intensity. Rule constraints can also be introduced when necessary: for example, setting a higher weight for the voice in determining the opening and closing of the mouth shape and the basic expression, while a higher weight for the text in the subtle expressions, to ensure that on the one hand, the lip movements are accurately synchronized with the voice, and on the other hand, the expression changes on the upper half of the face are enriched. Through the above multi-level conflict handling, the system can still output natural and coordinated facial animations when facing conflicting information sources.

[0050] When the expression states corresponding to the voice-driven information and the text-driven information are inconsistent, first detect and determine the facial expression parameters that cause conflicts, and then apply a filter to smooth these expression parameters to relieve the sudden changes brought about by the conflicts, making the adjusted expression parameters smoother and more coherent in the transition, and finally output a natural and coordinated virtual digital human facial animation.

[0051] Furthermore, when conflicting emotions occur, the system activates a three-level arbitration mechanism: Primary arbitration: Dynamically balance the weights through the gating coefficient. When it is detected that the difference in emotional intensity between the voice and the text exceeds the threshold, automatically adjust the weight ratio of the conflict area to voice:text = 55:45 to generate a "repressed excitement" transitional expression.

[0052] Intermediate filtering: Apply a Gaussian time-domain smoothing window to the mutated expression parameters, forcing the eyebrow movements to transition at a gradual rate of 6 frames per second to avoid the mechanical feeling of "expression jumping".

[0053] Ultimate fault tolerance: When the continuous conflict exceeds 3 seconds, activate the emotional inertia decay algorithm, and the expression intensity on the upper half of the face decays exponentially at a rate of 2% per frame, and finally returns to the basic state dominated by the voice intonation, conforming to the biological law of the natural dissipation of human emotions.

[0054] This arbitration mechanism can reduce the emotional misjudgment rate to 3.1%.

[0055] To improve the system robustness, the fusion algorithm can automatically degrade to single-modal driving when one modality is missing or abnormal. If the text input is missing, the system will automatically set the gating coefficient to 1, making the fusion result completely rely on the speech features to drive the facial animation. At this time, although the text supplement is missing, the system can still generate basically credible expressions through the prosody and emotional tones of the speech. In the case of missing speech signals or poor audio quality, it is set to approach 0, and the system mainly uses text features to drive the expression changes. For moments without speech, if there is still content in the text, the system slowly changes the facial expression according to the text features; on the contrary, if the text is missing and the audio is silent, the output maintains a neutral expression to avoid generating unnecessary actions. In addition, the fusion unit has clamping and filtering mechanisms for input outliers. For example, if abnormal jitter of speech features or mutation of text features is detected, the weight of this modality will be temporarily reduced or the features will be smoothed to prevent the propagation of incorrect information to the output sequence. In summary, these fault tolerance strategies ensure that the system can operate smoothly regardless of whether a single modality is missing or not, and output a reasonable and natural sequence of facial animation features.

[0056] The output of the fusion unit is a sequence of temporally synchronized facial action feature vectors. This sequence aggregates information from both speech and text and is directly used to drive the face key-point animation prediction of the virtual digital human. Each time step corresponds to a -dimensional feature vector. The value of depends on the selected facial action representation method. For example, in the implementation based on 2D key-point driving, the feature vector contains the displacement or absolute coordinate position of the main key points on the face. If the face defines key points, and each key point is represented by two-dimensional coordinates , then each frame of features contains the offset or absolute coordinate position of these key points relative to the reference position, with a total of dimensions. For example, taking the typical 68 landmark points of the face as an example, each frame contains 68 pairs of two-dimensional coordinates, with a total of 136-dimensional features, which are used to depict the position state of each part of the face at that moment. These key-point features not only reflect the lower-face movement driven by speech but also superimpose the upper-face expressions driven by text. In other implementations, the features can also be represented as a set of facial action unit parameters or BlendShape coefficients, and their dimensions correspond to the number of predefined expression basis vectors, such as the sum of several mouth shape parameters and expression parameters. Similarly, this fused feature sequence can be input into the subsequent animation rendering module to map each frame of feature vector to the specific change in the position of the face key points, thereby driving the 2D avatar image to generate continuous expression animations.

[0057] In summary, the implementation logic of the multi-modal fusion algorithm generates a unified facial action feature sequence through the extraction and alignment of speech features and text features, a fusion mechanism that combines attention and gating, and conflict tolerance processing. This sequence fully depicts the facial action state of the virtual digital human at each time frame, providing precise and coherent control signals for the image-driven digital human facial animation synthesis.

[0058] 2. The animation synthesis module takes the fusion-driven feature sequence as input and generates frame-by-frame face images output in time series. Specifically, the input includes a static source human face image and a sequence of facial action parameters of this person varying over time. These inputs jointly drive the generation of the face animation. The output is a sequence of face image frames corresponding to each time step. Playing these frames continuously can form the facial animation video of the target person. Through the above input / output structure, the mapping generation from 2D image drive signals to dynamic facial expressions is realized.

[0059] Phase 1: Adopt a local image deformation method based on face key points to distort and deform the source static face image according to the key point positions in the drive sequence to obtain preliminary frame-by-frame face images. First, extract the key point coordinates of the target face for each frame of drive data and establish a mapping relationship with the corresponding key points of the source face. The key point coordinates are usually normalized to a unified coordinate system. For example, the coordinates are normalized using the width W and height H of the source image to ensure consistent data interfaces at different resolutions. Then, calculate the deformation mapping function based on the position differences between the source image and the target key point set. This mapping can be obtained through interpolation methods such as local affine transformation or thin plate spline, enabling each local area of the source image to smoothly migrate to the target position. For example, let the position of the source key point set be , and the position of the key point set of the target frame be , then the displacement vector of each key point is . Calculate the displacement vector for each pixel position through a weighted interpolation function to obtain the displacement field , where: Equation (5); In the formula, is the displacement field; is the number of key points; is the influence weight of the th key point for the pixel ; is the displacement vector of the th key point. Subsequently, apply this displacement field to the source image to complete the distortion and deformation, that is, move the pixels in the source image according to Map to a new position. Through this key-point-driven local deformation, the expression and pose of the source face gradually approximate the target shape of the driving sequence frame by frame, obtaining a preliminary synthesized sequence of face images. This sequence retains the basic appearance features of the source person and the large changes in the expression contour, but there may be distortions or blurs in the detailed texture.

[0060] While continuously generating facial expressions and lip-sync animations, the head pose perturbation unit is responsible for fine-tuning the head movement of the virtual human. In real life, people's heads rarely remain completely stationary when speaking. By adding perturbations based on the micro-motion model, the head pose of the virtual digital human has slight dynamic changes. On the one hand, the system continuously monitors the change of the virtual human's head pose over time in the animation. If it is found that the head remains almost stationary for a period of time (for example, maintaining the same pose for dozens of consecutive frames), the perturbation unit will apply a small random pose change to make the head swing slightly, simulating the subconscious movement when people speak for a long time. On the other hand, the perturbation module combines the rhythm and stress information of the speech. When detecting a stressed word or intonation emphasis in the speech, it drives the virtual human's head to make small nodding or tilting movements synchronized with the speech stress. For example, when the virtual human asks a question, the system can make its head slightly rise and tilt as the intonation increases, to match the raised eyebrows expression and show a questioning look. The amplitudes of these head micro-movements are carefully controlled, neither overshadowing the main facial expressions and lip-sync nor making the virtual human remain rigid for a long time, thus enhancing the vividness and realism of the overall animation. When watching, the audience will feel that the virtual human has more "vitality" rather than being an unchanging puppet.

[0061] After generating the fused facial animation sequence and adding appropriate head micro-movements, the system will obtain a series of preliminary rendered virtual human face animation frames. These preliminary animations already contain the change information of mouth shape, expression, and head pose. However, in order to further approximate the visual effect of real people, Stage 2: To enhance the realism and detail accuracy of the image, the system further introduces a generative adversarial network (GAN) to refine the image based on the preliminary key-point deformed image. The generator used has an encoder-decoder structure, and the key-point position information is used as a conditional input to refine the texture details in areas such as the lips, eyes, and eyebrows. The discriminator is used to evaluate the authenticity of the generated image. During training, a combination of adversarial loss (such as cross-entropy loss), reconstruction loss (such as L1 or L2 norm), and perceptual loss (such as the difference in the VGG feature space) is adopted, and the three are weighted and combined to form the final optimization target. This strategy effectively compensates for the defects of traditional geometrically deformed images in detail fidelity and improves the visual quality of facial animations. Specifically, it includes: The generative adversarial network is introduced to refine the preliminary face image sequence generated in the first stage. This stage includes a generator network and a discriminator network. The generator takes the distorted face image sequence obtained in the first stage as input and outputs a refined high-definition face image sequence . Preferably, the generator adopts an encoder-decoder convolutional neural network structure and fuses the driving feature conditions to reconstruct the detailed texture. For example, by providing the target key point heatmap or displacement field together with the input image to the network, the generator can perceive the expression change area. The discriminator, as part of the adversarial training, plays a game with the generator and is used to discriminate the authenticity of the output image. To ensure stable training and realistic details, a comprehensive loss function is designed in the GAN refinement stage, including a weighted combination of adversarial loss and reconstruction loss. For example, let the discriminator be , and the generator be , then the adversarial loss adopts the traditional cross-entropy form: Equation (6); In the formula, is the mathematical expectation; is the real image sample; is the distribution of the real image; is the output probability of the discriminator for the real image ; is the random noise vector; is the distribution of the noise vector; is the fake image generated by the generator from the noise ; is the output probability of the discriminator for the generated image .

[0062] The reconstruction loss uses the L1 norm or the L2 norm to measure the pixel difference between the generated image and the reference true value image, expressed as , in the formula, is the output of the reconstruction loss; is the high-resolution image output by the generator; is the real image; is the L1 norm, measuring the pixel-level difference. At the same time, the perceptual loss can be introduced, that is, comparing the difference between the output and the true value in the high-level feature space of the pre-trained classification network to improve the realism of the synthesized face details. The above losses are weighted to form the total loss function: Equation (7); In the formula, is the total loss; , , are the weight coefficients of each loss term; It is a perceptual loss. By adjusting these coefficients to balance the adversarial loss and the reconstruction accuracy, the final training objective of the generator is to minimize it. After training is completed, the GAN refinement unit can map any distorted face frame into a realistic face image frame, increasing skin texture details and correcting the distortion caused by interpolation deformation, making the output image closer to a real face. The sequence of per-frame images processed by the GAN refinement can form the final high-fidelity virtual face animation output.

[0063] The implementation of the above two-stage method relies on clear and standardized data interfaces and parameter standards. To ensure the efficient cooperation and replacement flexibility among system units, a standardized definition is made for the input and output data structures. The key-point sequence is represented by a two-dimensional array, with each frame containing 68 feature points, arranged in a unified order and normalized to the interval [0,1]. The order and meaning of the key points are clearly defined in the interface to ensure a one-to-one correspondence between the driving feature sequence and the deformation algorithm. Additionally, in terms of the image data standard, the source face image and the output frame are usually agreed to have a unified resolution and color format, so that the input and output dimensions of the generator network are fixed, facilitating model training and deployment. For example, the image frame resolution is fixed at 512×512, using the RGB three-channel format. If an external face key-point detector is used to provide the driving sequence, the reference coordinate system of the key-point coordinates needs to be agreed in the interface, and appropriate normalization conversion of the input key points should be performed before use. Through strict interface specifications, each unit can cooperate in a decoupled manner: the fusion unit outputs a standardized key-point sequence, and the image deformation unit can generate animation frames according to the established algorithm after receiving it, without caring about the original detailed units of feature extraction. Such a design ensures data compatibility when system units are replaced or upgraded. Each unit follows the standard interface protocol for data interaction, enhancing the modularity and scalability of the system.

[0064] As Figure 2 shown, the 2D image-driven face synthesis process is divided into two main stages: first, local image deformation based on key points is performed, and then high-definition frames are generated through GAN refinement. In the figure, the arrows between the units represent the data flow: number 1 represents the input source face image, and number 2 represents the driving face key-point sequence; both enter the local image deformation unit 3 to generate an intermediate sequence of distorted face frames; this intermediate result is then sent to the GAN refinement unit 4 to produce a clear and realistic output face frame sequence. The entire process combines the input key-point driving information with the appearance of the source face to generate high-quality face animation images frame by frame, achieving the purpose of driving the expression changes of the target face with 2D images.

[0065] 3. Combining the above system architecture, the method flow of the present invention includes the following steps in chronological order, as Figure 3 shown: Step 1: Speech / text collection.

[0066] This step obtains the original speech data and / or text data for driving the facial animation of the digital human. Depending on the application scenario, the system can accept speech input, text input, or multimodal input that combines both. During speech collection, the speech signal of the speaker is recorded through a microphone and converted into a digital audio sequence after analog-to-digital conversion. To ensure the collection quality, a sampling rate of 44.1 kHz or 48 kHz and a quantization precision of 16 bits are preferably used to ensure clear speech and retain details. During text collection, the text corresponding to the speech content is obtained, for example, directly input by the user or read from a pre-prepared script. After obtaining the text, basic preprocessing is performed, such as removing redundant whitespace, standardizing character encoding, and retaining punctuation marks for subsequent sentiment analysis.

[0067] When both speech and text are provided, it is preferred to obtain the corresponding transcribed text while collecting the speech signal. One way is to use a speech recognition model to transcribe the recorded speech into text for sentiment analysis in real time; or when there is a ready-made script, the script can be directly used as text input. The system correlates the collected speech data with the text data to ensure that the content of the two is consistent and corresponds to the same passage. For the convenience of subsequent multimodal synchronization processing, both speech data and text data are appended with timestamp information: the speech data records the start time and total duration; the text data records the total duration of the corresponding speech or the time position of each word. If the time alignment information is temporarily missing, it can also be inferred by an algorithm later. By unifying the time reference, the features of speech and text can be accurately aligned during subsequent fusion.

[0068] Output: The original speech signal and / or text content. The speech is represented as a continuous sequence of digital audio samples, and the text is represented as a string or a sequence of words, and they are stored separately for subsequent units to read, along with metadata such as the sampling rate and duration.

[0069] Boundary case handling: If there is only text without speech input, the text can be converted into speech audio through text-to-speech outside the system first to drive the lip movement, or the text features can be directly used to drive the expression change without outputting speech. Conversely, if there is only speech without text, the text recognition step is skipped, and the animation is directly driven by the speech features. If the input speech quality is poor, noise reduction processing is superimposed during the collection stage to improve the accuracy of subsequent feature extraction. For overly long input speech, it can be collected and processed in segments, and overly short speech segments are buffered and waited until they accumulate to a certain length before processing to ensure that subsequent units have sufficient data for analysis. Through these measures, the data acquisition module can robustly provide high-quality and synchronously aligned raw data.

[0070] Step 2: Speech feature extraction.

[0071] This step extracts parameters reflecting the speech content and intonation features from the collected speech data, which are used to drive the lip movements and speech-related expressions of the characters in the picture. First, preprocess the digital speech signal, such as pre-emphasis filtering to enhance the high-frequency components and suppress the DC component, thereby enhancing the speech clarity. Then, segment the speech signal into frames of a fixed length: for example, each frame is 20 - 30 milliseconds long, and the frame shift is about 50% of the frame length, obtaining a series of continuously overlapping short speech frames. Multiply each frame of speech by a window function to reduce the spectral leakage caused by the discontinuity at the frame boundaries. Next, perform a fast Fourier transform on each windowed frame of speech to calculate its spectrum, obtaining a sequence of speech spectra that changes over time, where the horizontal axis is the time frame index and the vertical axis is the frequency. This spectrum contains information such as the fundamental frequency and formants of the speech, which is the basic representation for subsequent feature extraction.

[0072] To obtain more compact and robust acoustic features, this step preferably calculates parameters such as Mel-frequency cepstral coefficients or Mel power spectra. Specifically, the power spectrum of each frame of speech is superimposed through a set of Mel filters, converting the linear frequency axis to the Mel scale that conforms to the human auditory perception, and taking the logarithm to compress the dynamic range. Subsequently, apply the discrete cosine transform to the logarithmic Mel energy spectrum and take its low-order coefficients to form a cepstral feature vector. Usually, around 13-dimensional cepstral coefficients are taken to characterize the short-term acoustic characteristics of the speech. If more detailed information needs to be retained, the Mel power spectrum can also be directly used as a feature. Through the above frequency-domain feature extraction, clues about the pronunciation content and a certain degree of intonation and prosody information in the speech signal are captured.

[0073] Model the temporal dependencies of the speech features and text features through a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters; the control parameters simultaneously encode the lip movement changes of the lower half of the face driven by speech and the expression changes of the upper half of the face driven by text. When generating the part of the control parameters corresponding to the lip movement changes of the lower half of the face, increase the contribution weight of the speech modality; when generating the part of the control parameters corresponding to the expression changes of the upper half of the face, increase the contribution weight of the text modality.

[0074] In a preferred embodiment, a deep learning model is further used to perform high-level representation learning on traditional spectral features to extract richer speech embedding features. For example, input the Mel spectrogram of each frame or the features after splicing several adjacent frames into a pre-trained speech coding network. This encoder uses a convolutional neural network to capture local spectral patterns, or a recurrent neural network to capture time series correlations, or a self-attention model such as a Transformer to obtain long-range dependencies. Through the non-linear transformation of the deep network, a high-dimensional speech embedding vector corresponding to each time frame can be obtained. These deep embedding features contain higher-level speech information and are more robust to noise interference.

[0075] After the above processing, this step outputs a sequence of speech features arranged in chronological order of time frames. If the frame shift is 10 ms, this sequence is approximately 100 frames per second, and each frame contains a set of speech feature parameters. These features will serve as the basic driving information for subsequent multimodal fusion and expression generation.

[0076] Boundary case handling: There may be silent segments or pauses in the speech signal. Silent frames are detected and marked through an energy threshold to distinguish them from frames with vocalizations. For silent frames, a feature vector of the same length as normal frames is still output, but its values can be set to all zeros or a predefined "silent" vector to indicate that there is no pronunciation content at this time while maintaining the continuity of the time axis. For speech with high background noise, noise reduction measures can be added before and after spectrum extraction, such as filtering out abnormal peak frequency bands or reducing the noise component through an autoregressive model, to ensure that the extracted features more accurately reflect the speech itself. If the input audio amplitude is too large resulting in clipping distortion, amplitude normalization needs to be performed during acquisition or processing to prevent the generation of abnormal feature values. In extreme cases, such as when the speech duration is too short to complete a frame analysis, resampling can be repeated to supplement the frame length or wait until enough speech length has been accumulated before performing feature extraction. If the speech duration does not match the expected animation duration, the system continues to extract features at a unified time step: the excessive part can be truncated or otherwise processed later, and the too short part can be extended frame by frame through interpolation later to match the video length. Since each speech feature frame carries a timestamp, subsequent units can accurately align the speech modality information with that of other modalities based on this.

[0077] Step 3: Text recognition.

[0078] This step performs semantic understanding and sentiment analysis on the input text content, extracts high-level semantic / sentiment features, and maps them onto the time axis to guide facial expression changes. The role of the text recognition module is to infer the speaker's emotional attitude or tone from the language content so that the corresponding expression elements can be reflected in the generated facial animation. The input is the text data obtained in Step 1. The text can be a string of a whole sentence or multiple sentences, and may also have been tokenized or have the word boundaries marked. First, necessary preprocessing is performed on the original text, including tokenization of Chinese text, unifying case, removing special symbols irrelevant to semantics, etc. At the same time, the end-of-sentence punctuation is retained because punctuation often conveys tone pauses and emotional information.

[0079] The cleaned text is input into a semantic analysis model to obtain its semantic representation and preliminary sentiment tendency. A pre-trained deep language model is used to encode the text. For example, the BERT model is utilized to extract the vector representation of each word in the context, or a GPT-like Transformer model is adopted to generate the implicit semantic vector for the entire text segment. Such models are trained based on a large amount of corpus and can map the text into high-dimensional vectors, containing semantic content and context information. On this basis, to obtain sentiment information, sentiment classification or regression can be performed on the semantic vectors. One approach is to design a sentiment recognition sub-network to output the probability distribution of sentiment categories or the sentiment intensity value for the semantic vectors. Another approach is to use sentiment lexicons or rule matching for auxiliary judgment: for example, when emotion words such as "happy" and "angry" are detected in the text, corresponding sentiment labels are assigned. Preferably, the sentiment classification model is trained in an end-to-end manner, making judgments based on the comprehensive context rather than relying solely on individual emotion keywords, thereby improving the accuracy.

[0080] After obtaining the sentiment features of the text, it is necessary to map them to the speech timeline to form a sentiment sequence that evolves over time, which is used to guide dynamic expressions. Ideally, if both speech and text are provided simultaneously and their contents match, their alignment information can be utilized to map the text to the corresponding speech process according to time segments. For example, the start and end times of each word in the speech are obtained through forced alignment algorithms or speech recognition results, and based on this, the sentiment label or sentiment vector of each word is applied to the corresponding time interval. More specifically, let the start time of a certain semantic segment in the speech signal be t0 and the end time be t1. Then, within this time interval, the sentiment features can be regarded as the constant sentiment vector of this segment, or appropriate transitions are made at the boundaries. To make the sentiment features change smoothly over time, linear interpolation is used to connect at the junctions of adjacent segments: for example, between the end of a segment and the start of the next segment, transitional sentiment values are inserted by weighting according to the time distance to avoid sudden changes in sentiment parameters. If more refined speech alignment information can be obtained, sentiment values can also be assigned at the sub-word and syllable levels, enabling the sentiment features to change within a corresponding extremely short time window.

[0081] In the absence of precise alignment information, a rough temporal mapping is performed according to empirical rules. For example, the emotional features obtained from the analysis of the entire sentence text are regarded as the main emotion during the entire speech period, assuming that the emotion remains the same throughout the sentence; if the text contains multiple sentences, the total speech duration can be evenly divided by the number of sentences, and each sentence is assigned an emotional feature value. For the case where there is a lack of finer-grained information within a single sentence, it is assumed that the emotional value of the sentence remains unchanged throughout the duration of the entire sentence. When necessary, the pause duration can also be inferred based on punctuation marks. For example, a short pause is assumed at the end of a period or a comma. These rules ensure that an approximately reasonable sequence of emotions changing over time can still be generated when the information is incomplete. After the above temporal mapping is completed, a sequence of text features indexed by time can be obtained. Usually, the time step of this emotional sequence is adjusted to be consistent with the speech feature sequence for subsequent frame-by-frame fusion with the speech features.

[0082] Step 4: Multimodal feature fusion.

[0083] In this step, the speech feature sequence obtained in Step 2 and the text feature sequence obtained in Step 3 are fused to generate a unified driving feature, which is used to comprehensively utilize the information of both speech and text modalities to guide the generation of facial animations, as Figure 4 shown. The goal of fusion is to superimpose the expression changes indicated by the text semantic emotion while ensuring lip synchronization with the speech, so that the facial animation of the digital human can not only accurately match the lip movements but also vividly reflect the speaking emotion. Input: speech feature sequence and text feature sequence. First, it is necessary to ensure the alignment of the two sequences on the time axis. As described above, if the time resolution of the emotional feature sequence is lower than the speech frame rate, the emotional sequence can be interpolated and extended: for example, linear interpolation is performed between every two known emotional data points to generate emotional feature values corresponding to the speech frames. In special cases where the emotion only changes at a few key time points, the emotional feature can be approximately regarded as a constant function within this interval. After time alignment and interpolation processing, an emotional feature sequence that corresponds one-to-one with the speech features and has the same length can be obtained.

[0084] Two types of strategies are adopted: simple fusion or advanced fusion that introduces an attention mechanism. In simple fusion, the speech feature vector and the text feature vector corresponding to each time step are directly concatenated into a longer vector, and then mapped to the required dimension through a series of fully connected layers to obtain the fused feature. For example, a fully connected network f(·) is defined to map the concatenated vector: Equation (8); where is the fused feature vector, and its dimension is determined by the network design; is a non-linear activation function; is a trainable weight matrix; is the time step The aligned text feature vector; is a trainable bias term. Through this concatenation mapping, the neural network automatically learns the interaction relationship between speech and text features.

[0085] Preferably, the fusion unit introduces an attention mechanism to dynamically weigh the contributions of the two modal information. One implementation is to add a gating factor on the basis of the above simple concatenation: calculate a time-varying weight coefficient (taking values from 0 to 1) to balance the proportion of speech and text features, and generate a fusion vector accordingly. For example, define as the gating weight at the current time step, then the fusion output is expressed as , which is consistent with the expression in Equation (2) above. The weight itself is obtained by another small neural network according to the current and predicted in real time, or directly obtained as an additional parameter and trained together with the main network. In this way, in the scenario mainly driven by speech for lip movement, the network automatically learns to adjust to a value close to 1, making the speech features account for the main proportion; while at the moment when facial expressions and emotions need to be expressed, the text features are allowed to play a greater role.

[0086] Another attention fusion method is the cross-attention mechanism: using the speech feature sequence as the query, the text sequence as the key and value, and calculating the multi-head attention to obtain the fusion representation. This method refers to Equation (1) and related descriptions, that is, the model can select relevant information from the entire text sequence according to the current speech content and integrate it into the output. When the text implies that there should be a specific expression somewhere but the speech features do not clearly show it, the attention mechanism introduces this additional information to make up for the deficiency of a single modality.

[0087] The fused features carry both the low-level information of the current pronunciation and the high-level information of the speaking tone, serving as the comprehensive features to drive the subsequent animation. Output: The fused feature sequence. The sequence length is the same as the input speech feature sequence, and each is a fused feature vector.

[0088] In the scenario where there are conflicts in multi-modal information, to ensure the naturalness of facial expressions, the system sets the modal priority rules: the speech modality has a higher priority in driving the opening and closing of the mouth to ensure the strict synchronization of speech and mouth shape; while the text modality has a higher weight in the emotional expression of the upper half of the face (such as eyebrows and eyes), thereby enhancing the richness and semantic consistency of facial expressions.

[0089] Boundary case handling: The fusion unit needs to adapt to the possible absence of different modality inputs. When there are only speech features without text, the emotion features are set to an all-zero vector or a fixed vector representing neutral emotion, making the fusion result almost entirely determined by speech, ensuring that lip synchronization is not affected. Conversely, when there is only text without speech, the speech features are set to blank. In this case, the fusion output mainly depends on the text features. By making extreme treatments on or the attention weights in the implementation, it is ensured that the system can still output results smoothly when degenerating to single-modal driving, without errors occurring.

[0090] Step 5: Key point prediction.

[0091] In this step, based on the fused feature sequence, the position sequence of the face key points required for the digital human facial animation and the head pose parameters are predicted. Here, the key points usually refer to the two-dimensional coordinates of a series of representative feature points on the human face, used to depict detailed expressions and lip shape changes; the head pose parameters represent the rotation angles of the head, used to reflect head movements. Input: The fused feature sequence output by Step 4. A time series model is used to fit the mapping relationship from the fused features to the face key points. After training, the model can output the position of the face key points in the next frame based on the fused features of the current and several previous frames. Considering the continuity of facial expressions and lip shape changes, the model will learn a smooth path from the key points at the previous moment to the key points at the current moment. In addition to end-to-end learning, it can also be implemented step by step: for example, first train a regression model to predict the key point displacements or key point positions of each time frame, and then adjust the global coordinates according to the overall head movement. Output: The face key point sequence and the head pose sequence. To avoid jitter in prediction from causing unstable animation, a smoothing filter is applied to the output key point sequence to remove high-frequency jitter. At the same time, ensure that the change rate of the key points and poses does not exceed the physically reasonable range, so that the facial movements are smooth and natural.

[0092] Boundary case handling: During the prediction process, if the input features are missing or abnormal, the prediction model will tend to output neutral positions or maintain the positions of the previous frame, avoiding severe distortion due to abnormal fused features. When deploying the model, rules can be introduced: such as restricting the maximum value of the mouth opening angle to prevent excessive distortion of the lip shape; setting upper and lower limits for the eyelid blinking frequency, etc., to be close to the statistical laws of real human expressions.

[0093] Step 6: Animation frame generation.

[0094] This step generates the digital human face image frames at the corresponding moments based on the face key point sequence and head pose parameters output in step 5. Its core includes two stages: local deformation of the reference face and generative refinement. Input: the smoothed key point positions, head pose, and the reference face model or reference image of the digital human. Here, the reference image usually selects a high-definition face image of the digital human in a neutral expression and facing forward, or the initial frontal face frame rendered from the 3D digital human head model. At the same time, the set of face feature point coordinates in this reference state is required as the deformation reference.

[0095] First, for each time frame, according to the differences between the currently predicted face key points and the reference key points, calculate the affine transformation parameters or displacement fields of each local area. Apply these transformations to the reference face image to obtain the initially rendered distorted image. At the same time, according to the head pose parameters, rotate and translate the entire image to simulate the influence of head movement on the image.

[0096] Then, send the sequence of distorted images into a pre-trained generative refinement model to output a sequence of high-quality face images. This GAN model repairs the initial images frame by frame, enhances texture details, and improves the authenticity. Output: a sequence of high-definition virtual face animation frames. These frames can be directly used to compose the final video.

[0097] In the system training stage, the GAN refinement unit uses real face video frames as the supervision target, uses the key point-driven synthesized frames aligned with the real frames as the input and labels, and optimizes the generator through adversarial learning to improve the image authenticity. At the same time, introduce perceptual loss to improve the detail fidelity, making the model closer to the actual movement of the human face.

[0098] Boundary case handling: If the GAN refinement network generates artifacts or unstable outputs in some frames, the quality of adjacent frames can be detected and interpolation correction can be performed. For example, when it is detected that the brightness or color distribution of the output frame is abnormal, the previous and next frames can be used for linear interpolation to replace this frame to ensure the smoothness and consistency of the final sequence. In addition, for expressions that are extremely beyond the training sample distribution, GAN may not be able to generate realistic details. In this case, the system can degrade and use the original distorted frames to avoid serious distortion.

[0099] Step 7: Animation rendering and audio synchronization.

[0100] This step synthesizes the previously generated sequence of face image frames with the original speech audio, renders a complete animated scene, and ensures synchronous output of sound and video. The final result is a digital human speaking video file with sound, or an audio-video stream for synchronous playback in a real-time interaction scenario. Input: The sequence of image frames output from Step 6, and the original speech audio captured in Step 1. Before entering the rendering synchronization stage, these frames have been corresponding to the speech feature frames at a time step of 10 milliseconds or finer, so the quantity strictly matches the audio duration. The system first organizes the image frames at a constant frame rate, and at the same time resamples or frames the audio to align it to the same timeline. Then each frame of the image is associated with the audio of the corresponding time segment, and the video and sound are synthesized in the video container. Rendering uses a standard video encoding library to compress the sequence of frames into a video stream. In a real-time output scenario, the image may be directly overlaid on the background and the audio played using a graphical interface or engine. Finally, the animated scene of the digital human speaking in real time is obtained. Whether it is an offline file or an online stream, both ensure audio-visual synchronization and high-quality presentation.

[0101] Output: The animated scene of the virtual digital human speaking after completing audio-video synthesis. The common format is usually a video file such as MP4, or it is output to the display end through a streaming protocol. This output can be viewed by users or used as material for other downstream systems.

[0102] Boundary case handling: If it is detected during the animation rendering stage that the frame rate cannot keep up with the real-time audio playback, some frames can be discarded or the time scale of the audio can be micro-adjusted to realign the synchronization. At the same time, necessary metadata such as the time code information of the video and the sampling rate information of the audio are added during the packaging output to ensure strict synchronization in various playback environments. For the case of pure text-driven but also requiring sound output, TTS is called to convert the text to speech for audio synthesis. For the pantomime scenario without sound, only the video frames are output and the audio stream can be omitted. Through flexible synchronization control, the animated scene of the virtual human speaking with highly matching sound and expressions is finally achieved.

[0103] Since the processing of each unit follows a unified timeline, ideally the sequence of image frames and the audio are already synchronized. For example, assuming that the duration of each frame is 0.04 seconds, then the nth frame of the image should correspond to the speech content in the nth 0.04-second interval of the audio. Therefore, the image sequence and the audio stream can be directly mixed and output after matching with time stamps. The audio segment is processed to obtain feature vectors, combined with emotional information to generate face key points, and then the image frames are rendered and output. By processing each continuous time step in this way, a sequence of animated frames arranged in chronological order is finally obtained, which is strictly synchronized with the input speech.

[0104] In actual implementation, if a slight mismatch between the image and the audio is detected, the system adopts a fine-tuning alignment strategy. For example, if the image frame lags slightly behind the audio at a certain moment, a very short segment of audio samples is skipped; conversely, if the image frame ends too quickly, a static image frame with a very short duration is inserted in the middle to realign the frame sequence with the audio. Additionally, before starting the rendering, the total number of frames to be generated can be calculated based on the total length of the audio and the predetermined frame rate. If the actual number of generated frames is more or less than the calculated value, it is corrected by deleting or duplicating the last frame at the end of the sequence to make the total duration exactly the same as the audio length. After the above calibration, the image sequence and the audio sequence are precisely synchronized on the time axis.

[0105] After the frame sequence and the audio are aligned, the system will perform the final rendering and composition of the images frame by frame. For example, an appropriate background image is selected and the face frame is superimposed on it; at the same time, the corresponding voice segment is embedded in the audio track of the video. Each time a frame of the image is generated and presented, the corresponding audio segment of that frame is played synchronously. In real-time applications, the streaming media protocol is used to generate and push the video and audio streams to the display end while generating; in offline applications, all frames and audio are synthesized into a video file in a standard format. Output: Finally, an animated video containing the continuous speaking scenes of the digital human is output. When output in file form, it is a packaged video and audio file; when output in real-time form, it is a continuously playing video and audio stream.

[0106] During the synthesis stage, abnormal situations such as mismatches in the length of the audio or video need to be considered. For example, if the generated image frame sequence ends earlier than the audio for some reason, the system keeps the last frame static at the end until the audio ends to avoid premature interruption of the video; conversely, if the audio ends first, the frame update can also be stopped or the last frame of audio silence can be repeated to align. Similarly, if there is a frame processing delay during the real-time streaming process, resulting in temporarily falling behind the audio, the video frame output frame rate can also be briefly reduced for smooth transition, and then restored to the normal frame rate after catching up with the time axis. Through these mechanisms, it is ensured that the final output animated images are strictly synchronized with the sound and are smooth and continuous.

[0107] 4. During system initialization, the pre-trained model parameters are preferentially loaded from the local. If it is detected that the local model file does not exist or the version does not match, the system will automatically obtain the latest model file from the remote server and cache it locally to ensure subsequent offline use. The model loading process includes the initialization of the model structure and weight parameters, loading the model into memory and completing the deployment on the device. To improve the loading efficiency, the system adopts direct memory mapping or zero-copy technology. When conditions permit, the model data is directly loaded from the memory reference to avoid repeated file read and write operations. For example, when the mobile platform only allows access to the model in memory form, the system embeds the model as a static resource into the program and directly loads it through memory mapping, avoiding time-consuming file read and write operations.

[0108] After loading the model, during the inference execution process, the system adopts refined memory management and multi-threaded asynchronous mechanisms to ensure real-time performance and stability. On the one hand, the system pre-allocates the required tensor and cache spaces, and reuses the intermediate result buffer to avoid frequent memory allocation and deallocation. For the intermediate data that is no longer needed after each frame is processed, it will be released in a timely manner to reduce the peak memory occupancy. This memory management strategy combined with the memory pool technology cycles through the already allocated memory blocks during the inference process to ensure that the memory occupancy remains stable and avoid increasing the memory burden due to the accumulation of frame processing.

[0109] On the other hand, the system utilizes multi-threading and asynchronous execution mechanisms to fully exploit the parallel capabilities of multi-core hardware. For example, different functions such as audio processing, feature extraction, expression fusion, and image generation are executed by independent threads or asynchronous tasks, and the producer-consumer queue is used to synchronize the data stream. The audio acquisition and feature extraction thread runs continuously and puts the extracted features into the buffer queue; another thread monitors the queue and retrieves the latest features for emotion fusion and image generation. Such an asynchronous pipeline design ensures that each processing stage works in parallel as much as possible: when one frame of the image is being rendered, the audio features of the next time frame are also being calculated in parallel, thus reducing the overall waiting time. For the part accelerated by the GPU, the system uses asynchronous GPU compute streams to concurrently execute kernel computations and data copies whenever possible, further improving the throughput.

[0110] When loading the model, the system also takes into account the cross-platform deployment differences and selects an appropriate platform inference engine for loading. The system supports the use of mainstream cross-platform inference frameworks such as ONNX Runtime, TensorRT, CoreML, NCNN, etc. to make full use of the hardware acceleration capabilities of different platforms. Specifically as follows: On Windows or Linux desktops, when an NVIDIA GPU is detected to be available, the TensorRT engine is preferentially used for model loading, and the inference speed is greatly improved through TensorRT optimization; if there is no dedicated GPU, ONNXRuntime is used to call the local CPU or other backends for inference.

[0111] On the Android platform, the system loads the converted model through the NCNN engine, which is extremely optimized for ARM processors and uses NEON instructions and multi-core parallelism to accelerate the model operation. At the same time, for devices that support Android NNAPI or Vulkan, NCNN calls for hardware acceleration.

[0112] On the iOS platform, the system converts the model into CoreML format and utilizes the iPhone's Neural Engine to accelerate the inference, providing better energy efficiency.

[0113] By encapsulating a unified loading interface, the system can automatically select the best inference framework and model format for loading according to the running environment. If the loading of a certain engine fails, the system has a fallback mechanism. For example, when TensorRT fails to load, the system will fallback to execute using the default backend of ONNX Runtime to ensure the continuous availability of the system functions; on mobile devices, if the hardware acceleration fails, it will fallback to the CPU computing mode. The entire loading logic ensures that the model can be correctly initialized regardless of device differences and can provide alternative solutions to ensure the normal operation of the core functions.

Claims

1. A method for generating facial animations of virtual digital humans driven by multi-modalities, characterized in that, Including: Obtain voice input and text input, and extract voice features and text features; Fuse the voice features and text features to obtain control parameters for driving the facial animation of a virtual digital human. The control parameters include facial expression parameters and head pose parameters, and adopt a driving strategy with different facial upper expressions and lower facial expressions to dynamically adjust the contribution weights of the voice modality and text modality to the control parameters; Perform local deformation on the facial image of the virtual digital human according to the control parameters to generate an initial facial animation; Use a generative adversarial network to refine the initial facial animation to obtain a refined facial animation.

2. The method according to claim 1, wherein: Obtain the timestamp of the text feature in the voice signal through time alignment, and extend the text feature to be consistent with the frame rate of the voice feature by linear interpolation or replication to form a temporally synchronized text feature sequence.

3. The method according to claim 1, wherein: The fusion of the voice features and text features uses an attention mechanism to achieve information interaction between the two, and dynamically adjusts the contribution weights of the voice modality and text modality to the control parameters through a gating mechanism.

4. The method according to claim 3, wherein: When there is a conflict between the driving information carried by the voice input and the driving information carried by the text input, perform smoothing filtering on the generated facial animation.

5. The method according to claim 1, wherein: Perform time-dependent modeling on the voice features and text features through a long short-term memory network or a Transformer architecture to predict smooth and continuous control parameters.

6. The method according to claim 5, wherein: The control parameters simultaneously encode the mouth shape changes of the lower half of the face driven by voice and the expression changes of the upper half of the face driven by text; The voice-driven mouth shape changes of the lower half of the face include: obtaining the voice signal to be broadcasted, extracting audio features from the voice signal to obtain a time series of voice features; generating corresponding mouth movement parameters based on the voice features, and driving the mouth shape movement of the lower half of the virtual digital human's face to generate a preliminary lower half face animation; The text-driven expression changes of the upper half of the face include: obtaining the text content corresponding to the voice signal, performing semantic parsing and emotion recognition on the text, and extracting text emotion features; generating corresponding facial upper half expression parameters based on the emotion features, and driving the expression changes of the upper half of the virtual digital human's face to generate a preliminary upper half face animation.

7. The method according to claim 6, wherein: It further includes an adaptive multi-head fusion strategy for dynamically adjusting the weight distribution of each attention head of the attention according to the emotion intensity of the voice signal or the context change of the text content; The adaptive multi-head fusion strategy dynamically adjusts the weights of multiple attention heads of the cross-attention by monitoring emotion intensity parameters such as pitch and volume of the voice modality and the context semantic changes of the text modality, so as to assign a higher weight to the voice modality when the voice emotion is strong, and increase the weight of the text modality in a flat context.

8. The method according to claim 1, characterized in that: Based on the initial full-face animation, a head pose perturbation feedback mechanism is introduced to apply subtle random or regular perturbations to the head pose parameters of the virtual digital human, simulating the inadvertent micro-movements of the human head. The head pose perturbation feedback mechanism detects the changes in the head pose of the virtual digital human in consecutive animation frames. If it detects that the head remains stationary for a long time, it applies subtle random perturbations; or generates synchronized small head movements according to the rhythm and stress of the voice signal.

9. The method according to claim 1, characterized in that: The generation of the facial animation includes: first performing local image deformation on the facial image of the virtual digital human to generate the initial facial animation, and then using a generative adversarial network to refine and optimize the initial facial animation.

10. A multi-modal driven virtual digital human facial animation generation system, characterized in that: When the system runs, it executes the method according to any one of claims 1-9. The system includes: A feature extraction unit for obtaining voice input and text input and extracting voice features and text features. A fusion unit for fusing the voice features and text features to generate facial animation control parameters, where the control parameters include facial expression parameters and head pose parameters, and an attention mechanism and a gating mechanism are used to control the fusion process of the voice features and text features. An adaptive fusion control unit for dynamically adjusting the multi-head attention weight allocation in the fusion unit according to the change in the emotional intensity of the voice or the change in the context of the text. A head pose processing unit for determining the head pose change of the virtual digital human according to the control parameters. A head pose perturbation unit for performing micro-movement modeling on the head pose in the data of the initial full-face animation, generating subtle head pose perturbations and feeding them back to the animation data. An image deformation unit for locally deforming the facial image of the virtual digital human according to the control parameters to generate the initial facial animation. A GAN refinement unit for performing refinement processing of the generative adversarial network on the initial facial animation. A smoothing processing unit for performing smoothing filtering processing on the facial animation when it detects a conflict between the voice-driven information and the text-driven information.

Images