Speech-driven 3D face animation generation method based on dynamic emotion perception and related equipment

Abstract

The embodiment of the application provides a speech-driven 3D face animation generation method based on dynamic emotion perception and related equipment, and belongs to the technical field of computer vision and digital human. The method comprises the following steps: extracting a semantic feature sequence and an emotion feature sequence of input speech; inputting the emotion feature sequence into an emotion dynamic processor to capture emotion dynamic changes in multiple time scales and output refined emotion features; inputting the semantic feature sequence and the refined emotion feature sequence into an emotion guide intensity gating unit to adaptively weight the refined emotion features and obtain context-aware scaled emotion features; and inputting the semantic features and the scaled emotion features after fusion into a decoder to generate a 3D facial animation sequence synchronized with the speech. The embodiment of the application solves the problems of emotion modeling staticization, emotion and content fusion conflict and lack of high-level semantic supervision in the prior art, realizes dynamic balance of emotion richness and lip accuracy, and generates facial animation with accurate emotion expression and high calculation efficiency.

Description

Technical Field

[0001] This application relates to the fields of computer vision, digital human and artificial intelligence, and in particular to a voice-driven 3D face animation generation method and related equipment based on dynamic emotion perception. Background Technology

[0002] With the rapid development of multimedia technology, the metaverse, and artificial intelligence, voice-driven 3D facial animation has become a research hotspot in the fields of computer vision and graphics. This technology aims to automatically generate highly realistic 3D facial motion sequences that match the input voice signal, and it has enormous application value and commercial potential in scenarios such as virtual digital humans, film and animation production, online conferencing, live streaming interaction, and human-computer interaction.

[0003] Existing technologies mainly fall into two categories: one is to map phonemes and rhythms to specific facial expressions through manual creation or complex rule mapping (such as the mapping between phonemes and visual lip movements), i.e., artificially fixed mapping between speech and face, which belongs to traditional 3D facial animation generation. However, this type of method requires a lot of manpower, and its effects are limited and inefficient.

[0004] Another type is the deep learning-based end-to-end framework. These methods typically utilize pre-trained speech encoders (such as wav2vec 2.0) to extract semantic features from speech and then convert them into displacement or expression coefficients of 3D facial vertices through a decoder. This approach primarily focuses on lip-sync and emotional expression. However, existing deep learning-based end-to-end methods have two fundamental limitations: 1) Emotion is treated as a static, holistic attribute, failing to capture subtle temporal changes—for example, the intensity of emotion naturally fluctuates with the tone of voice; 2) They inject emotional cues uniformly during generation, which often overemphasizes emotional guidance, potentially suppressing or distorting language-related facial, especially lip, movements. Summary of the Invention

[0005] The main objective of this application is to propose a voice-driven 3D face animation generation method, electronic device, storage medium, and program product based on dynamic emotion perception. This method can dynamically capture subtle temporal changes in emotions and adaptively adjust the intensity of emotion injection according to the voice content, thereby generating 3D facial animations that are rich in emotion and accurate in lip movements.

[0006] To achieve the above objectives, one aspect of this application proposes a voice-driven 3D face animation generation method based on dynamic emotion perception, the method comprising: Acquire input speech data and extract semantic feature sequences and emotional feature sequences from the input speech data; The emotional feature sequence is input into the Emotion Dynamic Processor (EDP) to obtain a refined emotional feature sequence. The EDP is used to capture the dynamic changes of emotions at multiple time scales in the emotional feature sequence and to reorganize the captured multi-scale emotional features. The semantic feature sequence and the refined emotional feature sequence are input into the Emotion Guidance Intensity Gated (EIG) unit. The emotional weight of each frame is dynamically generated in a context-aware manner. The refined emotional feature sequence is then weighted using the emotional weight to obtain a context-adaptive scaled emotional feature sequence. The semantic feature sequence is fused with the scaled sentiment feature sequence to obtain a joint feature representation; The joint feature representation is input into the decoder to generate a 3D facial animation sequence synchronized with the input speech data.

[0007] In some embodiments, the Emotion Dynamic Processor (EDP) includes a Multi-granularity Emotion Aggregation Module (MEA) and a Subspace Routing Module (SFR). The step of inputting the emotional feature sequence into the Emotion Dynamic Processor (EDP) to obtain a refined emotional feature sequence specifically includes: The emotional feature sequence is input into the multi-granularity emotional aggregation module MEA. Through a multi-head attention mechanism and multiple pooling operations of different time lengths, the feature representations of emotional features at the original granularity and at least one other time granularity are captured respectively. The feature representations of different granularities are then concatenated to obtain multi-granularity emotional features. The multi-granularity emotional features are input into the subspace routing module SFR for feature recombination, and the refined emotional feature sequence is output.

[0008] In some embodiments, before inputting the emotional feature sequence into the multi-granularity emotional aggregation module (MEA), the method further includes: Multi-granularity positional encoding (MPE) is applied to the emotional feature sequence. The MPE is obtained by superimposing and averaging multiple positional codes of different frequencies.

[0009] In some embodiments, the Emotion Guidance Intensity Gated EIG unit includes a cross-modal attention module and a gating module; The step of inputting the semantic feature sequence and the refined sentiment feature sequence into the sentiment-guided intensity-gated EIG unit to dynamically generate the sentiment weight for each frame in a context-aware manner specifically includes: The semantic feature sequence is used as the query term, and the refined sentiment feature sequence is used as the key and value term. The result is input into the cross-modal attention module to obtain temporal sentiment features related to the semantic context. The temporal sentiment features are input into the gating module to calculate the sentiment weight for each frame.

[0010] In some embodiments, the weighting of the refined emotional feature sequence using the emotional weights specifically involves: The emotional weight of each frame is element-wise multiplied with the feature vector of the corresponding frame in the refined emotional feature sequence.

[0011] In some embodiments, the decoder is a Transformer decoder; The step of inputting the joint feature representation into the decoder to generate a 3D facial animation sequence synchronized with the input speech data specifically involves: The joint feature representation is used as a query term, and the semantic feature sequence is used as a key and value term. These are then input into the Transformer decoder to generate the 3D facial animation sequence.

[0012] In some embodiments, the method further includes the step of constructing a total loss function to perform end-to-end training on the generative network; the total loss function is a weighted sum of at least two of the following loss functions: Reconstructing Vertex Loss Function This is used to minimize the vertex coordinate error between the generated 3D facial animation sequence and the real 3D facial animation sequence. Velocity loss function This is used to minimize the inter-frame motion speed error between the generated 3D facial animation sequence and the real 3D facial animation sequence, so as to reduce motion jitter during rendering of the generated 3D facial animation and ensure animation smoothness. Potential consistency loss function This is used to ensure that lip-reading features extracted from generated animations are consistent with text features extracted from speech in the latent space. Text consistency loss function It is used to evaluate the difference between the text sequence decoded from the generated animation and the real text sequence corresponding to the speech; Facial expression consistency loss function , used to minimize the cosine distance between the generated 3D facial animation sequence and the real 3D facial animation sequence in the expression embedding space defined by the pre-trained expression encoder.

[0013] In some embodiments, the total loss function also includes the reconstructed vertex loss function. The velocity loss function The potential consistency loss function The text consistency loss function and the expression consistency loss function .

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0017] Compared with the prior art, this application has the following beneficial effects: 1) Achieved frame-level dynamic emotion perception: Through the multi-granularity emotion aggregation and subspace routing mechanism in the emotion dynamic processor, this application can capture the subtle dynamic fluctuations of emotion in speech with changes in intonation and stress, breaking through the limitations of traditional methods that treat emotion as a static label, making the generated facial animation more delicate, natural and full of vitality.

[0018] 2) Achieving an adaptive balance between emotion and content: Through the emotion guidance intensity gating unit, this application can dynamically and adaptively adjust the injection intensity of emotion features in each frame according to the contextual semantics of the speech content. Emotion guidance is enhanced when it is necessary to emphasize emotional expression, and weakened when it is necessary to ensure accurate lip-syncing, thus cleverly resolving the "competitive conflict" between emotional richness and lip-syncing accuracy, achieving an optimal balance between the two.

[0019] 3) High-level semantic supervision is introduced to enhance the realism of facial expressions: By introducing an expression consistency loss function based on a pre-trained expression encoder, this application explicitly constrains the consistency between the generated animation and the real animation in the high-level emotional semantic space during training. Compared with methods that rely solely on geometric loss, this supervision approach is more effective in ensuring that the generated facial movements are more realistic and accurate at the emotional cognition level.

[0020] 4) High computational efficiency and strong real-time performance: Both the emotion dynamic processor and the emotion guidance intensity gating unit are based on the multi-head attention mechanism, which avoids the O(n) complexity of sequence operations required by traditional recurrent neural networks (such as BiLSTM). The comprehensive sequence operation complexity is O(1), which significantly improves computational efficiency and has higher practical value for application scenarios with high real-time requirements (such as live interaction). Attached Figure Description

[0021] Figure 1 A flowchart of a voice-driven 3D face animation generation method based on dynamic emotion perception provided in an embodiment of this application.

[0022] Figure 2 This is an overall framework diagram of the voice-driven 3D facial animation generation method based on dynamic emotion perception provided in the embodiments of this application.

[0023] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0027] 1) EDP: Emotion Dynamic Processor.

[0028] 2) EIG: Emotion Intensity Gate.

[0029] 3) MPE: Multi-granularity Position Encoding.

[0030] 4) MEA: Multi-granularity Emotion Aggregation.

[0031] 5) SFR: Subspace Feature Routing.

[0032] In the process of implementing this invention application, the inventors discovered that existing deep learning-based end-to-end methods have the following drawbacks: 1) Emotion modeling is too simplistic and "static," losing temporal details: Existing technologies typically treat speech emotion as a global, static label or a single vector. This modeling approach assumes that the emotional intensity within a speech segment is constant. However, in real-world scenarios, emotional intensity fluctuates subtly with the intonation (prosody) and stress distribution of speech. Because existing technologies lack the ability to capture this fine temporal scale, the generated facial animations are too flat and mechanical in their emotional expression, failing to reproduce the characteristic of "emotion fluctuating with semantics" in real human speech.

[0033] 2) The fixed mechanism for injecting emotional features leads to a mismatch between lip-sync and facial expression richness: Existing methods for synthesizing animation typically involve a simple linear superposition or uniform injection of emotional features and speech content features. Since emotional features and speech content features (such as phoneme information) may have representational conflicts, overly strong emotional guidance can lead to inaccurate lip movements, while underlying guidance can result in a blank expression. This uniform injection method makes vivid emotional expression and accurate lip-sync incompatible.

[0034] 3) Lack of effective supervision of high-level facial expression semantics: Current model training mainly relies on geometric loss functions based on vertex coordinates. This approach only calculates displacement deviations in physical space. Due to the complexity of facial expressions, the same geometric error may correspond to completely different emotional meanings in visual perception. Existing technologies lack a mechanism to explicitly constrain the accuracy of emotional expression in high-level semantic space, resulting in insufficient accuracy and consistency in the emotional transmission of generated animations.

[0035] In view of this, this application addresses the shortcomings of existing voice-driven 3D face generation methods by proposing a voice-driven 3D face animation generation method, electronic device, storage medium, and program product based on dynamic emotion perception. This scheme enables dynamic contextual emotion perception and consists of two modules: an Emotion Dynamic Processor (EDP) and an Emotion Guidance Intensity Gating (EIG). The Emotion Dynamic Processor comprises multi-granularity emotion aggregation and subspace routing, capturing fine-grained changes in emotion features across different temporal and spatial scales. The Emotion Intensity Gating aligns content and emotion through multi-head attention and adaptively weights emotion features, dynamically adjusting the influence of emotion features during the generation process in a context-aware manner. This adaptive mechanism balances the expression of emotion and content.

[0036] The voice-driven 3D face animation generation method provided in this application relates to the fields of computer vision, digital human, and artificial intelligence. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the voice-driven 3D face animation generation method, but is not limited to the above forms.

[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0038] like Figure 1 As shown, this embodiment provides a voice-driven 3D face animation generation method based on dynamic emotion perception, which specifically includes the following steps: S1: Acquire input speech data and extract semantic feature sequences and emotional feature sequences from the input speech data; S2: Input the emotional feature sequence into the Emotion Dynamic Processor (EDP) to obtain a refined emotional feature sequence; wherein, the Emotion Dynamic Processor (EDP) is used to capture the emotional dynamic changes at multiple time scales in the emotional feature sequence and to reorganize the captured multi-scale emotional features; S3: Input the semantic feature sequence and the refined sentiment feature sequence into the sentiment-guided intensity-gated EIG unit, dynamically generate the sentiment weight of each frame in a context-aware manner, and use the sentiment weight to weight the refined sentiment feature sequence to obtain a context-adaptive scaled sentiment feature sequence. S4: Fuse the semantic feature sequence with the scaled sentiment feature sequence to obtain a joint feature representation; S5: Input the joint feature representation into the decoder to generate a 3D facial animation sequence synchronized with the input speech data.

[0039] The following is a detailed description and explanation of the proposed embodiments, in conjunction with the accompanying drawings and specific application examples.

[0040] Figure 2 This demonstrates the architecture of the method in this embodiment (referred to as EPTalk). Given a speech sequence... As input, the model generates the corresponding 3D face vertex offset sequence. This can be expressed by the following equation:

[0041] in, This represents the learnable parameters in the model.

[0042] (1) Feature extraction This embodiment uses two pre-trained encoders for speech feature extraction, where wav2vec 2.0 is used to extract semantic features. emotion2vec is used to extract emotional features. .

[0043] (2) Emotional dynamic processor The Emotion Dynamics Processor captures fine-grained emotional dynamics cues, and its three components are: Multi-granularity Location Encoding (MPE), Multi-granularity Emotion Aggregation (MEA), and Subspace Routing (SFR).

[0044] The first step in this embodiment is to propose multi-granularity sentiment aggregation—an improved positional encoding method that sums and averages multiple frequency positional codes, then injects them into the sentiment features. The second step involves inputting the sentiment features into a multi-head attention system and performing average pooling with different strides. The pooling results are then concatenated to obtain the sentiment features after multi-granularity sentiment feature aggregation. This process can be represented by the following equation:

[0045]

[0046]

[0047] in, This indicates the pooling step size.

[0048] Finally, this embodiment uses subspace routing to reconstruct sentiment features. Refined :

[0049] (3) Emotional guidance intensity gating The relative importance of emotional and content cues varies throughout the discourse. To dynamically balance the expressive power of emotion and content on facial features, this embodiment proposes Emotion Guiding Intensity Gating (EIG), which adjusts the influence of emotional features in a context-aware manner.

[0050] Emotional guidance intensity gating based on semantic features and refined emotional characteristics For input. First, [the following will be used]. and Input cross-modal attention mechanism to obtain semantically relevant, time-varying sentiment features. .Then, The emotional weight of each frame will be obtained through gating. Finally, emotional weight. will with Perform element-wise multiplication to scale the intensity of the emotional guidance:

[0051] in This indicates element-wise multiplication. The result is... It is a context-adaptive emotion representation that achieves richness of expression while maintaining semantic accuracy.

[0052] (4) Generation of speaking heads This embodiment will use semantic features And scaled emotions Perform a join to obtain a joint representation. Then, this union representation is used as a query. These are used as key and value inputs to the transformer encoder, ultimately yielding the facial offset sequence. :

[0053] (5) Network training This embodiment uses multiple objective loss functions for model training, totaling five: 5.1) Reconstructing Vertex Loss The mean squared error per frame (MSE) is used to evaluate the difference between the predicted action and the ground truth action.

[0054] in , Indicates the number of frames. This indicates the number of vertices in a 3D face.

[0055] 5.2) Velocity loss By measuring the difference between predicted and true values ​​between frames, the model is guided to reduce inter-frame jitter and ensure smooth motion.

[0056] 5.3) Potential Consistency Loss MSE (Mean Sequence Equation) is used to measure the difference between latent text features and features extracted by the lip-reading text encoder, ensuring consistency between the learned lip-reading text features and the text features corresponding to the audio. The audio text encoder uses a frozen WAV2Vec 2.0 CTC encoder, while the lip-reading text encoder uses a single-layer transformer and participates in parameter optimization during training.

[0057] in, This represents the features extracted by the audio text encoder. This represents the features extracted by the lip-reading text encoder. Indicates the number of samples.

[0058] 5.4) Text consistency loss The difference between the predicted character sequence and the real text is evaluated. This embodiment will extract audio text features. As a pseudo-ground truth text transcription, lip-reading text features are input into a softmax layer to obtain text logits. The CTC loss can then be defined as:

[0059] Wherein, the mapping function B represents the mapping function B. After merging duplicate characters, it is mapped to .

[0060] 5.5) Loss of facial consistency This is used to minimize the cosine similarity between the ground truth and predicted facial expression embeddings for each frame. To obtain such features, this embodiment trains a facial expression encoder using the AffectNets dataset. Then and Input to We obtain the facial expression embedding features and then calculate the cosine similarity of each frame.

[0061] The overall training objective is a weighted sum of the above losses:

[0062] in For hyperparameters. We adopt , , .

[0063] (6) Summary In summary, the method of this application solves the following problems of the prior art: 6.1) This application addresses the "static" and "coarse-grained" problems in emotion feature modeling in voice-driven facial animation: Existing technologies typically treat extracted speech emotion as a global, static label or vector, ignoring the subtle dynamic fluctuations of emotion over time with speech prosody, stress, and rhythm. This application solves the problem of how to capture and represent such frame-level, multi-scale, fine-grained emotion evolution trends, thereby avoiding the problem of stiff and lifeless animated expressions.

[0064] 6.2) This paper addresses the "competitive conflict" between the intensity of emotional expression and the fidelity of speech content (lip-sync): Existing methods often employ a uniform injection intensity when fusing emotional features. Since emotional features and speech content features (such as phoneme information) may have representational conflicts, excessively strong emotional guidance can lead to inaccurate lip movements, while insufficient guidance results in a "blank expression." This application addresses how to adaptively and dynamically adjust the injection intensity of emotional features for each frame based on the contextual semantics of the speech content, achieving an optimal balance between emotional richness and lip-sync accuracy.

[0065] 6.3) This application addresses the problem of insufficient supervision at the "emotional semantic level" by traditional loss functions: Existing technologies mostly employ geometric loss functions based on vertex coordinate differences and velocity differences. These functions only focus on the physical proximity of surfaces and are insufficient to constrain the consistency between the generated animation and real expressions in terms of deep emotional meaning. This application solves the problem of how to introduce an alignment mechanism of high-level facial semantic features during training to ensure that the generated 3D facial movements are more natural and realistic at the emotional cognition level.

[0066] In summary, this application presents a novel speech-driven 3D face animation generation scheme based on dynamic emotion perception. This framework aims to address the key challenge of capturing subtle and time-varying emotional dynamics from speech. The core innovation of this application lies in its dynamic, context-aware framework that fully utilizes emotional cues. This is achieved through the synergistic effect of two core components: EDP, which constructs a rich, multi-scale emotion representation as an emotion guide, and EIG, which adaptively balances this emotion guide according to the linguistic content. Experimental results demonstrate that the proposed scheme has considerable effectiveness in speech-driven 3D face animation, outperforming existing technologies.

[0067] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0068] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0069] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 using the methods described above in the embodiments of this application. Input / output interface 303 is used to implement information input and output; The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304); The processor 301, memory 302, input / output interface 303 and communication interface 304 are connected to each other within the device via bus 305.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0071] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0072] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0074] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0075] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0076] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0079] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0080] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0082] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A voice-driven 3D face animation generation method based on dynamic emotion perception, characterized in that, The method includes the following steps: Acquire input speech data and extract semantic feature sequences and emotional feature sequences from the input speech data; The emotional feature sequence is input into the Emotion Dynamic Processor (EDP) to obtain a refined emotional feature sequence. The EDP is used to capture the dynamic changes of emotions at multiple time scales in the emotional feature sequence and to reorganize the captured multi-scale emotional features. The semantic feature sequence and the refined emotional feature sequence are input into the Emotion Guidance Intensity Gated (EIG) unit. The emotional weight of each frame is dynamically generated in a context-aware manner. The refined emotional feature sequence is then weighted using the emotional weight to obtain a context-adaptive scaled emotional feature sequence. The semantic feature sequence is fused with the scaled sentiment feature sequence to obtain a joint feature representation; The joint feature representation is input into the decoder to generate a 3D facial animation sequence synchronized with the input speech data.

2. The method according to claim 1, characterized in that, The Emotion Dynamic Processor (EDP) includes a multi-granularity emotion aggregation module (MEA) and a subspace routing module (SFR). The step of inputting the emotional feature sequence into the Emotion Dynamic Processor (EDP) to obtain a refined emotional feature sequence specifically includes: The emotional feature sequence is input into the multi-granularity emotional aggregation module MEA. Through a multi-head attention mechanism and multiple pooling operations of different time lengths, the feature representations of emotional features at the original granularity and at least one other time granularity are captured respectively. The feature representations of different granularities are then concatenated to obtain multi-granularity emotional features. The multi-granularity emotional features are input into the subspace routing module SFR for feature recombination, and the refined emotional feature sequence is output.

3. The method according to claim 2, characterized in that, Before inputting the emotional feature sequence into the multi-granularity emotional aggregation module MEA, the method further includes: Multi-granularity positional encoding (MPE) is applied to the emotional feature sequence. The MPE is obtained by superimposing and averaging multiple positional codes of different frequencies.

4. The method according to claim 1, characterized in that, The Emotion Guidance Intensity Gated EIG unit includes a cross-modal attention module and a gating module; The step of inputting the semantic feature sequence and the refined sentiment feature sequence into the sentiment-guided intensity-gated EIG unit to dynamically generate the sentiment weight for each frame in a context-aware manner specifically includes: The semantic feature sequence is used as the query term, and the refined sentiment feature sequence is used as the key and value term. The result is input into the cross-modal attention module to obtain temporal sentiment features related to the semantic context. The temporal sentiment features are input into the gating module to calculate the sentiment weight corresponding to each frame.

5. The method according to claim 4, characterized in that, The process of weighting the refined emotional feature sequence using the emotional weights specifically involves: The emotional weight of each frame is element-wise multiplied with the feature vector of the corresponding frame in the refined emotional feature sequence.

6. The method according to claim 1, characterized in that, The decoder is a Transformer decoder; The step of inputting the joint feature representation into the decoder to generate a 3D facial animation sequence synchronized with the input speech data specifically involves: The joint feature representation is used as a query term, and the semantic feature sequence is used as a key and value term. These are then input into the Transformer decoder to generate the 3D facial animation sequence.

7. The method according to claim 1, characterized in that, The method further includes the step of constructing a total loss function to perform end-to-end training on the generative network; the total loss function is a weighted sum of at least two of the following loss functions: Reconstructing Vertex Loss Function This is used to minimize the vertex coordinate error between the generated 3D facial animation sequence and the real 3D facial animation sequence. Velocity loss function This is used to minimize the inter-frame motion speed error between the generated 3D facial animation sequence and the real 3D facial animation sequence, so as to reduce motion jitter during rendering of the generated 3D facial animation and ensure animation smoothness. Potential consistency loss function This is used to ensure that lip-reading features extracted from generated animations are consistent with text features extracted from speech in the latent space. Text consistency loss function It is used to evaluate the difference between the text sequence decoded from the generated animation and the real text sequence corresponding to the speech; Facial expression consistency loss function , used to minimize the cosine distance between the generated 3D facial animation sequence and the real 3D facial animation sequence in the expression embedding space defined by the pre-trained expression encoder.

8. The method according to claim 7, characterized in that, The total loss function also includes the vertex reconstruction loss function. The velocity loss function The potential consistency loss function The text consistency loss function and the expression consistency loss function .

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images