3D face animation generation method and device, computer device, and storage medium

Abstract

Embodiments of the present application disclose a 3D face animation generation method and device, computer equipment and a storage medium. The method can be applied to the digital human generation scene in the financial field and the medical and health field, and comprises the following steps: obtaining voice information and expression type guide information, and determining a target expression type based on the expression type guide information; analyzing the voice information to obtain content features, and obtaining an initial emotion vector without assignment; updating the initial emotion vector according to the target expression type to obtain a reference emotion feature; extracting audio emotion features and text semantic features from the voice information respectively; fusing the audio emotion features and the text semantic features to determine a target emotion intensity sequence; adjusting the vector length of the reference emotion feature according to the target emotion intensity sequence while keeping the vector direction unchanged to obtain a dynamic emotion feature; and fusing the content features and the dynamic emotion feature to generate a continuous 3D face animation. The method improves the generation quality of the 3D face animation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to 3D facial animation generation methods, apparatus, computer equipment and storage media. Background Technology

[0002] With the rapid development of virtual digital human technology, solutions based on voice-driven 3D facial animation generation have been widely applied in many fields such as human-computer interaction, virtual anchors, and intelligent customer service. For example, they are used in the financial field as virtual financial advisors, and in the medical and health field as virtual psychological counselors and emotional companion digital humans.

[0003] Existing audio-driven 3D facial animation generation technologies have several significant shortcomings. First, most methods use static emotion tags for control, keeping the emotional state unchanged throughout the speech segment. This results in the generated 3D facial animation lacking natural emotional fluctuations. Second, the existing technology couples lip movements with facial features, which may affect the accuracy of lip synchronization when adjusting emotions, easily leading to mismatches between mouth shape and speech. This feature entanglement problem severely restricts further improvement in the quality of 3D facial animation generation.

[0004] In summary, 3D facial animations generated using existing technologies suffer from a lack of natural emotional nuances and an entanglement between lip shapes and facial features, which affects the quality of 3D facial animation generation. Summary of the Invention

[0005] This application provides a method, apparatus, computer equipment, and storage medium for generating 3D facial animation, which can solve the problems of lacking natural emotional fluctuations and entanglement between lip shape and facial expression features in 3D facial animation, thereby improving the generation quality of 3D facial animation.

[0006] In a first aspect, embodiments of this application provide a method for generating 3D facial animation, which includes: Acquire voice information and facial expression type guidance information, and determine the target facial expression type based on the facial expression type guidance information; The speech information is parsed to obtain content features for controlling lip movements of the 3D face, and an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features is obtained. The initial emotion vector is updated according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face. Frame-level audio emotion features and frame-level text semantic features are extracted from the speech information, respectively. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; The vector magnitude of the baseline emotional feature is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged, to obtain the dynamic emotional feature that changes dynamically frame by frame; The content features are fused with the dynamic emotional features corresponding to each frame, and then input into a motion decoder to generate continuous 3D facial animation frame by frame.

[0007] Secondly, embodiments of this application also provide a 3D facial animation generation device, which includes: The transceiver unit is used to acquire voice information and facial expression type guidance information; The processing unit is configured to: determine the target expression type based on the expression type guidance information; parse the speech information to obtain content features for controlling lip movements of the 3D face, and obtain an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features; update the initial emotion vector according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face; extract frame-level audio emotion features and frame-level text semantic features from the speech information; fuse the audio emotion features and the text semantic features through a cross-modal attention mechanism to determine the target emotion intensity value for each corresponding animation frame, forming a continuous target emotion intensity sequence; adjust the vector magnitude of the baseline emotion feature according to the target emotion intensity sequence while keeping the vector direction unchanged to obtain a dynamic emotion feature that changes dynamically frame by frame; fuse the content features with the dynamic emotion features corresponding to each frame, input the result into a motion decoder, and generate a continuous 3D facial animation frame by frame.

[0008] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0009] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0010] This application provides a method, apparatus, computer device, and storage medium for generating 3D facial animation. On one hand, this application acquires voice information and expression type guidance information. After determining the target expression type based on the expression type guidance information, it parses the voice information to obtain content features (associated lip shape features) and an unassigned initial emotion vector. The content features and the initial emotion vector are independent and located in different embedding spaces. Therefore, this application can completely decouple lip shape features and emotion features, as they belong to independent embedding spaces. Adjusting expressions will not interfere with lip movement, significantly improving the matching degree between mouth shape and voice. On the other hand, this application updates the initial emotion vector according to the target expression type to obtain baseline emotion features; and extracts frame-level audio emotion features and frame-level text from the voice information. Semantic features; by fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; the vector magnitude of the baseline emotional features is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged, to obtain dynamic emotional features that change dynamically frame by frame; the content features are fused with the dynamic emotional features corresponding to each frame, and input into the motion decoder to generate continuous 3D facial animation frame by frame. It can be seen that, based on the design of fixed vector direction and adjustable magnitude of the baseline emotional features, this embodiment of the application realizes the dynamic change of emotional intensity while keeping the basic expression type unchanged, solving the problem of the lack of natural emotional fluctuations in the generated 3D facial animation and improving the generation quality of 3D facial animation. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the 3D facial animation generation method provided in this application embodiment; Figure 2 This is a schematic diagram of a sub-process of the 3D facial animation generation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the training process of the feature decoupling encoder in the 3D facial animation generation method provided in the embodiments of this application; Figure 4 This is another schematic diagram of a sub-process of the 3D facial animation generation method provided in the embodiments of this application; Figure 5This is another schematic diagram of a sub-process of the 3D facial animation generation method provided in the embodiments of this application; Figure 6 This is another schematic diagram of a sub-process of the 3D facial animation generation method provided in the embodiments of this application; Figure 7 A schematic block diagram of a 3D facial animation generation apparatus provided in the embodiments of this application; Figure 8 A schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been obtained by authorized entities (who have known and consented) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0015] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] This application provides a method, apparatus, computer equipment, and storage medium for generating 3D facial animation.

[0019] The execution subject of the 3D facial animation generation method can be the 3D facial animation generation device provided in the embodiments of this application, or a computer device that integrates the 3D facial animation generation device. The 3D facial animation generation device can be implemented in hardware or software. The computer device can be a terminal or a server. The terminal can be a smartphone, tablet computer, handheld computer, or laptop computer, etc.

[0020] The 3D facial animation generation method provided in this application embodiment can be applied in fields such as finance and healthcare that require the production of 3D facial animations for digital humans.

[0021] For example, in the financial field, it can be applied to the generation of digital human-like financial advisors and financial education anchors. When explaining market trends and financial knowledge, the digital human switches the intensity of its emotions according to the content being explained: calm and relaxed during the popular science phase, and serious during the risk warning phase.

[0022] For example, in the healthcare field, it can be applied to the generation of digital avatars for remote consultations and preoperative education. Doctors can use the digital avatar for remote follow-ups, chronic disease reminders, and medication guidance. The digital avatar is driven by voice scripts and can manually switch the emotions of the 3D facial animation based on the current content or the patient's current mood. When explaining the condition, surgical procedure, and rehabilitation precautions to patients and their families, the digital avatar needs to switch labels according to the current content. When talking about serious content (surgical risks, contraindications), it needs to switch to a serious expression, and when talking about reassuring content (postoperative rehabilitation, prognosis), it needs to switch to a friendly and soothing expression.

[0023] Figure 1 This is a flowchart illustrating the 3D facial animation generation method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps S110-S170.

[0024] S110. Obtain voice information and expression type guidance information, and determine the target expression type based on the expression type guidance information.

[0025] This embodiment is the data input and expression tone preset stage. The voice information is the original voice audio of the animation to be driven; the expression type guidance information is used to specify the overall expression style of the virtual face, and based on this, the target expression type that remains unchanged throughout the process is determined.

[0026] For example, in the 3D facial animation generation scenario of a virtual financial advisor in the financial field, the voice that needs to be broadcast (such as the voice of broadcasting the introduction of financial products or the voice of answering business questions) is collected as voice information, and the expression type guidance information is input, such as inputting the emotional tag of "professional and calm" as the expression type guidance information. Based on this, the system determines that the target expression type is a serious business expression.

[0027] For example, in the field of healthcare, a digital human for preoperative education can use a standard preoperative education voice message recorded by a professional nurse, such as "Hello, you will be undergoing laparoscopic cholecystectomy tomorrow morning. You must fast and abstain from water after 10:00 PM tonight before the surgery." The emotional tags "gentle and patient, professional and reassuring" can be entered, and the target expression type can be determined to be a friendly and reassuring expression from a medical professional.

[0028] In some embodiments, the expression type guidance information includes emotion tags, text descriptions, or reference images; please refer to [link / reference]. Figure 2 The step of determining the target expression type based on the expression type guidance information includes: S1101. If the expression type guidance information is an emotion tag, then the preset expression category corresponding to the emotion tag is directly determined as the target expression type.

[0029] Specifically, if the guiding information is an emotion tag, the target expression type can be directly matched based on the pre-stored mapping relationship between tags and expressions.

[0030] For example, if you input the emotion tag "gentle and friendly", the system will retrieve the preset mapping relationship and directly determine the target expression type as a friendly expression.

[0031] S1102. If the expression type guidance information is text description information, then perform semantic parsing on the text description information to determine the target expression type.

[0032] Specifically, if the guidance information is textual description information, then semantic analysis is performed on the natural language text to extract emotional keywords and determine the corresponding expression type.

[0033] For example, if the input text is "Please use soothing and patient emojis to interact", the keywords "soothing and patient" are extracted through semantic parsing, and the target emoji type is determined to be a soothing and friendly emoji.

[0034] S1103. If the expression type guidance information is a reference image, then perform facial expression recognition processing on the reference image to determine the target expression type.

[0035] Specifically, if the guiding information is a reference image, then facial expression recognition is performed on the input face image to extract facial expression features from the image, thereby determining the target facial expression type.

[0036] For example, upload a picture of a smiling person as a reference image. After facial expression recognition, the target expression type is determined to be a smiling expression.

[0037] As can be seen, the multimodal guidance mechanism provided in this embodiment supports three input methods: emotion tags, text descriptions, and reference images. The access method is flexible and can adapt to different hardware and business systems, greatly reducing the barrier to entry and expanding the applicable scenarios. S120. The speech information is parsed to obtain the content features for controlling the lip movements of the 3D face, and an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features.

[0038] This embodiment parses speech information to obtain content features for controlling lip movements. Simultaneously, it acquires an unassigned initial emotion vector located in a different embedding space from the content features. This initial emotion vector is a pre-defined blank emotion carrier, specifically designed to carry the expression type information subsequently specified by external guidance information (such as emotion tags). The content features and the initial emotion vector are stored in separate, independent embedding spaces, ensuring that subsequent emotion control operations do not affect the accuracy of lip movements.

[0039] For example, an 8-second audio clip of a financial advisor's business announcement is input into the target feature decoupler encoder, which outputs 128-dimensional content features. These content features are then bound to the speech, ensuring that the lip shape follows the pronunciation.

[0040] Specifically, in some embodiments, the two independent embedding spaces of content features and sentiment features in this scheme are constructed and optimized using a multi-stage self-supervised training strategy. The parsing operation of the speech information is implemented through a pre-trained target feature decoupling encoder. Please refer to [link to relevant documentation]. Figure 3 The target feature decoupling encoder is trained based on the following steps: S210. Collect multiple sets of speech samples with associated content features and emotional features to form a first training set, wherein the content features indicate real lip shape information and the emotional features indicate facial emotion information.

[0041] This embodiment collects massive amounts of speech data and labels the corresponding real lip movement data (content features) and facial emotions (emotional features) to construct a basic training dataset.

[0042] For example, tens of thousands of voice samples with different voice content and different emotions are collected, and each voice sample is bound to corresponding content features and emotional features, which are then integrated into the first training set.

[0043] S220. Based on the first training set, perform self-reconstruction phase training on the initial encoder and the corresponding initial decoder to obtain a first-level feature encoder and a first-level feature decoder.

[0044] In this embodiment, the self-reconstruction phase training allows the initial encoder and initial decoder to learn basic feature representation capabilities, ensuring that the resulting first-level feature encoder and first-level feature decoder can accurately reconstruct the original facial motion sequence.

[0045] Specifically, the first training set samples are input into the initial encoder, and the output features are then reconstructed by the initial decoder. The reconstruction results are constrained to be consistent with the real samples by the loss function. After training is completed, a first-level feature encoder and a first-level feature decoder are obtained.

[0046] S230. Perform feature exchange on speech samples with the same content features but different emotional features in the first training set to construct a second training set, and perform overlapping exchange training on the first-level feature encoder and the first-level feature decoder based on the second training set to obtain the second-level feature encoder and the second-level feature decoder.

[0047] In this embodiment, samples with the same wording but different emotions are selected and their emotional features are exchanged, forcing the encoder's corresponding model to distinguish between content and emotion, thus initially achieving feature decoupling.

[0048] Specifically, multiple voice samples with consistent wording but different emotions are selected, and their emotional features are exchanged to generate a second training set. This second training set is then input into the first-level model for training, weakening the correlation between content features and emotional features, thus obtaining a second-level encoding and decoding model.

[0049] For example, voice sample A: content features: "Hello", emotion features: happy; Voice Sample B: Content features: "Hello", Emotional features: Angry; By combining the content features of A with the sentiment features of B, and combining the sentiment features of A with the content features of B, we obtain the second training set.

[0050] S240. Select multiple groups of speech samples with consistent content features and different emotional features from the first training set and the second training set to construct a third training set, and perform a cyclic exchange phase training on the second-level feature encoder and the second-level feature decoder based on the third training set to obtain the target feature decoupling encoder.

[0051] This embodiment utilizes multiple sets of samples to repeatedly cross-exchange features, strengthens decoupling constraints, completely avoids feature confusion, and ultimately obtains a mature target feature decoupling encoder.

[0052] Specifically, multiple sets of samples with the same content but different emotions are selected to construct a third training set. The secondary model is then trained in multiple rounds to finally obtain a target feature decoupling encoder that can stably complete feature splitting.

[0053] The solution provided in this embodiment adopts a three-stage progressive self-supervised training method, which has strong feature decoupling ability, enabling lip movements and facial expressions to be completely independently controlled, fundamentally solving the feature entanglement problem. In addition, training is completed by relying on feature exchange, eliminating the need for massive amounts of finely labeled data and reducing data collection and labeling costs.

[0054] Furthermore, adopting a phased training strategy can ensure the stability of each training module, while keeping the entire model lightweight to meet the needs of real-time inference.

[0055] S130. Update the initial emotion vector according to the target expression type to obtain the baseline emotion feature, which is used to determine the basic expression tone of the 3D face.

[0056] In this embodiment, the target expression type is used to assign values ​​to the blank initial emotion vector to generate a baseline emotion feature. This feature determines the basic expression style of the entire animation process and will not be changed arbitrarily.

[0057] For example, by taking "professional and composed" as the target expression type, the initial emotion vector is assigned a value to obtain the baseline emotion feature, so that the virtual face can maintain a professional and calm basic expression throughout the process.

[0058] In some embodiments, specifically, please refer to Figure 4 Step S130 includes: S1301. Based on the pre-constructed correspondence between expression types and standard emotion vector bases, determine the target standard emotion vector base that matches the target expression type.

[0059] Specifically, in this embodiment, a mapping library between expression types and standard emotion vector bases is established in advance. Each type of expression corresponds to a unique vector base in a specific direction, and matching and retrieval are performed based on the target expression type.

[0060] For example, a pre-built mapping library contains expressions such as "professional and composed", "gentle and friendly", and "serious and warning" and their corresponding vector bases; if the target expression type is "professional and composed", then the corresponding target standard emotion vector base is retrieved.

[0061] S1302. The initial sentiment vector is updated by assigning values ​​to the target standard sentiment vector basis to obtain the benchmark sentiment feature.

[0062] In this embodiment, the matched standard vector basis is used to assign values ​​to the blank initial emotion vector to generate baseline emotion features and lock in the basic expression tone.

[0063] For example, the retrieved standard emotion vector base is assigned to the initial emotion vector to complete the vector update and obtain the baseline emotion feature. The vector direction of this feature is fixed to the "professional and calm" expression.

[0064] This solution defines expression types using standardized vector base quantization, resulting in unified and reusable expression classifications that facilitate batch deployment across multiple scenarios. Furthermore, by clarifying the implementation logic of vector assignment, it strengthens the core design of "vector direction corresponding to expression type," ensuring that the expression tone remains stable and unchanged.

[0065] S140. Extract frame-level audio emotion features and frame-level text semantic features from the speech information.

[0066] Specifically, the speech information is subjected to frame-level audio sentiment analysis to obtain the audio sentiment features; the speech information is transcribed into text to obtain frame-by-frame corresponding text content; and the text content is semantically encoded to obtain the text semantic features.

[0067] For example, emotion2vec is used to extract frame-level audio emotion features from speech information, and Whisper is used to transcribe the speech information into text and obtain RoBERTa-encoded text semantic features.

[0068] In this embodiment, the complete speech is divided into frame-by-frame data according to the animation frame rate. Audio emotional features corresponding to tone, speech rate and volume are extracted from the audio waveform, and text semantic features corresponding to sentence meaning and emotional tendency are extracted from the speech-to-text.

[0069] For example, according to the standard of 25 frames per second, an 8-second speech is divided into 200 animation frames; the audio waveform is analyzed frame by frame to obtain audio emotional features related to tone and speech rate; the speech is transcribed into text and encoded frame by frame to obtain text semantic features.

[0070] S150. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence.

[0071] This embodiment utilizes a cross-modal attention mechanism to weightedly fuse bimodal features, comprehensively determine the intensity of emotion in a single frame, and obtain the target emotion intensity value frame by frame after temporal constraint correction. All intensity values ​​are sorted by time to form a temporally coherent intensity sequence.

[0072] For example, when a financial advisor introduces product rules in a calm and even manner, the corresponding frame emotional intensity value is relatively low; when explaining risk warnings, the tone becomes more serious, and the corresponding frame intensity value increases; after fusion and temporal correction, the 200 frame intensity values ​​are arranged in an orderly manner to form a complete target emotional intensity sequence.

[0073] In some embodiments, specifically, please refer to Figure 5 Step S150 includes: S1501. The audio emotional features and the text semantic features are weighted and fused using the cross-modal attention mechanism to calculate the initial emotional intensity value of each animation frame, forming a continuous initial emotional intensity sequence.

[0074] This embodiment integrates bimodal features to calculate the original emotional intensity, but the initial sequence is not constrained, which can easily lead to numerical jumps between frames.

[0075] For example, by fusing audio and text features frame by frame, 200 initial emotional intensity values ​​are calculated and formed into an initial emotional intensity sequence. The values ​​of some adjacent frames may differ significantly.

[0076] S1502. Apply temporal smoothing constraints to the initial emotional intensity values ​​of adjacent animation frames in the initial emotional intensity sequence, and correct them to obtain the target emotional intensity value corresponding to each animation frame, thereby obtaining the target emotional intensity sequence.

[0077] This embodiment uses sliding window filtering, mean correction and other methods to limit the intensity difference between adjacent frames, smooth out numerical abrupt changes, and obtain a temporally smooth target intensity sequence.

[0078] For example, a 3-frame sliding window is used for smoothing, and the intensity value of the current frame is corrected by combining the values ​​of the previous and next frames to eliminate the jump between frames and generate a target emotion intensity sequence with a natural transition.

[0079] As can be seen, the introduction of temporal smoothing constraints in this embodiment effectively suppresses abrupt changes in emotional intensity between frames, solves the problems of facial expression jumps and visual inconsistencies, and significantly improves the naturalness of the animation.

[0080] S160. Adjust the vector magnitude of the baseline emotional feature according to the target emotional intensity sequence while keeping the vector direction unchanged to obtain the dynamic emotional feature that changes dynamically frame by frame.

[0081] In this embodiment, the vector direction corresponds to the expression type, and the vector magnitude corresponds to the intensity of the emotion. This step only scales the vector magnitude without changing the direction, ensuring that the basic expression remains unchanged and only realizing the dynamic fluctuation of the emotion intensity.

[0082] For example, if the baseline emotional feature direction is fixed as "professional and composed", the vector length is enlarged when the intensity value increases, and the serious expression is intensified; when the intensity value decreases, the vector length is reduced, and the expression tends to be softer, and the expression type does not switch throughout the process.

[0083] S170. The content features are fused with the dynamic emotional features corresponding to each frame, and the fusion is input into the motion decoder to generate continuous 3D facial animation frame by frame.

[0084] This embodiment integrates single-frame lip shape features and dynamic emotion features, which are then fed into a motion decoder for analysis, ultimately outputting a visual, temporally continuous 3D facial animation.

[0085] After fusing features frame by frame, the input is fed into the motion decoder to drive the 3D virtual face model and generate a complete facial animation with lip movements that perfectly match the speech, consistent basic expressions, and dynamic changes in the intensity of emotions.

[0086] In this embodiment, the motion decoder is a pre-trained deep learning network module, which is the core decoding unit for generating 3D facial animation in this embodiment. Its input is the fused content features and dynamic emotional features, and its output is facial driving parameters that can directly drive the 3D virtual face model, including facial key point coordinates, facial muscle deformation coefficients, eyebrow and mouth pose parameters, etc. Through these parameters, the movement of the 3D model mesh, skeleton and facial muscles is controlled, and finally, continuous 3D facial animation is generated.

[0087] In some embodiments, the network structure of the deep learning network module described above includes, but is not limited to, any of the following: (1) A decoder based on the Transformer architecture, the decoder includes a multi-head self-attention layer and a feedforward network layer, captures the long-range dependencies between elements in the fused frame features through the multi-head attention mechanism, and maps the fused frame features to the corresponding facial driving parameters; (2) A neural network based on an encoder-decoder architecture, wherein the motion decoder is the decoder part in the encoder-decoder architecture, receives the fused frame features as input, and outputs face driving parameters after being processed by several fully connected layers and / or convolutional layers; (3) A temporal decoding network based on the combination of convolutional neural network (CNN) and long short-term memory network (LSTM), wherein the CNN layer is used to extract the spatial features of the fused frame features, and the LSTM layer is used to capture the temporal dependencies between frames to generate a temporally coherent sequence of face driving parameters.

[0088] In some embodiments, specifically, please refer to Figure 6 Step S170 includes: S1701. The content features and dynamic emotion features of the same frame are spliced ​​or weighted and fused to obtain the fused frame features corresponding to each frame.

[0089] In this embodiment, feature concatenation involves directly concatenating the first and last parts of two feature vectors; weighted fusion involves assigning weights to the two sets of features and then superimposing them for calculation. The two methods can be flexibly selected according to the model structure.

[0090] For example, a single frame has 128-dimensional content features and 64-dimensional dynamic emotion features; after splicing, a 192-dimensional fusion feature is obtained; if weighted fusion is used, the content feature weight is set to 0.7 and the emotion feature weight is set to 0.3, and the fusion feature is generated by weighting.

[0091] S1702. The fused frame features corresponding to each frame are sent to the trained motion decoder for decoding, and the facial driving parameters corresponding to each frame are output.

[0092] In this embodiment, the motion decoder converts the fused frame features into quantized parameters (facial driving parameters) that can control the 3D face, including facial key point coordinates, muscle deformation coefficients, mouth / eyebrow pose parameters, etc.

[0093] For example, after frame-by-frame input of fused features, the decoder outputs the coordinates of 68 facial key points and facial deformation parameters as single-frame facial driving parameters.

[0094] S1703: Drive the preset 3D virtual character face based on the facial driving parameters of all frames, and render a continuous 3D facial animation.

[0095] This embodiment uses frame-by-frame facial driving parameters to control the posture of the 3D model, renders the scene frame by frame, and forms a complete animation after continuous playback.

[0096] For example, by applying 200 frames of facial driving parameters sequentially to a 3D virtual human model and rendering the image frame by frame, a 3D facial animation with continuous timing and complete visuals can be obtained.

[0097] In summary, on the one hand, after obtaining voice information and facial expression type guidance information, and determining the target facial expression type based on the facial expression type guidance information, the voice information is parsed to obtain content features (related lip shape features) and an unassigned initial emotion vector. The content features and the initial emotion vector are independent of each other and located in different embedding spaces. Therefore, the embodiments of this application can achieve complete decoupling of lip shape features and emotion features, as they belong to independent embedding spaces. Adjusting facial expressions will not interfere with lip movement, resulting in a significant improvement in the matching degree between mouth shape and voice. On the other hand, the embodiments of this application update the initial emotion vector according to the target facial expression type to obtain baseline emotion features; and extract frame-level audio emotion features and frame-level emotion features from the voice information. Text semantic features; by fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; the vector magnitude of the baseline emotional features is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged, to obtain dynamic emotional features that change dynamically frame by frame; the content features are fused with the dynamic emotional features corresponding to each frame, and input into the motion decoder to generate continuous 3D facial animation frame by frame. It can be seen that, based on the design of fixed vector direction and adjustable magnitude of the baseline emotional features, this embodiment of the application realizes the dynamic change of emotional intensity while keeping the basic expression type unchanged, solving the problem of the lack of natural emotional fluctuations in the generated 3D facial animation and improving the generation quality of 3D facial animation.

[0098] Figure 7 This is a schematic block diagram of a 3D facial animation generation device provided in an embodiment of this application. Figure 7 As shown, corresponding to the above-described 3D facial animation generation method, this application also provides a 3D facial animation generation apparatus 700. This 3D facial animation generation apparatus 700 includes a unit for performing the above-described 3D facial animation generation method, and can be configured in terminals such as desktop computers, tablet computers, and laptops. Specifically, please refer to... Figure 7 The 3D facial animation generation device 700 includes a transceiver unit 701 and a processing unit 702, wherein: The transceiver unit 701 is used to acquire voice information and facial expression type guidance information; Processing unit 702 is configured to: determine a target expression type based on the expression type guidance information; parse the speech information to obtain content features for controlling lip movements of the 3D face, and obtain an initial emotion vector that is located in a different embedding space from the content features and is not assigned a value; update the initial emotion vector according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face; extract frame-level audio emotion features and frame-level text semantic features from the speech information; fuse the audio emotion features and the text semantic features through a cross-modal attention mechanism to determine the target emotion intensity value for each corresponding animation frame, forming a continuous target emotion intensity sequence; adjust the vector magnitude of the baseline emotion feature according to the target emotion intensity sequence while keeping the vector direction unchanged to obtain a dynamic emotion feature that changes dynamically frame by frame; fuse the content features with the dynamic emotion features corresponding to each frame, input the fusion into a motion decoder, and generate a continuous 3D facial animation frame by frame.

[0099] In some embodiments, the expression type guidance information includes emotion tags, text description information, or reference images; when performing the step of determining the target expression type based on the expression type guidance information, the processing unit 702 is specifically used for: If the expression type guidance information is an emotion tag, then the preset expression category corresponding to the emotion tag is directly determined as the target expression type; If the expression type guidance information is text description information, then the text description information is semantically parsed to determine the target expression type; If the expression type guidance information is a reference image, then facial expression recognition processing is performed on the reference image to determine the target expression type.

[0100] In some embodiments, when performing the step of extracting frame-level audio emotion features and frame-level text semantic features from the speech information, the processing unit 702 is specifically used for: Frame-level audio sentiment analysis is performed on the speech information to obtain the audio sentiment features; The voice information is transcribed into text to obtain the corresponding text content frame by frame. The text content is semantically encoded to obtain the text semantic features.

[0101] In some embodiments, when the processing unit 702 performs the step of fusing the content features with the dynamic emotional features corresponding to each frame, inputting the result into a motion decoder, and generating a continuous 3D facial animation frame by frame, it is specifically used for: The content features and dynamic emotion features of the same frame are concatenated or weighted and fused to obtain the fused frame features corresponding to each frame. The fused frame features corresponding to each frame are fed into the trained motion decoder for decoding, and the facial driving parameters corresponding to each frame are output. The preset 3D virtual character face is driven by the facial driving parameters of all frames, and a continuous 3D facial animation is rendered.

[0102] In some embodiments, updating the initial emotion vector according to the target expression type to obtain baseline emotion features includes: Based on the pre-constructed correspondence between facial expression types and standard sentiment vector bases, determine the target standard sentiment vector base that matches the target facial expression type; The initial sentiment vector is updated by assigning values ​​to the target standard sentiment vector basis to obtain the baseline sentiment features.

[0103] In some embodiments, when the processing unit 702 performs the step of fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism to determine the target emotional intensity value for each corresponding animation frame and forming a continuous target emotional intensity sequence, it is specifically used for: The cross-modal attention mechanism is used to perform weighted fusion of the audio emotional features and the text semantic features to calculate the initial emotional intensity value of each animation frame, forming a continuous initial emotional intensity sequence. Temporal smoothing constraints are applied to the initial emotional intensity values ​​of adjacent animation frames in the initial emotional intensity sequence to correct and obtain the target emotional intensity value corresponding to each animation frame, thus obtaining the target emotional intensity sequence.

[0104] In some embodiments, the parsing operation of the speech information is implemented through a pre-trained target feature decoupling encoder. When training the target feature decoupling encoder, the processing unit 702 is specifically used for: The first training set is constructed by collecting multiple sets of speech samples with associated content features and emotion features. The content features indicate real lip shape information, and the emotion features indicate facial emotion information. Based on the first training set, a self-reconstruction phase training is performed on the initial encoder and the corresponding initial decoder to obtain a first-level feature encoder and a first-level feature decoder. The speech samples with the same content features but different emotional features in the first training set are subjected to feature exchange to construct a second training set. Based on the second training set, the first-level feature encoder and the first-level feature decoder are trained in an overlapping exchange phase to obtain the second-level feature encoder and the second-level feature decoder. Multiple groups of speech samples with consistent content features but different emotional features are selected from the first training set and the second training set to construct a third training set. The second-level feature encoder and the second-level feature decoder are trained by performing a cyclical exchange phase based on the third training set to obtain the target feature decoupled encoder.

[0105] In summary, on the one hand, after obtaining voice information and facial expression type guidance information, and determining the target facial expression type based on the facial expression type guidance information, the voice information is parsed to obtain content features (related lip shape features) and an unassigned initial emotion vector. The content features and the initial emotion vector are independent of each other and located in different embedding spaces. Therefore, the embodiments of this application can achieve complete decoupling of lip shape features and emotion features, as they belong to independent embedding spaces. Adjusting facial expressions will not interfere with lip movement, resulting in a significant improvement in the matching degree between mouth shape and voice. On the other hand, the embodiments of this application update the initial emotion vector according to the target facial expression type to obtain baseline emotion features; and extract frame-level audio emotion features and frame-level emotion features from the voice information. Text semantic features; by fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; the vector magnitude of the baseline emotional features is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged, to obtain dynamic emotional features that change dynamically frame by frame; the content features are fused with the dynamic emotional features corresponding to each frame, and input into the motion decoder to generate continuous 3D facial animation frame by frame. It can be seen that, based on the design of fixed vector direction and adjustable magnitude of the baseline emotional features, this embodiment of the application realizes the dynamic change of emotional intensity while keeping the basic expression type unchanged, solving the problem of the lack of natural emotional fluctuations in the generated 3D facial animation and improving the generation quality of 3D facial animation.

[0106] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned 3D facial animation generation device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0107] The aforementioned 3D facial animation generation device can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0108] Please see Figure 8 , Figure 8This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 800 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0109] See Figure 8 The computer device 800 includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0110] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions that, when executed, cause the processor 802 to perform a 3D facial animation generation method.

[0111] The processor 802 provides computing and control capabilities to support the operation of the entire computer device 800.

[0112] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a 3D facial animation generation method.

[0113] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 800 to which the present application is applied. The specific computer device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] The processor 802 is used to run a computer program 8032 stored in the memory to perform the following steps: Acquire voice information and facial expression type guidance information, and determine the target facial expression type based on the facial expression type guidance information; The speech information is parsed to obtain content features for controlling lip movements of the 3D face, and an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features is obtained. The initial emotion vector is updated according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face. Frame-level audio emotion features and frame-level text semantic features are extracted from the speech information, respectively. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; Adjusting the vector magnitude of the baseline emotional feature according to the target emotional intensity sequence while keeping the vector direction unchanged, yields a dynamic emotional feature that changes dynamically frame by frame. The content features are fused with the dynamic emotional features corresponding to each frame, and then input into a motion decoder to generate continuous 3D facial animation frame by frame.

[0115] It should be understood that in the embodiments of this application, the processor 802 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0116] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0117] Therefore, this application also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: Acquire voice information and facial expression type guidance information, and determine the target facial expression type based on the facial expression type guidance information; The speech information is parsed to obtain content features for controlling lip movements of the 3D face, and an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features is obtained. The initial emotion vector is updated according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face. Frame-level audio emotion features and frame-level text semantic features are extracted from the speech information, respectively. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; Adjusting the vector magnitude of the baseline emotional feature according to the target emotional intensity sequence while keeping the vector direction unchanged, yields a dynamic emotional feature that changes dynamically frame by frame. The content features are fused with the dynamic emotional features corresponding to each frame, and then input into a motion decoder to generate continuous 3D facial animation frame by frame.

[0118] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0121] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating 3D facial animation, characterized in that, include: Acquire voice information and facial expression type guidance information, and determine the target facial expression type based on the facial expression type guidance information; The speech information is parsed to obtain content features for controlling lip movements of the 3D face, and an initial emotion vector that is not assigned a value and is located in a different embedding space from the content features is obtained. The initial emotion vector is updated according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face. Frame-level audio emotion features and frame-level text semantic features are extracted from the speech information, respectively. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined, forming a continuous target emotional intensity sequence; The vector magnitude of the baseline emotional feature is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged, to obtain the dynamic emotional feature that changes dynamically frame by frame; The content features are fused with the dynamic emotional features corresponding to each frame, and then input into a motion decoder to generate continuous 3D facial animation frame by frame.

2. The method according to claim 1, characterized in that, The expression type guidance information includes emotion tags, text description information, or reference images; determining the target expression type based on the expression type guidance information includes: If the expression type guidance information is an emotion tag, then the preset expression category corresponding to the emotion tag is directly determined as the target expression type; If the expression type guidance information is text description information, then the text description information is semantically parsed to determine the target expression type; If the expression type guidance information is a reference image, then facial expression recognition processing is performed on the reference image to determine the target expression type.

3. The method according to claim 1, characterized in that, The step of extracting frame-level audio emotion features and frame-level text semantic features from the speech information includes: Frame-level audio sentiment analysis is performed on the speech information to obtain the audio sentiment features; The voice information is transcribed into text to obtain the corresponding text content frame by frame. The text content is semantically encoded to obtain the text semantic features.

4. The method according to claim 1, characterized in that, The process of fusing the content features with the dynamic emotional features corresponding to each frame, inputting the data into a motion decoder, and generating continuous 3D facial animation frame by frame includes: The content features and dynamic emotion features of the same frame are concatenated or weighted and fused to obtain the fused frame features corresponding to each frame. The fused frame features corresponding to each frame are fed into the trained motion decoder for decoding, and the facial driving parameters corresponding to each frame are output. The preset 3D virtual character face is driven by the facial driving parameters of all frames, and a continuous 3D facial animation is rendered.

5. The method according to claim 1, characterized in that, The step of updating the initial emotion vector according to the target expression type to obtain the baseline emotion features includes: Based on the pre-constructed correspondence between facial expression types and standard emotion vector bases, a target standard emotion vector base matching the target facial expression type is determined; The initial sentiment vector is updated by assigning values ​​to the target standard sentiment vector basis to obtain the baseline sentiment features.

6. The method according to claim 1, characterized in that, The process of fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism to determine the target emotional intensity value for each corresponding animation frame, forming a continuous target emotional intensity sequence, includes: The cross-modal attention mechanism is used to perform weighted fusion of the audio emotional features and the text semantic features to calculate the initial emotional intensity value of each animation frame, forming a continuous initial emotional intensity sequence. Temporal smoothing constraints are applied to the initial emotion intensity values ​​of adjacent animation frames in the initial emotion intensity sequence to correct and obtain the target emotion intensity value corresponding to each animation frame, thus obtaining the target emotion intensity sequence.

7. The method according to any one of claims 1 to 6, characterized in that, The parsing operation of the speech information is implemented through a pre-trained target feature decoupling encoder, which is trained based on the following steps: The first training set is constructed by collecting multiple sets of speech samples with associated content features and emotion features. The content features indicate real lip shape information, and the emotion features indicate facial emotion information. Based on the first training set, a self-reconstruction phase training is performed on the initial encoder and the corresponding initial decoder to obtain a first-level feature encoder and a first-level feature decoder. The speech samples with the same content features but different emotional features in the first training set are subjected to feature exchange to construct a second training set. Based on the second training set, the first-level feature encoder and the first-level feature decoder are trained in an overlapping exchange phase to obtain the second-level feature encoder and the second-level feature decoder. Multiple groups of speech samples with consistent content features but different emotional features are selected from the first training set and the second training set to construct a third training set. The second-level feature encoder and the second-level feature decoder are trained by performing a cyclical exchange phase based on the third training set to obtain the target feature decoupled encoder.

8. A 3D facial animation generation device, characterized in that, include: The transceiver unit is used to acquire voice information and facial expression type guidance information; The processing unit is used to determine the target expression type based on the expression type guidance information; The speech information is parsed to obtain content features for controlling lip movements of the 3D face, and an initial emotion vector that is located in a different embedding space from the content features and is not assigned a value is obtained; the initial emotion vector is updated according to the target expression type to obtain a baseline emotion feature, which is used to determine the basic expression tone of the 3D face; frame-level audio emotion features and frame-level text semantic features are extracted from the speech information respectively. By fusing the audio emotional features and the text semantic features through a cross-modal attention mechanism, the target emotional intensity value of each corresponding animation frame is determined to form a continuous target emotional intensity sequence. The vector magnitude of the baseline emotional features is adjusted according to the target emotional intensity sequence while keeping the vector direction unchanged to obtain dynamic emotional features that change dynamically frame by frame. The content features are fused with the dynamic emotional features corresponding to each frame and input into the motion decoder to generate continuous 3D facial animation frame by frame.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the 3D facial animation generation method as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the 3D facial animation generation method as described in any one of claims 1-7.

Images