Facial motion generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of 3D facial motion generation, and discloses a facial motion generation method, device and equipment and a storage medium, and the method comprises the steps: responding to multimedia data and emotion data inputted by a user, and determining a voice sequence according to the multimedia data; acquiring facial motion data, and adding noise according to the facial motion data to obtain facial noise motion; and inputting the voice sequence, the facial noise motion and the emotion data into a facial motion generation model to obtain a facial expression video, the facial expression video including a single-emotion facial expression video or a facial expression video with continuous emotion transformation. According to the method and the device, various and vivid facial expressions can be generated, and flexible and continuous emotion control is realized.

Description

Technical Field

[0001] The present application relates to the field of 3D facial motion technology, and in particular to a facial motion generation method, device, equipment and storage medium. Background Art

[0002] The animation generation system of 3D facial expressions with different emotions driven by speech and text has a wide range of applications in daily life, including film and television and animation production, game development, virtual reality and augmented reality, etc. Through the rapid development of the above fields, realistic and natural facial animation generation has gradually been applied to applications closely related to daily life such as education and medical care, which not only improves the visual and communication effects, but also plays an important role in various modern technologies and application scenarios. The principle is to design a 3D facial animation synthesis model for different emotions driven by speech or text according to the relevant content of speech and text. By adding emotion control to the input sequence, the model can realize 3D facial animation with different emotions under the same audio or text sequence. In order to further improve the ability of virtual people to express emotions, a continuous emotion transformation module is designed on this basis, which can realize continuous emotion transformation at the frame level under the same input sequence, and the process is coherent and natural. In general, the system can generate 3D facial animations with different emotions according to different audio or text sequences. The generation process is flexible and controllable, and the corresponding emotion transformation can be edited according to the emotional fluctuations in the audio or text. By accurately capturing and reflecting the emotional changes of the character in real time, the virtual character can show natural and smooth emotional changes in real time.

[0003] Current methods have difficulty in generating high-quality 3D facial motion. Most of the work on generating animations only focuses on learning the content of speech, audio and lip pronunciation synchronization, and style changes, ignoring the rich and realistic facial expression changes. However, emotional communication is crucial to enhancing the interactive experience of games and users. Therefore, voice-driven 3D facial animation needs to express different emotions through facial expressions while paying attention to the synchronization of audio and lip pronunciation. In terms of facial emotion expression, previous works use traditional neural network models to express emotions. However, this emotional facial animation generation method ignores the one-to-many relationship between audio and facial expression animation, limits the diversity of synthesized facial expressions, and lacks subtle emotional expression. In addition, existing emotional 3D facial animation generation methods lack flexible control and editing of facial emotions, and the transformation process between different emotions is too rigid, which cannot effectively process and express emotions.

[0004] Therefore, how to generate diverse and realistic facial expressions and achieve flexible and continuous emotion control is a problem that needs to be solved urgently.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a facial motion generation method, aiming to solve the technical problem of how to generate diverse and realistic facial expressions and achieve flexible and continuous emotion control.

[0007] To achieve the above objectives, the present application proposes a facial motion generation method, the method comprising:

[0008] In response to multimedia data and emotion data input by a user, determining a speech sequence according to the multimedia data;

[0009] Acquire facial motion data, and add noise according to the facial motion data to obtain facial noise motion;

[0010] The speech sequence, the facial noise movement and the emotion data are input into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0011] In one embodiment, determining the speech sequence according to the multimedia data includes:

[0012] When the multimedia data is text data, decomposing the text data into a phoneme sequence;

[0013] Converting the phoneme sequence into an acoustic feature sequence, and generating a digital signal according to the acoustic feature sequence;

[0014] Performing audio simulation according to the digital signal to generate a voice signal;

[0015] A speech sequence is generated according to the speech signal.

[0016] In one embodiment, the step of inputting the speech sequence, the facial noise motion, and the emotion data into a facial motion generation model to obtain a facial expression video includes:

[0017] Inputting the speech sequence, the facial noise motion and the emotion data into corresponding encoders in a facial motion generation model respectively to generate audio features, facial noise motion features and emotion features;

[0018] Inputting the audio features, the facial noise motion features and the emotion features into a facial motion decoder in a facial motion generation model to generate a facial motion sequence of a single emotion;

[0019] The speech sequence and the facial movement sequence of the single emotion are input into a facial expression rendering module in a facial movement generation model to generate a facial expression video of the single emotion.

[0020] In one embodiment, the step of inputting the speech sequence, the facial noise motion, and the emotion data into corresponding encoders in a facial motion generation model to generate audio features, facial noise motion features, and emotion features comprises:

[0021] Converting the speech sequence into audio features via an audio encoder in the facial movement generation model;

[0022] The facial noise motion is converted into facial noise motion features by a facial motion encoder in the facial motion generation model;

[0023] The emotion label and the emotion intensity in the emotion data are extracted by the emotion encoder in the facial movement generation model, and the emotion label and the emotion intensity are encoded as emotion features.

[0024] In one embodiment, the step of inputting the audio feature, the facial noise motion feature, and the emotion feature into a facial motion decoder in a facial motion generation model to generate a facial motion sequence of a single emotion comprises:

[0025] Input the audio feature, the facial noise motion feature and the emotion feature into a facial motion decoder in a facial motion generation model, and perform feature fusion on the audio feature and the facial noise motion feature through the facial motion decoder and inject them into a first time encoder to generate facial denoising motion;

[0026] The emotion feature and the facial denoising motion are injected into a second temporal encoder to generate a facial motion sequence of a single emotion.

[0027] In one embodiment, the step of inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model to generate a facial expression video of the single emotion comprises:

[0028] Inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model, and mapping the facial movement sequence into facial expression parameters and shape parameters through the facial expression rendering module;

[0029] Rendering is performed according to the facial expression parameters and the shape parameters to generate facial animation;

[0030] The facial animation and the speech sequence are synchronously synthesized to generate a facial expression video of a single emotion.

[0031] In one embodiment, the step of inputting the speech sequence, the facial noise motion, and the emotion data into a facial motion generation model to obtain a facial expression video further includes:

[0032] Determine different emotion tags according to the emotion data, and obtain emotion conversion frames;

[0033] Dividing the speech sequence into a plurality of audio sequences according to the emotion conversion frame;

[0034] Generate a plurality of facial movement sequences of different emotions according to the audio sequence, the emotion label and the emotion conversion frame;

[0035] Processing the facial movement sequences of different emotions at the emotion conversion frames by a multi-emotion smoothing algorithm to generate facial movement sequences of continuous emotions;

[0036] The speech sequence and the facial movement sequence of the continuous emotions are input into a facial expression rendering module in a facial movement generation model to generate a facial expression video with continuous emotion changes.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a facial motion generation device, the facial motion generation device comprising:

[0038] A determination module, configured to determine a speech sequence according to the multimedia data in response to multimedia data and emotion data input by a user;

[0039] A noise adding module, used for acquiring facial motion data, and performing noise adding according to the facial motion data to obtain facial noise motion;

[0040] The input module is used to input the speech sequence, the facial noise movement and the emotion data into the facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a facial motion generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the facial motion generation method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the facial motion generation method described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the facial motion generation method described above are implemented.

[0044] The present application provides a facial movement generation method, which first responds to multimedia data and emotional data input by a user, determines a voice sequence according to the multimedia data; obtains facial movement data, and adds noise according to the facial movement data to obtain facial noise movement; inputs the voice sequence, the facial noise movement and the emotional data into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes, and can generate a variety of realistic facial expressions to achieve flexible and continuous emotion control.

[0045] In summary, the present application responds to the multimedia data and emotional data input by the user, increases the diversity of generated samples by adding noise to the facial motion data, makes the facial motion data smoother, and obtains facial noise motion, and then uses the facial motion generation model to generate a facial expression video of a single emotion containing diverse and realistic facial expressions according to the speech sequence, facial noise motion and emotional data, or uses a multi-emotion smoothing algorithm to generate a facial expression video with continuous emotional changes, thereby achieving flexible and continuous emotional control, overcoming the technical defects of rough and insufficient diversity of facial emotional expression and rigid transformation process between different emotions, and being able to generate diverse and realistic facial expressions to achieve flexible and continuous emotional control. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the facial motion generation method of the present application;

[0049] Figure 2 A flow chart of reconstructing 3D facial motion data provided in Embodiment 1 of the facial motion generation method of the present application;

[0050] Figure 3A schematic diagram of a flow chart provided for the second embodiment of the facial motion generation method of the present application;

[0051] Figure 4 A schematic diagram of the structure of the facial noise reduction network MCHC provided in Embodiment 2 of the facial motion generation method of the present application;

[0052] Figure 5 A schematic diagram of a flow chart provided for Embodiment 3 of the facial motion generation method of the present application;

[0053] Figure 6 The overall flow chart of facial motion generation provided in the third embodiment of the facial motion generation method of the present application;

[0054] Figure 7 A schematic diagram of the structure of the 3D emotional facial motion generation model EmoDiffuser provided in the third embodiment of the facial motion generation method of the present application;

[0055] Figure 8 A continuous expression change flow chart provided for the third embodiment of the facial movement generation method of the present application;

[0056] Fig. 9 This is a schematic diagram of the module structure of the facial movement generation device according to an embodiment of the present application;

[0057] Fig.10 This is a schematic diagram of the device structure of the hardware operating environment involved in the facial motion generation method of the embodiment of the present application.

[0058] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0060] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0061] The main solution of the embodiment of the present application is: in response to multimedia data and emotional data input by a user, determine a voice sequence according to the multimedia data; obtain facial motion data, and add noise according to the facial motion data to obtain facial noise motion; input the voice sequence, the facial noise motion and the emotional data into a facial motion generation model to obtain a facial expression video, which includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0062] Current methods have difficulty in generating high-quality 3D facial motion. Most of the work on generating animation only focuses on learning the content of speech, audio and lip pronunciation synchronization, and style changes, ignoring the rich and realistic facial expression changes. However, emotional communication is crucial to enhancing the interactive experience of games and users. Therefore, voice-driven 3D facial animation needs to express different emotions through facial expressions while paying attention to the synchronization of audio and lip pronunciation. In terms of facial emotion expression, previous works use traditional neural network models to express emotions. However, this emotional facial animation generation method ignores the one-to-many relationship between audio and facial expression animation, limits the diversity of synthesized facial expressions, and lacks subtle emotional expression. In addition, the existing emotional 3D facial animation generation method lacks flexible control and editing of facial emotions, and the transformation process between different emotions is too rigid to effectively process and express emotions. Therefore, how to generate diverse and realistic facial expressions and achieve flexible and continuous emotional control is a problem that needs to be solved urgently.

[0063] The present application responds to multimedia data and emotional data input by the user, increases the diversity of generated samples by adding noise to facial motion data, makes the facial motion data smoother, and obtains facial noise motion. Then, through the facial motion generation model, a facial expression video of a single emotion containing diverse and realistic facial expressions is generated according to the speech sequence, facial noise motion and emotional data, or a facial expression video with continuous emotional changes is generated using a multi-emotion smoothing algorithm, thereby achieving flexible and continuous emotional control, overcoming the technical defects of rough and insufficient diversity of facial emotional expression and rigid transformation process between different emotions, and being able to generate diverse and realistic facial expressions to achieve flexible and continuous emotional control.

[0064] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a facial motion generation device, etc. The following takes the facial motion generation device as an example to illustrate this embodiment and the following embodiments.

[0065] Based on this, the embodiment of the present application provides a facial motion generation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the facial motion generation method of the present application. In this embodiment, the facial motion generation method includes steps S10 to S40:

[0066] Step S10, in response to the multimedia data and emotion data input by the user, determining a voice sequence according to the multimedia data.

[0067] It should be noted that multimedia data may include but is not limited to text files and audio files, among which text files may be text information input by a user through a keyboard, touch screen or other input device, and audio files may be voice recorded by a user through a microphone or other audio acquisition device or a pre-recorded voice clip.

[0068] It is understandable that the emotion data includes the emotion label and emotion intensity set by the user, and this embodiment does not impose any specific limitation on this.

[0069] In a specific implementation, when the multimedia data input by the user is text data, the text data is converted into a voice sequence; when the multimedia data input by the user is voice data, the voice data is directly used as the voice sequence. In the process of converting the text data into the voice sequence, the text-to-speech (TTS) technology can be used to convert the text information into the corresponding voice information, and this embodiment does not specifically limit this.

[0070] It is worth noting that the facial motion generation method in this embodiment can be applied to a facial motion generation system, which includes a facial motion generation module and a continuous emotion conversion module. The facial motion generation module is used to control the 3D facial motion generation of different emotions under any audio, thereby obtaining; the continuous emotion conversion module is used to perform frame-level continuous emotion conversion between multiple different types of emotions, and generate a facial expression video with continuous emotion changes.

[0071] In a feasible implementation, step S10 may include: when the multimedia data is text data, decomposing the text data into a phoneme sequence; converting the phoneme sequence into an acoustic feature sequence, and generating a digital signal based on the acoustic feature sequence; performing audio simulation based on the digital signal to generate a speech signal; and generating a speech sequence based on the speech signal.

[0072] It should be noted that the text is decomposed into phoneme sequences according to pronunciation. Phoneme sequences refer to a basic pronunciation unit that can represent the smallest speech segment in a specific language. The generation of phoneme sequences usually involves natural language processing technologies, such as word segmentation, part-of-speech tagging, named entity recognition, etc., to ensure that text data can be accurately converted into corresponding phoneme sequences.

[0073] It is understood that the acoustic feature sequence refers to a sequence of acoustic features including volume, intonation and pitch, which can reflect the physical properties of speech signals. In the process of generating the acoustic feature sequence, deep learning techniques such as recurrent neural networks (RNN) or convolutional neural networks (CNN) can be used to improve the accuracy of the conversion from phoneme sequences to acoustic feature sequences. The generated acoustic feature sequence can be further used to train the acoustic model, thereby improving the quality of speech synthesis.

[0074] It is worth noting that the acoustic features in the acoustic feature sequence can be converted into digital signals by using digital signal processing technology, such as Fast Fourier Transform (FFT) or Discrete Cosine Transform (DCT). Digital signals refer to digital forms that can represent sound waveforms. By converting the acoustic feature sequence into a digital signal, audio simulation and speech sequence generation can be performed more conveniently.

[0075] It is worth noting that the digital signal can be converted into an analog signal through an output device for audio simulation, using a device such as a digital-to-analog converter (DAC) to generate a speech signal. The speech signal can be further processed, such as optimized through filters, amplifiers and other devices to ensure that the generated speech sequence has high clarity and naturalness.

[0076] Step S20, acquiring facial motion data, and performing noise addition according to the facial motion data to obtain facial noise motion.

[0077] It should be noted that the facial movement in the present application can be 3D facial movement, 2D facial movement or other dimensions. This embodiment does not impose specific restrictions on this. This embodiment takes 3D facial movement as an example for explanation. Facial movement data is the 3D emotional facial movement data of the target, that is, the 3D kinematic information of the target facial expression, including a human facial mesh template based on FLAME topology, facial expression vertex position offset and facial shape parameters, etc., wherein FLAME (Faces Learned with an Articulated Model of Expression) is a 3D face model that contains detailed information on facial geometry and movement.

[0078] It can be understood that since there are a large number of 3D vertices in the facial motion data, the facial motion data is subjected to T-step Gaussian noise processing by the latent diffusion model of facial motion to obtain the kinematic information of the noisy face, namely, facial noise motion.

[0079] In a specific implementation, starting from the original 3D facial motion data, for each time step (t), a noise vector is generated according to a preset noise distribution (such as a Gaussian distribution), and the calculated noise vector is added to the data of the current time step to form noisy data. The above steps are repeated until a predetermined number of time steps (T) is reached, and noise is gradually added to the original data. This gradual noise addition method can effectively control the amount of noise, thereby better maintaining the structural characteristics of the data. The introduction of noise can increase the diversity of the generated samples, making the facial motion data generated by the model during the reconstruction process smoother and reducing unnatural jumps.

[0080] It is worth noting that the potential diffusion model of facial motion is based on Sohl's diffusion framework, which transforms an audio input A encoded by an audio encoder into 1：N With a series of GT facial motion sequences where N is the number of visual frames downsampled to 30 FPS specific to the dataset, and the latent diffusion model of facial motion is designed to be trained on the audio input A 1：N Predicting a facial motion sequence The prediction process is influenced by the sentiment label e i The guidance of which is represented as a one-hot vector. Noise sequence From the standard normal distribution The sampling dimension is similar to the GT facial motion sequence. To match, the latent diffusion model of facial motion directly generates vertex displacements relative to the neutral template, and facial action is generated as follows:

[0081]

[0082] in, is the predicted facial motion sequence, A 1：N is the input audio, t is the diffusion step length, is the noise sequence after diffusion t steps, e i is the emotion label, M∈R V×3 Represents a neutral facial mesh template with V vertices, where V represents 5023 mesh vertices.

[0083] In the diffusion process, the generation is modeled as a Markov denoising process. During the training process, the GT motion sequence of the face shape (N, V) in the training set is After T steps of Gaussian noise addition, the noise sequence is obtained V is the number of vertices, and the forward diffusion is t steps, as follows:

[0084]

[0085] Where N is the number of motion frames in the sequence, t is the diffusion step length randomly selected from [1, T], and βt is a constant noise at t and follows a predefined variance table, β t is a factor that controls the noise intensity and can be adjusted over time.

[0086] It is worth noting that the denoising model is trained to reverse the diffusion process. The model predicts the t-step noise, that is, relying on the conditional distribution function Motion is predicted by denoising. For 3D human and face motion, better results are achieved by directly predicting the motion data. Therefore, the latent diffusion model of facial motion is used through the face denoising network. From the noise sequence Directly predict facial motion in The reverse diffusion is as follows:

[0087]

[0088] Among them, α t =1-β t , A 1：N is the audio feature, e i are emotion labels, representing the facial denoising network Generate the required additional conditions.

[0089] Using simple loss as the main objective of the training model, using simple loss to learn variational bounds is beneficial to improving the quality of sampling results. Directly predict GT vertex displacement loss As follows:

[0090]

[0091] Understandably, it is worth noting that speech-to-text driven generation of 3D facial motions with different emotions relies on a large number of datasets with audio-3D emotional facial motion sequence matching. However, existing speech-driven 3D facial motion methods are trained on 3D scan datasets with paired audio, but there is no large-scale 3D scan dataset with emotional speech sequences. Therefore, the facial motion generation model is trained on the MEAD dataset, which is an emotional video dataset without 3D supervision. The dataset contains 7 emotion categories (anger, disgust, fear, happiness, contempt, sadness and surprise). In order to make full use of the dataset, the dataset is reconstructed using a monocular facial reconstruction model to obtain a 3D facial motion dataset with emotions. According to the monocular reconstruction method, EMOCAv2 can best restore emotional 3D faces, and SPECTRE improves lip motion synchronization. Therefore, SPECTRE's lip reading loss is used to enhance EMOCAv2. The pseudo Ground Truth (GT) 3D facial expression data of different emotional types and intensities are predicted from the MEAD video dataset through the adjusted model, thereby expanding the data diversity and solving the problem of insufficient 3D facial motion datasets and lack of diverse emotional expressions.

[0092] Reconstructing 3D facial motion data through the optimized EMOCAv2 model The optimized EMOCAv2 model is based on the FLAME facial model to reconstruct 3D facial motion data, such as Figure 2 As shown, Figure 2 To reconstruct the 3D facial motion data flow chart, the EMOCAv2 model predicts the FLAME Blendshape expression parameters express, head pose, and facial shape of facial expressions from the videos in the MEAD dataset. In order to focus on the generation of more refined emotional facial expression movements without being affected by the identity and posture of the character, we only use the FLAME Blendshape expression parameters express of facial expressions, and the head pose and facial shape are set to 0 (indicating the use of fixed head pose and facial shape). Finally, compared to the FLAME Blendshape data, the FLAME 3D mesh data has more vertices, which is conducive to the generation of more subtle facial details. We convert the obtained FLAME Blendshape parameters, express head pose, and facial shape into FLAME 3D mesh data and record them as The data dimension is (N, 5023, 3), where N is the number of frames, 5023 is the number of vertices in the facial mesh, and each vertex is represented by three coordinates (x, y, z). Then, the Python movipy library is used to extract the audio data from the corresponding video and save it in .m4a format, and the facial motion data based on the FLAME 3D mesh is converted into With the audio extracted from the video A 1：N 、Emotional tags i Correspondence matching, that is, each 3D facial motion data corresponds to audio and emotion labels.

[0093] Step S30, inputting the speech sequence, the facial noise movement and the emotion data into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0094] It should be noted that the facial motion generation model is a 3D emotional facial motion model driven by diffuse speech text, including a TTS text-to-speech module, an audio encoder, a facial motion encoder, an emotion encoder, a facial motion decoder and a 3D facial expression rendering module. The audio features, facial motion features and emotion features are encoded respectively by the audio encoder, the facial motion encoder and the emotion encoder, the features are fused by the facial motion decoder, and the audio and emotion features are injected in layers by the facial denoising module MCHC. It can generate facial animations corresponding to the emotions conveyed in the audio to match the speaking style of a specific emotion, and then generate facial expression videos of a single emotion or facial expression videos with continuous emotion changes.

[0095] It is understandable that the facial motion generation model can be built based on Diffusion+ neural network, using Python as the language, Intel(R) i7 13700K CPU and 64GB RAM, running on a LINUX system, and in the training phase, the framework uses NVIDIATM on the PyTorch platform All experiments were implemented on GeForce RTX 4090s GPU. The model was trained for 50 epochs using the Adam optimizer with a learning rate of 0.0001, a batch size of 1, a diffusion step of 1000, and a feature dimension of 512. In MCHC, the number of audio attention layers of the first temporal encoder (i.e., the first temporal transformer) and the number of emotion attention layers of the second temporal encoder (i.e., the second temporal transformer) are 4 and 8 respectively, and the number of attention heads is 4. During training, the audio feature extractor of HuBERT was frozen. In the inference phase, the noise is initialized by At time step t and audio A1：N In each step, the predicted motion is diffused again and fed to the next step until the final motion sequence A is obtained. 1：N .

[0096] It is worth noting that the facial expression video with continuous emotion transformation is obtained through continuous emotion transformation. Continuous emotion transformation is realized through the continuous emotion conversion module, which includes multi-emotion settings, 3D facial animation generation models of different emotions, and multi-emotion smoothing algorithms. Continuous and natural emotion changes can be achieved at the frame level in a single sequence, which is conducive to enhancing the naturalness of communication and improving emotional intelligence and interaction capabilities. The continuous emotion conversion algorithm can control the emotions expressed by 3D facial movements according to the emotional content required by the original audio in the same audio sequence, guiding the realization of fine-grained generation control.

[0097] It is understandable that continuous emotion transformation requires the user to input text or speech content through the keyboard, and input emotion transformation frames and emotion labels before and after the transformation. After receiving the speech content, emotion transformation frames and emotion labels, the system sends them to the 3D facial motion generation model of different emotions as the input of the model, and processes the facial movements between different emotions through the multi-emotion conversion algorithm to generate the final 3D facial motion sequence with continuous emotion transformation, and then generates a 3D facial expression video with continuous emotion transformation, completing a continuous emotion transformation.

[0098] It is worth noting that after generating facial expression videos, it is necessary to quantitatively evaluate the facial expressions in the video, including lip errors and emotional face errors, which are used to evaluate the quality of the generated 3D emotional facial expressions and the accuracy of lip synchronization. In order to ensure a fair comparison with other methods, the vertex error of previous work is used to measure lip synchronization. As a supplement, the facial vertex error is used to evaluate emotional faces, and new quantitative evaluation indicators are extended on the basis of the above methods to make the evaluation more comprehensive. According to the 3D facial motion sequence generated by the facial motion generation model, the lip vertex error of each frame of the generated 3D facial motion sequence and the corresponding lip vertex error in the Ground Truth facial motion is calculated, and it is used as a measure of overall lip synchronization. Then, the vertex error of the whole face and upper face corresponding to each frame of the generated 3D facial motion sequence and the Ground Truth facial motion is calculated, and it is used as a measure of emotional facial expressions. The quantitative evaluation results generated by the evaluation are compared with other methods to evaluate the quality of the generated 3D emotional facial expressions.

[0099] It is worth noting that when evaluating the quality of generated facial expressions, facial vertex errors are used to evaluate emotional faces. The evaluation criteria include lip vertex error (LVE), face vertex error (FVE) and emotion mean error (EVE). Lip Vertex Error-Mean (LVE-M) and Face Vertex Error-Mean (FVE-M) are new quantitative evaluation indicators extended on the basis of the above, making the evaluation more comprehensive. LVE lip synchronization plays a vital role in enhancing the realism of the entire face. The lip vertex error (LVE) method proposed in MeshTalk is used to evaluate the quality of generated facial expressions. The maximum L2 error between each frame and the corresponding lip vertex in GT is calculated, and its average value is used as a measure of overall lip synchronization. The LVE-mean indicator is introduced to add the average L2 error of lip vertices to the metric. In addition to lip movement, the focus of FVE and EME emotional 3D facial animation also depends on the entire facial area. FVE and EVE are proposed in FDM to evaluate model performance, which are expressed as vertex errors of the entire face and upper face area, respectively. Similarly, the FVE-mean indicator is introduced to add the average L2 error of facial vertices to the metric for a more comprehensive evaluation. The calculation formulas of EME and FVE are as follows:

[0100]

[0101] in, represents the motion of the vth vertex, V e and V f They represent the upper face vertex set and the entire face vertex set respectively, mean() represents the average L2 distance between vertices, and max() represents the maximum L2 distance between vertices.

[0102] The present embodiment provides a facial movement generation method, which first responds to multimedia data and emotional data input by a user, determines a voice sequence according to the multimedia data; obtains facial movement data, and adds noise according to the facial movement data to obtain facial noise movement; inputs the voice sequence, the facial noise movement and the emotional data into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes, and can generate diverse and realistic facial expressions to achieve flexible and continuous emotion control.

[0103] In summary, the present embodiment responds to the multimedia data and emotional data input by the user, increases the diversity of generated samples by adding noise to the facial motion data, makes the facial motion data smoother, and obtains facial noise motion. Then, through the facial motion generation model, a facial expression video of a single emotion containing diverse and realistic facial expressions is generated according to the speech sequence, facial noise motion and emotional data, or a facial expression video with continuous emotional changes is generated using a multi-emotion smoothing algorithm, thereby achieving flexible and continuous emotional control, overcoming the technical defects of rough and insufficient diversity of facial emotional expression and rigid transformation process between different emotions, and being able to generate diverse and realistic facial expressions to achieve flexible and continuous emotional control.

[0104] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 3 , the step S30 further includes steps S301-S303:

[0105] Step S301, inputting the speech sequence, the facial noise motion and the emotion data into corresponding encoders in a facial motion generation model respectively, to generate audio features, facial noise motion features and emotion features.

[0106] It should be noted that the facial motion generation model includes multiple encoders, namely, an audio encoder, a facial motion encoder, and an emotion encoder. The audio encoder is used to extract audio features from speech sequences, the facial motion encoder is used to extract facial noise motion features, and the emotion encoder is used to extract emotion features. These encoders process the input data separately to ensure that the subsequent generation model can accurately generate realistic facial expression videos based on these features.

[0107] It can be understood that the audio encoder accepts the speech sequence input by the user or converted from the TTS text-to-speech module, and uses the pre-trained large language model HuBERT to extract audio features from the original audio signal. In the pre-training stage, HuBERT uses hidden units to represent the audio signal, using the pre-trained version of the HuBERT architecture and the released HuBERT-base-ls960 version. The large language model HuBERT in the audio encoder takes the input audio A 1：N Mapped to a quantized feature vector with 1536 channels, then align the audio features and visual frames and map the input frame rate (the frame rate of the HuBERT encoder is 50fps) to the output frame rate 30FPS. Finally, a trainable linear layer is applied to project the feature vector from the 1536-dimensional channel to the 512-dimensional channel to obtain the audio feature A of N frames 1：N ∈RN×512 .

[0108] In a feasible implementation, step S301 may include: converting the speech sequence into audio features through an audio encoder in the facial movement generation model; converting the facial noise movement into facial noise motion features through a facial movement encoder in the facial movement generation model; extracting emotion labels and emotion intensities in the emotion data through an emotion encoder in the facial movement generation model, and encoding the emotion labels and the emotion intensities as emotion features.

[0109] It should be noted that the original speech signal is converted into audio features, and the Mel-frequency cepstral coefficients (MFCC) are usually used to obtain the audio feature matrix. The audio features are input into the HuBERT's Transformer encoder and processed by a multi-layer Transformer network to generate a high-level audio feature representation.

[0110] It can be understood that the facial motion encoder accepts an input 3D facial motion sequence, encodes the input facial motion sequence through convolution and pooling layers, and thus obtains facial motion features, that is, the kinematic information of the noisy face is input into the facial motion encoder to respectively obtain the temporal and spatial characteristics of the facial kinematics, that is, the facial noise motion features.

[0111] It is worth noting that the emotional features of facial motion data are divided into 7 emotional categories, each emotion corresponds to 3 levels of emotional intensity, and the emotion encoder encodes the 7 emotional labels (anger, disgust, fear, happiness, contempt, sadness and surprise) through one-hot encoding. i Then, a linear layer is applied to project the emotion labels to 512-dimensional channels respectively, and unsqueeze() is used to adjust the tensor dimension to be consistent with the feature dimension encoded by the audio encoder. The emotion features encoded by the emotion encoder are mainly used to control the emotion category and facial expressiveness of the generated 3D facial expressions. The emotion encoder encodes different emotion labels and emotion intensities obtained from the input emotion data, including 7 emotion types: anger, disgust, fear, happiness, contempt, sadness and surprise, and three levels of emotion intensity. The emotion label and emotion intensity are encoded in a one-hot encoding method and sent to the next layer as emotion features.

[0112] Step S302: input the audio features, the facial noise motion features, and the emotion features into a facial motion decoder in a facial motion generation model to generate a facial motion sequence of a single emotion.

[0113] It should be noted that the facial motion decoder is composed of a facial denoiser and a motion decoder, including a first time encoder Transformer and a second encoder time Transformer. By fusing four multimodal features of audio, facial motion, diffusion T and emotion, the kinematic information of the noisy face is subjected to T-step denoising to obtain the predicted result of the target facial motion, i.e., the facial motion sequence, which is controlled by different types of emotions.

[0114] In a feasible implementation, step S302 may include: inputting the audio features, the facial noise motion features and the emotional features into a facial motion decoder in a facial motion generation model, and performing feature fusion on the audio features and the facial noise motion features through the facial motion decoder and injecting them into a first time encoder to generate facial denoising motion; injecting the emotional features and the facial denoising motion into a second time encoder to generate a facial motion sequence of a single emotion.

[0115] It should be noted that the audio features encoded by the audio encoder, the facial noise motion features encoded by the facial motion encoder, and the diffusion step size T are feature fused and injected into the first time encoder (i.e., the first time Transformer) to obtain the facial denoising motion; then the emotion features encoded by the emotion encoder and the facial denoising motion are layered and injected into the second time encoder (i.e., the second time Transformer) to control the generation of the final facial denoising motion in an emotional conditional embedding manner, and the facial denoising motion predicted by the facial motion denoiser is injected into the motion decoder for decoding to obtain a predicted 3D facial motion sequence of a single emotion.

[0116] It can be understood that the first loss of the facial motion generation model is determined according to the predicted 3D facial motion sequence and the original Ground Truth facial motion sequence, and the parameters of the facial motion decoder are updated according to the first loss to obtain an updated facial motion decoder, and the first loss is the reconstruction loss. The denoising process is iterated in sequence, and the denoising process is controlled by the diffusion step size T. In each round of denoising, the noisy facial motion sequence of the previous step is used as input, and the output of the model is continuously generated until the entire denoising process is completed to generate the final 3D facial motion sequence.

[0117] In the specific implementation, the facial motion decoder is responsible for generating the final animation frame from the potential representation of audio features and noise. The decoder consists of a first time encoder (i.e., the first time Transformer) and a second time encoder (i.e., the second time Transformer). In order to use the diffusion model to gradually sample and generate more accurate facial expressions and lip movements, a multi-condition hierarchical cross-attention module (MCHC) is designed, such as Figure 4 As shown, Figure 4 Figure 1 is a schematic diagram of the MCHC structure of the facial denoising network. In the figure, the left side is the MCHC structure. The MCHC structure includes the first self-attention layer (Self-Attention) and the first cross-attention layer (Cross-Attention) of the first temporal encoder Transformer. The multi-head parameter is set to 4, and the attention feature dimension is 512. Among them, the 3D facial motion features are used as input to the first self-attention layer through the Periodic Positionsl Encoding (PPE) encoding features as Q, K, and V, and then the encoded audio features are used as K and V. The output of the first self-attention layer is used as Q and input to the first cross-attention layer. The encoded emotional features and the output of the first temporal Transformer are sequentially used as Q, K, and V to the second self-attention layer (Self-Attention) and the second cross-attention layer (Cross-Attention) of the second temporal Transformer, and the feature dimension remains unchanged. The right side of the figure is the detailed structure of the temporal Transformer. MCHC can better supervise audio and emotional features, and applies two forms of attention mechanisms: audio attention (Audio-Attention) and emotion attention (Emotion-Attention). These mechanisms are crucial to maintaining character emotions and regulating facial movements, respectively.

[0118] For t-step sampling, the MCHC network has a total of 4 inputs of different modalities. First, the audio, emotion, and facial motion features with noise are encoded by the audio encoder, facial motion encoder, and emotion encoder, respectively. The features are encoded using convolution, MLP, and linear layer (LL), respectively. The audio feature A 1：N Fusion with time step t yields F at . Through the noise encoder ε noise For noise sequence Encode to obtain F n . Emotional characteristics i Extract features F through linear layers emo, use MLP to extract features with the diffusion step length t, as follows:

[0119] F at =MLP(Concat(ε audio (A 1：N ), MLP(t))),

[0120]

[0121] Among them, ε audio represents the audio encoder, ε noise Represents a noisy facial motion encoder.

[0122] In the audio attention module, F n The global features of the face are obtained from the self-attention of the first time encoder Transformer. Then the facial features are combined with F at The audio attention module of the first time encoder Transformer is integrated to obtain the intermediate feature f t-1 , as follows:

[0123]

[0124] in, For self-attention, Cross attention for audio.

[0125] In the emotion attention module, use deeper emotion cross attention The network transforms the intermediate feature f t-1 and emotional characteristics F emo Fusion, obtain the input for the next sampling step As follows:

[0126]

[0127] The facial denoising decoding process iterates t steps in sequence until the final prediction is generated. Facial movement sequence, that is, the model completes one epoch of training.

[0128] Step S303, inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model to generate a facial expression video of the single emotion.

[0129] It should be noted that the facial expression rendering module is used to generate the final 3D facial movement sequence based on the FLAME model by driving the 3D facial movements of different emotions with speech text and render it into RGB images, and synthesize the images into videos.

[0130] It can be understood that the core of the FLAME model is a parameterized face model that represents 3D facial geometry and texture by linearly combining pre-trained basis vectors. This model is the PyTorch version of the previously proposed Basic Light Shape Modeling and Expression (FLAME) model, which specifically includes: using Pytorch to load the FLAME model and its pre-trained weights, importing the predicted 3D facial motion data of dimension (N, 5023, 3) into the FLAME model, superimposing the vertex parameters through the FLAME Mesh facial template, and then rendering N frames of RGB images based on the loaded FLAME model, and finally synchronizing the generated image with the corresponding original audio data to generate the final 3D emotional facial expression video.

[0131] It is worth mentioning that the specific rendering process based on FLAME includes: 1) The parametric modeling of the FLAME model controls the contour and size of the face through basic shape parameters, and the expression parameters are used to dynamically change facial expressions, such as smiling, frowning, etc. Through linear combination, a variety of facial expressions can be generated; 2) The lighting model is used to simulate the interaction between the light source and the surface. FlamePyTorch supports a variety of lighting models, such as Phong and Lambertian models, which can calculate the color value of each vertex, taking into account the position, intensity and normal direction of the light source. Use the Phong reflection model or physically based rendering (PBR) technology to simulate the lighting effect, making the rendered facial model more realistic; 3) Texture mapping is the process of applying a two-dimensional image (texture) to the surface of a 3D model. FlamePyTorch allows facial images to be mapped onto 3D models to enhance details and realism. Texture coordinates are associated with the vertices of the model to ensure that the texture can be displayed correctly; 4) View transformation. During the rendering process, 3D coordinates need to be converted into 2D image coordinates, which involves multiple transformation steps, including model transformation, view transformation, and projection transformation. Through these transformations, the final image can be generated from the camera's perspective; 5) Rasterization. Rasterization is the process of converting geometry into pixels. In this step, the computer determines which pixels should be drawn as specific colors based on previously calculated lighting and texture information. Finally, the rendering results are output in the form of images, which users can directly view or further process. The rendering process of FlamePyTorch combines multiple aspects such as geometric modeling, lighting calculation, texture mapping, and image processing to achieve high-quality 3D facial rendering through efficient algorithms and data structures. Its flexibility and scalability allow users to customize rendering effects according to their needs.

[0132] In a feasible implementation, step S303 may include: inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model, and mapping the facial movement sequence into facial expression parameters and shape parameters through the facial expression rendering module; rendering according to the facial expression parameters and the shape parameters to generate a facial animation; synchronously synthesizing the facial animation and the speech sequence to generate a facial expression video of a single emotion.

[0133] It should be noted that the facial expression rendering generation module is used to render the final 3D facial motion sequence into RGB images and synthesize the images into videos. The 3D facial motion is based on the 3D Mesh data of the FLAME model. The facial expression parameters and shape parameters are the FLAME model parameters. The 3DMesh of the FLAME model is driven by the facial expression parameters and shape parameters.

[0134] It can be understood that after receiving the 3D facial motion sequence, the facial expression rendering module performs the following operations: use Pytorch to load the FLAME model and its pre-trained weights, initialize the relevant 3D facial model parameters and network structure; map the final 3D facial motion sequence generated by the facial motion decoder to the FLAME model parameters, including facial expression parameters and shape parameters; input the 3D Mesh and its animation data that drive the FLAME model into the 3D rendering engine (such as Blender, Unity, etc.) for rendering; synchronize the rendered 3D facial animation with the original audio data to generate a 3D facial expression video, showing the 3D facial animation effects for different emotions.

[0135] In this embodiment, the corresponding encoder in the facial motion generation model is used to quickly and accurately encode the speech sequence, facial noise motion and emotion data into audio features, facial noise motion features and emotion features, and then the encoder generated features are decoded by the facial motion decoder to generate a facial motion sequence of a single emotion, and then the facial motion sequence and the speech sequence are rendered and synthesized into a facial expression video of a single emotion by the facial expression rendering module, thereby effectively improving the authenticity and diversity of facial emotional expression.

[0136] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the facial motion generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0137] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 5, the step S30' further comprises steps S301'-S305':

[0138] Step S301 ′, determining different emotion tags according to the emotion data, and acquiring emotion conversion frames.

[0139] It should be noted that when generating a facial expression video with continuous emotion changes, the user inputs multiple emotional expressions, i.e., emotion labels before and after the emotion changes. Emotion labels are identifiers used to describe specific emotional states, such as anger, disgust, fear, happiness, contempt, sadness, and surprise.

[0140] It is understandable that the emotion transition frame specifies the number of transition frames between different emotion states during the continuous emotion transformation process to ensure the smoothness and naturalness of the emotion change. The emotion label and emotion transition frame can be input by the user through a graphical user interface (GUI) or automatically obtained through speech recognition technology.

[0141] Step S302': dividing the speech sequence into a plurality of audio sequences according to the emotion conversion frames.

[0142] It can be understood that the continuous emotion conversion algorithm can perform frame-level emotion conversion by setting the emotion conversion frame i1i2…i n-1 Divide the audio into two or more segments to obtain multiple audio sequences.

[0143] Step S303 ′: generating a plurality of facial movement sequences of different emotions according to the audio sequence, the emotion label and the emotion conversion frame.

[0144] It should be noted that by setting different emotion labels e1e2…e n Emotion control is performed on two or more audio segments, and then n audio sequences, n emotion labels, and n-1 emotion conversion frames are sent together to the voice text-driven 3D facial motion generation model of different emotions to generate n 3D facial motion sequences of different emotions. Then, the multi-emotion smoothing algorithm is used to smooth the facial motion of the n 3D facial motion sequences with different emotions at the set emotion conversion frames to make the emotional changes of facial movements more natural and coherent.

[0145] It is understandable that the process of generating multiple facial motion sequences of different emotions through the 3D facial motion generation model is consistent with the process of generating a facial motion sequence of a single emotion, and this embodiment will not be repeated here. In the generated facial motion sequence, each sequence corresponds to a specific emotion label, and smoothing is performed at the emotion transition frame to ensure the coherence of the emotion change.

[0146] Step S304', processing the facial movement sequences of different emotions at the emotion conversion frames by a multi-emotion smoothing algorithm to generate facial movement sequences of continuous emotions.

[0147] It should be noted that in order to solve the problem of sudden changes in facial expressions under different emotional changes, a multi-emotion smoothing algorithm is used in the emotional change process to optimize the emotional conversion process, so that more natural continuous emotional changes can be made in a piece of audio. The diffusion process is controlled by setting the emotional change frame and the emotional label. The emotional motion change before and after the set frame is controlled by setting the key frame i and an emotional conversion window w. The different emotions of the corresponding frame are selected according to the window for multi-emotion smoothing optimization. The multi-emotion smoothing algorithm is as follows:

[0148]

[0149] Among them, F m Represents the intermediate motion sequence of a frame during the emotion conversion process, which is used to generate multiple smoothed emotion conversion frames. The motion sequence representing the first emotion in the iwth frame, represents the motion sequence of the second emotion in the i+wth frame, and δ is a hyperparameter.

[0150] Set the hyperparameter δ to 0, and increase it by 0.1 each time until δ = 1, and set w to 5. iw:i+w frame uses F m The replacement is carried out in sequence, and so on, to obtain a smoothed continuous emotional motion sequence.

[0151] Step S305 ′: input the speech sequence and the facial movement sequence of the continuous emotions into the facial expression rendering module in the facial movement generation model to generate a facial expression video with continuous emotion changes.

[0152] It should be noted that the smoothed continuous emotion motion sequence output by the multi-emotion smoothing algorithm is matched with the original audio and input into the expression rendering module to render and generate 3D facial motion video of continuous emotions, which can realize natural and multi-segment continuous emotion changes.

[0153] It can be understood that the process by which the facial expression rendering module in the facial motion generation model renders a speech sequence and a facial motion sequence of continuous emotions to generate a facial expression video with continuous emotion changes is consistent with the process of rendering a facial expression video of a single emotion by rendering a speech sequence and a facial motion sequence of a single emotion, and this embodiment will not be repeated here.

[0154] Specifically, different emotion labels and emotion conversion frames are set by a multi-emotion setting algorithm. The text sequence for continuous emotion conversion is converted into a speech signal using a TTS text-to-speech module. The input speech sequence or the setting parameters converted from the TTS text-to-speech module to the speech signal and the multi-emotion setting module are respectively input into the speech text driven 3D facial animation generation method for different emotions to generate a 3D facial motion sequence of the corresponding emotion. The generated emotional facial motion sequence and the emotional facial motion sequence are smoothed at the emotion conversion frame using a multi-emotion smoothing algorithm to obtain the final emotional conversion 3D facial motion sequence, which ensures the coherence and naturalness of the emotional conversion process. Finally, the final emotional conversion 3D facial motion sequence is rendered by a 3D facial expression rendering generation module to generate a 3D facial expression video with continuous emotional transformation. By setting more than two emotional labels and emotional conversion frames and repeating the above steps in sequence, multiple emotional facial expression conversions can be performed on one or more audio sequences, so that the virtual character can match the multiple different emotions embodied by the speech text in time, and a more natural and emotionally deep character performance is achieved. It is of great significance in the fields of animation, film and television production and games.

[0155] It is worth mentioning that the multi-emotion smoothing algorithm obtains the emotion and facial motion sequences before and after the i-th frame sequence, controls the emotional facial expression changes before and after the i-th frame through the facial motion sequence of the intermediate frames of two different emotion sequences, and smoothes the facial motions between different emotions in a linear smoothing manner.

[0156] like Figure 6 As shown, Figure 6 The overall flow chart of facial motion generation is shown. Different emotional 3D facial motion generation systems are driven by voice text. The functions of the facial motion generation system include 3D emotional facial motion generation and continuous emotional expression generation, which can work together. The 3D emotional facial motion generation model is trained by Diffusion+ neural network and the construction of 3D emotional facial motion dataset. The text Text input by the user is converted into a speech sequence Audio1 through TTS or the speech sequence Audio2 input by the user is input into the 3D emotional facial motion generation model, and the emotional labels and key frames are set and input into the 3D emotional facial motion generation model, so as to realize 3D emotional facial motion generation or continuous emotional expression generation, generate emotional 3D facial motion sequence or continuous emotional 3D facial motion sequence, and then perform 3D facial expression rendering to generate facial expression video.

[0157] like Figure 7 As shown, Figure 7This is a schematic diagram of the structure of the 3D emotional facial motion generation model EmoDiffuser. The 3D emotional facial motion generation model denoises the emotional facial motion of the noisy sequence based on audio features and emotional labels. In the diffusion processing stage, the pre-trained audio encoder processes the audio embedding. Subsequently, the facial denoising network MCHC is used for denoising. Two forms of attention mechanisms are applied in MCHC: audio attention (Audio-Attention) and emotional attention (Emo-Attention). In the inference stage, by initializing the noise At time step T and audio A 1：N In each step, the predicted motion is diffused again and fed to the next step until the final motion sequence is obtained.

[0158] By setting key frames to perform continuous emotion conversion in a single sequence, and extracting facial movements of different emotions through a multi-emotion smoothing algorithm to achieve coherent and natural conversion, a continuous emotion 3D facial movement sequence is finally generated, such as Figure 8 As shown, Figure 8 For the continuous expression transformation flow chart, the 1-frame speech sequence Audio1 and the emotion label emo1 are input into the 3D emotional facial motion generation model to generate the emotional 3D facial motion sequence Input the N-frame speech sequence Audio2 and the emotion label emo2 into the 3D emotional facial motion generation model to generate an emotional 3D facial motion sequence Processing emotional 3D facial motion sequences based on a multi-emotion smoothing algorithm Emotional 3D facial motion sequences Emotion 1 facial motion frames and emotional 3D facial motion sequences Emotion 2 facial motion frames in Finally, a continuous emotional 3D facial motion sequence is generated.

[0159] In this embodiment, the speech sequence is divided into multiple audio sequences through emotion conversion frames, and multiple facial movement sequences of different emotions are generated according to the multiple audio sequences, emotion labels and emotion conversion frames. Then, the multiple facial movement sequences of different emotions are smoothed and rendered to synthesize facial expression videos with continuous emotion changes, thereby achieving flexible and continuous emotion control.

[0160] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the facial motion generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0161] This application also provides a facial motion generation device, please refer to Fig. 9, the facial movement generating device comprises:

[0162] The determination module 10 is configured to respond to multimedia data and emotion data input by a user and determine a speech sequence according to the multimedia data.

[0163] The noise adding module 20 is used to obtain facial motion data and perform noise adding according to the facial motion data to obtain facial noise motion.

[0164] The input module 30 is used to input the speech sequence, the facial noise movement and the emotion data into the facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0165] The present embodiment provides a facial movement generation device. The present embodiment first responds to multimedia data and emotional data input by a user, determines a speech sequence according to the multimedia data; obtains facial movement data, and adds noise according to the facial movement data to obtain facial noise movement; inputs the speech sequence, the facial noise movement and the emotional data into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes, and can generate diverse and realistic facial expressions to achieve flexible and continuous emotion control.

[0166] In summary, the present embodiment responds to the multimedia data and emotional data input by the user, increases the diversity of generated samples by adding noise to the facial motion data, makes the facial motion data smoother, and obtains facial noise motion. Then, through the facial motion generation model, a facial expression video of a single emotion containing diverse and realistic facial expressions is generated according to the speech sequence, facial noise motion and emotional data, or a facial expression video with continuous emotional changes is generated using a multi-emotion smoothing algorithm, thereby achieving flexible and continuous emotional control, overcoming the technical defects of rough and insufficient diversity of facial emotional expression and rigid transformation process between different emotions, and being able to generate diverse and realistic facial expressions to achieve flexible and continuous emotional control.

[0167] Optionally, the determination module 10 is further used to: convert the phoneme sequence into an acoustic feature sequence, and generate a digital signal according to the acoustic feature sequence; perform audio simulation according to the digital signal to generate a speech signal; and generate a speech sequence according to the speech signal.

[0168] Optionally, the input module 30 is also used to input the speech sequence, the facial noise motion and the emotion data into the corresponding encoders in the facial motion generation model respectively to generate audio features, facial noise motion features and emotion features; input the audio features, the facial noise motion features and the emotion features into the facial motion decoder in the facial motion generation model to generate a facial motion sequence of a single emotion; input the speech sequence and the facial motion sequence of the single emotion into the facial expression rendering module in the facial motion generation model to generate a facial expression video of a single emotion.

[0169] Optionally, the input module 30 is also used to convert the speech sequence into audio features through the audio encoder in the facial movement generation model; convert the facial noise movement into facial noise motion features through the facial movement encoder in the facial movement generation model; extract emotion labels and emotion intensity in the emotion data through the emotion encoder in the facial movement generation model, and encode the emotion labels and the emotion intensity as emotion features.

[0170] Optionally, the input module 30 is also used to input the audio features, the facial noise motion features and the emotional features into a facial motion decoder in the facial motion generation model, and to perform feature fusion of the audio features and the facial noise motion features through the facial motion decoder and inject them into a first time encoder to generate facial denoising motion; and to inject the emotional features and the facial denoising motion into a second time encoder to generate a facial motion sequence of a single emotion.

[0171] Optionally, the input module 30 is also used to input the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in the facial movement generation model, and map the facial movement sequence into facial expression parameters and shape parameters through the facial expression rendering module; render according to the facial expression parameters and the shape parameters to generate a facial animation; and synchronously synthesize the facial animation and the speech sequence to generate a facial expression video of a single emotion.

[0172] Optionally, the input module 30 is also used to determine different emotion labels based on the emotion data and obtain emotion conversion frames; divide the speech sequence into multiple audio sequences according to the emotion conversion frames; generate multiple facial movement sequences of different emotions according to the audio sequences, the emotion labels and the emotion conversion frames; process the facial movement sequences of different emotions using a multi-emotion smoothing algorithm at the emotion conversion frames to generate facial movement sequences of continuous emotions; input the speech sequence and the facial movement sequence of continuous emotions into a facial expression rendering module in a facial movement generation model to generate a facial expression video with continuous emotion changes.

[0173] The facial motion generation device provided by the present application adopts the facial motion generation method in the above embodiment, which can solve the technical problem of how to generate diverse and realistic facial expressions and achieve flexible and continuous emotional control. Compared with the prior art, the beneficial effects of the facial motion generation device provided by the present application are the same as the beneficial effects of the facial motion generation method provided by the above embodiment, and the other technical features in the facial motion generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0174] The present application provides a facial motion generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the facial motion generation method in the above-mentioned embodiment one.

[0175] Reference below Fig.10 , which shows a schematic diagram of the structure of a facial motion generation device suitable for implementing the embodiment of the present application. The facial motion generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The facial motion generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0176] like Fig.10As shown, the facial motion generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the facial motion generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the facial motion generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a facial motion generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0177] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0178] The facial motion generation device provided by the present application adopts the facial motion generation method in the above embodiment, which can solve the technical problem of how to generate diverse and realistic facial expressions and achieve flexible and continuous emotional control. Compared with the prior art, the beneficial effects of the facial motion generation device provided by the present application are the same as the beneficial effects of the facial motion generation method provided by the above embodiment, and the other technical features in the facial motion generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0179] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0181] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the facial motion generation method in the above-mentioned embodiment.

[0182] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0183] The computer-readable storage medium may be included in the facial motion generation device; or may exist independently without being assembled into the facial motion generation device.

[0184] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a facial motion generation device, the facial motion generation device: responds to multimedia data and emotional data input by a user, determines a speech sequence according to the multimedia data; obtains facial motion data, and adds noise according to the facial motion data to obtain facial noise motion; inputs the speech sequence, the facial noise motion and the emotional data into a facial motion generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

[0185] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0186] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0188] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned facial motion generation method, and can solve the technical problem of how to generate diverse and realistic facial expressions and achieve flexible and continuous emotional control. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the facial motion generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0189] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the facial motion generation method as described above are implemented.

[0190] The computer program product provided by the present application can solve the technical problem of how to generate diverse and realistic facial expressions and achieve flexible and continuous emotion control. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the facial movement generation method provided by the above embodiment, which will not be repeated here.

[0191] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A facial motion generation method, characterized in that: The method comprises: In response to multimedia data and emotion data input by a user, determining a speech sequence according to the multimedia data; Acquire facial motion data, and add noise according to the facial motion data to obtain facial noise motion; The speech sequence, the facial noise movement and the emotion data are input into a facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

2. The method according to claim 1, characterized in that The step of determining a speech sequence according to the multimedia data comprises: When the multimedia data is text data, decomposing the text data into a phoneme sequence; Converting the phoneme sequence into an acoustic feature sequence, and generating a digital signal according to the acoustic feature sequence; Performing audio simulation according to the digital signal to generate a voice signal; A speech sequence is generated according to the speech signal.

3. The method according to claim 1, characterized in that The step of inputting the speech sequence, the facial noise movement and the emotion data into a facial movement generation model to obtain a facial expression video comprises: Inputting the speech sequence, the facial noise motion and the emotion data into corresponding encoders in a facial motion generation model respectively to generate audio features, facial noise motion features and emotion features; Inputting the audio features, the facial noise motion features and the emotion features into a facial motion decoder in a facial motion generation model to generate a facial motion sequence of a single emotion; The speech sequence and the facial movement sequence of the single emotion are input into a facial expression rendering module in a facial movement generation model to generate a facial expression video of the single emotion.

4. The method according to claim 3, characterized in that The step of inputting the speech sequence, the facial noise motion and the emotion data into corresponding encoders in the facial motion generation model to generate audio features, facial noise motion features and emotion features comprises: Converting the speech sequence into audio features via an audio encoder in the facial movement generation model; The facial noise motion is converted into facial noise motion features by a facial motion encoder in the facial motion generation model; The emotion label and the emotion intensity in the emotion data are extracted by the emotion encoder in the facial movement generation model, and the emotion label and the emotion intensity are encoded as emotion features.

5. The method according to claim 3, characterized in that The step of inputting the audio feature, the facial noise motion feature, and the emotion feature into a facial motion decoder in a facial motion generation model to generate a facial motion sequence of a single emotion comprises: Input the audio feature, the facial noise motion feature and the emotion feature into a facial motion decoder in a facial motion generation model, and perform feature fusion on the audio feature and the facial noise motion feature through the facial motion decoder and inject them into a first time encoder to generate facial denoising motion; The emotion feature and the facial denoising motion are injected into a second temporal encoder to generate a facial motion sequence of a single emotion.

6. The method according to claim 3, characterized in that The step of inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model to generate a facial expression video of the single emotion comprises: Inputting the speech sequence and the facial movement sequence of the single emotion into a facial expression rendering module in a facial movement generation model, and mapping the facial movement sequence into facial expression parameters and shape parameters through the facial expression rendering module; Rendering is performed according to the facial expression parameters and the shape parameters to generate facial animation; The facial animation and the speech sequence are synchronously synthesized to generate a facial expression video of a single emotion.

7. The method according to claim 1, characterized in that The step of inputting the speech sequence, the facial noise movement and the emotion data into a facial movement generation model to obtain a facial expression video further comprises: Determine different emotion tags according to the emotion data, and obtain emotion conversion frames; Dividing the speech sequence into a plurality of audio sequences according to the emotion conversion frame; Generate a plurality of facial movement sequences of different emotions according to the audio sequence, the emotion label and the emotion conversion frame; Processing the facial movement sequences of different emotions at the emotion conversion frames by a multi-emotion smoothing algorithm to generate facial movement sequences of continuous emotions; The speech sequence and the facial movement sequence of the continuous emotions are input into a facial expression rendering module in a facial movement generation model to generate a facial expression video with continuous emotion changes.

8. A facial motion generating device, characterized in that: The facial movement generating device comprises: A determination module, configured to determine a speech sequence according to the multimedia data in response to multimedia data and emotion data input by a user; A noise adding module, used for acquiring facial motion data, and performing noise adding according to the facial motion data to obtain facial noise motion; The input module is used to input the speech sequence, the facial noise movement and the emotion data into the facial movement generation model to obtain a facial expression video, wherein the facial expression video includes a facial expression video of a single emotion or a facial expression video with continuous emotion changes.

9. A facial motion generation device, characterized in that: The facial motion generation device comprises: a memory, a processor, and a facial motion generation program stored in the memory and executable on the processor, wherein the facial motion generation program is configured to implement the facial motion generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a facial motion generation program, and when the facial motion generation program is executed by the processor, the facial motion generation method according to any one of claims 1 to 7 is implemented.

Images