An audio-driven facial animation generation method, system, device and medium

Abstract

The application discloses an audio-driven facial animation generation method, system, device and medium. The audio-driven facial animation generation method comprises the following steps: audio feature extraction is performed by using combined embedding from two pre-trained audio encoders; the extracted audio features are fed into audio attention and time step embedding of a diffusion model; the diffusion model generates a key frame sequence at a low frame rate under the condition of audio input and identity frames; the diffusion model interpolates between key frames under the condition of two continuous frames in the key frame sequence; and the diffusion model is optimized in combination with a loss function of an RGB space and a latent feature space. The application combines key frame generation and interpolation, utilizes extended temporal context, can generate a video which maintains high quality and captures long-term temporal dependencies over time, and effectively maintains temporal consistency and realism of a long sequence.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video generation, and in particular to an audio-driven facial animation generation method, system, device and medium. BACKGROUND

[0002] With the development of generative models such as generative adversarial networks (GAN) and diffusion models (DM), the field of audio-driven facial animation has made significant progress. These methods have greatly enhanced the realism and expressiveness of facial animations, enabling promising applications in virtual assistants, education, virtual reality, and helping communication disorders. Therefore, there has been a sharp increase in high-resolution, natural, long-term audio-driven facial animations.

[0003] Although early methods of audio-driven facial animation were limited in head rotation or only focused on generating the mouth region, current methods have developed to produce results that are almost indistinguishable from real videos. Despite this progress, most methods struggle to handle longer audio inputs, with problems of identity drift and overall quality degradation after the first few seconds. To extend the generation length, some methods incorporate additional spatial information, such as target head positions or landmarks, as inputs to the model. While this can improve temporal consistency, it limits the animation to predefined facial movements, restricting expressiveness. Other methods use motion frameworks to provide context for previous motions, however, many autoregressive methods suffer from error accumulation over time, thus reducing overall quality. SUMMARY

[0004] The present application aims to solve the problems of identity drift and quality degradation in existing long video generation processes, and provides an audio-driven facial animation generation method, system, device and medium that combines keyframe generation and interpolation to generate videos that maintain high quality and capture long-term temporal dependencies over time.

[0005] To this end, the present application adopts the following technical solutions.

[0006] In a first aspect, the present application provides an audio-driven facial animation generation method, comprising:

[0007] Step 1), audio feature extraction using combined embeddings from two pre-trained audio encoders;

[0008] Step 2), feeding the extracted audio features into audio attention and time step embeddings of a diffusion model;

[0009] Step 3), the diffusion model generates a sequence of keyframes at a low frame rate conditioned on the audio input and identity frames;

[0010] Step 4), the diffusion model interpolates between keyframes given two consecutive frames in the keyframe sequence;

[0011] Step 5), the diffusion model is optimized by combining the loss functions in RGB space and latent feature space.

[0012] In the first stage, given the identity frame and audio input, the keyframe sequence is generated at a low frame rate, spanning multiple seconds and eliminating the need for motion frames. In the second stage, interpolation is used to fill in the intermediate frames, ensuring smooth transitions and temporal consistency. The invention divides the generation into two parts, implicitly separating motion and identity control, resulting in more natural motion and improved identity preservation over time. For longer sequences, this process can be repeated, interpolating to generate seamless transitions between segments.

[0013] Further, in step 1), the two pre-trained audio encoders are WavLM and BEATs, WavLM captures linguistic content from speech, and BEATs are trained to extract features from acoustic signals including non-speech sounds.

[0014] Further, in step 2), the mechanism of feeding audio features into the diffusion model includes:

[0015] 2.1) Audio Attention: Combined Embeddings As keys and values in the cross-attention layer within the U-Net architecture, the diffusion model can focus on relevant audio features;

[0016] where, WavLM represents the pre-trained audio encoder, BEATs represents the pre-trained audio encoder;

[0017] 2.2) Time Step Embedding: Add to the time step embedding , so , represents the fusion feature of the time step embedding and audio features, and MLP represents a fully connected neural network. Further promoting the alignment of the diffusion model between images and audio frames, improving lip synchronization.

[0018] Further, in step 3), the specific process of generating a keyframe sequence is:

[0019] 3.1) Given a noisy input sequence , the goal is to generate a sequence of a person speaking in sync with the given audio;

[0020] 3.2) To provide identity and background information, identity frames The VAE encoder is connected with a noise input, effectively utilizing the skip connections of the U-Net architecture to preserve input details;

[0021] 3.3) Conditioned on the audio input and the identity frame, a diffusion model generates a sequence of keyframes with length at a low frame rate, spaced frames apart, which effectively capture long-range temporal dependencies and serve as anchors for the subsequent interpolation stage.

[0022] Further, the specific process of step 4) is as follows:

[0023] 4.1) Obtain two consecutive frames and from the keyframe sequence as conditioning frames;

[0024] 4.2) Create a sequence where represents the learned embedding of the missing frames, by inputting the shape of the start frame and the end frame; the sequence is concatenated with the noise input channel-wise and interpolated by the diffusion model to obtain consecutive intermediate frames.

[0025] Further, the specific process of optimizing the diffusion model in step 5) is as follows:

[0026] 5.1) Decode the latent feature space back to the RGB space to obtain the decoded frame , apply L2 loss between the decoded frame and the ground truth frame , and add it to the L2 loss between the latent feature space and ; represents the ground truth feature space;

[0027] 5.2) Optimize the parameters of the diffusion model during backpropagation.

[0028] In a second aspect, the present application provides an audio-driven facial animation generation system for implementing the above-mentioned audio-driven facial animation generation method, which comprises:

[0029] An audio feature extraction unit: audio feature extraction is performed using combined embeddings from two pre-trained audio encoders;

[0030] An audio feature feeding unit: the extracted audio features are fed into the audio attention and time step embeddings of the diffusion model;

[0031] A keyframe sequence generation unit: a diffusion model generates a sequence of keyframes conditioned on the audio input and the identity frame at a low frame rate;

[0032] Interpolation unit: Diffusion model interpolates between keyframes given two consecutive frames in the sequence of keyframes as conditions.

[0033] Diffusion model optimization unit: Diffusion model is optimized in combination with loss functions in RGB space and latent feature space.

[0034] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the audio-driven facial animation generation method described above when executing the computer program.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the audio-driven facial animation generation method described above.

[0036] The present application has the beneficial effects that: the present application combines key frame generation and interpolation by key frame generation and interpolation, and can generate videos that maintain high quality and capture long-term temporal dependencies over time by using extended temporal context, effectively maintaining long sequence temporal consistency and realism. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A flowchart of an audio-driven facial animation generation method of the present application;

[0038] Figure 2 A process schematic diagram of an audio-driven facial animation generation method of the present application;

[0039] Figure 3 A composition diagram of an audio-driven facial animation generation system of the present application;

[0040] Figure 4 A logical structure schematic diagram of an electronic device provided in the specific embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described and explained with reference to the accompanying drawings and specific embodiments.

[0042] Example 1

[0043] As shown in Figure 1 and Figure 2 , the present embodiment is an audio-driven facial animation generation method, comprising the following steps:

[0044] Step 1), audio feature extraction using combined embeddings from two pre-trained audio encoders.

[0045] The two pre-trained audio encoders are WavLM and BEATs. WavLM captures linguistic content from speech; BEATs are trained to extract features from a wider range of acoustic signals, including non-speech sounds. With denoting the WavLM pre-trained audio encoder, denoting the BEATs pre-trained audio encoder.

[0046] Step 2), feed the extracted audio features into the audio attention and time step embedding of the diffusion model.

[0047] On the one hand, the combined embedding allows the diffusion model to focus on relevant audio features as keys and values in the cross-attention layer within the U-Net architecture; on the other hand, the is added to the diffusion time step embedding , so that , denotes the fusion feature of the time step embedding and the audio features, and MLP denotes a fully connected neural network. This further drives the diffusion model to align the image and audio frames, improving lip synchronization.

[0048] Step 3), given the audio input and identity frames, the diffusion model generates a sequence of key frames at a low frame rate.

[0049] First, given a noisy input sequence ; the goal is to generate a sequence of a person speaking in sync with the given audio;

[0050] Second, to provide identity and background information, the identity frames are connected to the noisy input through the VAE encoder, effectively utilizing the skip connections of the U-Net architecture to preserve input details;

[0051] Finally, given the audio input and identity frames, the diffusion model generates a sequence of key frames with length T and interval S frames, which effectively capture long-range temporal dependencies and serve as anchor points for the subsequent interpolation stage.

[0052] Step 4), given two consecutive frames in the key frame sequence, the diffusion model interpolates between the key frames.

[0053] First, two consecutive frames and are obtained from the key frame sequence as condition frames;

[0054] Then, to match the input shape of the starting frame and the ending frame, create a sequence where Learned embedding representing missing frames. This sequence is concatenated with the noisy input channel-wise and interpolated through the diffusion model to obtain coherent intermediate frames.

[0055] Step 5), loss function combining RGB space and latent feature space optimizes the diffusion model.

[0056] First, the latent feature space Decoding back to the RGB space to obtain the decoded frame , an L2 loss is applied between the decoded frame and the ground truth frame , and added to the L2 loss between the latent feature space and ; denotes the ground truth feature space;

[0057] Then, the parameters of the diffusion model are optimized during the backpropagation process.

[0058] Compared with the existing method of audio-driven face animation generation, the present application solves the problems of identity drift and quality degradation in the long video generation process, and realizes long-time, coherent and natural audio-driven face animation generation.

[0059] Application example

[0060] In order to verify the effectiveness of the method of the present application (i.e. Example 1), the training process of the method of the present application uses HDTF (High Definition Talking Face) and CelebV-Text as data sets. Among them, the HDTF data set focuses on the flow-oriented high-resolution audio visual face generation, involving nearly two years of speaker videos, containing 362 different characters, and the video stream is cut into 512*512 face; CelebV-Text is a data set containing 70,000 different face video clips, each clip has 20 text descriptions.

[0061] The performance of the model trained by the objective evaluation method of the present application is evaluated. For HDTF and CelebV-Text, the aesthetic quality index (AQ), Fr'echet initial distance (FID) and learning perceptual image patch similarity (LPIPS) are used as evaluation indexes of image quality; and Fr'echet video distance (FVD) and smoothness index are used to evaluate the general video quality.

[0062] Table 1 shows the quantitative results of the method of the present application on the HDTF and CelebV-Text data sets. The best results are in bold.

[0063] Table 1: Quantitative results of audio-driven face animation generation on HDTF and CelebV-Text data sets

[0064]

[0065] Results show that the method of the present application achieves the lowest FID and FVD, indicating higher realism and temporal coherence. At the same time, the method achieves the highest AQ and LPIPS, confirming the visual appeal of the generated animation. Although SadTalker achieves the highest smoothness, the method of the present application ranks second, outperforming SadTalker in other indicators and overall performance.

[0066] Embodiment 2

[0067] The present embodiment provides an audio-driven facial animation generation system for implementing the audio-driven facial animation generation method described in Embodiment 1, as shown in Figure 3 , which consists of an audio feature extraction unit, an audio feature feeding unit, a keyframe sequence generation unit, an interpolation unit, and a diffusion model optimization unit.

[0068] The audio feature extraction unit: uses combined embeddings from two pre-trained audio encoders for audio feature extraction; this unit is used to implement the function of step 1) in Embodiment 1, and will not be repeated here.

[0069] The audio feature feeding unit: feeds the extracted audio features into the audio attention and time step embeddings of the diffusion model; this unit is used to implement the function of step 2) in Embodiment 1, and will not be repeated here.

[0070] The keyframe sequence generation unit: with audio input and identity frame as conditions, the diffusion model generates a keyframe sequence at a low frame rate; this unit is used to implement the function of step 3) in Embodiment 1, and will not be repeated here.

[0071] The interpolation unit: with two consecutive frames in the keyframe sequence as conditions, the diffusion model interpolates between keyframes; this unit is used to implement the function of step 4) in Embodiment 1, and will not be repeated here.

[0072] The diffusion model optimization unit: combines the loss function of the RGB space and the latent feature space to optimize the diffusion model; this unit is used to implement the function of step 5) in Embodiment 1, and will not be repeated here.

[0073] It should be noted that each unit in the above audio-driven facial animation generation system can be realized by software, hardware and combinations thereof, in whole or in part. The above units can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in software form, so as to be called and executed by the processor to perform the operations corresponding to each unit. For specific limitations of an audio-driven facial animation generation system, refer to the limitations of an audio-driven facial animation generation method (i.e., embodiment 1) in the above, both of which have the same functions and effects, and will not be described here.

[0074] Embodiment 3

[0075] The embodiment provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication. The memory stores a computer program capable of being executed by the at least one processor, and the computer program is used for causing the electronic device to execute the method according to the embodiment 1 of the application when being executed by the at least one processor.

[0076] Embodiment 4

[0077] The embodiment provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is used for causing a computer to execute the method according to the embodiment 1 of the application when being executed by a processor of the computer.

[0078] Reference Figure 4 The structure block diagram of the electronic device 400 which can be a server or a client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent a variety of forms of digital electronic computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent a variety of forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in this document, their connections, and relationships, and their functions, are merely examples and are not intended to limit implementations of the present application described and / or claimed herein.

[0079] As Figure 4As shown, the electronic device 400 includes a computing unit 401 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 402 or a computer program loaded into a random access memory (RAM) 403 from a storage unit 408. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0080] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, an output unit 407, the storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device 400, and can receive inputted numerical or character information, as well as generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 407 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0081] The computing unit 401 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs various methods and processes described above. For example, in some embodiments, the aforementioned audio-driven facial animation generation system can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 can be configured to perform the aforementioned audio-driven facial animation generation system by any other appropriate means, such as by means of firmware.

[0082] Program code for carrying out operations of the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as part of a separate software package, and partially on a remote machine or entirely on a remote machine or server.

[0083] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0084] As used in the present application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or data to a programmable processor.

[0085] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0086] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0087] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0088] Those skilled in the art will readily observe that numerous modifications and alterations of the device and process can be made without departing from the scope of the present application. Accordingly, the above disclosure is intended to be illustrative only and not limiting of the scope of the present application. The present application is limited only as described in the following claims.

Claims

1. An audio-driven facial animation generation method, characterized in that: include: Step 1) audio feature extraction using the combined embeddings from two pre-trained audio encoders; Step 2) Feed the extracted audio features into the audio attention and time step embedding of the diffusion model; Step 3), based on the audio input and identity frame, the diffusion model generates a key frame sequence at a low frame rate; Step 4), based on two consecutive frames in the key frame sequence, the diffusion model interpolates between the key frames; Step 5), optimize the diffusion model by combining the loss function of RGB space and latent feature space; In step 1), the two pre-trained audio encoders are WavLM and BEATs, the WavLM captures language content from speech, and the BEATs are trained to extract features from acoustic signals including non-speech sounds; In step 2), the mechanism of feeding the audio features into the diffusion model includes: 2.1) Audio Attention: Combining Embeddings As keys and values ​​in the criss-cross attention layer within the U-Net architecture; in, represents the WavLM pre-trained audio encoder, represents the BEATs pre-trained audio encoder; 2.2) Time step embedding: Add to time step embedding In, therefore , represents the time step embedding The fusion features with audio features, MLP represents a fully connected neural network; The specific process of optimizing the diffusion model in step 5) is as follows: 5.1) The latent feature space Decode back to RGB space to get the decoded frame , in the decoded frame and ground truth frames Apply L2 loss between and add it to the latent feature space and In the L2 loss between represents the ground truth feature space; 5.2) Optimize the parameters of the diffusion model during backpropagation.

2. The audio-driven facial animation generation method according to claim 1, characterized in that: The specific process of generating the key frame sequence in step 3) is as follows: 3.1) Given a noisy input sequence ,The goal is to generate a sequence of a person speaking in sync with the given audio; 3.2) To provide identity and background information, the identity frame Connecting the VAE encoder to the noisy input effectively utilizes the skip connections of the U-Net architecture to preserve input details; 3.3) Based on the audio input and identity frame, a diffusion model is used to generate a A sequence of key frames, with an interval of These keyframes effectively capture long-range temporal dependencies and serve as anchor points for subsequent interpolation stages.

3. The audio-driven facial animation generation method according to claim 1, characterized in that: The specific process of step 4) is as follows: 4.1) Get two consecutive frames from the key frame sequence and as a conditional frame; 4.2) Create a sequence to match the start and end frame input shapes ,in Learned embeddings representing missing frames; this sequence is concatenated channel-wise with the noisy input and interpolated via a diffusion model to produce coherent intermediate frames.

4. An audio-driven facial animation generation system, used to implement the audio-driven facial animation generation method according to any one of claims 1 to 3, characterized in that: include: Audio feature extraction unit: Audio feature extraction using combined embeddings from two pre-trained audio encoders; Audio feature feeding unit: feeds the extracted audio features into the audio attention and time step embedding of the diffusion model; Key frame sequence generation unit: Based on the audio input and identity frame, the diffusion model generates a key frame sequence at a low frame rate; Interpolation unit: Based on two consecutive frames in the key frame sequence, the diffusion model interpolates between the key frames; Diffusion model optimization unit: optimizes the diffusion model by combining the loss functions of RGB space and latent feature space.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images