Face-driven video generation method and related device
By using a pre-trained inversion encoder and a 3D generative adversarial network, high 3D consistent face-driven videos are generated, solving the problems of multi-view consistency and detail preservation, and realizing the generation of high-quality face-driven videos in a single forward pass.
Patent Information
- Application Number
- CN202511078040.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-01
- Publication Date
- 2025-11-14
AI Technical Summary
Existing face-driven video generation methods are inadequate in terms of multi-view consistency and detail preservation, and require additional training or optimization, making it difficult to generate high-quality 3D consistent videos.
A pre-trained inversion encoder is used to obtain the 3D latent code representation of the target face image. Combined with a pre-trained 3D generative adversarial network, the modulation features of the generated video frames are injected through expression deviation and facial feature injection to achieve 3D consistency and expression control.
It generates highly consistent face-driven videos without additional training, maintaining 3D consistency under large angular transformations and preserving the texture details of the target face image.
Smart Images
Figure CN120953449A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence generation and relates to a method and related apparatus for generating face-driven videos. Background Technology
[0002] The development of deep learning and AI content generation models has made digital human generation possible. A key area of digital human generation is face-driven video generation, also known as facial reconstruction or image animation. This involves creating animations for target faces by combining control information extracted from different modalities such as video and audio to extract facial expressions and poses. In this process, the identity information of the source face image is preserved, driving video frames to provide pose and expression information, constructing dynamic and realistic face-driven videos. On the other hand, the introduction of neural radiation fields has sparked a revolution in 3D reconstruction. With advancements in 3D generative adversarial networks and their inversion techniques, the realism of generated 3D models is increasing, opening new avenues for generating more expressive face-driven videos.
[0003] Current face-driven video generation methods primarily utilize variational autoencoders and generative adversarial networks. These methods typically rely on facial representations, boundaries, and optical flow mappings from the target face image to the driving image to identify facial expressions and motions in the latent space. However, due to the lack of 3D information, these methods often cannot achieve multi-view... Figure 1 Consistent generation results. Some face-driven video generation methods explore creating face-driven videos with 3D consistency using pre-trained 3D generative adversarial network (GAN) models of faces. However, these methods either require additional training and optimization of the target image to obtain the corresponding latent code, or are limited to generating face-driven videos from random latent codes, significantly reducing the generality and ease of use of the methods. Furthermore, unlike 2D optical flow-based methods, 3D GANs often struggle to fully preserve the detailed information of the target face image. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art, such as inconsistent multiple views, inability to generate face-driven videos from a single specific face image, or the need for additional training and optimization, and to provide a face-driven video generation method and related apparatus.
[0005] To achieve the above objectives, the present invention employs the following technical solution:
[0006] In a first aspect, the present invention provides a method for generating face-driven video, comprising: acquiring a target face image and a driving video; extracting expression coefficients of the target face image and expression coefficients and face pose parameters of each video frame of the driving video, and acquiring a 3D latent code representation of the target face image based on a pre-trained inversion encoder; obtaining expression deviations of each video frame based on the expression coefficients of the target face image and the expression coefficients of each video frame, and adding them to the 3D latent code representation of the target face image to obtain 3D latent code representations of each video frame; decoding the 3D latent code representations of each video frame using a pre-trained 3D generative adversarial network to obtain initial 3D features of each video frame, and acquiring face features of the target face image and injecting them into the initial 3D features of each video frame to obtain modulated 3D features of each video frame; obtaining camera parameters of each video frame based on the face pose parameters of each video frame, and generating a target face video by combining the modulated 3D features of each video frame.
[0007] Optionally, the extraction of the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame of the driving video includes: using a three-dimensional deformable face model to extract the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame of the driving video.
[0008] Optionally, the step of obtaining the expression deviation of each video frame based on the expression coefficients of the target face image and the expression coefficients of each video frame, and adding it to the 3D latent code representation of the target face image, includes obtaining the 3D latent code representation of each video frame by using a pre-trained expression animator based on the expression coefficients of the target face image and the expression coefficients of each video frame; subtracting the expression latent code of each video frame from the expression latent code of the target face image to obtain the expression deviation of each video frame, and superimposing it on the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame; wherein, the expression animator includes an expression codebook and a multilayer perceptron, and the multilayer perceptron is used to map the expression coefficients to weight vectors on the orthogonal basis of the expression codebook to obtain the expression latent code.
[0009] Optionally, the pre-trained 3D generative adversarial network adopts a pre-trained geometry-aware 3D generative adversarial network.
[0010] Optionally, obtaining the facial features of the target face image includes: using a pre-trained convolutional neural network model to obtain the facial features of the target face image.
[0011] Optionally, the step of acquiring the facial features of the target face image and injecting them into the initial 3D features of each video frame includes: mapping the facial features of the target face image into key vectors and value vectors through convolutional layers; mapping the initial 3D features of each video frame into query vectors of each video frame through convolutional layers; wherein the query vectors and key vectors have the same dimension; obtaining the attention weights of each video frame by Softmax normalization based on the dot product similarity between the query vectors and key vectors of each video frame, and applying the attention weights of each video frame to the value vectors to generate alignment features of each video frame; and modulating the initial 3D features using adaptive instance normalization based on the alignment features of each video frame to obtain modulated 3D features of each video frame.
[0012] Optionally, the step of modulating the initial 3D features using adaptive instance normalization based on the alignment features of each video frame includes: modulating the initial 3D features using the following formula:
[0013]
[0014] Among them, F t To modulate 3D features; σ(F d ) is F d Standard deviation; F d For alignment features; F c For initial 3D features; σ(F c ) is F c Standard deviation; μ(F c ) is F c The mean; μ(F d ) is F d The mean.
[0015] In a second aspect, the present invention provides a face-driven video generation system, comprising: a data acquisition module for acquiring a target face image and a driving video; a preprocessing module for extracting expression coefficients of the target face image and expression coefficients and face pose parameters of each video frame of the driving video, and acquiring a 3D latent code representation of the target face image based on a pre-trained inversion encoder; an expression deviation module for obtaining expression deviations of each video frame based on the expression coefficients of the target face image and the expression coefficients of each video frame, and adding them to the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame; a feature modulation module for decoding the 3D latent code representation of each video frame using a pre-trained 3D generative adversarial network to obtain initial 3D features of each video frame, and acquiring face features of the target face image and injecting them into the initial 3D features of each video frame to obtain modulated 3D features of each video frame; and a video generation module for obtaining camera parameters of each video frame based on the face pose parameters of each video frame, and generating a target face-driven video by combining the modulated 3D features of each video frame.
[0016] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described face-driven video generation method.
[0017] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described face-driven video generation method.
[0018] Compared with the prior art, the present invention has the following beneficial effects:
[0019] This invention discloses a face-driven video generation method. It utilizes a pre-trained inversion encoder to obtain the 3D latent code representation of a target face image and generates a face-driven video for any 2D target face image in a single forward pass. Simultaneously, it combines a pre-trained 3D face generative adversarial network to generate the 3D latent code representation of any single target face image. This 3D latent code representation can be transformed into features suitable for 3D consistent rendering, enabling the method to maintain 3D consistency under large-angle transformations. Furthermore, this process requires no additional training or optimization. Simultaneously, based on the obtained facial expression deviations of each video frame, it is added to the 3D latent code representation of the target face image, serving as an effective facial animation control strategy to achieve precise facial expression control. Furthermore, by acquiring the facial features of the target face image and injecting them into the initial 3D features of each video frame, modulated 3D features of each video frame are obtained to preserve more texture details in the target face image. Attached Figure Description
[0020] Figure 1 This is a flowchart of a face-driven video generation method according to an embodiment of the present invention.
[0021] Figure 2 This is a schematic diagram illustrating the principle of the face-driven video generation method according to an embodiment of the present invention.
[0022] Figure 3 This is a structural block diagram of a face-driven video generation system according to an embodiment of the present invention. Detailed Implementation
[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0025] The present invention will now be described in further detail with reference to the accompanying drawings:
[0026] See Figure 1 In one embodiment of the present invention, a face-driven video generation method is provided, specifically a face-driven video generation method based on an inversion encoder, a 3D generative adversarial network, and 3D feature modulation, which can more reliably generate 3D consistent face-driven videos from any single target face image.
[0027] Specifically, the face-driven video generation method of the present invention includes the following steps:
[0028] S1: Acquire the target face image and drive video.
[0029] S2: Extract the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame of the driving video, and obtain the 3D latent code representation of the target face image based on the pre-trained inversion encoder.
[0030] S3: Based on the expression coefficients of the target face image and the expression coefficients of each video frame, the expression deviation of each video frame is obtained and added to the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame.
[0031] S4: Use a pre-trained 3D generative adversarial network to decode the 3D latent code representation of each video frame to obtain the initial 3D features of each video frame, and obtain the facial features of the target face image and inject them into the initial 3D features of each video frame to obtain the modulated 3D features of each video frame.
[0032] S5: Obtain the camera parameters of each video frame based on the face pose parameters of each video frame, and generate the target face video by combining the modulation 3D features of each video frame.
[0033] This invention discloses a face-driven video generation method. It utilizes a pre-trained inversion encoder to obtain the 3D latent code representation of a target face image and generates a face-driven video for any 2D target face image in a single forward pass. Simultaneously, it combines a pre-trained 3D face generative adversarial network to generate the 3D latent code representation of any single target face image. This 3D latent code representation can be transformed into features suitable for 3D consistent rendering, enabling the method to maintain 3D consistency under large-angle transformations. Furthermore, this process requires no additional training or optimization. Simultaneously, based on the obtained facial expression deviations of each video frame, it is added to the 3D latent code representation of the target face image, serving as an effective facial animation control strategy to achieve precise facial expression control. Furthermore, by acquiring the facial features of the target face image and injecting them into the initial 3D features of each video frame, modulated 3D features of each video frame are obtained to preserve more texture details in the target face image.
[0034] In one possible implementation, the extraction of the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame of the driving video includes: using a three-dimensional deformable face model to extract the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame of the driving video.
[0035] Interpretive, 3D deformable face (3DMM) model is a statistically based 3D face modeling method. By constructing a basis vector space containing a large amount of face data, any 3D face is represented as a linear combination of basis vectors, realizing parametric modeling and dynamic adjustment of face shape and texture.
[0036] For example, by inputting each frame of the target face image and the driving video into the fitting framework of a pre-trained 3DMM model, the global shape basis and local expression basis of the 3DMM model are used to parametrically decompose the face geometry, thereby accurately extracting the expression coefficients corresponding to static expressions in the target image. At the same time, the real-time expression coefficients and six-degree-of-freedom face pose parameters of the dynamic face in the driving video are analyzed frame by frame, ultimately achieving cross-modal expression feature decoupling and motion information extraction from two-dimensional image to three-dimensional parameter space.
[0037] In one possible implementation, the step of obtaining the expression deviation of each video frame based on the expression coefficients of the target face image and the expression coefficients of each video frame, and adding it to the 3D latent code representation of the target face image, includes: obtaining the expression latent code of the target face image and the expression latent code of each video frame through a pre-trained expression animator based on the expression coefficients of the target face image and the expression coefficients of each video frame; subtracting the expression latent code of each video frame from the expression latent code of the target face image to obtain the expression deviation of each video frame, and superimposing it on the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame; wherein, the expression animator includes an expression codebook and a multilayer perceptron, and the multilayer perceptron is used to map the expression coefficients to weight vectors on the orthogonal basis of the expression codebook to obtain the expression latent code.
[0038] Interpretive analysis reveals that the latent code representation extracted by the inversion encoder contains all the information of the target face image, including identity and facial expression information. A preliminary exploration of facial expression information is conducted here: First, a target face image S and two driving images are selected, where the target face image S and driving image C have the same standard facial expression, and driving images C and D are two video frames from the same driving video. The latent code representation w is obtained by performing latent code prediction on images S, C, and D respectively using the inversion encoder. s w c and w d At this time w s It contains the target face image's identity information and standard facial expression information, w c These are the identity information and standard facial expression information that drive the image, w d This represents the identity information and facial expression information driving the image. The following operations are performed on the latent code representation: w ani =w s +(w d -w c Intuitively speaking, under ideal conditions, w d -w c It can completely filter out the identity information of the driving image and express the expression deviation vector between the driving expression and the standard expression. s Adding this expression deviation vector represents shifting the latent code from the target face image identity + standard expression state to the target face image identity + driving expression state. Therefore, ideally, w ani It can fully represent the latent code representation of the objective of this invention and meets the requirements of facial animation.
[0039] Based on the above ideas, this embodiment proposes an expression animator to control the latent code representation of the target face image. Specifically, an additional expression animator is used to learn the expression deviation between the driving expression and the standard expression. This expression deviation removes the influence of other factors and retains only the expression as a control condition. The expression animator consists of an expression codebook and a multilayer perceptron (MLP). The MLP maps expression coefficients to weight vectors on an orthogonal basis of the expression codebook. It takes the expression coefficients as input and maps them into the latent code space. Since the expression coefficients do not contain identity information, the latent code here only represents the expression information of the input image. Similar to the preliminary attempt described above, the expression animator maps the expression coefficients of the target face image and the driving image into the latent code space and obtains the expression latent code. The expression latent code of the driving image is subtracted from the expression latent code of the target face image to obtain the expression deviation from the driving image to the target face image. This expression deviation is applied to the latent code representation of the target face image to obtain the 3D latent code representation of each video frame.
[0040] In one possible implementation, the pre-trained 3D generative adversarial network employs a pre-trained geometry-aware 3D generative adversarial network (EG3D).
[0041] Explained, in this embodiment, an inversion encoder is used to predict the latent code representation of the target face image, and then EG3D is used to reconstruct the target face image from the latent code representation as the initial step of inversion.
[0042] EG3D is a widely used 3D face generative adversarial network that combines the advantages of 2D generative adversarial networks and Neural Radiation Fields (NeRF), enabling the generation of face images with low computational cost. Similar to well-known generative adversarial networks, EG3D leverages the powerful generative capabilities of StyleGAN, accepting latent code representations as input to generate triplanar features of the image. These triplanar features are planar feature maps aligned to three axes. EG3D then decodes the color and transparency of sampled points from the three orthogonal planar features and uses volume rendering to render the output. This ensures 3D consistency of the generated image in the new view. The generative adversarial network inversion method can predict the latent code representation of any given target face image in a 3D face generative adversarial network. This embodiment employs an encoder-based inversion method (GOAE) to avoid training and optimization during the inference phase. Specifically, this embodiment uses the encoder structure of GOAE as the inversion encoder of this invention, which predicts the latent code representation and triplanar features of any given target face image.
[0043] In one possible implementation, obtaining the facial features of the target face image includes: using a pre-trained convolutional neural network (CNN) model to obtain the facial features of the target face image.
[0044] Optionally, the step of acquiring the facial features of the target face image and injecting them into the initial 3D features of each video frame includes: mapping the facial features of the target face image into key vectors and value vectors through convolutional layers; mapping the initial 3D features of each video frame into query vectors of each video frame through convolutional layers; wherein the query vectors and key vectors have the same dimension; obtaining the attention weights of each video frame by Softmax normalization based on the dot product similarity between the query vectors and key vectors of each video frame, and applying the attention weights of each video frame to the value vectors to generate alignment features of each video frame; and modulating the initial 3D features using adaptive instance normalization based on the alignment features of each video frame to obtain modulated 3D features of each video frame.
[0045] Interpretively, to fully preserve and fuse target face image information, it is desirable to inject more information from the target face image into the initial 3D features generated in EG3D. The generated image is aligned with the driving image, but due to the risk of identity leakage, feature adjustment cannot be performed using the driving image, whether for self-reconstruction or cross-identity reconstruction. Furthermore, there is a mismatch between the target face image and the generated image, which can be addressed using cross-attention and Adaptive Instance Normalization (ADAIN).
[0046] Specifically, the target face image is first processed by a CNN, which consists of several Resblocks to match the size of the initial feature map (initial 3D features) to obtain the face features of the target face image. In this embodiment, a 3D feature refiner is designed. The 3D feature refiner maps the face features of the target face image onto key vectors and value vectors through two independent convolutional layers. Then, the initial feature map is processed to obtain the query vector, and the face features of the target face image and the initial feature map are aligned based on cross-attention. The aligned features are then used to adjust the initial feature map to obtain the modulated 3D features of each video frame.
[0047] In one possible implementation, modulating the initial 3D features using adaptive instance normalization based on the alignment features of each video frame includes modulating the initial 3D features using the following formula:
[0048]
[0049] Among them, F t To modulate 3D features; σ(F d ) is F d Standard deviation; F d For alignment features; F c For initial 3D features; σ(F c ) is F c Standard deviation; μ(F c ) is Fc The mean; μ(F d ) is F d The mean.
[0050] This invention discloses a face-driven video generation method. First, a target face image and a driving video are acquired. A neural network is used to extract expression coefficients from the target face image and expression coefficients and facial pose parameters from each frame of the driving video. A 3D latent code representation of the target face is obtained based on a pre-trained inversion encoder. The expression deviation between the target face and the video frames is obtained based on the expression coefficients and added to the latent code representation of the target face to obtain the latent code representation of each video frame. A 3D generative adversarial network is used to decode the latent code representation, and the facial features of the target face are injected into the 3D features to obtain modulated 3D features. The camera parameters of the video frames are obtained based on the facial pose parameters, and the 3D features are adjusted frame by frame to render the target face video.
[0051] See Figure 2 The basic principle of the face-driven video generation method of this invention can be understood as follows: The target face image is mapped to a canonical latent space that can be reconstructed using a 3D generative adversarial network through an inversion encoder. The expression parameters of the target face image and the driving image are mapped to a latent code space representing expression deviations through an expression animator. Based on a 3D feature refiner, cross-attention is used to refine the target face image and 3D features, injecting more information from the target face image into the 3D features. Finally, the renderer uses the face pose parameters of the driving image to render the final output, achieving decoupled control of expression and pose. Using the above method, highly consistent 3D face-driven videos can be generated for any given 2D target face image.
[0052] In summary, this invention proposes a single-sample face-driven video generation framework. By utilizing a pre-trained inversion encoder to obtain the latent code corresponding to the target face image, and leveraging the encoder-based inversion method, the method can extract the latent code in a single forward pass, enabling face-driven generation of any 2D face portrait in a single forward pass. This process requires no additional training or optimization. Furthermore, this invention proposes an effective facial expression animation control strategy to achieve precise facial expression control. It introduces a self-supervised facial expression driving method, using the latent code deviation between the target face image and the driving image as a supervision signal to guide the facial expression animator to learn expression deviations from 3DMM expression coefficients. In addition, this invention proposes a 3D feature refiner to preserve more texture details in the target face image. Through the powerful 3D feature generation capabilities of 3D generative adversarial networks, the proposed method achieves higher 3D consistency and easily generates face portraits with different poses or novel perspectives. In application, this invention only requires the target face image as input, and expression coefficients and camera parameters as control conditions to generate face-driven video in a single forward pass.
[0053] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0054] See Figure 3 In another embodiment of the present invention, a face-driven video generation system is provided, which can be used to implement the above-mentioned face-driven video generation method. Specifically, the face-driven video generation system includes a data acquisition module, a preprocessing module, an expression deviation module, a feature modulation module, and a video generation module.
[0055] The system comprises the following modules: a data acquisition module for acquiring target face images and driving videos; a preprocessing module for extracting expression coefficients from the target face images and expression coefficients and facial pose parameters from each video frame of the driving video, and obtaining 3D latent code representations of the target face images based on a pre-trained inversion encoder; an expression deviation module for obtaining expression deviations for each video frame based on the expression coefficients of the target face images and the expression coefficients of each video frame, and adding these deviations to the 3D latent code representations of the target face images to obtain the 3D latent code representations of each video frame; a feature modulation module for decoding the 3D latent code representations of each video frame using a pre-trained 3D generative adversarial network to obtain the initial 3D features of each video frame, and for acquiring facial features from the target face images and injecting them into the initial 3D features of each video frame to obtain the modulated 3D features of each video frame; and a video generation module for obtaining camera parameters for each video frame based on the facial pose parameters of each video frame, and combining the modulated 3D features of each video frame to generate the target face driving video.
[0056] All relevant content of each step involved in the aforementioned embodiments of the face-driven video generation method can be referenced from the functional description of the corresponding functional module of the face-driven video generation system in the embodiments of the present invention, and will not be repeated here.
[0057] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0058] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a face-driven video generation method.
[0059] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the face-driven video generation method in the above embodiments.
[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.
[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for generating face-driven video, characterized in that, include: Acquire the target face image and driving video; The expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame driving the video are extracted, and the 3D latent code representation of the target face image is obtained based on the pre-trained inversion encoder. Based on the expression coefficients of the target face image and the expression coefficients of each video frame, the expression deviation of each video frame is obtained and added to the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame. The 3D latent code representation of each video frame is decoded using a pre-trained 3D generative adversarial network to obtain the initial 3D features of each video frame, and the facial features of the target face image are obtained and injected into the initial 3D features of each video frame to obtain the modulated 3D features of each video frame. The camera parameters for each video frame are obtained based on the facial pose parameters of each video frame, and the target face video is generated by combining the modulation 3D features of each video frame.
2. The face-driven video generation method according to claim 1, characterized in that, The extraction of expression coefficients from the target face image and expression coefficients and face pose parameters from each video frame of the driving video includes: using a three-dimensional deformable face model to extract expression coefficients from the target face image and expression coefficients and face pose parameters from each video frame of the driving video.
3. The face-driven video generation method according to claim 1, characterized in that, The step of obtaining the expression deviation of each video frame based on the expression coefficient of the target face image and the expression coefficient of each video frame, and adding it to the 3D latent code representation of the target face image, includes obtaining the 3D latent code representation of each video frame as follows: Based on the expression coefficients of the target face image and the expression coefficients of each video frame, the expression latent code of the target face image and the expression latent code of each video frame are obtained through a pre-trained expression animator. The expression latent code of each video frame is subtracted from the expression latent code of the target face image to obtain the expression deviation of each video frame, which is then superimposed on the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame. The facial expression animator includes an expression codebook and a multilayer perceptron. The multilayer perceptron is used to map the expression coefficients to weight vectors on the orthogonal basis of the expression codebook to obtain the expression latent code.
4. The face-driven video generation method according to claim 1, characterized in that, The pre-trained 3D generative adversarial network employs a pre-trained geometry-aware 3D generative adversarial network.
5. The face-driven video generation method according to claim 1, characterized in that, The facial features obtained from the target face image include: A pre-trained convolutional neural network model is used to obtain facial features of the target face image.
6. The face-driven video generation method according to claim 1, characterized in that, The process of acquiring facial features from the target face image and injecting them into the initial 3D features of each video frame includes: The facial features of the target face image are mapped into key vectors and value vectors through convolutional layers; the initial 3D features of each video frame are mapped into query vectors of each video frame through convolutional layers; the query vectors and key vectors have the same dimension; the attention weights of each video frame are obtained by Softmax normalization based on the dot product similarity between the query vectors and key vectors of each video frame, and the attention weights of each video frame are applied to the value vectors to generate alignment features of each video frame; and the initial 3D features are modulated using adaptive instance normalization based on the alignment features of each video frame to obtain modulated 3D features of each video frame.
7. The face-driven video generation method according to claim 1, characterized in that, The step of modulating the initial 3D features using adaptive instance normalization based on the alignment features of each video frame includes: The initial 3D features are modulated using the following formula: Among them, F t To modulate 3D features; σ(F d ) is F d Standard deviation; F d For alignment features; F c For initial 3D features; σ(F c ) is F c Standard deviation; μ(F c ) is F c The mean; μ(F d ) is F d The mean.
8. A face-driven video generation system, characterized in that, include: The data acquisition module is used to acquire the target face image and driving video; The preprocessing module is used to extract the expression coefficients of the target face image and the expression coefficients and face pose parameters of each video frame driving the video, and to obtain the 3D latent code representation of the target face image based on the pre-trained inversion encoder. The expression deviation module is used to obtain the expression deviation of each video frame based on the expression coefficient of the target face image and the expression coefficient of each video frame, and add it to the 3D latent code representation of the target face image to obtain the 3D latent code representation of each video frame. The feature modulation module is used to decode the 3D latent code representation of each video frame using a pre-trained 3D generative adversarial network to obtain the initial 3D features of each video frame, and to obtain the facial features of the target face image and inject them into the initial 3D features of each video frame to obtain the modulated 3D features of each video frame. The video generation module is used to obtain the camera parameters of each video frame based on the face pose parameters of each video frame, and combine the modulation 3D features of each video frame to generate a target face-driven video.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the face-driven video generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the face-driven video generation method as described in any one of claims 1 to 7.