Voice-driven facial animation generation method, device, equipment and storage medium

By introducing dynamic face radiation field and differentiable face deformation field based on few sample learning in voice-driven face animation synthesis, combined with the reference image mechanism, the problem of insufficient generalization in the existing technology is solved, and the Generate high-quality voice-driven face animation videos under a small amount of data.

CN114782596BActive Publication Date: 2024-11-22TSINGHUA UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210185835.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-28
Publication Date
2024-11-22
Estimated Expiration
2042-02-28

AI Technical Summary

Technical Problem

The existing voice-driven face animation synthesis methods have shortcomings in generalization, especially when a small amount of data is available, and the performance cannot be effectively generalized to new identity categories.

Method used

Using dynamic face radiation field and differentiable face deformation field based on few-sample learning, the generalization of the model is improved through reference image mechanism, and using image features and audio features to drive dynamic face neural radiation field for end-to-end training to generate Vivid and natural voice-driven face animation video.

Benefits of technology

It realizes rapid generalization to a new identity category on a small number of training samples, generates more realistic and accurate audio driver mouthpieces, and improves the generalization ability and image quality of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782596B_ABST
    Figure CN114782596B_ABST
Patent Text Reader

Abstract

The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for generating voice-driven facial animation, wherein the method comprises: extracting the corresponding image features under any query perspective based on any query perspective and multiple reference images; extracting the initial audio features of the audio frame by frame, and performing time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; using image features and audio features to drive the dynamic facial neural radiation field, and obtaining the generated image of the current frame after voxel rendering. Thus, the low generalization problem of the existing voice-driven facial animation synthesis method is solved, and a dynamic facial radiation field based on few-sample learning is proposed to more accurately model dynamic faces, and few-sample learning is achieved through a reference image mechanism to improve the generalization of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for generating voice-driven facial animation. Background Art

[0002] Speech-driven facial animation synthesis uses a spoken audio as a driving signal to control the mouth shape and generate a target face video that matches the given audio. This emerging technology has a wide range of application scenarios, such as movie dubbing, video conferencing, online education, and virtual stand-ins. Although a large number of related research has emerged recently, how to generate natural and realistic speech-driven facial animation videos is still a considerable challenge.

[0003] At present, voice-driven facial animation synthesis methods can be roughly divided into 2D-based methods and 3D-based methods. Among them, 2D-based methods usually rely on Generative Adversarial Networks (GAN). However, due to the lack of modeling of the three-dimensional structure of the head, most of these methods find it difficult to produce vivid and natural talking facial animations. The voice-driven facial animation synthesis genre relies on the 3D Morphable Model (3DMM). Thanks to the 3D modeling of the face, this type of method can generate more vivid talking faces than 2D-based methods. However, since the use of intermediate 3DMM parameters will cause some information loss, the audiovisual consistency of the generated video may be affected.

[0004] In the related technology, speech-driven facial animation synthesis methods based on Neural Radiance Field (NeRF) have made great progress. NeRF uses a deep fully connected network to store the 3D geometry and appearance information of objects in the form of voxels. The NeRF-based method can better capture the 3D structural information of the face. And it directly maps audio features to the neural radiance field for portrait rendering of the speaking face without introducing additional intermediate representations.

[0005] However, this method only encodes the 3D feature representation of a specific person into the network, so it cannot be generalized to new identities. For each new identity, a large amount of data is required to train a specific model, which leads to the performance of such methods being greatly limited in some practical application scenarios where only a small amount of data is available. Summary of the invention

[0006] The present application provides a voice-driven facial animation generation method, apparatus, device and storage medium to solve the low generalization problem of existing voice-driven facial animation synthesis methods. By proposing a dynamic face radiation field based on few-sample learning, dynamic faces can be modeled more accurately, and few-sample learning can be achieved through a reference image mechanism to improve the generalization of the model.

[0007] The first aspect of the present application provides a method for generating a voice-driven facial animation, comprising the following steps:

[0008] Based on any query perspective and multiple reference images, extracting image features corresponding to any query perspective;

[0009] Extracting initial audio features of the audio frame by frame, and performing time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; and

[0010] The image features and the audio features are used to drive the dynamic facial neural radiation field, and after voxel rendering, a generated image of the current frame is obtained.

[0011] Optionally, extracting corresponding image features under any query perspective based on any query perspective and multiple reference images includes:

[0012] Starting from any query perspective, emitting multiple rays to each pixel in the image to be rendered, and sampling a series of 3D sampling points on each ray;

[0013] Mapping any 3D sampling point to a corresponding 2D pixel position on each reference image, and extracting pixel-level features of the multiple reference images;

[0014] The image features are generated based on the fused pixel-level features.

[0015] Optionally, before mapping any one of the 3D sampling points to a corresponding 2D pixel position on each of the reference images, the method further includes:

[0016] The plurality of reference images are transformed into a preset space using a deformation field perceived by a preset audio information.

[0017] Optionally, the using the image features and the audio features to drive the dynamic facial nerve radiation field, and obtaining a generated image of the current frame after voxel rendering, includes:

[0018] For each image, obtaining a rotation vector and a translation vector of the head, and determining an actual position of the head according to the rotation vector and the translation vector;

[0019] Based on the actual position of the head, an equivalent camera viewing angle direction and a series of 3D spatial sampling points corresponding to the rays emitted from each face pixel are obtained;

[0020] Based on the coordinates of the 3D spatial sampling points, the viewing angle direction, the audio features and the image features, a multi-layer perceptron is used to obtain the RGB color and spatial density of the 3D sampling points.

[0021] Optionally, the method of driving a dynamic facial nerve radiation field using the image features and the audio features, and obtaining a generated image of the current frame after voxel rendering, further includes:

[0022] The RGB color and spatial density are integrated, and the integration result is used to synthesize the face image.

[0023] A second aspect of the present application provides a voice-driven facial animation generation device, comprising:

[0024] An extraction module, configured to extract image features corresponding to any query perspective based on any query perspective and a plurality of reference images;

[0025] A first acquisition module, configured to extract initial audio features of the audio frame by frame, and perform time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; and

[0026] The second acquisition module is used to drive the dynamic facial neural radiation field using the image features and the audio features, and to acquire the generated image of the current frame after voxel rendering.

[0027] Optionally, the extraction module is specifically used to:

[0028] Starting from any query perspective, emitting multiple rays to each pixel in the image to be rendered, and sampling a series of 3D sampling points on each ray;

[0029] Mapping any 3D sampling point to a corresponding 2D pixel position on each reference image, and extracting pixel-level features of the multiple reference images;

[0030] The image features are generated based on the fused pixel-level features.

[0031] Optionally, before mapping any one of the 3D sampling points to a corresponding 2D pixel position on each of the reference images, the extraction module is further configured to:

[0032] The plurality of reference images are transformed into a preset space using a deformation field perceived by a preset audio information.

[0033] Optionally, the second acquisition module is specifically used to:

[0034] For each image, obtaining a rotation vector and a translation vector of the head, and determining an actual position of the head according to the rotation vector and the translation vector;

[0035] Based on the actual position of the head, an equivalent camera viewing angle direction and a series of 3D spatial sampling points corresponding to the rays emitted from each face pixel are obtained;

[0036] Based on the coordinates of the 3D spatial sampling points, the viewing angle direction, the audio features and the image features, a multi-layer perceptron is used to obtain the RGB (RGB color mode) color and spatial density of the 3D sampling points.

[0037] Optionally, the second acquisition module is further used to:

[0038] The RGB color and spatial density are integrated, and the integration result is used to synthesize the face image.

[0039] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the voice-driven facial animation generation method as described in the above embodiment.

[0040] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the above-mentioned voice-driven facial animation generation method.

[0041] Therefore, the voice-driven facial animation generation method of the embodiment of the present application has the following advantages:

[0042] (1) Dynamic facial neural radiation field based on 3D perception image features. Thanks to the 3D perception image feature reference mechanism, the facial neural radiation field can be quickly generalized to new identity categories with only a few fine-tuning training iterations.

[0043] (2) Audio-aware differentiable face deformation field. With the help of this face deformation module, the embodiment of the present application can map all reference images into the standard space, more accurately model dynamic faces, and thus generate more realistic and accurate audio-driven mouth shapes;

[0044] (3) A speech-driven facial animation generation framework based on a differentiable facial deformation field and a dynamic facial neural radiation field. This framework can be trained end-to-end and can quickly generalize to new identity categories using only a small number of training samples, generating vivid and natural speech-driven facial animation videos.

[0045] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 A flowchart of a voice-driven facial animation generation method provided according to an embodiment of the present application;

[0048] Figure 2 A flowchart of a method for generating a voice-driven facial animation according to a specific embodiment of the present application;

[0049] Figure 3 Schematic diagram of processing of a differentiable face deformation module according to an embodiment of the present application;

[0050] Figure 4 This is an example diagram of a voice-driven facial animation generation device according to an embodiment of the present application;

[0051] Figure 5 is an exemplary diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0053] The following describes the voice-driven facial animation generation method, device, equipment and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the low generalization problem of the existing voice-driven facial animation synthesis method mentioned in the background technology center, the present application provides a voice-driven facial animation generation method, in which the corresponding image features under any query perspective can be extracted based on any query perspective and multiple reference images, and the initial audio features of the audio can be extracted frame by frame, and the audio features are time-series filtered to obtain audio features that meet the inter-frame smoothing conditions, and the image features and audio features are used to drive the dynamic facial neural radiation field, and after voxel rendering, the generated image of the current frame is obtained. Thus, the low generalization problem of the existing voice-driven facial animation synthesis method is solved, and a dynamic face radiation field based on few-sample learning is proposed to more accurately model the dynamic face, and few-sample learning is realized through the reference image mechanism to improve the generalization of the model.

[0054] Specifically, Figure 1 A flowchart of a voice-driven facial animation generation method provided in an embodiment of the present application.

[0055] like Figure 1 As shown, the voice-driven facial animation generation method includes the following steps:

[0056] In step S101, based on any query perspective and a plurality of reference images, corresponding image features under any query perspective are extracted.

[0057] Optionally, based on any query perspective and multiple reference images, corresponding image features at any query perspective are extracted, including: starting from any query perspective, emitting multiple rays to each pixel in the image to be rendered, and sampling a series of 3D sampling points on each ray; mapping any 3D sampling point to the corresponding 2D pixel position on each reference image, and extracting pixel-level features of multiple reference images; generating image features based on the fused pixel-level features.

[0058] Optionally, before mapping any 3D sampling point to a corresponding 2D pixel position on each reference image, the method further includes: transforming the plurality of reference images into a preset space using a deformation field perceived by a preset audio information.

[0059] Specifically, if Figure 2 As shown, Figure 2 This is a flowchart of a voice-driven facial animation generation method for a specific embodiment of the present application. The entire flowchart can be divided into three parts: image stream, audio stream, and facial neural radiation field. In the image stream, given any query perspective and N reference images, the embodiment of the present application can obtain the corresponding image features under the perspective. For each reference image, the embodiment of the present application uses a convolutional neural network (CNN) to extract pixel-level feature maps. From the query perspective, rays are emitted to each pixel in the image to be rendered, and a series of 3D points are sampled on each ray. For any 3D sampling point, the embodiment of the present application can map it to the corresponding 2D pixel position on each reference image. Taking into account the dynamics of the speaking face, the embodiment of the present application can further design an audio information-aware deformation field to transform all reference images into a standard space to eliminate the influence of facial deformation on the mapping between corresponding points. Afterwards, the pixel-level features corresponding to the N reference images are extracted, and a feature fusion module based on attention is used to fuse these N features into an overall feature.

[0060] In step S102, initial audio features of the audio are extracted frame by frame, and the audio features are time-series filtered to obtain audio features that meet inter-frame smoothing conditions.

[0061] Specifically, if Figure 2 As shown, in the audio stream, the embodiment of the present application can use a DeepSpeech module based on a recurrent neural network (RNN) to extract audio features frame by frame, and then use an audio attention module to perform temporal filtering to obtain smooth audio features between frames. After obtaining the above image features and audio features, the embodiment of the present application can use them as conditions to drive the dynamic facial neural radiation field, and after voxel rendering, the generated image of the frame can be obtained.

[0062] Next, the construction of the dynamic facial neural radiation field, the differentiable face deformation module and the final voxel rendering steps are introduced in detail.

[0063] In step S103, the dynamic facial neural radiation field is driven by using the image features and the audio features, and after voxel rendering, a generated image of the current frame is obtained.

[0064] Optionally, the dynamic facial neural radiation field is driven by image features and audio features, and after voxel rendering, the generated image of the current frame is obtained, including: for each image, the rotation vector and translation vector of the head are obtained, and the actual position of the head is determined according to the rotation vector and translation vector; based on the actual position of the head, an equivalent camera observation angle direction and a series of 3D space sampling points corresponding to the rays emitted from each facial pixel are obtained; based on the coordinates of the 3D space sampling points, the observation angle direction, audio features and image features, a multi-layer perceptron is used to obtain the RGB color and spatial density of the 3D sampling points.

[0065] Optionally, the dynamic facial neural radiation field is driven by using image features and audio features, and after voxel rendering, a generated image of the current frame is obtained, which also includes: integrating RGB color and spatial density, and synthesizing a facial image using the integration result.

[0066] Specifically, the dynamic facial neural radiation field of the embodiment of the present application uses a multilayer perceptron (MLP) as the backbone network. For each image, the embodiment of the present application can obtain the rotation vector R and the translation vector T of the head through face tracking technology to determine the position of the head, thereby obtaining an equivalent camera viewing angle direction and a series of 3D space sampling points corresponding to the rays emitted from each face pixel. The MLP network uses the coordinates of the 3D space sampling points, the direction of the viewing angle, the audio feature A and the reference image feature F as input, and outputs the RGB color and spatial density of the 3D sampling points. Among them, the acquisition of audio features uses the RNN-based DeepSpeech model. In order to enhance the inter-frame consistency of audio features, the embodiment of the present application introduces a time domain filtering module to further smooth the audio feature A. This time domain filter can be expressed as a fusion of adjacent frame audio features based on a self-attention mechanism. With audio features as control conditions, the embodiment of the present application can basically realize audio-driven face animation synthesis. However, since identity information is implicitly encoded into the facial neural radiance field, and no input about identity features is provided during rendering, a large amount of training data is required to optimize a separate facial neural radiance field for each facial identity category. This will result in huge computational costs and require long training video clips.

[0067] In order to eliminate these limitations, the embodiments of the present application further designs a reference image mechanism. Using the reference image as a guide for facial appearance, only a short video of the target face is needed to fine-tune the basic model, so that a fully pre-trained basic model can be quickly generalized to the new target identity category indicated by the reference image. Specifically, using n reference images and their corresponding camera positions as input, the embodiments of the present application use a two-layer convolutional neural network to obtain a pixel-level feature map of each reference image. Afterwards, for a 3D sampling point, the embodiments of the present application use its 3D spatial coordinates and internal and external parameters of the camera to map this 3D point to the corresponding pixel position (u, v) of the reference image through the transformation from the world coordinate system to the image coordinate system. And use this pixel position to index the corresponding reference image feature F.

[0068] For the differentiable face deformation module, the embodiment of the present application can map the query 3D point to the reference image space by simply transforming the world coordinate system to the image coordinate system. This simple transformation is based on the prior assumption that in the neural radiation field NeRF, the intersection of rays emitted from different angles should correspond to the same spatial physical position and the same color. This assumption is completely valid for rigid bodies, but it is not completely valid for highly dynamic objects such as human faces, which will cause deviations in the mapping.

[0069] In order to solve this problem, the embodiment of the present application designs a differentiable face deformation module conditioned on an audio signal to transform all reference images into a standard space to solve the impact of the dynamics of the face on coordinate mapping. Specifically, the embodiment of the present application uses a three-layer MLP network that perceives an audio signal to implement the face deformation module. The network uses 3D space coordinates, the corresponding coordinates (u, v) in the reference image obtained by mapping the 3D coordinates, and audio features A as input, and outputs a coordinate bias Δo = (Δu, Δv). By applying this coordinate bias to the corresponding coordinates (u, v) in the reference image, the corrected mapping coordinates (u+Δu, v+Δv) can be obtained, as shown in Figure 3 In order to control the predicted bias within a reasonable range, the embodiment of the present application also imposes a regularization constraint term Lr on the predicted amount, so that the second norm of the predicted bias is as small as possible.

[0070]

[0071] Where N is the number of reference images, and P is the set of 3D sampling points in the neural rendering field voxel space. Since the indexing operation has no gradient, in order to optimize the face deformation module, the embodiment of the present application can no longer directly index the image features according to the coordinates (u+Δu, v+Δv). To this end, the embodiment of the present application uses a bilinear interpolation strategy instead of a direct indexing operation to obtain the image feature F' at the corresponding position (u+Δu, v+Δv), such as Figure 3 (b) is shown. Under this strategy, the embodiment of the present application can obtain the gradient from the feature F' to the MLP parameters of the face deformation module, and achieve end-to-end optimization of the entire network. Compared with F, the use of the differentiable face deformation module can make the mapping relationship between images more accurate, so that a more accurate image feature F' can be obtained from the reference image.

[0072] Furthermore, voxel rendering integrates the RGB color c and density σ output by the dynamic facial neural radiation field to synthesize the facial image. In this embodiment of the present application, the background, torso, and neck can be used together as a new "background" and these backgrounds can be restored frame by frame from the original video. In this embodiment of the present application, the color of the last point of each ray can be set to the corresponding background color to render a natural background. Here, the embodiment of the present application can follow the settings in the original NeRF. Under the control of the audio signal A and the image feature F', the final RGB color C obtained by each camera ray using voxel rendering technology is:

[0073]

[0074] Where R and T are the rotation vector and translation vector used to determine the head position, respectively. θ and η are the network parameters corresponding to the neural radiation field MLP network and the differentiable face deformation module, respectively. near and z far are the near and far boundaries of the camera ray respectively. T is the integrated transparency along the camera ray, which can be expressed as:

[0075]

[0076] This embodiment of the application follows NeRF to design an MSE loss function L MSE =||CI|| 2 As the main supervisory signal, I is the real color and C is the color generated by the network. Combined with the regularization term L in the previous module r , the overall loss function can be expressed as:

[0077] L=L MSE +λ·L r ;

[0078] Here, the value of the coefficient λ is set to 5e-8.

[0079] It should be noted that, in the basic model training stage, the embodiment of the present application uses face images of different identity categories as training data to perform training from coarse to fine. MSE The facial radiation field is trained under the supervision of , the general modeling of the head structure is mastered, and a general mapping from audio to lip movement is established. Then in the detailed training stage, the embodiment of the present application adds a differentiable face deformation module to the overall network, and increases L r The loss function is related to L MSE Optimize together. After training the above-mentioned basic model, in the practical application stage, for a new identity category with only a very short available video segment, the embodiment of the present application only needs to use its 10-second speech video to fine-tune the trained basic model, and it can be quickly generalized to this new category. The embodiment of the present application emphasizes the importance of this fine-tuning process, because this process can learn personalized pronunciation methods, and the generated image quality can also be greatly improved after a very short number of iterations. After the fine-tuning process is completed, the fine-tuning model can be used for testing to synthesize various speech videos of this identity category.

[0080] According to the voice-driven facial animation generation method proposed in the embodiment of the present application, the corresponding image features under any query perspective can be extracted based on any query perspective and multiple reference images, and the initial audio features of the audio can be extracted frame by frame, and the audio features can be time-series filtered to obtain audio features that meet the inter-frame smoothing conditions, and the dynamic facial neural radiation field can be driven by image features and audio features, and the generated image of the current frame can be obtained after voxel rendering. In this way, the low generalization problem of the existing voice-driven facial animation synthesis method is solved, and a dynamic facial radiation field based on few-sample learning is proposed to more accurately model dynamic faces, and few-sample learning is realized through the reference image mechanism to improve the generalization of the model.

[0081] Next, the voice-driven facial animation generation device proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0082] Figure 4 It is a block diagram of a voice-driven facial animation generation device according to an embodiment of the present application.

[0083] like Figure 4 As shown, the voice-driven facial animation generation device 10 includes: an extraction module 100, a first acquisition module 200 and a second acquisition module 300.

[0084] The extraction module 100 is used to extract the image features corresponding to any query perspective based on any query perspective and multiple reference images;

[0085] The first acquisition module 200 is used to extract initial audio features of the audio frame by frame, and perform time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; and

[0086] The second acquisition module 300 is used to drive the dynamic facial neural radiation field using image features and audio features, and obtain the generated image of the current frame after voxel rendering.

[0087] Optionally, the extraction module 100 is specifically used for:

[0088] Starting from any query viewpoint, multiple rays are emitted to each pixel in the image to be rendered, and a series of 3D sampling points are sampled on each ray;

[0089] Map any 3D sampling point to the corresponding 2D pixel position on each reference image, and extract pixel-level features of multiple reference images;

[0090] Generate image features based on the fused pixel-level features.

[0091] Optionally, before mapping any 3D sampling point to a corresponding 2D pixel position on each reference image, the extraction module 100 is further configured to:

[0092] Using the deformation field perceived by the preset audio information, multiple reference images are transformed into the preset space.

[0093] Optionally, the second acquisition module 300 is specifically used for:

[0094] For each image, obtain the rotation vector and translation vector of the head, and determine the actual position of the head according to the rotation vector and translation vector;

[0095] Based on the actual position of the head, the equivalent camera viewing angle direction and a series of 3D spatial sampling points corresponding to the rays emitted from each face pixel are obtained;

[0096] Based on the coordinates of the 3D spatial sampling points, the viewing angle, the audio features and the image features, a multi-layer perceptron is used to obtain the RGB color and spatial density of the 3D sampling points.

[0097] Optionally, the second acquisition module 300 is further used for:

[0098] The RGB color and spatial density are integrated and the integration result is used to synthesize the face image.

[0099] It should be noted that the aforementioned explanation of the embodiment of the voice-driven facial animation generation method is also applicable to the voice-driven facial animation generation device of this embodiment, and will not be repeated here.

[0100] According to the speech-driven facial animation generation device proposed in the embodiment of the present application, the corresponding image features under any query perspective can be extracted based on any query perspective and multiple reference images, and the initial audio features of the audio can be extracted frame by frame, and the audio features can be temporally filtered to obtain audio features that meet the inter-frame smoothing conditions, and the dynamic facial neural radiation field can be driven by the image features and audio features, and the generated image of the current frame can be obtained after voxel rendering. In this way, the low generalization problem of the existing speech-driven facial animation synthesis method is solved, and a dynamic facial radiation field based on few-sample learning is proposed to more accurately model dynamic faces, and few-sample learning is realized through the reference image mechanism to improve the generalization of the model.

[0101] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0102] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0103] When the processor 502 executes the program, the voice-driven facial animation generation method provided in the above embodiment is implemented.

[0104] Furthermore, the electronic device further comprises:

[0105] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0106] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0107] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0108] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0110] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0111] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned voice-driven facial animation generation method is implemented.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0113] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0116] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0119] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A voice-driven facial animation generation method, characterized in that: The following steps are involved: Based on any query perspective and multiple reference images, extracting image features corresponding to any query perspective; Extracting initial audio features of the audio frame by frame, and performing time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; as well as Using the image features and the audio features to drive a dynamic facial neural radiation field, and after voxel rendering, obtaining a generated image of the current frame; The extracting the corresponding image features under any query perspective based on any query perspective and multiple reference images includes: starting from any query perspective, emitting multiple rays to each pixel in the image to be rendered, and sampling a series of 3D sampling points on each ray; mapping any 3D sampling point to a corresponding 2D pixel position on each reference image, and extracting pixel-level features of the multiple reference images; generating the image features based on the fused pixel-level features; Before mapping any one of the 3D sampling points to the corresponding 2D pixel position on each of the reference images, the method also includes: using a preset deformation field perceived by audio information to transform the multiple reference images into a preset space, extracting pixel-level features corresponding to the N reference images, and using an attention-based feature fusion module to fuse the N features into an overall feature.

2. The method according to claim 1, characterized in that The method of using the image features and the audio features to drive the dynamic facial neural radiation field and obtaining a generated image of the current frame after voxel rendering includes: For each image, obtaining a rotation vector and a translation vector of the head, and determining an actual position of the head according to the rotation vector and the translation vector; Based on the actual position of the head, an equivalent camera viewing angle direction and a series of 3D spatial sampling points corresponding to the rays emitted from each face pixel are obtained; Based on the coordinates of the 3D spatial sampling points, the viewing angle direction, the audio features and the image features, a multi-layer perceptron is used to obtain the RGB color and spatial density of the 3D sampling points.

3. The method according to claim 2, characterized in that The method of using the image features and the audio features to drive the dynamic facial nerve radiation field and obtaining a generated image of the current frame after voxel rendering also includes: The RGB color and spatial density are integrated, and the integration result is used to synthesize the face image.

4. A voice-driven facial animation generation device, characterized in that: include: An extraction module, configured to extract image features corresponding to any query perspective based on any query perspective and a plurality of reference images; A first acquisition module is used to extract initial audio features of the audio frame by frame, and perform time-series filtering on the audio features to obtain audio features that meet inter-frame smoothing conditions; as well as A second acquisition module is used to drive the dynamic facial nerve radiation field using the image features and the audio features, and obtain a generated image of the current frame after voxel rendering; The extraction module is specifically used to: emit multiple rays to each pixel in the image to be rendered from any query perspective, and sample a series of 3D sampling points on each ray; map any 3D sampling point to a corresponding 2D pixel position on each reference image, and extract pixel-level features of the multiple reference images; Generating the image features based on the fused pixel-level features; Before mapping any one of the 3D sampling points to the corresponding 2D pixel position on each reference image, the extraction module is further used to: transform the multiple reference images into a preset space using a deformation field perceived by a preset audio information, extract pixel-level features corresponding to the N reference images, and use an attention-based feature fusion module to fuse the N features into an overall feature.

5. The device according to claim 4, characterized in that The second acquisition module is specifically used for: For each image, obtaining a rotation vector and a translation vector of the head, and determining an actual position of the head according to the rotation vector and the translation vector; Based on the actual position of the head, an equivalent camera viewing angle direction and a series of 3D spatial sampling points corresponding to the rays emitted from each face pixel are obtained; Based on the coordinates of the 3D spatial sampling points, the viewing angle direction, the audio features and the image features, a multi-layer perceptron is used to obtain the RGB color and spatial density of the 3D sampling points.

6. The device according to claim 5, characterized in that The second acquisition module is further used for: The RGB color and spatial density are integrated, and the integration result is used to synthesize the face image.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the voice-driven facial animation generation method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the voice-driven facial animation generation method as described in any one of claims 1 to 3.

Citation Information

Patent Citations

  • Action driving method and device of target object, equipment and storage medium

    CN113554737A

  • Real-time audio-driven face generation method and device and server

    CN113793408A