A speech-driven face reconstruction method with editable expression based on 3D Gaussian sputtering technology

Abstract

The present invention discloses a method for face reconstruction with editable expression and voice-driven based on three-dimensional Gaussian splashing technology, which belongs to the field of data processing technology; the method comprises: inputting video material to form a video data set; performing three-dimensional reconstruction of the face of the video frame of the video material; constructing a mapping of audio features to face parameters of a three-dimensional face model; constructing a dynamic explicit radiation field; accepting audio and expression feature inputs, regressing the expression parameters and jaw posture coefficients of the re-enacted face through the vector quantized variational encoder, and re-enacting the dynamic attributes of the Gaussian basis element constructed by combining the dynamic explicit radiation field constructed by other FLAME parameter inputs of the source face to synthesize the re-enacted video; editing the appearance and expression of the target person. The present invention greatly accelerates the training and rendering efficiency through parameterized learnable hash coding structure and explicit three-dimensional Gaussian splashing, and can significantly shorten the calculation time while ensuring high-quality rendering, thereby improving the feasibility of the model in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for reconstructing human faces with editable expressions and voice-driven operation based on three-dimensional Gaussian sputtering technology. Background Art

[0002] 3D Gaussian Splatting (3D-GS) is a major advancement in computer graphics that provides an explicit scene representation and novel view synthesis method without relying on neural networks such as NeRF. The core of 3D-GS technology is its ability to infer the 3D shape of an object or scene from a set of input images, which is achieved through a Gaussian point cloud method that covers multiple aspects such as input type, model structure, output representation, and training strategy.

[0003] Synthesizing audio-driven talking head videos is of great significance in various digital applications such as virtual reality, filmmaking, and human-computer interaction. By converting speech input into realistic facial movements and expressions, the user's immersive experience and interactive effect are enhanced. Early methods used generative models driven by machine learning, such as GANs and StyleGAN, to improve the accuracy of facial texture and geometry modeling. In recent years, with the vigorous development of neural representations, many methods have adopted neural radiance fields to improve the stability of 3D head structures and provide high-fidelity real-time rendering.

[0004] Early methods attempted to synthesize lip movements to fit training data of still facial images. The "Synthesizing Obama" method proposed by Suwajanakorn et al. uses a recurrent neural network (RNN) to learn the correspondence between audio and lip shapes, and synthesizes mouth textures through a principal component analysis (PCA) model, thereby achieving speech-driven face reenactment.

[0005] AD-NeRF is an end-to-end audio-to-talking-head-video neural radiance field renderer developed using neural radiance field technology. The renderer directly generates face images conditioned on audio features.

[0006] Based on AD-NeRF, a discrete learnable grid structure is introduced to greatly accelerate the training and inference process. This improvement significantly improves the training efficiency based on neural radiance field technology. In addition, this method optimizes the performance of the model when processing complex scenes, making the generated results more realistic and efficient. Through this innovative grid structure, Tang et al.'s method significantly shortens the calculation time while maintaining high-quality rendering.

[0007] The ER-NeRF method further accelerates training and rendering efficiency by converting discrete learnable grids into parameterized learnable hash coding structures. In addition, the method introduces a regional self-attention algorithm so that audio and movements such as blinking can adaptively affect various regions of the human head.

[0008] The GeneFace method learns the mapping from audio to action on a large-scale dataset, significantly improving the generalization ability of the model. This method uses landmarks as intermediate representations of actions, reducing the complexity and difficulty of cross-modal learning, making it possible to generate natural facial movements from audio features. In addition, GeneFace can better capture facial details and dynamic changes through this intermediate representation, thereby improving the realism and consistency of the final generated facial animation.

[0009] The existing techniques are generally divided into methods based on 2D generative models and 3D neural radiation fields.

[0010] For 2D generative models, they lack explicit 3D structure, and these 2D-based methods perform poorly in maintaining naturalness and consistency when head pose changes. Although methods such as "Synthesizing Obama" try to improve by learning additional audio-to-3D structure mappings, this mapping suffers from errors and information loss. Secondly, this mapping cannot map the audio to the motion distribution of a specific person, resulting in average synthesized faces.

[0011] Neural radiance fields were introduced as a three-dimensional representation of talking heads, providing realistic rendering effects. However, although these methods introduced many acceleration modules, there are still challenges in training efficiency and rendering time. In addition, most of these methods represent facial actions by changing the appearance of each sampled point, which puts the burden of learning jumps and non-smooth appearance changes on the network, resulting in distortion of facial features. Therefore, the generalization ability based on neural radiance fields is limited. In order to solve the problem of generalization ability of neural radiance fields, the GeneFace method proposes to use landmarks as intermediate representations of actions, reducing the complexity and difficulty of cross-modal learning. However, using spatial points as intermediate deformation conditions is not applicable to neural radiance fields with obvious three-dimensional structures, and there is a serious problem of information loss.

[0012] Existing speech-driven face reenactment methods usually directly map audio to video, but are unable to perform personalized editing of the shape and texture of the reenacted face. Summary of the invention

[0013] The purpose of the present invention is to provide a more efficient and ultra-high rendering efficiency expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology. In view of the shortcomings of the prior art in terms of generalization ability and information loss from audio to video, a more reliable intermediate representation of audio to action is established to reduce information loss and enhance the generalization ability of the model. In view of the problem that the prior art lacks stylization in audio-to-video conversion, an additional action mapping network is proposed, which converts the action distribution of audio mapping into the action distribution of a specific face, thereby realizing stylized talking head generation. In view of the problem that the prior art cannot directly edit the expression of the generated talking head, a solution is proposed, which can personalize the shape and expression of the reenacted face.

[0014] In order to solve the above technical problems, the present invention provides a method for face reconstruction with editable expression and voice-driven based on three-dimensional Gaussian sputtering technology, comprising the following steps:

[0015] S1, input video material to form a video data set;

[0016] S2, reconstructing the three-dimensional face of the video frame of the video material, the identity coefficient and facial parameters of the three-dimensional deformable model of the face; the facial parameters include expression coefficients and posture coefficients, and the posture coefficients include eye posture coefficients, jaw posture coefficients and head posture coefficients;

[0017] S3, extracting expression coefficients and jaw posture coefficients of the three-dimensional deformable model of the source face in the video data set, training a vector quantized variational encoder using the aligned audio features and facial parameters, and constructing a mapping from the audio features to the facial parameters of the three-dimensional face model;

[0018] S4, construct a dynamic explicit radiation field, use the face model as a proxy, input facial parameters to dynamically regress the color, opacity, position and covariance attributes of the Gaussian primitives, and control the splitting and aggregation of the Gaussian primitives according to the position gradient, and synthesize the character reenactment video through volume rendering technology;

[0019] S5, receiving audio and expression feature input, regressing expression parameters and jaw posture coefficients of the replayed face through the vector quantized variational encoder, and synthesizing the replayed video by regressing the dynamic attributes of the Gaussian primitives constructed by combining the dynamic explicit radiation field input of the source face and other three-dimensional deformable face model parameters;

[0020] S6. Editing the appearance and expression of the target person is achieved by editing the expression features or directly editing the expression parameters.

[0021] Furthermore, step S2 specifically includes:

[0022] Using FLAME as a 3D deformable face model, a convolutional neural network is used to predict face identity and facial parameters, expressed as:

[0023]

[0024] in, Respectively represent the identity, posture and expression coefficient of the face; after obtaining the facial parameters, the shape of the face is expressed by the linear mixed skinning formula as follows:

[0025]

[0026] in, represents the average shape of the human head model, Represented as orthogonal shape, pose, and expression bases, respectively, Expressed as a linear blend skinning formula, Indicates the standard point after adding expression and posture offset, The joint regressor is defined, Per-vertex skin weights representing smooth blending, Linear combination formulas for representing shapes, poses, and expression blends;

[0027] The principal component analysis is used to smooth the frame-by-frame predicted 3D face parameters, which can be expressed as:

[0028]

[0029] in, is expressed as the length of the smoothing interval, Indicates the interval number.

[0030] Furthermore, step S3 specifically includes:

[0031] S301, extracting audio features for each speech frame corresponding to the video frame in the data set;

[0032] S302, using the aligned audio features and facial parameters to train a vector quantized variational encoder to construct a mapping from audio to expression coefficients;

[0033] S303, stylizing the expression coefficients, and transferring the generated expression coefficients to the distribution of the customized character.

[0034] Furthermore, step S302 specifically includes:

[0035] Construct a vector quantized variational encoder to encode speech features into potential representations, map the potential representations to discrete codes through a vector quantization module, and finally decode them into expression coefficients, expressed as:

[0036]

[0037] Encoding audio features into latent representations;

[0038]

[0039] in, is the embedding vector of all codebooks A collection of; argmin is a function, which represents the value of the function parameter that obtains the minimum value;

[0040] Potentially represent Mapping to the nearest codebook entry ,in is the first embedding vectors;

[0041]

[0042] Will Replace with the most recent codebook entry ;

[0043]

[0044] The quantized representation The decoder D generates the reconstructed output , expressed as expression parameters and jaw posture.

[0045] Furthermore, step S303 specifically includes:

[0046] The reconstructed facial parameters are transferred to the expression distribution of a certain identity, expressed as:

[0047]

[0048] in, Represented as a convolutional neural network.

[0049] Furthermore, step S4 specifically includes:

[0050] S401, defining a face template with a neutral expression, and anchoring Gaussian points in each triangle of the face template;

[0051] S402, transforming the neutral Gaussian into a standard space according to the camera posture;

[0052] S403, splitting and aggregating Gaussian primitives according to the gradient of Gaussian position attributes, and adding Gaussian points to the entire head space;

[0053] S404, constructing a displayed deformation field according to the input facial parameters of the human face, and deforming the neutral Gaussian into a certain shape;

[0054] S405: Generate a replay result through volume rendering.

[0055] Furthermore, step S401 specifically includes:

[0056] Initially, each Gaussian primitive is anchored at the center of each triangle in the 3D face model; each Gaussian primitive learns the properties of Gaussian locally in each triangle, expressed as:

[0057]

[0058] in, They are respectively represented as the position, rotation direction, size, opacity, and color attributes of the Gaussian in the local triangular area.

[0059] Furthermore, step S403 specifically includes:

[0060] In the initialization phase, zero expression parameters are input, and neutral Gaussians learn low-frequency geometric details;

[0061] The Gaussian primitives are effectively split and aggregated according to the returned gradients of the global position attributes, filling the Gaussian primitives into the locations where details are missing.

[0062] Further, step S404 specifically includes:

[0063] The constructed deformation field adopts a coarse-to-fine architecture. In the coarse deformation stage, the Gaussian attribute is regressed to a valid position according to the deformation of Flame;

[0064] In order to capture the dynamic representation of details, a multi-layer perceptron is used to learn a low-dimensional deformation representation for each Gaussian point.

[0065] It is expressed as:

[0066]

[0067] in, Represented as a low-dimensional deformed representation; Represented as a parameterized three-plane hash table, is a multi-layer perceptron;

[0068] Second, the facial parameters are downsampled to a latent feature, represented as:

[0069]

[0070] in, Represented as latent features; Expressed as a downsampling function;

[0071] The additional deformation is finally expressed as:

[0072]

[0073]

[0074] in: represents the additional deformation of the Gaussian primitive in addition to the FLAME deformation, Represents the final state after FLAME deformation plus additional deformation.

[0075] Further, step S405 specifically includes:

[0076] Gaussian element is composed of points The 3D covariance matrix centered on definition:

[0077]

[0078] in, is the 3D covariance matrix ;

[0079] Define a parametric ellipsoid with a scaling matrix S and a rotation matrix R, and construct the covariance matrix as follows:

[0080]

[0081] in, is the covariance matrix;

[0082] The scale and rotation matrices are represented by a learnable scale vector and a learnable quaternion, respectively; finally, the color of a pixel can be determined by a mixture of overlapping Gaussian distributions:

[0083]

[0084] in: is the color of the pixel, is the color of each point represented by the spherical harmonic function, the mixing weight Multiply the opacity of each point by the 2D projection of the 3D Gaussian Calculated;

[0085] The overall loss function of the model is expressed as:

[0086]

[0087] in: Represents the overall loss function of the model; Represents the rendered image and the real RGB image respectively.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] The present invention greatly accelerates the training and rendering efficiency through parameterized learnable hash coding structure and explicit three-dimensional Gaussian splashing, which can significantly shorten the calculation time while ensuring high-quality rendering, thereby improving the practical application feasibility of the model.

[0090] The present invention uses FLAME coefficients as the intermediate representation from audio to motion, avoiding the loss of using geometric structure as the intermediate representation and enhancing the generalization ability of the model. This enables the model to generate more natural and consistent facial expressions under different audio inputs.

[0091] The present invention converts the motion distribution of audio mapping into the motion distribution of a specific face through an additional motion conversion network, thereby achieving stylized generation of a talking head.

[0092] The present invention provides a function of performing personalized editing on the shape and expression of a reproduced human face, thereby enhancing the diversity and controllability of generated videos.

[0093] The present invention represents facial movements by changing the properties of Gaussian primitives, reduces the burden of network learning jumps and rough appearance changes, and avoids distortion of facial features. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0095] Figure 1 It is a schematic diagram of the overall process structure;

[0096] Figure 2 It is a training flow chart from audio to the distribution of expression coefficients of a specific person;

[0097] Figure 3 Schematic diagram of a high-fidelity and editable reconstruction method of a human face based on three-dimensional Gaussian splattering. DETAILED DESCRIPTION

[0098] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0099] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0100] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0101] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0102] The present invention provides a method for reconstructing a face with editable expression and voice-driven expression based on three-dimensional Gaussian sputtering technology. Figure 1 As shown, the following steps are included:

[0103] S1, input video material to form a video data set;

[0104] S2, reconstructing the face in three dimensions from the input video material, extracting the identity coefficient, expression coefficient, and posture coefficients of the eyes, jaw, and head of the face three-dimensional deformable model (FLMAE), and segmenting and cutting out the face area image as the training truth value;

[0105] S3, extracting the expression coefficient and jaw posture of the source face in the video data set, using the aligned audio features and the expression coefficient and jaw posture to train a vector quantized variational encoder, and constructing a mapping from the audio features to the facial parameters of the three-dimensional face model;

[0106] S4. Construct a dynamic explicit radiation field, use the face model as a proxy, input facial parameters (expression coefficients and posture coefficients of eyes, jaw and head) to dynamically regress the color, opacity, position, covariance and other attributes of the Gaussian primitives. Control the splitting and aggregation of the Gaussian primitives according to the position gradient, and synthesize the character reenactment video through volume rendering technology;

[0107] S5, receiving audio and expression feature input, regressing expression parameters and jaw posture of the human face through the vector quantized variational encoder, and reproducing the dynamic attributes of the dynamic explicit radiation field regression Gaussian primitives synthesized by combining other FLAME parameters (three-dimensional deformable model parameters of the human face) of the source face input;

[0108] S6. The appearance and expression of the target person can be edited by editing the expression features or directly editing the expression parameters.

[0109] Furthermore, step S2 specifically includes:

[0110] S201, predicting three-dimensional face parameters frame by frame using a convolutional neural network;

[0111] S202: Smoothing the three-dimensional face parameters predicted in S101 using principal component analysis.

[0112] Furthermore, step S201 specifically includes:

[0113] Using FLAME as a 3D deformable face model, a convolutional neural network is used to predict face identity and facial parameters, expressed as:

[0114]

[0115] in, Respectively represent the identity, posture and expression coefficient of the face; after obtaining the facial parameters, the shape of the face is expressed by the linear mixed skinning formula as follows:

[0116]

[0117] in, represents the average shape of the human head model, Represented as orthogonal shape, pose, and expression bases, respectively, Expressed as a linear blend skinning formula, Indicates the standard point after adding expression and posture offset, The joint regressor is defined, Per-vertex skin weights representing smooth blending, A linear combination formula for representing blended shapes of shape, pose, and expression.

[0118] Furthermore, step S202 specifically includes:

[0119] The principal component analysis is used to smooth the frame-by-frame predicted 3D face parameters, which can be expressed as:

[0120]

[0121] in, is expressed as the length of the smoothing interval, Indicates the interval number.

[0122] Furthermore, if Figure 2 As shown, step S3 specifically includes:

[0123] S301, extracting Hubert features (i.e., audio features) for each speech frame corresponding to the video frame in the data set;

[0124] S302, using the aligned audio features and facial parameters to train a vector quantized variational encoder to construct a mapping from audio to expression coefficients;

[0125] S303, stylizing the expression coefficients predicted in step S302, and transferring the generated expression coefficients to the distribution of the customized character.

[0126] Furthermore, step S302 specifically includes:

[0127] Construct a vector quantized variational encoder to encode speech features into potential representations, map the potential representations to discrete codes through a vector quantization module, and finally decode them into expression coefficients, expressed as:

[0128]

[0129] Encode Hubert features (i.e., audio features) as latent representations;

[0130]

[0131] in, is the embedding vector of all codebooks A collection of; argmin is a function, which represents the value of the function parameter that obtains the minimum value;

[0132] Potentially represent Mapping to the nearest codebook entry ,in is the first embedding vectors;

[0133]

[0134] Will Replace with the most recent codebook entry ;

[0135]

[0136] The quantized representation The decoder D generates the reconstructed output , expressed as expression parameters and jaw posture.

[0137] Furthermore, step S303 specifically includes:

[0138] The reconstructed facial parameters are transferred to the expression distribution of a certain identity, expressed as:

[0139]

[0140] in, Represented as a convolutional neural network.

[0141] Furthermore, if Figure 3 As shown, step S4 specifically includes:

[0142] S401, defining a face template with a neutral expression, and anchoring Gaussian points in each triangle of the face template;

[0143] S402, transforming the neutral Gaussian into a standard space according to the camera posture;

[0144] S403, splitting and aggregating Gaussian primitives according to the gradient of Gaussian position attributes, and adding Gaussian points to the entire head space;

[0145] S404, constructing a displayed deformation field according to the input facial parameters of the human face, and deforming the neutral Gaussian into a certain shape;

[0146] S405: Generate a replay result through volume rendering.

[0147] Furthermore, step S401 specifically includes:

[0148] Initially, each Gaussian primitive is anchored at the center of each triangle in the 3D face model; each Gaussian primitive learns the properties of Gaussian locally in each triangle, expressed as:

[0149]

[0150] in, They are respectively represented as the position, rotation direction, size, opacity, and color attributes of the Gaussian in the local triangular area.

[0151] Furthermore, step S403 specifically includes:

[0152] The motion of the head avatar can be divided into rigid motion related to the head pose and non-rigid transformation related to the facial expression. The head pose, including rotation and translation, can be manipulated by the camera parameters. Based on this observation, we input zero expression parameters in the initialization stage to facilitate the neutral Gaussian to learn low-frequency geometric details. At the same time, the Gaussian primitives are effectively split and aggregated according to the backpropagated gradients of the global position attributes so that the Gaussian primitives can be automatically filled in the positions where the details are missing.

[0153] Further, step S404 specifically includes:

[0154] The constructed deformation field adopts a coarse-to-fine architecture. In the coarse deformation stage, the Gaussian attributes are regressed to a valid position according to the deformation of Flame.

[0155] In order to capture the dynamic representation of details, a multi-layer perceptron is used to learn a low-dimensional deformation representation for each Gaussian point, which is expressed as:

[0156]

[0157] in, Represented as a low-dimensional deformed representation; Represented as a parameterized three-plane hash table, is a multi-layer perceptron;

[0158] Second, the facial parameters are downsampled to a latent feature, represented as:

[0159]

[0160] in, Represented as latent features; Expressed as a downsampling function;

[0161] The additional deformation is finally expressed as:

[0162]

[0163]

[0164] in: represents the additional deformation of the Gaussian primitive in addition to the FLAME deformation, Represents the final state after FLAME deformation plus additional deformation.

[0165] Further, step S405 specifically includes:

[0166] Gaussian element is composed of points The 3D covariance matrix centered on definition:

[0167]

[0168] in, is the 3D covariance matrix ;

[0169] Define a parametric ellipsoid with a scaling matrix S and a rotation matrix R, and construct the covariance matrix as follows:

[0170]

[0171] in, is the covariance matrix;

[0172] The scale and rotation matrices are represented by a learnable scale vector and a learnable quaternion, respectively; finally, the color of a pixel can be determined by a mixture of overlapping Gaussian distributions:

[0173]

[0174] in: is the color of the pixel, is the color of each point represented by the spherical harmonic function, the mixing weight Multiply the opacity of each point by the 2D projection of the 3D Gaussian Calculated;

[0175] Furthermore, the overall loss function of the model is expressed as:

[0176]

[0177] in: Represents the overall loss function of the model; Represents the rendered image and the real RGB image respectively.

[0178] The present invention realizes the mapping of audio to action:

[0179] The present invention uses FLAME coefficients as the intermediate representation of audio to action, and uses VQ-VAE to establish an accurate mapping of audio to action. Through an additional action mapping network, the action distribution of the audio mapping is converted into the action distribution of a specific face, achieving the stylized generation of the talking head.

[0180] The present invention realizes that expressions can be edited:

[0181] The present invention adopts FLAME coefficient as the driving condition of Gaussian primitive deformation, and realizes the synthesis of three-dimensional speaking face with editable expression.

[0182] The present invention realizes the deformation architecture:

[0183] The present invention adopts a coarse-to-fine deformation architecture, including a rough deformation of the FLAME grid and a refined deformation predicted by a parameterized three-plane architecture. This architecture reduces the burden of network learning jumps and non-smooth appearance changes, and avoids distortion of facial features.

[0184] Through the above patent protection points, the present invention proposes innovative solutions in audio-to-action mapping, expression editability and deformation architecture, providing a reliable method for the generation and stylization of talking heads.

[0185] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0186] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0187] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for face reconstruction with editable expression and voice-driven operation based on three-dimensional Gaussian sputtering technology, characterized in that: The following steps are involved: S1, input video material to form a video data set; S2, reconstructing the three-dimensional face of the video frame of the video material, the identity coefficient and facial parameters of the three-dimensional deformable model of the face; the facial parameters include expression coefficients and posture coefficients, and the posture coefficients include eye posture coefficients, jaw posture coefficients and head posture coefficients; S3, extracting expression coefficients and jaw posture coefficients of the three-dimensional deformable model of the source face in the video data set, training a vector quantized variational encoder using the aligned audio features and facial parameters, and constructing a mapping from the audio features to the facial parameters of the three-dimensional face model; S4, construct a dynamic explicit radiation field, use the face model as a proxy, input facial parameters to dynamically regress the color, opacity, position and covariance attributes of the Gaussian primitives, and control the splitting and aggregation of the Gaussian primitives according to the position gradient, and synthesize the character reenactment video through volume rendering technology; S5, receiving audio and expression feature input, regressing expression parameters and jaw posture coefficients of the replayed face through the vector quantized variational encoder, and synthesizing the replayed video by regressing the dynamic attributes of the Gaussian primitives constructed by combining the dynamic explicit radiation field input of the source face and other three-dimensional deformable face model parameters; S6. Editing the appearance and expression of the target person by editing the expression features or directly editing the expression parameters; Step S2 specifically includes: Using FLAME as a 3D deformable face model, a convolutional neural network is used to predict face identity and facial parameters, expressed as: ； in, Respectively represent the identity, posture and expression coefficient of the face; after obtaining the facial parameters, the shape of the face is expressed by the linear mixed skinning formula as follows: ； in, represents the average shape of the human head model, Represented as orthogonal shape, pose, and expression bases, respectively, Expressed as a linear blend skinning formula, Indicates the standard point after adding expression and posture offset, The joint regressor is defined, Per-vertex skin weights representing smooth blending, Linear combination formulas for representing shapes, poses, and expression blends; The principal component analysis is used to smooth the frame-by-frame predicted 3D face parameters, which can be expressed as: ； in, is expressed as the length of the smoothing interval, Indicates the interval number.

2. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1 is characterized in that: Step S3 specifically includes: S301, extracting audio features for each speech frame corresponding to the video frame in the data set; S302, using the aligned audio features and facial parameters to train a vector quantized variational encoder to construct a mapping from audio to expression coefficients; S303, stylizing the expression coefficients, and transferring the generated expression coefficients to the distribution of the customized character.

3. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 2 is characterized in that: Step S302 specifically includes: Construct a vector quantized variational encoder to encode speech features into potential representations, map the potential representations to discrete codes through a vector quantization module, and finally decode them into expression coefficients, expressed as: ； Encoding audio features into latent representations; ； in, is the embedding vector of all codebooks A collection of; argmin is a function, which represents the value of the function parameter that obtains the minimum value; Potentially represent Mapping to the nearest codebook entry ,in is the first embedding vectors; ； Will Replace with the most recent codebook entry ; ； The quantized representation The decoder D generates the reconstructed output , expressed as expression parameters and jaw posture.

4. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 3 is characterized in that: Step S303 specifically includes: The reconstructed facial parameters are transferred to the expression distribution of a certain identity, expressed as: ； in, Represented as a convolutional neural network.

5. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that: Step S4 specifically includes: S401, defining a face template with a neutral expression, and anchoring Gaussian points in each triangle of the face template; S402, transforming the neutral Gaussian into a standard space according to the camera posture; S403, splitting and aggregating Gaussian primitives according to the gradient of Gaussian position attributes, and adding Gaussian points to the entire head space; S404, constructing a displayed deformation field according to the input facial parameters of the human face, and deforming the neutral Gaussian into a certain shape; S405: Generate a replay result through volume rendering.

6. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 5, characterized in that: Step S401 specifically includes: Initially, each Gaussian primitive is anchored at the center of each triangle in the 3D face model; each Gaussian primitive learns the properties of Gaussian locally in each triangle, expressed as: ； in, They are respectively represented as the position, rotation direction, size, opacity, and color attributes of the Gaussian in the local triangular area.

7. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 6 is characterized in that: Step S403 specifically includes: In the initialization phase, zero expression parameters are input, and neutral Gaussians learn low-frequency geometric details; The Gaussian primitives are effectively split and aggregated according to the returned gradients of the global position attributes, filling the Gaussian primitives into the locations where details are missing.

8. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 7, characterized in that: Step S404 specifically includes: The constructed deformation field adopts a coarse-to-fine architecture. In the coarse deformation stage, the Gaussian attribute is regressed to a valid position according to the deformation of Flame; In order to capture the dynamic representation of details, a multi-layer perceptron is used to learn a low-dimensional deformation representation for each Gaussian point, which is expressed as: ； in, Represented as a low-dimensional deformed representation; Represented as a parameterized three-plane hash table, is a multi-layer perceptron; Second, the facial parameters are downsampled to a latent feature, represented as: ； in, Represented as latent features; Expressed as a downsampling function; The additional deformation is finally expressed as: ； ； in: represents the additional deformation of the Gaussian primitive in addition to the FLAME deformation, Represents the final state after FLAME deformation plus additional deformation.

9. The expression-editable voice-driven face reconstruction method based on three-dimensional Gaussian sputtering technology according to claim 8, characterized in that: Step S405 specifically includes: Gaussian element is composed of points The 3D covariance matrix centered on definition: ； in, is the 3D covariance matrix ; Define a parametric ellipsoid with a scaling matrix S and a rotation matrix R, and construct the covariance matrix as follows: ； in, is the covariance matrix; The scale and rotation matrices are represented by a learnable scale vector and a learnable quaternion, respectively; finally, the color of a pixel can be determined by a mixture of overlapping Gaussian distributions: ； in: is the color of the pixel, is the color of each point represented by the spherical harmonic function, the mixing weight Multiply the opacity of each point by the 2D projection of the 3D Gaussian Calculated; The overall loss function of the model is expressed as: ； in: Represents the overall loss function of the model; Represents the rendered image and the real RGB image respectively.

Images