Monocular video face reconstruction and driving method based on Gaussian splashing and three planes

Abstract

The invention discloses a monocular video face reconstruction and driving method based on Gaussian splashing and three planes, and the method comprises the following steps: S1, obtaining a mesh explicitly expressed by a parameterized model from each frame of a monocular face video through a 3DMM module, and enabling the 3DMM module to drive the position of a 3D Gaussian ball to change preliminarily; s2, 3D Gaussian balls are initialized, the number of the 3D Gaussian balls is the number of the midpoints of the mesh, the positions of the 3D Gaussian balls are the positions of the midpoints of the mesh under the neutral expression, and attributes of the rest 3D Gaussian balls are initialized randomly; s3, optimizing the 3D Gaussian balls to obtain the rough attribute of each ball, encoding the input facial expression picture through an expression encoder, and decoding through a three-plane frame to obtain the detail change of each attribute of the 3D Gaussian balls; s4, ensuring that the result of the expression encoder is only related to the facial expression through a circulation consistency module; and S5, utilizing a plurality of regularization terms to restrain the attribute change of the 3D Gaussian ball obtained by the expression encoder. The method provided by the invention has important application potential in the field of monocular video face reconstruction and driving.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a monocular video face reconstruction and driving method based on Gaussian splashing and three planes. Background Art

[0002] Creating realistic 3D head avatars from monocular video has always been a challenging problem in computer graphics. Rendering lifelike avatars from arbitrary viewpoints can provide rich application scenarios in many fields such as gaming, video conferencing, augmented reality (AR) and virtual reality (VR). For these applications, the key issue is whether it is possible to reconstruct high-quality 3D head avatars and extend them to new poses and expressions.

[0003] Generally, the reconstruction of 3D head portraits is considered a basic task. Recently, neural radiance fields and their variants have achieved remarkable results in static scene reconstruction. Subsequent research has further extended neural radiance fields to the field of digital human modeling. Although these methods perform well in new perspective synthesis, they often lack controllability, which limits their generalization ability to new postures and expressions. Recent progress in 3D Gaussian splashing has surpassed neural radiance fields in rendering quality of new perspective synthesis. 3D Gaussian splashing represents space as 3D Gaussian spheres, each of which contains position, rotation, scale, opacity and spherical harmonic parameters. Alpha blending is used to splash the Gaussian spheres into the final rendered image. Due to the improvement in rendering quality, 3D Gaussian splashing has been rapidly applied to the field of digital humans and virtual avatar generation.

[0004] Currently, most methods rely on 3D deformable models that compactly describe the virtual avatar through a parameterized space that includes identity and facial expression. By optimizing the avatar in a general pose space, it is able to generate new poses and expressions through monocular video tracking. However, we found that the limited accuracy of facial geometry extracted from monocular videos poses a bottleneck for the virtual avatar generation task. In addition, the lack of network input and supervision information may lead to overfitting problems. The lack of constraints on 3D Gaussian point clouds may lead to image artifacts and image segmentation during rendering. Summary of the invention

[0005] One purpose of the present invention is to propose a monocular video face reconstruction and driving method based on Gaussian splashing and three planes. The scheme proposed in the present invention has important application potential in the field of monocular video face reconstruction and driving, and can provide a more accurate and stable solution for research and application in related fields.

[0006] According to an embodiment of the present invention, a monocular video face reconstruction and driving method based on Gaussian splashing and three planes includes the following steps: S1. The 3DMM module obtains the mesh explicitly expressed by the parameterized model from each frame of the monocular face video. The 3DMM module drives the initial change of the position of the 3D Gaussian ball. S2, initialize 3D Gaussian balls. The number of 3D Gaussian balls is the number of mesh midpoints, and the position is the position of the mesh midpoint under neutral expression. The other properties of the 3D Gaussian balls are randomly initialized. S3, the 3D Gaussian sphere obtains the rough attributes of each sphere through optimization, and encodes the input facial expression image through the expression encoder, and obtains the detailed changes of each attribute of the 3D Gaussian sphere through the three-plane framework decoding; S4, through the cycle consistency module to ensure that the result of the expression encoder is only related to the facial expression; S5. Use several regularization terms to constrain the attribute detail changes of the 3D Gaussian sphere obtained by the expression encoder.

[0007] Optionally, the 3DMM model inputs a facial expression picture and outputs points, expression coefficients and identity coefficients for 3D facial reconstruction.

[0008] Optionally, the 3D Gaussian splash in S2 adopts a point-based scene representation, each point corresponds to a 3D Gaussian sphere, and the 3D Gaussian sphere passes through its position , rotation q, scale vector s, opacity And the spherical harmonic coefficients SH describe: ; ; Among them, the attribute set u represents the attributes of the 3D Gaussian sphere, 3DGS represents the 3D Gaussian splash operation, I represents the rendered avatar, the attribute set is differentially rasterized and described by the intrinsic parameters K and extrinsic parameters E of the camera projection, and the loss function is defined as: ; in, Represents the real input avatar.

[0009] Optionally, the expression encoder is composed of a convolutional neural network with a stride of 2 and a channel dimension of (3, 32, 64, 128, 256), and a fully connected layer with an output dimension of 128 is added later: ; ; Among them, DeepFace represents the segmentation method, f represents the convolutional neural network and the fully connected network; At the same time, a three-plane decoding network is designed, and the expression encoding of the three-plane decoding network The input is fed into a 2D convolutional network based on StyleGAN to generate three-plane features Hxy, Hyz, Hzx. For each 3D Gaussian position in the canonical space, the position parameters are used to query three feature vectors Fxy, Fyz, Fzx by projecting them onto each of the three planes. The feature vectors Fxy, Fyz, Fzx are aggregated to obtain a merged feature vector F. The feature vector F is further processed by a lightweight multi-layer perceptron to generate detail changes. and : ; ; ; Optionally, the expression encoder in the cycle consistency module is only related to facial expressions, the expression results rendered from the new perspective are input into the expression encoder, and a cycle consistency loss is designed: ; ; Among them, MSE represents the loss function, I′ represents the rendered new perspective avatar, I′ represents the new perspective avatar part related to expression obtained by DeepFace method. Indicates from The expression codes obtained in An additional encoding step is performed on the final rendered new perspective avatar and the expression encoder framework is shared. Optionally, the plurality of regularization terms include detail variation regularization and Laplace regularization; The position and scale parameters are set in the detail change regularization. A regularization loss function is designed in the detail change regularization to constrain the position and scale parameter changes to approach zero. The regularization loss is expressed as: ; Laplace regularization, during training, minimizes the difference between the Laplacian operator of the 3D Gaussian position in the standard space and the 3D Gaussian position defined by the initial mesh. The loss term is calculated separately for each triangle face, and L2 regularization is used to penalize the distance between the 3D Gaussian point and the basic template mesh, denoted as Llap, and the same Laplace regularizer is applied to the scale and RGB value of the 3D Gaussian point.

[0010] The beneficial effects of the present invention are: The scheme of the present invention has significant advantages in monocular video face reconstruction and driving. By designing an expression encoder module based on three planes, detail expressions that cannot be represented by a 3DMM model can be obtained. The accuracy of expression encoding is guaranteed by using a cycle consistency module. In view of the unconstrained problem of detail changes decoded from expression encoding, a regularization term is designed to constrain the detail changes of the 3D Gaussian sphere obtained by the expression encoder. Therefore, the scheme proposed by the present invention has important application potential in the field of monocular video face reconstruction and driving, and can provide a more accurate and stable solution for research and application in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 The present invention provides a flowchart of a monocular video face reconstruction and driving method based on Gaussian splashing and three planes. DETAILED DESCRIPTION

[0012] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0013] refer to Figure 1 , a monocular video face reconstruction and driving method based on Gaussian splashing and three planes, comprising the following steps: S1. The 3DMM module obtains a mesh explicitly expressed by a parameterized model from each frame of the monocular face video. The 3DMM module drives the initial change of the position of the 3D Gaussian ball. S2, initialize 3D Gaussian balls. The number of 3D Gaussian balls is the number of mesh midpoints, and the position is the position of the mesh midpoint under neutral expression. The other properties of the 3D Gaussian balls are randomly initialized. S3, the 3D Gaussian sphere obtains the rough attributes of each sphere through optimization, and encodes the input facial expression image through the expression encoder, and obtains the detailed changes of each attribute of the 3D Gaussian sphere through the three-plane framework decoding; S4, through the cycle consistency module to ensure that the result of the expression encoder is only related to the facial expression; S5. Use several regularization terms to constrain the attribute detail changes of the 3D Gaussian sphere obtained by the expression encoder.

[0014] In this embodiment, the 3DMM model inputs a facial expression picture and outputs points, expression coefficients and identity coefficients for 3D facial reconstruction.

[0015] In this implementation, the 3D Gaussian splash in S2 uses a point-based scene representation, where each point corresponds to a 3D Gaussian sphere. The 3D Gaussian sphere passes through its position. , rotation q, scale vector s, opacity And the spherical harmonic coefficients SH describe: ; ; Among them, the attribute set u represents the attributes of the 3D Gaussian sphere, 3DGS represents the 3D Gaussian splash operation, I represents the rendered avatar, the attribute set is differentially rasterized and described by the intrinsic parameters K and extrinsic parameters E of the camera projection, and the loss function is defined as: ; in, Represents the real input avatar.

[0016] In this implementation, the expression encoder consists of a convolutional neural network with a stride of 2 and a channel dimension of (3, 32, 64, 128, 256), and a fully connected layer with an output dimension of 128 is added later: ; ; Among them, DeepFace represents the segmentation method, f represents the convolutional neural network and the fully connected network; At the same time, a three-plane decoding network is designed, and the expression encoding of the three-plane decoding network The input is fed into a 2D convolutional network based on StyleGAN to generate three-plane features Hxy, Hyz, Hzx. For each 3D Gaussian position in the canonical space, the position parameters are used to query three feature vectors Fxy, Fyz, Fzx by projecting them onto each of the three planes. The feature vectors Fxy, Fyz, Fzx are aggregated to obtain a merged feature vector F. The feature vector F is further processed by a lightweight multi-layer perceptron to generate detail changes. and : ; ; ; Among them, MSE represents the loss function, I′ represents the rendered new perspective avatar, I′ represents the new perspective avatar part related to expression obtained by DeepFace method. Indicates from The expression codes obtained in An additional encoding step is performed on the final rendered new view avatar and the expression encoder framework is shared. In this embodiment, the regularization terms include detail variation regularization and Laplace regularization; The position and scale parameters are set in the detail change regularization. A regularization loss function is designed in the detail change regularization to constrain the position and scale parameter changes to approach zero. The regularization loss is expressed as: ; Laplace regularization, during training, minimizes the difference between the Laplacian operator of the 3D Gaussian position in the standard space and the 3D Gaussian position defined by the initial mesh. The loss term is calculated separately for each triangle face, and L2 regularization is used to penalize the distance between the 3D Gaussian point and the basic template mesh, denoted as Llap, and the same Laplace regularizer is applied to the scale and RGB value of the 3D Gaussian point.

[0017] In the implementation, a novel monocular video face reconstruction and driving method is proposed. The framework improves the quality of monocular video face reconstruction and driving, reduces image artifacts and image segmentation problems, and proposes an expression encoder under a three-plane framework to address the problem of unclear detail expression. Through additional detail changes, a detail expression is provided for a 3D Gaussian sphere driven by a 3DMM model. The result of the expression encoder should only be related to the expression and has nothing to do with the camera pose. A cycle consistency module is proposed to make the expression encoding result more accurate. The present invention proposes several regularization terms to constrain the detail changes obtained by the expression encoder. Example

[0018] The embodiment uses the PyTorch framework to implement the solution of the present invention, and all experiments are carried out on a single Nvidia 3090 graphics card. The input video image size is 512×512. In the training phase, the network is optimized using the Adam optimizer, the initial learning rate is 1e-4, the maximum number of training rounds is 30,000, and the loss function is the image level loss and the regularization term loss. The scheme of the present invention reports the facial image reconstruction indicators including peak signal-to-noise ratio, structural similarity index and learning-perceptual image block similarity at the NeRFace data level. Each video clip contains approximately 2,000 to 3,500 frames, the last 350 frames are selected as the test clip, and the previous frames are used for training.

[0019] The results in the following table show: ; The proposed method performs well on the NeRFace dataset, significantly surpassing the existing technology and achieving the best results in monocular video face reconstruction and driving.

[0020] In summary, the scheme of the present invention has significant advantages in monocular video face reconstruction and driving. By designing an expression encoder module based on three planes, the detail expression that cannot be represented by the 3DMM model can be obtained. The cycle consistency module is used to ensure the accuracy of expression coding. In view of the unconstrained problem of detail changes decoded from expression coding, a regularization term is designed to constrain the detail changes of the 3D Gaussian sphere obtained by the expression encoder. Therefore, the scheme proposed in the present invention has important application potential in the field of monocular video face reconstruction and driving, and can provide a more accurate and stable solution for research and application in related fields.

[0021] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A monocular video face reconstruction and driving method based on Gaussian splashing and three planes, characterized in that: The steps include: S1. The 3DMM module obtains a mesh explicitly expressed by a parameterized model from each frame of the monocular face video. The 3DMM module drives the initial change of the position of the 3D Gaussian ball. S2, initialize 3D Gaussian balls. The number of 3D Gaussian balls is the number of mesh midpoints, and the position is the position of the mesh midpoint under neutral expression. The other properties of the 3D Gaussian balls are randomly initialized. S3, the 3D Gaussian sphere obtains the rough attributes of each sphere through optimization, and encodes the input facial expression image through the expression encoder, and obtains the detailed changes of each attribute of the 3D Gaussian sphere through the three-plane framework decoding; S4, through the cycle consistency module to ensure that the result of the expression encoder is only related to the facial expression; S5. Use several regularization terms to constrain the attribute detail changes of the 3D Gaussian sphere obtained by the expression encoder.

2. The monocular video face reconstruction and driving method based on Gaussian splashing and three planes according to claim 1, characterized in that: The 3DMM model inputs a facial expression picture and outputs points, expression coefficients and identity coefficients for 3D facial reconstruction.

3. The monocular video face reconstruction and driving method based on Gaussian splashing and three planes according to claim 1, characterized in that: The 3D Gaussian splash in S2 uses a point-based scene representation, each point corresponds to a 3D Gaussian sphere, and the 3D Gaussian sphere passes through its position , rotation q, scale vector s, opacity And the spherical harmonic coefficients SH describe: ； ； Among them, the attribute set u represents the attributes of the 3D Gaussian sphere, 3DGS represents the 3D Gaussian splash operation, I represents the rendered avatar, the attribute set is differentially rasterized and described by the intrinsic parameters K and extrinsic parameters E of the camera projection, and the loss function is defined as: ； in, Represents the real input avatar.

4. The monocular video face reconstruction and driving method based on Gaussian splashing and three planes according to claim 1, characterized in that: The expression encoder consists of a convolutional neural network with a stride of 2 and channel dimensions of (3, 32, 64, 128, 256), followed by a fully connected layer with an output dimension of 128: ； ； Among them, DeepFace represents the segmentation method, f represents the convolutional neural network and the fully connected network; At the same time, a three-plane decoding network is designed. The expression code θ of the three-plane decoding network is input into the 2D convolutional network based on StyleGAN to generate three-plane features Hxy, Hyz, Hzx. For each 3D Gaussian position in the standard space, the position parameters are used to query three feature vectors Fxy, Fyz, Fzx by projecting them onto each three planes. The feature vectors Fxy, Fyz, Fzx are aggregated to obtain a merged feature vector F. The feature vector F is further processed by a lightweight multi-layer perceptron to generate detail changes. and : ； ； 。 5. The monocular video face reconstruction and driving method based on Gaussian splashing and three planes according to claim 1, characterized in that: The expression encoder in the cycle consistency module is only related to facial expressions. The expression results rendered from the new perspective are input into the expression encoder, and a cycle consistency loss is designed: ； ； ； Among them, MSE represents the loss function, I′ represents the rendered new perspective avatar, I′ represents the new perspective avatar part related to expression obtained by DeepFace method. Indicates from The expression codes obtained in An additional encoding step is performed on the final rendered new-view avatar and the expression encoder framework is shared.

6. The monocular video face reconstruction and driving method based on Gaussian splashing and three planes according to claim 1, characterized in that: The plurality of regularization terms include detail variation regularization and Laplace regularization; The position and scale parameters are set in the detail change regularization. A regularization loss function is designed in the detail change regularization to constrain the position and scale parameter changes to approach zero. The regularization loss is expressed as: ； Laplace regularization, during training, minimizes the difference between the Laplacian operator of the 3D Gaussian position in the standard space and the 3D Gaussian position defined by the initial mesh. The loss term is calculated separately for each triangle face, and L2 regularization is used to penalize the distance between the 3D Gaussian point and the basic template mesh, denoted as Llap, and the same Laplace regularizer is applied to the scale and RGB value of the 3D Gaussian point.

Images