Monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model

Abstract

The present invention discloses a monocular video face reconstruction and driving method based on Gaussian splash and 3DMM model, comprising the following steps: S1, converting each frame of the face RGB video into an explicitly represented face grid based on the 3DMM model; S2, constructing a geometric enhancement module, and using an improved iterative nearest point algorithm to geometrically align the eye and tooth regions; S3, initializing a 3D Gaussian ball, and driving the dynamic change of the 3D Gaussian ball according to the point change of the face grid under different expressions; S4, assigning adaptively optimized learning strategy coefficients based on the characteristics of different facial regions; S5, adding confidence information to each 3D Gaussian ball, generating a confidence image by rendering and multiplying it with the rendered image, calculating the high-dimensional feature similarity with the input image, and optimizing the attribute parameters and rendering effect of the 3D Gaussian ball. The present invention combines Gaussian splash and 3DMM model to achieve high-precision reconstruction and driving of monocular video faces.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a monocular video face reconstruction and driving method based on Gaussian splashing and a 3DMM model. Background Art

[0002] Creating realistic 3D head avatars from monocular video has always been a challenging problem in computer graphics. Rendering lifelike avatars from arbitrary viewpoints can provide rich application scenarios in many fields such as gaming, video conferencing, augmented reality (AR) and virtual reality (VR). For these applications, the key issue is whether it is possible to reconstruct high-quality 3D head avatars and extend them to new poses and expressions.

[0003] Generally, the reconstruction of 3D head avatars is considered a basic task. Recently, neural radiance fields and their variants have achieved remarkable results in static scene reconstruction. Subsequent research has further extended neural radiance fields to the field of digital human modeling. Although these methods perform well in novel view synthesis, they often lack controllability, which limits their generalization ability to new poses and expressions. Recent progress in 3D Gaussian splashing has surpassed neural radiance fields in rendering quality for novel view synthesis. 3D Gaussian splashing represents space as 3D Gaussian spheres, each of which contains position, rotation, scale, opacity and spherical harmonic coefficients, and uses alpha blending to splash the Gaussian spheres into the final rendered image. Due to the improvement in rendering quality, 3D Gaussian splashing has been rapidly applied to the field of digital humans and virtual avatar generation.

[0004] Existing technologies usually rely on 3D Morphable Model (3DMM) to reconstruct faces from monocular videos. The 3DMM model can better capture and reconstruct facial features by representing the face as a set of parameterized geometric shapes and texture features. However, traditional 3DMM models often face the problem of insufficient details when dealing with complex facial expression changes. For example, details in areas such as the eyes, mouth, and teeth are difficult to capture accurately, which limits the accuracy of the model. In addition, the training and optimization of traditional 3DMM models often rely on limited data sets, and the diversity of these data sets is not enough to cover all possible facial feature changes, resulting in poor adaptability of the model in real scenes.

[0005] In terms of optimizing facial details, existing technologies often use geometric expression alignment methods. However, these methods generally rely on fixed geometric models or predefined key point alignment schemes, and it is difficult to dynamically adapt to the characteristics of different regions. In addition, although the existing iterative closest point (ICP) algorithm is widely used in geometric alignment tasks, it usually assumes that the global weight is evenly distributed and cannot give higher weight optimization to complex areas (such as eyes and teeth), which leads to limited alignment accuracy in local areas. At the same time, the traditional ICP algorithm also has shortcomings in convergence speed and global optimization capabilities, and is prone to falling into local optimal solutions.

[0006] In terms of dynamic driving, existing technologies usually use simple linear deformations or predefined expression-driven models. However, these methods perform poorly in complex expression changes and non-rigid deformations. For example, linear deformations cannot capture the nonlinear changes of facial muscles, while traditional expression-driven models are often limited to fixed expression types and lack flexible support for dynamic changes. In addition, in the dynamic driving process, existing technologies usually find it difficult to balance the relationship between global consistency and local details, especially in the training process of facial areas. There is a lack of effective learning strategy adjustment methods, which dilutes the training difficulty of complex areas (such as eyes and mouths), and simple areas occupy too many computing resources, thus affecting the overall effect.

[0007] In addition, in order to improve the rendering effect, many existing methods try to enhance the expressive power of the model by introducing three-dimensional point cloud or voxel representation. However, these methods are often computationally intensive and cannot meet the needs of real-time applications. In the rendering process, traditional methods mostly use simple loss functions to optimize the rendering results, such as mean square error (MSE) or structural similarity (SSIM), but these indicators cannot fully capture the perceptual similarity, resulting in a significant gap between the quality of the final generated image and the real image. Especially in detail rendering, the existing technology lacks the importance of different facial areas, resulting in poor detail quality in complex areas, affecting the overall visual effect of the rendering.

[0008] In summary, the existing technologies in monocular video face reconstruction and driving have the following major problems and deficiencies: First, the traditional 3DMM model has insufficient expression capabilities for detail areas, especially in the reconstruction accuracy of areas such as eyes and teeth, there is still much room for improvement; Second, the optimization capability of the geometric alignment method is limited, and it is impossible to strike a balance between complex areas and global optimization; Third, the dynamic driving model has defects in capturing nonlinear expression changes and improving local training efficiency; Fourth, the existing rendering optimization methods are insufficient in perceiving similarities and detail expression capabilities, and fail to make full use of regional characteristics for differentiated optimization.

[0009] Therefore, how to provide a monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model is a problem that technicians in this field urgently need to solve. Summary of the invention

[0010] One object of the present invention is to propose a monocular video face reconstruction and driving method based on Gaussian splash and 3DMM model. The present invention combines Gaussian splash and 3DMM model, constructs a geometric enhancement module, initializes 3D Gaussian ball and allocates adaptive optimization learning strategy coefficients, so as to achieve high-precision reconstruction and dynamic driving of human faces in monocular video. The present invention has the advantages of strong dynamic adaptability, accurate expression modeling and high rendering quality, can effectively improve the reconstruction accuracy under complex expression changes, optimize the detail expression of facial area, and reduce the computational redundancy of simple areas, significantly improve the overall modeling and rendering efficiency, and is suitable for a variety of complex scenes of face reconstruction and driving.

[0011] According to an embodiment of the present invention, a monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model includes the following steps:

[0012] The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model is characterized by comprising the following steps:

[0013] S1, based on the 3DMM model, each frame of the face RGB video is converted into an explicit face mesh;

[0014] S2, constructing a geometric enhancement module, optimizing the face mesh, using an improved iterative closest point algorithm to geometrically align the eye and tooth regions, and generating an optimized face mesh;

[0015] S3, initialize the 3D Gaussian sphere, use the coordinates of the neutral expression space point in the optimized face mesh as the initial position of the 3D Gaussian sphere, and randomly initialize the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere, and drive the dynamic change of the 3D Gaussian sphere according to the point changes of the face mesh under different expressions;

[0016] S4, based on the characteristics of different facial regions, allocate adaptively optimized learning strategy coefficients to dynamically adapt to the learning difficulty of different facial regions;

[0017] S5. Add confidence information to each 3D Gaussian sphere, generate a confidence image by rendering, multiply the confidence image with the rendered image, calculate the high-dimensional feature similarity with the input image, and optimize the attribute parameters and rendering effect of the 3D Gaussian sphere.

[0018] Optionally, the 3DMM model is constructed based on a self-supervised framework, and is trained using wild facial images and synthetic facial images to generate a face mesh including expression coefficients and identity coefficient displays, which is used to accurately represent the changes in different facial expressions and features.

[0019] Optionally, the S2 specifically includes:

[0020] S21, in the face mesh, geometric alignment processing is performed on the eye and tooth regions, a geometric enhancement module is constructed, key points are marked for the eye and tooth regions on the face mesh, and they are defined as target key point sets respectively. and the reference keypoint set ,in represents the target key points on the face mesh, Indicates the corresponding reference key point in the template model;

[0021] S22. Calculate the rotation matrix and translation vector based on the improved iterative closest point algorithm, and perform geometric alignment by minimizing the weighted Euclidean distance between the target keypoint and the reference keypoint:

[0022] ;

[0023] in, represents the weight factor of the target key point, express The rotation matrix describes the rotation transformation of the reference key point set; t represents Translation vector, describing the translation transformation of the reference keypoint set;

[0024] S23. In each iteration, the weight factor of the target key point is dynamically updated:

[0025] ;

[0026] in, represents the weight factor of the updated target key point, exp represents the exponential function, Indicates the target key point position of the current iteration, represents the Gaussian kernel bandwidth parameter;

[0027] S24. Use the current rotation matrix and translation vector to update the target key point position of the current iteration:

[0028] ;

[0029] in, Indicates the updated target key point position of the current iteration;

[0030] S25. Introduce an error evaluation mechanism to judge the convergence of alignment based on the weighted mean square error between the target key points and the reference key points:

[0031] ;

[0032] Among them, E represents the error, and N represents the total number of target key points;

[0033] S26. When the error E is lower than a predetermined threshold or reaches a set maximum number of iterations, the iteration is stopped and the optimized face mesh is output.

[0034] The improved iterative closest point algorithm introduces dynamic weight adjustment. The weight factor of the key point is dynamically adjusted according to the error of the key point in the geometric alignment process, which improves the iteration efficiency and significantly enhances the robustness of the geometric alignment. At the same time, the error evaluation mechanism is optimized, and the convergence is judged by weighted mean square error to avoid the local convergence problem in the traditional iterative closest point algorithm. In view of the high-precision requirements of the eye and tooth areas, the improved iterative closest point algorithm significantly improves the effect of local area geometric detail alignment.

[0035] Optionally, the S3 specifically includes:

[0036] S31. Each point in the optimized face grid corresponds to a 3D Gaussian sphere. The properties of the 3D Gaussian sphere include position , rotation q, scale vector s, opacity And the spherical harmonic coefficients , the attribute set is defined as:

[0037] ;

[0038] S32, the attributes of the 3D Gaussian sphere are initialized through a point-based scene representation, the coordinates of the neutral expression space point in the optimized face grid are used as the initial position of the 3D Gaussian sphere, and the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere are randomly initialized;

[0039] S33. During the rendering process, a final rendering image is generated:

[0040] ;

[0041] Among them, I represents the rendered image, represents 3D Gaussian splash, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera;

[0042] S34, comparing the rendered image I with the input real image Optimize the pixel-level error between:

[0043] ;

[0044] in, represents the real input image, Represents the absolute value of pixel difference, which is used to guide the property optimization of 3D Gaussian sphere;

[0045] S35. The rendering process is implemented through differential rasterization technology, so that the properties of the 3D Gaussian sphere are continuously optimized to improve the quality of the rendered image and the degree of matching with the input image.

[0046] Optionally, the S4 specifically includes:

[0047] S41, assigning a semantic information label to each facial region to initialize the semantic attributes of each facial region, wherein the semantic information label includes an eye region, a mouth region, a forehead region, and a cheek region;

[0048] S42. During training, dynamically track the semantic attributes of the facial regions associated with each 3D Gaussian sphere:

[0049] S43, explicitly labeling the semantic information labels of the facial regions in the 3D space, and allocating adaptively optimized learning strategy coefficients according to the semantic attributes and training difficulty of each facial region, so as to dynamically adjust the training rate and optimize the training difficulty of different facial regions:

[0050] ;

[0051] Among them, U represents the attribute set of the 3D Gaussian sphere, represents the position, q represents the rotation, s represents the scale vector, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, and SH represents the spherical harmonic coefficient.

[0052] Optionally, the S5 specifically includes:

[0053] S51. Introduce confidence information for each 3D Gaussian sphere, and generate a confidence map through 3D Gaussian splash rendering:

[0054] ;

[0055] in, represents the confidence map, represents 3D Gaussian splash, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, Represents confidence information, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera;

[0056] S52, generate the confidence map Multiplying with the rendered image I is used to strengthen the association between the attribute information of the 3D Gaussian sphere and the input image;

[0057] S53. Calculate the high-dimensional feature similarity between the rendered image and the input image through the perceptual loss function:

[0058] ;

[0059] in, represents the perceptual loss value, F represents the 2D facial encoder, represents the real input image;

[0060] S54. By optimizing the perceptual loss function, the attribute parameters of each 3D Gaussian sphere are dynamically adjusted to optimize the realism and detail quality of the rendered image.

[0061] The beneficial effects of the present invention are:

[0062] First, the present invention uses a 3DMM model to accurately convert each frame of the face RGB video into an explicitly represented face mesh. Compared with traditional methods, the present invention has significantly improved the accuracy of mesh expression, laying a solid foundation for subsequent steps. By constructing a geometric enhancement module and using an improved iterative closest point algorithm to geometrically align the eye and tooth areas, the expression of local features is optimized, making the optimized face mesh more accurate in detail processing in local areas, especially in alignment accuracy in key areas, which is better than traditional methods.

[0063] Secondly, in the initialization and dynamic driving process of the 3D Gaussian ball, the present invention randomly initializes the rotation, scale vector, opacity and spherical harmonic coefficients by combining the neutral expression space points of the optimized grid, and drives the dynamic changes of the 3D Gaussian ball according to the dynamic changes of the grid points. This mechanism enables the model to be adaptively adjusted in different expression scenes, thereby improving the modeling ability of complex expression changes, especially in terms of consistency and fluency in dynamic expression performance.

[0064] In addition, the present invention dynamically adjusts the learning rate of different regions by introducing adaptive optimization learning strategy coefficients based on facial region characteristics. This regional optimization strategy based on semantic information makes the training of complex regions (such as eyes and mouth) more accurate, while reducing redundant calculations of simple regions (such as forehead and cheeks), thereby improving the model training efficiency while further enhancing the ability to express details in each facial region.

[0065] In the rendering and confidence optimization stage, the present invention introduces confidence information for each 3D Gaussian sphere and generates a confidence map through 3D Gaussian splash rendering. After multiplying the confidence map with the rendered image, the high-dimensional feature similarity is calculated through the perceptual loss function, thereby dynamically optimizing the attribute parameters of each 3D Gaussian sphere. This process significantly improves the realism and detail quality of the rendered image, making the generated image more consistent with the input image in visual perception while maintaining rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0067] Figure 1 The overall flow chart of the monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model proposed by the present invention;

[0068] Figure 2 Visualization results of face reconstruction using the monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model proposed in the present invention and advanced methods on the NeRFace dataset;

[0069] Figure 3 This is the visualization result of face reconstruction on the NeRFBlendshape dataset using the monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model proposed in the present invention and the advanced method. DETAILED DESCRIPTION

[0070] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0071] refer to Figure 1-Figure 3 , a monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model includes the following steps:

[0072] The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model is characterized by comprising the following steps:

[0073] S1, based on the 3DMM model, each frame of the face RGB video is converted into an explicit face mesh;

[0074] S2, constructing a geometric enhancement module, optimizing the face mesh, using an improved iterative closest point algorithm to geometrically align the eye and tooth regions, and generating an optimized face mesh;

[0075] S3, initialize the 3D Gaussian sphere, use the coordinates of the neutral expression space point in the optimized face mesh as the initial position of the 3D Gaussian sphere, and randomly initialize the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere, and drive the dynamic change of the 3D Gaussian sphere according to the point changes of the face mesh under different expressions;

[0076] S4, based on the characteristics of different facial regions, allocate adaptively optimized learning strategy coefficients to dynamically adapt to the learning difficulty of different facial regions;

[0077] S5. Add confidence information to each 3D Gaussian sphere, generate a confidence image by rendering, multiply the confidence image with the rendered image, calculate the high-dimensional feature similarity with the input image, and optimize the attribute parameters and rendering effect of the 3D Gaussian sphere.

[0078] In this embodiment, the 3DMM model is constructed based on a self-supervised framework and is trained using 400,000 field facial images and 100,000 synthetic facial images to generate a face mesh that includes expression coefficients and identity coefficients for accurately representing changes in different facial expressions and features.

[0079] In this implementation, S2 specifically includes:

[0080] S21, in the face mesh, geometric alignment processing is performed on the eye and tooth regions, a geometric enhancement module is constructed, key points are marked for the eye and tooth regions on the face mesh, and they are defined as target key point sets respectively. and the reference keypoint set ,in represents the target key points on the face mesh, Indicates the corresponding reference key point in the template model;

[0081] S22. Calculate the rotation matrix and translation vector based on the improved iterative closest point algorithm, and perform geometric alignment by minimizing the weighted Euclidean distance between the target keypoint and the reference keypoint:

[0082] ;

[0083] in, represents the weight factor of the target key point, express The rotation matrix describes the rotation transformation of the reference key point set; t represents Translation vector, describing the translation transformation of the reference keypoint set;

[0084] S23. In each iteration, the weight factor of the target key point is dynamically updated:

[0085] ;

[0086] in, represents the weight factor of the updated target key point, exp represents the exponential function, Indicates the target key point position of the current iteration, represents the Gaussian kernel bandwidth parameter;

[0087] S24. Use the current rotation matrix and translation vector to update the target key point position of the current iteration:

[0088] ;

[0089] in, Indicates the updated target key point position of the current iteration;

[0090] S25. Introduce an error evaluation mechanism to judge the convergence of alignment based on the weighted mean square error between the target key points and the reference key points:

[0091] ;

[0092] Among them, E represents the error, and N represents the total number of target key points;

[0093] S26. When the error E is lower than a predetermined threshold or reaches a set maximum number of iterations, the iteration is stopped and the optimized face mesh is output.

[0094] The improved iterative closest point algorithm introduces dynamic weight adjustment. The weight factor of the key point is dynamically adjusted according to the error of the key point in the geometric alignment process, which improves the iteration efficiency and significantly enhances the robustness of the geometric alignment. At the same time, the error evaluation mechanism is optimized, and the convergence is judged by weighted mean square error to avoid the local convergence problem in the traditional iterative closest point algorithm. In view of the high-precision requirements of the eye and tooth areas, the improved iterative closest point algorithm significantly improves the effect of local area geometric detail alignment.

[0095] In this implementation, S3 specifically includes:

[0096] S31. Each point in the optimized face grid corresponds to a 3D Gaussian sphere. The properties of the 3D Gaussian sphere include position , rotation q, scale vector s, opacity And the spherical harmonic coefficients , the attribute set is defined as:

[0097] ;

[0098] S32, the attributes of the 3D Gaussian sphere are initialized through a point-based scene representation, the coordinates of the neutral expression space point in the optimized face grid are used as the initial position of the 3D Gaussian sphere, and the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere are randomly initialized;

[0099] S33. During the rendering process, a final rendering image is generated:

[0100] ;

[0101] Among them, I represents the rendered image, represents 3D Gaussian splash, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera;

[0102] S34, comparing the rendered image I with the input real image Optimize the pixel-level error between:

[0103] ;

[0104] in, represents the real input image, Represents the absolute value of pixel difference, which is used to guide the property optimization of 3D Gaussian sphere;

[0105] S35. The rendering process is implemented through differential rasterization technology, so that the properties of the 3D Gaussian sphere are continuously optimized to improve the quality of the rendered image and the degree of matching with the input image.

[0106] In this implementation, S4 specifically includes:

[0107] S41, assigning a semantic information label to each facial region to initialize the semantic attributes of each facial region, wherein the semantic information label includes an eye region, a mouth region, a forehead region, and a cheek region;

[0108] S42. During training, dynamically track the semantic attributes of the facial regions associated with each 3D Gaussian sphere:

[0109] S43, explicitly labeling the semantic information labels of the facial regions in the 3D space, and allocating adaptively optimized learning strategy coefficients according to the semantic attributes and training difficulty of each facial region, so as to dynamically adjust the training rate and optimize the training difficulty of different facial regions:

[0110] ;

[0111] Among them, U represents the attribute set of the 3D Gaussian sphere, represents the position, q represents the rotation, s represents the scale vector, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, and SH represents the spherical harmonic coefficient.

[0112] The core idea of ​​this scheme is to model the face as a structure containing semantic information, assigning specific regional attributes to each facial region. These parameters are variable and can be easily represented in 2D space. By injecting facial region semantic information into the 3D Gaussian optimization process, each 3D Gaussian within the facial region r obtains new semantic parameters. We initialize the facial semantic attributes of different facial regions using semantic labels. During training, we dynamically track the specific semantic attributes associated with each 3D Gaussian sphere, and adjust the training importance of different facial regions by multiplying the semantic attributes of the facial region with the opacity of the corresponding 3D Gaussian sphere. By explicitly annotating specific facial regions in 3D space, facial semantic labels are able to solve the problem of detail quality caused by the different rendering difficulties of different facial regions. The facial region initialized by 3D Gaussian is recorded in the scheme and divided into 16 regions, including additional initialized regions. The region information of each region is initialized to 0.5.

[0113] In this implementation manner, S5 specifically includes:

[0114] S51. Introduce confidence information for each 3D Gaussian sphere, and generate a confidence map through 3D Gaussian splash rendering:

[0115] ;

[0116] in, represents the confidence map, represents 3D Gaussian splash, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, Represents confidence information, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera;

[0117] S52, generate the confidence map Multiplying with the rendered image I is used to strengthen the association between the attribute information of the 3D Gaussian sphere and the input image;

[0118] S53. Calculate the high-dimensional feature similarity between the rendered image and the input image through the perceptual loss function:

[0119] ;

[0120] in, represents the perceptual loss value, F represents the 2D facial encoder, represents the real input image;

[0121] S54. By optimizing the perceptual loss function, the attribute parameters of each 3D Gaussian sphere are dynamically adjusted to optimize the realism and detail quality of the rendered image.

[0122] Embodiment 1:

[0123] In order to verify the feasibility of the present invention in implementation, the scheme of the present invention was implemented using the PyTorch framework, and all experiments were performed on a single Tesla V100 graphics card. The input video image size was 512×512. During the training phase, the network was optimized using the Adam optimizer with an initial learning rate of 1e-4. The maximum number of training rounds was 30,000. The loss functions were image-level loss and high-dimensional feature consistency loss.

[0124] The proposed scheme tests the facial image reconstruction indicators including peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and learning-perceptual image patch similarity (LPIPS) at NeRFace and NeRFBlendshape data levels. Each video clip contains about 2000 to 3500 frames. We select the last 350 frames as the test clip, and the previous frames are used for training.

[0125] Table 1 Performance comparison of different methods on NeRFace and NeRFBlendShape datasets

[0126]

[0127] According to the results in Table 1 above, the method proposed in the present invention performs well on the NeRFace dataset and the NeRFBlendshape dataset, significantly surpassing the existing technology, and achieves the best results in monocular video face reconstruction and driving.

[0128] On the NeRFace dataset, the solution of the present invention showed significant performance advantages. First, in terms of the PSNR (peak signal-to-noise ratio) indicator, the value of the solution of the present invention is 29.09, which is significantly improved compared to 28.07 of SplattingAvatar and 27.45 of Flashavatar, indicating that it has better performance in image reconstruction fineness and accuracy. Secondly, in terms of SSIM (structural similarity index), the solution of the present invention and SplattingAvatar are both 0.929, which is significantly higher than 0.890 of Flashavatar, showing an advantage in structural consistency. In addition, in terms of LPIPS (learning perceptual image block similarity) indicators, the value of the solution of the present invention is 0.0695, which is better than 0.0697 of SplattingAvatar and 0.0923 of Flashavatar, indicating that the images it generates are closer to real images in terms of perceptual similarity.

[0129] On the NeRFBlendShape dataset, the solution of the present invention also achieved excellent performance. In terms of the PSNR index, the solution of the present invention reached 30.24, which is much higher than SplattingAvatar's 28.71 and Flashavatar's 27.97, demonstrating its powerful reconstruction capability in complex scenes. In terms of the SSIM index, the solution of the present invention significantly exceeded SplattingAvatar's 0.919 and Flashavatar's 0.893 with 0.950, fully demonstrating its ability to retain the global structural characteristics of the image. In terms of the LPIPS index, the value of the solution of the present invention was 0.0938, which is slightly higher than SplattingAvatar's 0.0916, but lower than Flashavatar's 0.0965, showing that it still maintains a high level of visual perception quality.

[0130] In summary, the scheme of the present invention has significant advantages in monocular video face reconstruction and driving. By designing a new 3DMM module, the present invention can obtain a more accurate and explicitly represented face mesh from a monocular video, increase the number of 3D Gaussian spheres initialized by using geometric enhancement of the eyes and teeth, propose a training method for adaptively adjusting each facial area in view of the different learning difficulties of different facial areas, and add a confidence attribute to the 3D Gaussian sphere by using high-dimensional feature similarity. Therefore, the scheme proposed by the present invention has important application potential in the field of monocular video face reconstruction and driving, and can provide a more accurate and stable solution for research and application in related fields.

[0131] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model, characterized in that: The steps include: S1, based on the 3DMM model, each frame of the face RGB video is converted into an explicit face mesh; S2, constructing a geometric enhancement module, optimizing the face mesh, using an improved iterative closest point algorithm to geometrically align the eye and tooth regions, and generating an optimized face mesh; S3, initialize the 3D Gaussian sphere, use the coordinates of the neutral expression space point in the optimized face mesh as the initial position of the 3D Gaussian sphere, and randomly initialize the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere, and drive the dynamic change of the 3D Gaussian sphere according to the point changes of the face mesh under different expressions; S4, based on the characteristics of different facial regions, allocate adaptively optimized learning strategy coefficients to dynamically adapt to the learning difficulty of different facial regions; S5. Add confidence information to each 3D Gaussian sphere, generate a confidence image by rendering, multiply the confidence image with the rendered image, calculate the high-dimensional feature similarity with the input image, and optimize the attribute parameters and rendering effect of the 3D Gaussian sphere; The S2 specifically includes: S21, in the face mesh, geometric alignment processing is performed on the eye and tooth regions, a geometric enhancement module is constructed, key points are marked for the eye and tooth regions on the face mesh, and they are defined as target key point sets respectively. and the reference keypoint set ,in represents the target key points on the face mesh, Indicates the corresponding reference key point in the template model; S22. Calculate the rotation matrix and translation vector based on the improved iterative closest point algorithm, and perform geometric alignment by minimizing the weighted Euclidean distance between the target keypoint and the reference keypoint: ； in, represents the weight factor of the target key point, express The rotation matrix describes the rotation transformation of the reference key point set; t represents Translation vector, describing the translation transformation of the reference keypoint set; S23. In each iteration, the weight factor of the target key point is dynamically updated: ； in, represents the weight factor of the updated target key point, exp represents the exponential function, Indicates the target key point position of the current iteration, represents the Gaussian kernel bandwidth parameter; S24. Use the current rotation matrix and translation vector to update the target key point position of the current iteration: ； in, Indicates the updated target key point position of the current iteration; S25. Introduce an error evaluation mechanism to judge the convergence of alignment based on the weighted mean square error between the target key points and the reference key points: ； Among them, E represents the error, and N represents the total number of target key points; S26. When the error E is lower than a predetermined threshold or reaches a set maximum number of iterations, the iteration is stopped and the optimized face mesh is output.

2. The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model according to claim 1 is characterized in that: The 3DMM model is built based on a self-supervised framework and is trained using field facial images and synthetic facial images to generate a face mesh including expression coefficients and identity coefficient displays, which is used to accurately represent the changes in different facial expressions and features.

3. The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model according to claim 1 is characterized in that: The S3 specifically includes: S31. Each point in the optimized face grid corresponds to a 3D Gaussian sphere. The properties of the 3D Gaussian sphere include position , rotation q, scale vector s, opacity And the spherical harmonic coefficients , the attribute set is defined as: ； S32, the attributes of the 3D Gaussian sphere are initialized through a point-based scene representation, the coordinates of the neutral expression space point in the optimized face grid are used as the initial position of the 3D Gaussian sphere, and the rotation, scale vector, opacity and spherical harmonic coefficients of the 3D Gaussian sphere are randomly initialized; S33. During the rendering process, a final rendering image is generated: ； Among them, I represents the rendered image, represents 3D Gaussian splash, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera; S34, comparing the rendered image I with the input real image Optimize the pixel-level error between: ； in, represents the real input image, Represents the absolute value of pixel difference, which is used to guide the property optimization of 3D Gaussian sphere; S35. The rendering process is implemented through differential rasterization technology, so that the properties of the 3D Gaussian sphere are continuously optimized to improve the quality of the rendered image and the degree of matching with the input image.

4. The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model according to claim 1 is characterized in that: The S4 specifically includes: S41, assigning a semantic information label to each facial region to initialize the semantic attributes of each facial region, wherein the semantic information label includes an eye region, a mouth region, a forehead region, and a cheek region; S42. During training, dynamically track the semantic attributes of the facial regions associated with each 3D Gaussian sphere: S43, explicitly labeling the semantic information labels of the facial regions in the 3D space, and allocating adaptively optimized learning strategy coefficients according to the semantic attributes and training difficulty of each facial region, so as to dynamically adjust the training rate and optimize the training difficulty of different facial regions: ； Among them, U represents the attribute set of the 3D Gaussian sphere, represents the position, q represents the rotation, s represents the scale vector, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, and SH represents the spherical harmonic coefficient.

5. The monocular video face reconstruction and driving method based on Gaussian splashing and 3DMM model according to claim 1 is characterized in that: The S5 specifically includes: S51. Introduce confidence information for each 3D Gaussian sphere, and generate a confidence map through 3D Gaussian splash rendering: ； in, represents the confidence map, represents 3D Gaussian splash, Indicates opacity, represents the learning strategy coefficient of adaptive optimization, Represents confidence information, Represents the camera internal parameters and defines the camera projection matrix; Represents the camera external parameters, describing the position and direction of the camera; S52, generate the confidence map Multiplying with the rendered image I is used to strengthen the association between the attribute information of the 3D Gaussian sphere and the input image; S53. Calculate the high-dimensional feature similarity between the rendered image and the input image through the perceptual loss function: ； in, represents the perceptual loss value, F represents the 2D facial encoder, represents the real input image; S54. By optimizing the perceptual loss function, the attribute parameters of each 3D Gaussian sphere are dynamically adjusted to optimize the realism and detail quality of the rendered image.

Images