Sparse view angle drivable face reconstruction method based on two-dimensional Gaussian

Abstract

The invention provides a sparse view angle drivable face reconstruction method based on two-dimensional Gaussian. Through the two-dimensional Gaussian technology, each point is expressed as a two-dimensional orientation plane Gaussian disc, and a more compact and efficient expression mode is provided. Different from the three-dimensional Gaussian, the two-dimensional Gaussian internally provides a view-consistent geometry when modeling the surface. In order to accurately recover a thin surface and realize stable optimization, two-dimensional Gaussian realizes a perspective accurate two-dimensional Gaussian splashing process by utilizing light splashing intersection and rasterization. Besides, the method combines a depth normalization strategy to divide the rendered image of each visual angle and perform depth normalization on the whole and each local part, so as to better supervise the error of the unobvious change area. Compared with a traditional multi-view method, the method has the advantages that the data acquisition requirement is remarkably reduced, and meanwhile, high geometric accuracy and rendering efficiency are kept.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of face reconstruction, and particularly relates to a sparse-view drivable face reconstruction method based on two-dimensional Gaussian. BACKGROUND

[0002] Realistic three-dimensional faces can enhance the sense of participation and authenticity of users in remote communication and immersive interaction, and are an important technology of virtual reality. However, existing three-dimensional face reconstruction technologies often rely on a large amount of view data or complex scanning devices, which have large amounts of calculations and high data acquisition requirements, limiting their widespread use in real-time applications, especially under low-view or sparse-view input conditions.

[0003] Existing drivable face reconstruction methods mainly include a method based on neural radiance fields and a method based on three-dimensional Gaussian splatting. The neural radiance field technology encodes an object as an implicit function that can be optimized and learned, and reconstructs the object through a neural network. Wojciech Zielonka et al. modeled a dynamic neural radiance field based on a neural primitive embedded around a parametric face model, and realized the reconstruction of a face. However, the reconstructed neural radiance field requires more computing resources and has a long reconstruction time. The three-dimensional Gaussian splatting technology uses an explicit Gaussian point cloud to represent an object, directly optimizes the parameters of the Gaussian points, and uses a GPU acceleration rendering method to improve the rendering speed. Qian Shenhe based on a dynamic three-dimensional representation of three-dimensional Gaussian point cloud splatting, binds these Gaussian point clouds with a parametric face model, parameterizes each Gaussian point cloud through a local coordinate frame of a triangle, and optimizes the explicit displacement offset to obtain a more accurate geometric representation. However, this method requires more views for reconstruction, has high requirements for data acquisition, and has a high data acquisition cost. Moreover, the three-dimensional Gaussian splatting method has certain inconsistency under multiple views, which makes it less effective in surface reconstruction. SUMMARY

[0004] The purpose of the present application is to provide a method for reconstructing a three-dimensional face from sparse viewpoints based on two-dimensional Gaussian to overcome the shortcomings of the prior art. Through the two-dimensional Gaussian technology, each point is represented as a two-dimensional directional planar Gaussian disk, providing a more compact and efficient representation. Unlike three-dimensional Gaussian, two-dimensional Gaussian inherently provides a geometric shape when modeling a surface. In order to accurately recover thin surfaces and achieve stable optimization, two-dimensional Gaussian uses ray splatting intersection and rasterization to realize a perspective-accurate two-dimensional Gaussian splatting process. In addition, the method combines a depth normalization strategy, divides the rendered images of each view, and normalizes the depth of the whole and each local part to better supervise the error of areas with little change. Figure One

[0005] The technical solutions adopted by the present application are as follows: ​

[0006] A sparse viewpoint-driven face reconstruction method based on two-dimensional Gaussian geometry includes:

[0007] Obtain the face image dataset to be reconstructed, which includes 3 to 6 face images with different perspectives and expressions;

[0008] Obtain a coarse face mesh based on a face image dataset;

[0009] Based on the face image dataset, the corresponding monocular depth image is predicted, and based on the coarse face mesh, the monocular depth of the monocular depth image is converted to the depth value in the actual scene as the true depth value.

[0010] Each triangular element in the coarse face mesh is bound to a 2D Gaussian point, which is determined by six parameters: the triangular element number containing the Gaussian point, the spherical harmonic function parameter sh, the opacity α, the scaling parameter sc in the local coordinate system of the triangular element, the position parameter μ, and the rotation parameter r. Initially, the position parameter μ is set to the origin of the local coordinate system, the rotation parameter r is set to the unit rotation matrix, and the scaling parameter sc is set to the unit vector. The local coordinate system of the triangular element describes its positional relationship in global space. During rendering, the 2D Gaussian point is transformed to global space using the scaling parameter sc, the position parameter μ, the rotation parameter r, and the local coordinate system of the triangular element.

[0011] Two-dimensional Gaussian points are projected onto the image plane using a 2D-to-2D mapping in homogeneous coordinates for 2D Gaussian rendering. A bounding box in screen space is then calculated for each Gaussian point. Next, the Gaussian points are sorted based on their center depth, and organized into tiles according to the bounding boxes in sorted order. Finally, a rendered image is obtained by alpha blending of color features accumulated from front to back using volume rendering. Simultaneously, the depth value of each pixel is acquired to obtain a rendered depth image. The 2D Gaussian rendering employs a low-pass filter based on object space.

[0012] Based on the obtained rendering image and rendering depth image, the parameters of the two-dimensional Gaussian point are optimized by minimizing the loss function until the loss function converges or reaches a set number of iterations; wherein, in the optimization process, a two-dimensional Gaussian point is added as a child two-dimensional Gaussian point of the two-dimensional Gaussian point according to the view space position gradient; for the two-dimensional Gaussian point whose view space position gradient is greater than a first threshold value, if the size of the two-dimensional Gaussian point is greater than a second threshold value, the Gaussian point is split into two child two-dimensional Gaussian points as a parent two-dimensional Gaussian point, wherein the two child two-dimensional Gaussian points are in the same triangular patch element, and the number of the triangular patch element is the same as that of the parent two-dimensional Gaussian point; if the size of the two-dimensional Gaussian point is less than or equal to the second threshold value, the two-dimensional Gaussian point is cloned to obtain a new two-dimensional Gaussian point; the number of the triangular patch element where the new two-dimensional Gaussian point is located is the same as that of the original cloned two-dimensional Gaussian point; redundant two-dimensional Gaussian points are removed according to the opacity of each two-dimensional Gaussian point; wherein, the two-dimensional Gaussian points with opacity less than a third threshold value are removed; the opacity of all two-dimensional Gaussian points is periodically reset to a set value.

[0013] Further, the face image dataset includes at least 6 different expressions, 4 different emotions, and one free expression face image sequence; each group of face image sequence includes at least front, left, right, each face looking straight and looking up, a total of 6 viewing angles.

[0014] Further, the rough face mesh is obtained based on the face image dataset, specifically including:

[0015] The FLAME parameterized face model is used to represent the expression and action of the face as a series of parameters, including per-frame parameters: translation t i , joint pose θ i , expression ψ i ; and shared parameters: shape β, vertex offset Δυ and albedo A;

[0016] Based on each frame of face image of the face image dataset, facial feature points and color parameters are obtained, and the facial feature points do not include 17 facial contour feature points;

[0017] Based on the facial feature points and color parameters obtained from each frame of face image, the FLAME parameters are optimized by constructing feature point loss, color loss and regularization term for frame-by-frame learning, and finally a plurality of frames are randomly selected for fine-tuning to obtain the final FLAME parameters, i.e. to obtain the rough face mesh; wherein the parameters of the first frame of face image are randomly initialized, and the parameters of the subsequent frames are used as the initialization.

[0018] Further, the Depth Anything-V2 monocular depth estimation model is used to predict the corresponding monocular depth image.

[0019] Further, the monocular depth of the monocular depth image is converted to the depth value in the actual scene as the depth ground truth based on the coarse face mesh, specifically:

[0020] Based on the coarse face mesh, the vertices of the visible part from the corresponding view angle are taken, the median of the depths of the vertices is taken as the depth offset value of the actual scene, and the average of the differences between the depths of the vertices and the median is taken as the depth scaling ratio s of the actual scene; the median of the depths of the monocular depth image is taken as the monocular depth offset value t', and the average of the differences between the depths of the monocular depth image and the median is taken as the monocular depth scaling ratio s', and the monocular depth d' of the monocular depth image is converted to the depth value d in the actual scene using the following formula:

[0021]

[0022] Further, the alpha-weighted color features are accumulated from front to back by volume rendering alpha blending, which is represented as follows:

[0023]

[0024] In the formula, i represents the i-th sampling point sampled along the x-ray; wherein, color i represents the color value of the i-th sampling point, which is obtained by solving the spherical harmonic function sh i ; alpha i represents the opacity alpha of the i-th sampling point, x represents a homogeneous ray emitted from the camera and passing through the pixel (x, y) and intersecting the two-dimensional Gaussian point at a depth z; u(x) represents the intersection point of the ray x and the two-dimensional Gaussian point; c is the projection of the center p k of the two-dimensional Gaussian point; G(x) represents a Gaussian low-pass filter, which is a lower bound constraint on the screen space Gaussian low-pass filter with c as the center and a radius of sigma; wherein k parameters of the four-dimensional homogeneous plane h u , h v respectively; the two-dimensional Gaussian point is defined in a local tangent plane uv in the global space.

[0025] Further, the loss function includes: a loss based on the rendered image and the ground truth, a position information loss with threshold to ensure that the two-dimensional Gaussian point is always kept within the parent triangle, a scaling loss with threshold to ensure that the two-dimensional Gaussian point is smaller than its parent triangle, a local depth normalization loss, and a global depth normalization loss.

[0026] An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for reconstructing a sparse-view drivable face based on a two-dimensional Gaussian when executing the computer program.

[0027] A storage medium comprising computer executable instructions that, when executed by a computer processor, implement the method for reconstructing a sparse-view drivable face based on a two-dimensional Gaussian as described.

[0028] A computer program product comprising computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the steps of the method for reconstructing a sparse-view drivable face based on a two-dimensional Gaussian.

[0029] The present application has the following advantages:

[0030] (1) By sparse-view reconstruction, the amount of input data is reduced, and the data acquisition cost and reconstruction cost are reduced.

[0031] (2) Drivable: compared with traditional three-dimensional reconstruction methods, the face can be freely driven.

[0032] (3) The result is close to the real case.

[0033] Compared with traditional multi-view methods, the present method significantly reduces the data acquisition requirements, while maintaining high geometric accuracy and rendering efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the method for reconstructing a sparse-view drivable face based on a two-dimensional Gaussian of the present application;

[0035] Figure 2 is a flowchart of binding two-dimensional Gaussian points on a triangular patch and optimization;

[0036] Figure 3 is a qualitative result graph of an implementation case. DETAILED DESCRIPTION

[0037] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is with reference to the drawings, in which like numerals represent like elements throughout the several figures. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0038] The terms used in the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application.

[0039] As used in the present application and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any or all possible combinations of one or more of the associated listed items.

[0040] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one piece of information from another. For example, a first information could be termed a second information, and, similarly, a second information could be termed a first information without departing from the scope of the present application. As used herein, the word "if' can be interpreted to mean "when" or "upon" or "in response to determining" depending on the context.

[0041] One embodiment of the present application is a sparse-view-based face reconstruction method based on two-dimensional Gaussian, the basic flow of which is shown in Figure 1 The method comprises the following steps:

[0042] Step 1: Obtain a face image dataset to be reconstructed, which includes face images of different expressions and different views;

[0043] In a preferred embodiment, the face image dataset includes at least 6 different expression, 4 different emotion and one free expression face image sequence; each group of face image sequence includes at least front view, left view, right view, each view includes level view and upward view, a total of 6 views. The face image dataset can be extracted by video recording and frame image to form the corresponding face image sequence.

[0044] In a preferred embodiment, the face image is down-sampled to a resolution of 802x550.

[0045] Step 2: Obtain a rough face mesh based on the face image dataset;

[0046] In a preferred embodiment, a FLAME parameterized face model is used to construct a rough face mesh, which comprises the following sub-steps:

[0047] Step 2.1: Use the FLAME parameterized face model to represent the expression and action of the face as a series of parameters, including per-frame parameters: translation t i , joint pose θ i , expression ψ i ; and shared parameters: shape β, vertex offset Δυ and albedo A;

[0048] Step 2.2: Obtain facial landmark points and color parameters based on each frame of face image in the face image dataset, wherein the facial landmark points do not include 17 facial contour landmark points. In this process, the 17 facial contour landmark points are deliberately excluded to avoid inconsistency caused by occlusion.

[0049] Step 2.3: Obtain facial landmark points and color parameters based on each frame of face image, and perform frame-by-frame learning optimization of FLAME parameters by constructing landmark loss, color loss and regularization term, and finally randomly select several frames for fine-tuning to obtain the final FLAME parameters, i.e., obtain a rough face mesh; wherein the parameters of the first frame of face image are randomly initialized, and the parameters of the subsequent frames are initialized based on the parameters of the previous frame.

[0050] Step three: predict the corresponding monocular depth image based on the face image dataset, and convert the monocular depth of the monocular depth image to the depth value in the actual scene as the depth ground truth based on the rough face mesh;

[0051] This method needs to use depth to supervise the optimization process. To obtain monocular depth, this method uses the DepthAnything-V2 monocular depth estimation model to predict the depth of the frame image selected in the previous step as the ground truth of the depth. The depth value obtained directly using the monocular depth prediction model is relative depth, which has a difference in scaling range from the actual face mesh. Therefore, the obtained depth needs to be further processed: using the FLAME face mesh obtained in the previous step, taking the vertices of the part visible from the corresponding view, taking the median of the depths of these vertices as the depth offset value t in the actual scene, and taking the average of the differences between the depths of these vertices and the median as the depth scaling ratio s in the actual scene; taking the median of the depths of the monocular depth image as the monocular depth offset value t', and taking the average of the differences between the depths of the monocular depth image and the median as the monocular depth scaling ratio s', and using the following formula to convert the relative depth d' of the monocular depth to the depth value d in the actual scene:

[0052]

[0053] Step four: bind a two-dimensional Gaussian point to each triangle element of the coarse face mesh for subsequent optimization and rendering. The two-dimensional Gaussian point is determined by six parameters: the number of the triangle element where the two-dimensional Gaussian point is located, the spherical harmonic function parameter sh, the opacity a, the scaling parameter sc in the local coordinate system of the triangle element, the position parameter μ, and the rotation parameter r; initially, the position μ is set as the origin of the local coordinate system, the rotation r is set as the unit rotation matrix, the scaling sc is set as the unit vector, the spherical harmonic function parameter sh and the opacity a are randomly initialized, for example, as 0; the local coordinate system of the triangle element is used to describe the positional relationship of the triangle element in the global space, and in the rendering, the two-dimensional Gaussian point is converted to the global space through the scaling parameter sc, the position parameter μ, and the rotation parameter r of the local coordinate system of the triangle element and the local coordinate system of the triangle element;

[0054] In a preferred embodiment, the local coordinate system of the triangle element is determined by three parameters: the scaling parameter k of the triangle element, the position parameter T, and the rotation parameter R. Among them, the average position of the vertices of the triangle element is taken as its position parameter, that is, as the origin of the local space; the direction vector of an edge, the normal vector of the triangle, and their cross product are spliced as column vectors to form a rotation matrix as the rotation parameter R for describing the direction of the triangle in the global space. A scalar is calculated as the scaling parameter k through the average length of a certain edge and its perpendicular distance.

[0055] Specifically, the flow of the binding process is as follows Figure 2 The initialization process is as follows:

[0056] Constructing the local coordinate system of the triangle element: for each triangle element of the coarse face mesh obtained in the last step, its local coordinate system relative to the global coordinate system can be obtained. Specifically, given a triangle element vertex and edge, the average position T of the vertices of the triangle element is taken as the origin of the local space. Then, the direction vector of an edge, the normal vector of the triangle, and their cross product are spliced as column vectors to form a rotation matrix R for describing the direction of the triangle in the global space. In addition, a scalar k is calculated as the scaling parameter k through the average length of a certain edge and its perpendicular distance, which is used to describe the scaling of the triangle.

[0057] For each triangle patch, a 2D Gaussian is bound at its center: A 2D Gaussian is determined by six parameters: the index i of the triangle patch it belongs to, the spherical harmonic parameter sh, the opacity alpha, the scaling parameter s in the local coordinate system of the triangle patch, the position parameter mu, and the rotation parameter r. A 2D Gaussian is initialized as a standard disk with the center of the triangle patch as the origin and the normal of the triangle patch as the initial normal. Initially, the position mu is set to the origin of the local coordinate system, the rotation r is set to the identity rotation matrix, the scaling s is set to the identity vector, and the spherical harmonic parameter sh and the opacity alpha are initialized to 0. During rendering, these attributes are converted to the global space by the following equations:

[0058] r' = Rr# (2)

[0059] mu' = kRmu + T# (3)

[0060] s' = ks# (4)

[0061] The scaling factor k of the triangle patch is introduced in equations (3) and (4) so that the local position and scaling of the 2D Gaussian are relative to the absolute scale of the triangle patch. In this way, although the learning rate in the local space is constant, the learning step in the metric space is adaptive. For example, a 2D Gaussian paired with a smaller triangle patch moves slower than a Gaussian paired with a larger triangle patch in one iteration.

[0062] Step five: project the 2D Gaussian points onto the image plane for 2D Gaussian rendering using the 2D-to-2D mapping in homogeneous coordinates, and then calculate the bounding box of each 2D Gaussian point in screen space; then, sort the 2D Gaussian points based on the depth of their centers and organize them into tiles according to the sorting order based on the bounding box; finally, obtain the rendered image by accumulating alpha-weighted color features from front to back through volume rendering alpha blending; at the same time, obtain the depth value of each pixel to obtain the rendered depth image; wherein the 2D Gaussian rendering uses an object space-based low-pass filter;

[0063] The 2D Gaussian of the present application proposes a method for accurately reconstructing geometric information while maintaining the efficiency of generating new perspectives. Unlike the three-dimensional Gaussian, which models the entire angle radiation as a whole, the two-dimensional Gaussian simplifies three-dimensional modeling by using a two-dimensional Gaussian disk embedded in three-dimensional space. That is, when using a two-dimensional Gaussian model, the primitives are distributed in the plane with a disk density, and each two-dimensional Gaussian point is determined by the center point p k , two principal tangent vectors t u , t v , and a scaling vector S = (s u , s v) to characterize, the normal vector of a 2D Gaussian point is defined as the direction of the steepest density change, i.e., the normal vector is defined by two orthogonal tangent vectors as t w u v Based on the above definitions, the direction can be arranged as a 3x3 rotation matrix R = [t u v w ] and the scaling factor can be arranged as a 3x3 diagonal matrix S with the last item being zero. Thus, the 2D Gaussian is defined in a local tangent plane uv of the global space, and its parametric form is:

[0064] P(u,v) = p k + s u t u u + s v t v v = H[u,v,1,1] T (5)

[0065] where:

[0066]

[0067] where P(u,v) is the parametric representation of a 2D Gaussian point;

[0068] The matrix is a homogeneous transformation matrix representing the geometric shape of a 2D Gaussian point. In the uv space, for a point p = (u,v), its 2D Gaussian value can be calculated by the standard Gaussian function:

[0069]

[0070] Therefore, the projection of the 2D Gaussian disk onto the image plane can be described by a 2D-to-2D mapping in homogeneous coordinates. Let W e R 4×4 be the transformation matrix from the world space to the screen space, and the point in the screen space can be obtained by:

[0071] x = (xz,yz,z,z) T = WP(u,v) = WH(u,v,1,1) T (8)

[0072] where x represents a homogeneous ray that is emitted from the camera and passes through the pixel (x,y), intersecting the Gaussian point at a depth z. In order to rasterize the 2D Gaussian function, if M = (WH) -1 ​​​​The implicit method of the conic projection to the screen space introduces numerical instability in the inverse transformation, especially when a 2D Gaussian point is observed from the side, resulting in its degeneration to a line segment. Therefore, the 2D Gaussian solves this problem by efficiently locating the intersection of the ray-Gaussian point by finding three intersection points located on different planes. For an image coordinate (x, y), the ray of a pixel can be parameterized as the intersection of two orthogonal planes: the x-plane and the y-plane. Specifically, the x-plane is defined by the normal vector (-1, 0, 0) and the offset x, so the x-plane can be represented as the homogeneous four-dimensional plane h x = [-1, 0, 0, x] T . Similarly, the y-plane is h y = [-1, 0, 0, y] T . Therefore, the ray x = (x, y) is determined by the intersection of the x-plane and the y-plane.

[0073] After determining the x, y planes, the two planes are converted to the local coordinate system of the 2D Gaussian point, i.e., the uv coordinate system. Converting a point on a plane by a transformation matrix M is equivalent to converting the homogeneous plane parameters by the inverse transpose M -T . Therefore, M = (WH) -1 is equivalent to (WH) T , thus avoiding explicit matrix inversion, resulting in:

[0074] h u = (WH) T h x , h v = (WH) T h y #(9)

[0075] The points on the 2D Gaussian plane are represented as [u, v, 1, 1] T by equation (5), and the intersection point should fall on the transformed x-plane and y-plane. Therefore:

[0076] h u · [u, v, 1, 1] T = h v · [u, v, 1, 1] T = 0#(10)

[0077] This can efficiently solve the intersection point:

[0078]

[0079] where k is the k parameters of the four-dimensional homogeneous plane parameters. After solving the intersection point, the depth z of the intersection point can be obtained by equation (8), and the Gaussian value can be calculated using equation (7).

[0080] Notably, when viewed from an oblique perspective, a 2D Gaussian degenerates to a line in screen space and thus can be missed during rasterization. To combat this and stabilize the optimization, 2D Gaussian rendering employs an object-space based low-pass filter:

[0081]

[0082] where u(x) is given by equation (11) and c is the center p k of the projection. Intuitively, represents the lower bound constraint imposed by a screen-space Gaussian low-pass filter centered at c with a radius of σ. In this method, σ is set to to ensure that enough pixels are covered during rendering.

[0083] In summary, for the final rendering, 2D Gaussians follow a similar rasterization process as 3D Gaussians. First, a bounding box in screen space is computed for each 2D Gaussian point. Then, the 2D Gaussian points are sorted based on their depth and organized into tiles (16*16) according to their bounding boxes in the sorted order. Finally, alpha-blending is performed through volume rendering to accumulate alpha-weighted color features from front to back:

[0084]

[0085] i denotes the i-th sample point (2D Gaussian point) along the x-ray. Here, color i denotes the color value of the i-th sample point, which is obtained from the spherical harmonics sh i .

[0086] The depth value of each pixel is then obtained to get the rendered depth image.

[0087] Step six: based on the obtained rendering image and rendering depth image, the parameters of the two-dimensional Gaussian point are optimized by minimizing the loss function until the loss function converges or reaches a set number of iterations; wherein, in the optimization process, a two-dimensional Gaussian point is added as a child two-dimensional Gaussian point of the two-dimensional Gaussian point according to the view space position gradient; for the two-dimensional Gaussian point with a view space position gradient greater than a first threshold value, if the two-dimensional Gaussian point size is greater than a second threshold value, the Gaussian point is split into two smaller (smaller than the parent two-dimensional Gaussian point) child two-dimensional Gaussian points as the parent two-dimensional Gaussian point, wherein the two child two-dimensional Gaussian points are located in the same triangular element, and the number of the triangular element is the same as that of the parent two-dimensional Gaussian point; if the two-dimensional Gaussian point size is less than or equal to the second threshold value, the two-dimensional Gaussian point is cloned, and the cloned two-dimensional Gaussian point (smaller than the parent two-dimensional Gaussian point) is used as a new two-dimensional Gaussian point; the number of the triangular element where the new two-dimensional Gaussian point is located is the same as that of the original cloned two-dimensional Gaussian point; redundant two-dimensional Gaussian points are removed according to the opacity of each two-dimensional Gaussian point; wherein, the two-dimensional Gaussian points with an opacity lower than a third threshold value are removed; the opacity of all two-dimensional Gaussian points is periodically reset to a set value (0.1 in this embodiment). In this embodiment, the first threshold value is 0.0002, the second threshold value is 20, and the third threshold value is 0.05.

[0088] The loss function of the application mainly includes two parts: one part is the loss function based on the rendering image and the true value, and the other part is the loss function based on the rendering depth image and the monocular depth.

[0089] Loss based on rendering image

[0090] (1) In this experiment, L1 term and D-SSIM term are combined to supervise the rendering image:

[0091] L rgb =(1-λ)L1+λL D-SSIM #(14)

[0092] Wherein L D-SSIM is a structural dissimilarity loss function, which is 0 when two pictures are equal.

[0093] Wherein L SSIM is a structural similarity loss function, and L1 represents an L1 loss function.

[0094] In the present invention, λ denotes the weight of the D-SSIM term, and takes value from 0 to 1, e.g. λ = 0.2. This method has already achieved good rendering quality without extra supervision (e.g. depth or silhouette supervision). However, when trying to drive these Gaussian points to new expressions and poses by FLAME parameterized face, large-scale spike and speckle artifacts appear in the scene. This is due to the lack of supervision in the alignment between Gaussian points and face triangle primitives.

[0095] (2) Position information loss with threshold

[0096] The basic assumption of two-dimensional Gaussian rigidity in the present invention is that the Gaussian points should roughly match the underlying mesh, and their positions should also match; for example, the Gaussian point representing a certain position on the nose should not be bound to the triangle primitive on the cheek. Although the Gaussian points are initially located at the center of the triangle primitive, and the newly added Gaussian points are close to the existing Gaussian points, after optimization, it cannot be guaranteed that these original points will always remain within the parent triangle primitive. To solve this problem, the present invention regularizes the local position of each Gaussian point in the following way:

[0097] L position = ||max(μ,∈ position )||2#(15)

[0098] where ∈ position is a position error threshold within the scaling range of the parent triangle primitive, which takes a real value, e.g. ∈ position = 1.

[0099] (3) Scaling loss with threshold

[0100] In addition to position, the scaling of two-dimensional Gaussian points is crucial to the visual quality in animation. Specifically, if a certain Gaussian point is too large compared to its parent triangle primitive, a small rotation of the triangle primitive will be amplified by the Gaussian point, resulting in obvious artifacts. To alleviate this problem, the present invention also regularizes the local scaling of each Gaussian point in the following way:

[0101] L scaling = ||max(s,∈ scaling )||2#(16)

[0102] where ∈ scaling is a scaling threshold, which takes a value from 0 to 1, e.g. ∈ scaling = 0.6. When the scaling of the Gaussian point is less than 0.6 times that of its parent triangle primitive, this loss term will be disabled. Without this loss function, the Gaussian points will be excessively scaled down, and the camera ray will need to hit more Gaussian points to achieve zero transmittance, resulting in a decrease in rendering speed.

[0103] Depth-based loss

[0104] Previous neural radiance fields based on depth supervision usually establish depth loss on the original scale of depth map. This alignment measures all losses by a fixed scale based on large-area statistics. Therefore, this may ignore small errors, especially when dealing with multiple targets (such as color reconstruction or wide-range depth changes). This neglect may not be a big problem in previous works based on neural radiance fields, but the problem will be more serious in Gaussian rendering.

[0105] In the Gaussian scene, the movement of the two-dimensional Gaussian point is carried out at a smaller learning rate, so it is more difficult to correct small depth errors; in addition, if the Gaussian primitives are not correctly corrected in position during depth regularization, they will form floating points, especially in areas with rich appearance details.

[0106] To solve this problem, the present application introduces a normalization based on local blocks, so that the loss function focuses on small errors.

[0107] (1) Local depth normalization

[0108] The present application cuts the entire depth map into small blocks and normalizes the blocks of predicted depth and monocular depth, so that their mean value is 0 and the standard deviation is close to 1:

[0109]

[0110] where ∈ is a small value for numerical stability, taking 1 × 10 -4 . denotes the local normalized depth at pixel pixel in block , denotes the depth value at pixel pixel in block , denotes the depth value at all pixels in block , mean(*) denotes taking the mean value, and std(*) denotes taking the standard deviation. From this, each block is normalized on a local scale and the loss is calculated within the block:

[0111] where is the local normalized depth value obtained by rendering the corresponding block

[0112] , is the monocular depth ground truth value, i.e. the true value, of the corresponding block . L2 denotes L2 loss, L LN denotes the local normalized depth loss value of block . (2) Global depth normalization

[0113]

[0114] ​​In addition to focusing on local small errors, a global perspective is needed to learn the overall shape. To compensate for the lack of global scale, global depth normalization is further added in the depth regularization. This enables the depth loss to perceive the global scale while preserving the local correlation. Similar to local normalization, the source scale of depth is removed by block-wise normalization, focusing on local variations. The only difference is that the standard deviation of the whole image I is used to replace the standard deviation of the block:

[0115]

[0116] The global depth loss is:

[0117]

[0118] where D GN is the rendered global normalized depth value, is the corresponding monocular depth value, i.e., the ground truth.

[0119] The final loss function

[0120] The final loss function is:

[0121] L = L rgb + λ position L position + λ scaling L scaling + λ LN L LN + λ GN L GN #(19)

[0122] where λ position , λ scaling , λ LN , λ GN represent the weight of the position information loss with threshold, the scaling loss with threshold, the local normalized depth loss value of the block , and the global depth loss, respectively, for example, λ position = 0.1, λ scaling = 1, λ LN = 0.1, λ GN = 1. In the experiment, L position , L scaling , L LN , L GN are only applied to the visible Gaussian points. That is, these points are regularized and normalized only when there is color loss L rgb . This helps to maintain the structure of the areas that are often occluded, such as teeth and eyeballs.

[0123] ​Further, merely limiting the number of 2D Gaussians to the same number as the number of triangle primitives is not sufficient to capture the details. For example, a single triangle primitive on the scalp can intersect with multiple hairs, which require multiple 2D Gaussians to represent. Therefore, an adaptive density control strategy is needed to add and remove blobs according to the view-space position gradient and the opacity of each 2D Gaussian. For 2D Gaussians with a large view-space position gradient (larger than 0.0002), if their size is large (larger than 20), they are split into two smaller 2D Gaussians; if their size is small, they are cloned.

[0124] The method performs this operation in the local space and ensures that the newly created Gaussians are close to the original Gaussian that triggered the densification operation. Therefore, to enhance the fidelity of the local region, multiple Gaussians are bound within the same triangle primitive. To enhance the distinction, each 2D Gaussian needs to carry a parameter, i.e., the index of its parent triangle primitive, to achieve the binding inheritance during the densification process. In addition to the densification operation, the method also uses a clipping operation as part of the adaptive density control strategy. The method removes 2D Gaussians with an opacity lower than a threshold of 0.05; at the same time, as the operation progresses, the opacity of all 2D Gaussians gradually decreases, and to ensure the operability of the method, the opacity of all Gaussians is periodically reset to 0.1. This technique is very effective in suppressing floating artifacts, but it can cause problems in dynamic scenes. For example, areas in the face that are often occluded (such as the eye triangle) can be too sensitive to this clipping strategy, ultimately resulting in almost no Gaussians attached to these areas. To prevent this, the method tracks the number of Gaussians attached to each triangle primitive and ensures that each triangle primitive always has at least one Gaussian attached. Specifically, the number of Gaussians on each triangle primitive is recorded to ensure that the number of triangle primitives is at least the number of triangle primitives where the Gaussians are located.

[0125] The optimized process is specifically Figure 2 .

[0126] Compared with traditional multi-view methods, the method significantly reduces the data acquisition requirements while maintaining high geometric accuracy and rendering efficiency, providing a solution for reconstructing realistic human faces from few-view input.

[0127] The effects of the present application will be further described below in conjunction with a specific implementation case:

[0128] Step one: obtain the face image dataset to be reconstructed, the face image dataset including 3 to 6 different views and different expressions of human face images;

[0129] The data set of the embodiment is from video recordings of 9 subjects of the NeRSemble data set, and for each video recording, 6 view angles of front, left, right, level and upward are taken. For each subject, the data used consists of 10 sequences of 6 different expressions and 4 different emotions, and one sequence of free play by the subject. When image processing is performed, the image is down-sampled to a resolution of 802x550.

[0130] Step two: obtaining a coarse face mesh based on the face image data set; to obtain the coarse face mesh, the method adopts the FLAME parameterized face model, and the expression and action of the face are represented as a series of parameters, including per-frame parameters (translation t i , joint pose θ i , expression ψ i and shared parameters (shape β, vertex offset Δυ and albedo A). By constructing feature point loss, color loss and regularization term, the above parameters are optimized from the selected frame images of the few-view video. Specifically, the embodiment uses the STAR loss to obtain 68 facial feature points in the 300-W format. In this process, 17 facial contour feature points are deliberately excluded to avoid inconsistency caused by occlusion. At the same time, the FLAME mesh is rendered by using the NVDiffRast method, and the vertex position gradient related to the color loss is obtained by means of texture interpolation of internal pixels and anti-aliasing processing of the boundary. To achieve regularization, the method applies Laplacian smoothing term on the vertex offset and time smoothing term on the per-frame parameters. For the whole process, the FLAME parameters are first optimized on the first frame of the video stream until convergence, and then the parameters of the subsequent frames are initialized with the parameters of the previous frame, and the parameters are optimized for 50 rounds. Then, in order to unify different view angles, a part of all frames are randomly selected, and the final FLAME parameters are obtained by fine-tuning all parameters for 30 rounds.

[0131] Step three: predicting the corresponding monocular depth image based on the face image data set, and converting the monocular depth of the monocular depth image to the depth value in the actual scene as the depth ground truth value by using formula (1) based on the coarse face mesh;

[0132] Step four: Bind a 2D Gaussian point to each triangle element of the coarse human face mesh for subsequent optimization and rendering. The 2D Gaussian point is determined by six parameters: the number of the triangle element where the 2D Gaussian point is located, the spherical harmonic parameter h, the opacity a, the scaling parameter sc in the local coordinate system of the triangle element, the position parameter μ, and the rotation parameter r; initially, the position μ is set as the origin of the local coordinate system, the rotation r is set as the unit rotation matrix, the scaling sc is set as the unit vector, the spherical harmonic parameter sh and the opacity a are randomly initialized as 0; the local coordinate system of the triangle element is used to describe the position relationship of the triangle element in the global space, and in the rendering, the 2D Gaussian point is converted to the global space through the scaling parameter sc, the position parameter μ, and the rotation parameter r of the local coordinate system of the triangle element and the local coordinate system of the triangle element;

[0133] Step five: Project the 2D Gaussian point to the image plane for 2D Gaussian rendering by using the 2D-to-2D mapping in the homogeneous coordinates, and then calculate the bounding box of each Gaussian element in the screen space; then, sort the 2D Gaussian points based on the depth of the center of the 2D Gaussian point, and organize them into tiles (16*16) according to the sorting order according to the bounding box; finally, obtain the rendered image by accumulating the alpha-weighted color features from front to back through volume rendering alpha blending; at the same time, obtain the rendered depth image by obtaining the depth value of each pixel (formula 8); wherein the 2D Gaussian rendering adopts an object space-based low-pass filter;

[0134] Step six: Based on the obtained rendered image, minimize the loss function (formula (14)~(19), wherein λ position = 0.1, λ scaling = 1, λ LN = 0.1, λ GN=1) optimizing the parameters of the two-dimensional Gaussian points until the loss function converges or a set number of iterations is reached; wherein, in the optimization process, a two-dimensional Gaussian point is added as a child two-dimensional Gaussian point of the two-dimensional Gaussian point according to the view space position gradient; for a two-dimensional Gaussian point whose view space position gradient is greater than a first threshold value, if the size of the two-dimensional Gaussian point is greater than a second threshold value, the two-dimensional Gaussian point is split into two smaller child two-dimensional Gaussian points as a parent two-dimensional Gaussian point, wherein the two child two-dimensional Gaussian points are in the same triangular patch element as the parent two-dimensional Gaussian point; if the size of the two-dimensional Gaussian point is less than or equal to the second threshold value, the two-dimensional Gaussian point is cloned to obtain a new two-dimensional Gaussian point; the triangular patch element in which the new two-dimensional Gaussian point is located is the same as the triangular patch element in which the original cloned two-dimensional Gaussian point is located; redundant two-dimensional Gaussian points are removed according to the opacity of each two-dimensional Gaussian point; wherein, the two-dimensional Gaussian points whose opacity is less than a third threshold value are removed; the opacity of all two-dimensional Gaussian points is periodically reset to a set value.

[0135] The present embodiment uses an Adam optimizer to optimize parameters. For the position of the two-dimensional Gaussian point (a three-dimensional vector), the learning rate is set to 5x10 -3 , and for the scaling ratio (a two-dimensional vector), it is set to 5x10 -3 , and for the rotation parameter (a quaternion), the learning rate is 1x10 -3 , and the learning rate of the transparency is 5x10 -2 . In addition to the Gaussian point parameters, the present embodiment also fine-tunes the translation, joint rotation and expression parameters of the FLAME face for each time step, and the learning rates are 1x10 -6 , 1x10 -5 and 1x10 -3 . The experiment is trained for 60000 iterations, and the learning rate of the Gaussian point position is exponentially decayed until the final iteration, at which time the learning rate is reduced to 0.01 times the initial value. Starting from the 10000th iteration until the end, adaptive density control with binding inheritance is enabled every 2,000 iterations. Every 3000 iterations, the opacity of the Gaussian is reset.

[0136] The present application evaluates the results on the divided test set, compares the quality of the rendered images on the test view and expression. The evaluation of the results involves the following indicators:

[0137] (1) Peak Signal-to-Noise Ratio (PSNR): a measure of the ratio of the maximum possible power of a signal to the power of the noise affecting its representation accuracy. The higher the PSNR value, the closer the rendered image is to the real image, and the better the image quality.

[0138] (2) Structural Similarity Index (SSIM): a metric to measure the similarity between two images, which takes into account the brightness, contrast and structural information of the images. The value of SSIM is between 0 and 1, the closer to 1, the more similar the two images, the better the preservation of the structural information of the image.

[0139] (3) Perceptual Loss (LPIPS): a deep learning model-based image similarity measure, which measures the difference between images by comparing the distance in the feature space. LPIPS can better reflect the subjective evaluation of human vision on image quality, the lower the value, the smaller the perceptual difference between images, the better the image quality.

[0140] The quantitative results are shown in Table 1, and the qualitative results are shown in Figure 3 Compared with traditional multi-view methods, the present method significantly reduces the requirements of data acquisition, while maintaining high geometric accuracy and rendering efficiency.

[0141] Table 1 Quantitative results

[0142] PSNR SSIM LPIPS Gaussian Avatars Results 31.68 0.9153 0.1189 Invention Results 32.3 0.9215 0.1001

[0143] Corresponding to the foregoing embodiment of the two-dimensional Gaussian-based sparse view driveable face reconstruction method, the present application also provides an embodiment of an electronic device for implementing a two-dimensional Gaussian-based sparse view driveable face reconstruction method.

[0144] The electronic device provided by the embodiment of the present application comprises one or more processors for implementing the two-dimensional Gaussian-based sparse view driveable face reconstruction method in the above embodiment.

[0145] The embodiment of the electronic device of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer.

[0146] The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device in which it is located. From the hardware level, in addition to the processor, memory, network interface and non-volatile memory, the device in the embodiment is usually equipped with other hardware according to the actual functions of the device with data processing capability, which will not be described here.

[0147] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be described here.

[0148] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0149] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the two-dimensional Gaussian-based sparse view angle drivable face reconstruction method in the above embodiment.

[0150] The computer readable storage medium can be an internal storage unit of any data processing device in the above-mentioned embodiments, such as a hard disk or a memory. The computer readable storage medium can also be any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can also include an internal storage unit of any data processing device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0151] Obviously, the above embodiments are only examples for clear illustration, and are not limitations on the embodiments. For those skilled in the art, other different forms of changes or variations can be made on the basis of the above description. All the embodiments do not need to be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A two-dimensional Gaussian-based sparse-view-angle drivable face reconstruction method, characterized in that, The method comprises the following steps: acquiring a face image dataset to be reconstructed, the face image dataset comprising face images of different angles and different expressions; acquiring a coarse face mesh based on the face image dataset; predicting a corresponding monocular depth image based on the face image dataset, and converting monocular depth of the monocular depth image to a depth value in an actual scene as a depth ground truth based on the coarse face mesh; binding a two-dimensional Gaussian point to each triangular element of the coarse face mesh, the two-dimensional Gaussian point being determined by six parameters: a number of the triangular element where the two-dimensional Gaussian point is located, a spherical harmonic function parameter sh, an opacity a, a scaling parameter sc in a local coordinate system of the triangular element, a position parameter mu, and a rotation parameter r; in an initial state, the position parameter mu is set as an origin of the local coordinate system, the rotation parameter r is set as a unit rotation matrix, and the scaling parameter sc is set as a unit vector; the local coordinate system of the triangular element is used to describe a positional relationship of the triangular element in a global space; in rendering, the two-dimensional Gaussian point is converted to the global space through the scaling parameter sc, the position parameter mu, and the rotation parameter r in the local coordinate system of the triangular element and the local coordinate system of the triangular element; projecting the two-dimensional Gaussian point onto an image plane for two-dimensional Gaussian rendering by using a two-dimensional to two-dimensional mapping in homogeneous coordinates, and then calculating a bounding box of each two-dimensional Gaussian point in a screen space; then, the two-dimensional Gaussian points are sorted based on depths of centers of the two-dimensional Gaussian points, and the two-dimensional Gaussian points are organized into tiles according to the bounding boxes in a sorting order; finally, a rendering image is obtained by accumulating alpha-weighted color features from front to back through volume rendering alpha blending; meanwhile, a rendering depth image is obtained by acquiring a depth value of each pixel; wherein the two-dimensional Gaussian rendering adopts an object space-based low-pass filter; based on the obtained rendering image and the rendering depth image, optimizing parameters of the two-dimensional Gaussian points by minimizing a loss function until the loss function converges or a set number of iterations is reached; wherein in the optimization process, a two-dimensional Gaussian point is added as a child two-dimensional Gaussian point of the two-dimensional Gaussian point according to a view space position gradient; for a two-dimensional Gaussian point with a view space position gradient greater than a first threshold value, if a size of the two-dimensional Gaussian point is greater than a second threshold value, the Gaussian point is taken as a parent two-dimensional Gaussian point and is split into two child two-dimensional Gaussian points, wherein the two child two-dimensional Gaussian points have the same number of the triangular element where the parent two-dimensional Gaussian point is located; if the size of the two-dimensional Gaussian point is less than or equal to the second threshold value, the two-dimensional Gaussian point is cloned to obtain a new two-dimensional Gaussian point; the new two-dimensional Gaussian point has the same number of the triangular element where the original cloned two-dimensional Gaussian point is located; redundant two-dimensional Gaussian points are removed according to opacities of each two-dimensional Gaussian point; wherein two-dimensional Gaussian points with opacities lower than a third threshold value are removed; opacities of all two-dimensional Gaussian points are periodically reset to a set value.

2. The method of claim 1, wherein, The face image dataset includes at least 6 different expressions, 4 different emotions, and one free face image sequence; each group of face image sequences includes at least front, left, right, each with 6 viewing angles of looking straight and looking up.

3. The method of claim 1, wherein, The rough face mesh is obtained based on the face image dataset, and specifically includes: The FLAME parameterized face model is adopted, and expressions and actions of the face are represented as a series of parameters, including per-frame parameters: translation t i , joint pose θ i , expression ψ i ; and shared parameters: shape β, vertex offset Δυ and albedo A; The face feature points and color parameters are obtained based on each frame of face image of the face image dataset, and the face feature points do not include 17 face contour feature points; Based on the face feature points and color parameters of each frame of face image, the FLAME parameters are learned and optimized frame by frame by constructing feature point loss, color loss and regularization term, and finally a plurality of frames are randomly selected for fine tuning to obtain the final FLAME parameters, that is, the rough face mesh; wherein the parameters of the first frame of face image are randomly initialized, and the parameters of the subsequent frames are used as the initialization.

4. The method of claim 1, wherein, The corresponding monocular depth image is predicted by using a Depth Anything-V2 monocular depth estimation model.

5. The method of claim 1, wherein, The monocular depth of the monocular depth image is converted to the depth value in the actual scene as the depth real value based on the rough face mesh, and specifically: Based on the rough face mesh, the vertices of the visible part from the corresponding viewing angle are taken, the median of the depths of the vertices is taken as the depth offset value in the actual scene, and the average of the differences between the depths of the vertices and the median is taken as the depth scaling ratio s in the actual scene; The median of the depths of the monocular depth image is taken as the monocular depth offset value t', and the average of the differences between the depths of the monocular depth image and the median is taken as the monocular depth scaling ratio s', and the monocular depth d' of the monocular depth image is converted to the depth value d in the actual scene using the following formula:

6. The method of claim 1, wherein, The alpha-weighted color features are accumulated from front to back by volume rendering alpha blending, and are represented as follows: In the formula, i represents the i-th sampling point along the x-ray; where color i The color value of the i-th sampling point is represented by the spherical harmonic function sh. i We obtain α. i Let α represent the opacity of the i-th sampling point, x represent a homogeneous ray emitted from the camera, passing through pixel (x,y), and intersecting the 2D Gaussian point at depth z; u(x) represent the intersection point of ray x and the 2D Gaussian point; c is the center p of the 2D Gaussian point. k The projection of; G(x) represents a Gaussian low-pass filter, This represents the lower bound constraint on the Gaussian low-pass filter imposed by a screen space centered at c and with radius σ; where They are the four-dimensional homogeneous plane h u h v k parameters; the two-dimensional Gaussian point is defined in a local tangent plane uv in the global space.

7. The method of claim 1, wherein, The loss function includes: loss based on the rendered image and the real value, thresholded position information loss for ensuring that the two-dimensional Gaussian points are always kept within the parent triangle, thresholded scaling loss for ensuring that the two-dimensional Gaussian points are smaller than the parent triangle, local depth normalization loss, and global depth normalization loss.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the two-dimensional Gaussian-based sparse view angle drivable face reconstruction method of any one of claims 1-7.

9. A storage medium containing computer executable instructions which, when executed by a computer processor, implement a two-dimensional Gaussian-based sparse view angle drivable face reconstruction method of any one of claims 1-7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the two-dimensional Gaussian-based sparse view angle drivable face reconstruction method of any one of claims 1-7.

Images