A monocular video human gaussian avatar reconstruction method

Abstract

The application provides a monocular video human body Gaussian avatar reconstruction method, and the implementation steps are as follows: obtaining a training and test sample set; constructing a reconstruction network model comprising a cascaded UV mapping module, an encoder, a decoder module, a linear blending skinning LBS module and a three-dimensional Gaussian sputtering module; iteratively training the reconstruction network model and obtaining a reconstruction result. The UV mapping module in the application cuts the reconstruction result output by the SMPL-X grid generator based on the anatomical principle through an anatomical cutter, and maps the SMPL-X parameters of the human body model into continuous and uniformly distributed two-dimensional data through a Laplace expander and a key point uniform distributor, so that the three-dimensional geometric structure can be more effectively learned from the two-dimensional features, the information loss is reduced, and the reconstruction accuracy is effectively improved; meanwhile, the application overcomes the defect of relying on camera parameters, and improves the applicability in the uncalibrated scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a method for reconstructing a human Gaussian avatar from a monocular video, which can be applied to fields such as virtual reality, augmented reality, and metaverse. Background Technology

[0002] In the field of computer vision, human Gaussian avatar reconstruction is achieved through 3D Gaussian sputtering technology based on parametric human models. Specifically, firstly, parametric human models such as SMPL-X are used to estimate human pose and shape parameters from monocular video. Then, the human surface is unfolded onto a 2D plane through UV mapping or projection. A convolutional neural network is used to learn the 3D Gaussian meta-parameters corresponding to each position from the 2D representation, including center position, covariance matrix, color, and opacity. Finally, a Gaussian template in normal space is deformed to the target pose space through linear blending skinning, and image synthesis is performed using a differentiable 3D Gaussian sputtering renderer. The network parameters are then optimized by inversely analyzing the loss between the rendered image and the real image.

[0003] For example, patent document CN119741431A discloses a method for creating a Gaussian human avatar from monocular video based on multi-resolution image-text fusion. This invention first uses a custom text description of the human video as model input in the text image feature processing module; it then designs a multi-resolution feature enhancement module to improve temporal consistency; and finally, it renders the generated Gaussian point cloud and fused dynamic features using 3D Gaussian splashing technology. This invention captures global and local dynamic details by extracting and fusing features at different resolutions. The low-resolution path enhances the ability to capture the dynamic global appearance, while the high-resolution path focuses on restoring the fine features of the human avatar. By combining temporal consistency, high-resolution details can help capture subtle dynamic changes and alleviate cross-frame jitter, making the generated human avatar's movements more natural and coherent. However, the text image feature processing module of this invention uses a discontinuous UV mapping method in the process of acquiring the initial input features, resulting in discontinuities and loss of 3D geometric information during 2D mapping, affecting further improvement in reconstruction accuracy. Furthermore, this method is highly dependent on known camera parameters and is difficult to apply in uncalibrated scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the defects of the prior art and propose a monocular video human Gaussian avatar reconstruction method to solve the technical problems of low reconstruction accuracy and poor adaptability in the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain the training sample set and the test sample set:

[0007] Acquire images captured by a monocular camera, including The frame contains RGB images of the human model and a training sample set of its corresponding SMPL-X parameters, as well as... The test sample set of RGB images, where, , , This indicates the total number of frames in the RGB image;

[0008] (2) Constructing a human Gaussian avatar reconstruction network model:

[0009] Construct a human Gaussian avatar reconstruction network model that includes a cascaded UV mapping module, encoder, decoder module, linear hybrid skin LBS module, and 3D Gaussian sputtering module. The UV mapping module includes a cascaded SMPL-X mesh generator, anatomical cutter, Laplacian expander, keypoint uniform distributor, and boundary square distributor. It is used to cut the reconstructed SMPL-X mesh based on anatomical principles and then transform the SMPL-X parameters of the cut open mesh into continuous and uniformly distributed two-dimensional data.

[0010] (3) Iteratively train the human Gaussian avatar reconstruction network model:

[0011] The human Gaussian avatar reconstruction network model was trained using a sample set. Iterative training is performed, during which the model parameters are updated by applying L1 loss, SSIM loss (structural similarity), and LPIPS (contrast learning similarity) to the rendered image and its corresponding RGB image, resulting in a well-trained reconstruction network model. ;

[0012] (4) Obtain the results of the human Gaussian avatar reconstruction:

[0013] The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain The results of human Gaussian avatar reconstruction for each test sample.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] (1) In the process of iteratively training the human Gaussian avatar reconstruction network model and obtaining the reconstruction results, the UV mapping module cuts the reconstruction results output by the SMPL-X mesh generator based on the anatomical principle through the anatomical cutter, and maps the SMPL-X parameters of the human model into continuous and uniformly distributed two-dimensional data through the Laplacian expander and the key point uniform distributor. This can more effectively learn the three-dimensional geometric structure from the two-dimensional features, avoid the information loss caused by non-continuous UV mapping and non-uniform expansion in the prior art, and effectively improve the reconstruction accuracy.

[0016] (2) In the process of iteratively training the human Gaussian avatar reconstruction network model, the present invention updates the model parameters by using the L1 loss, SSIM loss and LPIPS loss of the rendered image and its corresponding RGB image, avoiding the defect of the prior art being highly dependent on camera parameters, and effectively improving the applicability in uncalibrated scenarios compared with the prior art. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of the human Gaussian avatar reconstruction network model of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Reference Figure 1 The present invention includes the following steps:

[0021] Step 1) Obtain the training sample set and the test sample set:

[0022] Acquire images captured by a monocular camera, including The frame contains RGB images of the human model and a training sample set of its corresponding SMPL-X parameters, as well as... The test sample set of RGB images, where, , , This indicates the total number of frames in the RGB image;

[0023] SMPL-X parameters are a parametric representation of the human body model based on vertices and triangles, including human pose parameters. Shape parameters and global translation vector Among them, attitude parameters Used to describe the rotation angles and shape parameters of various joints in the human body. Translation vectors are used to describe individual morphological differences in humans, such as height, weight, etc. Used to describe the overall position of the human body in three-dimensional space. A corresponding 3D human body mesh can be generated using the SMPL-X parameters. , its first The three-dimensional coordinates of the vertices are as follows: ;

[0024] In this embodiment, the SMPL-X parameters corresponding to the RGB images of the human model are obtained by estimating the RGB images containing the human model in a monocular video using a pre-trained Neural Local Localization Field Network (NLF). The NLF includes a sequentially connected neural local localization field extractor and a synchronization optimizer. The neural local localization field extractor is used to predict the two-dimensional pixel coordinates of human key points from the input RGB image. and three-dimensional spatial coordinates The synchronization optimizer is used to perform synchronous optimization based on the keypoint coordinates predicted by the neural localization field extractor by constructing a 3D keypoint alignment loss function and a 2D reprojection loss function. Defined as:

[0025] ;

[0026] in This indicates that the SMPL-X model has the following attitude parameters. and shape parameters Next The three-dimensional coordinates of each vertex. It is a global translation vector. This represents the total number of key points.

[0027] Two-dimensional reprojection loss function Defined as:

[0028] ;

[0029] in The parameter is The projection function that projects three-dimensional points onto a two-dimensional image plane;

[0030] The objective function for simultaneous optimization is defined as:

[0031] ;

[0032] The Kabsch algorithm and the least squares algorithm are used for simultaneous optimization until the function converges, and the fitted SMPL-X parameters are output.

[0033] Step 2) Construct a human Gaussian avatar reconstruction network model, the structure of which is as follows: Figure 2 As shown:

[0034] Construct a human Gaussian avatar reconstruction network model that includes a cascaded UV mapping module, encoder, decoder module, linear hybrid skin LBS module, and 3D Gaussian sputtering module. ,in:

[0035] The UV mapping module includes a cascaded SMPL-X mesh generator, anatomical cutter, Laplacian expander, keypoint uniform distributor, and boundary square distributor. It is used to cut the reconstructed SMPL-X mesh based on anatomical principles, and then map the SMPL-X parameters of the cut open mesh into continuous and uniformly distributed two-dimensional data before transformation.

[0036] The decoder module includes a parallel arrangement of geometry decoders and appearance decoders;

[0037] The linear hybrid skin LBS module includes a cascaded skin weight assigner, joint transformation calculator, and linear hybrid deformation actuator;

[0038] The 3D Gaussian sputtering module includes a cascaded projector and sorter, a differentiable sputtering renderer, and an image post-processor.

[0039] Step 3) Iteratively train the human Gaussian avatar reconstruction network model:

[0040] (3a) Initialize the number of iterations to be The maximum number of iterations is , , No. Second iteration to rebuild network model The horizontal and vertical focal lengths are respectively , and order ;

[0041] (3b) The UV mapping module uses anatomical principles to map the SMPL-X parameters in each training sample. Mapping to a two-dimensional plane involves the following steps:

[0042] (3b1) The SMPL-X mesh generator generates SMPL-X parameters for each training sample. Reconstruction is performed to obtain coordinates of multiple vertices. SMPL-X mesh;

[0043] (3b2) The anatomical cutter cuts the SMPL-X mesh along the back midline, inner arm, and inner thigh of the human model; the Laplace unfolder extracts the coordinates of multiple vertices in the resulting open mesh composed of multiple triangular facets. Mapping to a 2D plane yields multiple 2D unfolded coordinates. And Laplace smoothing constraints are used to maintain local geometric continuity. Defined as:

[0044] ;

[0045] in, As vertices 2D coordinates Let it be the set of its adjacent vertices. The total number of vertices;

[0046] (3b3) Keypoint uniform distribution for multiple 2D unfolded coordinates To optimize, a uniform distribution constraint based on the area of ​​Voronoi cells is adopted. Avoid focusing on key points, which is defined as:

[0047] ;

[0048] in, To predefine the number of key points on the human body, Key point The area of ​​the surrounding Voronoi cells, The average area is used; an area preservation constraint is introduced simultaneously. Maintaining the area ratio of the triangular facets, it is defined as:

[0049] ;

[0050] in, For noodles Jacobian matrix, This is the global scaling factor. This represents the total number of triangular facets.

[0051] The overall optimization objective is defined as:

[0052] ;

[0053] in, , and The weights of the Laplace smoothing constraint, the uniform distribution constraint, and the area preservation constraint are respectively represented. A phased optimization strategy is adopted. First, the Laplace and area preservation constraints are optimized, and then the uniform distribution constraint of key points is fine-tuned. The L-BFGS algorithm is used to solve the problem.

[0054] The bounding square distributor distributes the coordinates of multiple vertices uniformly on the optimized 2D plane. Affine transformation to a standard unit square yields the SMPL-X parameters. Corresponding two-dimensional vertex coordinates .

[0055] The encoder obtains the two-dimensional vertex coordinates through mapping. Encode the data to obtain a geometric texture map.

[0056] The geometry decoder in the decoder module upsamples the geometry texture map to obtain the center position of each pixel. Covariance Matrix The appearance decoder upsamples the geometry texture map to obtain the color corresponding to each pixel. and opacity ;

[0057] The skin weight allocator in the linear blended skin LBS module is based on the center position in the normal space corresponding to each pixel. Generate the skin weight vector for each pixel. The joint transformation calculator calculates the rotation matrix for each transformation unit. The linear hybrid deformation actuator utilizes the skin weight vector. For rotation matrix Perform linear blending to obtain the center position of each pixel in the target pose space. Covariance Matrix :

[0058] ;

[0059] ;

[0060] And the color corresponding to each pixel obtained by upsampling from the appearance decoder. and opacity Fusion generates a human Gaussian avatar template in the target pose space. ;

[0061] The projector and sorter in the 3D Gaussian sputtering module will generate a human Gaussian avatar template in the target pose space. Each pixel in the image is determined by the camera's horizontal and vertical focal lengths. , Projecting onto a 2D image plane yields the 2D projection of each pixel and its covariance matrix, and sorts all Gaussian elements according to the projection depth; the differentiable sputtering renderer generates the rendered image by calculating the color and opacity contributions of each pixel. Image post-processors process the rendered image. Normalization is performed to obtain the final rendered image and output it;

[0062] (3c) By rendering the image and its corresponding RGB image Calculate the loss value of the reconstructed network model and through For horizontal and vertical focal lengths , The network model is updated to obtain the reconstruction network model for this iteration. ,in:

[0063] Loss value The calculation formula is:

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] in: Indicates the rendered image Its corresponding RGB image L1 loss, express and Structural similarity SSIM loss, express and Image contrast learning similarity LPIPS loss, , , This represents the weight of the corresponding loss. Represents the total number of pixels in an RGB image. , express , The Middle Intensity of each pixel , express , The average value of the middle pixels. , express , Standard deviation of mid-pixel for and covariance, , It is a constant. express , Number of image contrast learning layers This indicates the feature channel normalization operation. Indicates the first Layer learning weights It represents the Hadamah accumulation. This indicates the operation of taking the 2-norm.

[0069] For horizontal and vertical focal lengths , The formula for updating is:

[0070] ;

[0071] ;

[0072] in, , express , The update results Indicates the learning rate. This represents partial differential operations.

[0073] In step 1) of this invention, the camera parameters and human pose have been optimized synchronously using the Neural Local Localization Field (NLF), and the initialized camera focal length has been obtained. In this stage, when further fine-tuning the focal length using image loss, based on projection geometry, the projected coordinates of the 3D human points after rotation and alignment in step 1) can be expressed as follows: , ,in This is the camera translation distance. When the shooting distance is relatively far... When the projection is approximately equal to , At this point, there is a coupling effect between the focal length and the translation distance, that is, the focal length and the translation distance... When scaled by the same ratio, the projection result remains essentially unchanged; therefore, the optimization may converge to an equivalent solution rather than a physically accurate solution, but this coupling does not affect the quality of the rendered image or the final reconstruction accuracy of the human Gaussian avatar; when the camera is close to the human body, the coupling naturally weakens, and more accurate camera parameters can be obtained; through this mechanism, the present invention can still reconstruct a high-quality human Gaussian avatar without any prior camera calibration.

[0074] (3d) judgment If true, then a well-trained reconstruction network model is obtained. Otherwise, let , Then proceed with step (3b).

[0075] (4) Obtain the results of the human Gaussian avatar reconstruction:

[0076] The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain The results of human Gaussian avatar reconstruction for each test sample.

[0077] The technical effects of the present invention will be further explained below with reference to simulation results:

[0078] 1. Experimental conditions and contents:

[0079] The experiment used the People-Snapshot and NeuMan datasets. The hardware platform was an NVIDIA RTX 4090 GPU, and the software platform was PyTorch.

[0080] The reconstruction accuracy of this invention and existing monocular video Gaussian human avatar methods based on multi-resolution image-text fusion was compared through simulation, including peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS). The results are shown in Table 1.

[0081] 2. Analysis of Experimental Results:

[0082] Referring to Table 1, the method of this invention was used to reconstruct multiple test objects on the People-Snapshot dataset. Table 1 presents a quantitative comparison of existing methods and the method of this invention on four test objects. The method of this invention outperforms existing methods that rely on camera parameters in terms of PSNR, SSIM, and LPIPS without requiring any prior camera parameters. Experimental results show that this invention, by using a continuous and uniform UV mapping strategy, can effectively improve the reconstruction accuracy of human Gaussian avatars while eliminating dependence on known camera parameters.

[0083] Table 1

[0084]

Claims

1. A method for reconstructing a human Gaussian avatar from a monocular video, characterized in that, Includes the following steps: (1) Obtain the training sample set and the test sample set: Acquire images captured by a monocular camera, including The frame contains RGB images of the human model and a training sample set of its corresponding SMPL-X parameters, as well as... The test sample set of RGB images, where, , , This indicates the total number of frames in the RGB image; (2) Constructing a human Gaussian avatar reconstruction network model: Construct a human Gaussian avatar reconstruction network model that includes a cascaded UV mapping module, encoder, decoder module, linear hybrid skin LBS module, and 3D Gaussian sputtering module. The UV mapping module includes a cascaded SMPL-X mesh generator, anatomical cutter, Laplacian expander, keypoint uniform distributor, and boundary square distributor. It is used to cut the reconstructed SMPL-X mesh based on anatomical principles and then transform the SMPL-X parameters of the cut open mesh into continuous and uniformly distributed two-dimensional data. (3) Iteratively train the human Gaussian avatar reconstruction network model: The human Gaussian avatar reconstruction network model was trained using a sample set. Iterative training is performed, during which the model parameters are updated by applying L1 loss, SSIM loss (structural similarity), and LPIPS (contrast learning similarity) to the rendered image and its corresponding RGB image, resulting in a well-trained reconstruction network model. ; (4) Obtain the results of the human Gaussian avatar reconstruction: The test sample set is used as the trained reconstruction network model. The input is propagated forward to obtain The results of human Gaussian avatar reconstruction for each test sample.

2. The method according to claim 1, characterized in that, The SMPL-X parameters mentioned in step (1) are obtained by estimating the RGB image of the human model in the monocular video through the pre-trained Neural Local Localization Field Network (NLF).

3. The method according to claim 1, characterized in that, The reconstruction of the network model described in step (2) ,in: The decoder module includes a parallel arrangement of geometry decoders and appearance decoders; The linear hybrid skin LBS module includes a cascaded skin weight assigner, joint transformation calculator, and linear hybrid deformation actuator; The 3D Gaussian sputtering module includes a cascaded projector and sorter, a differentiable sputtering renderer, and an image post-processor.

4. The method according to claim 3, characterized in that, The iterative training described in step (3) is implemented as follows: (3a) Initialize the number of iterations to be The maximum number of iterations is , , No. Second iteration to rebuild network model The horizontal and vertical focal lengths are respectively , and order ; (3b) The UV mapping module uses anatomical principles to map the SMPL-X parameters in each training sample. Mapped to a two-dimensional plane; the encoder processes the mapped two-dimensional vertex coordinates. Encode; The geometry decoder and appearance decoder in the decoder module upsample the encoded geometry texture map respectively; The linear hybrid skinning LBS module fuses the upsampled information from the geometry decoder and appearance decoder to form a human Gaussian avatar template in canonical space. Converted to human Gaussian avatar template in target pose space The 3D Gaussian sputtering module uses the camera's horizontal and vertical focal lengths... , right Rendering is performed to obtain a rendered image for each sample. ; (3c) By rendering the image and its corresponding RGB image Calculate the loss value of the reconstructed network model and through For horizontal and vertical focal length , The network model is updated to obtain the reconstruction network model for this iteration. ; (3d) Judgment If true, then a well-trained reconstruction network model is obtained. Otherwise, let , Then proceed with step (3b).

5. The method according to claim 4, characterized in that, The step (3b) describes the SMPL-X parameters in each training sample. Mapping to a two-dimensional plane involves the following steps: (3b1) The SMPL-X mesh generator generates SMPL-X parameters for each training sample. Reconstruction is performed to obtain coordinates of multiple vertices. SMPL-X mesh; (3b2) The anatomical cutter cuts the SMPL-X mesh along the back midline, inner arm, and inner thigh of the human model; the Laplace unfolder extracts the coordinates of multiple vertices in the resulting open mesh composed of multiple triangular facets. Mapping to a 2D plane yields multiple 2D unfolded coordinates. ; (3b3) Keypoint uniform distribution for multiple 2D unfolded coordinates Optimization is performed; the bounding square distributor distributes the coordinates of multiple vertices evenly on the optimized 2D plane. Affine transformation to a standard unit square yields the SMPL-X parameters. Corresponding two-dimensional vertex coordinates .

6. The method according to claim 4, characterized in that, The human Gaussian avatar template in the normative space described in step (3b) This includes the center position of each pixel obtained by upsampling the geometric texture map using the geometric decoder. Covariance Matrix And the color corresponding to each pixel obtained by upsampling the geometry texture map by the appearance decoder. and opacity , .

7. The method according to claim 4, characterized in that, The loss value described in step (3c) The calculation formula is: ； ； ； ； in: Indicates the rendered image Its corresponding RGB image L1 loss, express and Structural similarity SSIM loss, express and Image contrast learning similarity LPIPS loss, , , This represents the weight of the corresponding loss. Represents the total number of pixels in an RGB image. , express , The Middle Intensity of each pixel , express , The average value of the middle pixels. , express , Standard deviation of mid-pixel for and covariance, , It is a constant. express , Number of image contrast learning layers This indicates the feature channel normalization operation. Indicates the first Layer learning weights It represents the Hadamah accumulation. This indicates the operation of taking the 2-norm.

8. The method according to claim 4, characterized in that, The horizontal and vertical focal lengths described in step (4c) , The update is performed using the following formula: ； ； in, , express , The update results Indicates the learning rate. This represents partial differential operations.

Images