A local information guided human free-view rendering method and system

Abstract

The application discloses a local information guided human free-viewpoint rendering method, comprising the following steps: inputting a single-view human video dataset, fitting SMPL posture and shape parameters and performing random sampling to obtain three-dimensional point cloud data; constructing a Gaussian distribution based on the point cloud, initializing Gaussian distribution attributes, training parameter offsets by using a multilayer perception machine and performing fitting to obtain three-dimensional Gaussian positions after deformation, and combining joint rotation matrices to implement LBS transformation to obtain Gaussian distribution of an observation frame posture; decomposing and predicting Gaussian colors to obtain final colors; projecting the three-dimensional Gaussian to a two-dimensional plane according to a view angle parameter and performing rasterization to render a human two-dimensional image, and dynamically calculating a densification threshold according to a local perception adaptive density control strategy to control the density of the Gaussian. The application further discloses a local information guided human free-viewpoint rendering system, and the application can effectively improve the detail fitting degree and color prediction accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image rendering, specifically to a method and system for rendering human free viewpoints guided by local information. Background Technology

[0002] In recent years, with the continuous development of computer technology and the continuous improvement of equipment performance, people's demand for understanding and reconstructing three-dimensional scenes and objects has been increasing. Compared with text and images, three-dimensional data offers a richer and more three-dimensional presentation, and is more consistent with our real world. Among the massive amounts of data, human body data is the most common. With the continuous development of artificial intelligence technology, virtual digital humans have gradually entered our daily lives, and have been widely used in virtual reality, game and film production, digital e-commerce, and other fields and industries. High-quality free-viewpoint rendering has broad application prospects and potential commercial value in film and television entertainment, holographic communication, and other fields.

[0003] Human free-viewpoint image rendering technology is a process that uses computer graphics and computer vision techniques to recover the geometry and appearance of the human body from data sources such as images, videos, and depth data, to model the human body, and then render realistic human images from a specific viewpoint based on camera parameters. Achieving accurate, efficient, and realistic digital human modeling and free-viewpoint rendering using portable devices is a major challenge in this field.

[0004] One current technology is a depth-based human body rendering technique based on depth information, as described in Yu T et al.'s paper, "Function4d: Real-time human volumetric capture from very sparse consumer RGBD sensors." This technique utilizes sparse multi-view RGBD sensors to acquire RGBD images from multiple perspectives. It then uses dynamic sliding fusion technology to merge RGBD images from adjacent frames, obtaining a voxelized fusion result. This result is then re-rendered to create an RGBD image with the same viewpoint as the original. The re-rendered multi-view RGBD image is then processed by a multi-view image encoder, feature aggregation, and geometry and color decoders to ultimately reconstruct the human body completely. In scenarios requiring rendering of new perspectives, traditional rendering models such as BRDF are used to acquire human images from any viewpoint. The drawbacks of this technique are: the RGBD sensors required are not yet widely available, and multi-view image acquisition requires rigorous camera calibration, a level of expertise beyond the reach of most users. Furthermore, this technique generates a displayed 3D human body mesh, not an end-to-end new viewpoint human body image, consuming additional storage space without considering physical interaction.

[0005] The second existing technology is a novel human perspective synthesis technique based on neural radiation fields, as described in Jiang T et al.'s paper, "Instantavatar: Learning avatars from monocular video in 60 seconds." This technique represents the 3D human body as an implicit neural radiation field. Under the observation posture, light projection and point sampling are performed according to the camera's viewpoint, and the sampled points are mapped to the standard posture. For sampled points located outside the human body, this technique designs a blank space skipping strategy, maintaining a common occupancy grid for all observation frames in the standard space. This common occupancy grid is updated by integrating the medium probability density of sampled points from different observation frames. During inference, sampled points far from the human body surface are filtered out based on this occupancy grid. Finally, based on the coordinates of the sampled points under the standard posture, point features at different resolutions are retrieved through hash lookup, stitched together, and input into the neural radiation field to calculate the medium probability density and color of the points, rendering a human image from any viewpoint. The drawbacks of this technique are: it uses a multilayer perceptron as the storage medium in the neural radiation field, requiring dense point sampling during image rendering; and the time overhead for forward computation during training and inference is relatively large, making real-time performance impossible. Furthermore, using a continuous implicit neural radiation field to render images often results in blurring in edge regions with drastic color changes.

[0006] The third existing technology is a novel human body perspective synthesis technique based on 3D Gaussian mixture and sputtering rendering, as described in Kocabas M et al.'s "Hugs: Human Gaussian Splats." This technique samples 3D point clouds from a parametric template of the human body in a standard pose, establishes a learnable 3D Gaussian distribution for each point cloud, and stores attributes such as the mean position, covariance matrix, scaling vector, and spherical harmonic coefficients of the Gaussian distribution. The Gaussian position is sampled through a feature triplane to obtain features, which are then processed by a multilayer perceptron to output 3D Gaussian attributes and linear skinning weights. The 3D Gaussian distribution in the standard pose is transformed by linear skinning weights to obtain the 3D Gaussian distribution in the observation pose. Given an arbitrary viewpoint, a differentiable GPU parallel rasterization strategy is executed to render the human body image. The disadvantages of this technique are: it does not consider the combination of 3D Gaussian with other geometric information, as 3D Gaussian, as a special display point cloud, can be associated with other attributes with display features; furthermore, it does not consider the different detail granularities in different regions of the human body, resulting in a lack of high-frequency details. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing methods and propose a local information-guided human free-viewpoint rendering method and system. The main problems addressed by this invention are: 1) How to solve the problem that depth-information-based human free-viewpoint rendering technology requires a professional RGBD camera, which is difficult to acquire, and the generated 3D human body mesh has high storage requirements, hindering large-scale popularization and application; 2) How to solve the problem that neural radiation field-based human free-viewpoint synthesis technology uses a multilayer perceptron as the storage medium for the radiation field, requiring dense point sampling during image rendering, resulting in slow training and inference speeds and high time costs; 3) How to solve the problem that 3D Gaussian mixture and sputtering rendering-based human free-viewpoint synthesis technology does not consider the relationship and combination of 3D Gaussian with other geometric properties, and the fitting effect for high-frequency details is not ideal.

[0008] To address the aforementioned problems, this invention proposes a local information-guided free-viewpoint rendering method for the human body, the method comprising:

[0009] Input a single-view human video dataset. For each frame in the video in the dataset, fit the corresponding Skinned Multi-Person Linear (SMPL) model. At the same time, remove the background in the video frame according to the human mask provided in the dataset. Under standard pose, randomly sample the surface of the SMPL model to obtain three-dimensional point cloud data.

[0010] Based on the three-dimensional point cloud data, a three-dimensional Gaussian distribution is constructed for each point, and the properties of the Gaussian distribution are initialized. The position of the three-dimensional Gaussian distribution is passed into a multilayer perceptron MLP_1, and the offset of the stored parameters in the Gaussian distribution is obtained through training. This offset is then applied to the initial three-dimensional Gaussian distribution to obtain the three-dimensional Gaussian position after deformation.

[0011] The deformed 3D Gaussian position and the joint rotation matrix of the SMPL model are used to predict the linear blend skinning (LBS) transformation weights from the standard pose to the observation frame pose through a multilayer perceptron MLP_2, and the LBS transformation is performed to obtain the Gaussian distribution of the observation frame pose.

[0012] The Gaussian color is decomposed into specular and diffuse components, which are estimated separately based on the normal vector, and then combined to obtain the final predicted color.

[0013] Based on the viewpoint parameters, a 3D Gaussian is projected onto a 2D plane, the 2D Gaussian is rasterized, a 2D image of the human body is rendered, and a densification threshold is dynamically calculated from the neighborhood variance of the Gaussian's normal vector according to the local perception adaptive density control strategy, and the density of the Gaussian is controlled.

[0014] Preferably, the SMPL model is as follows:

[0015] The SMPL model is a vertex-skinned model that can accurately represent various body shapes in natural human poses. The model's parameters include shape and pose parameters, a standard pose template, and predefined hybrid skin weights. The shape parameters describe the approximate shape of the human body. The standard pose refers to the human body with its arms and legs outstretched, forming a "T" shape. The pose parameters in the SMPL model describe the degree of rotation and translation of the human body's joints relative to the standard pose. The standard pose template is the human body SMPL model in the standard pose, consisting of several points and triangular patches formed by connecting the points.

[0016] Preferably, based on the 3D point cloud data, a 3D Gaussian distribution is constructed for each point, and the attributes of the Gaussian distribution are initialized. The position of the 3D Gaussian distribution is input into a multilayer perceptron (MLP_1) to train and obtain the offset of the stored parameters in the Gaussian distribution. This offset is then applied to the initial 3D Gaussian distribution to obtain the deformed 3D Gaussian position. Specifically:

[0017] The mathematical definition of the three-dimensional Gaussian distribution is as follows:

[0018]

[0019] Where x is a point in the point cloud, μ is the center of the 3D Gaussian distribution, i.e., the position mean, and Σ is the covariance matrix, used to represent the size, shape, and orientation of the 3D Gaussian distribution, calculated from the scaling vector s and the rotation matrix R. The 3D Gaussian distribution stores the following parameters: position μ, rotation quaternion q, scaling vector s, normal vector n, and opacity parameter α. s It is a specular reflection color characteristic, f d This refers to diffuse color characteristics. The rotation quaternion is initialized to q = [1,0,0,0], which is equivalent to the identity matrix in the rotation matrix, meaning no rotation is performed. The scaling vector s is initialized to the logarithm of the distances from the point to its three nearest neighbors. The opacity parameter α is initialized to 1, and f... s and f d Initialize it as a vector of all zeros, where the rotation quaternion q = [w, x, y, z] is transformed into a rotation matrix R by the following formula:

[0020]

[0021] Based on the three-dimensional point cloud data, the three-dimensional Gaussian is initialized, wherein the position μ is initialized to the coordinates of the point cloud, and the normal vector n is initialized to the surface normal vector of the SMPL triangle patch where the sampling point is located.

[0022] Train a multilayer perceptron (MLP_1) to learn the offsets of three-dimensional Gaussian attributes, specifically:

[0023] (Δμ,Δq,Δs,Δn)=MLP_1(μ)

[0024] The output consists of the offsets of position μ, rotation quaternion q, scaling vector s, and normal vector n, i.e., the offsets of the stored parameters, used to fit the offset of clothing relative to the human body surface. MLP_1 consists of one input layer, two hidden layers, and one output layer, where the width of each hidden layer is 128. The predicted offsets are applied to the initialized 3D Gaussian distribution using the following formula:

[0025]

[0026] The · operation between q and Δq is equivalent to converting them into matrices and then performing matrix multiplication. e is an exponential function. The predicted normal vector offset needs to be converted into a rotation matrix and then multiplied by the original normal vector n.

[0027] Preferably, the deformed 3D Gaussian position and the joint rotation matrix of the SMPL model are used to predict the linear blend skinning (LBS) transformation weights from the standard pose to the observation frame pose using a multilayer perceptron (MLP_2), and then LBS transformation is performed to obtain the Gaussian distribution of the observation frame pose, specifically:

[0028] The initialization of the 3D Gaussian and the fitting of the clothing deformation are both completed in the standard pose space. The transformation from the standard pose space to the observation pose space is achieved through the linear blending skin (LBS) transformation. The LBS transformation regards the changes of the human body surface vertices caused by the action as being driven by the skeleton. The trajectory changes of the points are obtained by weighted calculation of the transformation of the joints on the skeleton. That is, the position of each vertex is the weighted average of the vertex positions after the joint rigid skin that affects it.

[0029] The SMPL model predicted from the observed frames includes pose parameters. The human joint bone rotation matrix is ​​calculated from these pose parameters. The deformed 3D Gaussian position and the SMPL joint rotation matrix are then fed into a multilayer perceptron (MLP_2) to predict the weights for the LBS transform.

[0030]

[0031] in, It is a skeletal rotation matrix, where k represents the k-th joint, K represents the total number of joints, and w k The transformation of the k-th joint relative to μ dThe weights are obtained by MLP_2, which consists of one input layer, three hidden layers, and one output layer. The width of the hidden layers is 128. After obtaining the 3D Gaussian LBS weights through MLP_2, the LBS transformation is performed according to the following formula:

[0032]

[0033] Where T is the transformation matrix, obtained by multiplying the bone weights predicted by MLP_2 by the bone transformation matrix calculated from SMPL, and R... d From rotation quaternion q d The rotation matrix is ​​calculated in the middle.

[0034] Preferably, the step of decomposing the Gaussian color into specular and diffuse components, estimating them separately based on the normal vector, and combining them to obtain the final predicted color specifically involves:

[0035] The Gaussian color is decomposed into specular and diffuse components, and the normal vector is used as one of the bases for predicting the two color components, as follows:

[0036]

[0037] Among them, f s It is the specular reflection color feature stored in Gaussian, f d d is the diffuse color feature stored in Gaussian, d is the camera viewpoint, and c is the diffuse color feature stored in Gaussian. s It is the specular reflection component, c d This refers to the diffuse reflection component. MLP_3 and MLP_4 have the same structure, both consisting of an input layer, a hidden layer, and an output layer. The width of the hidden layer is 64. The two components are then added together:

[0038] c = c s +c d

[0039] This yields the final predicted color c.

[0040] Preferably, the step of projecting a 3D Gaussian onto a 2D plane based on the viewpoint parameters, rasterizing the 2D Gaussian, rendering a 2D human body image, and dynamically calculating the densification threshold based on the neighborhood variance of the Gaussian's normal vector according to a local perception adaptive density control strategy, and controlling the density of the Gaussian, specifically involves:

[0041] Based on the viewpoint parameters, the three-dimensional Gaussian is projected onto a two-dimensional plane to obtain the corresponding two-dimensional Gaussian, as follows:

[0042]

[0043] Where i represents the i-th Gaussian distribution; αi μ represents the opacity parameter of the i-th Gaussian. i Let Σ′ represent the two-dimensional mean coordinates of the i-th 3D Gaussian vector after projection. i =(JWΣ i W T J T ) 1:2,1:2 is the two-dimensional covariance matrix obtained after projecting the i-th three-dimensional Gaussian, where W is the transformation matrix of the camera from the world coordinate system to the camera coordinate system, J is the Jacobian matrix of the perspective projection transformation; p is the coordinate of each pixel, and e is the exponential function.

[0044] The final RGB color value of each pixel in the rendered image is obtained through the alpha blending method, and its calculation formula is as follows:

[0045]

[0046] Among them, c i It is the i-th final predicted color, f i It is the probability value of the position of pixel p after projection with respect to the i-th two-dimensional Gaussian distribution. The final result C after α mixing is the RGB color value of pixel p.

[0047] A local perceptual adaptive density control strategy is applied to the 3D Gaussian to dynamically adjust the number and distribution of the 3D Gaussian.

[0048] The normal vector is viewed as a cue for the details of the human body surface, and the local variance of the normal vector is seen as a measure of Gaussian local similarity. During the splitting and cloning process, the Gaussian densification threshold is dynamically calculated, as follows:

[0049]

[0050] Here, var is the variance calculation operation. Based on the knnk nearest neighbor algorithm, it finds the k Gaussians that are closest to the 3D Gaussian at its location and calculates the variance of their normal vectors, var(n). d ), where a, β and b are constants. According to the local perception adaptive density strategy, the calculated splitting and cloning thresholds are lower, making densification more likely, which makes the number and distribution of Gaussians more reasonable, and further makes the final rendered image have richer high-frequency details.

[0051] The loss function for the training process is as follows:

[0052]

[0053] in, It is the mean error between each pixel in the rendered image and the corresponding pixel in the real image. This is the structural similarity loss between the rendered image and the real image, measuring the degree of similarity between the two images. Using ECON, the corresponding normal map is estimated from the observed frame as a supervision signal. Simultaneously, the RGB color value C of the pixel p is replaced with a Gaussian normal vector to render the normal map. The same loss as the image is calculated on the two normal maps. and λ1 and λ2 are the loss weights.

[0054] Accordingly, the present invention also provides a human free-viewpoint rendering system guided by local information, comprising:

[0055] The data acquisition unit is used to input a single-view human video dataset. For each frame in the video in the dataset, the corresponding Skinned Multi-Person Linear (SMPL) model is fitted. At the same time, the background in the video frame is removed according to the human mask provided in the dataset. Under standard pose, the surface of the SMPL model is randomly sampled to obtain three-dimensional point cloud data.

[0056] The parameter deformation unit is used to construct a three-dimensional Gaussian distribution for each point based on the three-dimensional point cloud data, initialize the properties of the Gaussian distribution, input the position of the three-dimensional Gaussian into the multilayer perceptron MLP_1, train to obtain the offset of the stored parameters in the Gaussian distribution, and apply it to the initial three-dimensional Gaussian distribution to obtain the three-dimensional Gaussian position after deformation.

[0057] The weight transformation unit is used to predict the linear blend skinning (LBS) transformation weights from the standard pose to the observation frame pose through the multilayer perceptron MLP_2, based on the deformed 3D Gaussian position and the joint rotation matrix of the SMPL model, and to perform LBS transformation to obtain the Gaussian distribution of the observation frame pose.

[0058] The color prediction unit is used to decompose Gaussian color into specular reflection component and diffuse reflection component, and estimate them separately based on the normal vector, and combine them to obtain the final predicted color.

[0059] The image rendering unit is used to project a 3D Gaussian onto a 2D plane according to the viewpoint parameters, rasterize the 2D Gaussian, render a 2D image of the human body, and dynamically calculate the densification threshold based on the neighborhood variance of the Gaussian's normal vector according to the local perception adaptive density control strategy, and control the density of the Gaussian.

[0060] Implementing this invention has the following beneficial effects:

[0061] This solution adds normal vectors to the properties of the 3D Gaussian, as a form of geometric information. This can guide the 3D Gaussian to pay more attention to the geometry of the human body surface during the network learning process, thereby assisting in the learning of other properties of the 3D Gaussian.

[0062] This solution takes into account that different parts of the human body have different levels of detail. Simply using a fixed threshold to control the Gaussian density ignores local information. Therefore, a locally perceptive adaptive density control strategy is used to dynamically calculate the densification threshold of each Gaussian, so that the Gaussian can fit more details of the human body surface.

[0063] This scheme combines traditional computer graphics knowledge to divide color into specular reflection and diffuse reflection components. Considering that the factors affecting the two are different, they are predicted separately. At the same time, the normal vector also plays a guiding role in color calculation and is therefore used as input to the color prediction model. In addition, the color decomposition strategy of this scheme can help the 3D Gaussian to better fit the human body surface color in areas with drastic color changes. Attached Figure Description

[0064] Figure 1 This is a flowchart of a local information-guided free-viewpoint rendering method for the human body according to an embodiment of the present invention;

[0065] Figure 2 This is a structural diagram of a human free-viewpoint rendering system guided by local information according to an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Figure 1 This is a flowchart of a local information-guided free-viewpoint rendering method for the human body according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0068] S1 is a single-view human video. For each frame in the video, the corresponding SMPL model is fitted, and the background in the video frame is removed according to the human mask provided in the dataset.

[0069] S2, In the standard pose space, the surface of the SMPL model, i.e. the standard pose template, is sampled to obtain three-dimensional point cloud data;

[0070] S3, Based on the point cloud data in step S2, construct a three-dimensional Gaussian distribution for each point and initialize the properties of the Gaussian distribution;

[0071] S4. The position of the three-dimensional Gaussian is fed into the multilayer perceptron MLP_1 to train and obtain the offset of the stored parameters in the Gaussian distribution, and then applied to the initial three-dimensional Gaussian distribution in step S3.

[0072] S5. The three-dimensional Gaussian position and SMPL joint rotation matrix after deformation are used by the multilayer perceptron MLP_2 to predict the LBS transformation weight from the standard pose space to the observation frame pose space, and then the LBS transformation is performed.

[0073] S6. For Gaussian color estimation, the color is decomposed into specular reflection component and diffuse reflection component, and estimated separately based on the normal vector, and then combined to obtain the final color.

[0074] S7 projects a three-dimensional Gaussian onto a two-dimensional plane based on camera parameters from a certain perspective.

[0075] S8 rasterizes a 2D Gaussian image to render a 2D human body image.

[0076] S9, based on the local perception adaptive density control strategy, dynamically calculates the densification threshold from the neighborhood variance of the Gaussian normal vector, and performs density control on the Gaussian.

[0077] Step S1 is as follows:

[0078] S1-1: The Skinned Multi-Person Linear (SMPL) model is a vertex-skinned model that can accurately represent various body shapes in natural human poses. The model's parameters include shape and pose parameters, a standard pose template, and predefined hybrid skinning weights. The shape parameters describe the approximate shape of the human body. The standard pose refers to a human body with arms and legs outstretched, forming a "T" shape. The pose parameters in SMPL describe the rotation and translation of the human body's joints relative to the standard pose. The standard pose template is the human body SMPL model in the standard pose, consisting of several points and triangular patches formed by connecting these points. Mature methods such as HMR2.0 can fit SMPL models from two-dimensional images.

[0079] Step S2 is as follows:

[0080] S2-1: The sampling method is random sampling to ensure that the sampled points uniformly cover the human body surface.

[0081] Step S3 is as follows:

[0082] S3-1: The mathematical definition of the three-dimensional Gaussian distribution is as follows:

[0083]

[0084] Here, μ is the center of the 3D Gaussian distribution, i.e., the position mean, and Σ is the covariance matrix, used to represent the size, shape, and orientation of the 3D Gaussian, calculated from the scaling vector s and the rotation matrix R. The 3D Gaussian distribution stores the following parameters: position μ, rotation quaternion q, scaling vector s, normal vector n, opacity parameter α, and color feature f. s and f d The rotation quaternion is initialized to q = [1,0,0,0], which is equivalent to the identity matrix in the rotation matrix, meaning no rotation is performed; the scaling vector s is initialized to the logarithm of the distances from the point to its three nearest neighbors; the opacity α is initialized to 1, and f s and f d Initialize it as a vector of all zeros. The rotation quaternion q = [w, x, y, z] can be transformed into a rotation matrix R using the following formula:

[0085]

[0086] S3-2: Based on the point cloud in S2, initialize the 3D Gaussian, where μ is initialized to the coordinates of the point cloud and n is initialized to the surface normal vector of the SMPL triangle patch where the sampling point is located.

[0087] Step S4 is as follows:

[0088] S4-1: The SMPL model does not include human clothing; directly fitting a single-view human body using a 3D Gaussian model obtained from SMPL introduces bias. Therefore, a multilayer perceptron (MLP_1) is trained to learn the offset of the 3D Gaussian attributes:

[0089] (Δμ,Δq,Δs,Δn)=MLP_1(μ)

[0090] The output includes the offsets of position, rotation quaternion, scaling vector, and normal vector, used to fit the offset of clothing relative to the human body surface. MLP_1 consists of one input layer, two hidden layers, and one output layer, where each hidden layer has a width of 128. The predicted offsets are applied to an initialized 3D Gaussian distribution using the following formula:

[0091]

[0092] The · operation between q and Δq is equivalent to converting them into matrices and then performing matrix multiplication. e is an exponential function, and the predicted normal vector offset needs to be converted into a rotation matrix and then multiplied by the original normal vector n.

[0093] Step S5 is as follows:

[0094] S5-1: The initialization of the 3D Gaussian model and the fitting of clothing deformation are both completed in the standard pose space. A transformation from the standard pose space to the observation pose space is required through Linear Blend Skinning (LBS). The LBS transformation treats the changes in the vertices of the human body surface caused by movement as being driven by the skeleton. The trajectory changes of the points can be calculated from the weighted average of the transformations of the joints on the skeleton. That is, the position of each vertex is a weighted average of the vertex positions after the joints are rigidly skinned, affecting it.

[0095] S5-2: The SMPL model predicted from the observed frames contains pose parameters, from which the rotation matrix of the human joint bones can be calculated. The deformed 3D Gaussian positions and the SMPL joint rotation matrices are then fed into a multilayer perceptron (MLP_2) to predict the weights for the LBS transform.

[0096]

[0097] in, This is the skeletal transformation matrix, where k represents the k-th joint, K represents the total number of joints, and w k The transformation of the k-th joint relative to μ d The weights are determined by the MLP_2 layer, which consists of one input layer, three hidden layers, and one output layer, with the hidden layers having a width of 128. After obtaining the 3D Gaussian LBS weights through MLP_2, the LBS transformation is performed according to the following formula:

[0098]

[0099] Among them, R d From rotation quaternion q d The rotation matrix is ​​calculated in the middle.

[0100] Step S6 is as follows:

[0101] S6-1: In traditional computer graphics, color is formed by the reflection of light, which can be divided into specular reflection and diffuse reflection. Based on this, this scheme also decomposes color into specular and diffuse components. In the traditional color calculation model of computer graphics, the normal vector also participates in the color calculation; therefore, the normal vector also contributes to the color calculation and serves as one of the bases for predicting the two color components.

[0102]

[0103] Among them, f s It is the specular reflection color feature stored in Gaussian, f dd is the diffuse color feature stored in Gaussian, d is the camera viewpoint, and c is the diffuse color feature stored in Gaussian. s It is the specular reflection color component, c d This is the diffuse color component. Because specular reflection is view-dependent while diffuse reflection is view-independent, d is used as the input to MLP_3 but not to MLP_4. MLP_3 and MLP_4 have the same structure, consisting of an input layer, a hidden layer, and an output layer, with the hidden layer having a width of 64. The final predicted color is obtained by adding the two components together.

[0104] c = c s +c d .

[0105] Step S7 is as follows:

[0106] S7-1: The calculation of a two-dimensional Gaussian is as follows:

[0107]

[0108] Where i represents the i-th Gaussian distribution; α i μ represents the opacity parameter of the i-th Gaussian. i Σ′ represents the two-dimensional mean coordinates obtained by projecting the i-th three-dimensional Gaussian vector; i =(JWΣ i W T J T ) 1:2,1:2 is the two-dimensional covariance matrix obtained after projecting the i-th three-dimensional Gaussian, where W is the transformation matrix of the camera from the world coordinate system to the camera coordinate system, J is the Jacobian matrix of the perspective projection transformation; p is the coordinate of each pixel; and e is the exponential function.

[0109] Step S8 is as follows:

[0110] S8-1: The final RGB color value of each pixel in the rendered image is obtained using the α blending method, and the calculation formula is as follows:

[0111]

[0112] c i It is the color of the i-th Gaussian digit calculated based on S6, f i It is the probability value of the position of pixel p after projection calculated in S7 according to the i-th two-dimensional Gaussian distribution. Finally, after α mixing, the result C is the RGB color value of pixel p.

[0113] Step S9 is as follows:

[0114] S9-1: Implement a local perception adaptive density control strategy for the three-dimensional Gaussian to dynamically adjust the number and distribution of the three-dimensional Gaussian.

[0115] During the Gaussian optimization process, every 100 iterations, Gaussian scalars are split, cloned, or pruned to adjust their number. The granularity of detail varies across different parts of the human body, thus requiring a different number of 3D Gaussian scalars for representation. This process leverages local information. Normal vectors can be seen as cues for surface details, and their local variance can be considered a measure of local Gaussian similarity. Based on this, the densification threshold of the Gaussian scalars is dynamically calculated during splitting and cloning.

[0116]

[0117] Here, var is the variance calculation operation. Based on the knnk nearest neighbor algorithm, it finds the k Gaussians that are closest to the 3D Gaussian at its location and calculates the variance of their normal vectors, var(n). d ), where a, β, and b are constants, a = 1.0, b = 0.0002, and b = 0.00003. For Gaussians with large variance in their neighborhood normal vectors, it can be assumed that their similarity to neighborhood Gaussians is low. The current number of Gaussians is insufficient to fit the human body part. Therefore, according to the local perception adaptive density strategy, the calculated splitting and cloning thresholds are also lower, making it more likely to be densified, thus making the number and distribution of Gaussians more reasonable, and further making the final rendered image have richer high-frequency details;

[0118] S9-2: The loss function for the training process is as follows:

[0119]

[0120] in, It is the average error between each pixel in the rendered image and the corresponding pixel in the real image; This is the structural similarity loss between the rendered image and the real image, measuring the degree of similarity between the two images. Using ECON, the corresponding normal map is estimated from the observed frames as a supervision signal. Simultaneously, replacing the colors in S8 with Gaussian normal vectors allows for the rendering of the normal map. The same loss is calculated for both, similar to the loss for the image. and λ1 and λ2 are the loss weights.

[0121] Accordingly, the present invention also provides a human free-viewpoint rendering system guided by local information, such as... Figure 2 As shown, it includes:

[0122] Data acquisition unit 1 is used to input a single-view human video dataset. For each frame in the video in the dataset, the corresponding Skinned Multi-Person Linear (SMPL) model is fitted. At the same time, the background in the video frame is removed according to the human mask provided in the dataset. Under standard pose, the surface of the SMPL model is randomly sampled to obtain three-dimensional point cloud data.

[0123] The parameter deformation unit 2 is used to construct a three-dimensional Gaussian distribution for each point based on the three-dimensional point cloud data, initialize the properties of the Gaussian distribution, input the position of the three-dimensional Gaussian into the multilayer perceptron MLP_1, train to obtain the offset of the stored parameters in the Gaussian distribution, and apply it to the initial three-dimensional Gaussian distribution to obtain the three-dimensional Gaussian position after deformation.

[0124] The weight transformation unit 3 is used to predict the linear blend skinning (LBS) transformation weights from the standard pose to the observation frame pose through the multilayer perceptron MLP_2, based on the deformed 3D Gaussian position and the joint rotation matrix of the SMPL model, and to perform LBS transformation to obtain the Gaussian distribution of the observation frame pose.

[0125] Color prediction unit 4 is used to decompose Gaussian color into specular reflection component and diffuse reflection component, and estimate them separately based on the normal vector, and combine them to obtain the final predicted color;

[0126] Image rendering unit 5 is used to project a three-dimensional Gaussian onto a two-dimensional plane according to the viewpoint parameters, rasterize the two-dimensional Gaussian, render a two-dimensional human body image, and dynamically calculate the densification threshold based on the neighborhood variance of the Gaussian's normal vector according to the local perception adaptive density control strategy, and perform density control on the Gaussian.

[0127] Therefore, this scheme adds normal vectors to the attributes of the 3D Gaussian model as a form of geometric information. This guides the 3D Gaussian model to focus more on the geometry of the human body surface during network learning, thereby assisting in the learning of other attributes of the 3D Gaussian model. Considering that different parts of the human body have different levels of detail, simply using a fixed threshold to control the Gaussian density ignores local information. Therefore, a locally perceptive adaptive density control strategy is used to dynamically calculate the densification threshold of each Gaussian model, enabling it to fit more details of the human body surface. This scheme combines traditional computer graphics knowledge to divide color into specular and diffuse components, and considering the different factors affecting them, they are predicted separately. Simultaneously, the normal vectors also provide clues for color calculation and are therefore used as input to the color prediction model. Furthermore, the color decomposition strategy of this scheme helps the 3D Gaussian model better fit the human body surface color in areas of dramatic color change.

[0128] The above provides a detailed description of the human free-viewpoint rendering method and system guided by local information provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for rendering human free-viewpoint based on local information, characterized in that, The method includes: Input a single-view human video dataset. For each frame in the video in the dataset, fit the corresponding skin multi-human linear SMPL model. At the same time, remove the background in the video frame according to the human mask provided in the dataset. Under standard pose, randomly sample the surface of the SMPL model to obtain three-dimensional point cloud data. Based on the three-dimensional point cloud data, a three-dimensional Gaussian distribution is constructed for each point, and the properties of the Gaussian distribution are initialized. The position of the three-dimensional Gaussian distribution is passed into a multilayer perceptron MLP_1, and the offset of the stored parameters in the Gaussian distribution is obtained through training. This offset is then applied to the initial three-dimensional Gaussian distribution to obtain the three-dimensional Gaussian position after deformation. The deformed 3D Gaussian position and the joint rotation matrix of the SMPL model are used to predict the linear hybrid skin LBS transformation weight from the standard pose to the observation frame pose through a multilayer perceptron MLP_2, and then LBS transformation is performed to obtain the Gaussian distribution of the observation frame pose. The Gaussian color is decomposed into specular and diffuse components, which are estimated separately based on the normal vector, and then combined to obtain the final predicted color. Based on the viewpoint parameters, the three-dimensional Gaussian is projected onto the two-dimensional plane, the two-dimensional Gaussian is rasterized, and a two-dimensional human body image is rendered. Based on the local perception adaptive density control strategy, the densification threshold is dynamically calculated from the neighborhood variance of the Gaussian's normal vector, and the density of the Gaussian is controlled. Specifically, based on the 3D point cloud data, a 3D Gaussian distribution is constructed for each point, and the attributes of the Gaussian distribution are initialized. The position of the 3D Gaussian distribution is then fed into a multilayer perceptron (MLP_1) to train and obtain the offset of the stored parameters in the Gaussian distribution. This offset is then applied to the initial 3D Gaussian distribution to obtain the deformed 3D Gaussian position, including: The following parameters are stored in the three-dimensional Gaussian distribution: position Rotation quaternion q, scaling vector s, normal vector n, opacity parameter Specular reflection color characteristics Diffuse color characteristics The rotation quaternion is initialized to q=[1,0,0,0], which is equivalent to the identity matrix in the rotation matrix, i.e., no rotation is performed; the scaling vector s is initialized to the logarithm of the distances from the point to its three nearest neighbors; the opacity parameter... Initialize to 1, and Initialize as a vector of all zeros; Based on the aforementioned 3D point cloud data, a 3D Gaussian model is initialized, wherein the position... Initialize the coordinates of the point cloud, and initialize the normal vector n to the face normal vector of the SMPL triangle patch where the sampling point is located; Train a multilayer perceptron (MLP_1) to learn the offsets of three-dimensional Gaussian attributes, specifically: ； The output is the position. The offsets of the rotation quaternion q, scaling vector s, and normal vector n, i.e., the offsets of the stored parameters, are used to fit the offset of the clothing relative to the human body surface. MLP_1 consists of one input layer, two hidden layers, and one output layer, where the width of each hidden layer is 128. The predicted offsets are applied to the initialized 3D Gaussian distribution using the following formula: ； in, and Between The operation is equivalent to converting the two into matrices and then performing matrix multiplication. Since it is an exponential function, the predicted normal vector offset needs to be converted into a rotation matrix and then multiplied by the original normal vector n.

2. The human body free viewpoint rendering method guided by local information as described in claim 1, characterized in that, The deformed 3D Gaussian position and the joint rotation matrix of the SMPL model are used by a multilayer perceptron (MLP_2) to predict the linear hybrid skinning (LBS) transformation weights from the standard pose to the observed frame pose. The LBS transformation is then performed to obtain the Gaussian distribution of the observed frame pose, specifically: The initialization of the 3D Gaussian and the fitting of the clothing deformation are both completed in the standard pose space. The transformation from the standard pose space to the observation pose space is achieved through the linear blending skin (LBS) transformation. The LBS transformation regards the changes of the human body surface vertices caused by the action as being driven by the skeleton. The trajectory changes of the points are obtained by weighted calculation of the transformation of the joints on the skeleton. That is, the position of each vertex is the weighted average of the vertex positions after the joint rigid skin that affects it. The SMPL model predicted from the observed frames includes pose parameters. The human joint bone rotation matrix is ​​calculated from these pose parameters. The deformed 3D Gaussian position and the SMPL joint rotation matrix are then fed into a multilayer perceptron (MLP_2) to predict the weights for the LBS transform. ； in, It is a skeletal rotation matrix. This represents the k-th joint, where K represents the total number of joints. The transformation of the k-th joint is relative to The weights are obtained by MLP_2, which consists of one input layer, three hidden layers, and one output layer. The width of the hidden layers is 128. After obtaining the 3D Gaussian LBS weights through MLP_2, the LBS transformation is performed according to the following formula: ； Where T is the transformation matrix, obtained by multiplying the bone weights predicted by MLP_2 by the bone transformation matrix calculated from SMPL, and R... d From rotation quaternion q d The rotation matrix is ​​calculated in the middle.

3. The human body free viewpoint rendering method guided by local information as described in claim 1, characterized in that, The process of decomposing Gaussian color into specular and diffuse components, estimating them separately based on the normal vector, and combining them to obtain the final predicted color is as follows: The Gaussian color is decomposed into specular and diffuse components, and the normal vector is used as one of the bases for predicting the two color components, as follows: ； in, These are the specular reflection color features stored in Gaussian. It is the diffuse color feature stored in Gaussian, where d is the camera viewpoint. It is the specular reflection component, This refers to the diffuse reflection component. MLP_3 and MLP_4 have the same structure, both consisting of an input layer, a hidden layer, and an output layer. The width of the hidden layer is 64. The two components are then added together: ； This yields the final predicted color. .

4. A human free-viewpoint rendering system guided by local information, characterized in that, The system includes: The data acquisition unit is used to input a single-view human video dataset. For each frame in the video in the dataset, the corresponding skin multi-human linear SMPL model is fitted. At the same time, the background in the video frame is removed according to the human mask provided in the dataset. Under standard pose, the surface of the SMPL model is randomly sampled to obtain three-dimensional point cloud data. The parameter deformation unit is used to construct a three-dimensional Gaussian distribution for each point based on the three-dimensional point cloud data, initialize the properties of the Gaussian distribution, input the position of the three-dimensional Gaussian into the multilayer perceptron MLP_1, train to obtain the offset of the stored parameters in the Gaussian distribution, and apply it to the initial three-dimensional Gaussian distribution to obtain the three-dimensional Gaussian position after deformation. The weight transformation unit is used to predict the linear hybrid skin LBS transformation weights from the standard pose to the observation frame pose through the multilayer perceptron MLP_2, based on the deformed 3D Gaussian position and the joint rotation matrix of the SMPL model, and to perform LBS transformation to obtain the Gaussian distribution of the observation frame pose. The color prediction unit is used to decompose Gaussian color into specular reflection component and diffuse reflection component, and estimate them separately based on the normal vector, and combine them to obtain the final predicted color. The image rendering unit is used to project a 3D Gaussian onto a 2D plane according to the viewpoint parameters, rasterize the 2D Gaussian, render a 2D image of the human body, and dynamically calculate the densification threshold based on the neighborhood variance of the Gaussian's normal vector according to the local perception adaptive density control strategy, and control the density of the Gaussian. Specifically, based on the 3D point cloud data, a 3D Gaussian distribution is constructed for each point, and the attributes of the Gaussian distribution are initialized. The position of the 3D Gaussian distribution is then fed into a multilayer perceptron (MLP_1) to train and obtain the offset of the stored parameters in the Gaussian distribution. This offset is then applied to the initial 3D Gaussian distribution to obtain the deformed 3D Gaussian position, including: The following parameters are stored in the three-dimensional Gaussian distribution: position Rotation quaternion q, scaling vector s, normal vector n, opacity parameter Specular reflection color characteristics Diffuse color characteristics The rotation quaternion is initialized to q=[1,0,0,0], which is equivalent to the identity matrix in the rotation matrix, i.e., no rotation is performed; the scaling vector s is initialized to the logarithm of the distances from the point to its three nearest neighbors; the opacity parameter... Initialize to 1, and Initialize as a vector of all zeros; Based on the aforementioned 3D point cloud data, a 3D Gaussian model is initialized, wherein the position... Initialize the coordinates of the point cloud, and initialize the normal vector n to the face normal vector of the SMPL triangle patch where the sampling point is located; Train a multilayer perceptron (MLP_1) to learn the offsets of three-dimensional Gaussian attributes, specifically: ； The output is the position. The offsets of the rotation quaternion q, scaling vector s, and normal vector n, i.e., the offsets of the stored parameters, are used to fit the offset of the clothing relative to the human body surface. MLP_1 consists of one input layer, two hidden layers, and one output layer, where the width of each hidden layer is 128. The predicted offsets are applied to the initialized 3D Gaussian distribution using the following formula: ； in, and Between The operation is equivalent to converting the two into matrices and then performing matrix multiplication. Since it is an exponential function, the predicted normal vector offset needs to be converted into a rotation matrix and then multiplied by the original normal vector n.

5. The human body free-viewpoint rendering system guided by local information as described in claim 4, characterized in that, The weight transformation unit is used to predict the linear hybrid skinning (LBS) transformation weights from the standard pose to the observed frame pose using a multilayer perceptron (MLP_2) through the deformed 3D Gaussian position and SMPL joint rotation matrix, and then perform the LBS transformation to obtain the Gaussian distribution of the observed frame pose. Specifically: The initialization of the 3D Gaussian and the fitting of the clothing deformation are both completed in the standard pose space. The transformation from the standard pose space to the observation pose space is achieved through the linear blending skin (LBS) transformation. The LBS transformation regards the changes of the human body surface vertices caused by the action as being driven by the skeleton. The trajectory changes of the points are obtained by weighted calculation of the transformation of the joints on the skeleton. That is, the position of each vertex is the weighted average of the vertex positions after the joint rigid skin that affects it. The SMPL model predicted from the observed frames includes pose parameters. The human joint bone rotation matrix is ​​calculated from these pose parameters. The deformed 3D Gaussian position and the SMPL joint rotation matrix are then fed into a multilayer perceptron (MLP_2) to predict the weights for the LBS transform. ； in, It is a skeletal rotation matrix. This represents the k-th joint, where K represents the total number of joints. The transformation of the k-th joint is relative to The weights are obtained by MLP_2, which consists of one input layer, three hidden layers, and one output layer. The width of the hidden layers is 128. After obtaining the 3D Gaussian LBS weights through MLP_2, the LBS transformation is performed according to the following formula: ； Where T is the transformation matrix, obtained by multiplying the bone weights predicted by MLP_2 by the bone transformation matrix calculated from SMPL, and R... d From rotation quaternion q d The rotation matrix is ​​calculated in the middle.

6. The human body free-viewpoint rendering system guided by local information as described in claim 4, characterized in that, The color prediction unit is used to decompose Gaussian color into specular reflection and diffuse reflection components, estimate them separately based on the normal vector, and combine them to obtain the final predicted color, specifically: The Gaussian color is decomposed into specular and diffuse components, and the normal vector is used as one of the bases for predicting the two color components, as follows: ； in, These are the specular reflection color features stored in Gaussian. It is the diffuse color feature stored in Gaussian, where d is the camera viewpoint. It is the specular reflection component, This refers to the diffuse reflection component. MLP_3 and MLP_4 have the same structure, both consisting of an input layer, a hidden layer, and an output layer. The width of the hidden layer is 64. The two components are then added together: ； This yields the final predicted color. .

Images