A three-dimensional Gaussian dynamic human body modeling method, system and application based on compressed triplanes

Abstract

The application discloses a three-dimensional Gaussian dynamic human body modeling method based on compressed three planes, and comprises the following steps: obtaining a camera pose parameter sequence, human body SMPL shape information and a posture information sequence from a single-view video sequence, and obtaining an initialized three-dimensional Gaussian point cloud; obtaining high-dimensional features according to the three-dimensional Gaussian point cloud, and predicting information of the three-dimensional Gaussian in a standard space; distilling skin weights of each vertex of a standard SMPL model and smoothing; generating high-dimensional position coding of the three-dimensional Gaussian by splicing, combining SMPL posture information and three-plane high-dimensional features, predicting basic attribute transformation values, and correcting Gaussian attributes under different human body postures; calculating a transformation matrix by using the corrected skin weights and the SMPL posture information, and obtaining a transformed three-dimensional Gaussian point cloud by transformation; rendering the transformed three-dimensional Gaussian point cloud, combining an image loss function and attribute constraints of neighbor Gaussians, and jointly optimizing to obtain a drivable human body three-dimensional Gaussian model. The application also discloses a system for realizing the above method, and has wide application scenarios.

Description

Technical Field

[0001] This invention relates to the field of dynamic virtual human body modeling, and in particular to a three-dimensional Gaussian dynamic human body modeling method, system and application based on compressed three planes. Background Technology

[0002] 3D reconstruction is a crucial research problem in computer vision, aiming to create 3D models of scenes or objects using 2D images or other data sources. 3D models are fundamental resources for many everyday applications, including entertainment, human-computer interaction, medical diagnosis, virtual reality, and autonomous driving. The description of 3D models relies on different 3D data representations, including traditional explicit and novel implicit representations. Both types of representations have many classic algorithms in 3D reconstruction. Traditional explicit methods include algorithms based on mesh representations, point cloud representations, and volume pixel representations; novel implicit methods include algorithms based on signed distance fields and neural radiance fields. Although algorithms based on different 3D representations have achieved significant breakthroughs in various directions, they are also limited by the inherent nature of explicit and implicit representations.

[0003] In 2023, Inria proposed the concept of 3D Gaussian splatting (ACM Transactions on Graphics). This work is based on the theory that "a mixture of multiple Gaussian distributions can fit any distribution," treating the spatial geometry of a 3D model as a complex distribution and attempting to fit it using a 3D Gaussian distribution. As an explicit discrete representation of space with implicit features, 3D Gaussian splatting can not only efficiently model surface information through learnable parameters but also drive the 3D model by applying simple rotations, scaling, and translations to the 3D representation. Currently, representative algorithms such as GART (Gaussian Articulated Template Models) and GEA (Expressive 3D Gaussian Avatars) improve performance by either more accurately modeling surface motion information or by initializing the spatial position of the 3D Gaussian splatting to more closely approximate the real 3D surface, respectively. Most of these algorithms leverage the inherent advantages of 3D Gaussian splashing to improve performance, without specifically considering the relationship between the spatial distribution constraints of 3D Gaussian splashing during the optimization process and the task scenario, as well as the spatial distribution changes and neighbor information during the movement of 3D Gaussian splashing. This leads to over-reconstruction problems. Summary of the Invention

[0004] To better address the overreconstruction problem in digital human body reconstruction tasks based on 3D Gaussian splashing and reduce the number of 3D Gaussians required for the 3D model, this invention provides a 3D human body modeling method based on 3D Gaussian splashing to generate and drive the 3D human body. The method in this invention solves the overreconstruction problem in existing technologies, making the trained model more lightweight.

[0005] This invention provides a 3D Gaussian dynamic human body modeling method based on compressed three-planes. The method first provides a single-viewpoint human motion video and uses a pose estimation algorithm to extract human shape and pose information from the video. This information guides the 3D Gaussian transformation and attribute correction. Each Gaussian is assigned an LBS (Linear Blend Skinning) weight correction value. The remaining appearance and geometric attributes of the Gaussian are stored as high-dimensional features in the compressed three-planes. A decoder restores the high-dimensional features extracted from the three planes, and an MLP (Multilayer Perception) decodes the high-dimensional features into specific attributes. The position and human pose information of the 3D Gaussian are then input into another MLP to obtain the attribute correction values ​​of the 3D Gaussian in deformation space. The corrected Gaussian is then transformed to the target pose using linear blend skinning. Finally, in the new viewpoint or pose, Gaussian splashing is used for real-time rendering. Compared with other current 3D Gaussian splashing methods, this method can achieve the same quality with fewer parameters and shorter training time, and achieve a higher frame rate.

[0006] The steps of the three-dimensional Gaussian dynamic human body modeling method based on compressed three planes in this invention are as follows:

[0007] Step 1: Extraction and Initialization of 3D Gaussian Point Cloud from SMPL (Skinned Multi-Person Linear Model) Sequence Information: Using the ReFit algorithm (Recurrent fitting network for 3D human recovery), the corresponding camera pose parameter sequence and the corresponding SMPL shape and pose information sequence of the human body are obtained from the pre-provided single-view video sequence. Based on the model vertex information provided by the SMPL shape information, the positions of the 3D Gaussian point cloud in standard space are initialized. Each Gaussian point cloud contains a set of learnable linear mixture skinning weights to adapt to the differences in skinning weights for different human bodies.

[0008] Step 2: Extract and decode the three-plane high-dimensional features to predict basic attributes:

[0009] In step two, the decoding module involves using an attention mechanism during model training to learn how to encode, compress, and then decode the high-dimensional features of the three planes. During inference, the decoder is used to restore the features in the compressed three planes.

[0010] Specifically, for a given 3D Gaussian, high-dimensional features are obtained from a compressed three-plane model based on its coordinate projection in standard space. A decoder is then used to restore the compressed three-plane high-dimensional features to higher-dimensional features. These high-dimensional features include the appearance information, geometric information, and information about neighboring Gaussians. The appearance information includes color and transparency; the geometric information includes scaling; and the neighboring Gaussian information includes the high-dimensional features extracted from other Gaussians surrounding the current Gaussian in the three planes. A neural network decodes these high-dimensional features to obtain specific basic attributes. For the high-dimensional features obtained from the three planes, a multilayer perceptron is used to predict the appearance and geometric information of the 3D Gaussian in standard space.

[0011] Step 3: Basic Skin Weight Distillation: The skin weights of each vertex in the standard SMPL model are distilled in standard space using the KNN (K-Nearest Neighbors) algorithm onto a partitioned coarse voxel grid. A 3D Gaussian algorithm is then used to obtain smoothed skin weights based on their positions in the coarse voxel grid in standard space using trilinear interpolation.

[0012] Step 4: Correction of Basic Attributes and Linear Hybrid Skin Weights: A positional encoding embedding generator maps positional information into a high-dimensional space, and concatenates the 3D information to obtain a high-dimensional positional encoding of a 3D Gaussian. The high-dimensional positional encoding, SMPL pose information, and three-plane high-dimensional features are concatenated as input, and a deformation prediction neural network is used to predict the transformation values ​​of basic attributes. The transformation values ​​of basic attributes are used to correct the Gaussian attributes under different human poses. The transformation values ​​of basic attributes are added to the basic attributes to obtain the final basic attribute values. For the linear hybrid skin weights, each Gaussian also contains a set of learnable linear hybrid skin weight correction values, which are added to the smooth skin weights to obtain the final skin weights.

[0013] Step 5, Linear Blending Skin: Using the corrected skin weights and SMPL pose information, calculate the transformation matrix for each joint. For the Gaussian in the corrected standard space from Step 4, combine the corrected skin weights with the linear blending skin method to perform a weighted summation of the joint transformation matrices using the skin weights as coefficients to obtain the transformation matrix. The transformation matrix can be decomposed into three transformations: translation, scaling, and rotation. Apply translation, scaling, and rotation transformations to the corrected Gaussian to transform it from the standard space to the deformation space of the corresponding target human pose, obtaining the transformed 3D Gaussian point cloud.

[0014] Step Six: Obtaining the Optimal 3D Gaussian Human Model: The 3D Gaussian point cloud transformed in Step Five is rendered using the 3D Gaussian splash rasterization method. Then, the 3D Gaussian coordinates, three-plane parameters, and learnable skin weight correction values ​​are jointly optimized using the image loss function, the neighbor Gaussian properties selected based on KNN, and the multiple perceptron to obtain the optimal, driveable 3D Gaussian human model.

[0015] Step six may also include: real-time arbitrary viewpoint rendering: given the target human body pose and the camera pose parameters corresponding to the rendering viewpoint, drive the obtained optimized 3D Gaussian model and use Gaussian splashing to obtain the final human body rendering image.

[0016] Furthermore, the specific steps for obtaining a 3D model using 3D Gaussians as basic units in this invention are as follows: using the high-dimensional features extracted from the compressed three-plane using the position of each 3D Gaussian in standard space as input, a multi-perceptron is used to predict the 3D Gaussian parameters and SMPL pose information correction amount under standard pose in standard space, and the SMPL pose information and high-dimensional features are used as input to the multi-perceptron to predict the 3D Gaussian parameter transformation value under standard space; then, the skin weights obtained by interpolation from the vertices of the coarse voxel mesh are added to the learnable skin weight correction value to obtain the final skin weight of each Gaussian; the Gaussian is transformed to deformation space by a linear blending skin method; and finally, Gaussian splashing is used for real-time rendering.

[0017] In this invention, the Refit algorithm (Recurrent Fitting Network for 3D Human Recovery) is a deep learning method for 3D human reconstruction; its main goal is to recover the shape, pose, and camera parameters of a 3D human body from a single-view video sequence. In this invention, it is used to extract SMPL shape information, SMPL pose information sequences, and camera parameters.

[0018] SMPL Model: SMPL (Skinned Multi-Person Linear Model) is a standardized 3D model for human body modeling, widely used in computer vision, computer graphics, and virtual reality. It is a human body model based on shape and pose parameters, used to generate and represent the three-dimensional shapes and poses of different human bodies. In this invention, it is used to describe the shape and pose information of the human body in a video sequence.

[0019] LBS Algorithm: Linear Blend Skinning (LBS) is an algorithm used for 3D model deformation, widely applied in skeletal animation, such as human body modeling and animation. Through linear interpolation, it combines the position of a vertex on the model with the influence of multiple joints to deform the model and generate vertex positions in the target space. The linear blend skinning weights describe the degree to which vertices on the model are affected by different joints.

[0020] KNN Algorithm: KNN (K-Nearest Neighbors) is a commonly used machine learning algorithm, widely applied to tasks such as classification, regression, and search. In this invention, it is mainly used for attribute constraints of Gaussian neighbors selected based on KNN in the generation and optimization stage of linear mixture skinning weights for voxel mesh vertices.

[0021] Furthermore, step one mainly includes the following sub-steps:

[0022] Step 1.1. Process the single-view video sequence using the ReFit algorithm (Refit: Recurrent fitting network for 3D human recovery) to obtain camera parameters, human-machine masking parameters, human SMPL shape information, and human SMPL pose information. The human SMPL shape information is expressed using an SMPL linear hybrid skinning model, which includes the positions of 6890 vertices and 24 joints. The human SMPL pose information uses axis-angle to represent the pose of K=24 joints, and the rotation of each joint is represented by a 3D vector, for a total of 72 dimensions (24×3).

[0023] Step 1.2. Under standard space and standard pose, initialize the three-dimensional Gaussian positions using the vertex positions of the SMPL model. The initial number of three-dimensional Gaussians is also 6890; the skin weight correction value of each three-dimensional Gaussian is initialized to null (0).

[0024] Furthermore, step two mainly includes the following sub-steps:

[0025] Step 2.1. The three planes consist of the XY plane, the YZ plane, and the ZX plane. Each plane is a tensor of shape (H, W, C), where H is the length, W is the width, and C is the dimension of the high-dimensional features. During training, a custom encoder is used based on an attention mechanism. HWC The module compresses each plane of the original shape (H, W, C) to (H / 2, W / 2, C_compress), and then uses a custom decoder. HW The shape of each plane is restored to (H, W, C / 3).

[0026] Step 2.2. For a given 3D Gaussian, extract compressed high-dimensional features from the initially reconstructed three planes (H, W, C / 3) using trilinear interpolation based on its coordinates in standard space. Based on the position of the 3D Gaussian in standard space, obtain the high-dimensional features on each plane using bilinear interpolation, and apply a decoder to each high-dimensional feature. C A second reduction is performed to obtain a high-dimensional feature with shape (1,1,C). Then, the high-dimensional features that have undergone the second reduction on different planes are accumulated to obtain the final high-dimensional feature.

[0027] In one specific implementation, the length and width are selected as 64, and the depth is 144.

[0028] Step 2.3. For a given high-dimensional feature, feed the high-dimensional feature into an L... feature Layers, each layer is H wide feature Output transparency and scaling in MLP.

[0029] Step 2.4. Feed the geometric features into an L SH Layers, each layer is H wide SH The first-order spherical harmonic function used in MLP to describe Gaussian color properties.

[0030] Furthermore, step three mainly includes the following sub-steps:

[0031] Step 3.1. Based on the vertices of the SMPL model in Step 1, construct an axis-aligned bounding box with the length, width, and height all being slightly larger than the difference between the maximum and minimum values ​​of all vertices on each axis; on this bounding box, divide the x-axis into 128 equal parts, the y-axis into 128 equal parts, and the z-axis into 32 equal parts to obtain a coarse voxel mesh with a total of 128*128*32 vertices;

[0032] As the Gaussian model's spatial position is optimized during subsequent training to make its spatial distribution closer to the actual human body in the video, some space needs to be reserved to prevent the Gaussian model from moving outside the voxel grid. In one specific implementation, 1.2 times the difference between the maximum and minimum values ​​is selected.

[0033] Step 3.2. Use the KNN algorithm on each vertex to obtain the j closest SMPL model vertices, and perform a weighted sum of the skin weights of each vertex based on the Euclidean distance to these vertices, and distill the skin weights of the SMPL model onto the coarse voxel mesh vertices.

[0034] Step 3.3. Determine the unit voxel grid where the 3D Gaussian is located based on its position in standard space. Perform trilinear interpolation on the skin weights of the eight vertices of the voxel grid based on the relative position of the 3D Gaussian in the voxel grid to obtain the uncorrected and smoothed skin weights of the Gaussian.

[0035] Furthermore, step four mainly includes the following sub-steps:

[0036] Step 4.1. Generate D using cosine position coding and sine position coding respectively from the 3D position information. PE The total number of dimensions is 2*D PE The location encoding information is then combined with the original 3D information to synthesize the final (2*D) image. PE +3) dimensional position encoding information, then concatenate the pose parameters and three-plane high-dimensional features of SMPL with the position encoding information, and use them as input to a set of linear layers to predict the output high-dimensional feature information.

[0037] Step 4.2. Concatenate the three-plane high-dimensional features, position encoding information, and motion pose information into a variable with an L-shape. Deform Layers, each layer is H wide Deform The output of the MLP includes the changes in Gaussian parameters such as position, scaling, rotation, 0th-order spherical harmonic function parameters, and 1st-order spherical harmonic function parameters.

[0038] Furthermore, step six mainly includes the following steps:

[0039] Step 6.1. Based on the obtained transformed Gaussian, use the Gaussian splashing method to obtain the rendered image;

[0040] Step 6.2. In order to optimize the various properties in Gaussian splash and the multi-perceptron neural network used, the loss is calculated based on the rendered image and ground truth image obtained in Step 6.1, and the loss is also calculated based on the different properties of Gaussian and its neighboring Gaussian.

[0041] Furthermore, step 6.2 includes the following steps:

[0042] Step 6.2.1. The losses based on the predicted and rendered images include: absolute error loss (L1 Loss), structural similarity loss (SSIM Loss), perceptual loss (LPIPS Loss), and contour loss (Mask Loss). The contour loss is mainly used to ensure that the 3D Gaussian is constrained within the correct human body area, preventing the neural network from failing to converge.

[0043] Step 6.2.2. The loss based on neighbor Gaussian attributes mainly uses the KNN method to find the K nearest neighbors of each Gaussian in space. The loss is obtained by taking the squared difference between the mean of the skinning weights, spherical harmonic parameters, size, rotation, and other parameters of the Gaussian and the parameters of the K nearest neighbors. By introducing a loss based on the KNN method that considers the consistency of the attributes of the 3D Gaussian with its neighbor Gaussians, the neighbor Gaussian attribute constraint is mainly used to ensure the continuity of the attributes of the spatially nearest Gaussians and prevent severe overfitting.

[0044] In a specific embodiment of the present invention, the K value is a hyperparameter. This loss is introduced to constrain the properties of neighboring Gaussians to be similar. If the K value is too small, there will be insufficient constraints; if the K value is too large, the properties of Gaussians will be too similar over a large range, thus making it impossible to fit high-frequency features.

[0045] The loss is obtained by taking the squared difference of the mean of the parameters described in this step to measure continuity, with the goal of making spatially similar Gaussian properties similar as well.

[0046] Furthermore, real-time arbitrary viewpoint rendering mainly includes the following steps:

[0047] a. Based on the obtained optimized 3D Gaussian model and multiple perceptron (MLP) neural network parameters, for a given action and camera parameters, a Gaussian transformation is performed, and rendering is carried out using the Gaussian splashing method.

[0048] b. When only camera parameters are changed, the already transformed Gaussian image is rendered directly without needing to be transformed again.

[0049] The present invention also provides a system for implementing the above method, the system comprising: a data extraction and initialization module, a three-plane encoding compression and restoration module, a feature extraction and weight calculation module, an attribute correction and deformation module, and a rendering and real-time optimization module;

[0050] The data extraction and initialization module extracts camera pose, human SMPL information and vertex data from single-view video sequences, and initializes the three-dimensional Gaussian point cloud and related parameters.

[0051] The three-plane encoding compression and restoration module is used to perform dimensional compression and one or more restorations on the original shape of the plane;

[0052] The feature extraction and weight calculation module extracts Gaussian high-dimensional features through three planes and uses the KNN algorithm to generate smooth skin weights to predict basic attributes and weights for Gaussian point clouds.

[0053] The attribute correction and deformation module combines position encoding and SMPL pose information to correct the Gaussian attributes and skin weights, and generate the joint transformation matrix required for deformation.

[0054] The rendering and real-time optimization module uses linear blending skinning and Gaussian splashing techniques to render the target pose, and optimizes the Gaussian model through a loss function to support real-time rendering.

[0055] The present invention also provides the application of the above-described method or system in digital human body modeling and driving, etc.

[0056] The beneficial effects of this invention include: when training a 3D Gaussian model using a single-view video sequence, it can effectively reduce training time and use fewer 3D Gaussians without sacrificing rendering quality. Due to the efficiency of this invention, both new viewpoint synthesis and new human body synthesis can achieve real-time speeds, allowing for real-time observation of virtual human bodies from different camera positions and enabling virtual human bodies to perform new actions. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of the present invention.

[0059] Figure 2 This is a structural diagram of the three-plane encoding compression and decoding restoration module of the present invention.

[0060] Figure 3 This is a Gaussian rendering result of the present invention in standard space.

[0061] Figure 4 This is the final result diagram in the deformation space of the Gaussian transformation after attribute correction according to the present invention.

[0062] Figure 5 This is a comparison chart of evaluation indicators for different methods in the embodiments of the present invention. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0064] This invention provides a three-dimensional Gaussian dynamic human body modeling method based on compressed three planes, such as... Figure 1As shown, the core of this invention lies in training a 3D Gaussian model using a single-view video sequence. While single-view videos are generally simpler to acquire than multi-view videos, they contain relatively less information. Single-view videos can be used with consumer-grade graphics cards to train the model in a short time. The trained 3D Gaussian model, for a given target human pose, can render human images or high-frame-rate videos with high-frequency texture details, including clothing folding effects, under any camera parameters.

[0065] In this invention, the terms "three-dimensional Gaussian" and "Gaussian" refer to the same thing.

[0066] This invention uses the Refit algorithm to obtain a set of frames from a pre-provided single-view video sequence. i The corresponding camera pose parameter sequence ParamsCam is obtained from this. f Human body corresponding to SMPL shape information (Shape), SMPL pose information sequence (θ) f and SMPL vertex information smpl :{svert s |s=1,2,...,6890}, each SMPL vertex is also associated with an SMPL vertex forward blending weight W. smpl :{W s |s=1,2,...,6890};The SMPL vertex information refers to a set of vertex positions initialized based on the shape information.

[0067] Guided by a series of initial information obtained through the Refit algorithm, this invention learns a set of three-dimensional Gaussian representations {G1, G2, ..., G...} defined in a standard space. N Each Gaussian is associated with a set of basic properties: position pos i Skin weight correction value ΔWG i Other information about Gauss, such as the spherical harmonic coefficient SH, is also available. i Transparency a i And scaling factor s i Then by using MLR Triplane The high-dimensional information F in the three planes i Decoded.

[0068] Meanwhile, in order to record the feature information of the model more compactly, this invention uses a three-plane compression method based on an attention mechanism to compress the plane from the original shape (H,W,C) to (H / 2,W / 2,C_compress).

[0069] Specifically, Encoder HWCIt consists of multiple TriplanetokenBlocks. Each TriplanetokenBlock mainly comprises two parts: Windowed Multi-Head Self-Attention and Global-Local FFN. Its main structure is as follows: Figure 2 As shown.

[0070] More specifically, for Windowed Multi-Head Self-Attention, taking the uncompressed plane (H,W,C) as input, it is divided into non-overlapping windows, each of size M*M, then we have:

[0071]

[0072] Where M is the width and height of the window, and N is the number of windows.

[0073] Then, reshaping each window into a sequence yields:

[0074]

[0075] For high-frequency branches, the original window is directly fed into multi-head attention, and standard Multi-Head Self-Attention (MHA) is performed on each window:

[0076] Q (n) =X (n) W Q ,K (n) =X (n) W K V (n) =X (n) W V

[0077] in It is a learnable linear projection matrix, and d is the dimension of each attention head.

[0078] Attention output is:

[0079]

[0080] After concatenating multiple attention points, a linear transformation is performed to obtain the high-frequency output of this window:

[0081]

[0082] in, This is the output projection matrix. The high-frequency outputs of all windows are reconstructed into the complete image.

[0083]

[0084] For low-frequency branches, first perform a global average pooling (GAP) operation on the channel dimension for each window:

[0085]

[0086] g (n) Repeat M 2 Next, construct a new low-frequency sequence:

[0087]

[0088] Similarly, multi-head self-attention calculations are performed on it:

[0089]

[0090]

[0091] Ultimately, we obtain low-frequency features at the full-image level:

[0092]

[0093] The output dimension of the attention module is typically hd, where h is the number of attention heads and d is the dimension C = h·d for each attention head. To restore it to the standard channel dimension C, a shared linear mapping weight is introduced:

[0094]

[0095] The above W O It can be shared with or set independently for high- and low-frequency paths, and is used to uniformly map the attention results of each window back to the channel dimension C. After rearranging and concatenating the outputs of all windows, a complete spatial feature map is obtained:

[0096]

[0097] To enhance global modeling capabilities while preserving spatial structure information, this invention also introduces a local-global feedforward network. This module divides the input features into two parts: one part is used for local modeling, and the other part is used for global channel modeling, and finally the two parts are fused together for the output.

[0098] Similarly, using the original features of the three planes as input:

[0099] First, the input is normalized using Layer Normalization:

[0100]

[0101] Then it is divided into two parts according to the channel dimension:

[0102]

[0103] in, Used for local branches.

[0104] Used for global branches.

[0105] Within a local path, depthwise separable convolutions are used to extract local spatial information:

[0106]

[0107] in, The features extracted from the local path are represented by DepthConv, which is a 5×5 depthwise convolution, and Conv, which is a 1×1 pointwise convolution.

[0108] In the global path, global average pooling (GAP) is performed first at the channel level:

[0109]

[0110] Among them, g n Represents the global mean vector along the channel dimension.

[0111] Then g n Input two 1×1 convolutional layers to obtain channel attention weights:

[0112]

[0113] Among them, a n This represents the channel attention weights of the Sigmoid output, ranging from [0,1].

[0114] After channel weight broadcast and

[0115] Multiplication yields the global modulation features:

[0116]

[0117] in, This represents the weighted global path features.

[0118] Concatenate local and global features along the channel dimension:

[0119]

[0120] in, This represents the fused feature map.

[0121] The final output is obtained by adding the residual connection to the input:

[0122]

[0123] Among them, X n+1 Output feature maps for the module, keeping the dimensions unchanged.

[0124] Finally, the overall structure of TriplaneTokenBlock is as follows:

[0125] Y=MLGFFN(SaSA(X)+X)+(SaSA(X)+X)

[0126] Encoder HWC The TriplaneTokenBlock extracts contextual semantics (invariant shape) while simultaneously compressing both spatial and channel dimensions using a downsampling module. Specifically:

[0127] Encoder HWG Contains L TriplaneTokenBlocks, updating features layer by layer:

[0128] X l =TriplaneTokenBlock(X l-1 ), l=1,2,...,L

[0129] in, The structure does not change the spatial or channel dimensions, but only enhances the feature representation.

[0130] After passing through several layers of TokenBlock, a downsampling module is used to compress the features to the target dimension:

[0131]

[0132] Where DH, DW, and DC are the compression coefficients for length, width, and depth, respectively.

[0133] Final Encoder HWC The output is:

[0134]

[0135] The decoder then restores the width and height of the planar features. First, it uses an upsampling module to restore the spatial dimensions to the original image size, and then restores the channel dimensions (e.g., to C′).

[0136]

[0137] in, Other settings are intermediate representations of compressed channels.

[0138] The input then passes through M layers of TriplaneTokenBlock, progressively enhancing the context information:

[0139] G m =TriplaneTokenBlock(G m-1 ), m=1,2,...,M

[0140] Among them, G M This represents the final output feature of the decoder.

[0141] Then, based on Gauss's projection onto the three planes, directly on G... M Upsampling and bilinear interpolation are used to obtain compressed features, which are then processed by the Decoder. C Restore features.

[0142] Specifically, for a given 3D Gaussian position information pos i and plane lane xy Plane yz Plane zx Three planes containing tensors of shape (H, W, C). Let the three-dimensional Gaussian model be {G1, G2, ..., G...}. N If the axisymmetric bounding box of} is AABB, then:

[0143] Center X =(Max(pos1.x,pos2.x,...,pos N .x)+Min(pos1.x,pos2.x,...,pos N .x)) / 2

[0144] Center Y =(Max(pos1.y,pos2.y,...,pos N .y)+Min(pos1.y,pos2.y,...,pos N .y)) / 2

[0145] Center Z =(Max(pos1.z,pos2.z,...,pos N .z)+Min(pos1.z,pos2.z,...,pos N .z)) / 2

[0146] X min =Center X +(Min(pos1.x,pos2.x,...,pos N .x)-Center X )*k

[0147] X max =Center X +(Max(pos1.x,pos2.x,...,pos N .x)-Center X )*k

[0148] Y min =Center Y +(Min(pos1.y,pos2.y,...,pos N .y)-Center Y )*k

[0149] Y max =Center Y +(Max(pos1.y,pos2.y,...,pos N .y)-Center Y )*k

[0150] Z min =Center Z +(Min(pos1.z,pos2.z,...,pos N .z)-Center Z )*k

[0151] Z max =Center Z +(Max(pos1.z,pos2.z,...,pos N .z)-Center Z )*k

[0152] Where k is a reserved extra percentage (in a specific implementation, the reserved extra percentage of k can be chosen to be 1.2) to prevent Gaussian from moving outside the bounding box during the model optimization process. Center is the center coordinate of the AABB bounding box, X min With X max The bounding box has its minimum and maximum boundaries on the X-axis, and Y-axis... min With Y max The bounding box has its minimum and maximum boundaries on the Y-axis, and Z... min With Z max These are the minimum and maximum boundaries of the bounding box on the Z-axis.

[0153] Based on the position pos i The relative positions of Gaussians on the axes of the bounding box are used to extract features from the corresponding three planes, specifically F from the xy plane. xy_compressed For example:

[0154]

[0155] Among them, X relative It's a point pos i In the normalized relative position along the X-axis of the bounding box, Y relative It's a point pos i Normalized relative position in the Y-axis direction of the bounding box.

[0156] Use X relative and Y relative (Value range is 0 to 1) for Plane xy The high-dimensional features stored on the upper layer are obtained using bilinear interpolation to obtain F. xy_compressed .

[0157] Then use Decoder C That is, a small MLP decode The final feature F is obtained by restoration. xy :

[0158] F xy =MLP decode (F xy_compressed )

[0159] F was obtained using the same method. yz F zx , will F xy F yz F zx F is obtained by splicing Triplane Using a neural network MLP feature ,

[0160] Get a i s i F geo Then use a neural network MLP SH Derive the first-order spherical harmonic function:

[0161] a i s i F geo =MLP feature (F Triplane )

[0162] SH i [0] = MLP SH (F geo )

[0163] SH i [1] = MLP SH (F geo )

[0164] Among them, a iIt's transparency, s i It's the scaling factor, F. geo It is a geometric feature, SH i These are the spherical harmonic coefficients used to represent colors.

[0165] Each Gaussian also requires a linear blending skin weight WG. i To support the model's motion, specifically, based on the AABB bounding box boundary mentioned earlier, a voxel mesh is created. This voxel mesh is divided into 128 equal parts along the x-axis, 128 equal parts along the y-axis, and 32 equal parts along the z-axis, thus obtaining a set of vertices for the voxel mesh: Vertex. voxel :{vert v |v=1,2,...,2 19 Each voxel mesh vertex is also associated with a linear blending skin weight W. voxel :{W v |v=1,2,...,2 19}

[0166] The SMPL vertex forward blending weight W smpl :{W s |s=1,2,...,6890} is distilled onto the vertex set of this voxel mesh. Specifically, the KNN algorithm is used on each vertex to obtain the j nearest (in one specific implementation, j=30) SMPL vertices. The linear blending skin weights associated with the SMPL vertices are weighted according to the distance from the voxel vertex to the SMPL vertex, and the blending weights are distilled onto the voxel mesh vertices. Then:

[0167] KNN(vert i ):{kvert j |j=1,2,..,30}

[0168] Distance ij =‖vert i -kvert j ||2

[0169]

[0170] Where σ is the distance decay parameter, controlling the distribution of the weights, vert i It is the vertex of the grid, Distance ij It is the Euclidean distance between the two vertices, w ij W represents the contribution of the j-th SMPL vertex surrounding a voxel mesh vertex to the linear blending skin weights of the voxel mesh vertices. v It is the final linear blending skin weight on the voxel mesh vertices.

[0171] Then, by using the relative position of the Gaussian within the voxel grid, trilinear interpolation is applied to the 8 nearest voxel grid vertices to obtain the uncorrected WG. i The uncorrected linear blended skin weights are compared with the learnable skin weight correction value ΔWG. i The final linear blending skin weights are obtained by adding them together:

[0172] WG i * =WG i +ΔWG i

[0173] Among them, WG i It is the uncorrected linear blended skin weight, ΔWG i It is a learnable skin weight correction value, WG i * It is the final linear blending skin weight for each Gaussian.

[0174] At this point, we have obtained the Gaussian expression in standard space. To refine the Gaussian expression in deformable space after transformation, we still need to consider the specific motion posture information θ. f The properties of the Gaussian need to be modified, including the transparency 'a'. i Scaling ratio s i spherical harmonic coefficient SH i and linear hybrid skin weight WG i Specifically, based on the target human pose θ in a certain frame f. f The position of Gauss in standard space is predicted using a deformation prediction neural network (MLP). Deform Predict the correction value for the Gaussian property:

[0175] Position encoding maps coordinates to a high-dimensional space and associates them with coordinate pos. i and θ f splicing:

[0176] PE(x)=[sin(x·ω1),cos(x·ω1),...,sin(x·ω 10 ),cos(x·ω 10 )],

[0177]

[0178] Δa i Δs i ΔSH i =MLP Deform (concate(PE(pos i .x), PE(pos i .y), PE(pos i.z),pos i ,θ f ))

[0179]

[0180] Where PE(x) is the position code, base is a hyperparameter for controlling the frequency, typically chosen as 10,000; ω i The frequency parameter is used for coding at different scales, Δa i It is the correction value for transparency, Δs i This is the scaling correction value, ΔSH i These are the spherical harmonic coefficient correction values ​​used to represent color. It is ultimate transparency. This is the final scaling ratio. These are the final spherical harmonic coefficients used to represent colors.

[0181] After Gaussian correction, Gaussian needs to be driven into the deformation space through linear blending skin. Specifically, θ f The transformation matrix P of each joint can be obtained by including the rotation parameters of each joint in the target pose. k For each Gaussian, use its corresponding mixing weights:

[0182] WG i * ={wg i1 * ,wg i2 * ,...,wg i24 *}

[0183] Among them, WG i * wg represents the final linear blending skin weights for each Gaussian. ik * This represents the weight of the i-th Gaussian on the k-th bone.

[0184] The corresponding transformation matrix can be calculated:

[0185]

[0186] Among them, P k PG is the transformation matrix of the k-th bone. i It is the i-th Gaussian transformation matrix.

[0187] To transform the Gaussian matrix to the current human pose space, the transformation matrix PG is... i Decomposed into scaling matrix S i Rotation matrix R i and displacement matrix Ti Gaussian will be transformed according to the following formula:

[0188]

[0189] Here, ⊙ represents element-wise multiplication. It is the final transformed position, R i It is a rotation matrix, S i It is a scaling matrix, T i It is a displacement matrix. and This refers to undriven scaling and rotation in standard space. and It refers to the scaling and rotation driven within the deformation space.

[0190] Finally, for the transformed Gaussian Applying the Gaussian splashing method, given the camera parameters ParamsCam... f The final rendered image I is obtained by rendering the following steps. Render .

[0191] The second stage of this invention is the training of the loss function and the model.

[0192] In order to train a 3D high-resolution model, in addition to using rendered images I Render Compared with the ground truth image I provided in the dataset Target In addition to calculating the loss, an attribute constraint of Gaussian spatial neighbors was added to update the Encoder. HWC Decoder HW Decoder C position pos i Skin weight correction value ΔWG i Plane xy Plane yz Plane zx MLP Triplane MLP feature MLP SH MLP Deform .

[0193] There are four loss functions for images: L1 Loss (absolute error loss), Mask Loss (contour loss), SSIM Loss (structural similarity loss), and LPIPS Loss (perceptual loss).

[0194] L1 Loss refers to the difference between the rendered image and the real image within the masking range M. target The distance L1 within, where the mask M target This refers to the area occupied by the human body in the ground truth image, where M represents the area occupied by the human body.target The value is 1, and all other values ​​are 0:

[0195]

[0196] Among them, I Render This is the final rendered image, I Target It is a real image, M target The area of ​​the human body masking in the real image (value is 0 or 1, only the image loss within the masking area is calculated), Loss L1 The rendered image and the real image are within the masked range M. target The L1 distance within.

[0197] Mask loss refers to the area M occupied by the human body in the rendered image. render The area M occupied by the human body in a real image target L2 distance within.

[0198]

[0199] Among them, M render M is the area of ​​human body masking in the rendered image. target It is the area of ​​human body masking in the real image, Loss Mask M is the area M occupied by the human body in the rendered image. render The area M occupied by the human body in a real image target L2 distance within.

[0200] The attribute constraints of spatially neighboring Gaussians are also applied through a loss mechanism. Specifically, the spatial loss is obtained by finding the K nearest neighbors of each Gaussian in space using the KNN method. The loss is calculated by taking the squared difference between the skinning weights, spherical harmonic parameters, size, rotation, and other parameters of the Gaussian and the mean of these parameters of the K nearest neighbors.

[0201] KNN (gauss) i ):{kgauss j |j=1,2,..,30}

[0202]

[0203] Among them, gauss i .attributes is the set of attributes of the i-th Gaussian, Mean(KNN(gauss i ).attributes are the mean of the parameters of the K nearest neighbors of the i-th Gaussian, and Loss neighbor Property constraint loss of spatially neighboring Gaussians.

[0204] The final loss is:

[0205] Loss = λ1Loss L1 +λ2Loss Mask +λ3Loss SSIM +λ4Loss LPIPS +λ5Loss neighbor

[0206] Among them, λ1=1.0, λ2=0.01, λ3=0.01, λ4=0.1, λ4=0.1.

[0207] During the training phase, the gradient magnitudes of the image loss and neighbor Gaussian attribute constraint loss backpropagated to each Gaussian are used as the criteria for Gaussian increment / decrement. If the gradient exceeds a set threshold, a strategy of adding a new Gaussian is implemented.

[0208] Example

[0209] This embodiment implements the present invention on a desktop computer equipped with an i5-13600KF CPU, 64GB of RAM, and an NVIDIA RTX 4090 graphics processor. Training can converge within 10 minutes. On new perspective synthesis tasks, this method can achieve an average rendering speed of 120 frames per second, and on new human pose synthesis tasks, it can achieve a rendering speed of 70 frames per second, which fully meets the requirements of real-time rendering.

[0210] Specifically, this invention uses the ReFit algorithm to extract the camera pose parameter sequence ParamsCam from videos in the ZJU-MoCap dataset. f Human body corresponding to SMPL shape information (Shape), SMPL pose information sequence (θ) f and SMPL vertex information smpl .

[0211] Based on the spatial location provided by the SMPL vertex information, a 3D Gaussian model is initialized at each vertex, resulting in a set of 3D Gaussian models {G1, G2, ..., G...} defined in standard space. N Each Gaussian is associated with a set of basic properties: position pos i Skin weight correction value ΔWG i Other information about Gauss, such as the spherical harmonic coefficient SH, is also available. i Transparency a i And scaling factor s i Then by using MLP Triplane The high-dimensional information F in the three planes i Decoded.

[0212] Meanwhile, in order to record the feature information of the model more compactly, this invention uses a three-plane compression method based on an attention mechanism to compress the plane from the original shape (H,W,C) to (H / 2,W / 2,C_compress).

[0213] Specifically, Encoder HWC It consists of multiple TriplanetokenBlocks. Each TriplanetokenBlock mainly comprises two parts: Windowed Multi-Head Self-Attention and Global-Local FFN. Its main structure is as follows: Figure 2 As shown.

[0214] More specifically, for Windowed Multi-Head Self-Attention, taking the uncompressed plane (H,W,C) as input, it is divided into non-overlapping windows, each of size M*M, then we have:

[0215]

[0216] Where M is the width and height of the window, and N is the number of windows; in one specific implementation, H=64, W=64, C=144, and M=8 are set.

[0217] Then, reshaping each window into a sequence yields:

[0218]

[0219] For high-frequency branches, the original window is directly fed into multi-head attention, and standard Multi-Head Self-Attention (MHA) is performed on each window:

[0220] Q (n) =X (n) W Q ,K (n) =X (n) W K V (n) =X (n) W V

[0221] in It is a learnable linear projection matrix, and d is the dimension of each attention head; in one specific implementation, d = 36.

[0222] Attention output is:

[0223]

[0224] After concatenating multiple attention points, a linear transformation is performed to obtain the high-frequency output of this window:

[0225]

[0226] in, This is the output projection matrix. The high-frequency outputs of all windows are reconstructed into the complete image.

[0227]

[0228] For low-frequency branches, first perform a global average pooling (GAP) operation on the channel dimension for each window:

[0229]

[0230] g (n) Repeat M 2 Next, construct a new low-frequency sequence:

[0231]

[0232] Similarly, multi-head self-attention calculations are performed on it:

[0233]

[0234]

[0235] Ultimately, we obtain low-frequency features at the full-image level:

[0236]

[0237] The output dimension of the attention module is typically hd, where h is the number of attention heads and d is the dimension C = h·d for each attention head. To restore it to the standard channel dimension C, a shared linear mapping weight is introduced:

[0238]

[0239] The above W O It can be shared with or set independently for high- and low-frequency paths, and is used to uniformly map the attention results of each window back to the channel dimension C. After rearranging and concatenating the outputs of all windows, a complete spatial feature map is obtained:

[0240]

[0241] To enhance global modeling capabilities while preserving spatial structure information, this invention also introduces a local-global feedforward network. This module divides the input features into two parts: one part is used for local modeling, and the other part is used for global channel modeling, and finally the two parts are fused together for the output.

[0242] Similarly, using the original features of the three planes as input:

[0243] First, the input is normalized using Layer Normalization:

[0244]

[0245] Then it is divided into two parts according to the channel dimension:

[0246]

[0247] in, Used for local branches. Used for global branches.

[0248] Within a local path, depthwise separable convolutions are used to extract local spatial information:

[0249]

[0250] in, The features extracted from the local path are represented by DepthConv, which is a 5×5 depthwise convolution, and Conv, which is a 1×1 pointwise convolution.

[0251] In the global path, global average pooling (GAP) is performed first at the channel level:

[0252]

[0253] Among them, g n Represents the global mean vector along the channel dimension.

[0254] Then g n Input two 1×1 convolutional layers to obtain channel attention weights:

[0255]

[0256] Among them, a n This represents the channel attention weights of the Sigmoid output, ranging from [0,1].

[0257] After channel weight broadcast and Multiplication yields the global modulation features:

[0258]

[0259] in, This represents the weighted global path features.

[0260] Concatenate local and global features along the channel dimension:

[0261]

[0262] in, This represents the fused feature map.

[0263] The final output is obtained by adding the residual connection to the input:

[0264]

[0265] Among them, X n+1 Output feature maps for the module, keeping the dimensions unchanged.

[0266] Finally, the overall structure of TriplaneTokenBlock is as follows:

[0267] Y=MLGFFN(SaSA(X)+X)+(SaSA(X)+X)

[0268] Encoder HWC The TriplaneTokenBlock extracts contextual semantics (invariant shape) while simultaneously compressing both spatial and channel dimensions using a downsampling module. Specifically:

[0269] Encoder HWC Contains L TriplaneTokenBlocks, updating features layer by layer:

[0270] X l =TriplaneTokenBlock(X l-1 ), l=1,2,…,L

[0271] in, The structure does not change the spatial or channel dimensions, but only enhances the feature representation; in one specific implementation, L=3;

[0272] After passing through the L-layer TokenBlock, a downsampling module is used to compress the features to the target dimension:

[0273]

[0274] Where DH, DW, and DC are the compression coefficients for length, width, and depth, respectively.

[0275] Final Encoder HWC The output is:

[0276]

[0277] In one specific implementation, DH = 2, DW = 2, and DC = 3.

[0278] The decoder then restores the width and height of the planar features. First, it uses an upsampling module to restore the spatial dimensions to the original image size, and then restores the channel dimensions (e.g., to C′).

[0279]

[0280] in, Other settings are intermediate representations of compressed channels.

[0281] The input then passes through M layers of TriplaneTokenBlock, progressively enhancing the context information:

[0282] G m =TriplaneTokenBlock(G m-1 ), m=1,2,…,M

[0283] In one specific implementation, M = 2.

[0284] Among them, G M This represents the final output feature of the decoder.

[0285] Then, based on Gauss's projection onto the three planes, directly on G... M Upsampling and bilinear interpolation are used to obtain compressed features, which are then processed by the Decoder. C Restore features.

[0286] Specifically, for a given 3D Gaussian position information pos i and plane lane xy Plane yz Plane zx Three planes containing tensors of shape (H, W, C). Let the three-dimensional Gaussian model be {G1, G2, ..., G...}. N If the axisymmetric bounding box of} is AABB, then:

[0287] Center X =(Max(pos1.x,pos2.x,...,pos N .x)+Min(pos1.x,pos2.x,...,pos N .x)) / 2

[0288] Center Y =(Max(pos1.y,pos2.y,...,pos N .y)+Min(pos1.y,pos2.y,...,pos N .y)) / 2

[0289] Center Z=(Max(pos1.z,pos2.z,...,pos N .z)+Min(pos1.z,pos2.z,...,pos N .z)) / 2

[0290] X min =Center X +(Min(pos1.x,pos2.x,...,pos N .x)-Center X )*k

[0291] X max =Center X +(Max(pos1.x,pos2.x,...,pos N .x)-Center X )*k

[0292] Y min =Center Y +(Min(pos1.y,pos2.y,...,pos N .y)-Center Y )*k

[0293] Y max =Center Y +(Max(pos1.y,pos2.y,...,pos N .y)-Center Y )*k

[0294] Z min =Center Z +(Min(pos1.z,pos2.z,...,pos N .z)-Center Z )*k

[0295] Z max =Center Z +(Max(pos1.z,pos2.z,...,pos N .z)-Center Z )*k

[0296] Where k is a reserved extra percentage (in a specific implementation, the reserved extra percentage of k can be chosen to be 1.2) to prevent Gaussian from moving outside the bounding box during the model optimization process. Center is the center coordinate of the AABB bounding box, X min With X maxThe bounding box has its minimum and maximum boundaries on the X-axis, and Y-axis... min With Y max The bounding box has its minimum and maximum boundaries on the Y-axis, and Z... min With Z max These are the minimum and maximum boundaries of the bounding box on the Z-axis.

[0297] Based on the position pos i The relative positions of Gaussians on the axes of the bounding box are used to extract features from the corresponding three planes, specifically F from the xy plane. xy_compressed For example:

[0298]

[0299] Among them, X relative It's a point pos i In the normalized relative position along the X-axis of the bounding box, Y relative It's a point pos i Normalized relative position in the Y-axis direction of the bounding box.

[0300] Use X relative and Y relative (Value range is 0 to 1) for Plane xy The high-dimensional features stored on the upper layer are obtained using bilinear interpolation to obtain F. xy_compressed .

[0301] Then use Decoder C That is, a small MLP decode The final feature F is obtained by restoration. xy :

[0302] F xy =MLP decoder (F xy_compressed )

[0303] F was obtained using the same method. yz F zx , will F xy F yz F zx F is obtained by splicing Triplane Using a neural network MLP feature

[0304] Get a i s i F geo Then use a neural network MLP SH Derive the first-order spherical harmonic function:

[0305] a i s i Fgeo =MLP feature (F Triplane )

[0306] SH i [0] = MLP SH (F geo )

[0307] SH i [1] = MLP SH (F geo )

[0308] Among them, a i It's transparency, s i It's the scaling factor, F. geo It is a geometric feature, SH i These are the spherical harmonic coefficients used to represent colors.

[0309] Each Gaussian also requires a linear blending skin weight WG. i To support the model's motion, specifically, based on the AABB bounding box boundary mentioned earlier, a voxel mesh is created. This voxel mesh is divided into 128 equal parts along the x-axis, 128 equal parts along the y-axis, and 32 equal parts along the z-axis, thus obtaining a set of vertices for the voxel mesh: Vertex. voxel :{vert v |v=1,2,...,2 19 Each voxel mesh vertex is also associated with a linear blending skin weight W. voxel :{W v |v=1,2,...,2 19}

[0310] The SMPL vertex forward blending weight W smpl :{W s |s=1,2,...,6890} is distilled onto the vertex set of this voxel mesh. Specifically, the KNN algorithm is used on each vertex to obtain the 30 nearest SMPL vertices. The linear blending skin weights associated with the SMPL vertices are weighted according to the distance from the voxel vertex to the SMPL vertex, and the blending weights are distilled onto the voxel mesh vertices. Then:

[0311] KNN(vert i ):{kvert j |j=1,2,..,30}

[0312] Distance ij =‖vert i -kvert j ||2

[0313]

[0314] Where σ is the distance decay parameter, controlling the distribution of the weights, vert i It is the vertex of the grid, Distance ij It is the Euclidean distance between the two vertices, w ij W represents the contribution of the j-th SMPL vertex surrounding a voxel mesh vertex to the linear blending skin weights of the voxel mesh vertices. v It is the final linear blending skin weight on the voxel mesh vertices.

[0315] Then, by using the relative position of the Gaussian within the voxel grid, trilinear interpolation is applied to the 8 nearest voxel grid vertices to obtain the uncorrected WG. i The uncorrected linear blended skin weights are compared with the learnable skin weight correction value ΔWG. i The final linear blending skin weights are obtained by adding them together:

[0316] WG i * =WG i +ΔWG i

[0317] Among them, WG i It is the uncorrected linear blended skin weight, ΔWG i It is a learnable skin weight correction value ΔWG i WG i * It is the final linear blending skin weight for each Gaussian.

[0318] At this point, we have obtained the Gaussian expression in standard space. To refine the Gaussian expression in deformable space after transformation, we still need to consider the specific motion posture information θ. f The properties of the Gaussian need to be modified, including the transparency 'a'. i Scaling ratio s i spherical harmonic coefficient SH i and linear hybrid skin weight WG i Specifically, based on the target human pose θ in a certain frame f. f The position of Gauss in standard space is predicted using a deformation prediction neural network (MLP). Deform Predict the correction value for the Gaussian property:

[0319] Position encoding maps coordinates to a high-dimensional space and associates them with coordinate pos. i and θ f splicing:

[0320] PE(x)=[sin(x·ω1),cos(x·ω1),...,sin(x·ω 10),cos(x·ω 10 )],

[0321]

[0322] Δa i Δs i ΔSH i =MLP Deform (concate(PE(pos i .x), PE(pos i .y), PE(pos i .z),pos i ,θ f ))

[0323]

[0324] Where PE(x) is the position code, with a base of 10,000; ω i The frequency parameter is used for coding at different scales, Δa i It is the correction value for transparency, Δs i This is the scaling correction value, ΔSH i These are the spherical harmonic coefficient correction values ​​used to represent color. It is ultimate transparency. This is the final scaling ratio. These are the final spherical harmonic coefficients used to represent colors.

[0325] After Gaussian correction, Gaussian needs to be driven into the deformation space through linear blending skin. Specifically, θ f The transformation matrix P of each joint can be obtained by including the rotation parameters of each joint in the target pose. k For each Gaussian, use its corresponding mixing weights:

[0326] WG i * ={wg i1 ** ,wg i2 ** ,...,wg i24 *}

[0327] Among them, WG i * wg represents the final linear blending skin weights for each Gaussian. ik * This represents the weight of the i-th Gaussian on the k-th bone.

[0328] The corresponding transformation matrix can be calculated:

[0329]

[0330] Among them, P k PG is the transformation matrix of the k-th bone. i It is the i-th Gaussian transformation matrix.

[0331] To transform the Gaussian matrix to the current human pose space, the transformation matrix PG is... i Decomposed into scaling matrix S i Rotation matrix R i and displacement matrix T i Gaussian will be transformed according to the following formula:

[0332]

[0333] Here, ⊙ represents element-wise multiplication. It is the final transformed position, R i It is a rotation matrix, S i It is a scaling matrix, T i It is a displacement matrix. and This refers to undriven scaling and rotation in standard space. and It refers to the scaling and rotation driven within the deformation space.

[0334] Finally, for the transformed Gaussian Applying the Gaussian splashing method, given the camera parameters ParamsCam... f The final rendered image I is obtained by rendering the following steps. Render .

[0335] The second stage of this invention is the training of the loss function and the model.

[0336] In order to train a 3D high-resolution model, in addition to using rendered images I Render Compared with the ground truth image I provided in the dataset Target In addition to calculating the loss, an attribute constraint of Gaussian spatial neighbors was added to update the position pos. i Skin weight correction value ΔWG i ,plae xy Plane yz Plane zx MLPT riplane MLP feature MLP SH MLP Deform .

[0337] First, let's discuss the loss function for the encoding / compression and decoding / restoration modules:

[0338] Loss Compressed =||F xy -F orig_xy ‖+‖F yz -F orig_yz ‖+‖F zx -F zx ||

[0339] Where F xy Features such as F are obtained after decoding and reconstruction. orig_xy Features are obtained by interpolation from the original three planes.

[0340] There are four loss functions for images: L1 image loss, MaskLoss, SSIM loss, and LPIPS loss.

[0341] L1 Loss refers to the difference between the rendered image and the real image within the masking range M. target The distance L1 within, where the mask M target This refers to the area occupied by the human body in the ground truth image, where M represents the area occupied by the human body. target The value is 1, and all other values ​​are 0:

[0342]

[0343] Among them, I Render This is the final rendered image, I Target It is a real image, M target The area of ​​the human body masking in the real image (value is 0 or 1, only the image loss within the masking area is calculated), Loss L1 The rendered image and the real image are within the masked range M. target The L1 distance within.

[0344] Mask loss refers to the area M occupied by the human body in the rendered image. render The area M occupied by the human body in a real image target L2 distance within.

[0345]

[0346] Among them, M render M is the area of ​​human body masking in the rendered image. target It is the area of ​​human body masking in the real image, Loss Mask M is the area M occupied by the human body in the rendered image. render The area M occupied by the human body in a real image target L2 distance within.

[0347] The attribute constraints of spatially neighboring Gaussians are also applied through a loss mechanism. Specifically, the spatial loss is obtained by finding the K nearest neighbors of each Gaussian in space using the KNN method. The loss is calculated by taking the squared difference between the skinning weights, spherical harmonic parameters, size, rotation, and other parameters of the Gaussian and the mean of these parameters of the K nearest neighbors.

[0348] KNN (gauss) i ):{kgauss j |j=1,2,..,30}

[0349]

[0350] Among them, gauss i .attributes is the set of attributes of the i-th Gaussian, Mean(KNN(gauss i ).attributes are the mean of the parameters of the K nearest neighbors of the i-th Gaussian, and Loss neighbor Property constraint loss of spatially neighboring Gaussians.

[0351] The final loss is:

[0352] Loss = λ1Loss L1 +λ2Loss Mask +λ3Loss SSIM +λ4Loss LPIPS +λ5Loss neighbor

[0353] Among them, λ1=1.0, λ2=0.01, λ3=0.01, λ4=0.1, λ4=0.1.

[0354] During the backpropagation phase, the gradient magnitude of the image loss and the neighbor Gaussian attribute constraint loss backpropagated to each Gaussian is used as the criterion for Gaussian addition or subtraction. If the gradient exceeds a set threshold, a strategy of adding a new Gaussian is executed.

[0355] After 6000 iterations, the following can be obtained: Figure 3 The diagram shows a Gaussian model in standard space, and several MLPs used for Gaussian property correction. The Gaussian property values ​​stored in the Gaussian model in standard space are: a i Indicates transparency, s i Indicates scaling ratio, SH i The spherical harmonic function is uncorrected because the correction values ​​for the Gaussian properties depend on the specific pose information. The linear blending skin weights for each Gaussian, however, are corrected.

[0356] For the specific driving process, in this embodiment, a pose θ conforming to the SMPL pose information format can be freely specified. f For each Gaussian in the 3D Gaussian model, the position of the Gaussian is encoded with θ. f splicing as MLP Deform The input yields Δa for each Gaussian. i Δs is the correction value for transparency. i ΔSH is the scaling correction value. i This is the spherical harmonic correction value for color. The Gaussian attribute value in standard space is added to the Gaussian attribute correction value to obtain the final Gaussian attribute value. Then, a linear blending skin method is used to transform the Gaussian model from standard space to deformation space, and a 3D Gaussian splash rendering method can be used to obtain, as shown below. Figure 4 The rendered image shown.

[0357] This invention tested its method on different datasets, verifying that it can achieve high-quality visual effects in tasks involving synthesizing human images from new perspectives or with new human poses, including the deformation of clothing according to human pose. Compared to previous methods, this method can achieve the same quality with fewer parameters and shorter training time, while also achieving a higher frame rate.

[0358] like Figure 5 As shown, this invention outperforms other methods on subsets 386, 387, 392, 393, and 3945 of the ZJU-MoCap dataset in multiple metrics, including PSNR, SSIM, and LPIPS (the data for other methods are their own experimental data published in their papers). Furthermore, compared to the HUGS method, which is also based on 3D Gaussian splashing, this invention is faster and lighter, outperforming it in several metrics. Specifically, the HUGS method requires over 30 minutes of training and approximately 200,000 Gaussians, while this invention only requires 4 minutes of training time and approximately 20,000 Gaussians.

[0359] exist Figure 5 The method uses color intensity to emphasize its leading position on the dataset metrics; the darker the color, the higher the metric ranking.

[0360] in Figure 5 The various control methods used in this study all originated from international scientific research conferences.

[0361] Neural body:Peng S,Zhang Y,Xu Y,et al.Neural body:Implicit neuralrepresentations with structured latent codes for novel view synthesis ofdynamic humans[C] / / Proceedings of the IEEE / CVF conference on computer visionand pattern recognition.2021:9054-9063.

[0362] HumanNerf:Weng C Y,Curless B,Srinivasan P P,et al.Humannerf:Free-viewpoint rendering of moving people from monocular video[C] / / Proceedings ofthe IEEE / CVF conference on computer vision and pattern Recognition.2022:16210-16220.

[0363] MonoHuman:Yu Z,Cheng W,Liu X,et al.Monohuman:Animatable human neuralfield from monocular video[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:16943-16953.

[0364] Instant-NVR:Jiang Y,Yao K,Su Z,et al.Instant-nvr:Instant neuralvolumetric rendering for human-object interactions from monocular rgbd stream[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2023:595-605.

[0365] HUGS: Kocabas M, Chang JHR, Gabriel J, et al. Hugs: Human gaussian splats[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2024:505-515.

[0366] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A three-dimensional Gaussian dynamic human body modeling method based on compressed three planes, characterized in that, Includes the following steps: Step 1: Use the ReFit algorithm to obtain the camera pose parameter sequence, human SMPL shape information and human SMPL pose information sequence from the single-view video sequence. Generate the vertices of the model based on the human SMPL shape information and initialize the three-dimensional Gaussian point cloud at the vertices. Step 2: Based on the 3D Gaussian point cloud obtained in Step 1, use the position projection of the 3D Gaussian to extract compressed high-dimensional features from the three planes, use the decoding module to restore the high-dimensional features of the three planes, and use the restored high-dimensional features of the three planes to predict the appearance information, geometric information and adjacent Gaussian information of the 3D Gaussian in standard space. Step 3: Distill the skinning weights of each vertex in the standard SMPL model based on the KNN algorithm, and smooth the skinning weights using an interpolation algorithm; Step three further includes: Step 3.1: Based on the vertices of the SMPL model, construct an axis-aligned bounding box, and then divide the x, y, and z axes equally to generate a coarse voxel mesh to cover all vertices; Step 3.2: Use the KNN algorithm to find the j vertices of each coarse voxel grid vertex that are closest to the SMPL model, and sum the skinning weights of each vertex in a weighted manner, then distill the weights to the grid vertices; In step 3.2, the skinning weights of the vertex are weighted and summed by calculating the Euclidean distance between the vertex and the j nearest SMPL model vertices; Step 3.3: Based on the position of the 3D Gaussian in standard space, determine the voxel mesh in which it is located, and calculate the smooth skinning weight of the Gaussian point by trilinear interpolation; The final skin weights are obtained by adding the learnable linear blending skin weight correction value contained in each Gaussian in the linear blending skin weights to the smoothing skin weights in step 3.

3. Step 4: Use position encoding and 3D information to generate a high-dimensional position encoding of a 3D Gaussian, and combine SMPL pose information and three-plane high-dimensional features to predict the transformation values ​​of basic attributes through a deformation prediction neural network, and correct the Gaussian attributes under different human poses. Step 5: Using the corrected skin weights and SMPL pose information, calculate the transformation matrix, and transform the Gaussian point cloud to the target human pose deformation space through the transformation matrix to obtain the transformed 3D Gaussian point cloud. Step 6: Render the transformed 3D Gaussian point cloud using the Gaussian splashing method, and combine the image loss function with the attribute constraints of neighboring Gaussians to jointly optimize the 3D Gaussian coordinates, three-plane parameters, skin weight correction value and multiple perceptron to obtain a 3D Gaussian model that can drive the human body. Step six further includes: Step 6.

1. Based on the obtained transformed Gaussian, use the Gaussian splashing method to obtain the rendered image; Step 6.

2. In order to optimize the various properties in Gaussian splash and the multi-perceptron neural network used, the loss is calculated based on the rendered image and ground truth image obtained in Step 6.1, and the loss is also calculated based on the different properties of Gaussian and its neighboring Gaussian. Step 6.2 further includes: Step 6.2.1: Based on the error between the predicted image and the rendered image, optimize the properties of Gaussian by using absolute error loss, structural similarity loss, perceptual loss, and contour loss to limit the range of the 3D Gaussian. Step 6.2.2: Find the nearest neighbor Gaussian set for each 3D Gaussian using the KNN method, calculate the squared difference loss between the skin weight, spherical harmonic parameter, size and rotation attributes and the mean of the nearest neighbor Gaussian attributes, and introduce neighbor constraints to ensure the continuity of Gaussian attributes in space.

2. The method as described in claim 1, characterized in that, Step one further includes the following steps: Step 1.

1. Use the ReFit algorithm to process the single-view video sequence to obtain the camera pose parameter sequence including camera parameters and human-machine mask parameters, human SMPL shape information, and human SMPL pose information; Step 1.

2. Under standard space and standard pose, initialize the 3D Gaussian position using the vertex position of the SMPL model, and initialize the skin weight correction value of each 3D Gaussian to null.

3. The method as described in claim 1, characterized in that, In step two, the three planes include the XY plane, the XZ plane, and the ZX plane; the high-dimensional features include the appearance information, geometric information, and adjacent Gaussian information of the Gaussian splash. And / or, During training, the shape of the original plane is compressed and restored once, and the compressed high-dimensional features on the plane are extracted after the first restoration. During inference, the high-dimensional features that have been extracted and interpolated are restored a second time using the decoder. And / or, The appearance information includes color and transparency; the geometric information includes scaling ratio; the adjacent Gaussian information includes high-dimensional features extracted from other Gaussians around the current Gaussian in the three planes. And / or, Bilinear interpolation is used to obtain high-dimensional features on each plane, and secondary restoration is performed. The high-dimensional features obtained after secondary restoration on different planes are accumulated to obtain the final high-dimensional features. And / or, High-dimensional features are processed through a multilayer perceptron and a fully connected layer to output transparency, scaling, and spherical harmonic features, and Gaussian color properties are described by a spherical harmonic function.

4. The method as described in claim 1, characterized in that, Step four further includes: Step 4.1: The position coding information is generated by combining sine and cosine position coding and three-dimensional information. After being concatenated with SMPL pose parameters and three-plane high-dimensional features, it is input into the linear layer to predict and generate high-dimensional feature information. Step 4.2: Concatenate the three-plane high-dimensional features, position encoding information and motion pose information and send them into the MLP to output the changes in Gaussian parameters, including position, scaling, rotation and spherical harmonic function parameters.

5. The method as described in claim 1, characterized in that, In step five, the transformation matrix of each joint is calculated using the corrected skin weights and SMPL pose information. For the Gaussian in the corrected standard space, the transformation matrix of the joint is obtained by weighted summation using the skin weights as coefficients, with the corrected skin weights combined with the linear hybrid skinning method. The corrected Gaussian is then subjected to translation, scaling and rotation transformations decomposed by the transformation matrix to transform it from the standard space to the deformation space of the corresponding target human pose, thus obtaining the transformed three-dimensional Gaussian point cloud.

6. The method as described in claim 1, characterized in that, Step six also includes: based on the obtained optimized 3D Gaussian model and multi-perceptron (MLP) neural network parameters, performing a Gaussian transformation for a given action and camera parameters, and rendering using the Gaussian splashing method.

7. A system for implementing the method as described in any one of claims 1-6, characterized in that, The system includes: a data extraction and initialization module, a three-plane encoding compression and restoration module, a feature extraction and weight calculation module, an attribute correction and deformation module, and a rendering and real-time optimization module; The data extraction and initialization module extracts camera pose, human SMPL information and vertex data from single-view video sequences, and initializes the three-dimensional Gaussian point cloud and related parameters. The three-plane encoding compression and restoration module is used to perform dimensional compression and one or more restorations on the original shape of the plane; The feature extraction and weight calculation module extracts Gaussian high-dimensional features through three planes and uses the KNN algorithm to generate smooth skin weights to predict basic attributes and weights for Gaussian point clouds. The attribute correction and deformation module combines position encoding and SMPL pose information to correct the Gaussian attributes and skin weights, and generate the joint transformation matrix required for deformation. The rendering and real-time optimization module uses linear blending skinning and Gaussian splashing techniques to render the target pose, and optimizes the Gaussian model through a loss function to support real-time rendering.

Images