A modeling and rendering method, device and medium based on layered three-dimensional Gaussian representation

Abstract

The application relates to the technical field of three-dimensional human body modeling, and particularly discloses a modeling and rendering method and device based on layered three-dimensional Gaussian representation and a medium, method steps comprising: acquiring an image sequence of a target, initializing three-dimensional Gaussian points in a canonical space based on SMPL human body template vertices; taking the initialized three-dimensional Gaussian points as the first layer on the innermost side, constructing a layered Gaussian structure from coarse to fine, and establishing conditional dependence between layers through a parent-child node connection mode; setting layered motion conditions, and respectively predicting rigid body and non-rigid body deformation transformation parameters of each layer of Gaussian points by using a rigid body transformation network and a non-rigid body transformation network; uniformly mapping the layered Gaussian points to an observation space according to the predicted transformation parameters, and performing image synthesis by using a three-dimensional Gaussian rendering technology. Compared with the prior art, the application can naturally adapt to and express deformation characteristics of different levels in human body motion, and realizes high-quality reconstruction and real-time rendering of a complex dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of 3D human body modeling technology, and in particular to a temporal modeling and rendering method based on hierarchical 3D Gaussian representation. Background Technology

[0002] In recent years, 3D human avatar modeling and rendering technologies have been widely used in virtual reality, intelligent interaction, and other fields. With the development of neural rendering methods, this technology has transformed from static 3D reconstruction to dynamic, actuated human image reconstruction. Among them, 3D Gaussian Splatting (3DGS) technology, with its efficient rendering performance and strong geometric modeling capabilities, has become one of the mainstream methods for constructing animated digital humans.

[0003] 3DGS technology achieves high-fidelity human portrait reconstruction by initializing Gaussian point clouds in canonical space and performing joint optimization with rendering supervision. To achieve controllable human animation effects, researchers combine the human parametric model SMPL or SMPL-X with 3DGS, using linear blend skinning (LBS) to transform the Gaussian points of the human body in canonical space to the observation space of each frame, thus realizing pose-driven human animation. This type of method simplifies human deformation modeling to skeleton-driven rigid body transformation, improving efficiency and maneuverability, and has good engineering applicability.

[0004] Representative approaches include GauHuman and 3DGS-Avatar, which use the skeletal pose estimated by SMPL as a condition, utilize LBS to calculate the rigid body transformation of Gaussian points, and perform joint optimization with image supervision to achieve efficient portrait rendering. These methods typically treat all Gaussian points as homogeneous structures, lacking the ability to organize and model more finely. Since human motion usually involves multi-scale, non-rigid local dynamic changes (such as clothing swaying and secondary limb movements), the above methods have significant limitations in detail restoration and non-rigid representation.

[0005] To enhance temporal continuity and dynamic performance, some studies have introduced temporal embedding mechanisms, such as HumanNeRF, to improve rendering quality by encoding temporal features or guiding temporal consistency. However, these methods are mostly based on implicit neural representations, rely on a large amount of computational resources, and lack structured driving capabilities, making it difficult to support flexible human animation control.

[0006] While existing SMPL-driven 3D Gaussian portrait modeling methods offer high rendering efficiency, they still have significant shortcomings in non-rigid deformation modeling. On one hand, these methods typically rely on linear blending skins (LBS) to rigidly drive Gaussian points, making it difficult to accurately model the complex deformations of non-rigid structures such as clothing caused by pose changes, resulting in a lack of realistic dynamic details in the rendered images. On the other hand, existing methods generally employ a flattened Gaussian structure of the human body, ignoring the spatial and structural hierarchy of human motion, making it difficult to capture local motion patterns from multiple scales. Furthermore, while some NeRF-based methods introduce temporal context to improve dynamic consistency, they suffer from high computational costs, slow rendering speeds, and a lack of real-time performance. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology in non-rigid human body modeling, multi-scale motion structure representation, and temporal consistency, and to provide a temporal human body modeling method that introduces hierarchical Gaussian structure and hierarchical motion modeling.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] As a first aspect of the present invention, a modeling and rendering method based on hierarchical 3D Gaussian representation is provided, the steps of which include:

[0010] Obtain the dynamic image sequence of the target, and initialize three-dimensional Gaussian points in normal space based on the vertices of the SMPL human body template;

[0011] Starting with the initial three-dimensional Gaussian points as the innermost first layer, a layered Gaussian structure is constructed from coarse to fine. Conditional dependencies are established between each layer through parent-child node connections.

[0012] The preceding frame set is sampled and the layered motion conditions are obtained. Based on the layered motion conditions, the rigid body transformation network and the non-rigid body transformation network are used to predict the transformation parameters of rigid body and non-rigid body deformation of Gaussian points in each layer.

[0013] The layered Gaussian points are uniformly mapped to the observation space according to the predicted transformation parameters, and the three-dimensional Gaussian rendering technology is used to synthesize images from any viewpoint.

[0014] As a preferred technical solution, the hierarchical Gaussian structure is constructed in stages based on the backpropagation gradient received by each Gaussian point during training, and the steps are as follows:

[0015] The Gaussian points obtained by initializing the 3D Gaussian representation with SMPL human body template points are used as the Gaussian points of the first layer.

[0016] Based on the number of layers n layer Divide the entire training process into n equal parts. layer-1 training phases, where the i-th training phase is used to construct the (i+1)-th sub-Gaussian representation;

[0017] In the i-th training phase, each time the parameters of all layers before the i-th Gaussian point are updated according to the gradient of backpropagation, the parameters of the i-th layer are fixed, and the Gaussian parameters of the i-th layer are updated according to the gradient.

[0018] For a Gaussian point in the i-th layer whose Gaussian gradient value is greater than a set threshold, assign it a child Gaussian node; the child Gaussian node inherits the attributes of its parent node and also participates in the optimization of the current stage, and the child node is added to the Gaussian point in the (i+1)-th layer.

[0019] As a preferred technical solution, the layered motion conditions include:

[0020] The coarse-grained skeleton motion condition is the change in skeleton parameters between adjacent frames;

[0021] The fine-grained point-level velocity condition is the difference in position of the human body Gaussian point in the observation space between two consecutive frames.

[0022] As a preferred technical solution, the rigid body transformation is a transformation matrix describing the Gaussian point from the standard space to the observation space; the non-rigid body transformation is a three-dimensional vector describing the positional offset of each Gaussian point in the three-dimensional space under the standard space.

[0023] As a preferred technical solution, the rigid body transformation and non-rigid body transformation of the first layer of Gaussian points are set as follows:

[0024] The position of each initial Gaussian point is obtained by estimating the human pose using SMPL and input into LBS MLP. The LBS MLP predicts and outputs the weight of each Gaussian point for each human joint. Based on the predicted weights, the weighted sum of all human joints is calculated, and the rigid body transformation of each Gaussian point in the first layer is obtained.

[0025] The non-rigid transformation of the first layer of Gaussian points is set to 0.

[0026] As a preferred technical solution, the method of using a rigid-body and non-rigid-body transformation network to predict the rigid-body and non-rigid-body deformation transformation parameters of each Gaussian point is as follows:

[0027] The rigid and non-rigid deformation results of the first layer of Gaussian points are transformed into the observation space according to the skeleton parameters of the current frame using the linear hybrid skin LBS algorithm.

[0028] Following the hierarchical structure, the subsequent sub-layers are traversed sequentially. The transformation result of the parent node of each sub-layer Gaussian point, the hierarchical motion condition of the parent node, and the spatial position of the child node itself are input into the rigid body transformation network and the non-rigid body transformation network respectively to predict the residuals of the rigid body and non-rigid body transformations of the child node relative to the parent node.

[0029] Based on the rigid and non-rigid transformations of the current child node corresponding to the parent node, and the predicted residuals of the rigid and non-rigid transformations of the child node relative to the parent node, the rigid and non-rigid transformations of the current child node are obtained.

[0030] As a preferred technical solution, during the transformation parameter prediction process...

[0031] For the image sequence of the target, multiple historical frames are sampled from the current frame backward at different sampling intervals to form a set of preceding frames;

[0032] All the layered motion conditions obtained by downsampling at all sampling intervals are used as the transformation parameters for predicting Gaussian points in both rigid body transformation networks and non-rigid body transformation networks.

[0033] As a preferred technical solution, during the transformation parameter prediction process:

[0034] The point-level velocity conditions of the nearest multiple points for each Gaussian point are concatenated and input into the non-rigid transformation network to predict the residuals of the non-rigid transformation.

[0035] As a second aspect of the present invention, a modeling and rendering apparatus based on hierarchical three-dimensional Gaussian representation is provided, comprising:

[0036] One or more processors;

[0037] Memory, used to store one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the modeling and rendering method based on hierarchical 3D Gaussian representation as described above.

[0039] As a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the modeling and rendering method based on hierarchical three-dimensional Gaussian representation as described above.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1) This invention proposes a temporal human body modeling method based on layered Gaussian structure and layered motion modeling, introducing a layered human body Gaussian structure and a layered motion condition-driven mechanism. By combining prior information about the human skeleton and local geometric motion information, the method enhances the expressive power and rendering quality of dynamic human figures in non-rigid deformation scenes. Furthermore, by utilizing coarse-to-fine motion conditions and a layered non-rigid body transformation mechanism, the expressive power and deformation accuracy of Gaussian modeling are structurally improved. This enables more detailed, stable, and real-time 3D modeling and rendering of dynamic human figures.

[0042] 2) The hierarchical human Gaussian structure proposed in this invention first initializes a three-dimensional Gaussian point representation in standard space based on the vertices of the SMPL human body template, and constructs a hierarchical human body representation system. All Gaussian points are organized into multiple layers, with each layer establishing conditional dependencies through parent-child node connections to achieve a more structured expression. This hierarchical design does not explicitly divide the human body into skeleton, shape, or clothing layers, but rather, through layer-by-layer refinement of modeling capabilities, enables the system to naturally adapt to and express the deformation characteristics of different levels during human movement, such as global movements driven by the skeleton and local dynamics of surface clothing.

[0043] 3) This invention proposes a layered motion condition modeling mechanism, combining coarse-grained skeleton pose differences and fine-grained point velocities as conditional inputs for non-rigid deformation, thereby enhancing the expressive power of motion-driven processes. To further enhance the system's modeling effect on temporal continuity and dynamic details, a spatiotemporal multi-scale sampling strategy is designed. By extracting motion trends and local details at different time intervals, it can capture long-term human motion trends while preserving inter-frame dynamic details, thus achieving high-quality reconstruction and real-time rendering of complex dynamic scenes. Attached Figure Description

[0044] Figure 1 The flowchart shows the human body modeling and rendering method based on layered 3D Gaussian representation of the present invention.

[0045] Figure 2 This is a schematic diagram of the temporal human rendering framework for hierarchical Gaussian structure and hierarchical motion modeling of the present invention.

[0046] Figure 3 This is a schematic diagram illustrating the motion conditions at different particle sizes and multi-scale sampling of the present invention;

[0047] Figure 4 This is a schematic diagram of the human body layered Gaussian representation and the human body transformation prediction based on the layered representation of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] This invention proposes a modeling and rendering method based on hierarchical 3D Gaussian representation, aiming to solve the problem that traditional skeleton-driven modeling methods struggle to accurately predict subtle changes in human appearance, especially under conditions of loose clothing or complex dynamics where there is a one-to-many appearance fuzzy mapping. By constructing a structured hierarchical Gaussian representation and a spatiotemporal multi-scale conditional modeling mechanism, this invention significantly improves the modeling capability and rendering quality for non-rigid body motion.

[0051] like Figure 1 , 2 As shown, this embodiment takes human body modeling as an example, and the specific implementation process of the method is as follows:

[0052] S1. Human body initialization and hierarchical Gaussian structure construction

[0053] This method first acquires a sequence of human training images and estimates the human pose parameters for each frame using the SMPL model. Based on the SMPL template vertices, three-dimensional Gaussian points are initialized in the normalized space as the basic geometric representation of the human body. Building upon this, the invention introduces a hierarchical Gaussian structure, organizing the human body Gaussian points into a multi-layered structure with parent-child dependencies, forming a coarse-to-fine modeling system. Gaussian points in different layers have different resolutions and degrees of freedom in representation, facilitating the modeling of human motion characteristics at different granularities, from the skeletal framework to surface details (such as clothing dynamics). This hierarchical structure does not directly divide into explicit categories such as "skeleton layer" or "clothing layer," but rather enhances the modeling capability for different motion levels through structural organization.

[0054] The hierarchical Gaussian points are constructed in stages based on the backpropagation gradient received by each Gaussian point during training. Specifically, the number of layers is n. layer As a hyperparameter, it needs to be manually specified, and is generally set to 3 layers. Layering Gaussian points does not explicitly define what each layer represents, but rather aims to adaptively learn different levels of the human body throughout the training and optimization process. Below is the specific division and construction method of layered Gaussian points during training and optimization:

[0055] 1.1) Initialize the 3D Gaussian representation using SMPL human body template points, and use the resulting Gaussian points as the first layer (i.e., the top layer Gaussian points).

[0056] 1.2) Based on the number of layers n layer Divide the entire training process into n equal parts. layer -1 stage. The i-th training stage is due to the construction of the (i+1)-th layer sub-Gaussian representation.

[0057] 1.3) In the i-th training phase, each time the gradient is updated based on backpropagation, the parameters of the i-th Gaussian layer before (i.e., layers 1 to i-1) are fixed, and the Gaussian parameters of the i-th layer are updated based on the gradient. For Gaussian gradient values ​​in the i-th layer that are greater than a threshold σ, split Gaussian point (σ split If the hyperparameter is , then a child Gaussian node is assigned to it. The attributes of the child Gaussian node are inherited from its parent node, and it also participates in the optimization of the current stage. The child node is then added to the (i+1)th layer of Gaussian.

[0058] S2. Layered motion condition modeling

[0059] This invention designs a motion modeling mechanism based on spatiotemporal information sampling for the current frame t in a temporal input sequence, such as... Figure 3 As shown. In terms of time, multiple historical frames are sampled backwards from frame t to form a preceding frame set, with a sampling interval of s:

[0060]

[0061] Regarding the sampling interval, the initial sampling interval is s0, and we are given a sampling interval growth hyperparameter Δs. Then, the i-th sampling interval s... i =s0+iΔs, a total of L intervals are sampled.

[0062] Spatially, two levels of motion conditions are constructed respectively:

[0063] Coarse-grained skeleton motion condition ΔP t This indicates the change in skeleton parameters between adjacent frames;

[0064] Fine-grained point-level velocity condition V t The velocity is obtained by calculating the position difference of the human body's Gaussian point in the observation space between two consecutive frames.

[0065] These two elements combine to form a hierarchical motion context, which serves as a conditional input to a non-rigid transformation network (multilayer perceptron). This is used to predict the rigid and non-rigid body deformations at Gaussian points in each layer, starting from the second layer. The velocity condition ΔP is obtained from sampling at all sampling intervals. t and V t Both will be used as inputs in the transformation of the predicted Gaussian points in the MLP.

[0066] S3. Hierarchical Transformation Propagation Mechanism and Transformation Parameter Prediction

[0067] For the first layer of Gaussian points (i.e., points initialized using the SMPL template), since they have no parent node, the transformation of these Gaussian points directly uses the human pose estimated by SMPL, and LBS MLP is used to predict the weights of each Gaussian point for each human joint. The input of LBS MLP is the position of each Gaussian point, and its output is the weight of each Gaussian point for each human joint. These weights are then used to calculate the weighted sum of all human joints to obtain the rigid body transformation of each Gaussian point. In this application, only the first layer of Gaussian points needs to use LBS MLP to predict the weights of each Gaussian point for each joint. The non-rigid body transformation of the first layer of Gaussian points is set to 0.

[0068] The rigid and non-rigid deformation results of the first layer are transformed into the observation space using a Linear Blend Skinning (LBS) algorithm based on the skeleton parameters of the current frame. Then, subsequent sub-layers are traversed sequentially according to the layered structure. Each sub-layer Gaussian point depends on the transformation results (including rigid and non-rigid transformations) of its parent node and the layered motion conditions of the parent node, combined with the spatial position of the sub-node itself. Another MLP predicts its transformation residual relative to the parent node, thus constructing a top-down conditional propagation and refinement process. This mechanism ensures structural consistency in the Gaussian point transformation and improves the rationality and stability of non-rigid deformation in complex regions (such as clothing edges).

[0069] Specifically, such as Figure 4 As shown, the transformation parameters of each Gaussian child node include rigid body transformation and non-rigid body transformation. These are obtained by using MLP to predict the transformation residuals based on the transformation of its parent node. Furthermore, it is necessary to traverse layer by layer to calculate the transformation of each Gaussian point, as detailed below:

[0070]

[0071] in, This refers to the rigid and non-rigid transformations of the current child node. The rigid transformation is a 3x4 matrix that describes the transformation from the canonical space to the Gaussian point in the observation space, including rotation and translation. The non-rigid transformation is a three-dimensional vector that describes the offset of the three-dimensional position of each Gaussian point in the canonical space. and For the rigid body transformation and non-rigid body transformation of the parent node of the current child node; and The residuals of the rigid and non-rigid transformations of the current child node are obtained from the MLP prediction. The prediction method is described in [link to prediction]. Figure 4 Specifically:

[0072]

[0073] in, ΔP t For the point-level velocity conditions and skeleton motion conditions of the parent node, Pos cur This represents the position of the current child node.

[0074] Spatiotemporal multi-scale sampling mechanism: To enhance the stability and generalization ability of predictions, this invention further designs a spatiotemporal multi-scale motion modeling mechanism. In the time dimension, multiple sampling intervals are employed. It can capture long-term human motion trends while preserving dynamic details between frames. Spatially, the point-level velocity conditions of the k nearest points for each Gaussian point are concatenated and input into the non-rigid body transformation network to predict non-rigid body transformations.

[0075] S4. End-to-end optimization and high-quality rendering

[0076] After uniformly mapping the hierarchical Gaussian points to the observation space according to the predicted transformation parameters, 3D Gaussian rendering technology is used to synthesize images from any viewpoint, and the images are supervised by real images. The Gaussian parameters and neural network weights are optimized through end-to-end training. This framework can achieve high-fidelity reconstruction of dynamic portraits without explicitly modeling complex texture changes and occlusion relationships.

[0077] Not limited to the human body modeling and rendering example provided in this embodiment, the solution of the present invention can be used not only for human body reconstruction, but also for the reconstruction of other objects with joint movement, such as animals.

[0078] In summary, this invention achieves accurate geometric modeling and high-quality rendering of complex human body movements by introducing hierarchical structure modeling, multi-scale spatiotemporal motion modeling, and non-rigid body prediction mechanisms, exhibiting excellent flexibility, scalability, and generalization ability. Compared with existing skeleton-driven dynamic human body modeling methods, this invention significantly improves the expressive ability for complex non-rigid body movements (such as clothing swaying and muscle deformation) by introducing a hierarchical Gaussian point structure and a hierarchical motion modeling mechanism. Simultaneously, the temporal fusion design of coarse-grained skeleton motion and fine-grained point-level velocity effectively alleviates the one-to-many mapping problem between the skeleton and the appearance, improving the accuracy and consistency of modeling. Furthermore, the multi-scale spatiotemporal sampling strategy enhances the method's adaptability to rapid or slow motion changes, making the model more stable in real-world scenarios and significantly improving the accuracy and generalization of non-rigid body human body modeling and rendering.

[0079] Example 2

[0080] As a second aspect of the present invention, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the modeling and rendering method based on hierarchical 3D Gaussian representation as described above. In addition to the processors, memory, and interfaces described above, any data processing device in the embodiments may also include other hardware depending on the actual function of the data processing device, which will not be elaborated further.

[0081] Example 3

[0082] As a third aspect of the present invention, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the modeling and rendering method based on hierarchical three-dimensional Gaussian representation as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0083] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A modeling and rendering method based on hierarchical 3D Gaussian representation, characterized by the following steps: include: Obtain the dynamic image sequence of the target, and initialize three-dimensional Gaussian points in normal space based on the vertices of the SMPL human body template; Starting with the initial 3D Gaussian points as the innermost layer, a hierarchical Gaussian structure is constructed from coarse to fine. Conditional dependencies are established between each layer through parent-child node connections. This hierarchical Gaussian structure is constructed in stages based on the backpropagation gradient received by each Gaussian point during training, as follows: The Gaussian points obtained by initializing the 3D Gaussian representation with SMPL human body template points are used as the Gaussian points of the first layer. Based on the number of layers n layer Divide the entire training process into equal parts. n layer - 1 training phase, in which the first i The training phase is used to construct the first... i + 1 layer sub-Gaussian representation; In the i In each training phase, the gradient update based on backpropagation is fixed. i The parameters of all layers before the Gaussian point are updated according to the gradient. i Gaussian parameters of the layer; For the i For Gaussian points in a layer whose Gaussian gradient value is greater than a set threshold, a child Gaussian node is assigned to them. This child Gaussian node inherits the attributes of its parent node and also participates in the optimization of the current stage. The child node is then added to the next layer. i + 1 layer of Gaussian points; The preceding frame set is sampled and the layered motion conditions are obtained. Based on the layered motion conditions, the transformation parameters of rigid and non-rigid body deformations at each Gaussian point are predicted using rigid body transformation networks and non-rigid body transformation networks, as follows: The rigid and non-rigid deformation results of the first layer of Gaussian points are transformed into the observation space according to the skeleton parameters of the current frame using the linear hybrid skin LBS algorithm. Following the hierarchical structure, the subsequent sub-layers are traversed sequentially. The transformation result of the parent node of each sub-layer Gaussian point, the hierarchical motion condition of the parent node, and the spatial position of the child node itself are input into the rigid body transformation network and the non-rigid body transformation network respectively to predict the residuals of the rigid body and non-rigid body transformations of the child node relative to the parent node. Based on the rigid and non-rigid transformations of the current child node corresponding to the parent node, and the predicted residuals of the rigid and non-rigid transformations of the child node relative to the parent node, the rigid and non-rigid transformations of the current child node are obtained. The layered Gaussian points are uniformly mapped to the observation space according to the predicted transformation parameters, and the three-dimensional Gaussian rendering technology is used to synthesize images from any viewpoint.

2. The modeling and rendering method based on hierarchical 3D Gaussian representation according to claim 1, characterized in that, The aforementioned layered motion conditions include: The coarse-grained skeleton motion condition is the change in skeleton parameters between adjacent frames; The fine-grained point-level velocity condition is the difference in position of the human body Gaussian point in the observation space between two consecutive frames.

3. The modeling and rendering method based on hierarchical 3D Gaussian representation according to claim 1, characterized in that, Rigid body transformation is a transformation matrix describing the Gaussian point from standard space to the observation space; non-rigid body transformation is a three-dimensional vector describing the positional offset of each Gaussian point in three-dimensional space under standard space.

4. The modeling and rendering method based on hierarchical 3D Gaussian representation according to claim 1, characterized in that, The rigid body transformation and non-rigid body transformation settings for the Gaussian points in the first layer are as follows: The position of each initial Gaussian point is obtained by estimating the human pose using SMPL and input into LBS MLP. The LBS MLP predicts and outputs the weight of each Gaussian point for each human joint. Based on the predicted weights, the weighted sum of all human joints is calculated, and the rigid body transformation of each Gaussian point in the first layer is obtained. The non-rigid transformation of the first layer of Gaussian points is set to 0.

5. The modeling and rendering method based on hierarchical 3D Gaussian representation according to claim 1, characterized in that, During the transformation parameter prediction process: For the image sequence of the target, multiple historical frames are sampled from the current frame backward at different sampling intervals to form a set of preceding frames; All the layered motion conditions obtained by downsampling at all sampling intervals are used as the transformation parameters for predicting Gaussian points in both rigid body transformation networks and non-rigid body transformation networks.

6. The modeling and rendering method based on hierarchical 3D Gaussian representation according to claim 1, characterized in that, During the transformation parameter prediction process: The point-level velocity conditions of the nearest multiple points for each Gaussian point are concatenated and input into the non-rigid transformation network to predict the residuals of the non-rigid transformation.

7. A modeling and rendering device based on hierarchical 3D Gaussian representation, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the modeling and rendering method based on hierarchical 3D Gaussian representation as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the modeling and rendering method based on hierarchical 3D Gaussian representation as described in any one of claims 1-6.

Images