A multi-level-of-detail training method based on Gaussian splatting

Abstract

The present invention discloses a multi-level detail training method based on Gaussian splashing, which relates to the fields of virtual reality and augmented reality technology. In application scenarios such as virtual reality and augmented reality, different levels of Gaussian models are used for rendering targets at different distances within the field of view, which can accelerate the rendering of the model in the scene. First, the number of points in the point cloud is graded, and then the Gaussian splashing models of the target scene or object are trained in order from small to large according to the number of points in the point cloud, respectively as Gaussian splashing models of different levels. In actual application scenarios, the appropriate level is selected for rendering according to the distance between the target and the observer to save computing resources.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual reality and augmented reality, and in particular to a multi-level-of-detail training method based on Gaussian splashing. Background Art

[0002] Patent No. 202310041887.8 provides a large-scene LOD cloud rendering method. This technology realizes large-scene multi-level of detail cloud rendering, including the following steps: converting the cloud rendering original model and original map into a multi-level of detail model and a multi-level of detail map respectively, and calculating the error value of the multi-level of detail model; determining the pre-rendered multi-level of detail level used by the multi-level of detail model and the multi-level of detail map respectively; and performing real rendering according to the error value, the multi-level of detail model and the multi-level of detail map corresponding to the pre-rendered multi-level of detail level to obtain the scene model.

[0003] This method uses an explicit mesh model for modeling. The modeling process requires a lot of manual carving, which consumes a lot of labor costs and time costs. In addition, the model occupies a large amount of storage space. The memory usage of some rendering tasks can reach hundreds of GB or even higher, which requires the help of cloud rendering.

[0004] The paper, "GaussianAvatars: Photorealistic Head Avatars with Rigged 3DGaussians," provides a training method for dynamic 3D Gaussian splatter models. This method drives the 3D Gaussian model through a pre-set mesh model, eliminating the need for complex neural networks for training and rendering. By optimizing the parameters of the Gaussian point cloud model and combining cloning and pruning operations to adjust the number of Gaussian points, high-quality real-time free-viewpoint image rendering is achieved.

[0005] This method optimizes all Gaussian points with each Gaussian iteration, resulting in redundant calculations for frames with similar data. Furthermore, only one set of high-precision Gaussian models is trained for each target object. This model can render high-level details when the camera is close to the target object. However, when the camera is far away, detailed information is no longer necessary, so using this set of Gaussian models will result in redundant computations.

[0006] By default, Gaussian splatting uses the same number of points to render objects at different distances. This can waste GPU computing resources and affect the performance of the VR experience. Therefore, implementing lightweight dynamic Gaussian training is an urgent problem to be solved. Summary of the Invention

[0007] In view of this, the present invention provides a multi-level detail training method based on Gaussian splashing, which can aim to reduce the computational complexity of human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and train human body Gaussian models at multiple detail levels to improve rendering efficiency.

[0008] To achieve the above object, the technical solution of the present invention includes the following steps:

[0009] Step 1: Obtain multi-view and multi-pose image data for the target human body, and construct a corresponding mesh model for each pose. The mesh model is composed of triangular faces covering the target human body; the mesh model corresponding to the neutral pose is the neutral mesh model, and the vertices of all triangular faces in the neutral mesh model are taken as the positions of the initial Gaussian points to obtain the initialized Gaussian point cloud data.

[0010] Step 2: Set the maximum number of Gaussian point clouds to N n , the corresponding level is LOD n ; The lightweight training process is completed to obtain the corresponding level LOD n Dynamic Gaussian splash model; the initialization value of n is greater than or equal to 3.

[0011] Execute the lightweight training process; the lightweight training process is specifically as follows:

[0012] S1: Take the image data corresponding to the neutral posture for Gaussian training to obtain a Gaussian model that is strictly aligned with the neutral grid model, which is recorded as the trained neutral Gaussian splash model. Each Gaussian point is bound to a triangular face of the neutral grid model.

[0013] S2: Random iterative training of the driving effect of the neutral Gaussian model trained in S1 under various other postures. Each Gaussian point in the neutral Gaussian splash model is deformed into the current posture Gaussian splash model according to the translation vector and rotation matrix of its bound triangle face; the rotation degree of the mesh model of the current posture relative to each triangle face of the neutral mesh model is calculated, and the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken, and recorded as the triangle face to be optimized. The Gaussian points bound to the triangle face to be optimized are selected to participate in Gaussian training, and the Gaussian parameters of other Gaussian points are frozen; after the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained;

[0014] In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splatter model needs to be densified and pruned based on the gradient value; when the number of Gaussian points reaches the set upper limit, the Gaussian densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed.

[0015] Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level LOD is read. n+1 Dynamic Gaussian splash model, and freeze the Gaussian parameters; set the maximum number of Gaussian point clouds to N n =N n+1 +δ n , the corresponding level is LOD n , continue to execute the lightweight training process, and obtain the corresponding level LOD after the lightweight training process is completed n Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process.

[0016] Outputs a dynamic Gaussian splatter model at each level.

[0017] Furthermore, the method further comprises the following steps:

[0018] After training the dynamic Gaussian splash model at different levels of detail, select the dynamic Gaussian splash model at different levels based on the distance between the object and the camera; the selected LOD level is calculated according to the following formula:

[0019]

[0020] Among them, X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, n is the total number of LOD levels, and floor means rounding down, that is, calculating the LOD level that should be used.

[0021] Furthermore, for each other posture, the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken and recorded as the triangle face to be optimized, specifically:

[0022]

[0023] θ=arccos((trace(Δ)-1) / 2)

[0024] Where Δ is the median value, Represents the world rotation matrix of the i-th triangle of the neutral mesh model, represents the world rotation matrix of the i-th triangle of a certain other pose mesh model, trace is the trace of the calculated matrix, and the final θ represents the angle between the two triangles belonging to the neutral mesh model and the other pose mesh model; when θ is greater than the set threshold, the corresponding triangle is recorded as the triangle to be optimized.

[0025] Furthermore, the indices of the triangles whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds a set threshold are taken as the triangles to be optimized, and the set threshold is taken as 60°.

[0026] Preferably, in the Gaussian training of S1 and S2, the loss function of the Gaussian training consists of an L1 loss of the color value between the rendered image and the real image and a structural consistency loss.

[0027] Preferably, n is set to 3, and a Gaussian splash model of LOD3 to LOD0 levels is finally obtained; wherein N3 is set to 20,000, and δ2 to δ0 are both 10,000.

[0028] Beneficial effects:

[0029] The present invention provides a multi-level-of-detail training method based on Gaussian splatting, which constructs a dynamic Gaussian splatting model for multi-level detail rendering. A dynamic Gaussian splatting model for multi-level detail rendering involves training Gaussian splatting models with different point counts for the same object, used to render images with varying degrees of detail. A Gaussian model with the fewest points is first trained as the model for the lowest level of detail. After training is complete, the number of points is further increased based on this trained model to train a finer model for the next level, and so on. The number of levels of models and the number of points at each level can be defined based on the actual scene conditions. When an object in a scene moves away from the user, the visible detail decreases compared to objects closer to the user. However, by default, Gaussian splatting uses the same point cloud to render objects at different distances, which can waste GPU computing resources and affect the performance of the virtual reality experience. Therefore, the dynamic Gaussian splatting model is graded based on the number of points in the point cloud, depending on the distance of the object from the camera. Dynamic Gaussian splatting models with more points are used for rendering when the object is closer to the camera, while dynamic Gaussian splatting models with fewer points are used when the object is closer to the camera. The present invention reduces the computational complexity of human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and trains human body Gaussian models at multiple detail levels to improve rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Neutral image data for the example;

[0031] Figure 2 Select the effect for the triangles with a large degree of rotation (the left image shows the neutral posture, the right image shows other postures);

[0032] Figure 3 This is the algorithm flow chart (taking 4-level LOD as an example);

[0033] Figure 4 The following is a graph showing the training results of different LOD models (taking level 4 LOD as an example). DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] The present invention aims to reduce the computational complexity of human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and train human body Gaussian models at multiple levels of detail to improve rendering efficiency.

[0036] The Gaussian model for multi-level detail rendering involves training dynamic Gaussian splatter models with varying numbers of points for the same object, enabling rendering of images at varying levels of detail. A Gaussian model with the fewest points is trained first, serving as the model for the lowest level of detail. After training is complete, the number of points is further increased based on this trained model to train the next level of finer detail, and so on. The number of levels of models and the number of points at each level can be defined based on the actual scene conditions. When an object in a scene moves away from the user, the visible detail decreases compared to objects closer to the user. However, by default, Gaussian splatter uses the same point cloud to render objects at different distances, which can waste GPU computing resources and affect VR performance. Therefore, the Gaussian splatter model is graded based on the number of points in the point cloud, depending on the distance of the object from the camera. Dynamic Gaussian splatter models with higher points are used for rendering closer objects, while models with lower points are used for rendering closer objects.

[0037] The current mainstream approach for training dynamic Gaussian models is to use multi-view, multi-pose human images and optimize the parameters of all Gaussian points simultaneously. This means that even if the pose differences between two frames are minimal, the Gaussian point parameters still need to be optimized again, which significantly increases the computational workload. Therefore, training in each pose optimizes only the parameters of selected Gaussian points, avoiding the need to optimize all Gaussian points and effectively reducing the computational workload.

[0038] For human body dynamic Gaussian, the present invention can achieve faster training and rendering speed and higher rendering quality with smaller memory usage, further enhancing user experience.

[0039] The present invention provides a lightweight Gaussian splatter model training method for multi-level-of-detail rendering, which is used to save GPU computing resources during training and rendering. It can be applied in a wide range of fields, including but not limited to game development, computer graphics, medical image processing, engineering modeling and other fields.

[0040] The Gaussian splattering algorithm is a method for representing 3D scenes and rendering new views. It first reconstructs the Gaussian point cloud data of the target object and the position of the shooting camera. Next, it optimizes the parameters of the Gaussian point cloud (including position, rotation, scale, transparency, color, quantity, etc.) and finally uses rasterization to render the Gaussian point cloud on the target perspective image.

[0041] Step 1: Acquire multi-view, multi-pose images of the target human body. Before training, a multi-view, multi-pose dataset of the target human body must be established, and a homotopological mesh model corresponding to each action must be obtained. Multi-poses include, but are not limited to, leg raising, bending, leaning back, raising arms to the side, and bending elbows, and must encompass as many limb movements as possible. Multi-view refers to arranging multiple cameras in the scene space to obtain images of the human body from different angles. The preferred approach is to evenly arrange multiple cameras around the human body to capture all the detailed information about the human body.

[0042] A corresponding mesh model is constructed for each posture. The mesh model consists of triangular faces covering the target body. The mesh model corresponding to the neutral posture is called the neutral mesh model. The vertices of all triangular faces in the neutral mesh model are used as the positions of the initial Gaussian points to obtain the initial Gaussian point cloud data.

[0043] Step 2: Set the maximum number of Gaussian point clouds to Nn, and the corresponding level is LODn; after the lightweight training process, obtain the dynamic Gaussian splash model corresponding to level LODn; the initialization value of n is greater than or equal to 3.

[0044] Execute the lightweight training process; the lightweight training process is specifically as follows:

[0045] S1: Perform Gaussian training on the image data corresponding to the neutral posture to obtain a Gaussian model that is strictly aligned with the neutral grid model. This is recorded as the trained neutral Gaussian splash model. Each Gaussian point is bound to a triangular face of the neutral grid model.

[0046] S2: Random iterative training of the driving effect of the neutral Gaussian model trained in S1 under various other postures. Each Gaussian point in the neutral Gaussian splash model is deformed into the current posture Gaussian splash model according to the translation vector and rotation matrix of its bound triangle face; the rotation degree of the mesh model of the current posture relative to each triangle face of the neutral mesh model is calculated, and the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken, and recorded as the triangle face to be optimized. The Gaussian points bound to the triangle face to be optimized are selected to participate in Gaussian training, and the Gaussian parameters of other Gaussian points are frozen; after the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained.

[0047] In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splatter model needs to be densified and pruned based on the gradient value. When the number of Gaussian points reaches the set upper limit, the Gaussian densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed.

[0048] This step is a specific implementation method of lightweight Gaussian training, which can effectively reduce the number of parameters in each step of Gaussian iterative optimization, thereby improving the efficiency of training. Before training, it is necessary to establish a multi-view and multi-pose dataset of the target human body and obtain the same topological mesh model corresponding to each action. Define the first frame as the neutral pose of the dataset, Figure 1 This is a multi-view dataset for the neutral pose. The data for other poses are similar. Compare each mesh model for other poses with the neutral mesh model, calculate the index value of the triangle face with a larger rotation relative to the neutral mesh model and save it. This data is used as a reference for dynamic training to select the Gaussian points for optimization. Calculate according to formulas (1) and (2):

[0049]

[0050] θ=arccos((trace(Δ)-1) / 2) (2)

[0051] Where Δ is the median value, Represents the world rotation matrix of the i-th triangle of the neutral model (which can be calculated based on the spatial positions of the three vertices of the triangle), The world rotation matrix of the i-th triangle of a certain pose model is represented by trace. The resulting θ represents the angle between the two triangles. A threshold of 60° is set, and the indices i of the triangles that are greater than the threshold are recorded (and stored). Figure 2 This is the selection result, where the left side is the neutral posture mesh and the right side is the other posture meshes.

[0052] The training process first trains the image data of the neutral posture to obtain a human body Gaussian model that is strictly aligned with the neutral posture mesh model. Each Gaussian point is bound to a triangular face of the mesh model.

[0053] Next, we train image data for other poses and calculate the translation vector and rotation matrix of each triangle of the mesh model relative to the neutral model in other poses. Each Gaussian point trained in the first step deforms the neutral Gaussian model into a Gaussian model of the new pose based on the translation vector and rotation matrix of the triangle to which it is bound (calculated based on the three vertices of the triangle). Based on the previously calculated triangle indices with the largest degree of rotation for each pose, the Gaussian points bound to these triangles are selected for optimization, and the position, rotation, scale, color, and transparency parameters of the other Gaussian points are frozen. The training loss function consists of the L1 loss of the color value between the rendered image and the real image and the structural consistency loss. Through a large amount of pose data, the parameters of the Gaussian model are optimized so that it can vividly express these poses. After the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any pose is obtained.

[0054] Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level LOD is read. n+1 Dynamic Gaussian model, and freeze the Gaussian parameters; set the maximum number of Gaussian point clouds to N n =N n+1 +δ n , the corresponding level is LOD n , continue to execute the lightweight training process, and obtain the corresponding level LOD after the lightweight training process is completed n Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process;

[0055] This step reconstructs models of different levels of detail (LOD) by limiting the number of Gaussian point clouds in different levels, which are used for rendering different scene requirements. Figure 3 Taking level 4 LOD as an example, the overall process of the algorithm is introduced. The final output includes four levels of models: LOD3, LOD2, LOD1, and LOD0. First, the LOD3 model is trained, which is the lowest quality model. The maximum number of point clouds for the lowest LOD model can be selected according to actual conditions. Training is performed using the lightweight training method mentioned above. The training process limits the number of Gaussian point clouds to a maximum of 20,000. The loss function consists of the L1 loss and structural consistency loss of the color values ​​between the rendered image and the real image. After training is completed, the parameters of the Gaussian point cloud of the LOD3 model are frozen, including Gaussian point position, color, rotation, and transparency information. The LOD2 model is trained based on the LOD3 model. On the basis of the LOD3 model, 10,000 Gaussian points are added for training, and a model with 30,000 Gaussian points is output. This is repeated and so on. The final LOD0 model, as the model with the highest level of detail, reaches 50,000 Gaussian points. Figure 2 The number of LODs and Gaussian points of the process are not limited to this and can be adjusted according to actual usage.

[0056] The total number of LOD levels, the number of initial Gaussian points, and the difference in the number of Gaussian points between each LOD level are not fixed to the values ​​in the above example. Taking 6 levels of LOD as an example, the final output includes four models: LOD5, LOD4, LOD3, LOD2, LOD1, and LOD0. The LOD5 model is trained first, and the training process limits the number of its Gaussian point clouds to a maximum of 50,000. The Gaussian models of the other LODs are then trained step by step. LOD4 adds 15,000 Gaussian points to LOD5, LOD3 adds 10,000 Gaussian points to LOD4, LOD2 adds 10,000 Gaussian points to LOD3, LOD1 adds 5,000 Gaussian points to LOD2, and LOD0 adds 3,000 Gaussian points to LOD1.

[0057] After training the dynamic Gaussian models of different levels of detail, the dynamic Gaussian models of different levels are selected according to the distance between the object and the camera. The actual selected LOD level can be calculated according to formula (3),

[0058]

[0059] Where X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, n is the total number of LOD levels, and floor indicates rounding down to the nearest integer. This calculates the appropriate LOD level. When an object is far from the user, a model with fewer points is used for rendering; when an object is close to the user, a model with more points is used for rendering to save computing resources. Figure 4 Shows detailed comparison of different LOD models.

[0060] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-level detail training method based on Gaussian splatting, characterized in that: The steps include: Step 1: Obtain multi-view and multi-pose image data for the target human body, and construct a corresponding mesh model for each pose. The mesh model consists of triangular faces covering the target human body. The mesh model corresponding to the neutral pose is called the neutral mesh model. The vertices of all triangular faces in the neutral mesh model are used as the positions of the initial Gaussian points to obtain the initial Gaussian point cloud data. Step 2: Set the maximum number of Gaussian point clouds to , the corresponding level is ; The lightweight training process ends and the corresponding level is obtained Dynamic Gaussian splash model; n is the total number of LOD levels, and the initialization value of n is greater than or equal to 3; Execute a lightweight training process; the lightweight training process is specifically as follows: S1: Perform Gaussian training on the image data corresponding to the neutral posture to obtain a Gaussian model that is strictly aligned with the neutral grid model. This is recorded as the trained neutral Gaussian splash model. Each Gaussian point is bound to a triangular face of the neutral grid model. S2: Random iterative training of the driving effect of the neutral Gaussian model trained in S1 under various other postures. Each Gaussian point in the neutral Gaussian splash model is deformed into the current posture Gaussian splash model according to the translation vector and rotation matrix of its bound triangle face; the rotation degree of the mesh model of the current posture relative to each triangle face of the neutral mesh model is calculated, and the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken, and recorded as the triangle face to be optimized. The Gaussian points bound to the triangle face to be optimized are selected to participate in Gaussian training, and the Gaussian parameters of other Gaussian points are frozen; after the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained; In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splatter model needs to be densified and pruned based on the gradient value. When the number of Gaussian points reaches the set upper limit, the Gaussian densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed. Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level is read. Dynamic Gaussian splash model, and freeze the Gaussian parameters; set the maximum number of Gaussian point clouds to , the corresponding level is , continue to execute the lightweight training process, and obtain the corresponding level after the lightweight training process is completed Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process; Outputs a dynamic Gaussian splatter model at each level.

2. The multi-level detail training method based on Gaussian splatting according to claim 1, characterized in that: The method further comprises the steps of: After training the dynamic Gaussian splash model at different levels of detail, select the dynamic Gaussian splash model at different levels based on the distance between the object and the camera; the selected LOD level is calculated according to the following formula: Among them, X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, and floor means rounding down, that is, calculating the LOD level that should be used.

3. The multi-level detail training method based on Gaussian splatting according to claim 1, characterized in that: In each of the other postures, the index of the triangle face whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken and recorded as the triangle face to be optimized, specifically: in is the middle value, Represents the world rotation matrix of the i-th triangle of the neutral mesh model, Represents the world rotation matrix of the i-th triangle of a certain other posture mesh model. Trace is the trace of the calculated matrix. The final result is Indicates the angle between the two triangles belonging to the neutral mesh model and the other posture mesh model; when When it is greater than the set threshold, the corresponding triangle is recorded as the triangle to be optimized.

4. The multi-level detail training method based on Gaussian splatting according to claim 3, characterized in that: The triangle face index of the mesh model corresponding to the current posture whose rotation degree relative to the neutral mesh model exceeds a set threshold is recorded as the triangle face to be optimized, and the set threshold is 60°.

5. The multi-level-of-detail training method based on Gaussian splatting according to any one of claims 1 to 4, characterized in that: In the Gaussian training of S1 and S2, the loss function of the Gaussian training is composed of the L1 loss of the color value between the rendered image and the real image and the structural consistency loss.

6. The multi-level detail training method based on Gaussian splatting according to claim 1, characterized in that: Set n to 3, and you will get Level Gaussian splash model; where the setting is 20000, Both are 10,000.

Images