Multi-level-of-detail training method based on Gaussian splashing

Abstract

The invention discloses a multi-level-of-detail training method based on Gaussian splashing, relates to the technical field of virtual reality and augmented reality, and aims to realize rendering acceleration of models in scenes by adopting Gaussian models of different levels for targets at different distances in a visual field range in application scenes of virtual reality, augmented reality and the like. Firstly, the number of points in the point cloud is graded, and then according to the number of the points in the point cloud, Gaussian splash models of a target scene or object are sequentially trained from small to large to serve as Gaussian splash models of different hierarchies. In an actual application scene, according to the distance between a target and an observer, a proper hierarchy is selected for rendering, so that operation resources are saved.

Description

Technical Field

[0001] The invention relates to the technical field of virtual reality and augmented reality, and in particular to a multi-detail level training method based on Gaussian splashing. Background Art

[0002] Patent No. 202310041887.8 provides a large-scene LOD cloud rendering method, which realizes large-scene multi-level of detail cloud rendering, including the following steps: converting the cloud rendering original model and the original map into a multi-level of detail model and a multi-level of detail map, respectively, and calculating the error value of the multi-level of detail model; determining the pre-rendered multi-level of detail level used by the multi-level of detail model and the multi-level of detail map respectively; performing real rendering according to the error value, the multi-level of detail model and the multi-level of detail map corresponding to the pre-rendered multi-level of detail level to obtain the scene model.

[0003] This method uses an explicit mesh model for modeling. The modeling process requires a lot of manual carving, which consumes a lot of labor costs and time costs. In addition, the model occupies a large amount of storage space. The memory usage of some rendering tasks can reach hundreds of GB or even higher, which requires the help of cloud rendering.

[0004] Paper: GaussianAvatars:Photorealistic Head Avatars with Rigged 3DGaussians, provides a training method for dynamic 3D Gaussian splash models. The 3D Gaussian model is driven by a pre-set grid model, without the need for complex neural networks for training and rendering. By optimizing the parameters of the Gaussian point cloud model and combining cloning and pruning operations to adjust the number of Gaussian points, high-quality real-time free-viewpoint image rendering is achieved.

[0005] This method optimizes all Gaussian points in each Gaussian iteration, and redundant calculations will be performed for two frames of data that are not much different. And only one set of high-precision Gaussian models is trained for each target object. When the camera is close to the target object, this model can present good detail effects, but when the camera is too far away from the target object, too much detail information is no longer needed, so using this set of Gaussian models will generate redundant calculations.

[0006] It can be seen that by default, Gaussian splashing will use point clouds with the same number of points to render objects at different distances, which may lead to a waste of GPU computing resources and thus affect the performance of the virtual reality experience. Therefore, how to achieve lightweight dynamic Gaussian training is an urgent problem to be solved. Summary of the invention

[0007] In view of this, the present invention provides a multi-detail level training method based on Gaussian splashing, which can aim to reduce the computational complexity of human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and train human body Gaussian models at multiple detail levels to improve rendering efficiency.

[0008] To achieve the above object, the technical solution of the present invention comprises the following steps:

[0009] Step 1: Obtain multi-view and multi-pose image data for the target human body, and construct a corresponding mesh model for each pose. The mesh model is composed of triangular faces covering the target human body. The mesh model corresponding to the neutral pose is the neutral mesh model. The vertices of all triangular faces in the neutral mesh model are taken as the positions of the initialization Gaussian points to obtain the initialization Gaussian point cloud data.

[0010] Step 2: Set the maximum number of Gaussian point clouds to N n , the corresponding level is LOD n ; The lightweight training process ends and the corresponding level LOD is obtained n Dynamic Gaussian splash model; the initialization value of n is greater than or equal to 3.

[0011] Execute the lightweight training process; the lightweight training process is specifically as follows:

[0012] S1: Take the image data corresponding to the neutral posture for Gaussian training to obtain a Gaussian model that is strictly aligned with the neutral grid model, recorded as the trained neutral Gaussian splash model, and each Gaussian point is bound to a triangular face of the neutral grid model.

[0013] S2: Randomly iterate the training of the driving effect of the neutral Gaussian model trained in S1 under various other postures. Each Gaussian point in the neutral Gaussian splash model is deformed into the current posture Gaussian splash model according to the translation vector and rotation matrix of the bound triangle face; the rotation degree of the mesh model of the current posture relative to each triangle face of the neutral mesh model is calculated, and the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken, and recorded as the triangle face to be optimized. The Gaussian points bound to the triangle face to be optimized are selected to participate in Gaussian training, and the Gaussian parameters of other Gaussian points are frozen; after the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained;

[0014] In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splatter model needs to be densified and pruned based on the gradient value; when the number of Gaussian points reaches the set upper limit, the densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed.

[0015] Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level LOD is read. n+1 Dynamic Gaussian splash model, and freeze Gaussian parameters; set the maximum number of Gaussian point clouds to N n =N n+1 +δ n , the corresponding level is LOD n , continue to execute the lightweight training process, and obtain the corresponding level LOD after the lightweight training process is completed n Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process.

[0016] Outputs a dynamic Gaussian splatter model at each level.

[0017] Furthermore, the method further comprises the following steps:

[0018] After training the dynamic Gaussian splash models of different detail levels, select dynamic Gaussian splash models of different levels according to the distance between the object and the camera; the selected LOD level is calculated according to the following formula:

[0019]

[0020] Among them, X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, n is the total number of LOD levels, and floor means rounding down, that is, calculating the LOD level that should be used.

[0021] Furthermore, in each other posture, the index of the triangle face whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken and recorded as the triangle face to be optimized, specifically:

[0022]

[0023] θ=arccos((trace(Δ)-1) / 2)

[0024] Where Δ is the middle value, Represents the world rotation matrix of the i-th triangle of the neutral mesh model, represents the world rotation matrix of the i-th triangle of a certain other posture mesh model, trace is the trace of the calculated matrix, and the final θ represents the angle between the two triangles belonging to the neutral mesh model and the other posture mesh model; when θ is greater than the set threshold, the corresponding triangle is recorded as the triangle to be optimized.

[0025] Furthermore, the indices of the triangle faces whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds a set threshold are taken as the triangle faces to be optimized, and the set threshold is taken as 60°.

[0026] Preferably, in the Gaussian training of S1 and S2, the loss function of the Gaussian training consists of an L1 loss of color values ​​between the rendered image and the real image and a structural consistency loss.

[0027] Preferably, n is set to 3, and the final LOD is obtained 3 ~LOD 0 Gaussian splash model of level; where N is set 3 is 20000, δ 2 ~δ 0 Both are 10,000.

[0028] Beneficial effects:

[0029] The present invention provides a multi-level training method based on Gaussian splashing, and constructs a dynamic Gaussian splashing model with multiple details. The dynamic Gaussian splashing model for rendering at multiple detail levels refers to training Gaussian splashing models with different points for the same object, which is used to render images with different fineness. First, the Gaussian model with the least number of points is trained as the model with the lowest level of details. After the training is completed, the number of points is further increased based on the trained model, and a more refined next-level model is trained. By analogy, the level of the model and the number of points at each level can be defined according to the actual situation of the scene. When an object in the scene is far away from the user, the visible details will be reduced relative to the object close to the user. However, by default, Gaussian splashing uses point clouds with the same number of points to render objects at different distances, which may cause a waste of GPU computing resources, thereby affecting the performance of the virtual reality experience. Therefore, according to the distance between the object and the camera, the dynamic Gaussian splashing model is graded according to the number of point cloud points. The closer the distance to the camera is, the more points the dynamic Gaussian splashing model is used for rendering, and vice versa. The dynamic Gaussian splashing model with fewer points is used. The present invention reduces the amount of calculation for human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and trains human body Gaussian models at multiple detail levels to improve rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Neutral image data for the example;

[0031] Figure 2 Select the effect for the triangles with a large degree of rotation (the left picture shows the neutral posture, and the right picture shows other postures);

[0032] Figure 3 The following is the algorithm flow chart (taking level 4 LOD as an example);

[0033] Figure 4This is a graph of training results for different LOD models (taking level 4 LOD as an example). DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] The present invention aims to reduce the computational complexity of human body dynamic Gaussian training to achieve lightweight dynamic Gaussian training, and train human body Gaussian models at multiple detail levels to improve rendering efficiency.

[0036] The Gaussian model for multi-level detail rendering refers to training a dynamic Gaussian splash model with different points for the same object, which is used to render images with different fineness. First, the Gaussian model with the least number of points is trained as the model with the lowest level of detail. After the training is completed, the number of points is further increased based on the trained model to train a more refined next-level model. And so on. The level of the model and the number of points at each level can be defined according to the actual situation of the scene. When an object in the scene is far away from the user, the visible details will be reduced compared to the object close to the user. However, by default, Gaussian splash uses the same number of point clouds to render objects at different distances, which may cause a waste of GPU computing resources and affect the performance of the virtual reality experience. Therefore, according to the distance between the object and the camera, the Gaussian splash model is graded according to the number of point cloud points. The closer the distance to the camera, the more points of the dynamic Gaussian splash model are used for rendering, and vice versa.

[0037] For the training of dynamic Gaussian models, the current mainstream method is to use multi-view multi-pose human body images and optimize the parameters of all Gaussian points together. This means that if the human body postures of two frames are not much different, the Gaussian point parameters still need to be optimized again, which invisibly increases the amount of training calculations. Therefore, in the training under each posture, only the selected Gaussian point parameters are optimized, so as to avoid all Gaussian points from participating in the optimization process, which can effectively reduce the amount of training calculations.

[0038] For human body dynamic Gaussian, the present invention can achieve faster training and rendering speed and higher rendering quality with smaller memory usage, further enhancing user experience.

[0039] The present invention provides a lightweight Gaussian splatter model training method for multi-level-of-detail rendering, which is used to save GPU computing resources during training and rendering, and can be applied in a wide range of fields, including but not limited to game development, computer graphics, medical image processing, engineering modeling and other fields.

[0040] The Gaussian splattering algorithm is a method for representing 3D scenes and rendering new views. First, the Gaussian point cloud data of the target object and the position of the shooting camera are reconstructed. Then the parameters of the Gaussian point cloud (including position, rotation, scale, transparency, color, quantity and other information) are optimized. Finally, the Gaussian point cloud is rendered on the target perspective image using a rasterization method.

[0041] Step 1: Obtain multi-view and multi-pose images of the target human body. Before training, it is necessary to establish a multi-view and multi-pose dataset of the target human body and obtain the same topological mesh model corresponding to each action. Multi-poses include but are not limited to actions such as raising legs, bending, leaning back, raising arms to the side, bending elbows, etc., and need to include as many limb movements as possible. Multi-view means arranging multiple cameras in the scene space to obtain images of the human body from different angles. The preferred solution is to evenly arrange multiple cameras around the human body to obtain all the detailed information of the human body.

[0042] A corresponding mesh model is constructed for each posture, and the mesh model is composed of triangular faces covering the target human body; the mesh model corresponding to the neutral posture is the neutral mesh model, and the vertices of all triangular faces in the neutral mesh model are taken as the positions of the initialization Gaussian points to obtain the initialization Gaussian point cloud data;

[0043] Step 2: Set the maximum number of Gaussian point clouds to Nn, and the corresponding level is LODn; after the lightweight training process, a dynamic Gaussian splash model of the corresponding level LODn is obtained; the initialization value of n is greater than or equal to 3.

[0044] Execute the lightweight training process; the lightweight training process is specifically as follows:

[0045] S1: Take the image data corresponding to the neutral posture for Gaussian training to obtain a Gaussian model that is strictly aligned with the neutral grid model, which is recorded as the trained neutral Gaussian splash model. Each Gaussian point is bound to a triangular face of the neutral grid model.

[0046] S2: Random iterative training S1 trained neutral Gaussian model driving effect in various other postures, each Gaussian point in the neutral Gaussian splash model according to its bound triangle translation vector and rotation matrix, the neutral Gaussian splash model is deformed to the current posture Gaussian splash model; calculate the current posture mesh model relative to each triangle of the neutral mesh model rotation degree, take the current posture corresponding mesh model relative to the neutral mesh model rotation degree exceeds the set threshold triangle index, record it as the triangle to be optimized, select the Gaussian points bound to the triangle to be optimized to participate in Gaussian training, and freeze the Gaussian parameters of other Gaussian points; Gaussian training is completed to obtain a dynamic Gaussian splash model that can follow the mesh model to drive to any posture.

[0047] In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splash model needs to be densified and pruned based on the gradient value; when the number of Gaussian points reaches the set upper limit, the Gaussian densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed;

[0048] This step is a specific implementation method of lightweight Gaussian training, which can effectively reduce the number of parameters in each step of Gaussian iterative optimization, thereby improving the efficiency of training. Before training, it is necessary to establish a multi-view and multi-pose dataset of the target human body and obtain the same topological mesh model corresponding to each action. Define the first frame as the neutral pose of the dataset. Figure 1 is a multi-view dataset for neutral poses. The data for other poses are similar. Compare each mesh model for other poses with the neutral mesh model, calculate and save the index value of the triangle face with a larger rotation relative to the neutral mesh model. This data is used as a reference for dynamic training to select Gaussian points for optimization. Calculate according to formulas (1) and (2):

[0049]

[0050] θ=arccos((trace(Δ)-1) / 2) (2)

[0051] Where Δ is the middle value, Represents the world rotation matrix of the i-th triangle of the neutral model (which can be calculated based on the spatial positions of the three vertices of the triangle). Represents the world rotation matrix of the i-th triangle of some other posture model. trace is the trace of the calculated matrix. The final θ represents the angle between the two triangles. Set the threshold to 60° and record the triangle index value i that is greater than the threshold (save the index value). Figure 2 This is the selection result, where the left one is the neutral posture mesh and the right one is the other posture mesh.

[0052] The training process first trains the image data of the neutral posture to obtain a human body Gaussian model that is strictly aligned with the neutral posture mesh model, and each Gaussian point is bound to a triangular face of the mesh model.

[0053] Next, we train image data of other postures, calculate the translation vector and rotation matrix of each triangle face of the mesh model relative to the neutral model in other postures, and in the first step, each trained Gaussian point is transformed from the neutral Gaussian model to the new posture Gaussian model according to the translation vector and rotation matrix of the bound triangle face (calculated according to the three vertices of the triangle face). According to the previously calculated triangle face index with a large degree of rotation for each posture, the Gaussian points bound to these triangle faces are selected for optimization, and the position, rotation, scale, color, and transparency parameters of other Gaussian points are frozen. The training loss function consists of the L1 loss of the color value between the rendered image and the real image and the structural consistency loss. Through a large amount of posture data, the parameters of the Gaussian model are optimized so that it can vividly express these postures. After the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained.

[0054] Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level LOD is read. n+1 Dynamic Gaussian model, and freeze Gaussian parameters; set the maximum number of Gaussian point clouds to N n =N n+1 +δ n , the corresponding level is LOD n , continue to execute the lightweight training process, and obtain the corresponding level LOD after the lightweight training process is completed n Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process;

[0055] In this step, the number of Gaussian point clouds is restricted in stages to reconstruct models of different levels of detail (LOD, hereinafter referred to as LOD) for rendering in different scene requirements. Figure 3 Taking level 4 LOD as an example, the overall process of the algorithm is introduced. The final output includes four levels of models, including LOD3, LOD2, LOD1, and LOD0. First, the LOD3 model is trained, which is the model with the lowest quality. The maximum number of point clouds of the lowest LOD model can be selected according to the actual situation. The lightweight training method mentioned above is used for training. The training process limits the maximum number of Gaussian point clouds to 20,000. The loss function consists of the L1 loss and structural consistency loss of the color value between the rendered image and the real image. When the training is completed, the parameters of the Gaussian point cloud of the LOD3 model are frozen, including the Gaussian point position, color, rotation, and transparency information. The LOD2 model is trained based on the LOD3 model. On the basis of the LOD3 model, 10,000 Gaussian points are added for training, and a model with 30,000 Gaussian points is output. By analogy, the final LOD0 model, as the model with the highest detail, reaches 50,000 Gaussian points. Figure 2The number of LODs and Gaussian points of the process are not limited to this and can be adjusted according to actual usage.

[0056] The total number of LOD levels, the number of initialized Gaussian points, and the number of Gaussian points between each LOD level are not fixed to the values ​​in the above example. Taking 6 levels of LOD as an example, the final output includes four models: LOD5, LOD4, LOD3, LOD2, LOD1, and LOD0. First, the LOD5 model is trained. The training process limits the maximum number of its Gaussian point clouds to 50,000. Then continue to train the Gaussian models of other LODs step by step. LOD4 adds 15,000 Gaussian points on the basis of LOD5, LOD3 adds 10,000 Gaussian points on the basis of LOD4, LOD2 adds 10,000 Gaussian points on the basis of LOD3, LOD1 adds 5,000 Gaussian points on the basis of LOD2, and LOD0 adds 3,000 Gaussian points on the basis of LOD1.

[0057] After training the dynamic Gaussian models of different detail levels, the dynamic Gaussian models of different levels are selected according to the distance between the object and the camera. The actual selected LOD level can be calculated according to formula (3):

[0058]

[0059] Among them, X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, n is the total number of LOD levels, and floor means rounding down to calculate, and the LOD level that should be used can be calculated. When the object is far away from the user, a model with fewer points is used for rendering; when the object is close to the user, a model with more points is used for rendering to save computing resources. Figure 4 Shows detailed comparison of different LOD models.

[0060] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-level-of-detail training method based on Gaussian splashing, characterized in that: The steps include: Step 1: Obtain multi-view and multi-pose image data for the target human body, and construct a corresponding mesh model for each pose. The mesh model is composed of triangular faces covering the target human body. The mesh model corresponding to the neutral pose is the neutral mesh model. The vertices of all triangular faces in the neutral mesh model are taken as the positions of the initialization Gaussian points to obtain the initialization Gaussian point cloud data. Step 2: Set the maximum number of Gaussian point clouds to N n , the corresponding level is LOD n ; The lightweight training process ends and the corresponding level LOD is obtained n Dynamic Gaussian splash model; the initialization value of n is greater than or equal to 3; Execute a lightweight training process; the lightweight training process is specifically: S1: Take the image data corresponding to the neutral posture for Gaussian training to obtain a Gaussian model that is strictly aligned with the neutral grid model, which is recorded as the trained neutral Gaussian splash model. Each Gaussian point is bound to a triangular face of the neutral grid model. S2: Randomly iterate the training of the driving effect of the neutral Gaussian model trained in S1 under various other postures. Each Gaussian point in the neutral Gaussian splash model is deformed into the current posture Gaussian splash model according to the translation vector and rotation matrix of the bound triangle face; the rotation degree of the mesh model of the current posture relative to each triangle face of the neutral mesh model is calculated, and the triangle face index whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken, and recorded as the triangle face to be optimized. The Gaussian points bound to the triangle face to be optimized are selected to participate in Gaussian training, and the Gaussian parameters of other Gaussian points are frozen; after the Gaussian training is completed, a dynamic Gaussian splash model that can follow the mesh model to drive to any posture is obtained; In the Gaussian training of S1 and S2, each iteration first optimizes the Gaussian parameters, and then determines whether the Gaussian splash model needs to be densified and pruned based on the gradient value; when the number of Gaussian points reaches the set upper limit, the Gaussian densification and pruning steps of each iteration are skipped, and only the Gaussian parameter optimization step is executed until the training is completed; Step 3: n is decremented by 1, and then it is determined whether n is greater than or equal to 0. If so, the level LOD is read. n+1 Dynamic Gaussian splash model, and freeze Gaussian parameters; set the maximum number of Gaussian point clouds to N n =N n+1 +δ n , the corresponding level is LOD n , continue to execute the lightweight training process, and obtain the corresponding level LOD after the lightweight training process is completed n Dynamic Gaussian splash model; repeat step 3 until n is less than 0, and end the training process; Outputs a dynamic Gaussian splatter model at each level.

2. A multi-level-of-detail training method based on Gaussian splashing as claimed in claim 1, characterized in that: The method further comprises the steps of: After training the dynamic Gaussian splash models of different detail levels, select dynamic Gaussian splash models of different levels according to the distance between the object and the camera; the selected LOD level is calculated according to the following formula: Among them, X is the actual distance between the object and the camera, near is the closest distance between the object and the camera, far is the farthest distance between the object and the camera, n is the total number of LOD levels, and floor means rounding down, that is, calculating the LOD level that should be used.

3. A multi-level-of-detail training method based on Gaussian splashing as claimed in claim 1, characterized in that: In each other posture, the index of the triangle face whose rotation degree of the mesh model corresponding to the current posture relative to the neutral mesh model exceeds the set threshold is taken and recorded as the triangle face to be optimized, specifically: θ=arccos((trace(Δ)-1) / 2) Where Δ is the middle value, Represents the world rotation matrix of the i-th triangle of the neutral mesh model, R i k represents the world rotation matrix of the i-th triangle of a certain other posture mesh model, trace is the trace of the calculated matrix, and the final θ represents the angle between the two triangles belonging to the neutral mesh model and the other posture mesh model; when θ is greater than the set threshold, the corresponding triangle is recorded as the triangle to be optimized.

4. A multi-level-of-detail training method based on Gaussian splatting as claimed in claim 3, characterized in that: The triangular face index of the mesh model corresponding to the current posture whose rotation degree relative to the neutral mesh model exceeds the set threshold is recorded as the triangular face to be optimized, and the set threshold is 60°.

5. A multi-level-of-detail training method based on Gaussian splashing as claimed in any one of claims 1 to 4, characterized in that: In the Gaussian training of S1 and S2, the loss function of Gaussian training consists of L1 loss of color values ​​between the rendered image and the real image and structural consistency loss.

6. A multi-level-of-detail training method based on Gaussian splashing as claimed in claim 1, characterized in that: When n is set to 3, a Gaussian splash model of LOD3 to LOD0 level is finally obtained; wherein N3 is set to 20000, and δ2 to δ0 are both set to 10000.

Images