A three-dimensional scene model pruning method based on rendering importance

Abstract

The application discloses a three-dimensional scene model pruning method based on rendering importance, and belongs to the field of computer vision and robot technology.The method comprises the following steps: receiving key frames and key points needing to be reconstructed as input, initializing the key points as geometric bodies in a 3D scene, and constructing a key frame set in time sequence; a key frame is randomly selected as a target frame for rendering, the parameters of the geometric bodies are optimized through back propagation, so that the rendering result is as close to the real scene as possible; the rendering importance of all geometric bodies in the current scene is calculated, the geometric bodies with the rendering importance lower than a preset threshold are removed, and a three-dimensional scene rendering result after pruning is obtained; and the rendering importance is multiplied by the opacity factor, the perspective factor and the scaling factor of the geometric body in the current three-dimensional scene, and then normalized to obtain the rendering importance.The application effectively reduces the number of redundant geometric bodies under the premise of ensuring that the rendering effect is not affected, reduces the calculation cost and storage demand, and improves the rendering efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and robotics, and specifically relates to a three-dimensional scene model pruning method based on rendering importance. BACKGROUND

[0002] The application of three-dimensional reconstruction technology in real scenes faces many challenges, one of which is the accumulation of redundant geometry in the scene. When performing three-dimensional scene reconstruction, traditional methods usually construct scene models through sparse point clouds or other geometric representations. However, as information accumulates during the reconstruction process, these methods often produce a large number of overlapping, repetitive feature points or geometric elements, resulting in wasted computing resources and potentially introducing noise and unnecessary details. Especially in large-scale and dynamic environments, the model often produces redundant data. These redundancies not only cause storage and computing burdens, but also can affect the quality of the final reconstruction results.

[0003] Therefore, how to efficiently process and optimize these redundant data to avoid unnecessary repetition and invalid computation is the key to improving the performance of three-dimensional reconstruction. Solving this problem not only requires considering how to effectively represent each part of the scene, but also requires pruning and optimizing redundant information to improve computational efficiency, reduce storage requirements, and ultimately achieve more accurate three-dimensional reconstruction.

[0004] Existing classic three-dimensional reconstruction methods, such as feature point matching, Bundle Adjustment, point cloud compression, and multi-view geometry optimization, while being able to effectively handle redundant geometry, often have high computational complexity, large memory consumption, and precision loss problems, especially in large-scale datasets or large-scale scenes, making it difficult to achieve real-time and high efficiency. While deep learning-based methods have shown greater ability in handling redundant data, the training process requires a large amount of labeled data, and the high computational overhead of the model and its adaptability to large-scale environments are still bottlenecks that restrict its application. Therefore, how to reduce computational resources and storage requirements while maintaining high accuracy is a key problem in improving the efficiency of three-dimensional reconstruction. SUMMARY

[0005] To solve the above problems, the present application provides a three-dimensional scene model pruning method based on rendering importance, which aims to remove redundant information through the "rendering importance" of the geometry in the scene, thereby significantly improving computational efficiency and reducing storage requirements while ensuring three-dimensional reconstruction accuracy.

[0006] The three-dimensional scene model pruning method based on rendering importance provided by the present application includes the following steps:

[0007] Step 1, the initialization module receives the key frame and key point that need to be reconstructed, initializes the key point as a geometric body in the 3D scene, and maintains the key frame in chronological order into a key frame set. At this time, the geometric body in the scene is taken as the basic unit of the scene, and the three-dimensional representation is started to be constructed.

[0008] Step 2, the optimization module randomly selects a frame from the key frame set as a target frame, uses the target frame as a rendering reference; according to the camera pose of the target frame, the scene is rendered and compared with the real scene, and the rendering error loss is calculated; finally, the parameters of the geometric body are optimized through back propagation to improve the scene rendering effect, so that the rendering result is as close as possible to the real image of the target frame.

[0009] Step 3, after each back propagation ends, the rendering importance pruning module calculates the rendering importance of all geometric bodies in the current three-dimensional scene, and according to the set threshold τ, the geometric bodies with rendering importance lower than the threshold τ are removed to obtain the three-dimensional scene rendering result after pruning; it is judged whether there is a new key point to join, if yes, continue to step 1, otherwise, end the current three-dimensional scene rendering task.

[0010] Among them, the product of the opacity factor α, the perspective factor V and the scaling factor S of the geometric body in the current three-dimensional scene is obtained d = a · V · S, and then normalized to obtain the rendering importance score of the geometric body I = N(d), the normalization function x max and x min respectively correspond to the maximum value d max and the minimum value d min of the rendering importance of all geometric bodies in the current three-dimensional scene.

[0011] In step 1, in the 3D Gaussian rendering scene, the initialization module initializes the input key point as a 3D Gaussian geometric body, and the 3D Gaussian geometric body is represented by a Gaussian sphere. The parameters of each Gaussian sphere include position, covariance matrix, opacity and size scaling value.

[0012] In step 3, the opacity factor a of the geometric body is calculated as follows:

[0013]

[0014] Where w a is the opacity weight, and w a = 2.0; opacity is the opacity of the geometric body.

[0015] The perspective factor V of the geometric body is calculated as follows:

[0016]

[0017] wherein, θ is the angle between the center of the geometry and the axis of the camera frustum, is the vector from the geometry to the camera in the camera coordinate system of the current three-dimensional scene, is the camera direction of the current three-dimensional scene; V min is the minimum frustum weight, set V min = 0.3; w v is the frustum weight, set w v = 1.0;

[0018] The scaling factor S of the geometry is calculated as follows:

[0019]

[0020] wherein, w s is the scaling weight, set w s = 0.5; scale the size scaling value of the geometry, N(scale) is the size scaling value processed using the normalization function N(x), and x max and x min respectively correspond to the maximum and minimum values of the size scaling value of all geometries in the current three-dimensional scene.

[0021] The advantages and positive effects of the present application are as follows:

[0022] (1) The method of the present application effectively reduces the number of redundant geometries while ensuring that the rendering effect is not affected, reduces the computational overhead and storage requirements, improves the rendering efficiency, and improves the real-time processing capability of three-dimensional scene reconstruction. In addition, the method of the present application can adapt to scenes of different scales, whether static scenes or dynamic environments, and can achieve efficient scene reconstruction.

[0023] (2) The method of the present application generates geometries from key points, generates a three-dimensional scene by randomly extracting key frames, and reversely optimizes the geometry parameters, and then designs the opacity factor, the perspective factor and the scaling factor of the optimized geometry to effectively represent the rendering importance of the geometry. According to the rendering importance, the threshold-based automatic elimination of geometries is carried out. The present application improves the calculation method of the rendering importance, and the overall method is organically combined, reduces the computational resources and storage requirements, realizes more effective removal of redundant geometries, maintains the quality stability of the reconstructed scene, and improves the three-dimensional reconstruction efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the overall implementation flowchart of the three-dimensional scene model pruning method based on rendering importance of the embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0026] The present application discloses a three-dimensional scene model pruning method based on rendering importance. First, the key frames and corresponding key points that need to be reconstructed are received as input. The key points are initialized as 3D Gaussian geometry, and the input key frame set is used as the basis for mapping. Then, a key frame is randomly selected as the rendering target frame. By rendering the frame and training the parameters of the 3D Gaussian geometry, the rendering result is as close to the real scene as possible. In this way, the system gradually establishes a three-dimensional representation of the scene.

[0027] To solve the problem of redundant scene geometry, especially in large-scale or dynamic environments, the system defines and calculates the rendering importance of all geometries after training. Specifically, rendering importance measures the contribution of geometry in the scene. Lower importance geometries usually contribute less to the reconstruction result, so they can be removed. To achieve this goal, the method sets an importance threshold τ. Geometries below this threshold are automatically removed. In this way, the quality of scene reconstruction is maintained while the rendering rate is greatly improved.

[0028] Taking 3D Gaussian rendering as an example, as shown in Figure 1 The three-dimensional scene model pruning method based on rendering importance of the present application embodiment includes the following three steps. The system corresponding to the implementation of the present application method includes an initialization module, an optimization module, and a pruning module based on rendering importance.

[0029] Step 1, initialization. The initialization module performs initialization operations according to the input key point data, constructs the geometry in the 3D scene, obtains the initial three-dimensional scene, and organizes and maintains the input key frames into a key frame set in a certain way.

[0030] In the reconstruction of the 3D Gaussian scene, the initialization module processes the input key points and key frames, initializes the key points as 3D Gaussian geometry, and maintains the received key frames into a key frame set in chronological order. At this time, the 3D Gaussian geometry is used as the basic unit of the scene, and the three-dimensional representation is started.

[0031] Step 1.1, initialize 3D Gaussian geometry.

[0032] The input three-dimensional key points are first initialized as 3D Gaussian geometry, which is not only a simple point, but also a Gaussian distribution described by a covariance matrix. The covariance matrix reflects the spatial uncertainty of the point, which is usually related to the local environment of the point, such as the texture of the image and the angle of view of the camera. The key points received by the method of the present application represent objects in a three-dimensional scene, and each key point will be initialized as a geometry according to the position of the key point. The position of the geometry will be optimized in the optimization module.

[0033] In three-dimensional reconstruction, 3D Gaussian geometry can be used to represent irregular or complex scene shapes, especially in dynamic or sparse environments. These Gaussian geometries are represented by Gaussian spheres, and the parameters of each Gaussian sphere include the three-dimensional coordinates of the geometry position, covariance information, rotation matrix, opacity, scaling size, color or appearance parameters, etc. The color or appearance parameters are encoded using surface texture SH spherical harmonics. The parameters of the Gaussian sphere will be optimized through training. The three-dimensional coordinates and covariance information of the Gaussian sphere obtained by the embodiment of the present application are represented as follows:

[0034] p = (x, y, z)

[0035]

[0036] where p is the position of the 3D Gaussian geometry, represented by the center position of the Gaussian sphere, and initially set as the three-dimensional spatial coordinates corresponding to the key point; the covariance matrix Σ is a 3x3 matrix representing the uncertainty of the key point position, including the uncertainty of the three-dimensional point in different directions. xy ,σ xz ,σ zy represent the covariance between the x, y, and z coordinate axes, represent the variance in the x, u, and z coordinate axis directions.

[0037] Step 1.2, maintain a set of key frames.

[0038] The three-dimensional scene uses an ordered queue to maintain the received set of key frames. Each key frame usually contains key frame pose, key frame feature points, etc. As new key frames are added, the system will continuously expand this set; at the same time, the queue will also periodically delete expired key frames.

[0039] Step 2, the optimization module randomly selects a frame from the set of key frames maintained in step 1 as the target frame, and uses the target frame as the rendering reference. The 3D Gaussian scene is rendered according to the camera pose of the target frame, and compared with the real scene to calculate the rendering error loss. Finally, the parameters of the 3D Gaussian geometry are optimized through backpropagation to make the rendering result as close as possible to the real image of the target frame.

[0040] Step 2.1, Select target frame. A frame is randomly selected from the set of keyframes as the target frame to compute the error between the current scene and the real scene. Embodiments of the invention use a random selection strategy, i.e., a frame is randomly selected from the set of keyframes as the target frame.

[0041] Step 2.2, Scene rendering. According to the pose of the selected target frame, the projection matrix P of the camera is first calculated as follows:

[0042] P = K - [R | t]

[0043] where K is the intrinsic matrix of the camera, and R and t are the rotation matrix and translation vector, respectively.

[0044] Then, by projecting each Gaussian sphere in the 3D Gaussian scene, the position on the 2D image is obtained. This is done by matrix multiplication of the 3D position (x, y, z) of each Gaussian sphere with the camera projection matrix to get the 2D coordinates (u, v):

[0045]

[0046] After projection, the position of each Gaussian sphere in the scene in the image can be obtained.

[0047] Step 2.3, Compute rendering error loss. The goal of the loss function is to minimize the error between the rendered scene and the real image. Specifically, embodiments of the invention compare the rendered result of the target frame with the real image, and calculate the loss function as follows:

[0048]

[0049] where I r is the rendered image of the target frame, I gt is the real image from the target frame, SSIM(I r , I gt ) represents the structural similarity between the two images, and λ is a weight factor used for balancing.

[0050] Once the total loss is calculated, backpropagation can be used to calculate the gradient of the loss with respect to the 3D Gaussian geometry parameters such as position, covariance, opacity, scaling factor, etc., and the Adam optimization algorithm is used to update these parameters. By minimizing the loss function, the optimization process gradually adjusts the parameters of the Gaussian spheres so that the rendered result is closer to the true value.

[0051] Step 3, after each back propagation, the system calculates the rendering importance of all 3D Gaussian geometry in the scene. According to the importance value of these geometries, an importance threshold τ is set. For the geometries whose importance is lower than the threshold τ, they will be automatically removed. Through this process, the system can effectively remove redundant geometries, maintain the quality stability of the 3D Gaussian scene, and reduce the computational overhead and improve the rendering efficiency.

[0052] Step 3.1, the rendering importance pruning module based on the present application calculates a current rendering importance of all geometries according to their features including three-dimensional coordinates, opacity, size, texture, etc. in the current scene at the end of each iteration training, and performs pruning operation.

[0053] The embodiment of the present application multiplies the factors a, V and S of the geometries in the current three-dimensional scene to obtain the rendering importance d, and then normalizes d according to the maximum value d max and the minimum value d min of the rendering importance of all geometries in the current three-dimensional scene.

[0054] The embodiment of the present application calculates the rendering importance score I of a certain geometry as follows:

[0055] I=N(α·V·S)

[0056] Wherein: a is the opacity factor, V is the viewing angle factor, and S is the scaling factor; N(x) is a normalization function, which normalizes the input value s as follows:

[0057] Here x, x max and x min correspond to d, d max and d min respectively.

[0058] The opacity is crucial to the influence of the geometry on rendering. Since the system will be based on depth sorting when rendering, and then render in turn according to the transmittance, i.e. the remaining opacity. Therefore, the greater the opacity of the geometry, the greater the influence on the scene, and vice versa. The smaller the opacity of the geometry, the smaller the influence on the scene rendering, and the lower the importance. The embodiment of the present application calculates the opacity factor a as follows:

[0059]

[0060] Wherein w a =2.0 is the opacity weight, and opacity is the opacity of the 3D Gaussian geometry.

[0061] The most concerned object in three-dimensional reconstruction is usually the object near the camera center. Therefore, the method of the present application sets a view cone factor to represent the relative distance of the geometric body from the camera center, specifically using the cosine value of the angle θ between the geometric body center and the camera view cone axis. The larger the cosine value, the closer the geometric body is to the camera center, and the higher the importance. The view angle factor V is calculated as follows in the embodiment of the present application:

[0062]

[0063] wherein, is the vector of the 3D Gaussian geometric body to the camera in the camera coordinate system of the current scene, is the camera direction of the current scene, V min = 0.3 is the minimum view cone weight, w v is the view cone weight, which is set to w v = 1.0 in the embodiment of the present application.

[0064] The scaling factor S of the geometric body is calculated as follows:

[0065]

[0066] wherein, N(scale) is the normalization of the size scaling value of the geometric body. When using N(x) for normalization processing, x max and x min correspond to the maximum and minimum values of the size scaling values of all geometric bodies in the current three-dimensional scene, respectively. w s = 0.5 is the scaling weight, and scale is the scaling value of the 3D Gaussian geometric body. Both opacity and scale are assigned an initial value to be optimized in the initialization stage, and then optimized in the optimization module.

[0067] Step 3.2, the pruning module performs geometric body pruning based on the rendering importance.

[0068] After calculating the rendering importance of all 3D Gaussian geometric bodies, pruning of the 3D Gaussian is performed. According to the characteristic analysis of different indoor scene data sets, a suitable importance threshold τ is set. For the 3D Gaussian geometric bodies below this threshold, they are removed to obtain the rendering result of the pruned three-dimensional scene.

[0069] Then it is judged whether there is a new key point added. If yes, step 1 is continued to be executed, otherwise, the current three-dimensional scene rendering task is ended.

[0070] In general, the various embodiments of the disclosure can be implemented in hardware or special-purpose circuits, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software which can be executed by a controller, microprocessor or other computing device, Although the various aspects of the disclosure can be implemented in a number of manners, the embodiments presented herein are not limited to any particular manner in which such aspects can be implemented. It is to be understood that other aspects can be utilized and / or

[0071] In addition to the technical features described in the specification, they are known to those skilled in the art. The present application omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present application. The embodiments described in the above embodiments do not represent all embodiments consistent with the present application. Various modifications or variations made by those skilled in the art without creative labor on the basis of the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A method for rendering importance-based three-dimensional scene model pruning, characterized in that, Comprise: Step 1, the initialization module receives key frames and key points, initializes each key point as a geometric body in a three-dimensional scene, and maintains the received key frames in chronological order into a key frame set; Step 2, the optimization module randomly selects a frame from the current key frame set as the target frame, uses the camera pose of the target frame to render the scene, compares it with the real scene, calculates the rendering error loss, optimizes the parameters of the geometric body through back propagation, and improves the rendering effect of the scene; Step 3, after each back propagation is completed, the rendering importance pruning module calculates the rendering importance of all geometric bodies in the current three-dimensional scene, removes the geometric bodies with rendering importance lower than the threshold τ according to the set threshold τ, and obtains the rendering result of the pruned three-dimensional scene; judge whether there is a new key point added, if yes, continue to step 1, otherwise, end the current three-dimensional scene rendering task; wherein a product of an opacity factor a, a view factor V and a scale factor S of the geometry in the current three-dimensional scene is obtained as a rendering importance d = a · V · S, and a rendering importance score I = N(d) of the geometry is obtained by normalizing the rendering importance d = a · V · S, the normalizing function N being x max and x min corresponding to a maximum value d max and a minimum value d min of the rendering importance of all geometries in the current three-dimensional scene, respectively. The opacity factor α of the geometric body is calculated as follows: where w a is the opacity weight, set w a = 2.0; opacity is the opacity of the geometry; The view factor V of the geometric body is calculated as follows: where θ is the angle between the center of the geometry and the axis of the camera frustum, is the vector from the camera to the geometry in the camera coordinate system of the current three-dimensional scene, is the camera direction of the current three-dimensional scene; V min is the minimum frustum weight, set V min = 0.3; w v is the frustum weight, set w v = 1.0; The scaling factor S of the geometric body is calculated as follows: where w s is a scaling weight, set w s = 0.5; scale is a size scaling value of the geometry, N(scale) is a processing using a normalization function N(x) to the size scaling value, x max and x min correspond to a maximum value and a minimum value of the size scaling value of all geometries in the current three-dimensional scene, respectively.

2. The method of claim 1, wherein, In step 1, in the 3D Gaussian rendering scene, the initialization module initializes the input key points as 3D Gaussian geometric bodies, and the 3D Gaussian geometric bodies are represented by Gaussian spheres. The parameters of each Gaussian sphere include position, covariance matrix, opacity and size scaling value.

Images