A sparse mesh radiation field representation method based on point cloud initialization and depth supervision

Abstract

This invention discloses a sparse mesh radiation field representation method based on point cloud initialization and depth supervision. This radiation field can be used for rendering from new perspectives. The method includes obtaining at least two camera color images, obtaining two camera depth images, generating a 3D point cloud from the depth images, and generating rays from each pixel using camera parameters (including intrinsic and extrinsic parameters) to obtain a set of rays for all pixels. Since real-world scenes are often largely blank, this invention uses a sparse mesh to represent the scene, reducing memory costs. It utilizes the completed 3D reconstruction as a geometric constraint to enhance geometric learning capabilities and uses the completed 3D reconstruction as a guide for initialization, reducing the number of parameters during the mesh generation stage, improving optimization efficiency, and reducing storage costs.

Description

Technical Field

[0001] This invention relates to a scene radiation field representation technique, more specifically, a sparse mesh radiation field representation method based on point cloud initialization and depth supervision, which can be used for rendering new perspectives. Background Technology

[0002] Research on three-dimensional spatial perception and interaction has always been a hot topic. Whether it's robot localization, mapping, and navigation tasks, or XR tasks such as AR, VR, and MR, all rely on environmental perception and interaction. Especially in recent years, XR has become a market hotspot. On November 1, 2022, five departments, including the Ministry of Industry and Information Technology, issued the "Action Plan for the Integration and Development of Virtual Reality and Industry Applications (2022-2026)," proposing that by 2026, the overall scale of my country's virtual reality industry would exceed 350 billion yuan, and sales of virtual reality terminals would exceed 25 million units. XR stands for "Extended Reality," a collective term for various forms such as Virtual Reality, Augmented Reality, and Mixed Reality. XR requires the combination of hardware and software to create a virtual human-computer interaction environment. To achieve this environment, the core steps are mainly two: environmental perception and environmental interaction. First, a 3D reconstruction of the environment must be completed, i.e., active environmental perception; then, interaction within the known environment must be achieved, i.e., interactive environmental rendering. Current radiation fields have the following problems:

[0003] 1. Neural radiation fields based on multilayer perceptrons often require a long training time and are inefficient;

[0004] 2. Currently, supervision using purely visual images often results in a large number of images being required for geometry to be learned correctly;

[0005] 3. Radiation fields based on voxel meshes often require a large amount of memory. This invention provides an efficient radiation field representation method for 3D scenes, overcoming the above problems, and can be used for rendering from new perspectives, thus helping XR applications to develop better. Summary of the Invention

[0006] To address the aforementioned problems in existing radiation fields, this invention solves the corresponding problems through the following approaches. The core ideas of this invention lie in the following four points: 1. Voxel-based radiation field characterization; 2. Geometric initialization and sparse mesh; 3. Sampling method based on sparse mesh; 4. Depth supervision (geometric constraints).

[0007] This invention is achieved through the following technical solution:

[0008] This invention discloses a sparse mesh radiation field representation method based on point cloud initialization and depth supervision, comprising:

[0009] Obtain at least two camera color images;

[0010] Obtain two camera depth images and generate a 3D point cloud from the depth images;

[0011] Using camera parameters (including intrinsic and extrinsic parameters), generate rays for each pixel in all the acquired camera images to obtain a set of rays for all pixels;

[0012] The 3D point cloud is input into the occupancy network to generate an occupancy grid. Each grid vertex stores the occupancy probability (0-1) of that point. The occupancy grid is used as the geometric initialization prior of the voxel radiation field to sparsify the voxel grid in the voxel radiation field and obtain a sparse voxel radiation field.

[0013] All rays are sampled, and a sparse voxel grid is used as an auxiliary sampling method to optimize the sampling strategy. Only the non-blank parts of the sparse grid are sampled. For a sampling point, trilinear interpolation is used to interpolate the adjacent voxel vertices to obtain the geometric parameters and color parameters of each point.

[0014] Voxel rendering is performed on all sampling points on the ray to obtain RGB color images and depth images respectively;

[0015] A loss function (color constraint and geometric constraint) is constructed using the acquired color and depth images and the rendered RGB color and depth images;

[0016] Gradient propagation is performed using a loss function to optimize the parameters of the voxel radiation field until all parameters converge, ultimately yielding a sparse voxel radiation field representation of the current scene.

[0017] As a further improvement, the sparse voxel radiation field of the present invention is specifically as follows:

[0018] Sparse voxels are obtained by sparsifying dense voxel grids. Sparse voxels only retain the space occupied by objects in the scene, which aims to greatly reduce unnecessary information storage costs. At the same time, each voxel stores information including 1-dimensional geometric information and color information of 3 dimensions or more. The voxel grid and the stored geometric and color information together form the sparse voxel radiation field.

[0019] As a further improvement, the method of using sparse voxel mesh as an aid to optimize the sampling strategy described in this invention is as follows:

[0020] For a ray passing through a sparse voxel field in space, ray sampling skips all empty grids in space and only samples from the sparse grids. This method can significantly reduce the number of samples while ensuring sampling quality.

[0021] As a further improvement, the loss function constructed by the present invention using the acquired color image and depth image and the rendered RGB color image and depth image is specifically as follows:

[0022] The color constraints are formed by calculating the squared error between the RGB color image rendered from the radiation field and the actual color image captured. The depth constraints are formed by calculating the squared error between the depth image rendered from the radiation field and the actual depth image captured. The specific formulas are as follows:

[0023] ;

[0024] .

[0025] As a further improvement, the use of the occupied grid as the geometric initialization prior for the voxel radiation field described in this invention is specifically as follows:

[0026] Using the acquired depth images, a 3D point cloud is generated through image parameters. This point cloud is then processed through an occupancy network to generate an occupancy probability grid. An occupancy probability threshold is set, and grid cells with probabilities below the threshold are deleted, resulting in a sparse voxel radiation field. Furthermore, during initialization, the occupancy probability values ​​in the occupancy probability grid are used as initial geometric density values.

[0027] Compared to existing methods for representing neural radiation fields, the advantages of this invention are as follows:

[0028] 1. Since real-world scenes are often mostly blank, this invention uses sparse meshes to represent the scene, reducing memory costs;

[0029] 2. Since implicitly learning geometric structures through voxel rendering requires a large number of images, this invention utilizes the already completed 3D reconstruction as geometric constraints to enhance geometric learning capabilities.

[0030] 3. This invention utilizes the completed 3D reconstruction as a guide for initialization, reducing the number of parameters during the mesh generation stage, improving optimization efficiency and reducing storage costs.

[0031] 4. Because sparse meshes avoid invalid sampling in ray sampling, the number of samples required for each ray is greatly reduced. Therefore, this invention significantly improves the speed during both training and rendering, achieving a 1000-fold speedup for neural radiation fields based on multilayer perceptrons and a 5-fold speedup for radiation fields based on voxel meshes. During rendering, the speed of this invention can reach 20Hz, approaching real-time.

[0032] 5. Because this invention utilizes the 3D point cloud of the scene as a geometric prior, it can reconstruct the target effect with fewer images. This avoids the need for large-scale data collection and improves scene reconstruction efficiency. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the use of point clouds for sparse initialization of the mesh, as described in this invention.

[0034] 101 represents the 3D point cloud of the scene; 102 represents placing the 3D point cloud in the scene into a dense mesh according to the maximum bounding box; 103 represents using the 3D point cloud to sparsify the dense mesh to obtain a sparse mesh.

[0035] Figure 2 This is a schematic diagram illustrating the application of the present invention in rendering;

[0036] 204 represents the sparse mesh radiation field; 201 represents the current camera position; 202 represents the current ray's frontal direction; 203 represents the final rendered image (including color and depth maps);

[0037] Figure 3 This is a flowchart illustrating the algorithm application of the present invention during actual training and rendering. Detailed Implementation

[0038] This invention discloses a sparse mesh radiation field representation method based on point cloud initialization and depth supervision:

[0039] First, this invention uses a voxel-based scene representation, that is, using voxel meshes to represent the scene, avoiding the use of multilayer perceptrons, avoiding redundant network parameters and calculations, and improving speed. Simultaneously, this invention utilizes spherical harmonic coefficients as color parameters in the voxel mesh, effectively modeling the different colors observed at different viewing angles for the same point.

[0040] Secondly, this invention incorporates existing 3D reconstruction information from the scene (such as point clouds and depth maps). For example... Figure 1 As shown, 101 is the 3D point cloud for scene reconstruction. The scene point cloud is aligned with the dense voxel mesh 102. This invention uses the 3D point cloud as an initialization guide to remove meshes without point clouds in space. Finally, a sparse mesh as shown in 103 is used for subsequent training.

[0041] Furthermore, when rendering images using a mesh, such as Figure 2As shown in Figure 201, 201 represents the current camera position, and 202 represents the ray direction of the current pixel. By sampling points in space along the current pixel direction and weighting the sum of the color and distance of all sampled points, the color and depth information of the current pixel can be obtained. Sampling all pixels yields the camera's color and depth maps at the current position, as shown in Figure 203. Since the voxel mesh shown in Figure 204 is sparse, only the mesh regions traversed by the ray need to be sampled. Generally, a ray traverses a small number of meshes. Therefore, this sparse mesh-based sampling method can significantly accelerate the training convergence speed. Furthermore, during rendering, thanks to the rapid acceleration of the sparse mesh, the radiation field proposed in this invention can be rendered in real time.

[0042] Finally, the color and depth maps rendered from the current radiation field are compared with the real color and depth maps provided in the dataset. RGB and geometric errors are calculated separately, and gradients are calculated for each error. These gradients are then passed to each voxel vertex to optimize the voxel's geometric and color parameters, resulting in the final image. Figure 3 As shown, color constraints and geometric constraints work together to effectively train the radiation field of this invention, ultimately enabling the radiation field to render realistic scene images and structures.

[0043] The technical solution of the present invention will be further illustrated below through specific implementation examples:

[0044] Step 1: This invention uses sensors to acquire color images and depth images respectively, and records the acquired color images as... Depth image denoted as After data acquisition, an image localization algorithm is used to locate all acquired images and convert the depth images into a 3D point cloud of the scene, denoted as . The following formula is used for calculation in the conversion from depth image to 3D point cloud:

[0045]

[0046] Step 2: Using the positioned color and depth images, combined with the camera's intrinsic and extrinsic parameters, first convert each pixel of the color and depth images into a ray in the camera coordinate system. Then, using the camera's extrinsic parameters, represent each ray in the world coordinate system to obtain the set of rays in the world coordinate system corresponding to all image pixels.

[0047] Step 3: Put Occupation networks, where the occupancy grid is composed of a 3D convolutional neural network and a fully connected network, are used to generate an occupancy probability grid, denoted as . Let the resolution of the occupancy probability grid be... Each grid vertex stores the occupancy probability of the corresponding spatial point, ranging from 0 to 1, where 1 indicates that the point is occupied by an object and 0 indicates that the point is an empty area.

[0048] Step 4: Utilize the occupancy probability grid By selecting an appropriate threshold (e.g., 0.5), all grid cells with an occupancy probability less than 0.5 are removed, thus constructing a sparse voxel grid radiation field. Simultaneously, the geometric information of the vertices of the sparse grid radiation field is initialized using the occupancy probability value. This invention utilizes a... The tensor stores the sparse information of the mesh; for a single mesh in a dense mesh... That is, if the stored value in this tensor is -1, it indicates that the grid is empty. Therefore, there are a total of Each point stores a value greater than -1, and the stored value of these points is... The integers in the set are unique. Then use... The matrix represents the geometry of the scene, and the stored information is the probability that the current grid is occupied, denoted as . , The matrix represents the colors of the scene, where When stored directly using RGB, the value is 3. If represented using spherical harmonic coefficients, then... The values ​​can be 3, 12, 27, etc. These geometric parameters and color parameters together constitute the sparse voxel grid radiation field of this invention. After initialization, the geometric parameters in the sparse voxel grid radiation field are occupancy probabilities, and their initial values ​​are the values ​​in the corresponding occupancy probability grids. The color parameters are uniformly initialized to 0.

[0049] Step 5: Sample all rays from Step 2. The nearest and farthest points are pre-defined based on the scene. Generally, the nearest point is the origin of the current ray, and the farthest point is obtained from the intersection of the ray and the scene mesh. During ray sampling, the sparse voxel mesh obtained in Step 4 is used for acceleration. If the mesh corresponding to the current sampling point does not fall within the sparse voxel mesh, the current sampling point is skipped. Therefore, using the sparse voxel mesh can avoid invalid sampling during the sampling process and greatly speed up the sampling process while ensuring sampling quality. For sampling points that fall within the sparse voxel mesh, trilinear interpolation is performed on each sampling point to obtain the parameters (occupancy probability and color parameters) of the sampling point. The trilinear interpolation calculates the weight relationship between the current point and each vertex by using the distance from the current point to the voxel vertex. Generally, the closer the distance, the greater the weight. Finally, these weights are used to sum the parameters of the voxel vertices to obtain the parameters of the current sampling point.

[0050] Step Six: Perform voxel rendering on all the sampling points from Step Five. Assume a certain ray has been effectively sampled a total of [number missing]. Point, record the first The probability of each point being occupied The color is The depth distance is First, use the occupancy probability. The sampling weight of the current sampling point is calculated by multiplying the occupancy probability of the current point by the non-occupancy probabilities of all points preceding the current point. After calculating the weight, for depth rendering, the depth value of the current pixel is the weighted sum of the depth values ​​of all sampling points on the ray corresponding to the current pixel; for color rendering, the color value of the current pixel is the weighted sum of the color values ​​of all sampling points on the ray corresponding to the current pixel. Therefore, during voxel integration, the following formula is used for weight calculation and weighted summation to obtain the voxel rendering result of the ray:

[0051]

[0052]

[0053]

[0054] By performing voxel rendering on the rays corresponding to all pixels, a predicted image and a predicted depth image can be generated.

[0055] Step 7: Construct errors (color error and depth error) by comparing the predicted images from Step 6 with the actual images acquired in Step 1. The color error represents the difference in the RGB three channels of each pixel, and the depth error represents the difference in depth of each pixel. These errors correspond to the texture and geometric information in the scene, respectively. The formulas for calculating the color and depth errors are as follows:

[0056]

[0057]

[0058]

[0059] Step 8: Backpropagate the gradient of the error function to update the parameters in the sparse grid radiation field. When solving for the gradients of all errors, one approach is to use libraries like PyTorch that can automatically calculate the derivatives. Another approach is to theoretically derive the first derivative of each loss term with respect to each voxel parameter and then use optimization methods to correct all voxel parameters.

[0060] Step 9: Once the sparse mesh radiation field has completed parameter training, it means that the current radiation field can be used to characterize the current scene. At this time, if a brand new camera pose and camera intrinsic parameters are given, the present invention can use steps 2, 5 and 6 to render the camera image with any brand new pose and camera intrinsic parameters, thereby enabling the image rendering result to be obtained from any viewpoint in the scene.

[0061] The above examples are merely preferred embodiments of the present invention. The present invention is not limited to the above embodiments. Other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A sparse mesh radiation field representation method based on point cloud initialization and depth supervision, characterized in that, include: Obtain at least two camera color images; Obtain two camera depth images and generate a 3D point cloud from the depth images; Using camera parameters, including intrinsic and extrinsic parameters, generate rays for each pixel in all acquired camera images to obtain a set of rays for all pixels; The 3D point cloud is input into the occupancy network to generate an occupancy grid. Each grid vertex stores the occupancy probability of that point. The occupancy grid is used as the geometric initialization prior of the voxel radiation field to sparsify the voxel grid in the voxel radiation field and obtain a sparse voxel radiation field. All rays are sampled, and a sparse voxel grid is used as an auxiliary sampling method to optimize the sampling strategy. Only the non-blank parts of the sparse grid are sampled. For a sampling point, trilinear interpolation is used to interpolate the adjacent voxel vertices to obtain the geometric parameters and color parameters of each point. Voxel rendering is performed on all sampling points on the ray to obtain RGB color images and depth images respectively; A loss function, including color constraints and geometric constraints, is constructed using the acquired color and depth images and the rendered RGB color and depth images. Gradient propagation is performed using a loss function to optimize the parameters of the voxel radiation field until all parameters converge, ultimately yielding a sparse voxel radiation field representation of the current scene.

2. The sparse mesh radiation field representation method based on point cloud initialization and depth supervision as described in claim 1, characterized in that, The sparse voxel radiation field is specifically as follows: Sparse voxels are obtained by sparsifying dense voxel meshes. Sparse voxels retain only the space occupied by objects in the scene to reduce unnecessary information storage costs. At the same time, each voxel stores information including 1-dimensional geometric information and color information of 3 dimensions or more. The voxel mesh and the stored geometric and color information together form the sparse voxel radiation field.

3. The sparse mesh radiation field representation method based on point cloud initialization and depth supervision as described in claim 1 or 2, characterized in that, The aforementioned optimization of the sampling strategy using sparse voxel meshes as an aid is specifically as follows: For a ray passing through a sparse voxel field in space, ray sampling will skip all empty grids in space and only sample in the sparse grid.

4. The sparse mesh radiation field representation method based on point cloud initialization and depth supervision as described in claim 3, characterized in that, The loss function constructed by combining the acquired color and depth images with the rendered RGB color and depth images is as follows: The color constraints are formed by calculating the squared error between the RGB color image rendered from the radiation field and the actual color image captured. The depth constraints are formed by calculating the squared error between the depth image rendered from the radiation field and the actual depth image captured. The specific formulas are as follows: ； 。 5. The sparse mesh radiation field representation method based on point cloud initialization and depth supervision as described in claim 1, 2, or 4, characterized in that, The use of this occupied grid as a geometric initialization prior for the voxel radiation field specifically refers to: Using the acquired depth images, a three-dimensional point cloud is generated through image parameters. Then, the three-dimensional point cloud is used to generate an occupancy probability grid through an occupancy network. An occupancy probability threshold is set, and grids with occupancy probabilities less than the threshold are deleted to obtain a sparse voxel radiation field. During initialization, the occupancy probability values ​​in the occupancy probability grid are used as the initial values ​​for the geometric density.

Images