An online three-dimensional scene reconstruction method and system based on neural radiation field and multi-sensor fusion

Abstract

The present application belongs to the field of information science and technology and robot, and relates to an online three-dimensional scene reconstruction method and system based on neural radiance field and multi-sensor fusion. The method comprises: acquiring camera images and laser radar point clouds of a vehicle, and extracting fusion sensor features; obtaining predicted color values and predicted depth values of space points based on the fusion sensor features through a multi-layer perception network and volume rendering of a neural radiance field; jointly optimizing and training the feature extraction network and the multi-layer perception network; sampling multiple new perspectives around the current position of the vehicle and synthesizing depth images under the new perspectives by using the trained feature extraction network and multi-layer perception network, and generating a triangular mesh model of the scene according to the depth images. The present application can realize real-time explicit three-dimensional reconstruction of the scene around the vehicle, finely perceive special-shaped obstacles in the scene, and provide a high-quality scene model for environment perception of unmanned driving.

Description

Technical Field

[0001] This invention belongs to the field of information science and technology and robotics, and specifically relates to an online three-dimensional scene reconstruction method and system based on neural radiation field and multi-sensor fusion. Background Technology

[0002] Environmental object perception is the prerequisite and foundation for the safe, autonomous, and reliable operation of autonomous vehicles. It relies on multiple sensors and requires the fusion of various sensor information to leverage their complementary strengths. The environment contains many irregularly shaped objects, and their detection and modeling are inevitable during the perception process. Introducing self-supervised learning methods to utilize massive amounts of unlabeled data and avoid the cost of acquiring labeled data is a crucial research problem for improving environmental perception performance when training the environmental perception model.

[0003] Current mainstream methods for environmental perception in autonomous vehicles use 3D object detection models to represent objects as 3D rectangular boxes. This method is only applicable to a limited number of object categories and struggles to accurately represent irregularly shaped obstacles. Therefore, this invention uses 3D reconstruction to replace object detection, providing a refined model of the scene's 3D structure.

[0004] Traditional 3D reconstruction techniques include Simultaneous Localization and Mapping (SLAM), Structure of Motion (SfM), and Multi-View Stereo Vision (MVS). SLAM incurs high computational costs during map building, typically reducing map update frequency and thus prioritizing localization accuracy over mapping quality. Furthermore, it constructs sparse point cloud maps lacking collision volumes, making it difficult to directly use for obstacle avoidance path planning. The latter relies on the assumption that the reconstructed objects or scenes remain spatially invariant over a considerable period, an assumption that is almost entirely invalid in autonomous vehicle scenarios. Additionally, it usually requires offline operation, resulting in high computational costs and long processing times, making it unsuitable for online scene reconstruction. Finally, neither algorithm can reconstruct occluded areas.

[0005] Neural Radiation Field (NeRF) is a novel 3D reconstruction method that constructs a continuous, refined, and resolution-independent 3D scene by representing it as a continuous neural network function. However, NeRF is an implicit representation of the scene, while collision detection in autonomous vehicles requires acquiring 3D entities, necessitating explicit scene reconstruction. Furthermore, traditional NeRF methods are primarily designed for offline new view synthesis of static scenes, resulting in slow training and rendering speeds, and their reliance on image information makes them difficult to directly apply to online multi-sensor environmental perception tasks in autonomous vehicles.

[0006] In summary, how to effectively integrate LiDAR and camera data to achieve real-time, high-precision 3D reconstruction of complex driving scenarios containing irregular obstacles without manual annotation is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0007] The purpose of this invention is to provide an online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion. This method can fuse camera images and LiDAR point cloud data, utilize self-supervised learning based on the NeRF concept to reduce data costs, and achieve explicit scene reconstruction through multi-view depth synthesis and TSDF (truncated symbolic distance function) fusion. Ultimately, this method can perform real-time explicit 3D reconstruction of the scene around a vehicle, accurately perceive irregular obstacles in the scene, and provide a high-quality scene model for environmental perception in autonomous driving.

[0008] The technical solution adopted in this invention is as follows: An online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion includes the following steps: Acquire camera images and LiDAR point clouds of the vehicle, and use a feature extraction network to extract fused sensor features; Based on the fusion of sensing features, the predicted color value and predicted depth value of spatial points are obtained through volume rendering of multilayer perceptron network and neural radiation field. The feature extraction network and the multilayer perceptron network are jointly optimized and trained using color loss, depth loss and multi-view geometric consistency loss. Using a trained feature extraction network and a multilayer perceptron network, multiple new perspectives are sampled around the vehicle's current position and depth images from these new perspectives are synthesized. A triangular mesh model of the scene is then generated based on the depth images.

[0009] Furthermore, the step of extracting fused sensing features using a feature extraction network includes: using two neural networks as feature extraction networks to extract camera image features from a perspective view and laser point cloud features from a bird's-eye view, respectively, and then stitching them together to obtain fused sensing features of spatial points.

[0010] Furthermore, the step of obtaining the predicted color value and predicted depth value of a spatial point based on fused sensing features through volume rendering of a multilayer perceptron network and a neural radiation field includes: emitting a ray from the observation viewpoint to the pixel point according to the camera model, sampling several discrete points on the ray; constructing a multilayer perceptron network, taking the position of the spatial point, the observation viewpoint, and the fused sensing features as inputs, outputting the color value and volume density value of the spatial point, and calculating the predicted color value and predicted depth value of the pixel point through a differentiable volume rendering formula.

[0011] Furthermore, the color loss calculation is the mean square error between the rendered color and the pixel color of the real camera image.

[0012] Furthermore, the depth loss calculation calculates the L1 error between the rendered depth and the sparse depth truth obtained from the lidar projection.

[0013] Furthermore, regarding the first-person perspective... i 1 pixel ( The depth of the object is estimated, and the estimated depth is used to reconstruct the 3D spatial points and project them onto the second viewpoint to obtain the pixel points in the second viewpoint. ), restore pixel points ( The position of ) in three-dimensional space is projected back to the first-person viewpoint to obtain the pixel. ),in Represents the x-coordinate of a pixel. , , The vertical coordinate of a pixel is represented; the multi-view geometric consistency loss is calculated using the following formula: in, This indicates the number of pixels in the first-person perspective.

[0014] Furthermore, the step of generating a triangular mesh model of the scene based on the depth image includes: converting the depth image into a truncated sign distance function voxel mesh, and then generating a triangular mesh model of the scene using an isosurface extraction algorithm.

[0015] An online 3D scene reconstruction system based on neural radiation field and multi-sensor fusion, comprising: The sensor feature extraction module is used to acquire camera images and LiDAR point clouds of the vehicle, and to extract and fuse sensor features using a feature extraction network. The color and depth synthesis module is used to obtain the predicted color and depth values ​​of spatial points based on the fused sensing features through volume rendering of multilayer perceptron networks and neural radiation fields. The model training module is used to jointly optimize and train the feature extraction network and the multilayer perceptron network using color loss, depth loss and multi-view geometric consistency loss. The online 3D scene reconstruction module uses a trained feature extraction network and a multilayer perceptron network to sample multiple new perspectives around the vehicle's current position and synthesize depth images from these new perspectives. Based on the depth images, it generates a triangular mesh model of the scene.

[0016] The present invention also provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0017] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the above-described method.

[0018] Compared with the prior art, the beneficial effects of this invention are as follows: First, this invention abandons the traditional bounding box detection approach and adopts an online 3D reconstruction method, which can perform refined modeling of diverse environmental objects with different shapes (especially irregular obstacles).

[0019] Second, this invention integrates camera and lidar information, which improves the robustness and accuracy of scene reconstruction compared to single-sensor methods.

[0020] Third, this invention reduces the reliance of model training on manual data annotation by using a self-supervised training method based on neural radiation fields, thus achieving self-supervised learning. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the steps of an online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion according to an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the overall process architecture of the method of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the calculation principle of geometric consistency loss for multiple views.

[0024] Figure 4 This is a schematic diagram of a triangular mesh generated based on the Marching Cubes algorithm.

[0025] Figure 5 This invention presents a novel perspective on color and depth synthesis effects on the KITTI dataset.

[0026] Figure 6 This invention presents a novel perspective on color and depth synthesis effects on the nuScenes dataset.

[0027] Figure 7 This invention demonstrates the synthetic effect of dynamic objects.

[0028] Figure 8 This is the 3D reconstruction effect of the present invention in the KITTI scene.

[0029] Figure 9 This is the 3D reconstruction effect of the present invention in the nuScenes scene.

[0030] Figure 10 This is a schematic diagram of the module composition of an online three-dimensional scene reconstruction method system based on neural radiation field and multi-sensor fusion according to an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0032] This invention provides an online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion. It combines the rich information provided by cameras and LiDAR, utilizes the self-supervised training paradigm of NeRF to drive parameter optimization of the neural radiation field network without human labels, and transforms the implicit field into an explicit triangular mesh model through real-time TSDF fusion and isosurface extraction algorithms.

[0033] In one embodiment, the online 3D scene reconstruction method of the present invention based on neural radiation field and multi-sensor fusion, such as... Figure 1 As shown, it includes the following steps: Acquire camera images and LiDAR point clouds of the vehicle, and use a feature extraction network to extract fused sensor features; Based on the fusion of sensing features, the predicted color value and predicted depth value of spatial points are obtained through volume rendering of multilayer perceptron network and neural radiation field. The feature extraction network and the multilayer perceptron network are jointly optimized and trained using color loss, depth loss and multi-view geometric consistency loss. Using a trained feature extraction network and a multilayer perceptron network, multiple new perspectives are sampled around the vehicle's current position and depth images from these new perspectives are synthesized. A triangular mesh model of the scene is then generated based on the depth images.

[0034] In one embodiment, the step of extracting fused sensing features using a feature extraction network includes: using two neural networks as feature extraction networks to extract camera image features from a perspective view and laser point cloud features from a bird's-eye view, respectively, and then stitching them together to obtain fused sensing features of spatial points.

[0035] In one embodiment, obtaining the predicted color and predicted depth values ​​of spatial points based on fused sensing features through volume rendering of a multilayer perceptron network and a neural radiation field includes: emitting a ray from the observation viewpoint to the pixel point according to the camera model, sampling several discrete points on the ray; constructing a multilayer perceptron network, taking the position of the spatial point, the observation viewpoint, and the fused sensing features as inputs, outputting the color and volume density values ​​of the spatial point, and calculating the predicted color and predicted depth values ​​of the pixel point using a differentiable volume rendering formula.

[0036] In one embodiment, the color loss is calculated as the mean square error between the rendered color and the pixel color of the actual camera image.

[0037] In one embodiment, the depth loss calculation calculates the L1 error between the rendered depth and the sparse depth truth obtained from LiDAR projection.

[0038] In one embodiment, for the first-person perspective... i 1 pixel ( The depth of the object is estimated, and the estimated depth is used to reconstruct the 3D spatial points and project them onto the second viewpoint to obtain the pixel points in the second viewpoint. ), restore pixel points ( The position of ) in three-dimensional space is projected back to the first-person viewpoint to obtain the pixel. ),in Represents the x-coordinate of a pixel. , , The vertical coordinate of a pixel is represented; the multi-view geometric consistency loss is calculated using the following formula: in, This indicates the number of pixels in the first-person perspective.

[0039] In one embodiment, generating a triangular mesh model of the scene based on the depth image includes: converting the depth image into a truncated sign distance function voxel mesh, and then generating a triangular mesh model of the scene using an isosurface extraction algorithm.

[0040] In one embodiment, the online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion of the present invention includes the following steps: 1. Sensor feature extraction of camera-LiDAR fusion: Acquire camera image data and LiDAR point cloud data of the vehicle at the current moment, use two neural networks as feature extraction networks to extract camera image features under perspective view (PV) and LiDAR point cloud features under bird's-eye view (BEV) respectively, and stitch them together to obtain the fused sensor features of spatial points.

[0041] 2. Constructing a novel perspective synthesis based on neural radiation fields and volume rendering: According to the camera model, rays are emitted from the observation viewpoint towards the pixel, and several discrete spatial points are sampled along the rays. A multilayer perceptron (MLP) network is constructed, taking the position of the spatial point, the observation viewpoint, and fused sensor features as input, and outputting the color value and volume density value of the spatial point. Finally, the predicted color value and predicted depth value of the pixel are calculated using a differentiable volume rendering formula. Here, a pixel is a point on the two-dimensional plane of the camera image, and a spatial point is a point in three-dimensional space.

[0042] 3. Implement self-supervised training of the model and design loss functions: Select frames that are temporally or spatially adjacent to the current input frame as rendering frames, use the original camera image and laser point cloud depth of the rendering frames as supervision information, calculate color loss, depth loss and multi-view geometric consistency loss, and jointly optimize and train the feature extraction network and the multilayer perceptron network.

[0043] 4. Online Explicit Reconstruction of 3D Scenes: Using a trained model (feature extraction network and multilayer perceptron network), multiple new perspectives are sampled around the current vehicle pose to synthesize depth images from the new perspectives; the depth images are converted into truncated signed distance function (TSDF) voxel meshes, and a triangular mesh model of the scene is generated by TSDF fusion and Marching Cubes algorithm.

[0044] In one embodiment, the present invention provides an online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion, such as... Figure 2 As shown, it mainly includes three stages: sensor feature extraction, new perspective synthesis, and 3D scene reconstruction.

[0045] (1) Sensor feature extraction from camera-lidar fusion.

[0046] Real-time acquisition of camera images during autonomous vehicle operation and lidar point cloud To fully utilize multimodal information, this embodiment designs a dual-branch feature extraction network. For camera images, a neural network is used to extract camera feature maps under the perspective view (PV). For the laser point cloud, after voxelization, a neural network is used to extract the laser feature map from the bird's-eye view (BEV). .

[0047] To obtain any point in space Based on the sensing characteristics, this invention employs a projection query mechanism: First, the points... Camera features are obtained by projecting camera intrinsic and extrinsic parameters onto the image plane and sampling them. If the projected point exceeds the image range, the feature value is set to 0. Then, the point... Projected onto the BEV plane, laser features are obtained by sampling. If the value exceeds the BEV range, the feature is set to 0. Finally, and By splicing together, the fused sensing features of that point are formed. This step combines PV and BEV features, enabling autonomous vehicles to process distant scenes using PV features and solve the problem of occlusion by nearby objects using BEV features.

[0048] (2) New perspective on color and depth synthesis and model training.

[0049] This embodiment employs Neural Radiation Field (NeRF) volume rendering technology. A multilayer perceptron network is defined. Its input is the spatial point position. Observation direction and fusion of sensing features The output is this point. color and bulk density : For a ray passing through a pixel Sampling on it For each point, the desired color is calculated using the volume rendering formula. and expected depth .

[0050] Desired color The calculation formula is: in, For the first The color of each sampling point N The number of sampling points. For the first The weight of each sampling point represents the probability that the light ray reaches that point and terminates there. The calculation method is as follows: in, The distance between adjacent sampling points. For the first Volume density of each sampling point.

[0051] Expected depth The calculation formula is: in For the first The distance from each sampling point to the optical center of the camera.

[0052] The model training employs a self-supervised approach, jointly optimizing and training the feature extraction network and the multilayer perceptron network, with the loss function... It consists of three parts: First, color loss. : Calculate the mean square error between the rendered color and the pixel color of the real camera image.

[0053] Secondly, depth loss : Calculate the L1 error between the rendered depth and the true sparse depth obtained from LiDAR projection.

[0054] Third, loss of geometric consistency across multiple views : Pixels under viewpoint 1 The three-dimensional spatial points are obtained by reconstructing the depth from the estimated viewpoint 1. Then project to obtain the pixels under viewpoint 2. Then restore from perspective 2. Position in three-dimensional space and will Projecting back to viewpoint 1 yields the pixel. ,like Figure 3 As shown. The formula for the multi-view geometric consistency loss is: in, This indicates that there are n pixels in viewpoint 1.

[0055] (3) Online 3D scene reconstruction.

[0056] After training the model (MLP network and feature extraction network), three new viewpoints are selected each in the forward and backward directions from the vehicle's current position. The trained model is then used to render and synthesize high-resolution depth images from these viewpoints. Subsequently, the synthesized depth images are converted into truncated signed distance function (TSDF) voxels in 3D space. For each voxel in space, the final TSDF value is the minimum of the TSDF values ​​for each viewpoint, used to remove noise and fill in occlusions. Finally, the Marching Cubes algorithm (e.g., ...) is used... Figure 4As shown, isosurfaces with a TSDF value of 0 are extracted to generate a mesh model containing triangular facets. This model is the final reconstructed 3D scene, which can be directly input into the subsequent path planning module for drivable area analysis.

[0057] Figure 5 and Figure 6 The new perspectives on color and depth synthesis effects are shown on the KITTI and nuScenes datasets, respectively. Figure 7 This demonstrates the effect of the present invention on the synthesis of dynamic objects. Figure 8 and Figure 9 The 3D reconstruction results in the KITTI and nuScenes scenes are demonstrated respectively. Experiments show that the color images synthesized by the method of this invention are clear, the depth maps accurately restore the geometric structure of the scene, the generated triangular mesh models effectively reconstruct static environmental objects such as vehicles and trees, and have a certain modeling capability for dynamic objects, proving the effectiveness of the method.

[0058] Another embodiment of the present invention provides an online 3D scene reconstruction system based on neural radiation field and multi-sensor fusion, such as... Figure 10 As shown, it includes: The sensor feature extraction module is used to acquire camera images and LiDAR point clouds of the vehicle, and to extract and fuse sensor features using a feature extraction network. The color and depth synthesis module is used to obtain the predicted color and depth values ​​of spatial points based on the fused sensing features through volume rendering of multilayer perceptron networks and neural radiation fields. The model training module is used to jointly optimize and train the feature extraction network and the multilayer perceptron network using color loss, depth loss and multi-view geometric consistency loss. The online 3D scene reconstruction module uses a trained feature extraction network and a multilayer perceptron network to sample multiple new perspectives around the vehicle's current position and synthesize depth images from these new perspectives. Based on the depth images, it generates a triangular mesh model of the scene.

[0059] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding process in the aforementioned method embodiments. Each module can be implemented entirely or partially through software, hardware, or a combination thereof.

[0060] Another embodiment of the present invention provides a computer device (computer, server, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0061] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0062] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. An online 3D scene reconstruction method based on neural radiation field and multi-sensor fusion, characterized in that, Includes the following steps: Acquire camera images and LiDAR point clouds of the vehicle, and use a feature extraction network to extract fused sensor features; Based on the fusion of sensing features, the predicted color value and predicted depth value of spatial points are obtained through volume rendering of multilayer perceptron network and neural radiation field. The feature extraction network and the multilayer perceptron network are jointly optimized and trained using color loss, depth loss and multi-view geometric consistency loss. Using a trained feature extraction network and a multilayer perceptron network, multiple new perspectives are sampled around the vehicle's current position, and depth images from these new perspectives are synthesized. A triangular mesh model of the scene is then generated based on the depth images.

2. The method according to claim 1, characterized in that, The extraction of fused sensing features using a feature extraction network includes: using two neural networks as feature extraction networks to extract camera image features from a perspective view and laser point cloud features from a bird's-eye view, respectively, and then stitching them together to obtain fused sensing features of spatial points.

3. The method according to claim 1, characterized in that, The method of obtaining predicted color and depth values ​​of spatial points based on fused sensing features through volume rendering of a multilayer perceptron network and neural radiation field includes: emitting rays from the observation viewpoint to the pixel point according to the camera model, sampling several discrete points on the ray; constructing a multilayer perceptron network, taking the position of the spatial point, the observation viewpoint, and the fused sensing features as inputs, outputting the color and volume density values ​​of the spatial point, and calculating the predicted color and depth values ​​of the pixel point using a differentiable volume rendering formula.

4. The method according to claim 1, characterized in that, The color loss calculation is based on the mean square error between the rendered color and the pixel color of the actual camera image.

5. The method according to claim 1, characterized in that, The depth loss calculation calculates the L1 error between the rendered depth and the sparse depth truth obtained from LiDAR projection.

6. The method according to claim 1, characterized in that, For the first-person perspective i 1 pixel ( The depth of the object is estimated, and the estimated depth is used to reconstruct the 3D spatial points and project them onto the second viewpoint to obtain the pixel points in the second viewpoint. ), restore pixel points ( The position of ) in three-dimensional space is projected back to the first-person viewpoint to obtain the pixel. ),in Represents the x-coordinate of a pixel. , , The vertical coordinate of a pixel is represented; the multi-view geometric consistency loss is calculated using the following formula: in, This indicates the number of pixels in the first-person perspective.

7. The method according to claim 1, characterized in that, The process of generating a triangular mesh model of the scene based on the depth image includes: converting the depth image into a truncated sign distance function voxel mesh, and then generating a triangular mesh model of the scene using an isosurface extraction algorithm.

8. An online 3D scene reconstruction system based on neural radiation field and multi-sensor fusion, characterized in that, include: The sensor feature extraction module is used to acquire camera images and LiDAR point clouds of the vehicle, and to extract and fuse sensor features using a feature extraction network. The color and depth synthesis module is used to obtain the predicted color and depth values ​​of spatial points based on the fused sensing features through volume rendering of multilayer perceptron networks and neural radiation fields. The model training module is used to jointly optimize and train the feature extraction network and the multilayer perceptron network using color loss, depth loss and multi-view geometric consistency loss. The online 3D scene reconstruction module uses a trained feature extraction network and a multilayer perceptron network to sample multiple new perspectives around the vehicle's current position and synthesize depth images from these new perspectives. Based on the depth images, a triangular mesh model of the scene is generated.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images