Scene three-dimensional reconstruction method based on camera-lidar fusion and related device

Abstract

The application belongs to the technical field of scene three-dimensional reconstruction, and discloses a scene three-dimensional reconstruction method based on camera laser radar fusion and a related device; wherein the scene three-dimensional reconstruction method comprises the following steps: firstly, based on a selected scene, the relative pose between a camera and a laser radar and the camera image and laser radar point cloud sensor data of the scene are obtained; then, based on 3DGS, sensor tracking, map updating and loop detection are sequentially performed to obtain a scene three-dimensional reconstruction result; wherein in the sensor tracking stage, pose optimization is performed by minimizing photometric loss and geometric loss; in the map updating stage, laser radar point clouds meeting preset conditions are added to a Gaussian point cloud map. The technical scheme provided by the application can realize accurate pose estimation and high-quality image rendering when performing scene three-dimensional reconstruction in environments such as strong light changes and textureless areas.

Description

Technical Field

[0001] This invention belongs to the field of scene 3D reconstruction technology, and specifically relates to a scene 3D reconstruction method and related apparatus based on camera-LiDAR fusion. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) has been an active research area for decades, aiming to simultaneously reconstruct unknown environments and localize sensor poses. As a fundamental task of 3D computer vision, SLAM is crucial for more advanced tasks such as autonomous driving, embodied intelligence, and robot navigation.

[0003] Currently, traditional SLAM methods use point clouds, meshes, and surface clouds as map representations. These methods demonstrate excellent real-time performance and scalability; however, their performance is poor in environments with strong lighting changes and textureless regions.

[0004] With the development of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) in the field of new-view image synthesis, these two methods have been applied to some SLAM systems, providing better surface modeling and more robust noise handling. Compared to NeRF, 3DGS renders images faster; furthermore, NeRF uses neural networks to represent the scene, while 3DGS uses millions of Gaussian ellipsoids distributed throughout space. By adding more Gaussian ellipsoids, map capacity can be easily expanded, making 3DGS more suitable for incremental map building processes in SLAM. In summary, using 3DGS in SLAM enables more accurate pose estimation and higher-quality rendering.

[0005] The application of 3DGS in vision-LiDAR SLAM systems is mainly focused on the map building stage. These methods first determine the sensor pose based on traditional LiDAR SLAM algorithms, then initialize a Gaussian map using LiDAR (Light Detection and Range) point clouds, and finally optimize the 3D Gaussian map using image rendering loss. During the initialization and optimization of the Gaussian map, the sensor pose remains constant. This indicates that 3DGS is primarily used for map building in vision-LiDAR SLAM systems, and its potential in pose tracking has not been fully utilized. Furthermore, it is important to emphasize that in existing 3DGS-based SLAM methods, pose estimation is affected by floating Gaussians, which are introduced by gradient-based Gaussian denser strategies, leading to geometric inconsistencies. Summary of the Invention

[0006] The purpose of this invention is to provide a method and related apparatus for 3D scene reconstruction based on camera-LiDAR fusion, in order to solve one or more of the aforementioned technical problems. The technical solution provided by this invention enables accurate pose estimation and high-quality image rendering for 3D scene reconstruction in environments with strong lighting changes and textureless regions.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for scene 3D reconstruction based on camera-LiDAR fusion, comprising the following steps:

[0009] Based on the selected scene, the relative pose between the camera and the LiDAR, as well as the camera image and LiDAR point cloud sensor data of the scene are obtained.

[0010] Based on the relative pose between the camera and the lidar, as well as the camera images and lidar point cloud sensor data of the scene, sensor tracking, map updating and loop closure detection are performed sequentially based on 3DGS to obtain the scene 3D reconstruction results.

[0011] In the sensor tracking phase, the poses of the camera and LiDAR are first initialized using a linear motion assumption. Then, pose optimization is performed by minimizing photometric and geometric losses to obtain the optimized poses of the camera and LiDAR. After sensor tracking is completed, the map update phase begins. First, a transparency map and a depth map are rendered based on the current camera pose. The LiDAR point cloud that meets the preset conditions is added to the Gaussian point cloud map. Then, the properties of the Gaussian point cloud are optimized based on photometric and depth losses to obtain the Gaussian map. After the map update is completed, the loop closure detection phase begins, where the Gaussian map is optimized and the optimized Gaussian map is used as the result of the scene's 3D reconstruction.

[0012] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0013] In the step of optimizing pose by minimizing photometric and geometric losses to obtain the optimized pose of the camera and LiDAR, the overall loss function L is used. track Represented as:

[0014]

[0015] In the formula, λ pho , λ geo These are the weighting terms for photometric loss and geometric loss, respectively; L pho L geo These are photometric loss and geometric loss, respectively.

[0016] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0017] The overall loss function L track In the calculation expression:

[0018]

[0019] In the formula, λ1 is a hyperparameter; ∥*∥1 represents the L1 norm; Represents the rendered image Structural similarity to the ground truth image I;

[0020]

[0021] In the formula, ω ij It is a weighted term; d ij It is the distance between two points, d ij =p i -p j p i It is a lidar point cloud P c Point p in j It is a local keyframe point cloud P l p i The nearest neighbor; ∑′ j It is point p j The corresponding covariance matrix; T c This represents the current pose of the lidar; ∑ i ′ is point p i The corresponding covariance matrix.

[0022] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0023] In the step of optimizing the properties of Gaussian point clouds based on photometric and depth losses to obtain a Gaussian map, scale regularization loss and size loss are also added when optimizing the properties of the Gaussian point clouds, and the overall optimization loss function L map The calculation expression is:

[0024]

[0025] In the formula, λ d , λ reg , λ size These are the weights of the depth loss, scale regularization loss, and size loss, respectively; L d For depth loss; L reg For scale regularization loss; L size This is due to dimensional loss.

[0026] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0027] The overall optimization loss function L map In the calculation expression:

[0028]

[0029] In the formula, I d It's a depth map; x i d i They are points p i The two-dimensional projection points obtained by projecting onto the image plane and the corresponding projection depth;

[0030]

[0031] In the formula, max(s) i ), min(s) i ) represent the elements that have obtained the maximum and minimum values, respectively; Represents the scale of the Gaussian ellipsoid in three directions;

[0032]

[0033] In the formula, f is the focal length of the camera.

[0034] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0035] In the step of optimizing the attributes of the Gaussian point cloud based on photometric loss and depth loss to obtain a Gaussian map, the attributes include: rotation, scaling, opacity, and RGB color.

[0036] A further improvement of the scene 3D reconstruction method of the present invention lies in that,

[0037] The steps of optimizing the Gaussian map and using the optimized Gaussian map as the result of scene 3D reconstruction include:

[0038] First, based on the scene's camera images, LiDAR point cloud sensor data, and optimized camera and LiDAR poses, the cumulative error of the camera and LiDAR poses is corrected by detecting loop closure frames and optimizing the pose map to obtain the corrected pose. Then, the Gaussian map is optimized based on the corrected pose, and the optimized Gaussian map is used as the result of the scene's 3D reconstruction.

[0039] In a second aspect, the present invention provides a scene 3D reconstruction system based on camera-LiDAR fusion, comprising:

[0040] The data acquisition module is used to acquire the relative pose between the camera and the LiDAR, as well as the camera images and LiDAR point cloud sensor data of the scene, based on the selected scene.

[0041] The 3D reconstruction module is used to obtain the scene 3D reconstruction results by performing sensor tracking, map updating and loop closure detection in sequence based on 3DGS, according to the relative pose between the camera and LiDAR, as well as the camera images and LiDAR point cloud sensor data of the scene.

[0042] In the sensor tracking phase, the poses of the camera and LiDAR are first initialized using a linear motion assumption. Then, pose optimization is performed by minimizing photometric and geometric losses to obtain the optimized poses of the camera and LiDAR. After sensor tracking is completed, the map update phase begins. First, a transparency map and a depth map are rendered based on the current camera pose. The LiDAR point cloud that meets the preset conditions is added to the Gaussian point cloud map. Then, the properties of the Gaussian point cloud are optimized based on photometric and depth losses to obtain the Gaussian map. After the map update is completed, the loop closure detection phase begins, where the Gaussian map is optimized and the optimized Gaussian map is used as the result of the scene's 3D reconstruction.

[0043] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scene 3D reconstruction method based on camera-LiDAR fusion as described in any one of the first aspects of the present invention.

[0044] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the scene 3D reconstruction method based on camera-LiDAR fusion as described in any one of the first aspects of the present invention.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] This invention discloses a scene 3D reconstruction method based on camera-LiDAR fusion, proposing a method that simultaneously uses 3DGS for map representation and camera tracking, which can be called VLGS-SLAM (Vision LiDAR Gaussian Splatting SLAM). The inputs to this method are strictly synchronized camera images and LiDAR point cloud data, as well as the relative pose between the two sensors. During the sensor tracking phase, pose is optimized by minimizing photometric and geometric losses. During the map update phase, the accuracy of pose estimation is enhanced by treating the LiDAR point cloud as a Gaussian ellipsoid, avoiding the geometric inconsistency problem caused by a floating Gaussian ellipsoid. Ultimately, accurate pose estimation and high-quality image rendering can be achieved.

[0047] In existing 3DGS optimization processes, photometric loss leads to the magnification of distant Gaussian ellipsoids, causing these Gaussian ellipsoids to be used to render the scene's background. However, as the camera moves closer, these background Gaussian ellipsoids become more visible, resulting in visual artifacts that interfere with tracking and rendering. To address these issues, the preferred embodiment of this invention introduces a loss function to regularize the scale of the Gaussian ellipsoid, improving the robustness of pose estimation.

[0048] In existing 3DGS-based SLAM methods, pose estimation is affected by floating Gaussians, which are introduced by gradient-based Gaussian densification strategies. To reduce their influence, VLGS-SLAM assigns additional properties to LiDAR points (e.g., rotation, scaling, opacity, and RGB color) to form new Gaussian ellipsoids, thus avoiding the introduction of floating Gaussians. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a scene 3D reconstruction method based on camera-LiDAR fusion in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the principle of a scene 3D reconstruction method based on camera-LiDAR fusion in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram of a scene 3D reconstruction system based on camera-LiDAR fusion in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0054] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0055] Please see Figure 1 and Figure 2 The present invention provides a method for scene 3D reconstruction based on camera-LiDAR fusion, comprising the following steps:

[0056] Step 1: Based on the selected scene, acquire the relative pose between the camera and the LiDAR, as well as the camera image and LiDAR point cloud sensor data of the scene.

[0057] Step 2: Based on the relative pose between the camera and the LiDAR, as well as the camera images and LiDAR point cloud sensor data of the scene, perform sensor tracking, map updating and loop closure detection in sequence using 3DGS to obtain the scene 3D reconstruction results.

[0058] In the sensor tracking phase, the poses of the camera and LiDAR are first initialized using a linear motion assumption. Then, pose optimization is performed by minimizing photometric and geometric losses to obtain the optimized camera and LiDAR poses. After sensor tracking is completed, the map update phase begins. First, a transparency map and a depth map are rendered based on the current camera pose. LiDAR point clouds that meet preset conditions are added to the Gaussian point cloud map. Then, the attributes of the Gaussian point cloud are optimized based on photometric and depth losses to obtain the Gaussian map. After the map update is completed, the loop closure detection phase begins, where the Gaussian map is optimized and used as the result of the scene's 3D reconstruction. In a further exemplary technical solution, in the loop closure detection phase, based on the scene's camera images, LiDAR point cloud sensor data, and the optimized camera and LiDAR poses, the cumulative errors of the camera and LiDAR poses are corrected by detecting loop closure frames and optimizing the pose map to obtain the corrected poses. Then, the Gaussian map is optimized based on the corrected poses to obtain the scene's 3D reconstruction result.

[0059] The technical solution disclosed in this invention utilizes a 3D Gaussian ellipsoid for map representation and camera tracking, achieving significant technological advancements. The core of the method lies in fusing two different types of sensor data—camera images and LiDAR point clouds—and leveraging their strict synchronization and relative pose information to achieve accurate pose estimation and high-quality image rendering. This method not only improves the accuracy of 3D reconstruction but also enhances the system's robustness and adaptability. More specifically, camera images provide rich texture and color information, contributing to the construction of more realistic 3D scenes; LiDAR point clouds provide accurate depth information and geometric structure, forming the foundation for 3D reconstruction; and the relative pose between the two sensors ensures accurate fusion of camera and LiDAR data. This invention uses 3DGS for map representation, enabling flexible description of complex 3D scenes while maintaining high representation efficiency; by combining camera images and LiDAR point clouds and optimizing algorithms, accurate tracking of the camera and LiDAR is achieved, thereby acquiring continuous pose information.

[0060] Furthermore, by assigning additional attributes (such as rotation, scaling, opacity, and RGB color) to LiDAR points, this invention forms a new Gaussian ellipsoid, effectively avoiding the introduction of floating Gaussians and improving the accuracy of attitude estimation.

[0061] Explained, 3DGS is a method for representing three-dimensional scenes, where the entire scene is represented by multiple (typically millions) 3D Gaussian ellipsoids, each of which is a three-dimensional Gaussian distribution. During training, the following attributes of each Gaussian ellipsoid need to be optimized: center (i.e., the mean of the Gaussian distribution), scale (i.e., the variance of the Gaussian distribution), transparency, and color. In the technical solution provided by the embodiments of the present invention, each Gaussian ellipsoid has the following attributes: transparency o∈[0,1], color... Rotation scale mean Using these properties and the camera's pose, each Gaussian ellipsoid can be projected onto the image, thereby rendering an image.

[0062] In a specific exemplary technical solution, the following describes the specific rendering process for each pixel:

[0063] Assuming the current pixel is p, first arrange the Gaussian ellipsoids in space according to their distance from the camera plane from near to far. Then calculate the transparency of each Gaussian ellipsoid to that pixel. The calculation expression is:

[0064]

[0065] In the formula, α i It is the current Gaussian transparency of pixel p; αi ′ represents the transparency of the Gaussian ellipsoid itself; Σ is the projection location of the center (mean) of the Gaussian ellipsoid onto the image plane; Σ is the variance of the projection of the Gaussian ellipsoid onto the image plane.

[0066] Next, the weight of each Gaussian ellipsoid with respect to the current pixel can be obtained, calculated as follows:

[0067]

[0068] In the formula, α j It represents the transparency of the other Gaussians arranged before the current Gaussian; ω i These are the current Gaussian weights;

[0069] Finally, the expression for calculating the rendered color of the current pixel is:

[0070]

[0071] In the formula, It is the rendered color of the current pixel p; c i is the color of each Gaussian ellipsoid; M is the number of all Gaussian ellipsoids;

[0072] Repeat the above process for each pixel of the current image to obtain the rendered image using Gaussian rendering. Then render the image The photometric loss is calculated using the ground truth image I, and the calculation expression is:

[0073]

[0074] In the formula, Represents the structural similarity between two images; ∥*∥1 represents the L1 norm; λ1 is a hyperparameter used to balance the two types of errors.

[0075] Finally, by backpropagating the loss, all properties of the Gaussian ellipsoid can be updated.

[0076] In one embodiment of the present invention, since the camera pose is used during the rendering process, the gradient of the loss function is also backpropagated to the camera pose, thereby optimizing the camera pose. In the technical solution of this embodiment, the Gaussian properties are fixed during the pose estimation stage, allowing the photometric loss to optimize only the camera pose; during the mapping stage, the camera pose is fixed, allowing the photometric loss to optimize only the Gaussian properties. Furthermore, since the Gaussian ellipsoid in the method of this invention is obtained from the LiDAR point cloud, its position is not optimized; the position of the Gaussian ellipsoid only changes when the position of the LiDAR changes.

[0077] Based on the above embodiments, in a preferred embodiment of the present invention, during sensor tracking, the poses of the camera and LiDAR are first initialized using a linear motion assumption, and then further optimized by minimizing photometric and geometric losses. In this embodiment, an image is rendered based on a Gaussian map using the current camera pose, and the photometric loss is calculated using the aforementioned photometric loss calculation formula. To avoid the influence of newly observed areas, the method of the present invention also renders an opacity image and only calculates the loss corresponding to pixels with an opacity greater than a preset threshold (exemplarily, 0.9). Similar to other 3DGS-based SLAM methods, the present invention does not use SSIM loss and sets λ1 = 0 during the sensor tracking stage. The current LiDAR point cloud P is used. c and local keyframe point cloud P l The plane-to-plane distance between them is used as the geometric loss. For example, here P... l It is a collection of LiDAR point clouds from the first 5 keyframes.

[0078] In a specific exemplary technical solution, firstly, the lidar point cloud is downsampled using a voxel grid with a resolution of (0.5m, 0.5m, 0.5m); then, for each point p in the point cloud... i The k nearest neighbors are calculated using the K-nearest neighbor algorithm, and the set of these points is denoted as .

[0079] The covariance matrix is:

[0080]

[0081] In the formula, p j It is the current point p i The nearest neighbor; yes The mean;

[0082] Then, the covariance matrix is ​​decomposed using SVD to obtain ∑ i =U i D i V i T ;

[0083] Next, put the diagonal matrix D i Become Here, ε is a very small quantity, typically set to 0.001;

[0084] Through this parameterization, the final covariance matrix is ​​∑′ i =U i D i ′V i T ;

[0085] This covariance matrix represents that the uncertainty of the direction of the normal vector of the current point in the current plane is very low, but the uncertainty of its specific position in the current plane is relatively high; that is, it is certain that the point is on the plane, but it is uncertain where it is exactly in the plane.

[0086] For LiDAR point cloud P c and local keyframe point cloud P l Repeat the above process for all; for P c Each point p in i In P l Find its nearest neighbor p j ∈P l The distance between these two points is d. ij =p i -p j .

[0087] The final geometric loss can be defined as:

[0088]

[0089] In the formula, It is geometric loss; T c This is the current pose of the lidar; ω ij It is a weight term used to filter out outliers; ∑′ j It is point p j The corresponding covariance matrix; ∑ i ′ is point p i The corresponding covariance matrix; if ∥d ij If ∥2>0.5 then ω ij =0, otherwise ω ij =1.

[0090] During the sensor tracking phase, the total loss is:

[0091]

[0092] In the formula, λ pho , λ geo These are two weighted terms;

[0093] The entire tracking process is iterative, requiring 30 iterations to obtain a satisfactory camera and LiDAR pose. During the tracking phase, the properties of the 3D Gaussian ellipsoid are fixed; only the camera and LiDAR poses are variable.

[0094] The keyframe selection strategy in VLGS-SLAM of this invention is as follows: if the distance between the current frame and the previous keyframe exceeds n... kfmeters, or the camera's rotation angle since the last keyframe exceeding r. kf If the frame is decremented, then the frame is considered a new keyframe.

[0095] In this embodiment of the invention, after the sensor tracking is completed, the poses of the camera and lidar are fixed, and the Gaussian map corresponding to the new observation area needs to be further optimized in order to better represent the current scene.

[0096] In a specific exemplary technical solution, firstly, a transparency map I is rendered based on the current camera pose. o and depth map I d Then, the current lidar point cloud P c Each point p i All are projected onto the image plane to obtain a two-dimensional projection point x. i =[u i v i ] T and the corresponding projection depth d i If any of the following conditions are met, then p is set to... i Add to Gaussian point cloud map:

[0097] 1)I o (x i <0.9;

[0098] 2)d i <0.8*I d (x i ).

[0099] After adding the new Gaussian, data from the current frame and the previous 5 frames are needed to update the local map; the map update process continues for 30 iterations. In each iteration, a frame is randomly selected, and an RGB image I is rendered. rgb and depth image I d .

[0100] Next, calculate the photometric loss, and then calculate the depth loss using the following formula:

[0101]

[0102] In the formula, L d This is a deep loss.

[0103] In the initial 3DGS, the scale of each Gaussian ellipsoid was unconstrained, allowing it to better capture scene details and render more accurate images. However, this also resulted in the Gaussians exhibiting a needle-like shape, with a larger scale in one direction and a smaller scale in the other two. These needle-shaped Gaussians negatively impact camera tracking because their appearance changes significantly with different viewpoints, leading to numerous artifacts in the rendered image. Furthermore, Gaussians farther from the camera, rendered as background, typically have a larger scale, but as the camera approaches, these distant Gaussians gradually move closer, affecting most pixels due to their larger scale, potentially resulting in poor rendering quality. To address these issues, this invention introduces scale regularization loss and size loss. For each Gaussian ellipsoid, this invention uses... Let represent its scale in three directions, then the scale regularization loss can be expressed as:

[0104]

[0105] In the formula, L reg For scale regularization loss; max(s) i ), min(s) i ) represent the elements that have obtained the maximum and minimum values, respectively;

[0106] The method for calculating dimensional loss is as follows:

[0107]

[0108] In the formula, L size For size loss; f is the camera's focal length; d i It is the depth of Gauss;

[0109] The meaning of size loss is to ensure that the radius of all Gaussian ellipsoids projected onto the image is less than 100 pixels, thus limiting the size of the Gaussians.

[0110] In summary, the total losses during the map update phase are:

[0111]

[0112] In the formula, λ pho , λ d , λ reg , λ size It is a weighting term.

[0113] During the map update phase, only the properties of the Gaussian ellipsoid are optimized, while the sensor pose remains fixed. Specifically, the overall process of the technical solution in this embodiment of the invention is divided into three stages: sensor tracking, map update, and loop closure detection. In the sensor tracking stage, sensor pose is estimated based on the existing map; during this stage, the sensor pose is variable, while the map is fixed. In the map update stage, the existing map is updated based on the existing sensor pose, so the sensor pose must be fixed, and only the map is optimized. Here, the Gaussian ellipsoid is the basic unit constituting the map; optimizing the properties of the Gaussian ellipsoid is equivalent to optimizing the map.

[0114] In this embodiment of the invention, loop closure detection in a SLAM system inevitably involves accumulated errors, as small tracking errors in each frame accumulate over time, eventually leading to significant trajectory drift. The loop closure detection module can correct these accumulated errors by detecting loop closure frames and optimizing the pose graph, thus solving this problem. Visualization results before and after loop closure are shown below. Figure 2 The final part, the core of loop closure detection, is identifying loop closure frames.

[0115] In an exemplary technical solution, for the current frame (I) c P c For example, using the NetVLAD algorithm, we can find the 10 keyframes that are closest to it {(I i P i ), i = 1, 2, ..., 10}. Then, for each LiDAR point cloud p in these candidate loopback frames. i Using geometric loss L geo Register the current point cloud. If the average point-to-point distance between two point clouds is less than 5cm, the registration is considered good, and the current frame and the i-th frame can be considered to have formed a loop.

[0116] To accelerate the loop closure detection phase, each current frame generates at most two loop closure constraints, and other candidate frames are discarded. Subsequently, a pose graph is used to optimize all frames within the loops; this optimization framework is based on the g2o framework. In the pose graph, each vertex represents the pose of a data frame, and the edges between vertices represent the relative pose constraints from the tracked pose and loop closure. Loop closure constraints are crucial during optimization because inaccurate loop closures can lead to pose optimization errors. To enhance robustness and accuracy, pose graph optimization is only performed when loop closures are detected in three consecutive frames. After optimization, all keyframes within the loops and the mapping loss L are used to calculate the loop closure. map Update the local map. Once loop closure detection is complete, the processing of the current input frame is finished. The system then loads the next frame from the dataset and performs sensor tracking, map updating, and loop closure detection steps again.

[0117] In summary, cameras and LiDAR are two commonly used sensor types for 3D reconstruction. Camera-based vision methods are susceptible to the effects of textureless regions and lighting, while LiDAR-based methods tend to degrade in scenes lacking significant structural features. Therefore, the current mainstream approach is to fuse images and LiDAR data for 3D reconstruction. Currently, the 3D Gaussian splashing method is very effective in the vision domain. This invention proposes an algorithm based on 3DGS to fuse image and LiDAR information for pose estimation and 3D reconstruction, which can be called VLGS-SLAM. To ensure the accuracy of scene geometry and reduce pose estimation errors caused by floating Gaussians, this invention uses LiDAR points as 3D Gaussian ellipsoids. To enhance pose estimation performance, this invention applies regularization to the scale property of the Gaussian ellipsoids, restricting the shape of each Gaussian ellipsoid. Furthermore, for loop closure detection, this invention combines image similarity and LiDAR point cloud distance to effectively detect and close loops.

[0118] In specific exemplary experiments, the experimental results show that the VLGS-SLAM of the present invention has reached the state-of-the-art level in the field of 3DGS-based SLAM, and is superior to many traditional SLAM algorithms.

[0119] In a specific embodiment of the present invention, the experimental comparisons are as follows:

[0120] 1) Test Dataset: The KITTI dataset was used to test the performance of the algorithm, and VLGS-SLAM was compared with other SLAM methods. The KITTI dataset is a publicly available large-scale outdoor dataset containing 11 sequences with a total track length of 22 kilometers.

[0121] 2) Evaluation Metrics: The Absolute Trajectory Error (ATE), commonly used in the SLAM field, is used for evaluation. This error is calculated by taking the distance between the optical center of each camera and the ground truth, and then calculating the root mean square of all distances to obtain the final trajectory error. This error metric is expressed in meters; the lower the error, the more accurate the trajectory.

[0122] 3) Comparison method: Several different types of SLAM methods were used; among them, the visual SLAM methods were ORBSLAM3 and OpenVSLAM; the LiDAR SLAM methods were MULLS, Fast-LIO2, and LeGO-LOAM; the image-LiDAR fusion SLAM method was CamVox; and the 3DGS-based SLAM methods were MonoGS and SplaTAM.

[0123] 4) Results Comparison: The trajectory errors of each method on the KITTI dataset are shown in Table 1; where X represents the inability to complete the current sequence; bold black text represents the best result of the current sequence, and underline represents the second best result of the current sequence.

[0124] Table 1. Comparison Results of Trajectory Errors

[0125]

[0126] As can be seen from Table 1, the method disclosed in the embodiments of the present invention achieves the best results on 5 sequences and is also the second best on the other 6 sequences. The final average result is also the best, which is a significant improvement and progress compared with the existing scheme.

[0127] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0128] Please see Figure 3 In this embodiment of the invention, a scene 3D reconstruction system based on camera-LiDAR fusion is provided, comprising:

[0129] The data acquisition module is used to acquire the relative pose between the camera and the LiDAR, as well as the camera images and LiDAR point cloud sensor data of the scene, based on the selected scene.

[0130] The 3D reconstruction module is used to obtain the scene 3D reconstruction results by performing sensor tracking, map updating and loop closure detection in sequence based on 3DGS, according to the relative pose between the camera and LiDAR, as well as the camera images and LiDAR point cloud sensor data of the scene.

[0131] In the sensor tracking phase, the poses of the camera and LiDAR are first initialized using a linear motion assumption. Then, pose optimization is performed by minimizing photometric and geometric losses to obtain the optimized poses of the camera and LiDAR. After sensor tracking is completed, the map update phase begins. First, a transparency map and a depth map are rendered based on the current camera pose. The LiDAR point cloud that meets the preset conditions is added to the Gaussian point cloud map. Then, the properties of the Gaussian point cloud are optimized based on photometric and depth losses to obtain the Gaussian map. After the map update is completed, the loop closure detection phase begins, where the Gaussian map is optimized and the optimized Gaussian map is used as the result of the scene's 3D reconstruction.

[0132] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to execute the operation of a scene 3D reconstruction method based on camera-LiDAR fusion.

[0133] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the scene 3D reconstruction method based on camera-LiDAR fusion in the above embodiments.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction of a scene based on camera-lidar fusion, characterized in that, The method comprises the following steps: Based on the selected scene, the relative pose between the camera and the laser radar and the camera image and the laser radar point cloud sensor data of the scene are obtained; According to the obtained relative pose between the camera and the laser radar and the camera image and the laser radar point cloud sensor data of the scene, sensor tracking, map updating and loop detection are sequentially performed based on 3DGS to obtain a scene three-dimensional reconstruction result; In the sensor tracking stage, the pose of the camera and the laser radar is first initialized by linear motion assumption, then the pose is optimized by minimizing photometric loss and geometric loss, and finally the optimized pose of the camera and the laser radar is obtained; after the sensor tracking is completed, the map updating stage is entered, the transparency map and the depth map are first rendered according to the current pose of the camera, the laser radar point cloud meeting the preset condition is added to the Gaussian point cloud map, then the properties of the Gaussian point cloud are optimized according to the photometric loss and the depth loss, and the Gaussian map is obtained; after the map updating is completed, the loop detection stage is entered, the Gaussian map is optimized, and the optimized Gaussian map is taken as the scene three-dimensional reconstruction result.

2. The method according to claim 1, wherein The step of optimizing the pose by minimizing photometric loss and geometric loss to finally obtain the optimized pose of the camera and the lidar L track is represented as: ； wherein , are the weight terms for photometric loss, geometric loss, respectively; L pho , L geo are the photometric loss, geometric loss, respectively.

3. The method according to claim 2, wherein The overall loss function L track In the computational expression of: ； wherein is a hyper-parameter; ‖*‖1 represents the L1 norm; SSIM( , ) represents the structural similarity of the rendered image to the ground truth image I . ； wherein is a weight term; is the distance between two points, , is a point in the laser radar point cloud , is the nearest neighbor in the local key frame point cloud , is the covariance matrix corresponding to the point , is the pose of the current laser radar; is the covariance matrix corresponding to the point .​ 4. The method according to claim 1, wherein In the step of optimizing the attributes of the Gaussian point cloud according to the photometric loss and the depth loss to obtain the Gaussian map, scale regularization loss and size loss are further added when the attributes of the Gaussian point cloud are optimized, and an overall optimization loss function is obtained L map The calculation expression is: ； wherein, is a weight term for photometric loss, L pho is a photometric loss; , , are weights for depth loss, scale regularization loss, and size loss, respectively; L d is a depth loss; L reg is a scale regularization loss; L size is a size loss.

5. The method according to claim 4, wherein the overall optimization loss function L map in the computational expression of: ； wherein, is a depth map; , are respectively a point a two-dimensional projection point and a corresponding projection depth obtained by projecting the point a lidar point cloud; ； max( min( x, y ), z ) = max( min( x, z ), y ) s i max( min( x, y ), z ) = max( min( x, z ), y ) s i max( min( x, y ), z ) = max( min( x, z ), y ) s i max( min( x, y ), z ) = max( min( x, z ), y ) 3 max( min( x, y ), z ) = max( min( x, z ), y ) ； In the formula, f is the focal length of the camera.

6. The method according to claim 1, wherein In the step of optimizing the properties of the Gaussian point cloud according to the photometric loss and the depth loss to obtain the Gaussian map, the properties include rotation, scaling, opacity and RGB color.

7. The method according to claim 1, wherein The step of optimizing the Gaussian map and taking the optimized Gaussian map as the scene three-dimensional reconstruction result comprises: First, the cumulative error of the pose of the camera and the laser radar is corrected by detecting the loop frame and the pose graph optimization according to the camera image and the laser radar point cloud sensor data of the scene and the optimized pose of the camera and the laser radar, to obtain the corrected pose, and then the Gaussian map is optimized according to the corrected pose, and the optimized Gaussian map is taken as the scene three-dimensional reconstruction result.

8. A camera-lidar fusion based scene 3D reconstruction system, characterized in that, The method comprises: A data acquisition module is configured to obtain the relative pose between the camera and the laser radar and the camera image and the laser radar point cloud sensor data of the scene based on the selected scene; A three-dimensional reconstruction module is configured to sequentially perform sensor tracking, map updating and loop detection based on 3DGS according to the obtained relative pose between the camera and the laser radar and the camera image and the laser radar point cloud sensor data of the scene, to obtain a scene three-dimensional reconstruction result. In the sensor tracking stage, the poses of the camera and the lidar are initialized by a linear motion assumption, then pose optimization is performed by minimizing photometric loss and geometric loss, and finally the optimized poses of the camera and the lidar are obtained; after the sensor tracking is completed, the map updating stage is entered, the transparency map and the depth map are rendered according to the current pose of the camera, the lidar point cloud meeting the preset condition is added to the Gaussian point cloud map, then the properties of the Gaussian point cloud are optimized according to the photometric loss and the depth loss, and the Gaussian map is obtained; after the map updating is completed, the loop detection stage is entered, the Gaussian map is optimized, and the optimized Gaussian map is taken as the scene three-dimensional reconstruction result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the camera-lidar fusion based scene three-dimensional reconstruction method in any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the camera-lidar fusion based scene three-dimensional reconstruction method in any one of claims 1 to 7 when executed by the processor.

Images