Intensive vision mapping system based on three-dimensional Gaussian distribution

Abstract

The invention discloses a dense vision mapping system based on three-dimensional Gaussian distribution, which combines binocular vision, an inertial measurement unit (IMU) and 3D Gaussian distribution to greatly improve the construction speed and precision of a three-dimensional map. Compared with the prior art, the problem of map holes in a traditional SLAM method is solved, RGB color information of a scene is reserved, and a more vivid scene reconstruction effect is achieved. Meanwhile, in the aspect of time consumption, compared with a current mainstream coomap reconstruction system, the time for obtaining a single pose is shortened by about 30%.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and unmanned aerial vehicle navigation technology, and is particularly applicable to a dense visual mapping system based on three-dimensional Gaussian distribution. Background Art

[0002] SLAM technology (Simultaneous Localization and Mapping), which means simultaneous positioning and mapping, has made rapid progress in recent years in the fields of drones, autonomous robots, and augmented reality. The core goal of SLAM technology is to gradually build a map of the three-dimensional environment through sensor data, and at the same time determine its own precise position in the map.

[0003] Existing SLAM systems are mainly divided into two categories: laser SLAM and visual SLAM. Among them, visual SLAM has received widespread attention and application due to its advantages such as low cost and easy integration.

[0004] Visual SLAM systems such as ORB-SLAM2 proposed by Raul Mur-Artal, JMM Montiel, and Juan D. Tardos, “ORB-SLAM: a versatile and accurate monocular SLAM System,” IEEE Transactions on Robotics, can achieve accurate 3D scene reconstruction using a monocular camera. However, they have many shortcomings when dealing with complex scenes, such as scale uncertainty, inability to estimate pure monocular rotation, easy loss due to fast motion, and interference from image occlusion and moving objects.

[0005] The paper Tong Qin, Peiliang Li, Shaojie Shen, VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, IEEE Transactions on Robotics proposed a method of performing positioning and attitude estimation with VINS-Fusion by simultaneously fusing data from visual cameras and inertial measurement units, providing more accurate and robust positioning and attitude estimation results. However, there is still a problem of map holes. The created map is grayscale, which cannot achieve a realistic map effect and loses the effective RGB information of the scene.

[0006] In recent years, Neural Radiance Field (NeRF) has been proposed as a new scene representation method, which provides the ability to generate realistic images under complex geometry and lighting conditions. However, the training and rendering process of NeRF is time-consuming and cannot meet the real-time requirements. In addition, NeRF uses implicit scene representation, which performs well in generating new perspective images, but does not allow intuitive acquisition of 3D scenes. Summary of the invention

[0007] The present invention aims to provide a dense visual mapping system based on three-dimensional Gaussian distribution, aiming to solve the problems of map holes, invalid scene RGB information and insufficient real-time performance in the current SLAM system.

[0008] To achieve the above object, the present invention adopts the following technical solutions: The dense visual mapping system based on three-dimensional Gaussian distribution of the present invention comprises the following steps: S1, initialize the mapping system; take the first frame RGB image as the origin of the world coordinate system, perform coordinate system synchronization, gravity alignment, time synchronization, and posture information deviation correction; S2, select a key frame from the RGB image, add it to the sliding window, and obtain the feature points and predicted pose information of the current frame image based on the key frame in the current sliding window; S3, removing bad points and unobserved feature points according to the reprojection error and depth judgment of the feature points to obtain an initial point cloud, and assigning an initial normal vector and color value to the initial point cloud; S4, optimize the global pose through Bundle Adjustment; S5, generating a 3D Gaussian ellipsoid for each of the initial point clouds in step S3, converting the pose optimized in step S4 into a camera-to-world matrix, calculating projection parameters according to camera parameters and image resolution, and projecting the initial point cloud onto a 2D plane; S6, delete the points whose depth value is less than or equal to the threshold, and use the rasterizer to perform fast and micro rasterization to achieve fast overall rendering; S7, determining the rectangular area projected onto the 2D plane and the loss between the pixel rendering result and the RGB image within the rectangular area by calculating the influence range of the 3D Gaussian ellipsoid; S8, performs backpropagation and updates parameters using the Adam optimizer, combining the adaptive density control loop and the pose optimized in step S4 to output a color perspective image.

[0009] Furthermore, the RGB image is acquired by a binocular camera; the posture information is captured by an IMU, including position, speed and attitude, and is solved using a PnP method.

[0010] Furthermore, the posture information deviation correction includes deviation correction of the IMU gyroscope and accelerometer.

[0011] Furthermore, the depth of the feature point is obtained by matching the feature point pairs through binocular triangulation.

[0012] Furthermore, obtaining the feature points of the current frame image includes, after synchronizing the left and right eye RGB images of the binocular camera according to the timestamp, tracking the current frame using the forward optical flow method of the feature points of the previous frame image, obtaining the successfully tracked feature points, and extracting new corner feature points.

[0013] Furthermore, the predicted pose information of the current frame image extracts the pose information obtained by the IMU of the previous frame into the current frame as the initial value of the nonlinear optimization, and predicts the pose information of the current frame.

[0014] Furthermore, the parameters updated using the Adam optimizer include initial point cloud coordinates, opacity, SH spherical harmonics, rotation matrix, scaling factor, and a two-dimensional tensor of maximum radius.

[0015] Furthermore, the projection parameters include horizontal and vertical viewing angles, a projection matrix, and a clipping coordinate system matrix.

[0016] The advantage of the present invention is that it combines binocular vision, IMU (inertial measurement unit), and 3D Gaussian distribution to greatly improve the speed and accuracy of building three-dimensional maps. Compared with the prior art, the present invention solves the problem of map holes in traditional SLAM methods, retains the RGB color information of the scene, and achieves a more realistic scene reconstruction effect. At the same time, in terms of time consumption, compared with the current mainstream colmap reconstruction system, the time to obtain a single posture is reduced by nearly 30%. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure is a flow chart of the system of the present invention.

[0018] Figure 2 The images are captured in real scenes.

[0019] Figure 3 This is a comparison chart of point cloud output between the present invention and the traditional SLAM method.

[0020] Figure 4 This is a comparison chart of the present invention and ORB-SLAM2 on the ICL-NUIM dataset.

[0021] Figure 5 New perspective images synthesized by the present invention in indoor and corridor environments.

[0022] Figure 6 This is a comparison diagram of three different types of initial point cloud inputs indoors for the present invention.

[0023] Figure 7 This is a comparison diagram of three different types of initial point cloud inputs in the corridor of the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The dense visual mapping system based on three-dimensional Gaussian distribution of the present invention comprises a quadcopter, an Intel RealSense D435i binocular camera, an IMU (inertial measurement unit) and a GPU graphics processor based on RTX 4060. The Intel RealSense D435i binocular camera is mounted on the quadcopter.

[0026] The Intel RealSense D435i binocular camera is responsible for collecting RGB images and depth information. The IMU is used to capture the acceleration, angular velocity and other motion data of the quadcopter, that is, to obtain the position, velocity and attitude of the binocular camera, and the PnP method is used to solve the position and attitude of the binocular camera.

[0027] The Intel RealSense D435i binocular camera and IMU collect environmental data in real time, providing the necessary input for positioning and mapping of the dense visual mapping system based on three-dimensional Gaussian distribution.

[0028] The core theory and method of the dense visual mapping system based on three-dimensional Gaussian distribution described in the present invention is to combine binocular vision, IMU fusion and 3D Gaussian distribution model, synchronize the IMU data with the image captured by the camera, build a more robust attitude estimation model, and realize high-precision positioning and trajectory estimation of the UAV through the visual inertial fusion algorithm, thereby achieving accurate posture tracking and trajectory estimation.

[0029] like Figure 1 As shown, the dense visual mapping system based on three-dimensional Gaussian distribution of the present invention comprises the following steps: S1, initialize the mapping system; initialize the first frame of RGB image as the origin of the world coordinate system, perform coordinate system synchronization, gravity alignment, time synchronization, and posture information deviation correction. The posture information deviation correction includes the deviation correction of the IMU gyroscope and accelerometer.

[0030] S2, select a key frame from the RGB image, add it to the sliding window, and obtain the feature points and predicted pose information of the current frame image based on the key frame in the current sliding window.

[0031] Due to the large amount of image information, it is unrealistic to maintain all frames in a real-time SLAM system. The current common practice is to extract a portion of image frames as key frames from all frame images through reasonable judgments, such as whether there are enough feature points, whether the variance is small, whether loops can be detected, and other logic. At the same time, several of Ajing's most recent key frames form a sliding window, such as 10 key frames. When a new image frame comes in, the earliest frame in the sliding window will be discarded through marginalization without affecting the amount of information. The key frames retained in the sliding window will participate in the current visual mapping.

[0032] When obtaining the feature points required for the current visual mapping, we first need to synchronize the left and right eye RGB images of the binocular camera according to the timestamp, and then use the forward optical flow method of the feature points of the previous frame image to track the current frame, obtain the successfully tracked feature points, remove the invisible feature points and extract new corner feature points.

[0033] When shooting the current frame, the position information of the binocular camera is the predicted pose information. The pose information obtained by the IMU data of the previous frame is extracted to the current frame as the initial value of nonlinear optimization to predict the pose information of the current frame. Nonlinear optimization can choose to use methods such as Gauss-Newton and gradient descent according to the situation.

[0034] S3, performing nonlinear optimization to remove bad points based on the reprojection error and depth judgment of the feature points, and moving the sliding window to remove unobserved feature points, to obtain an initial point cloud, and assigning an initial normal vector and color value to the initial point cloud. The depth of the feature point can be obtained by matching the feature point pairs through binocular triangulation.

[0035] S4, optimize the global pose through Bundle Adjustment.

[0036] S5, generating a 3D Gaussian ellipsoid for each of the initial point clouds in step S3, converting the optimized pose in step S4 into a camera-to-world matrix, calculating projection parameters according to camera parameters and image resolution, and projecting the initial point cloud onto a 2D plane. The projection parameters include horizontal and vertical field of view angles, projection matrix, and clipping coordinate system matrix.

[0037] S6, deleting points whose depth values ​​are less than or equal to a threshold, specifically points whose depth values ​​are less than or equal to 0.3. And using a rasterizer to perform fast and micro-rasterization to achieve fast overall rendering.

[0038] S7, by calculating the influence range of the 3D Gaussian ellipsoid, determining the rectangular area projected onto the 2D plane and the loss between the pixel rendering result in the rectangular area and the RGB image.

[0039] S8, performs back propagation and uses the Adam optimizer to update parameters, including the initial point cloud coordinates, opacity, SH spherical harmonics, rotation matrix, scaling factor, and the maximum radius of the two-dimensional tensor. Combined with the adaptive density control loop and the pose optimized in step S4, the color perspective image is output.

[0040] That is to say, the dense visual mapping system based on three-dimensional Gaussian distribution of the present invention adopts a 3D Gaussian distribution method and uses a three-dimensional Gaussian distribution as a scene representation model. An initial sparse point cloud is generated by a binocular camera, and then the three-dimensional coordinates of each point are calculated using the optimized pose information. These coordinates are then used as the center point, and a plurality of Gaussian ellipsoids are generated using a 3D Gaussian distribution as a point cloud representation of the scene.

[0041] In the process of generating point clouds, the Gaussian model is adaptively controlled in density to avoid over-reconstruction or under-reconstruction. At the same time, in order to eliminate floating points and excessive Gaussian distribution, the system periodically resets the Gaussian density and removes unreasonable points.

[0042] Compared with the prior art, the dense visual mapping system based on three-dimensional Gaussian distribution described in the present invention uses 3D Gaussian distribution to model the scene, which greatly improves the density and detail expression of the map and effectively solves the map hole problem commonly seen in traditional SLAM systems.

[0043] like Figure 2 As shown in Figure 1, it is a color image of the interior and corridor obtained in a real scene. Figure 2 a~c are indoor color images, Figure 2 d~f are color images of the corridor. Figure 3 , which is a comparison diagram of the dense point cloud output by the dense visual mapping system based on three-dimensional Gaussian distribution described in the present invention and the dense point cloud output by the VINS-Fusion+Dense Surfel Mapping method. Figure 3 a and b are the dense point clouds output by the VINS-Fusion+Dense Surfel Mapping method. Figure 3 c and d are dense point clouds output by the dense visual mapping system based on three-dimensional Gaussian distribution of the present invention. It can be seen from the figure that the method of the present invention can obtain a more realistic color point cloud, and it is shown in the red frame area that the present invention can fill the map holes, while the Dense Surfel Mapping method cannot.

[0044] Figure 4The point cloud map output by the system of the present invention is compared with the point cloud map output by ORB-SLAM2 on the ICL-NUIM dataset. Figure 4 a~d are point cloud maps output by the system of the present invention, wherein 4a is a point cloud map including the ceiling. Figure 4 e~h are the point cloud maps output by ORB-SLAM2, where 4e is the point cloud map including the ceiling. Figure 4 In the second to fourth columns of the figure, the top ceiling point cloud is deleted to provide a more intuitive view of the internal structure. The red box in the figure shows that the map reconstructed by ORB-SLAM2 has holes, while the present invention does not produce holes in these areas, effectively improving the map hole problem in the traditional SLAM method.

[0045] The present invention randomly inputs a camera-to-world matrix or a camera-to-world matrix along an arc trajectory and uses a trained model to output rendered images from multiple angles. Figure 5 As shown, high-quality and highly realistic unknown perspective images can be obtained.

[0046] Figure 6 and Figure 7 The colored point cloud maps obtained by inputting different types of initial point clouds in two different scenes, indoors and in corridors, are shown in the figure. It can be seen that the image quality performance of the present invention when inputting sparse point clouds is similar to that when inputting dense point clouds, indicating its excellent performance.

[0047] Experiments also show that the reconstruction accuracy of the dense visual mapping system based on three-dimensional Gaussian distribution in the indoor and corridor environment tests is 2.416 cm and 2.231 cm respectively, which is significantly better than 6.632 cm and 5.527 cm of VINS-Fusion, and 3.278 cm and 3.158 cm of VINS-Fusion+DenseSurfel Mapping. The average time to obtain a single pose is 0.530 seconds, which is nearly 30% faster than the 0.774 seconds of the colmap system.

[0048] In the ICL-NUIM data set, the reconstruction accuracy of the present invention and some SLAM methods are compared, and the output point cloud maps are compared. The average accuracy of the point cloud map output by the present invention is 1.925cm, which is nearly 30% higher than the average accuracy of the point cloud map output by the best ORB-SLAM2, which is 2.825cm. The maps reconstructed by traditional SLAM methods have holes, while the present invention does not produce holes in these areas, indicating that the present invention can effectively improve the mapping hole problem in traditional SLAM methods.

[0049] In summary, the innovation of the system of the present invention lies in the combination of IMU information with 3D Gaussian field for the first time, optimizing the attitude estimation, and has been verified on the actual UAV platform. Through this method, the system can generate dense, high-quality 3D point cloud maps with less input point cloud, and further improve the fitting accuracy of the model through adaptive density control of Gaussian model. In the experimental results, the performance of this method on the ICL-NUIM dataset (A Benchmark for RGB-D Visual Odometry, 3D Reconstruction and SLAM) is better than that of existing SLAM systems such as VINS-Fusion, especially in terms of map integrity and new perspective image synthesis.

Claims

1. A dense visual mapping system based on three-dimensional Gaussian distribution, characterized in that: The following steps are involved: S1, initialize the mapping system; take the first frame RGB image as the origin of the world coordinate system, perform coordinate system synchronization, gravity alignment, time synchronization, and posture information deviation correction; S2, select a key frame from the RGB image, add it to the sliding window, and obtain the feature points and predicted pose information of the current frame image based on the key frame in the current sliding window; S3, removing bad points and unobserved feature points according to the reprojection error and depth judgment of the feature points to obtain an initial point cloud, and assigning an initial normal vector and color value to the initial point cloud; S4, optimize the global pose through Bundle Adjustment; S5, generating a 3D Gaussian ellipsoid for each of the initial point clouds in step S3, converting the pose optimized in step S4 into a camera-to-world matrix, calculating projection parameters according to camera parameters and image resolution, and projecting the initial point cloud onto a 2D plane; S6, delete the points whose depth value is less than or equal to the threshold, and use the rasterizer to perform fast and micro rasterization to achieve fast overall rendering; S7, determining the rectangular area projected onto the 2D plane and the loss between the pixel rendering result and the RGB image within the rectangular area by calculating the influence range of the 3D Gaussian ellipsoid; S8, performs backpropagation and updates parameters using the Adam optimizer, combining the adaptive density control loop and the pose optimized in step S4 to output a color perspective image.

2. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The RGB image is acquired by a binocular camera; the position information is captured by an IMU, including position, speed and attitude, and is solved using a PnP method.

3. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The posture information deviation correction includes deviation correction of the IMU gyroscope and accelerometer.

4. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The depth of the feature point is obtained by matching the feature point pairs through binocular triangulation.

5. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: Obtaining the feature points of the current frame image includes, after synchronizing the left and right eye RGB images of the binocular camera according to the timestamp, tracking the current frame using the forward optical flow method of the feature points of the previous frame image, obtaining the successfully tracked feature points, and extracting new corner feature points.

6. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The predicted pose information of the current frame image extracts the pose information obtained by the IMU of the previous frame into the current frame as the initial value of the nonlinear optimization, and predicts the pose information of the current frame.

7. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The parameters updated using the Adam optimizer include initial point cloud coordinates, opacity, SH spherical harmonics, rotation matrix, scaling factor, and a two-dimensional tensor of the maximum radius.

8. The dense visual mapping system based on three-dimensional Gaussian distribution according to claim 1, characterized in that: The projection parameters include horizontal and vertical viewing angles, a projection matrix, and a clipping coordinate system matrix.

Images