Dynamic instant positioning and mapping method based on 3D Gaussian

Abstract

The invention belongs to the technical field of simultaneous localization and mapping in the aspect of computer vision, and particularly relates to a dynamic simultaneous localization and mapping (SLAM) method based on 3D Gaussian, which comprises the following steps of: aligning an RGB image captured by a sensor with a depth image, and processing the RGB image by using a semantic segmentation algorithm to obtain a depth image; setting a semantic tag for a pixel point where the semantic target is located; for the RGB image, an ORB feature point and a SuperPoint feature point are extracted by using a FAST algorithm and a SuperPoint algorithm respectively; feature points with semantic tags are recognized and removed, and then pose estimation of the current frame is optimized in combination with the depth image; projecting a semantic target of a historical frame into a current frame image through pose transformation, recognizing a dynamic object according to the similarity of the semantic target of the historical frame and the semantic target of the current frame and the intersection-union ratio of a semantic target mask, and respectively constructing a dynamic map and a static map; and further optimizing the static map by using a 3DGS technology to obtain a dense map finally represented by 3D Gaussian.

Description

Technical Field

[0001] The present invention belongs to the technical field of simultaneous localization and mapping in computer vision, and particularly relates to a dynamic simultaneous localization and mapping (SLAM) method based on 3D Gaussian. Background Art

[0002] With the rapid development of computer vision, deep learning, and robotics technology, SLAM systems have been widely used in various fields. However, in complex dynamic environments, traditional SLAM methods usually face problems such as positioning errors and map distortion due to their inability to effectively handle the interference of dynamic objects. Therefore, some dynamic SLAM algorithms for dynamic objects have emerged successively, aiming to detect and suppress the influence of dynamic objects in real time to improve the stability and robustness of the system.

[0003] Existing dynamic SLAM algorithms usually rely on feature point detection and matching to achieve pose estimation and map construction. However, the dense mapping results of feature point-based SLAM algorithms often do not have a high degree of scene restoration. In recent years, as an effective 3D point cloud rendering and representation method, 3DGS can efficiently process dense map data and provide a smooth geometric representation. However, its real-time performance still needs to be improved. Summary of the Invention

[0004] In view of this, the present invention provides a dynamic simultaneous localization and mapping method based on 3D Gaussian. The method uses an RGB-D visual odometer based on semantic information to estimate the pose of the system, and performs mapping based on 3DGS and a dynamic object discrimination method. By separating static and dynamic objects, a 3D Gaussian point cloud containing motion information is generated, thereby constructing an accurate environmental map.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A dynamic simultaneous localization and mapping method based on 3D Gaussian, comprising the following steps:

[0007] Step 1: Align the RGB image and the depth image captured by the sensor, and use a semantic segmentation algorithm to process the RGB image, and set semantic labels for the pixel points where semantic targets are located;

[0008] Step 2: For the RGB image, use the FAST algorithm and the SuperPoint algorithm to extract ORB feature points and SuperPoint feature points respectively;

[0009] Step 3: For the ORB feature points and SuperPoint feature points extracted in Step 2, identify and remove the feature points with semantic labels, and then optimize the pose estimation of the current frame in combination with the depth image;

[0010] Step 4: Project the semantic targets of the historical frame into the current frame image through the pose transformation estimated in Step 3. Identify dynamic objects based on the similarity between the semantic targets of the historical frame and the current frame and the intersection over union of the semantic target masks, and construct a dynamic map and a static map respectively;

[0011] Step 5: Further optimize the static map using 3DGS technology to obtain a final dense map represented by 3D Gaussian.

[0012] Optionally, the ORB feature points in the present invention include their positions and descriptors, and the SuperPoint feature points include their positions and descriptors.

[0013] Optionally, the pose estimation of the current frame by combining the depth image in the present invention is as follows: First, consider the pose estimation based on ORB feature points. When the pose estimation based on ORB feature points fails to track, then use the pose estimation based on SuperPoint feature points.

[0014] Optionally, the pose estimation of the ORB feature points in the present invention is as follows:

[0015]

[0016] The pose estimation of the SuperPoint feature points is as follows:

[0017]

[0018] where is the 3D coordinate of the feature point in the current frame image, is the 3D coordinate of the matching feature point corresponding to the previous frame image;

[0019] The 3D coordinate pi corresponding to the feature point (xi, yi) is:

[0020]

[0021] where d is the depth value corresponding to the feature point (xi, yi) in the depth image DDepth(t), and K is the internal parameter matrix of the camera.

[0022] Optionally, the structural similarity index of the semantic target i between the historical frame and the current frame in the present invention is:

[0023]

[0024] where μt and μt-1 are the means of the current frame and the previous frame image, and represent variances, σt ,t-1 is the covariance, and c1 and c2 are constants.

[0025] Optionally, the intersection over union of the computed semantic target mask in the present invention is as follows:

[0026]

[0027] where M i (t) and M i (t - 1) are the masks of the semantic targets in the current frame and the previous frame, respectively.

[0028] Optionally, in the present invention, when both the similarity and the intersection over union are less than a set threshold, the point cloud on the map semantic target is classified as and a static map is generated

[0029]

[0030] where pi is a static point that does not belong to the dynamic point cloud.

[0031] Optionally, the specific process of step 5 in the present invention is as follows:

[0032] First, using the 3DGS technology, static objects are modeled as 3D Gaussian point clouds;

[0033] Second, according to the pose of the current frame, the 3D Gaussian point clouds are projected onto the imaging plane, and the RGB image and the depth image are rendered. Further, the RGB rendering error and the depth rendering error are calculated:

[0034] Finally, the relevant parameters of the Gaussian point clouds are updated by minimizing the rendering error to obtain the static map with optimized parameters

[0035] Advantageous effects:

[0036] First, by combining semantic information and feature point information, the present invention can effectively reduce the interference of dynamic objects, extract features from each frame of image more precisely, and thus significantly improve the accuracy and robustness of pose estimation.

[0037] Second, the present invention adopts a dynamic object recognition and separation strategy to ensure that the finally generated map only contains static objects. This feature is particularly important for map construction and positioning in dynamic environments (such as mobile platforms, robots, etc.). The effective separation of dynamic objects can prevent them from interfering with map updates, thereby improving the accuracy of static environment mapping.

[0038] Third, by introducing the dense mapping method of 3DGS, a more refined and accurate dense 3D map can be obtained. This dense mapping method based on Gaussian point clouds can significantly improve the mapping accuracy when dealing with complex dynamic scenes and provide a continuous and smooth environmental representation.

[0039] Fourth, the present invention combines a 3D Gaussian distribution model with traditional SLAM technology, which can effectively eliminate the influence of dynamic objects in a dynamic environment and balance the real-time performance and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a basic flowchart of the dynamic SLAM method based on 3D Gaussian of the present invention.

[0042] Figure 2 It is the Gaussian rendering process based on semantic object masks of the present invention.

[0043] Figure 3 It is a comparison of the mapping results of the present invention with other algorithms under the walking_static sequence of the TUM RGB-D dataset. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The following will describe the embodiments of the present invention in detail with reference to the drawings.

[0045] It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other; and, based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present disclosure.

[0046] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement a device and / or practice a method. Additionally, this device and / or this method can be implemented using other structures and / or functions in addition to one or more of the aspects described herein.

[0047] An embodiment of the present application is a dynamic simultaneous localization and mapping (SLAM) method based on 3D Gaussian, as Figure 1 shown, and the specific process is as follows:

[0048] Step 1, data preprocessing: Each frame of RGB image I RGB (t) captured by the sensor of the scene to be located is aligned with the depth image I Depth (t) to ensure that the image data at the same timestamp corresponds, and calibration is performed using the nearest neighbor method. The RGB image is processed using a semantic segmentation algorithm, and semantic labels are set for the pixel points where semantic targets are located (such as people, vehicles, buildings, etc.).

[0049] Step 2, extract feature point information from the RGB image: Use the FAST algorithm and the SuperPoint algorithm to extract ORB feature points and SuperPoint feature points for the RGB image. Specifically:

[0050] Use the FAST algorithm to extract ORB feature points from the RGB image Each feature point contains its position and descriptor: Among them, (x i , y i ) is the position of the feature point, and d ORB (x i , y i ) is the descriptor.

[0051] Use the SuperPoint algorithm to extract SuperPoint feature points from the RGB image Each feature point f SuperPoint also contains position and descriptor. Among them, d SuperPoint (x i , y i ) is the descriptor of the feature point.

[0052] Step 3, process each frame of image in chronological order. Denote the image frame to be processed at the current time t as the current frame. Based on the depth image and semantic information obtained in Step 1 and the ORB feature points and SuperPoint feature points extracted in Step 2, optimize the pose estimation of the current frame.

[0053] It mainly includes the following two steps:

[0054] (1) Remove the feature points on the semantic target

[0055] Identify the feature points belonging to the semantic target These feature points belonging to the semantic target will be removed from the original feature point set:

[0056]

[0057] Among them, and They are the set of effective ORB feature points and SuperPoint feature points after removing semantic targets respectively.

[0058] (2) Optimize the pose based on depth information and effective feature points

[0059] Combine with the depth image D Depth (t) Use the effective ORB feature points and SuperPoint feature points after removing semantic targets for pose estimation. Specifically: First, consider the pose estimation based on ORB feature points. When the tracking fails in the pose estimation based on ORB, then consider using the pose estimation based on SuperPoint feature points.

[0060] Pose estimation of ORB feature points: For each frame of image, optimize the pose according to the matching pairs of feature points with the previous frame:

[0061]

[0062] Pose estimation of SuperPoint feature points: For each frame of image, optimize the pose according to the matching pairs of feature points with the previous frame:

[0063]

[0064] Among them, is the 3D coordinate of the feature point in the current frame image, is the 3D coordinate of the matching feature point corresponding to the previous frame image.

[0065] The 3D coordinate p of the feature point (x i , y i ) is calculated by the formula:

[0066]

[0067] Among them, d is the depth value corresponding to the feature point (x i , y i ) in the depth image D Depth (t), and K is the internal parameter matrix of the camera.

[0068] In this embodiment, to ensure the real-time performance of pose estimation, priority is given to using ORB feature points for pose estimation. When the tracking fails in the pose estimation based on ORB, then consider using the pose estimation based on SuperPoint feature points, so as to achieve the purpose of balancing the robustness and real-time performance of the algorithm. This step enhances the pose estimation accuracy of the system in a dynamic environment by combining semantic labels with the depth image.

[0069] Step 4, Dynamic object recognition and construction of dynamic and static maps.

[0070] By comparing the similarity (ssim) of semantic objects and the intersection over union (IoU) of the semantic object mask M between the historical frame and the current frame, the motion of the object is determined. According to the motion situation, the current scene map is separated into a dynamic map containing only dynamic objects and a static map without dynamic objects. Specifically, it is divided into the following 4 steps: i (1) Calculate relevant metrics.

[0071] (1) Calculate relevant metrics.

[0072] The pose T estimated in step 3 is used to project the semantic object i of the historical frame into the current frame. The structural similarity (ssim) metric between the semantic object i of the historical frame and the current frame is compared. The calculation formula is:

[0073]

[0074] where μ t and μ t-1 are the means of the current frame and the previous frame images, and are their variances, σ t,t-1 is the covariance, and c1 and c2 are constants used to avoid a zero denominator.

[0075] Calculate the intersection over union (IoU) of the semantic object mask:

[0076]

[0077] where M i (t) and M i (t - 1) are the masks of the semantic objects of the current frame and the previous frame.

[0078] (2) Discrimination of dynamic objects.

[0079] If the above metrics are all less than a certain threshold, the point cloud on the map semantic object is classified as Then, according to the discrimination result of dynamic objects, the dynamic objects are removed and an accurate map of static objects is generated.

[0080]

[0081] Finally, according to the motion of the dynamic objects, the 3D point cloud of the dynamic objects is stored separately

[0082] Step 5: Further optimize the static map using 3DGS technology to obtain a final dense map represented by 3D Gaussians.

[0083] First, use 3DGS technology to model static objects as 3D Gaussian point clouds. The point cloud of each static object is represented by a Gaussian distribution:

[0084]

[0085] Among them, μ i and ∑ i are the mean and covariance of the i-th point cloud respectively, representing the spatial distribution of the point cloud.

[0086] Secondly, project the 3D Gaussian point cloud onto the imaging plane according to the pose of the current frame. The projection process is as Figure 2 shown, and the RGB image and depth image are rendered. Furthermore, the RGB rendering error and depth rendering error are calculated:

[0087]

[0088] E RGB (t) = ||I RGB,render (t) - I RGB (t)|| 2

[0089] E Depth (t) = ||I Depth,render (t) - I Depth (t)|| 2

[0090] Among them, π(·) represents the function of projecting a three-dimensional Gaussian point onto the image plane. In this way, the rendering error can be quantified as the pixel error in the image.

[0091] Finally, update the relevant parameters of the Gaussian point cloud by minimizing the rendering error to obtain the static map with optimized parameters

[0092] The present invention can directly provide accurate pose information for robots or other intelligent devices. It adopts an efficient pose estimation method and a high-fidelity map representation method, so that while ensuring accuracy, it also has good real-time performance. This technology has broad application prospects and can be used in multiple fields such as robot navigation, virtual reality, augmented reality, and driverless driving, providing high-precision positioning and map construction support for intelligent devices in these fields, and promoting the development and innovation of related technologies. Especially in a dynamic environment, it improves the accuracy and robustness of the positioning and mapping of robots or intelligent devices.

[0093] Next, the effectiveness of the method of the present invention will be tested through simulation experiments.

[0094] The dynamic simultaneous localization and mapping (SLAM) method based on 3D Gaussian was experimentally evaluated on two dynamic datasets, TUM RGB-D and Tartanair, to verify its effectiveness in dynamic environments. For this purpose, the proposed algorithm was compared with the SplaTAM algorithm in the experiment, and the pose estimation accuracy, mapping accuracy, and real-time performance were mainly tested.

[0095] On the TUM RGB-D dataset, the experimental results show that the algorithm of the present invention has significantly better pose estimation accuracy than the SplaTAM algorithm in most dataset sequences. Table 1 shows the comparison of the SplaTAM and GSD-SLAM algorithms on the TUM RGB-D dataset in terms of evaluation metrics for image reconstruction (psnr, ssim, lpips), depth reconstruction (rmse), pose evaluation (ape), etc. The results show that the method of the present invention successfully reduces the mapping error and localization error in dynamic scenes through accurate pose estimation and dynamic object separation. In the TUM RGB-D dataset, the method of the present invention improves by about 22.23% compared with SplaTAM in the psnr (↑) metric, by about 24.35% in the ssim (↑) metric, decreases by about 48.37% compared with SplaTAM in the lpips (↓) metric, decreases by about 43.41% in the rmse (↓) metric, and decreases by about 69.12% in the ape (↓) metric compared with SplaTAM. This performance improvement and error reduction are mainly due to the strategy of the method of the present invention in dealing with dynamic objects, avoiding the influence of dynamic objects on the reconstruction of the static environment. Figure 3 The comparison of the mapping results between the method of the present invention and the SplaTAM algorithm in the "walking_static" sequence of the TUM RGB-D dataset is shown. It can be seen that the method of the present invention can better separate dynamic objects from the static background and maintain a more accurate map update, avoiding map distortion caused by dynamic objects.

[0096] On the Tartanair dataset, especially in scenes such as "RoadCrossing04" and "RoadCrossing05" that contain a large number of dynamic objects, SplaTAM cannot achieve normal tracking, while the method of the present invention shows better mapping accuracy and localization robustness. Table 2 shows the comparison of the SplaTAM and GSD-SLAM algorithms on the Tartanair dataset in terms of evaluation metrics for image reconstruction (psnr, ssim, lpips), depth reconstruction (rmse), pose evaluation (ape), etc. It can be seen that SplaTAM cannot track normally on some datasets, while the method of the present invention has better robustness.

[0097] For real-time performance, we compared the performance of the two algorithms in terms of the image processing time per frame on the TUM RGB-D and Tartanair datasets. Table 3 lists the average processing times of the two algorithms in each data sequence. The experimental results show that the method of the present invention is superior to SplaTAM in real-time performance on both the tracking and mapping threads. This advantage is mainly due to the fact that the method of the present invention uses a more efficient and accurate pose estimation method.

[0098] The method of the present invention shows excellent performance on the two dynamic datasets of TUM RGB-D and Tartanair, especially having obvious advantages over the SplaTAM algorithm in terms of pose estimation accuracy, mapping accuracy, and real-time performance. This indicates that the proposed method can effectively cope with the challenges in dynamic environments, having better robustness and broad application potential.

[0099] Table 1 Comparison of various indicators of SplaTAM and GSD-SLAM algorithms on the TUM RGB-D dataset

[0100]

[0101] Table 2 Comparison of various indicators of SplaTAM and GSD-SLAM algorithms on the Tartanair dataset

[0102]

[0103] Table 3 Comparison of the real-time performance of SplaTAM and GSD-SLAM

[0104]

[0105] In summary, the above are only the preferred embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A dynamic real-time positioning and mapping method based on 3D Gaussian, characterized in that: The steps include: Step 1: Align the RGB image captured by the sensor with the depth image, and use the semantic segmentation algorithm to process the RGB image and set semantic labels for the pixels where the semantic target is located; Step 2: For the RGB image, use the FAST algorithm and the SuperPoint algorithm to extract ORB feature points and SuperPoint feature points respectively; Step 3: For the ORB feature points and SuperPoint feature points extracted in step 2, identify and remove the feature points with semantic labels, and then optimize the pose estimation of the current frame in combination with the depth image; Step 4: Project the semantic target of the historical frame into the current frame image through the pose transformation estimated in step 3. According to the similarity of the semantic targets of the historical frame and the current frame and the intersection-over-union ratio of the semantic target masks, identify the dynamic objects and construct the dynamic map and static map respectively. Step 5: Use 3DGS technology to further optimize the static map and obtain the final dense map represented by 3D Gaussian.

2. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 1, characterized in that: The ORB feature point includes its position and descriptor, and the SuperPoint feature point includes its position and descriptor.

3. The method for dynamic real-time positioning and mapping based on 3D Gaussian according to claim 1, characterized in that: The method of optimizing the pose estimation of the current frame in combination with the depth image is as follows: firstly, the pose estimation based on the ORB feature points is considered; when the pose estimation based on the ORB feature points fails to track, the pose estimation based on the SuperPoint feature points is used.

4. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 3, characterized in that: The pose estimation of the ORB feature point is: The pose estimation of the SuperPoint feature point is: in, is the 3D coordinate of the feature point in the current frame image, It is the 3D coordinates of the matching feature points corresponding to the previous frame image; Feature point (x i ,y i ) corresponds to the 3D coordinate p i for: Among them, d is the feature point (x i ,y i ) in the depth image D Depth (t) corresponds to the depth value, and K is the intrinsic parameter matrix of the camera.

5. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 1, characterized in that: The structural similarity index of the semantic target i between the historical frame and the current frame is: Among them, μ t and μ t-1 is the mean of the current frame and the previous frame image, and represents the variance, σ t,t-1 is the covariance, c1 and c2 are constants.

6. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 1 or 5, characterized in that: The calculation of the intersection-over-union ratio of the semantic target mask is: Among them, M i (t) and M i (t-1) is the mask of the semantic object of the current frame and the previous frame.

7. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 6, characterized in that: When the similarity and intersection-over-union ratio are both less than the set threshold, the point cloud on the map semantic target is classified as And generate a static map Among them, p i is a static point that does not belong to the dynamic point cloud.

8. The 3D Gaussian-based dynamic real-time positioning and mapping method according to claim 1, characterized in that: The specific process of step 5 is as follows: First, the static object is modeled as a 3D Gaussian point cloud using 3DGS technology; Secondly, the 3D Gaussian point cloud is projected onto the imaging plane according to the pose of the current frame, and the RGB image and depth image are rendered. The RGB rendering error and depth rendering error are further calculated: Finally, the relevant parameters of the Gaussian point cloud are updated by minimizing the rendering error to obtain a static map with optimized parameters.

Images