Robot loop detection method based on multi-task low-dimensional hidden code and neural radiation field

Abstract

This invention provides a robot loop closure detection method based on multi-task low-dimensional implicit codes and neural radiation fields, comprising the following steps: first, extracting the current frame information and tracking the camera pose; then, determining whether the current frame is a keyframe; if it is a keyframe, determining whether the current frame is a location frame or a common-view frame based on the keyframe; performing loop closure detection based on the clustering results of the location frames; after successfully detecting a loop, using a second-order graph optimization algorithm to adjust the camera pose; finally, performing mapping and global bundle adjustment based on the existing camera pose information to perform 3D modeling. This invention fully utilizes existing feature point information and the point cloud information provided by the image itself for loop closure detection. The frame information required for loop closure detection is directly extracted from the stored information without secondary processing, while still maintaining high detection accuracy and simplifying the loop closure detection process.

Description

Technical Field

[0001] This invention relates to the field of robot localization technology, specifically to a robot loop closure detection method based on multi-task low-dimensional hidden codes and neural radiation fields. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that enables intelligent devices such as robots or self-driving cars to navigate autonomously in unknown environments. It has wide applications in many fields. Currently, this technology typically integrates data from multiple sensors, such as LiDAR, cameras, and IMUs (Inertial Measurement Units), to improve the accuracy and robustness of localization and mapping.

[0003] The classic SLAM framework mainly consists of four parts: visual odometry, backend optimization, mapping, and loop closure detection. Visual odometry primarily relies on frame sequences to estimate the camera's trajectory, calculating the camera's relative pose change by analyzing changes in feature points between consecutive frames. Backend optimization is responsible for minimizing the overall system error using nonlinear optimization algorithms. Loop closure detection identifies whether the robot has returned to previously visited locations, which is crucial for correcting accumulated errors and improving long-term localization accuracy.

[0004] SLAM systems typically consist of multiple parts, such as tracking, mapping, and loop closure detection. Each part usually requires the same frame sequence to perform its function. Existing robot loop closure detection methods often perform feature extraction again when using frame data in different parts, resulting in redundancy. Furthermore, research on the detection methods and optimization strategies after loop closure detection in existing robot methods is still insufficient.

[0005] Latent code refers to a set of low-dimensional variables used to control generated data in generative models (such as generative adversarial networks GANs). Typically, latent code is only used as an encoding tool for local geometric features in maps, mainly serving scene reconstruction, and is not fully utilized to capture local geometric features of the scene. Summary of the Invention

[0006] The first objective of this invention is to overcome the shortcomings of the prior art and provide a robot loop closure detection method based on multi-task low-dimensional hidden codes and neural radiation fields. On the one hand, it focuses on using the same features for tracking and loop closure detection, and on the other hand, it provides a new loop closure optimization approach.

[0007] The technical solution adopted in this invention is as follows:

[0008] Step S1: Extract the information of the current frame in the image and encode it to obtain a low-dimensional hidden code;

[0009] Step S2: Estimate the pose of the current frame based on the pose of the previous frame (if it is the first frame, use the actual pose as the estimated pose), and optimize the pose of the current frame using the RGB information and geometric information obtained by low-dimensional hidden code and neural radiation field (NeRF). Then determine whether the current frame is a key frame. If it is, proceed to step S3; otherwise, proceed to step S6.

[0010] Step S3: Determine whether the current frame is a location frame. If so, randomly sample the pixels of the current frame and store the relevant information. Subsequently, directly extract the low-dimensional hidden code information of all the stored pixels and the geometric features generated by the Neural Radiation Field (NeRF) as local features for clustering. Otherwise, do not store. Then determine whether the current frame is a common view frame. If so, record the index number of the current common view frame. Otherwise, do not record.

[0011] The overlap between each current frame and the point cloud of all existing location frames is used as a criterion to determine whether the current frame should be added to the location frame sequence. The overlap formula is as follows: (Where, cnt is the number of effective projection points, and H and W are image parameters). If the overlap is less than 0.2, the current frame is added to the location frame sequence as a new location frame. The determination of a co-view frame only calculates the overlap with the previous co-view frame. (A location frame is always a co-view frame, but a co-view frame is not necessarily a location frame).

[0012] When a new location frame is generated, its local features are extracted and added to the local feature set of historical location frames. KMEANS clustering is then performed based on this set. The extracted local features originate from a low-dimensional implicit code and are the same set of features used for tracking camera pose and global bundle adjustment. They are encoded using a combination of low-dimensional implicit code and NeRF, meaning that the geometric and RGB information generated by combining low-dimensional implicit code and neural implicit field can be used for both step S2 (camera pose estimation) and step S3 (loop closure detection) preparation.

[0013] The system detects loop closures, generates VLAD (Vector of Locally Aggregated Descriptors) descriptors based on clustering results, and creates a ball tree. It then uses this tree for efficient nearest neighbor search, identifying the ten location frames most similar to the current frame. These ten frames are then filtered based on two conditions: first, their overlap with the current frame's point cloud is greater than 0.75 (calculated using the formula above); second, the distance between the two frames is greater than 30. The frame that meets both conditions and has the highest overlap (called the index frame) is selected to form a loop with the current frame. Based on the error, the poses of all co-view frames between the index frame and the current frame are adjusted. Finally, based on the final co-view frames and their relative poses, the poses of all keyframes are adjusted. Finally, based on the final keyframe poses, the poses of all frames between the index frame and the keyframes are adjusted.

[0014] Step S4: Perform loop closure detection based on the clustering results;

[0015] Step S5: After successfully detecting loop closure, use the second-order graph optimization algorithm to adjust the camera pose;

[0016] Construction of the loop-closure position graph: Read the estimated poses of all co-view frames between the index frame and the current frame, and use them as nodes of the graph; calculate the pose transformation between adjacent co-view frames based on the estimated poses, and use them as edges of the graph. Extract the point cloud information of the two adjacent frames before and after the index frame, the index frame, and the current frame, and use the point cloud ICP algorithm to calculate the pose transformation between the two frames, and use them as edges (loop-closure edges) of the graph.

[0017] The constructed graph represents the loop closure position. The ordinary edges and loop closure edges of the graph should be "closed," meaning that the positional changes from the loop closure frame to the current frame and from the current frame to the loop closure frame should theoretically be of the same magnitude but opposite direction. The relevant information of the graph is converted into a special Euclidean group form (SE3), and the Levenberg-Marquardt (LM) optimizer is initialized. The errors of the two "edges" (the pose transformation from the index frame to the current frame obtained by accumulating the estimated pose frame by frame and the pose transformation from the current frame to the index frame directly calculated by ICP) are used as the optimizer's loss function. By adjusting the graph nodes (i.e., the poses of the frames), the errors are reduced, thus completing the loop closure position optimization.

[0018] Step S6: Based on the existing camera pose information, perform mapping and global bundle adjustment, and then perform 3D modeling.

[0019] A second objective of the present invention is to provide a loop closure detection device for performing the above-described method, comprising:

[0020] The image acquisition module is used to acquire images collected by the robot.

[0021] The frame extraction module is used to identify and record keyframes, location frames, and shared view frames of an image, and to cluster location frames.

[0022] The loop closure detection module is used to perform loop closure detection based on the clustering results of the location frames.

[0023] A third object of the present invention is to provide a robot comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the loop closure detection method described above.

[0024] A fourth object of the present invention is to provide a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the above-described method.

[0025] The invention employing the above technical solution has the following advantages:

[0026] 1. The method of this invention makes full use of low-dimensional implicit code information. The local features required for tracking and loop closure detection and the features used by the global bundle are extracted from the same underlying features. They are all generated by combining low-dimensional implicit codes with NeRF, without the need for secondary extraction, and can still maintain high detection accuracy. This simplifies the loop closure detection process, reduces the consumption of system computing resources, improves positioning efficiency and system robustness. In this way, the potential for efficient positioning can be achieved without relying on additional feature extraction. This overcomes the shortcomings of existing technologies that only use inefficient optimization methods in the pose estimation process, without realizing that these low-dimensional implicit codes used for local geometric feature encoding can be directly used to guide the matching process.

[0027] 2. In this invention, loop closure optimization uses a second-order linear optimizer, which expands the flexible application of the second-order linear optimizer in this field. Attached Figure Description

[0028] Figure 1 This is a flowchart of the loop closure detection method;

[0029] Figure 2 Two frames represent the loop closure position. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] like Figure 1 As shown, this invention provides a robot loop closure detection method based on multi-task low-dimensional hidden codes and neural radiation fields, which is described in detail below:

[0032] The current frame is read in, and the pixel coordinates in the current frame are converted into continuous representations in the latent space through coordinate encoding. Then, the low-dimensional latent codes of these points are further extracted using a sparse parameter encoding strategy for subsequent calculations and processing.

[0033] If the current frame is the first frame, set the true pose to the estimated pose and insert the current frame into the keyframe sequence and the common view frame sequence;

[0034] If the current frame is not the first frame, continue with the following processing:

[0035] First, the pose of the current frame is estimated based on the relevant positional information of the previous frame. Then, a pixel is randomly selected from the current frame, and its low-dimensional implicit code information is read and input into NeRF. NeRF, based on the knowledge learned by the network, predicts RGB (color) and SDF (geometric) information and calculates the loss. The optimizer gradient is then used to update the network parameters, iteratively optimizing network performance and pose information until the model converges. After iteration, it is determined whether the current frame is a keyframe. If it is, no further processing is performed, and the next step continues. If not, the relative pose is calculated and stored for subsequent optimization.

[0036] Next, it's determined whether the frame is a keyframe. If it is, the algorithm determines whether it's a location frame or a shared-view frame. If it's a location frame, pixels are randomly sampled and used for subsequent clustering and loop closure detection. Shared-view frames are used for pose optimization after loop closure detection. The next step is loop closure detection. The local features used for loop closure detection come from the preceding low-dimensional latent code and do not require additional extraction. Next, loop closure optimization is performed. Before optimization, the influence of keyframes on ordinary frames during global bundle adjustment needs to be eliminated. A graph is constructed based on the loop closure edges and shared-view frames, and a second-order optimizer is used to draw the graph, thereby optimizing the pose information. After loop closure optimization, the pose between the two frames at the loop closure position is adjusted step by step. Finally, it's determined whether to perform global bundle adjustment and 3D reconstruction.

[0037] Example 1:

[0038] like Figure 2 The image shows the current frame tracked during the tracking process, and the 10 nearest keyframes found in the location frame. The overlap ratio is represented by scores1 and score2. The detailed process involves calculating the point cloud transformation pcd_cur between two frames, using the following formula: Calculate the projected coordinates pcd_u and pcd_v using the following formula: pcd_u= f x × ( pcd_cur[0,:] pcd_cur[2,:] )+ C x and pcd_ v = f y × ( pcd_cur[0,:] pcd_cur[2,:] )+ C y ; Calculate whether the projection point is within the image area and The formula is: ; The formula for calculating the number of effective projection points is: ; Calculate the percentage, the formula is as follows The scores for frames 1 to 2 and 2 to 1 are calculated separately: 0.79 and 0.81, respectively, both greater than 0.75, and the interval between the two frames is greater than 30, thus identifying them as loop-loop frames. The graph required for second-order optimization is constructed by extracting the estimated poses (est_c2w_data) of all common-view frames and converting them into SE3 form as graph nodes. The transformations of all adjacent common-view frames are calculated as edges of the graph, using the following formula: The inter-frame transform is directly calculated using ICP, and the loop closure edge is denoted as the edge. The error loss is calculated using the following formula: Initialize the linear solver, set the trust domain policy, create the LM optimizer, and create the scheduler. Set up the points and edges of the graph, use the loss as the optimization error, balance the error across each node, and finally return the updated pose, updating all previously estimated poses according to dependencies.

[0039] To further verify the effectiveness of the method of this invention, it was run on the ScanNet dataset and the Replica dataset, and the average ATE RMSE (cm) results are shown in Table 1 and Table 2 below, respectively. The following results demonstrate that the method of this invention achieves more accurate tracking results without reducing the robot's loop closure detection speed.

[0040] Table 1. Average ATE RMSE (cm) results for the ScanNet dataset.

[0041] Scene ID 0000 0059 0106 0169 0181 0207 iMAP* 55.95 32.06 17.50 70.51 32.10 11.91 NICE-SLAM 8.64 12.25 8.09 10.28 12.93 5.59 Co-SLAM 7.13 11.14 9.36 5.90 11.81 7.14 Ours 5.9 10.7 8.4 5.4 10.5 6.2

[0042] Table 2. Average ATE RMSE (cm) results for the ScanNet dataset.

[0043] Scene ID Room-0 Room-1 Room-2 iMAP* 5.23 3.09 2.58 NICE-SLAM 1.69 2.04 1.55 Co-SLAM 0.65 1.13 1.43 Ours 0.7 0.79 0.95

Claims

1. A method for detecting loop closures in robots based on multi-task low-dimensional hidden codes and neural radiation fields, characterized in that, Includes the following steps: Step S1: Extract the information of the current frame in the image and encode it to obtain a low-dimensional hidden code; Step S2: Estimate the pose of the current frame based on the pose of the previous frame, and record it as the estimated pose. If the current frame is the first frame, use the actual pose as the estimated pose. Optimize the estimated pose using the RGB information and geometric information obtained from the low-dimensional hidden code and neural radiation field. Then determine whether the current frame is a keyframe. If it is, proceed to step S3; otherwise, proceed to step S6. Step S3: Determine if the current keyframe is a location frame. If so, randomly sample pixels from the current frame, generate RGB and geometric information of the pixels using neural radiation fields, and store them. Subsequently, extract all stored RGB and geometric information of the pixels as local features for clustering. Otherwise, do not store them. Then, determine if the current frame is a shared-view frame. If so, record the index number of the shared-view frame; otherwise, do not record it. The specific conditions for determining a location frame are as follows: The current frame is determined to be a location frame based on the point cloud overlap between each current frame and all location frames. If the point cloud overlap is less than 0.2, it is a location frame. The specific conditions for determining co-view frames are as follows: The current frame is determined to be a co-view frame based on the point cloud overlap between the current frame and the previous co-view frame. If the point cloud overlap is less than 0.15, it is a co-view frame. Step S4: Perform loop closure detection based on the clustering results; Step S5: After successfully detecting loop closure, the camera pose is adjusted using a second-order graph optimization algorithm; specifically: S401: After successfully finding the final loop, use matrix multiplication to obtain the pose transformation of adjacent co-view frames using the camera pose obtained from step S2, and use it as part of the edge of the graph. Estimate the pose of all co-view frames between the index frame and the current frame as the vertices of the graph. S402: Extract the point cloud information of the index frame and the current frame, and use the ICP algorithm to obtain the pose transformation of the index frame and the current frame as another part of the graph edge; S403: Adjust the parameters of the Levenberg-Marquardt optimizer to optimize the graph; Step S6: Based on the existing camera pose information, perform mapping and global bundle adjustment, and then perform 3D modeling.

2. The robot loop closure detection method based on multi-task low-dimensional hidden codes and neural radiation fields according to claim 1, characterized in that, Step S4 is as follows: To determine whether the current frame forms a loop with all co-viewing frames, the method is as follows: VLAD descriptors are generated based on existing categories, and the list of closest matching frames is matched using a ball tree. Iterate through the frame list, checking each frame for compliance with the loop closure constraint. If the following two conditions are met: ① the overlap between the current frame's point cloud and the nearest frame in the frame list is greater than 0.75; ② the difference between the index number of the nearest frame in the frame list and the index number of the current frame is greater than 30, then a loop closure is formed. Then, select the frame with the largest overlap with the current frame's point cloud from all frames that form the loop closure as the index frame, and form the final loop with the current frame. If the conditions are not met, proceed to step S6.

3. The robot loop closure detection method based on multi-task low-dimensional hidden codes and neural radiation fields according to claim 1, characterized in that, The parameters of the Levenberg-Marquardt optimizer include a linear optimizer and a trust domain policy.

4. A loop closure detection device for implementing the method according to any one of claims 1-3, characterized in that, include: The image acquisition module is used to acquire images collected by the robot. The frame extraction module is used to identify and record keyframes, location frames, and shared view frames of an image, and to cluster location frames. The loop closure detection module is used to perform loop closure detection based on the clustering results of location frames.

5. A robot, characterized in that, include: At least one processor; The system also includes a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the robot loop closure detection method according to any one of claims 1-3.

6. A machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the robot loop closure detection method according to any one of claims 1-3.

Images