Robot loopback detection method based on multi-task low-dimensional implicit code and neural radiation field

Abstract

The invention provides a robot loopback detection method based on a multi-task low-dimensional implicit code and a neural radiation field, and the method comprises the following steps: firstly extracting the information of a current frame, tracking the pose of a camera, then judging whether the current frame is a key frame, if the current frame is the key frame, judging whether the current frame is a place frame and a common-view frame according to the key frame, and if the current frame is not the key frame, executing the next step; loopback detection is carried out according to the clustering result of the place frames, and after loopback is successfully detected, a second-order optimization algorithm of the graph is used for camera pose adjustment; and finally, mapping and global beam adjustment are carried out according to the existing camera pose information, and three-dimensional modeling is carried out. According to the method, the existing feature point information and the point cloud information provided by the picture are fully utilized to carry out loopback detection, the frame information required by the loopback detection is directly extracted from the stored information, secondary processing is not needed, high detection precision can still be kept, and the loopback detection process is simplified.

Description

Technical Field

[0001] The present invention relates to the field of robot positioning technology, and in particular to a robot loop detection method based on multi-task low-dimensional latent codes and neural radiation fields. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that allows intelligent devices such as robots or self-driving cars to navigate autonomously in unknown environments. It is widely used in many fields. Currently, this technology usually integrates multiple sensor data, such as LiDAR, camera, IMU (inertial measurement unit), etc., to improve the accuracy and robustness of positioning and mapping.

[0003] The classic SLAM framework mainly consists of four parts: visual odometry, backend optimization, mapping, and loop detection. Visual odometry mainly relies on frame sequences to estimate the motion trajectory of the camera, and calculates the relative position change of the camera by analyzing the changes in feature points between consecutive frames. Backend optimization is responsible for minimizing the error of the entire system through nonlinear optimization algorithms. Loop detection identifies whether the robot has returned to a previously visited location, which is crucial for correcting accumulated errors and improving the accuracy of long-term positioning.

[0004] SLAM systems usually include multiple parts, such as tracking, mapping, and loop detection. Usually, each part needs to use the same frame sequence to perform its own function. Existing robot loop detection methods usually perform feature extraction again when using frame data in different parts, resulting in redundant work. In addition, there is still a lack of research on the detection method of existing robot loop detection methods and the optimization method after loop detection.

[0005] A low-dimensional latent code refers to a set of low-dimensional variables used to control the generated data in a generative model (such as a generative adversarial network (GAN)). Usually, low-dimensional latent codes are only used as an encoding tool for local geometric features in a map, mainly serving scene reconstruction, and are not fully utilized to capture the local geometric features of the scene. Summary of the invention

[0006] The first purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a robot loop detection method based on multi-task low-dimensional latent code and neural radiation field. On the one hand, it is committed to using the same feature for tracking and loop detection, and on the other hand, it provides a new loop optimization idea.

[0007] The technical solution adopted by the present invention is as follows: Step S1, extracting information of the current frame in the picture and encoding it to obtain a low-dimensional hidden code; Step S2, estimate the pose of the current frame based on the pose of the previous frame (if it is the first frame, use the actual pose as the estimated pose), and optimize the pose of the current frame through the RGB information and geometric information obtained by the low-dimensional latent code and the neural radiance field (NeRF), and then determine whether the current frame is a key frame, if so, go to step S3, if not, go to step S6; Step S3, determine whether the current frame is a location frame, if so, randomly sample the pixels of the current frame and store the relevant information, and then directly extract the low-dimensional hidden code information of all the stored pixels and the geometric features generated by the neural radiation field (NeRF) as local features for clustering, otherwise do not store; then determine whether the current frame is a common view frame, if so, record the index number of the current common view frame, otherwise do not record; The overlap between each current frame and all existing location frame point clouds is used as a standard to determine whether the current frame should be added to the location frame sequence. The overlap formula is: (where cnt is the number of valid projection points, H and W are image parameters). If the overlap is less than 0.2, the current frame is a new location frame and is added to the location frame sequence. The judgment of the common view frame only calculates the overlap with the previous common view frame. (A location frame must be a common view frame, but a common view frame is not necessarily a location frame).

[0008] When a new location frame is generated, the local features of the new location frame are extracted and added to the local feature set of the historical location frame, and KMEANS clustering is performed based on the set. The extracted local features are derived from the low-dimensional latent code, which is the same set of features used for tracking camera pose and global bundle adjustment. They are encoded by combining low-dimensional latent code with NeRF, that is, the geometric information and RGB information generated by combining low-dimensional latent code with neural implicit field can be used for both step S2 to estimate camera pose and S3 to prepare for loop detection.

[0009] Detect whether there is a loop, generate a VLAD (Vector of Locally Aggregated Descriptors) descriptor based on the clustering results, and create a ball tree, which is used for efficient nearest neighbor search to obtain the top ten location frames with the closest similarity to the current frame. Filter these ten frames. Condition 1: The overlap with the point cloud of the current frame is greater than 0.75, and the calculation formula is as above. Condition 2: The distance between the two frames is greater than 30. Select the frame that meets the above two conditions and has the largest overlap (called the index frame) to form a loop with the current frame. According to the error, adjust the pose of all co-view frames between the index frame and the current frame. Adjust the pose of all key frames according to the final co-view frame and relative pose, and adjust the pose of all frames between the index frame and the key frame according to the final key frame pose.

[0010] Step S4: performing loop detection according to the clustering results; Step S5: After the loop is successfully detected, the camera posture is adjusted using the second-order optimization algorithm of the graph; Construction of loop position graph: read the estimated poses of all co-view frames between the index frame and the current frame as the nodes of the graph; calculate the pose transformation between adjacent co-view frames based on the estimated poses as the edges of the graph. Extract the point cloud information of the two adjacent frames before and after the index frame, the index frame and the current frame, and use the point cloud ICP algorithm to calculate the pose transformation of the two frames as the edges of the graph (loop edges).

[0011] The constructed graph represents the loop position. The normal edges of the graph and the loop edges should be "closed", that is, the position changes from the loop frame to the current frame and from the current frame to the loop frame should be of the same size and opposite in direction in theory. The relevant information of the graph is converted into a special Euclidean group form (SE3), the Levenberg-Marquardt (LM) optimizer is initialized, and the error of the two "edges" (the pose transformation from the index frame to the current frame obtained by accumulating the estimated pose frame by frame and the pose transformation from the current frame to the index frame directly calculated by ICP) is calculated as the loss function of the optimizer. The error is reduced by adjusting the graph nodes (i.e. the pose of the frame) to complete the loop position optimization.

[0012] Step S6: perform mapping and global bundle adjustment based on the existing camera pose information, and then perform 3D modeling.

[0013] A second object of the present invention is to provide a loop detection device for executing the above method, comprising: An image acquisition module, used to acquire images collected by the robot; The frame extraction module is used to judge and record the key frames, location frames, and common view frames of the image, and cluster the location frames; The loop detection module is used to perform loop detection based on the clustering results of the location frame.

[0014] The third object of the present invention is to provide a robot, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned loop detection method.

[0015] A fourth object of the present invention is to provide a machine-readable storage medium storing machine-executable instructions, which, when called and executed by a processor, prompt the processor to implement the above method.

[0016] The invention adopting the above technical solution has the following advantages: 1. The method of the present invention makes full use of low-dimensional latent information. The local features required for tracking and loop detection and the features used by the global bundle are extracted from the same underlying features. They are all generated by combining low-dimensional latent codes with NeRF, without the need for secondary extraction, and can still maintain high detection accuracy, simplifying the loop detection process, reducing system computing resource consumption, and improving positioning efficiency and system robustness. In this way, the potential for efficient positioning can be achieved without relying on additional feature extraction, overcoming the defect that the prior art only uses low-efficiency optimization methods in the process of pose estimation without realizing that these low-dimensional latent codes used for encoding local geometric features can be directly used to guide the matching process; 2. The loop optimization in the present invention uses a second-order linear optimizer, which expands the flexible application of the second-order linear optimizer in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of loop detection method; Figure 2 There are two frames indicating the loop position. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0019] like Figure 1 As shown, the present invention provides a robot loop detection method based on multi-task low-dimensional latent code and neural radiation field, which is described in detail as follows: The current frame is read in, and the pixel coordinates in the current frame are converted into continuous representations in the latent space through coordinate encoding. Then, the sparse parameter encoding strategy is used to further extract the low-dimensional latent codes of these points for subsequent calculations and processing.

[0020] If the current frame is the first frame, set the true pose to the estimated pose, and insert the current frame into the key frame sequence and the common view frame sequence; If the current frame is not the first frame, continue to process as follows: First, the pose of the current frame is estimated based on the relevant position information of the previous frame, and the pixels of the current frame are randomly selected, and the low-dimensional hidden information of the pixels is read and input into NeRF. NeRF predicts RGB (color) and sdf (geometry) and other information based on the knowledge learned by the network and calculates the loss, so as to use the optimizer gradient to update the network parameters, so as to iteratively optimize the network performance and pose information until the model converges. After the iteration is completed, it is determined whether the current frame is a key frame. If it is, it will not be processed and continue to the next step. If not, the relative pose is calculated and stored for subsequent optimization.

[0021] Next, determine whether it is a key frame. If it is a key frame, determine whether it is a location frame or a common view frame according to the corresponding algorithm. If it is a location frame, randomly sample the pixels of the current location frame and use them for subsequent clustering and loop detection; the common view frame is used for pose optimization after the loop position is detected. The next step is loop detection. The local features used for loop detection come from the previous low-dimensional hidden code and do not require additional extraction. Next, loop optimization is performed. Before optimization, it is necessary to eliminate the influence of key frames relative to ordinary frames during global bundle adjustment. According to the loop edge and common view frame construction graph, a second-order optimizer is used to draw the graph to optimize the pose information. After loop optimization, the pose between the two frames of the loop position is adjusted step by step. Next, determine whether to perform global bundle adjustment and whether to perform three-dimensional reconstruction.

[0022] Embodiment 1: like Figure 2 As shown in the figure, the 10 most adjacent key frames found in the location frame for the current frame tracked during the tracking process. The overlap rate is represented by scores score1 and score2. The detailed process is to calculate the point cloud transformation pcd_cur between the two frames. The calculation formula is ; Calculate the projection coordinates pcd_u and pcd_v, the calculation formula is pcd_u= f x × ( pcd_cur[0,:] pcd_cur[2,:] )+ C x and pcd_ v = f y × ( pcd_cur[0,:] pcd_cur[2,:] )+ C y ; Calculate whether the projection point is within the image range and , the formula is; calculate the number of effective projection points, the formula is ; Calculate the proportion score, the formula is ; Calculate the scores from frame 1 to frame 2 and frame 2 to frame 1 respectively, and the scores are 0.79 and 0.81, both greater than 0.75, and the interval between the two frames is greater than 30, which is determined to be a loop frame. Construct the graph required for the second-order optimization, extract all the common view frame estimated poses (est_c2w_data) and convert them into SE3 form as graph nodes, calculate all adjacent common view frame transformations as graph edges, and the calculation formula is , use ICP to directly calculate the transformation between two frames as the loop edge and record it as edge, and calculate the error loss. The formula is ; Initialize the linear solver, set the trust region strategy, create the LM optimizer, and create the scheduler. Set the points and edges of the graph, use the loss as the optimization error, balance the error to each node, and finally return the updated pose, and update all the estimated poses according to the dependencies.

[0023] To further verify the effectiveness of the method of the present invention, the method is run on the ScanNet dataset and the Replica dataset, and the average ATE RMSE (cm) results are shown in Tables 1 and 2. The following results show that the tracking results of the method of the present invention are more accurate while the robot loop detection speed is not reduced.

[0024] Table 1 Average ATE RMSE (cm) results of ScanNet dataset Scene ID 0000 0059 0106 0169 0181 0207 iMAP* 55.95 32.06 17.50 70.51 32.10 11.91 NICE-SLAM 8.64 12.25 8.09 10.28 12.93 5.59 Co-SLAM 7.13 11.14 9.36 5.90 11.81 7.14 Ours 5.9 10.7 8.4 5.4 10.5 6.2 Table 2 Average ATE RMSE (cm) results of ScanNet dataset Scene ID Room-0 Room-1 Room-2 iMAP* 5.23 3.09 2.58 NICE-SLAM 1.69 2.04 1.55 Co-SLAM 0.65 1.13 1.43 Ours 0.7 0.79 0.95

Claims

1. A robot loop detection method based on multi-task low-dimensional latent code and neural radiation field, characterized in that: The following steps are involved: Step S1, extracting information of the current frame in the image and encoding it to obtain a low-dimensional hidden code; Step S2, estimate the pose of the current frame according to the pose of the previous frame of the current frame, recorded as the estimated pose, if the current frame is the first frame, use the actual pose as the estimated pose; optimize the estimated pose through the RGB information and geometric information obtained by the low-dimensional latent code and the neural radiation field; then determine whether the current frame is a key frame, if so, go to step S3, if not, go to step S6; Step S3, determine whether the current key frame is a location frame, if so, randomly sample the pixels of the current frame, generate and store the RGB information and geometric information of the pixels through the neural radiation field, and then directly extract the RGB information and geometric information of all the stored pixels as local features for clustering, otherwise do not store; then determine whether the current frame is a common view frame, if so, record the index number of the current common view frame, otherwise do not record; Step S4: performing loop detection according to the clustering results; Step S5: After the loop is successfully detected, the camera posture is adjusted using the second-order optimization algorithm of the graph; Step S6: perform mapping and global bundle adjustment based on the existing camera pose information, and then perform 3D modeling.

2. The robot loop detection method based on multi-task low-dimensional latent code and neural radiation field according to claim 1 is characterized in that: The conditions for determining the location frame in step S3 are specifically: Determine whether the current frame is a location frame based on the point cloud overlap between each current frame and all location frames. If the point cloud overlap is less than 0.2, it is a location frame. The conditions for determining the common viewing frame are specifically: The point cloud overlap between each current frame and the previous common view frame is used to determine whether the current frame is a common view frame. If the point cloud overlap is less than 0.15, it is a common view frame.

3. The robot loop detection method based on multi-task low-dimensional latent code and neural radiation field according to claim 1 is characterized in that: Step S4 is specifically as follows: Determine whether the current frame and all co-viewing frames form a loop. The determination method is: Generate VLAD descriptors based on existing categories and use ball trees to match the closest frame list; The frame list is traversed in a loop, and each frame is judged to meet the loop constraint conditions. If the following two conditions are met: ① the overlap between the point cloud of the similar frames in the frame list and the current frame is greater than 0.75; ② the difference between the index number of the similar frames in the frame list and the index number of the current frame is greater than 30, then a loop is formed, and then the frame with the largest overlap with the point cloud of the current frame is selected from all the frames that constitute the loop as the index frame, and the final loop is formed with the current frame; if not met, enter S6.

4. The robot loop detection method based on multi-task low-dimensional latent code and neural radiation field according to claim 1 is characterized in that: Step S5 is specifically as follows: S401: After successfully finding the final loop, use matrix multiplication to obtain the pose transformation of adjacent common view frames using the camera pose tracked in step S2 as part of the edge of the graph, and use the estimated poses of all common view frames between the index frame and the current frame as the vertices of the graph; S402: extracting the point cloud information of the index frame and the current frame, and using the ICP algorithm to obtain the pose transformation between the index frame and the current frame as another part of the edge of the graph; S403: Adjust the Levenberg-Marquardt optimizer parameters to optimize the graph.

5. The robot loop detection method based on multi-task low-dimensional latent code and neural radiation field according to claim 4 is characterized in that: The Levenberg-Marquardt optimizer parameters include a linear optimizer and a trust region strategy.

6. A loop detection device for implementing the method according to any one of claims 1 to 5, characterized in that: include: An image acquisition module, used to acquire images collected by the robot; The frame extraction module is used to judge and record the key frames, location frames, and common view frames of the image, and cluster the location frames; The loop detection module is used to perform loop detection based on the clustering results of the location frame.

7. A robot, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robot loop detection method described in any one of claims 1-5.

8. A machine-readable storage medium storing machine-executable instructions, wherein when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the robot loop detection method described in any one of claims 1-5.

Images