Visual slam loop closure detection method based on domestic embedded platform

Abstract

The application explores the application of deep learning in the field of visual loop detection, focuses on deploying a lightweight deep learning feature extraction algorithm on an RK3588 platform, a high-performance flagship SoC chip, and integrating it into the loop detection algorithm of VINS-MONO. The application realizes a visual positioning and mapping (SLAM) loop detection algorithm based on deep learning feature extraction on RK3588, and on this basis, deploys the loop detection algorithm based on deep learning feature extraction on the RK3588 platform. This work has important significance for deploying efficient and accurate loop detection algorithms on embedded platforms, and also has certain significance for supporting the development of domestic chips. After testing, the method proposed in the application is superior to the classical VINS-MONO algorithm, and can accurately detect loops in environments with large angle differences and strong light changes, and the inference speed of the deep learning-based feature detection algorithm on RK3588 can reach 10fps.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically to the deployment and implementation of a learning-based loop closure detection method on a domestically produced embedded platform. Background Technology

[0002] Loop closure detection algorithms are a crucial component of Simultaneous Localization and Mapping (SLAM). SLAM refers to the process by which mobile devices, using sensors such as cameras or radar, acquire their own position and map their environment in unknown conditions. Visual SLAM has a wide range of applications, including drones, automated ground cleaning / lawn mowing (small ground-based mobile robots), pedestrian navigation / AR games (head-mounted devices), automated inspection, and food delivery. A visual SLAM system typically comprises three main modules: visual odometry, backend optimization, and loop closure detection. Because visual sensors are significantly affected by changes in lighting conditions, researchers have fused inertial sensors with data sensors to achieve a complementary advantage between the two, creating a visual-inertial SLAM system. Accordingly, in the visual odometry module, IMU information is incorporated as a visual inertial odometry system to estimate the device's camera pose and other information in real time. In the backend optimization section, all trajectory information is used to globally optimize the visual inertial odometry, improving positioning accuracy. Due to environmental factors and random sensor errors, the positioning results of the visual inertial SLAM algorithm will contain errors. During operation, these errors accumulate and negatively impact positioning and mapping over time. Therefore, a loop closure detection module is introduced into the SLAM system. By detecting loops in the device's trajectory, the predicted trajectory is corrected, eliminating accumulated errors in stages, thereby improving the positioning and mapping performance in visual inertial SLAM applications.

[0003] Current loop closure detection methods for visual-inertial SLAM rely on appearance similarity to determine loop closures. Early methods extracted features from images using various manually created features to achieve loop closure detection. Bag-of-words (BOB)-based loop closure detection methods treat local features as "vocabularies," constructing a bag-of-words structure by clustering descriptors obtained from manually extracted features, vectorizing the image, and calculating the similarity between image vectors to determine loop closures. Common traditional feature extraction methods include SIFT. [1] (Scale-Invariant Feature Transform), SURF [2] (Speeded Up Robust Features) and ORB [3](Oriented FAST and Rotated BRIEF). SIFT detects feature points at different scales, exhibiting good rotation and scale invariance, but it is computationally complex and lacks real-time performance. SURF is an improvement on SIFT, using integral images to accelerate feature detection and descriptor computation, while employing Hessian matrices to detect keypoints. ORB combines FAST and BRIEF descriptors and introduces directional information, resulting in fast computation and suitability for real-time applications. Traditional hand-designed feature extraction methods often fail to extract features in environments with strong lighting changes or low texture. With the development of deep learning, learning-based feature extraction methods have been proposed. Deep learning methods can be trained on large datasets, enabling the network to autonomously extract image features and demonstrating greater robustness in dealing with changes in viewpoint, lighting, and scale. With the development of deep learning, learning-based methods have gradually been applied and practiced in loop closure detection. In 2018, SuperPoint... [4] The proposed network model achieves self-supervised interest point detection and description. By using deep learning methods, it realizes end-to-end interest point detection and descriptor generation, avoiding to some extent the tedious process of manually designing features. YUE et al. were the first to apply SuperPoint to a loop closure detection algorithm. [5] By utilizing the SuperPoint network to extract keypoints and descriptors, and training the visual vocabulary incrementally, a topological graph-based verification method is proposed to determine candidate loop closures, taking into full account the relationships between consecutive training images. [6] Other researchers also used the SuperPoint model to obtain feature points and descriptors, and proposed the LRO (Local Relative Orientation) matching algorithm to calculate the correspondence between image pairs. This technique shows significant robustness in scenarios with factors such as viewpoint changes and dynamic objects.

[0004] This invention uses the SuperPoint network, based on VINS-MONO. [7] The algorithm is improved by using a SuperPoint network deployed on RK3588 to extract descriptors of tracked feature points in keyframes from the visual inertial odometry, as well as additional feature point descriptors, during the loop closure detection stage. This is based on SP-LOOP. [8]The additional extracted feature descriptors are binarized. A SuperPoint-based vocabulary, combining SuperPoint and DBoW3, is used to represent keyframes as vectors. Vector similarity between two frames is calculated in the keyframe database to identify loop closure frames. Geometric verification is then performed. Once verification is successful, the relative pose information between the two frames is calculated, and pose optimization is performed. This invention deploys the complete loop closure detection algorithm on RK3588 and tests it on a dataset, demonstrating that the improved algorithm outperforms the classic VINS-MONO algorithm. Summary of the Invention

[0005] This invention aims to deploy a learning-based loop closure detection algorithm on the domestically produced heterogeneous CPU-NPU chip RK3588, based on existing technologies. The overall algorithm framework adopts the loop closure detection part of VINS-MONO. The research content of this invention mainly focuses on the deployment of the learning-based feature extraction network SuperPoint on RK3588, and the implementation of the loop closure detection algorithm.

[0006] This invention is a learning-based loop closure detection algorithm based on a domestically produced heterogeneous CPU-NPU chip. Utilizing the multi-core NPU capabilities of the RK3588, the network model is converted and deployed on the RK3588, with parallel inference using three NPU cores to reduce processing time. The key feature of this method is the use of a learned feature extraction algorithm to extract image features, employing a vocabulary trained using SuperPoint. The image is vectorized, and in the loop closure detection stage, feature descriptors are extracted using the SuperPoint model for loop closure identification. Finally, pose optimization is performed. The steps include:

[0007] 1) Deploy the SuperPoint network model on RK3588 and perform testing and inference time analysis.

[0008] 2) Extract image feature points and feature descriptors for the current frame using the SuperPoint model, and use the vocabulary obtained from offline training of SuperPoint to represent the image feature descriptors as vectors.

[0009] 3) Perform loop closure detection on the current frame.

[0010] 4) In the loop closure detection stage, candidate loop closure frames are filtered based on similarity. Finally, the loop closure frames are matched with the current frame to find the correspondence between the feature points of the current frame and the loop closure frames, and the points that fail to match are removed.

[0011] 5) Finally, the relative pose between the keyframe and the loopback frame is calculated using the PnP method, and this information is stored. In the subsequent pose graph optimization process, this information is added to the graph as an additional constraint. The more reliable the constraint, the more accurate the estimation result will be. Attached Figure Description

[0012] Figure 1 This is a hardware resource diagram of the EVM3588-B embedded motherboard.

[0013] Figure 2 This is a schematic diagram of the SuperPoint network structure.

[0014] Figure 3 This is the main block diagram of RKNN-Toolkit2.

[0015] Figure 4 This is an analysis of the time consumption and memory usage of the SuperPoint network model during inference on the RK3588 interconnect board.

[0016] Figure 5 The results are obtained by extracting and matching feature points on the test image using the SuperPoint network model on the RK3588.

[0017] Figure 6 This is a flowchart of the loop closure detection algorithm of the present invention.

[0018] Figure 7 This is an example of a loop closure detected by the loop closure detection algorithm of this invention during actual operation. Detailed Implementation

[0019] This invention proposes a deep learning-based loop closure detection method based on a domestically developed embedded platform. The invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] This invention proposes a loop closure detection algorithm based on a domestically developed embedded platform, deployed on the EVM3588-B development board. This development board is an embedded artificial intelligence motherboard with the RK3588 eight-core 64-bit processor as its core. Its hardware resource diagram is shown below. Figure 1 As shown. This development board has a maximum clock speed of 2.4GHz, integrates a quad-core Mali-G610 GPU, and has a built-in AI accelerator NPU with a computing power of 6.0TOPs, featuring low power consumption and high performance.

[0021] This invention employs the SuperPoint model as the feature extraction network. The SuperPoint network model is a feature extraction network trained using an unsupervised fully convolutional neural network architecture, as shown in the network structure diagram below. Figure 2 As shown. The output includes feature points and feature descriptors. Based on the SP-Loop literature... [8]The SuperPoint descriptor is binarized, and DBoW3 is used to cluster the large-scale image training dataset to construct a vocabulary based on the SuperPoint descriptor. In this invention, after extracting the SuperPoint descriptor for keyframes, the original descriptor is written to a binary file to create a binarized descriptor. This binarized descriptor is then used to construct vectors for subsequent loop closure frame queries and database construction. The original SuperPoint descriptor is a 256-dimensional high-dimensional vector. Since loop closure detection requires extracting at least 500 additional feature points, the algorithm's memory usage increases over time. Therefore, a binary file is used to write the original descriptor, and it is only read from the file during matching. Binary files save space when storing numeric data, and for more precise data, binary storage avoids the loss of significant bits, while also being very fast for reading and writing.

[0022] This invention uses the RKNN-Toolkit2 tool to transform the network model. RKNN-Toolkit2 is a development kit provided by Rockchip Microelectronics for researchers to perform network model transformation, inference, and performance evaluation. Its main block diagram is shown below. Figure 3 As shown. The converted model is deployed to the Rockchip development board using the RKNPU2 software stack provided by Rockchip's official RKNN SDK. RKNPU2 provides RKNN C API and Python API to allow users to deploy the model on the development board. The deployment steps of the SuperPoint network model in this invention on the RK3588 development board are as follows:

[0023] 1) Download and install the RKNN-Toolkit2 environment on your computer. The current version of RKNN-Toolkit2 is 2.3. Use the conda tool to create a virtual Python environment for toolkit2. In this environment, you can import the RKNN class for use during subsequent model conversions.

[0024] 2) Load the prepared SuperPoint.pth model using PyTorch and import it into the ONNX library. First, convert the model to ONNX format, fixing the network input size. The network model's input tensor is [1, 1, 480, 640]. Write a Python script to convert the converted SuperPoint ONNX model to a SuperPoint RKNN model using the RKNN Python API. This step should be performed in the toolkit2 virtual environment created in step one. Add a board-connected inference module to the conversion script file. Before execution, connect the development board to the computer using a cable, then execute the script to obtain the model's memory consumption and inference time during board-connected inference. The analysis results are as follows: Figure 4 As shown.

[0025] 3) Install the RKNPU2 environment on the board system. The development board system used in this invention is a Linux system, specifically Ubuntu 20.04. RKNPU provides CAPI for processing network models.

[0026] 4) The SuperPoint network output is processed. First, feature point information is filtered using a threshold, then the feature points are sorted. A grid sampling method is used to obtain the descriptors corresponding to the filtered feature points from the descriptor information. Two images are used for testing. The extracted feature information is matched using OpenCV's BFMatcher. The matching results are as follows: Figure 5 As shown, the total inference time for SuperPoint on the RK3588 is approximately 80ms. The above steps outline the specific procedures for deploying and testing the SuperPoint network on the RK3588.

[0027] This invention deploys SuperPoint on RK3588 and improves the loop closure detection part based on VINS-MONO: SuperPoint replaces the traditional feature point detection method, and SuperPoint is combined with DBoW using DBoW version 3, which supports both binary and floating-point descriptors. Since the algorithm runs on an embedded platform with limited memory resources, the high-dimensional SuperPoint descriptor needs to be processed into a binary descriptor. Using a binary descriptor results in faster retrieval speed and lower memory usage compared to a floating-point descriptor. The SuperPoint descriptor is a 256-dimensional floating-point descriptor, based on SP-LOOp... [8]The proposed binarization method performs binarization before loop closure frame retrieval. To ensure the accuracy of feature matching, the original SuperPoint descriptor is used when matching the loop closure frame with the current frame. To reduce memory consumption, the extracted SuperPoint descriptor is written to a binary file. The flowchart of the loop closure detection algorithm is shown below. Figure 6 As shown. The steps of the loop closure detection algorithm are as follows:

[0028] 1) Selected tracking features for keyframes can be obtained from the front-end visual odometry. Then, the SuperPoint network is used to extract the corresponding feature descriptors, and additional feature descriptors also need to be extracted. These additional feature descriptors can increase the feature information of the keyframes, which is helpful for loopback frame retrieval and can improve position recognition performance.

[0029] 2) For keyframes, the SuperPoint model deployed on RK3588 is used to extract descriptors for selected tracking feature points and additional feature descriptors. The extracted descriptors are written to a binary file and stored in the development board's storage space. The additional descriptors are binarized for loopback frame retrieval.

[0030] 3) Loopback Frame Retrieval. For keyframes, the descriptors extracted using SuperPoint (after binarization of additional extracted descriptors) are represented as visual vocabulary vectors using a SuperPoint-based vocabulary, and then retrieved from the keyframe database. The keyframe database consists of all processed keyframe vectors. For keyframe K... t During processing, its additional binary descriptors are converted into a visual vocabulary vector V using a vocabulary list. t The database is searched for vectors similar to it. The similarity threshold is set to 0.015. A set of candidate loop-closing frames with high similarity is obtained. If no candidate loop-closing frame option is found that meets the requirements, then the keyframe K is... t Add it to the database. Otherwise, select the earliest appearing candidate loopback frame from the candidate loopback frame options. This is because the estimation of earlier keyframes is more reliable, and the loopback information is more accurate for subsequent optimization. Set the keyframe index threshold to 50, meaning that initially, the first 50 frames are not searched for loopbacks, but are directly processed and added to the keyframe database, and are not searched within the first 50 frames of the current frame.

[0031] 4) Feature matching. The BFMatcher function in the OpenCV library is used to match the tracking features of the keyframes with the additional features of the candidate loop closure frames. To ensure the accuracy of the matching, the matching order of the two frames is swapped and the matching is performed again. Feature points that are consistent in both matches are retained.

[0032] 5) Eliminate outliers based on geometric relationships. If the number of matches in step four reaches a certain condition, calculate the current keyframe K. t and candidate loopback frame K c The relative pose transformation between the points is performed. RANSAC is used for F-matrix testing to filter 2D-2D points. Then, using the known 3D positions of feature points and the 2D points of candidate loop closure frames, PnP is performed. After two steps of filtering and removing feature points, if the number of remaining feature point pairs exceeds a certain threshold, the candidate loop closure frame can be considered the correct loop closure detection and subsequent relative pose transformation can be performed.

[0033] 6) Pose optimization. The current keyframe is placed in the optimization queue to obtain the pose information of the front-end visual inertial odometry, and the actual pose is calculated based on the offset, and then the pose is updated.

[0034] The above outlines the basic steps of the algorithm proposed in this invention. Furthermore, the algorithm implemented in this invention was tested using datasets such as Euroc. The absolute trajectory error was calculated using the evo tool to compare the obtained pose results with the ground truth, and the RMSE of the absolute trajectory error was used as the comparison metric. The results are shown in the table below, in meters. The numbers in parentheses indicate the number of detected loop closures. Since the front-end uses VINS-MONO visual inertial odometry, only the results with loop closures are compared here. It can be seen that using the SuperPoint network model significantly increases the number of detected loop closures and also improves the accuracy of the pose. The SuperPoint network is characterized by its ability to detect loop closures under conditions of large viewpoint or illumination changes. Figure 7 As shown.

[0035] Table 1

[0036] Euroc vins-mono-sp(loop) vins-mono(loop) V1_01_easy 0.044327(162) 0.049838(55) v1_02_medium 0.043842(67) 0.063909(15) v1_03_difficult 0.085462(27) 0.172625(2) v2_01_easy 0.053845(66) 0.062638(13) v2_02_medium 0.079046(74) 0.107884(27) MH_01_easy 0.080635(168) 0.077483(78) MH_02_easy 0.061758(111) 0.071327(35) MH_03_medium 0.059131(203) 0.078985(89) MH_04_difficult 0.111628(91) 0.136050(48) MH_05_difficult 0.118552(114) 0.145313(42)

[0037] References

[0038] [1]LOWE D G.Object recognition from local scale-invariant features[C] / / Proceedings of the seventh IEEE international conference on computervision.Kerkyra:IEEE,1999:1150-1157.

[0039] [2]BAY H,TUYTELAARS T,GOOL L V.SURF:speeded up robust features[C] / / European conference on computer vision.Berlin:Springer,2006:404-417.

[0040] [3]RUBLEE E,RABAUD V,KONOLIGE K,et al.ORB:an efficient alternative toSIFT or SURF[C] / / 2011 international conference on computer vision.Barcelona:IEEE,2011:2564-2571.

[0041] [4]D.DeTone,T.Malisiewicz,and A.Rabinovich,SuperPoint:Selfsupervisedinterest point detection and description,Proc.IEEE Conf.Comput.Vis.PatternRecog.,2018:224-236.

[0042] [5]YUE H,MIAO J,YU Y,et al.Robust loop closure detection based on bagof SuperPoints and graph verification[C] / / 2019 IEEE / RSJ internationalconference on intelligent robots and systems(IROS).Macau:IEEE,2019:3787-3793.

[0043] [6]MA J,YE X,ZHOU H,et al.Loop-closure detection using local relativeorientation matching[J].IEEE Transactions on Intelligent TransportationSystems,2022,23(7):7896-7909.

[0044] [7]Qin,T.;Li,P.;Shen,S.Vins-mono:A robust and versatile monocularvisual-inertial state estimator.IEEE Trans.Robot.2018,34,1004-1020.

[0045] [8]J.Ma,X.Ye,H.Zhou,X.Mei and F.Fan,"Loop-Closure Detection UsingLocal Relative Orientation Matching,"in IEEE Transactions on IntelligentTransportation Systems,vol.23,no.7,pp.7896-7909,July 2022,doi:10.1109 / TITS.2021.3074520.

Claims

1. A visual SLAM loop closure detection method based on a domestically developed embedded platform, characterized by deploying a learning-based feature extraction model on the domestically developed embedded platform and then integrating it into a traditional bag-of-words model-based loop closure detection method. The main steps of this method are as follows: 1) Deploy the learning-based feature extraction model on the RK3588. Select the SuperPoint model. The original model format is PyTorch, so first convert the PyTorch format to ONNX format. Due to the limitations of the RK3588 NPU platform, the input size of the model needs to be manually set during conversion to fix the input dimensions. Set the input image size to 480×640. The post-processing part requires processing the output of the SuperPoint network, including non-maximum suppression, thresholding, and normalizing feature descriptors. 2) Download and configure the Rknn-toolkit2 tool provided by Rockchip for RKNPU. Configure the environment on the PC to convert ONNX to the RKNN format that can run on RKNPU. After conversion, perform board-to-board inference and check the inference time and other information. 3) Loopback Frame Retrieval: Keyframes are extracted using the SuperPoint model to extract feature points and corresponding descriptors, which are recorded in a binary file. The SuperPoint descriptors are binarized, and the binarized descriptors are used to retrieve loopback frames from the database using the bag-of-words model. 4) Loop Closure Detection: Feature matching is performed on keyframes and candidate loop closure frames. This involves reading the previously saved binary descriptor file to obtain the descriptor information for both frames. After feature matching, geometric constraints are applied, and RANSAC and fundamental matrix calculations are used to verify whether the two frames constitute a loop closure. 5) After determining the presence of loops in step four, perform pose optimization.

2. In claim 1, the loop closure algorithm uses the back-end pose optimization part of the VINS-MONO algorithm, and upgrades the DOBW library of the bag-of-words model from version 2 to version 3 to support the SuperPoint descriptor, which also improves the retrieval speed.

Images