A loop-back detection method based on pulse neural network

Abstract

The application provides a loop detection method based on a pulse neural network, which can be applied to the fields of robot autonomous positioning and navigation, augmented reality and the like. In view of the problem of loop frame detection in SLAM visual positioning, the application trains and uses an SNN model as a loop detector to reduce cumulative errors. The application uses the SNN model to extract features of an image, searches for the corresponding feature vectors of historical frames after the feature vectors are obtained, calculates the Euclidean distance from the current frame, finds the image frames that may constitute a loop, constructs a non-linear optimization error term, and then eliminates the cumulative error. The application integrates a fast and lightweight neural network into a SLAM system, and takes into account the efficiency and accuracy. After comparison and analysis with a traditional method, the application has better positioning effect in some scenes.

Description

Technical Field

[0001] This invention belongs to the fields of robot autonomous localization and navigation, augmented reality, and specifically relates to a loop closure detection method based on a spiking neural network. Background Technology

[0002] Spiking Neural Networks (SNNs) are a type of neural network model that differs from traditional approaches, simulating the biological nervous system. The spiking neuron is the core component of a spiking neural network. By mimicking the behavior of biological neurons, using the spiking neuron as the basic unit, it processes time-series information with good biological plausibility and energy efficiency. Because information transmission in spiking neural networks occurs in the form of pulses, it can utilize energy more efficiently compared to traditional continuous signal transmission methods. This makes spiking neural networks promising for applications in low-power devices and large-scale neural networks.

[0003] SLAM (Simultaneous Localization and Mapping) is a technology used to simultaneously achieve autonomous localization and environmental mapping. It has wide applications in unmanned systems, robotics, and augmented reality. SLAM technology achieves autonomous localization in unknown environments and simultaneously constructs a 3D map of the environment by fusing sensor data and motion models. SLAM technology typically includes the following main modules: data acquisition, feature extraction, data association, state estimation, map updating, and loop closure detection. Among these, the loop closure detection module plays a crucial role. The main goal of the loop closure detection module is to identify and process instances where the system traverses the same area during movement, i.e., detecting locations that have already been explored. Without loop closure detection, localization errors may accumulate, affecting the accuracy and consistency of the map. The core task of loop closure detection is to find matching locations in data from different time points and connect them to form a closed trajectory loop, thereby reducing accumulated errors. By detecting and processing loops, the system can eliminate the cumulative effect of localization errors and improve localization accuracy.

[0004] Previous loop closure detection schemes typically used the Bag-of-Words (DBoW) model for feature extraction and search, but these methods have several shortcomings. As time accumulates, the number of images captured by the camera in a scene increases significantly, while the limited number of words in the Bag-of-Words model cannot accommodate a sufficiently large environment, resulting in suboptimal performance. Furthermore, the dictionary is space-consuming, requiring a large dictionary to be loaded into memory before the SLAM system can run. The process of searching for loop closure frames is slow and computationally expensive, making real-time operation impossible and incurring significant time and storage costs. Additionally, the Bag-of-Words model has poor robustness to repetitive textures, making it ineffective at detecting loop closures.

[0005] In short, traditional bag-of-words (BOB) model-based loop closure detection schemes suffer from low efficiency, high memory and runtime costs, and insufficient adaptability. Therefore, this invention uses a subnet-based neural network (SNN) to replace the traditional BOB model, achieving loop closure detection with higher efficiency and thus improving the localization performance of SLAM. Summary of the Invention

[0006] To address the problems in existing technologies, this invention proposes a loop closure detection method based on spiking neural networks (SNNs). Regarding the high computational cost of existing algorithms, SNNs offer better biocompatibility and energy efficiency. Because information transmission in SNNs is in the form of pulses, they can utilize energy more efficiently compared to traditional continuous signal transmission methods. This makes SNNs promising for applications in low-power devices and large-scale neural networks. Regarding the limitation of bag-of-words models in terms of word count and their inability to accommodate sufficiently large environments, neural networks learn and store the necessary information for environmental feature extraction into the model's parameters, thus overcoming limitations imposed by scene size. SNNs can be deployed on lightweight devices with low computational requirements, making them suitable for use in SLAM algorithms.

[0007] The technical solution of the present invention is as follows:

[0008] This invention first provides a loop closure detection method based on a spiking neural network. The method uses an SNN model as the loop closure detection module to find frames with loops in the historical image frames and the current frame. During the movement of the device to be located, after the loop closure detection module finds the loop, the loop closure correction module performs feature matching between the loop frame and the current frame to construct a reprojection error. The cumulative error in the pose estimation module is reduced by nonlinear optimization.

[0009] The method includes the following steps:

[0010] S1: Define two ordered lists in the spiking neural network, the first list... The second list stores the index IDs of the image frames. Storing the feature vector of an image frame ;

[0011] S2: Obtain the current frame image and historical frame images, and store the image IDs in a list. In the process, construct an image ID database. ; Utilizing a neural network based on SNN to extract features from images, transforming image data into feature vectors of a specified dimension. And save it to a list. In the process, an image feature vector database is constructed. ;

[0012] S3: Search within a defined range in the image feature vector database and calculate the feature vectors therein. With the feature vector of the current frame European distance Select Euclidean distance Smaller historical frames are selected, and one frame is chosen from these historical frames as the detected loopback frame and added to the image ID database. The corresponding image ID is found and returned to the loop closure correction module; the loop closure correction module performs loop closure correction accordingly, thereby correcting the accumulated error in the pose estimation module.

[0013] As a preferred embodiment of the present invention, the feature extraction of the image using an SNN-based neural network described in S2 specifically involves: encoding the image using a Vgg16 autoencoder; the input of the Vgg16 model is a single-frame RGB image, and the output is a latent vector of a set dimension; thus, the latent vector is obtained. Then, it is input into the NetVLAD implementation based on SNN to check the latent vector. Decoding is performed; the input to NetVLAD, which is based on SNN, is the latent vector obtained from image encoding, and the output is a feature vector of a set dimension. .

[0014] As a preferred embodiment of the present invention, the search within a predetermined range in the image feature vector database described in S3 specifically includes:

[0015] Considering that image frames with a time interval close to the current frame have image content that is close to the current frame, a frame number threshold is set, and data from the image feature vector database that is more than a set frame number threshold away from the current frame is taken as the search range.

[0016] As a preferred embodiment of the present invention, the selection of Euclidean distance described in S3 Smaller historical frames, specifically: setting a Euclidean distance threshold. When Euclidean distance Less than the threshold At that time, it was assumed that their images contained similar content, forming a loop; if the Euclidean distance was too large, it indicated that the differences between the images were large, and the possibility of forming a loop was low.

[0017] As a preferred embodiment of the present invention, the step of selecting one frame from these historical frames as the detected loopback frame in S3 specifically means: selecting the image with the smallest timestamp as the loopback frame.

[0018] As a preferred embodiment of the present invention, the pose estimation module receives image data acquired by the camera of the device to be located and solves the camera pose; the loop closure correction module receives image frames sent by the pose estimation module, creates and maintains the ID of each frame of image data, and sends a request to the loop closure detection module to perform loop closure detection; after receiving the result from the loop closure detection module, the loop closure correction module performs feature matching between the current frame and the loop closure frame, constructs the reprojection error, and performs nonlinear optimization to reduce the cumulative error of the pose estimation module.

[0019] Compared with traditional loop closure detection methods based on the bag-of-words model, the advantages of this invention are:

[0020] 1) SNN is better at finding loop closures, which improves the positioning accuracy of the trajectory after loop closure detection.

[0021] 2) It is not limited by scene size and can still detect loop closures in a wide range of scenes, reducing cumulative errors. Attached Figure Description

[0022] Figure 1 This is an overall framework diagram of the present invention;

[0023] Figure 2 This invention searches for loop closures on the EuRoC dataset and compares the results with the DBoW scheme;

[0024] Figure 3 This invention finds loops on the Kitti dataset and compares its performance with that of the DBoW scheme. Detailed Implementation

[0025] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0026] This invention is a loop closure detection method based on a spiking neural network, which needs to be deployed within an existing pose estimation algorithm. Depending on the odometry method, the system can input image data and IMU data, or monocular, binocular, or RGBD image data, and output the robot's motion trajectory and a map of the environment.

[0027] like Figure 1 As shown, this invention is implemented within the ROS framework and consists of three modules: a pose estimation module, a loop closure correction module, and a loop closure detection module, abbreviated as Est module, Loop module, and SNN module, respectively. Image data communicates with the Est and Loop modules via a publish-subscribe model. After image data is published to a topic, it is subscribed to by the Est and Loop modules. Upon receiving the image, the Est module runs an odometry algorithm to solve for the camera pose and map construction. The Loop and SNN modules communicate via a client-server model. After subscribing to image data, the Loop module sends a request to the SNN module, sending the image and its ID. After finding the loop closure, the SNN module sends the ID of the loop closure frame back to the Loop module. The Loop module then constructs a reprojection error based on the loop closure result and performs nonlinear optimization to reduce the accumulated error of the Est module.

[0028] The first module is the pose estimation module, which runs the pose estimation algorithm. This module references VINS-Fusion, receives image data and other sensor data, extracts feature points from the image, calculates their BRIEF descriptors, and correlates multiple consecutive frames of images by tracing the optical flow of these feature points. Between correlated frames, the feature points are reprojected based on the camera pose, and a reprojection error is constructed using the optically traced feature points. The camera pose is then determined by nonlinearly optimizing the reprojection error until it is minimized.

[0029] The second module is the loop closure correction module, which runs the loop closure correction algorithm. This module, also referencing VINS-Fusion, receives image frames sent by the EST module, creates and maintains the ID for each frame of image data, and sends a request to the third module for loop closure detection. Upon receiving the result from the third module, it performs feature matching between the current frame and the loop closure frame based on the BRIFE descriptor calculated in the EST module, thus establishing a connection between the two frames. Then, similar to the EST module, it reprojects the feature points based on the relative poses of the current frame and the loop closure frame, constructing a reprojection error with the matched feature points. By performing nonlinear optimization on the reprojection error, the relative poses of the current frame and the loop closure frame are obtained, thereby correcting the accumulated error over this period.

[0030] The third module is the loop closure detection module, which runs the SNN-based loop closure detection algorithm (hereinafter referred to as the SNN module). It is responsible for receiving requests from the Loop module, performing loop closure detection, and returning the ID of the loop closure frame.

[0031] This invention mainly includes the following steps:

[0032] This invention provides a loop closure detection method based on a spiking neural network. The method employs an SNN-based model as the loop closure detection module to find frames in historical image frames that have a loop with the current frame. During the movement of the device to be located, after the loop closure detection module finds a loop, a loop closure correction module performs feature matching between the loop closure frame and the current frame to construct a reprojection error. This reprojection error is then reduced in the pose estimation module through nonlinear optimization. The method includes the following steps:

[0033] Step 1: Define two ordered lists in the spiking neural network, the first list The second list stores the index IDs of the image frames. Storing the feature vector of an image frame ;

[0034] Step 2: Obtain the current frame image and historical frame images, and store the image IDs in a list. In the process, construct an image ID database. ; Utilizing a neural network based on SNN to extract features from images, transforming image data into feature vectors of a specified dimension. And save it to a list. In the process, an image feature vector database is constructed. ;

[0035] Step 3: Search within a defined range in the image feature vector database and calculate the feature vectors therein. With the feature vector of the current frame European distance Select Euclidean distance Smaller historical frames are selected, and one frame is chosen from these historical frames as the detected loopback frame and added to the image ID database. The corresponding image ID is found and returned to the loop closure correction module; the loop closure correction module performs loop closure correction accordingly, thereby correcting the accumulated error in the pose estimation module.

[0036] As a preferred embodiment of the present invention, the feature extraction of the image using an SNN-based neural network in step 2 specifically involves: firstly, encoding the image using a general-purpose Vgg16 autoencoder. The input to the Vgg16 model is a single-frame RGB image, and the output is a latent vector of a certain dimension. The latent vector is then obtained. Then, it is input into NetVLAD (i.e., SNN-NetVLAD, where the NetVLAD structure is described in the following literature: NetVLAD: CNN architecture forweakly supervised place recognition, CVPR 2016) to process the latent vectors. Decoding is then performed. The input to SNN-NetVLAD is the latent vector obtained from image encoding, and the output is a feature vector of a certain dimension. The model was trained on the Pittsburgh dataset. During training, the Vgg16 autoencoder used publicly available pre-trained models, training only SNN-NetVLAD, thus achieving good results while saving training time.

[0037] As a preferred embodiment of the present invention, the search within a certain range of the image feature vector database in step 3 specifically involves: only detecting image frames from a certain number of frames ago; the feature vectors of the images within the nearest certain number of frames will not be used for loop retrieval. Image frames whose time interval is closer to the current frame have image content that is closer to the current frame; after feature extraction, their Euclidean distance to the feature vector of the current frame will be closer, making them more likely to outperform other candidate frames and be selected as loop retrieval frames. However, since the EST module already constructs nonlinear optimization constraints with the most recent frames, selecting them as loop retrieval frames would be redundant.

[0038] As a preferred embodiment of the present invention, the selection of Euclidean distance in step 3 The smaller historical frames, specifically: This invention uses the Faiss tool to retrieve a certain feature vector from the database. The Faiss tool will output a certain number of candidate frames. And the Euclidean distance between the feature vectors of these image frames and the feature vector of the current frame. .when When the distance is less than a certain threshold, the algorithm considers the images to have similar content, forming a loop. If the distance is too large, it indicates that the images are significantly different, and the likelihood of a loop is low. Therefore, we set a threshold. Iterate through all candidate loopback frames. If its distance Greater than If the value is less than 1, it is considered not to constitute a loop; if it is greater than 1, it is retained.

[0039] As a preferred embodiment of the present invention, step 3, which involves selecting one frame from these historical frames as the detected loopback frame, specifically means selecting the image with the smallest timestamp as the loopback frame. The robot may pass through the same location multiple times, and the longer the algorithm runs, the greater the accumulated error. The image with the smallest timestamp indicates that the robot passed through this location earliest, resulting in a relatively smaller accumulated error. Selecting it as the loopback frame will effectively correct the accumulated error during this period.

[0040] Example

[0041] To demonstrate the effectiveness of this invention, it was compared with traditional bag-of-words (DBoW) methods and NetVlad methods based on traditional convolutional neural networks (CNNs) on the EuRoC and KITTI datasets. The comparison metric was the ATE metric, commonly used in SLAM algorithms, which describes the translation error between the predicted trajectory and the true trajectory.

[0042] The loop closure detection method of this invention can be applied as a loop closure detection module in a general SLAM system. To verify the effectiveness of the invented loop closure detection method based on a spiking neural network, we replaced the loop closure detection module of the well-known open-source SLAM system VINS-Fusion with an SNN neural network.

[0043] The original VINS-Fusion is implemented based on the DBoW method. The input to the loopback lookup module is a query image and its ID, and an image dictionary database and a corresponding image ID database are maintained in the background. The output of the module is the ID of the loopback frame.

[0044] The SNN scheme of this system does not change its input and output. It still takes an image and ID as input and outputs the ID of the loopback frame. The SNN module maintains an image feature vector database and a corresponding ID database.

[0045] In contrast, the CNN approach replaces the SNN-NetVlad in this system with CNN-NetVlad, while keeping all other input and output interfaces the same.

[0046] Experiment 1: Comparison of localization errors on EuRoC and Kitti datasets

[0047] As shown in Tables 1 and 2, Table 1 compares the localization accuracy of SNN, CNN, and DBoW methods on the EuRoC dataset, with the ATE (Alternative Location) metric being the standard accuracy (AAC). A smaller AAC indicates a smaller localization error and higher accuracy. Table 2 compares the results on the KITTI dataset. The EuRoC dataset primarily consists of indoor scenes, while KITTI mainly comprises outdoor urban roads, with larger scene ranges and scales. Analyzing the results in the tables, it is evident that the SNN-based method achieves better results in multiple sequences across both the EuRoC and KITTI datasets.

[0048] Table 1: Comparison of ATE metrics for various methods on the EuRoC dataset

[0049]

[0050] Table 2: Comparison of ATE metrics for various methods on the KITTI dataset

[0051]

[0052] Experiment 2: Qualitative Comparison of Loop Closure Detection Results

[0053] like Figure 2 , Figure 3 As shown, Figure 2 This shows the loop closure result found in a specific frame of the EuRoC dataset. Figure 1 The top left and top right images are from the same moment. The bottom left image shows the historical loop closure frame found by the SNN, and the bottom right image shows the loop closure frame found by the DBoW. It can be seen that the loop closure frame found by the SNN has a co-view relationship with the current frame, which can be used to construct reprojection error. However, the loop closure frame found by the DBoW has almost no co-view relationship with the current frame, and cannot be used to construct loop closure.

[0054] Figure 3 This shows the results of loop closure finding in a specific frame on the KITTI dataset. The top image is the current frame, the middle image is the loop closure frame found by SNN, and the bottom image is the loop closure frame found by DBoW. It is still evident that the frame found by SNN can establish a co-view relationship with the current frame, while DBoW cannot.

[0055] Based on the comparative experimental results of the two datasets (indoor small-scale and outdoor large-scale), it can be concluded that SNN is better than traditional DBoW in finding loop closures. In these test scenarios, SNN can improve the localization effect of SLAM algorithm and enhance localization accuracy.

[0056] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A loop closure detection method based on a spiking neural network, characterized in that, The method employs an SNN-based model as a loop closure detection module to find frames that have loop closures with the current frame in historical image frames. During the movement of the device to be located, after the loop closure detection module finds the loop closure, the loop closure correction module performs feature matching between the loop closure frame and the current frame to construct a reprojection error. The cumulative error in the pose estimation module is reduced through nonlinear optimization. The method includes the following steps: S1: Define two ordered lists in the spiking neural network, the first list... The second list stores the index IDs of the image frames. Storing the feature vector of an image frame ; S2: Obtain the current frame image and historical frame images, and store the image IDs in a list. In the process, construct an image ID database. ; Utilizing a neural network based on SNN to extract features from images, transforming image data into feature vectors of a specified dimension. And save it to a list. In the process, an image feature vector database is constructed. ; Specifically, feature extraction of images is performed using an SNN-based neural network, as follows: The image is encoded using a Vgg16 autoencoder; the input to the Vgg16 model is a single-frame RGB image, and the output is a latent vector of a specified dimension; thus, the latent vector is obtained. Then, it is input into the NetVLAD implementation based on SNN to check the latent vector. Decoding is performed; the input to NetVLAD, which is based on SNN, is the latent vector obtained from image encoding, and the output is a feature vector of a set dimension. ; S3: Search within the defined range of the image feature vector database and calculate the feature vectors therein. With the feature vector of the current frame European distance Select Euclidean distance Smaller historical frames are selected, and one frame is chosen from these historical frames as the detected loopback frame and added to the image ID database. The corresponding image ID is found and returned to the loop closure correction module; the loop closure correction module performs loop closure correction accordingly, thereby correcting the accumulated error in the pose estimation module. Specifically, the search within a defined range in the image feature vector database is as follows: Considering that image frames with a time interval close to the current frame have image content that is close to the current frame, a frame number threshold is set, and data from the image feature vector database that is more than a set frame number threshold away from the current frame is taken as the search range. Select Euclidean distance Smaller historical frames, specifically: setting a Euclidean distance threshold. When Euclidean distance Less than the threshold At that time, it was assumed that their images contained similar content, forming a loop; if the Euclidean distance was too large, it indicated that the images were significantly different, and the possibility of forming a loop was low. Select one frame from these historical frames as the detected loopback frame, specifically: select the image with the smallest timestamp as the loopback frame.

2. The loop closure detection method based on SNN according to claim 1, characterized in that, The pose estimation module receives image data acquired by the camera of the device to be located and solves the camera pose; the loop closure correction module receives image frames sent by the pose estimation module, creates and maintains the ID of each frame of image data, and sends a request to the loop closure detection module to perform loop closure detection; after receiving the result from the loop closure detection module, the loop closure correction module performs feature matching between the current frame and the loop closure frame, constructs the reprojection error, and performs nonlinear optimization to reduce the cumulative error of the pose estimation module.

Images