Binocular vision SLAM method based on deep learning feature extraction and matching algorithm

Abstract

The invention belongs to a synchronous localization and mapping (SLAM) method in the field of robots, and discloses a binocular vision SLAM method based on a deep learning feature extraction and matching algorithm. The system comprises three modules: a front-end tracking module, a local mapping module and a loopback detection module. For each frame of input binocular image, firstly, feature points of the image are extracted, and then frame-to-frame matching is used to track a current image frame and determine whether the current frame is set as a key frame. In the local mapping module, matching from a key frame to a local map is used to obtain a more accurate pose and a global map, and in addition, the loopback detection module is used for inhibiting accumulative errors of a large scene. Experiments on a public data set and a data set collected by a robot platform show that robust, accurate and globally consistent pose estimation and mapping are realized by the method, and quick real-time operation can be realized on an embedded platform at a speed exceeding 10 FPS.

Description

Technical Field

[0001] The present invention relates to a method for simultaneous localization and mapping (SLAM) in the field of robotics, and more particularly to a binocular vision SLAM method based on a deep learning feature extraction and matching algorithm. Background Art

[0002] Simultaneous localization and mapping (SLAM) is one of the core technologies in the fields of robotics, autonomous driving, and augmented reality. The SLAM system estimates its own pose in real time through sensor data and constructs an environmental map, which is the basis for realizing autonomous navigation and environmental understanding. Visual SLAM uses a camera as the main sensor and has advantages such as low cost and rich information, and has been widely studied and applied in recent years. However, traditional visual SLAM systems still face many challenges in complex scenarios such as dynamic environments, weak texture regions, and lighting changes, such as insufficient robustness of feature extraction, limited matching accuracy, and accumulation of trajectory errors.

[0003] In recent years, deep learning technology has made remarkable progress in the field of computer vision, providing new ideas for the improvement of visual SLAM systems. Feature extraction and matching algorithms based on deep learning can automatically learn robust feature representations from data, significantly improving performance in complex scenarios.

[0004] Traditional feature extraction methods rely on manually designed feature descriptors, while deep learning methods automatically learn feature representations in images through convolutional neural networks (CNNs). For example, networks such as SuperPoint and D2-Net can extract highly discriminative feature points and their descriptors, showing stronger robustness in weak texture and lighting change scenarios. In addition, learning-based feature extraction methods can also generate dense feature points, providing richer information for the SLAM system.

[0005] Feature matching is a key step in visual SLAM. Traditional methods (such as nearest neighbor matching based on descriptors) are easily affected by noise and mismatches. Deep learning methods can significantly improve the accuracy and robustness of matching by learning the geometric and semantic relationships between feature points. For example, networks such as SuperGlue and LightGlue use graph neural networks (GNNs) and attention mechanisms to achieve efficient and accurate feature matching. Summary of the Invention

[0006] (I) Technical Problems to be Solved

[0007] In order to overcome the difficulty in simultaneously ensuring the accuracy, robustness, and efficiency of visual SLAM in the prior art, and the lack of robust feature extraction and association in existing visual SLAM methods based on traditional feature points, the present invention proposes a binocular visual SLAM method based on a deep learning feature extraction and matching algorithm.

[0008] The technical solution of the present invention is as follows: A binocular visual SLAM method based on a deep learning feature extraction and matching algorithm. The binocular image frames are processed by a front-end tracking module to obtain the camera pose between adjacent frames and the matching feature point pairs between adjacent frames, by a local mapping module to obtain the locally optimized key frame pose and the locally optimized local map points, and by a loop detection module to obtain loop constraints and the camera trajectory and global map after global pose graph optimization.

[0009] Further, the front-end tracking module includes feature extraction based on a convolutional neural network, feature matching, pose estimation, and key frame selection based on a graph neural network;

[0010] Set the first binocular image frame as a key frame, and set its pose as the initial pose of the map. Extract all feature points from the first binocular image frame and create them as map points;

[0011] In each pair of binocular images, perform feature extraction on the left-eye image based on a convolutional neural network to obtain feature points and their descriptors;

[0012] Match the left-eye image feature points and descriptors with the left-eye image of the previous key frame based on a graph neural network to establish data association between frames;

[0013] Perform pose estimation based on the data association result to obtain the pose of the current frame;

[0014] Combine the pose of the current frame and perform key frame selection according to the key frame determination criterion to determine whether the current frame is a new key frame.

[0015] The specific process of performing feature extraction based on a convolutional neural network is as follows:

[0016] The convolutional neural network includes a shared VGG encoder and two independent decoders;

[0017] The binocular image frame is first input into a shared VGG encoder; for an input image I of H×W, the VGG encoder maps it to an intermediate vector C of Hc×Wc×128, where Hc = H / 8 and Wc = W / 8;

[0018] Two independent decoders are responsible for extracting the key-point pixel positions and calculating the corresponding descriptors respectively; in the decoder for extracting the key-point pixel positions, the intermediate vector C is first mapped to X ∈ Hc×Wc×65 through a convolutional layer; the 65 channels of X correspond to an 8×8 local region on the input image I and a Dustbin flag used to indicate that there is no key-point in this region; after channel Softmax and reshape, it is restored to the original image size to obtain a probability map M with a size of H×W; each pixel in the probability map M represents the probability that the corresponding pixel on the input image I is a key-point. Perform NMS on the probability map M to remove redundant points, and only retain the points with a confidence higher than the set threshold τ. These high-confidence points are the output key-point pixel positions p.

[0019] In the decoder for calculating the corresponding descriptors, the intermediate vector C is mapped to a semi-dense descriptor grid D ∈ Hc×Wc×256 through a convolutional layer, where 256 represents the dimension of the output descriptor; each grid in the descriptor grid D represents a descriptor shared by an 8×8 pixel block in the input image I. Perform bilinear interpolation on the descriptor grid D to obtain the descriptor for each pixel in the input image I; perform L2 normalization to obtain a 256-dimensional floating-point descriptor d corresponding to each key-point.

[0020] The VGG encoder is mainly composed of convolutional layers, pooling layers, and non-linear activation functions.

[0021] The specific process of feature matching based on the graph neural network is as follows:

[0022] The graph neural network is mainly composed of a graph attention mechanism layer and an optimal matching layer; considering two images A and B, each image has a set of locally extracted features and which contain the key-point pixel positions p and the corresponding descriptors d mentioned above;

[0023] In the graph attention mechanism layer, the pixel positions of the key-points and the corresponding descriptors are input into a key-point encoder to be aggregated into an initial representation x i ; the pixel positions of the key-points are embedded into a high-dimensional vector by a multi-layer perceptron MLP, which is expressed by formula (1) as:

[0024] x i = d i + MLP enc (p i ) (1)

[0025] The initial representation x is enhanced multiple times using self-attention and cross-attention i ;

[0026] Let The intermediate representation of element i of image A in the l-th layer, information m ε→i is the result aggregated from all key points {j: (i, j) ∈ ε}, where {ε ∈ {ε self , ε cross}, then the information transfer update residual representation for all elements i in image A is expressed by formula (2):

[0027]

[0028] where, [·||·] represents aggregation;

[0029] At the same time, the same update as in image A is performed on all key points in image B; it is iterated for a fixed number L of rounds with different parameters, and gradually refined through alternating aggregation of self-attention and cross-attention;

[0030] After the attention aggregation is completed, the feature representation after attention aggregation is transformed into a vector f through linear projection; the vector f enters the optimal matching layer; the matching score matrix is obtained by calculating the inner product, and the optimal assignment matrix P is solved using the sinkhorn algorithm; each element p i,j represents the probability that feature i in image A and feature j in image B are a pair of correct matches, obtaining the feature point matching relationship between the two images and outputting the matching point pairs

[0031] The specific process of the pose estimation is as follows:

[0032] Extract the feature points in the first frame image I0 and initialize the map by assuming its pose T0 as the identity matrix. At the same time, use the internal parameter matrix K of the camera and the depth information to back-project the feature points into the three-dimensional space to generate the initial map points Its calculation process is expressed by formula (3):

[0033]

[0034] According to the feature points of the second frame image I1 and the matching point pairs that match the first frame feature points use the PnP algorithm to calculate the relative pose T 1,0 = [R|t], where R is the rotation matrix and t is the translation vector; generate new map points through the triangulation method and verify whether its reprojection error is within the threshold to complete the initialization and verify the rationality of the map points;

[0035] After the initialization is successful, continuous tracking is performed; for each new image, first extract the feature points and according to the pose T of the previous framet-1 or predict the pose of the current frame using a uniform motion model Determine the local map area; select map points {X within the field of view of the current frame from the global map n}, and construct a sparse local map; the feature points of the current frame are projected and matched or descriptor-matched with the map points {X n} in the local map to obtain the matching point pairs Eliminate false matches through geometric constraints; based on the matching feature point pairs, construct a reprojection error function:

[0036]

[0037] where π is the projection function, K is the camera intrinsic matrix, R t and t t are the rotation matrix and translation vector of the current frame respectively; use the nonlinear optimization method Levenberg-Marquardt to optimize the pose of the current frame and minimize the reprojection error to improve the accuracy of pose estimation.

[0038] The process of determining the key frame is as follows:

[0039] After solving the pose of the current frame, determine whether to set it as a new key frame according to the number of feature points, matching quality, and motion amplitude of the current frame; if it is set as a key frame, add its feature points to the local map and update the local map; when the image frame meets any of the following conditions, it is recognized as a key frame:

[0040] 1) The number of extracted feature points is more than 60;

[0041] 2) The distance from the previous key frame exceeds 30 frames;

[0042] 3) The angle from the previous key frame exceeds 5 degrees;

[0043] 4) More than 25% of the feature points are not tracked.

[0044] The process of the local mapping module is as follows:

[0045] The local mapping module only processes key frames; when a new key frame K i is inserted, first perform feature matching with the adjacent key frames {K j} in the covisibility graph, establish the observation relationship of the map points {X k}, and generate new map points {X new} through triangulation, and its three-dimensional coordinates are calculated as:

[0046]

[0047] Among them and are the matched feature points in the key frame K i and the matched feature points in the key frame K j respectively, and T i and T j are the poses of the key frame K i and the pose of the key frame K j respectively;

[0048] Optimize the local map by minimizing the reprojection error through local BA:

[0049]

[0050] Among them, K is the camera intrinsic matrix, R i , t i and R j , t j are the rotation matrix and translation vector of the key frames K i and K j respectively; During the optimization process, redundant key frames and abnormal map points are removed to maintain the compactness and consistency of the map; Finally, the local mapping thread gradually constructs an accurate and dense local environment representation by continuously inserting key frames, generating map points, and optimizing the local map.

[0051] The process of the loop detection module is as follows:

[0052] The loop detection module fuses CNN deep features and DBoW3 bag-of-words model, including offline training, bag-of-words retrieval, similarity measurement, geometric verification, and loop correction;

[0053] In the offline training stage, an adaptive hierarchical K-means++ clustering algorithm is adopted to perform clustering analysis on the depth descriptors extracted by the CNN network from the public dataset; First, the 256-dimensional descriptors extracted by the CNN are compressed to 64 dimensions through PCA dimensionality reduction to improve the calculation efficiency, while retaining more than 95% of the feature information; Then, a density-based initial center selection strategy is adopted to avoid the local optimum problem that may be caused by the traditional K-means; Finally, the constructed visual vocabulary contains 10 5 visual words, forming a six-layer tree structure, providing fine-grained feature discrimination ability while ensuring the retrieval efficiency;

[0054] The objective function for constructing the visual vocabulary is:

[0055]

[0056] Among them, k is the number of visual words, μ i is the i-th clustering center, and λ is the spacing penalty coefficient;

[0057] In the bag-of-words retrieval stage, the descriptors extracted by the CNN network are quantified into sparse word frequency vectors using a pre-built visual dictionary, and the inverted index technology is adopted for fast retrieval; a dynamic key-frame database based on a time sliding window is maintained, and fast deduplication is achieved through a Bloom filter to ensure that the retrieval efficiency is not affected by the increase in running time;

[0058] In the similarity measurement stage, loop closures are detected and the consistency of the global map is optimized by identifying the similarity between the current key frame K t and the historical key frames {K h}; first, the bag-of-words descriptor D t of the current key frame K t and the bag-of-words descriptors {D h} of the historical key frames {K h} are extracted, and the similarity score is calculated by comparing the word frequency vectors of the two, and the candidate loop closure frames are screened out Then, the relative pose T t,h = [R|t] is calculated using the PnP algorithm, and it is verified whether the reprojection error satisfies the threshold:

[0059]

[0060] where K is the camera intrinsic matrix, is the map point corresponding to the historical key frame K h ;

[0061] In the geometric verification stage, a multi-stage progressive verification strategy is adopted; first, candidate pairs with an inlier rate exceeding 60% are quickly screened out by the PROSAC algorithm; then, geometric consistency constraints based on the essential matrix are constructed, and the M-estimator is used to suppress the influence of outliers; subsequently, 3D-2D reprojection verification is performed, and the map points of the candidate frame are projected onto the current frame, requiring that at least 50% of the matching point pairs have a reprojection error less than 1.5 pixels; finally, descriptor consistency verification is introduced, and bidirectional nearest neighbor matching is constructed to ensure that the descriptor distance of the feature point pairs is less than the adaptive threshold τ_d:

[0062] τ_d = μ + 2σ (10)

[0063] where μ and σ are the mean and standard deviation of the descriptor distances of the current frame;

[0064] In the loop closure correction stage, when a valid loop closure is detected, the hierarchical optimization mechanism is initiated. First, local pose graph optimization is performed, and the Dog-leg algorithm is used for rapid convergence. Then, global BA optimization is carried out, and the Schur complement technique is adopted to achieve efficient large-scale sparse matrix solution. To balance accuracy and efficiency, the optimization granularity is dynamically adjusted. When the loop closure interval is less than 100 frames, only the local window is optimized. When a large-scale loop closure is detected, full map optimization is initiated, and the poses of key frames {T i} and map points {X k} are adjusted through pose graph optimization to minimize the global error E({T}, {X}). At this time, the poses of all key frames are updated, and the running trajectory of the robot is also corrected accordingly;

[0065]

[0066] where L represents the set of loop closure constraints, and T ij is the relative pose between loop closure frames.

[0067] Advantages of the present invention: The present invention proposes a binocular vision SLAM method based on a deep learning feature extraction and matching algorithm. The method mainly includes three modules: a front-end tracking module, a local mapping module, and a loop closure detection module. Front-end tracking module: Feature points are extracted from the input binocular frame images for pose estimation, and the tracking of the current frame is achieved through data association between frames. Local mapping module: The local map is optimized by processing newly added key frames to expand the environmental representation. Loop closure detection module: Loop closures are detected by identifying the similarity between the current key frame and historical key frames, and the consistency of the global map is optimized. Experiments on public datasets and robot platforms show that the system has high accuracy and good real-time performance. Due to the importance of SLAM in unmanned driving and robotics, the present invention has broad application prospects. Description of the Drawings

[0068] Figure 1 is the flow chart of the binocular vision SLAM method based on the deep learning feature extraction and matching algorithm.

[0069] Figure 2 are the feature points and feature matching results extracted from the binocular images collected in the production workshop environment.

[0070] Figure 3 are the trajectory results of the method running on the public EuRoC dataset; (a) is the trajectory of the MH_01 sequence, (b) is the trajectory of the MH_03 sequence, (c) is the trajectory of the MH_05 sequence, and (d) is the trajectory of the V1_03 sequence.

[0071] Figure 4The trajectory results of the method running on the publicly available KITTI dataset; (a) is the trajectory of the KITTI_00 sequence, (b) is the trajectory of the KITTI_01 sequence, (c) is the trajectory of the KITTI_03 sequence, and (d) is the trajectory of the KITTI_05 sequence.

[0072] Figure 5 It is the running result of the robot platform in the real environment. Specific implementation manner

[0073] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the following further describes the present invention in detail with reference to specific embodiments and the accompanying drawings.

[0074] The present invention includes three parts: a robust front-end tracking module based on a deep learning feature extraction and matching algorithm, a local mapping module based on key frames, and a loop closure detection module based on the bag-of-words model:

[0075] The specific process of the front-end tracking module is as follows:

[0076] The front-end tracking thread performs pose estimation by extracting feature points from the input images. When the system starts, the first binocular image frame is set as the key frame, and its pose is set as the initial pose of the map. All feature points extracted from this frame are created as map points. After successful initialization, for each pair of binocular images, we use a convolutional neural network (CNN) to extract features from the left-eye image to obtain feature points and their descriptors. Then, we use a graph neural network (GNN) to match the features with the previous key frame to establish data associations between frames, thereby performing the tracking process.

[0077] For the real-time tracking of each binocular image frame, there are mainly the following two parts: First, the pose of the current frame is solved according to the initial pose estimation; after obtaining the pose estimation of the current frame, it is then determined whether the current frame needs to be set as a new key frame.

[0078] Feature extraction is implemented based on a convolutional neural network, and the process is as follows:

[0079] The stereo image frame is first input into a shared VGG encoder to reduce the image size. The shared VGG encoder consists of convolutional layers, pooling layers, and nonlinear activation functions. For an H×W input image I, the encoder maps it to an intermediate vector C of Hc×Wc×128, where Hc=H / 8 and Wc=W / 8. Two independent decoders are responsible for extracting the pixel positions of key points and calculating the corresponding descriptors. In the decoder for extracting the pixel positions of key points, the intermediate vector C is first mapped to X∈Hc×Wc×65 through a convolutional layer. The 65 channels of X correspond to an 8×8 local area on the original image plus a Dustbin marker to indicate that there are no key points in the area. After channel Softmax and reshape, X is restored to the original image size, resulting in a probability map M of size H×W. Each pixel in the probability map M represents the probability that the corresponding pixel on the original image is a key point.

[0080] In order to make the local area contain only one key point, we perform a 3×3 grid NMS on all key points. Only points with confidence higher than the set threshold τ are retained, and these high-confidence points are the output key point pixel positions p.

[0081] In the calculation of the corresponding descriptor decoder, the intermediate vector C is mapped to a semi-dense descriptor grid D∈Hc×Wc×256 through a convolutional layer, where 256 represents the dimension of the descriptor to be output. D does not contain descriptors for all pixels, but rather an 8×8 pixel block in the original image shares a descriptor. Next, D needs to be interpolated to obtain the descriptor for each pixel. Since the bicubic interpolation algorithm is time-consuming and not suitable for systems such as SLAM that require high real-time performance, we use bilinear interpolation. L2 normalization is then performed to obtain a 256-dimensional floating-point descriptor corresponding to each key point. At this point, the feature points and their descriptors in a frame of image have been extracted.

[0082] Feature matching is implemented based on graph neural network, and the process is as follows:

[0083] The graph neural network consists of two main modules: the graph attention mechanism layer and the best matching layer. Consider two images A and B, each of which has a set of extracted local features, including key point locations p and corresponding descriptors d. In the graph attention mechanism layer, the feature points and corresponding descriptors are input into a key point encoder to aggregate into an initial representation x i Among them, the key point position is embedded into a high-dimensional vector by a multi-layer perceptron (MLP), which is expressed by formula (1):

[0084] x i =d i +MLP enc (p i ) (1)

[0085] This encoder enables graph neural networks to reason about appearance and location simultaneously, especially when the descriptor d is combined with attention. Subsequently, the initial representation x is enhanced multiple times using self-attention and cross-attention. i Graph neural networks compute updated representations at each layer by aggregating messages simultaneously. is the intermediate representation of element i of image A in layer l, and information m ε→i is the result of aggregation from all key points {j:(i,j)∈ε}, where {ε∈{ε self ,ε cross}. Then the information transfer update residual of all elements i in image A is expressed as formula (2):

[0086]

[0087] where [·||·] denotes aggregation. Similar updates can be performed on all keypoints in image B simultaneously. A fixed number L of layers with different parameters are concatenated and aggregated alternately with self-attention and cross-attention.

[0088] After the attention aggregation is completed, the representation x is transformed into a vector f through linear projection. Then, the vector f enters the optimal matching layer. The matching score matrix is ​​obtained by calculating the inner product, and the sinkhorn algorithm is used to solve the optimal allocation matrix P. Each element p i,j It represents the probability that feature i in image A and feature j in image B are a pair of correct matches. So far, the feature point matching relationship between the two images has been obtained.

[0089] The specific process of initial pose estimation to solve the current frame pose is as follows:

[0090] Initial pose estimation is a key step in starting the system. First, we extract the feature points in the first frame image I0. And assume that its pose T0 is the unit matrix (indicating that the camera coordinate system is aligned with the world coordinate system) to initialize the map, and use the camera's intrinsic parameter matrix K and depth information Back-project the feature points into three-dimensional space to generate initial map points The calculation process is expressed by formula (3):

[0091]

[0092] Then, by extracting the feature points of the second frame image I1 And match it with the feature points of the first frame to get the matching point pair Based on these matching point pairs, the PnP algorithm is used to calculate the relative pose T between the two frames. 1,0= [R|t], where R is the rotation matrix and t is the translation vector. Generate new map points through triangulation method And verify whether its reprojection error is within the threshold to complete the initialization and verify the rationality of the map points.

[0093] After successful initialization, the system enters the continuous tracking process. For each new image frame, first extract feature points And based on the pose T of the previous frame t-1 Or predict the pose of the current frame using a constant velocity motion model Determine the local map area. Select map points {X n} within the field of view of the current frame from the global map to construct a sparse local map. Then, project and match the feature points of the current frame With the map points {X n} in the local map to obtain matching point pairs And eliminate false matches through geometric constraints. Based on the matching feature point pairs, construct a reprojection error function:

[0094]

[0095] Where, π is the projection function, K is the camera intrinsic matrix, R t And t t Are the rotation matrix and translation vector of the current frame respectively. Use the nonlinear optimization method Levenberg - Marquardt algorithm to optimize the pose of the current frame and minimize the reprojection error to improve the accuracy of pose estimation.

[0096] The specific process of key frame determination is as follows:

[0097] Once the pose of the current frame is solved, determine whether to set it as a new key frame according to the number of feature points of the current frame, the matching quality, and the motion amplitude. If it is set as a key frame, add its feature points to the map and update the local map. The key frame selection strategy needs to ensure that the system obtains an accurate and globally consistent pose estimation without loss of accuracy. An image frame is set as a key frame if it meets any of the following conditions:

[0098] 1) The number of extracted feature points is more than 60.

[0099] 2) The distance from the previous key frame exceeds 30 frames.

[0100] 3) The angle from the previous key frame exceeds 5 degrees.

[0101] 4) More than 25% of the feature points are not tracked.

[0102] When the robot undergoes rapid movement or encounters a complex environment, the key-frame selection strategy of the present invention is more stable. These conditions not only ensure that the established local map contains sufficient feature information, but also enable the local map to provide reasonable environmental coverage to guarantee fast and accurate tracking. According to the above criteria, it can be judged whether the current frame needs to be set as a key frame.

[0103] The specific process of the local mapping module based on key frames is as follows:

[0104] The local mapping module only processes key frames, and optimizes the local map and expands the environmental representation by processing newly added key frames. When a new key frame K i is inserted, first, it performs feature matching with adjacent key frames {K j} in the co-visibility graph, establishes the observation relationship of map points {X k}, and generates new map points {X new} through triangulation. The calculation of its three-dimensional coordinates is as follows:

[0105]

[0106] Where, and are the matched feature points in key frames K i and K j respectively, and T i and T j are their poses.

[0107] Then, the local map is optimized by minimizing the reprojection error through local BA (Bundle Adjustment):

[0108]

[0109] Where, K is the camera intrinsic matrix, R i , t i and R j , t j are the rotation matrix and translation vector of key frames K i and K j respectively. During the optimization process, the system will also remove redundant key frames and abnormal map points to maintain the compactness and consistency of the map. Finally, the local mapping thread gradually constructs an accurate and dense local environmental representation by continuously inserting key frames, generating map points, and optimizing the local map, providing a reliable basis for the optimization of the global map and loop detection.

[0110] The specific process of the loop detection module based on the bag-of-words model is as follows:

[0111] The loop detection module fuses CNN deep features and the DBoW3 bag-of-words model, including offline training, bag-of-words retrieval, similarity measurement, geometric verification, and loop correction;

[0112] In the offline training stage, an adaptive hierarchical K-means++ clustering algorithm is used to perform clustering analysis on the depth descriptors extracted by the CNN network from the public dataset; first, the 256-dimensional descriptors extracted by the CNN are compressed to 64 dimensions through PCA dimensionality reduction to improve computational efficiency while retaining more than 95% of the feature information; then, a density-based initial center selection strategy is adopted to avoid the local optimum problem that may be caused by traditional K-means; finally, the constructed visual vocabulary contains 10 5 visual words, forming a six-layer tree structure, providing fine-grained feature discrimination ability while ensuring retrieval efficiency;

[0113] The objective function for constructing the visual vocabulary is:

[0114]

[0115] where k is the number of visual words, μ i is the i-th clustering center, and λ is the spacing penalty coefficient;

[0116] In the bag-of-words retrieval stage, the pre-constructed visual vocabulary is used to quantize the descriptors extracted by the CNN network into sparse word frequency vectors, and the inverted index technology is adopted for fast retrieval; a dynamic key frame database based on a time sliding window is maintained, and fast deduplication is achieved through a Bloom filter to ensure that the retrieval efficiency is not affected by the increase in running time;

[0117] In the similarity measurement stage, the loop is detected and the consistency of the global map is optimized by identifying the similarity between the current key frame K t and the historical key frames {K h}; first, the bag-of-words descriptor D t of the current key frame K t and the bag-of-words descriptors {D h} of the historical key frames {K h} are extracted, and the similarity score is calculated by comparing their word frequency vectors to screen out candidate loop frames Then, the relative pose T t,h =[R|t] is calculated using the PnP algorithm, and it is verified whether the reprojection error satisfies the threshold:

[0118]

[0119] where K is the camera internal parameter matrix, is the map point corresponding to the historical key frame K h ;

[0120] In the geometric verification stage, a multi-stage progressive verification strategy is adopted. First, the PROSAC algorithm is used to quickly screen out candidate pairs with an inlier rate exceeding 60%. Then, geometric consistency constraints based on the essential matrix are constructed, and the M-estimator is used to suppress the influence of outliers. Subsequently, 3D-2D reprojection verification is performed, projecting the map points of the candidate frame onto the current frame, requiring that at least 50% of the matching point pairs have a reprojection error less than 1.5 pixels. Finally, descriptor consistency verification is introduced, constructing bidirectional nearest neighbor matching to ensure that the descriptor distance of the feature point pairs is less than the adaptive threshold τ_d:

[0121] τ_d = μ + 2σ (10)

[0122] where μ and σ are the mean and standard deviation of the descriptor distances of the current frame;

[0123] In the loop closure correction stage, when an effective loop closure is detected, a hierarchical optimization mechanism is started. First, local pose graph optimization is performed, using the Dog-leg algorithm for fast convergence. Then, global BA optimization is carried out, adopting the Schur complement technique to achieve efficient large-scale sparse matrix solution. To balance accuracy and efficiency, the optimization granularity is dynamically adjusted. When the loop closure interval is less than 100 frames, only the local window is optimized. When a large-scale loop closure is detected, full-map optimization is started, adjusting the poses of the key frames {T i} and the map points {X k} to minimize the global error E({T}, {X}). At this time, the poses of all key frames are updated, and the running trajectory of the robot is also corrected accordingly;

[0124]

[0125] where L represents the set of loop closure constraints, and T ij is the relative pose between the loop closure frames.

[0126] The present invention proposes a new binocular vision SLAM method based on deep learning feature extraction and matching algorithms. This method can achieve real-time positioning and mapping in complex environments, with relatively high positioning trajectory accuracy and reliable stability. This method has good scalability and real-time performance.

[0127] The present invention is mainly divided into three modules: the front-end tracking module of binocular image frames, the local mapping module, and the loop closure detection module. The method block diagram is as Figure 1 shown.

[0128] Refer to Figure 1, the process of front - end tracking is as follows: For an image in a binocular frame, the present invention first uses a convolutional neural network model to extract feature points and descriptors in the image, and then uses the idea of a graph neural network to achieve feature matching to obtain the data association between two frames. After that, initial pose estimation is performed, that is, an initial value of motion estimation is obtained according to a uniform motion model or the matching relationship between two frames, so as to achieve image tracking and pose estimation. After obtaining the pose estimation of the current frame, the system also needs to determine whether to set the current frame as a new key frame.

[0129] Refer to Figure 1 , the process of local mapping is as follows: For the real - time performance of the system, the present invention only uses key frames for mapping. For each new key frame, the present invention first performs feature matching with adjacent key frames in the covisibility graph, establishes the observation relationship of map points, and generates new map points through triangulation; then the system optimizes the local map by minimizing the reprojection error through local BA (Bundle Adjustment). During the optimization process, the system also removes redundant key frames and abnormal map points to maintain the compactness and consistency of the map, and gradually constructs an accurate and dense local environment representation.

[0130] Refer to Figure 1 , the calculation process of loop detection is as follows: The present invention uses a method based on the bag - of - words model to implement the loop detection process. For each key frame, the system extracts the bag - of - words (BoW) descriptor of the current key frame and performs similarity matching with the bag - of - words descriptors of historical key frames to screen out candidate loop frames. Then, loop confirmation is further carried out through geometric verification, that is, the relative pose is calculated using feature point matching and the PnP algorithm, and it is verified whether the reprojection error meets the threshold. If a loop is formed, a new edge is added to the established pose graph, and the pose of the new key frame and map points are optimized by optimizing the pose graph. Through loop detection and optimization, the system can significantly reduce the cumulative error and improve the global consistency of the map, so as to achieve long - term stable positioning and mapping.

[0131] Figure 2 Shows the results of feature extraction and feature matching on binocular image frames in a production workshop environment by this system. The small dots are the feature points extracted from the two images, and the line segments are the matching relationships between these feature points.

[0132] Figure 3 Shows some demonstration results of this method on the publicly available EuRoC dataset, MH_01(a), MH_03(b), MH_05(c), V1_03(d). The solid lines represent the trajectory results of this method, the dashed lines represent the ground truth, and different colors represent the deviation between the running trajectory of the method and the ground truth. Table 1 shows the quantitative results of this method on the EuRoC dataset.

[0133] Table 1 Quantitative test results of this method on the publicly available EuRoC dataset

[0134]

[0135]

[0136] Figure 4 Shows partial display results of this method on the publicly available KITTI dataset, KITTI_00(a), KITTI_01(b), KITTI_03(c), KITTI_05(d). The solid line represents the trajectory result of this method, the dashed line represents the ground truth, and different colors represent the deviation between the running trajectory of the method and the ground truth. Table 2 shows the quantitative results of this method on the KITTI dataset.

[0137] Table 2 Quantitative test results of this method on the publicly available KITTI dataset

[0138]

[0139]

[0140] Figure 5 Is the real-time running result of this method walking around an indoor scene. The green line represents the running trajectory, the blue dots represent the established map points, and the red short lines represent the key frame poses. The trajectory length is approximately 400m.

[0141] Table 3 shows the average running time of each frame of each module of the method. It can be seen that the speed of each module exceeds 10 FPS.

[0142] Table 3 Average running time of each module test of this method

[0143] Module Front-end tracking Local mapping Loop detection Running time (ms) 32.81 74.86 97.7

[0144] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A binocular vision SLAM method based on deep learning feature extraction and matching algorithm, characterized in that: The binocular image frames are processed by the front-end tracking module to obtain the camera poses between adjacent frames and the matching feature point pairs between adjacent frames. The local mapping module obtains the locally optimized key frame poses and locally optimized local map points. The loop detection module obtains the camera trajectory and global map after loop constraints and global pose graph optimization.

2. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 1, characterized in that, The front-end tracking module includes feature extraction based on convolutional neural network, feature matching based on graph neural network, pose estimation and key frame selection; The first binocular image frame is set as a key frame, and its position is set as the initial position of the map, and all feature points are extracted from the first binocular image frame to create map points; In each pair of binocular images, feature extraction is performed on the left image based on a convolutional neural network to obtain feature points and their descriptors; The feature points and descriptors of the left eye image are matched with the left eye image of the previous key frame based on the graph neural network to establish data association between frames; Perform pose estimation based on the data association results to obtain the pose of the current frame; Combined with the pose of the current frame, key frame selection is performed according to the key frame determination criteria to determine whether the current frame is a new key frame.

3. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 2, characterized in that, The specific process of feature extraction based on convolutional neural network is as follows: The convolutional neural network includes a shared VGG encoder and two independent decoders; The stereo image frame is first input into a shared VGG encoder; for the H×W input image I, the VGG encoder maps it to an intermediate vector C of Hc×Wc×128, where Hc=H / 8, Wc=W / 8; Two independent decoders are responsible for extracting the key point pixel position and calculating the corresponding descriptor respectively; in the decoder for extracting the key point pixel position, the intermediate vector C is first mapped to X∈Hc×Wc×65 through a convolutional layer; the 65 channels of X correspond to an 8×8 local area on the input image I and a Dustbin mark used to indicate that there is no key point in the area; after channel Softmax and reshape, it is restored to the original image size to obtain a probability map M with a size of H×W; each pixel in the probability map M represents the probability that the corresponding pixel on the input image I is a key point, and NMS is performed on the probability map M to remove redundant points, and only points with confidence higher than the set threshold τ are retained. These high-confidence points are the output key point pixel positions p; In the decoder that calculates the corresponding descriptor, the intermediate vector C is mapped to a semi-dense descriptor grid D∈Hc×Wc×256 through a convolutional layer, where 256 represents the output descriptor dimension; Each grid in the descriptor grid D represents a descriptor shared by an 8×8 pixel block in the input image I. The descriptor grid D is bilinearly interpolated to obtain the descriptor of each pixel in the input image I; L2 normalization is performed to obtain a 256-dimensional floating-point descriptor d corresponding to each key point.

4. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 3, characterized in that, The VGG encoder is mainly composed of convolutional layers, pooling layers and non-linear activation functions.

5. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 2, characterized in that, The specific process of feature matching based on graph neural network is as follows: The graph neural network mainly consists of a graph attention mechanism layer and an optimal matching layer; consider two images A and B, each with a set of local features that have been extracted and It contains the key point pixel position p and the corresponding descriptor d; In the graph attention mechanism layer, the pixel positions of the keypoints and the corresponding descriptors are input into a keypoint encoder to aggregate into an initial representation x i ; The pixel position of the key point is embedded into a high-dimensional vector by a multi-layer perceptron MLP, which is expressed as: x i =d i +MLP enc (p i ) (1) Use self-attention and cross-attention to enhance the initial representation x multiple times i ; set up is the intermediate representation of element i of image A in layer l, and information m ε→i is the result of aggregation from all key points {j:(i,j)∈ε}, where {ε∈{ε self ,ε cross }, then the information transfer update residual in all elements i in image A is expressed as formula (2): Among them, [·||·] represents aggregation; Simultaneously perform the same updates on all keypoints in image B as in image A; iterate for a fixed number of rounds L with different parameters, and gradually refine the matches as self-attention and cross-attention are alternately aggregated; After the attention aggregation is completed, the feature representation after attention aggregation is converted into a vector f through linear projection; the vector f enters the optimal matching layer; the matching score matrix is ​​obtained by calculating the inner product, and the sinkhorn algorithm is used to solve the optimal allocation matrix P; each element p i,j It indicates the probability that feature i in image A and feature j in image B are a pair of correct matches, and obtains the feature point matching relationship between the two images, and outputs the matching point pair 6. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 2, characterized in that, The specific process of the pose estimation is as follows: By extracting the feature points in the first frame image I0 And assume that its pose T0 is the unit matrix to initialize the map, and use the camera's intrinsic parameter matrix K and depth information Back-project the feature points into three-dimensional space to generate initial map points The calculation process is expressed by formula (3): According to the feature points of the second frame image I1 Matching point pairs that match the feature points of the first frame The relative pose T between two frames is calculated using the PnP algorithm. 1,0 = [R|t], where R is the rotation matrix and t is the translation vector; new map points are generated by triangulation method And verify whether its reprojection error is within the threshold, complete the initialization and verify the rationality of the map points; After successful initialization, continuous tracking is performed; For each new frame of image, first extract the feature points And according to the posture T of the previous frame t-1 Or the uniform motion model predicts the pose of the current frame Determine the local map area; Select a map point from the global map that is within the field of view of the current frame {X n }, build a sparse local map; current frame feature points With the map point {X n } Perform projection matching or descriptor matching, and obtain the matching point pairs Eliminate mismatches through geometric constraints; construct a reprojection error function based on the matched feature point pairs: Among them, π is the projection function, K is the camera intrinsic parameter matrix, R t and t t are the rotation matrix and translation vector of the current frame respectively; the nonlinear optimization method Levenberg-Marquardt is used to optimize the pose of the current frame and minimize the reprojection error to improve the accuracy of pose estimation.

7. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 2, characterized in that, The key frame determination process is as follows: After solving the pose of the current frame, determine whether to set it as a new keyframe based on the number of feature points, matching quality, and motion amplitude of the current frame; if it is set as a keyframe, add its feature points to the local map and update the local map; when an image frame meets any of the following conditions, it is considered a keyframe: 1) The number of extracted feature points is more than 60; 2) The distance from the previous key frame exceeds 30 frames; 3) The angle with the previous keyframe exceeds 5 degrees; 4) More than 25% of the feature points are not tracked.

8. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 1, characterized in that, The process of the local mapping module is as follows: The local mapping module only processes key frames; when a new key frame K i When it is inserted, it is first compared with the adjacent keyframes {K j }Perform feature matching and establish map points {X k } observation relationship, and generate a new map point {X new }, its three-dimensional coordinates are calculated as: in and The key frames K i The feature points and key frames K matched in j The matching feature points in T i and T j is the key frame K i The pose and key frame K j 's posture; Optimize the local map and minimize the reprojection error through local BA: Among them, K is the camera internal parameter matrix, R i ,t i and R j ,t j The key frames K i and K j The rotation matrix and translation vector of the local environment are obtained by denoting the rotation matrix and translation vector of the local environment. During the optimization process, redundant keyframes and abnormal map points are removed to keep the map compact and consistent. Finally, the local mapping thread gradually builds an accurate and dense local environment representation by continuously inserting keyframes, generating map points and optimizing the local map.

9. The binocular vision SLAM method based on deep learning feature extraction and matching algorithm according to claim 1, characterized in that, The process of the loop detection module is as follows: The loop detection module integrates CNN deep features and the DBoW3 bag-of-words model, including offline training, bag-of-words retrieval, similarity measurement, geometric verification, and loop correction; In the offline training stage, an adaptive hierarchical K-means++ clustering algorithm is used to perform cluster analysis on the deep descriptors extracted by the CNN network in the public dataset. First, the 256-dimensional descriptors extracted by CNN are compressed to 64 dimensions through PCA dimensionality reduction to improve computational efficiency while retaining more than 95% of the feature information. Then, a density-based initial center selection strategy is adopted to avoid the local optimal problem that may be caused by traditional K-means. The final visual dictionary contains 10 5 visual words, forming a six-layer tree structure, which provides fine-grained feature differentiation capabilities while ensuring retrieval efficiency; The objective function for constructing a visual dictionary is: Where k is the number of visual words, μ i is the i-th cluster center, λ is the spacing penalty coefficient; In the bag-of-words retrieval stage, the descriptors extracted by the CNN network are quantized into sparse word frequency vectors using a pre-built visual dictionary, and the inverted index technology is used for fast retrieval. A dynamic keyframe database based on a time sliding window is maintained, and Bloom filters are used to achieve fast deduplication, ensuring that the retrieval efficiency is not affected by the increase in running time. In the similarity measurement stage, by identifying the current key frame K t With the historical keyframe {K h } to detect loop closure and optimize the consistency of the global map; first extract the current key frame K t The bag-of-words descriptor D t and historical keyframes {K h }'s bag-of-words descriptor {D h }, by comparing the word frequency vectors of the two to calculate the similarity score, the candidate loop frames are screened out Next, the PnP algorithm is used to calculate the relative pose T t,h = [R|t], and verify whether the reprojection error meets the threshold: Where K is the camera intrinsic parameter matrix, is the historical key frame K h The corresponding map point; The geometric verification stage adopts a multi-stage progressive verification strategy. First, the PROSAC algorithm is used to quickly screen out candidate pairs with an inlier rate of more than 60%. Then, a geometric consistency constraint based on the essential matrix is ​​constructed, and the influence of outliers is suppressed using the M estimator. Subsequently, a 3D-2D reprojection verification is performed to project the map points of the candidate frame to the current frame, requiring that at least 50% of the matching point pairs have a reprojection error of less than 1.5 pixels. Finally, descriptor consistency verification is introduced to construct a bidirectional nearest neighbor match to ensure that the descriptor distance of the feature point pair is less than the adaptive threshold τ_d: τ_d=μ+2σ (10) Where μ and σ are the mean and standard deviation of the descriptor distance of the current frame; In the loop correction stage, when a valid loop is detected, the hierarchical optimization mechanism is started; first, the local pose graph is optimized, and the Dog-leg algorithm is used for rapid convergence; then the global BA optimization is performed, and the Schur complement technique is used to achieve efficient large-scale sparse matrix solution; in order to balance accuracy and efficiency, the optimization granularity is dynamically adjusted; when the loop interval is less than 100 frames, only the local window is optimized; when a large-scale loop is detected, the whole graph optimization is started, and the key frame pose is adjusted through pose graph optimization {T i } and map point {X k }, minimize the global error E({T},{X}), at which point the poses of all key frames are updated and the robot's trajectory is corrected accordingly; Where L represents the loop constraint set, T ij is the relative pose between loop closure frames.

Images