A synchronous tracking and mapping method fusing deep learning feature matching

Abstract

The application discloses a kind of synchronous tracking and mapping methods of fusing deep learning feature matching, it includes the following steps: step S1, acquisition vision front end image, initialization is carried out;Step S2, motion pose is estimated by feature matching, track local map, determine next key frame;Step S3, according to key frame information, constructs local map;Step S4, loop detection and optimization.The application provides a kind of synchronous tracking and mapping methods of fusing deep learning feature matching, solves the technical problems that existing SLAM method is poor in tracking effect, pose solution error is big in dynamic, low texture etc.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, in particular to the technical field of computer vision, and specifically relates to a synchronous tracking and mapping method fusing deep learning feature matching. BACKGROUND

[0002] The synchronous tracking and mapping (PTAM, Parallel Tracking And Mapping) technology is a simultaneous localization and mapping algorithm (SLAM, Simultaneous Localization And Mapping), which belongs to the core technical field of visual navigation and intelligent robot control, and is currently widely used in unmanned aerial vehicles, indoor intelligent robots, unmanned driving and the like.

[0003] Most of the existing SLAM systems are based on ORBSLAM2 and ORBSLAM3, and the biggest advantage of the ORB feature matching algorithm is fast algorithm speed, which can meet the real-time requirement of the SLAM system. However, in dynamic and low-texture environments, the target feature recognition accuracy is low, the feature points are insufficient, the selection and application of key frames are affected, and the tracking difficulty is also large. In the matching process, there may be false matching, which leads to errors in the selection of map points and the estimation of motion poses.

[0004] In recent years, the feature matching method based on deep learning has gradually become popular. Tang et al. were inspired by the deep learning feature extraction algorithm SuperPoint and proposed GCNv2-SLAM, which replaced the feature point extraction and description method in the ORB algorithm and met the real-time requirement of SLAM, but the system did not work well in the environment of occlusion and low texture. Shen et al. were inspired by Superglue and proposed an end-to-end feature matching method LoFTR, which still cannot cope with the occlusion environment, and the algorithm time is difficult to meet the real-time requirement of SLAM. The above Superglue and LoFTR algorithms can obtain a large number of matching pairs in images, which can solve the problem of large tracking difficulty and pose estimation error in the SLAM system due to the lack of feature points in dynamic and low-texture environments to a certain extent.

[0005] In the existing literature, the paper "S-PTAM:Stereo Parallel Tracking and Mapping" provides a binocular vision SLAM method, which also uses ORB matching method, and only provides constant speed motion model and repositioning in the tracking process, and in dynamic and low texture environment, enough number of feature points cannot be extracted, and more tracking loss situations cannot be coped with, and mapping is also affected. Combining the SLAM method with the feature matching of deep learning can solve the above problems to a certain extent. SUMMARY

[0006] The application provides a synchronous tracking and mapping method fusing deep learning feature matching, and solves the technical problems of poor tracking effect and large pose solution error of the existing SLAM method in dynamic and low texture environments.

[0007] The technical scheme adopted by the application is as follows:

[0008] A synchronous tracking and mapping method fusing deep learning feature matching, comprising the following steps:

[0009] Step S1, collecting visual front-end images and initializing;

[0010] Step S2, estimating motion pose through feature matching, tracking local map, and determining the next key frame;

[0011] Step S3, constructing a local map according to key frame information;

[0012] Step S4, loop detection and optimization.

[0013] In step S1, the following steps are included:

[0014] The visual front-end images are collected by using a binocular camera, the LoFTR algorithm is used for matching, the matching result is used as an initialization tracking thread, and the map is initialized.

[0015] The specific steps are as follows:

[0016] Step S101, calling a gpu, selecting a first frame binocular image as a key frame, using the LoFTR algorithm to match through gpu operation, and outputting the format as [mkpts0, mkpts1, mconf]; wherein mkpts0 and mkpts1 are coordinate information of feature points in left and right image, and mconf is a matching pair confidence;

[0017] Step S102, record the feature point coordinates of the matching pair on the left and right eye images; use the Brief descriptor extractor to extract the respective corresponding descriptors at the feature points in the format [desp0, desp1]; use the feature point coordinates for feature point triangulation to calculate the 3D coordinates of the feature points and update the initial pose information;

[0018] Step S103, create and initialize the map, insert the key frame into the map, and initialize the first key frame 3D coordinates as the map point. Compared with the ORB algorithm, LoFTR can match more feature points and solve more accurate pose information, which is beneficial to the accuracy of subsequent tracking. Although the time consumption is increased, the time consumption in the initialization process does not affect the real-time performance of the subsequent system.

[0019] In step S101, when matching using the LoFTR algorithm, the following steps are adopted:

[0020] Step S101-1, extract coarse level feature map F1 and fine level feature map F3 through the feature extraction network

[0021] Step S101-2, position encoding is performed on the coarse level feature map F1, and multi-head cross attention is calculated, and the feature map F1 is output ′ ;

[0022] Step S101-3, the feature map F1 ′ is matched using the dual-softmax algorithm, the mutual nearest neighbor MNN algorithm is used to screen the matching pairs, and the coarse level feature matching Mc is finally output;

[0023] Step S101-4, map the coarse level feature matching Mc in the fine level feature map F3, crop a window of target size, extract features again and match, and finally screen out the fine level feature matching Mf as the final output result.

[0024] In step S2, the motion pose is estimated by feature matching tracking, the local map is tracked, and the next key frame is determined;

[0025] Specifically, the following steps are included:

[0026] Step S201, to ensure the real-time performance of the tracking main thread, use ORB feature matching, and sequentially pass through three tracking models of step S201-1, step S201-2, and step S201-3 for inter-frame tracking, output the camera pose information, and use it for local map tracking; any model tracking is successful, after the step of the tracking model is completed, enter step S202;

[0027] Step S201-1, use the constant speed model for tracking, and the steps are as follows:

[0028] Step (1) estimates the initial pose of the current frame by using the constant velocity motion model based on the pose of the previous frame, and saves the current constant velocity model;

[0029] Step (2) projects the temporary map points of the previous frame to the current coordinate system, and matches the descriptors in the two frames. When the number of matching points with a Euclidean distance less than a threshold is greater than a certain number, the tracking is successful.

[0030] Step (3) uses the re-projection error as the objective function, and optimizes the pose of the current frame using the least squares method, and removes outliers in the optimization process.

[0031] The re-projection error calculation formula is as follows:

[0032]

[0033] Where e is the re-projection error; P is the coordinate of the feature point in the world coordinate system of the previous frame, u is the pixel coordinate of the previous frame projected in the current frame, s is the conversion parameter of the depth value and the actual distance, K is the camera intrinsic parameter, ζ is the Lie algebra corresponding to the camera pose at this time, exp(ζ) is the logarithmic mapping from vector to matrix of ζ; the sum of the re-projection errors of all tracked feature points is used as the optimization objective function, and the least squares method is used for optimization.

[0034] Step S201-2, track by referring to the information of the previous key frame, and the steps are as follows:

[0035] Step (1) select the first-level and second-level key frames with co-view relationship as reference;

[0036] Step (2) use the bag-of-words (BoW) to accelerate the matching of the current frame and the reference frame. If the number of matching points with a Euclidean distance less than a threshold is greater than a certain number, use the pose of the previous frame as the initial value of the current frame;

[0037] Step (3) optimize the current pose using the re-projection error and calculate the inliers. If the number of inliers is greater than a certain number, the tracking is successful.

[0038] Step S201-3, track using relocalization, and the steps are as follows:

[0039] Step (1) calculate the BoW vector of the current frame, and find similar candidate key frames using the bag-of-words;

[0040] Step (2) perform feature matching and pose estimation between the candidate key frames and the current frame;

[0041] Step (3) perform re-projection error optimization. If the number of inliers is greater than a certain number, the tracking is considered successful.

[0042] Preferably, the temporary map points in step S201-1 are not included in the map points, and are deleted after tracking is completed.

[0043] Step S202, determine whether the current frame is a key frame. Step S202 conditions are as follows:

[0044] Condition (1) the ratio of the number of matching points tracked by the current frame to the total number of matching points is less than the threshold, and the key frame is inserted;

[0045] Condition (2) the number of feature points tracked by the current frame is less than a certain number, and the key frame is inserted;

[0046] Condition (3) when the local mapping is idle, or does not satisfy any one of conditions (1) and (2), a key frame is inserted every certain number of frames;

[0047] If any of the above conditions are met, the current frame is determined to be a key frame, and step S3 is entered; At the same time, the condition of step S203 is judged. Step S202 modifies the key frame determination condition to maintain a certain interval between key frames, avoiding process disorder caused by frequent opening of subsequent sub-threads.

[0048] Step S203, create a key frame sub-thread; When a new key frame is determined, and the interval between the current key frame and the previous key frame is greater than a certain number, the main thread continues, and the sub-thread is started;

[0049] Step S203-1, in the sub-thread, use LoFTR algorithm to match by gpu operation;

[0050] Comprising the following steps:

[0051] Step (1) use LoFTR algorithm to re-match left and right image in key frame, get 2D feature matching coordinates, and select matching pairs with confidence mconf greater than threshold;

[0052] Step (2) use Brief descriptor extractor to extract descriptor at feature point, calculate 3D coordinates of feature point, and update pose information;

[0053] Step (3) additional LoFTR calculation results are added in the key frame information, the format is [pose, points, mkpts, desp], pose is pose information, points is 3D coordinates of feature points, mkpts is 2D feature points, and desp is descriptor.

[0054] Step S203-2, when the sub-thread step S203-1 is completed, determine:

[0055] Condition (1) the next key frame has not been selected; Take the union of 3D coordinates in step S203-1 and the current local map points; Take the union of 2D feature points, descriptors and information of the key frame solved by ORB in the main thread; Replace the pose information of the key frame in the main thread;

[0056] The condition (2) is that a next key frame has been selected or a local mapping thread is idle; the 3D coordinates in step S203-1 are taken and collected with the current local map points. In this way, the sub-thread starting condition is used to avoid the main thread disorder caused by the frequent starting of the sub-thread; meanwhile, enough time is reserved for the LoFTR algorithm, so that the LoFTR algorithm can update the LoFTR solving information of the key frame before the next two key frames are judged, and the information can be applied to step S1 and step three in the main thread in time, so that better tracking and mapping effects can be achieved through more detailed key frame information.

[0057] When step S203-2 is performed, if the main thread is in a local map tracking stage or a local mapping stage, the step is continued after the main thread stage is completed. In this way, the main thread can be smoothly performed when the sub-thread and the main thread conflict.

[0058] In step S3, the following steps are specifically used:

[0059] Step S301, a local map is inserted after the key frame is judged;

[0060] Step S302, key frames with a co-view relationship are selected, and map points are fused and selected;

[0061] Step S303, local BA optimization is performed.

[0062] In step S4, the following steps are specifically used:

[0063] Step S401, a current key frame queue is detected, grouped by a co-view relationship, and local key frame continuity is detected;

[0064] Step S402, a Sim3 similarity transformation is calculated, and map points and key frame poses are optimized;

[0065] Step S403, map loop fusion is performed.

[0066] Compared with the prior art, the present application has the following technical effects:

[0067] 1) The LoFTR algorithm is used for initialization during initialization, and compared with the traditional ORB algorithm, the LoFTR deep learning feature matching can obtain more image matching pairs, and the accuracy is higher than that of the ORB. Although the initialization time is improved, the 3D feature point information obtained by the initialization is also increased, and more accurate initial pose information is provided, which is beneficial to the accuracy of subsequent tracking.

[0068] 2) In the key frame, a new sub-thread method is proposed, which uses the LoFTR deep learning image matching method to perform image matching again without affecting the main thread, improving the key frame pose accuracy and expanding the map point information, which is beneficial to the reference key frame tracking, local map tracking and obtaining more 3D map point information. The sub-thread is synchronized with the main thread, and the sub-thread opening condition is reserved for the sub-thread algorithm time, and the sub-thread is not frequently opened to cause process disorder, so that the thread does not conflict and runs in real time.

[0069] 3) The application re-matches the key frame, expands the key frame information, ensures the smooth progress of the tracking thread and the local mapping thread, improves the robustness of the SLAM system under dynamic and low texture conditions, and improves the accuracy of the SLAM system. BRIEF DESCRIPTION OF DRAWINGS

[0070] The application will be further described below in conjunction with the drawings and examples:

[0071] Figure 1 The whole flowchart of the application is shown in the figure:

[0072] Figure 2 The sub-thread flowchart of the application is shown in the figure:

[0073] Figure 3 The KITTI dataset experimental result example graph is shown in the figure: DETAILED DESCRIPTION

[0074] As shown in the figure, a synchronous tracking and mapping method fusing deep learning feature matching: Figure 1

[0075] Step S1, collect visual front-end images and initialize;

[0076] Step S2, estimate motion pose by feature matching, track local map and determine next key frame;

[0077] Step S3, construct local map according to key frame information;

[0078] Step S4, loop detection and optimization.

[0079] In step S1, the following steps are used:

[0080] Use binocular camera to collect visual front-end images, use LoFTR algorithm for matching, use the matching result as the initialization tracking thread, and perform initial mapping.

[0081] The specific steps are as follows:

[0082] ​Step S101, call the gpu, select the first frame binocular image record as the key frame, use LoFTR algorithm to match through gpu operation, and the output format is [mkpts0, mkpts1, mconf]. Wherein mkpts0, mkpts1 are the coordinate information of feature points in left and right eye images respectively, and mconf is the matching confidence;

[0083] Step S102, record the coordinates of the matching pairs in the left and right eye images; use the Brief descriptor extractor to extract the respective descriptors at the feature points, and the format is [desp0, desp1]; use the feature point coordinates to triangulate the feature points, calculate the 3D coordinates of the feature points, and update the initial pose information;

[0084] Step S103, create and initialize the map, insert the key frame into the map, and initialize the first key frame 3D coordinates as the map point.

[0085] Preferably, compared with the ORB algorithm, LoFTR can match more feature points and solve more accurate pose information, which is beneficial to the accuracy of subsequent tracking. Although the time consumption is increased, the time consumption in the initialization process does not affect the real-time performance of the subsequent system.

[0086] In step S101, when using the LoFTR algorithm for matching, the following steps are adopted:

[0087] Step S101-1, extract the coarse level feature map F1 and the fine level feature map F3 through the feature extraction network;

[0088] Step S101-2, position encoding is performed on the coarse level feature map F1, and multi-head cross attention is calculated, and the feature map F1 ′ is outputted;

[0089] Step S101-3, the feature map F1 ′ is matched using the dual-softmax algorithm, the mutual nearest neighbor MNN algorithm is used to screen the matching pairs, and finally the coarse level feature matching Mc is outputted;

[0090] Step S101-4, map the coarse level feature matching Mc in the fine level feature map F3, crop the window of the target size, extract the features again and match, and finally screen out the fine level feature matching Mf as the final output result.

[0091] In step S2, the motion pose is estimated by feature matching tracking, the local map is tracked, and the next key frame is determined.

[0092] Specifically, the following steps are included:

[0093] Step S201, in order to ensure the real-time tracking of the main thread, using ORB feature matching, sequentially through step S201-1, step S201-2, step S201-3 three tracking models for inter-frame tracking, output camera pose matrix, for local map tracking; any model tracking success, after waiting for the tracking model step is completed, then enter step S202;

[0094] Step S201-1, using constant speed model for tracking, the steps are as follows:

[0095] Step (1) using the last frame of the solved pose, through the constant speed motion model to estimate the current initial pose, save the current constant speed model;

[0096] Step (2) using the last frame of the solved temporary map point, project to the current coordinate system, match the descriptors in the two frames, when the number of matching points with the Euclidean distance less than the threshold is greater than 20, the tracking is successful;

[0097] Step (3) the re-projection error as the objective function, using the least square method to optimize the current frame pose, eliminate the outliers in the optimization process (the optimization method is referred to as BA optimization).

[0098] The re-projection error calculation formula is as follows:

[0099]

[0100] Where e is the re-projection error; P is the position of the last frame feature point in the world coordinate system, u is the pixel coordinate of the last frame projected in the current frame, s is the conversion parameter of the depth value and the actual distance, K is the camera intrinsic parameter, ζ is the Lie algebra corresponding to the camera pose at this time, exp(ζ) is the logarithmic mapping from vector to matrix of ζ; using the sum of all tracking feature points re-projection error as the optimization objective function, using the least square method to optimize;

[0101] Step S201-2, reference the last key frame information for tracking, the steps are as follows:

[0102] Step (1) select the first level, second level key frame reference with co-view relationship;

[0103] Step (2) using the bag of words (BoW) to accelerate the matching of the current frame and the reference frame, if the number of matching points with the Euclidean distance less than the threshold is greater than 20, the last frame pose is used as the initial value of the current frame;

[0104] Step (3) using the re-projection error to optimize the current pose and calculate the inliers, if the number of inliers is greater than 15, the tracking is successful.

[0105] Step S201-3, using reposition for tracking, the steps are as follows:

[0106] Step (1) Calculate the BoW bag-of-words vector of the current frame, and find similar candidate key frames with the bag-of-words;

[0107] Step (2) Feature matching and pose estimation between the candidate key frame and the current frame;

[0108] Step (3) Reprojection optimization, if the number of inliers is greater than 10, it is considered that the tracking is successful.

[0109] Preferably, the temporary map points in step S201-1 are not counted in the map points, and are deleted after tracking is completed.

[0110] Step S202, judge whether the current frame is a key frame. The conditions of step S202 are as follows:

[0111] Condition (1) The ratio of the number of inliers to the total number of matching points tracked by the current frame is less than 0.25, and the key frame is inserted;

[0112] Condition (2) The number of feature points tracked by the current frame is less than 20 points, and the key frame is inserted;

[0113] Condition (3) When the local mapping is idle, or any one of conditions (1) and (2) is not met, a key frame is inserted every certain number of frames;

[0114] If any of the above conditions is met, the current frame is judged as a key frame, and step S3 is entered; at the same time, the condition of step S203 is judged.

[0115] Step S203, create a key frame sub-thread. When the new key frame is determined, and the interval between the current key frame and the previous key frame is >=8, the main thread continues, and the sub-thread is started:

[0116] Step S203-1, in the sub-thread, use the LoFTR algorithm to perform matching through gpu operation. The steps are as follows:

[0117] Step (1) Use the LoFTR algorithm to re-match the left and right images in the key frame, obtain 2D feature matching coordinates, and select matching pairs with a confidence mconf greater than 0.8;

[0118] Step (2) Use the Brief descriptor extractor to extract descriptors at the feature points, calculate the 3D coordinates of the feature points, and update the pose information;

[0119] Step (3) Additional LoFTR calculation results are added in the key frame information, in the format of [pose, points, mkpts, desp], pose is the pose information, points is the 3D coordinates of the feature points, mkpts is the 2D feature points, and desp is the descriptor.

[0120] Step S203-2, when the sub-thread step S203-1 is calculated, a judgment is made:

[0121] Condition (1) the next key frame is not selected. The 3D coordinates in step S203-1 are taken with the current local map points; the 2D feature points and the descriptors are taken with the information of the ORB in the main thread to solve the key frame; the pose information of the key frame in the main thread is replaced;

[0122] Condition (2) the next key frame is selected or the local mapping thread is idle. The 3D coordinates in step S203-1 are taken with the current local map points;

[0123] Preferably, when step S203-2 is performed, if the main thread is in the local map tracking stage or the local mapping stage, it is waited for the main thread stage to be completed before continuing. In this way, when the sub-thread and the main thread conflict, the main thread is given priority to proceed smoothly.

[0124] Preferably, step S203-2 updates the pose information of the last key frame and saves more local map point results, and intervenes in step S201 in the tracking process, improving the success rate of reference key frame tracking. More 3D point information is provided for the local map and the global map, which is beneficial to map tracking and construction.

[0125] Step S3, constructing a local map according to the key frame information. The steps are as follows:

[0126] Step S301, judging whether the key frame is inserted into the local map and updated;

[0127] Step S302, selecting key frames with co-view relationship, and performing map point fusion and screening;

[0128] Step S303, performing local BA optimization.

[0129] Step S4, loop detection and optimization. The steps are as follows:

[0130] Step S401, detecting the current key frame queue, grouping by co-view relationship, and detecting the continuity of local key frames;

[0131] Step S402, calculating Sim3 similarity transformation, and optimizing map points and key frame poses;

[0132] Step S403, performing map loop fusion.

[0133] The traditional ORB algorithm and the LoFTR deep learning algorithm are tested for matching on the first frame or so of the left eye image and the tenth frame or so of the left eye image in the KITTI-00 dataset. The results are shown in Tables 1 and 2.

[0134] Table 1 ORB vs. LoFTR 1st frame comparison

[0135]

[0136] Table 2 ORB vs. LoFTR 10th frame comparison

[0137]

[0138] From the above table, it can be seen that the ORB algorithm is far less than LoFTR in terms of matching time consumption and RANSAC optimization time consumption, but the precision and the number of matching pairs obtained are far less than LoFTR. And the initialization time consumption and the 10th frame time consumption have a large gap, the reason is that the first time the GPU reads and processes the image is very time-consuming, and after the sub-thread is started, there is no such problem. After the sub-thread is started, the LoFTR algorithm time consumption is greatly shortened; at the same time, in order to ensure the real-time performance of the sub-thread, steps S202 and S203 are needed to filter the key frames, and the sub-thread is started according to the filtering conditions, and the time for the sub-thread is reserved to ensure that the calculation results of the sub-thread can be applied to the main thread in time, while avoiding process disorder; and the calculation results of the sub-thread are filtered through S203, to avoid too much feature point information of the key frame, which leads to too long calculation time of the subsequent process and fails to meet the real-time requirement. The initialization does not need to worry about this problem.

[0139] According to the SLAM algorithm provided by the application, the SLAM system runs in the Ubantu 18.04, python 3.6 environment, and the schematic diagram of the running completion on the KITTI-00 data set is as shown in Figure 3 It can be known from Figure 3 that the SLAM system integrates the trajectory and key frame calculation information in the main thread into the visual map. The visual information includes the tracking process trajectory of the SLAM system, the position of all key frames in the tracking process, the global map established according to the key frame information, and all map points obtained in the whole process.

Claims

1. A synchronous tracking and mapping method integrating deep learning feature matching, characterized in that, It includes the following steps: Step S1: Acquire images from the visual front end and initialize it; Step S2: Estimate motion pose through feature matching, track the local map, and determine the next keyframe; Step S3: Construct a local map based on the keyframe information; Step S4: Loop closure detection and optimization; In step S1, a stereo camera is used to acquire visual front-end images, the LoFTR algorithm is used for matching, and the matching results are used to initialize the tracking thread and initialize the map. In step S2, motion pose is estimated by feature matching and tracking, the local map is tracked, and the next keyframe is determined. Specifically, the following steps are included: Step S201: To ensure the real-time performance of the main tracking thread, ORB feature matching is used. Inter-frame tracking is performed sequentially through three tracking models: Step S201-1, Step S201-2, and Step S201-3. Camera pose information is output for local map tracking. Once any model successfully tracks, the process waits for the tracking model to complete its steps before proceeding to Step S202. Step S201-1: Track using a constant velocity model, the steps are as follows: Step (1) Using the pose calculated in the previous frame, estimate the current initial pose through the constant velocity motion model and save the current constant velocity model; Step (2) Use the temporary map points calculated in the previous frame to project onto the current coordinate system and match the descriptors in the two frames. When the number of matching points with a Euclidean distance less than the threshold is greater than a certain number, the tracking is successful. Step (3) uses the reprojection error as the objective function, and optimizes the pose of the current frame using the least squares method, eliminating outliers during the optimization process; The formula for calculating reprojection error is as follows: ； in e This is for reprojection error; P These are the coordinates of the feature points in the world coordinate system from the previous frame. u Here, represents the pixel coordinates of the previous frame projected onto the current frame, and s is the conversion parameter between the depth value and the actual distance. K For camera internal parameters, This is the Lie algebra corresponding to the camera pose at this moment. yes Logarithmic mapping from vector to matrix; optimization using least squares method with the sum of reprojection errors of all tracked feature points as the objective function; Step S201-2: Tracking is performed with reference to the information from the previous keyframe. The steps are as follows: Step (1) Select primary and secondary keyframe references that have a co-view relationship; Step (2) Use bag-of-words (BoW) to accelerate the matching between the current frame and the reference frame. If the number of matching points with a Euclidean distance less than the threshold is greater than a certain number, use the pose of the previous frame as the initial value of the current frame. Step (3) Use reprojection error to optimize the current pose and calculate interior points. If the number of interior points is greater than a certain number, the tracking is successful. Step S201-3: Use relocation for tracking, the steps are as follows: Step (1) Calculate the current frame's BoW bag-of-words vector and use the bag-of-words vector to find similar candidate keyframes; Step (2) Perform feature matching and pose estimation between the candidate keyframes and the current frame; Step (3) optimizes the reprojection error. If the number of interior points is greater than a certain number, the tracking is considered successful. In step S201-1, temporary map points are not counted as map points and are deleted after tracking is completed; Step S202: Determine whether the current frame is a keyframe; the conditions for step S202 are as follows: Condition (1) If the ratio of the number of points within the matching points tracked in the current frame to the total number of matching points is less than the threshold, insert a keyframe; Condition (2) If the number of feature points tracked in the current frame is less than a certain number, insert a keyframe; Condition (3) When the local mapping is idle, or when either (1) or (2) is not met, keyframes are inserted every certain number of frames; If any of the above conditions are met, the current frame is determined to be a key frame, and the process proceeds to step S3; at the same time, the condition judgment in step S203 is performed. Step S203: Create a keyframe sub-thread; when a new keyframe is determined and the interval between the new keyframe and the previous keyframe is greater than a certain number of frames, the main thread continues and the sub-thread is started. Step S203-1: In the child thread, the LoFTR algorithm is used to perform matching via GPU computation; Includes the following steps: Step (1) Use the LoFTR algorithm to rematch only the left and right eye images in the keyframe, obtain the 2D feature matching coordinates, and filter out the matching pairs with confidence mconf greater than the threshold. Step (2) Use the Brief descriptor extractor to extract descriptors at feature points, calculate the 3D coordinates of feature points, and update pose information; Step (3) Add the LoFTR calculation result to the keyframe information. The format is [pose, points, mkpts, desp], where pose is the pose information, points are the 3D coordinates of the feature points, mkpts are the 2D feature points, and desp is the descriptor. Step S203-2: When the sub-thread completes the calculation in step S203-1, a judgment is made: Condition (1) The next keyframe has not yet been selected; take the union of the 3D coordinates in step S203-1 with the current local map points; take the union of the 2D feature points, descriptors and the information of the keyframe solved by ORB in the main thread; replace the pose information of the keyframe in the main thread; Condition (2) The next keyframe has been selected or the local mapping thread is idle; take the union of the 3D coordinates in step S203-1 with the current local map points.

2. The method according to claim 1, characterized in that, In step S1, the specific steps are as follows: Step S101: Call the GPU, select the first frame of the stereo image as the keyframe, and use the LoFTR algorithm to perform matching through GPU operation. The output format is [mkpts0, mkpts1, mconf]; where mkpts0 and mkpts1 are the coordinate information of the feature points in the left and right eye images, respectively, and mconf is the confidence score of the matching pair. Step S102: Record the feature point coordinates of the matching pair on the left and right eye images; use the Brief descriptor extractor to extract the corresponding descriptors at the feature points, in the format [desp0, desp1]; use the feature point coordinates to perform feature point triangulation, calculate the 3D coordinates of the feature points, and update the initial pose information. Step S103: Create and initialize the map, insert keyframes into the map, and initialize the 3D coordinates of the first keyframe as map points.

3. The method according to claim 2, characterized in that, In step S101, when using the LoFTR algorithm for matching, the following steps are performed: Step S101-1: Extract coarse-level feature maps using a feature extraction network. Fine-level feature maps ; Step S101-2: Process the coarse-level feature map Perform positional encoding and compute multi-head cross-attention to output feature maps. ; Step S101-3: Process the feature map The dual-softmax algorithm is used for matching, and the nearest neighbor MNN algorithm is used to filter matching pairs. The final output is a coarse-level feature matching Mc. Step S101-4: Map the coarse-level feature matching Mc onto the fine-level feature map In the process, a window of the target size is cropped, features are extracted and matched again, and finally, the fine-level feature matching Mf is selected as the final output.

4. The method according to claim 1, characterized in that, If the main thread is in the local map tracking stage or the local mapping stage when performing step S203-2, it will wait for the main thread stage to complete before continuing.

5. The method according to claim 1, characterized in that, In step S3, the following steps are specifically adopted: Step S301: After determining the keyframes, insert a local map and update the information; Step S302: Select keyframes with shared viewing relationships and perform map point fusion and filtering; Step S303: Perform local BA optimization.

6. The method according to claim 1, characterized in that, In step S4, the following steps are specifically adopted: Step S401: Detect the current keyframe queue, group them by co-view relationship, and detect the continuity of local keyframes; Step S402: Calculate Sim3 similarity transformation to optimize map point and keyframe pose; Step S403: Perform map loop fusion.

Images