Synchronous positioning and mapping method and system based on deep learning

Through deep learning models, image features are extracted and screened, combined with graph neural network and visual bag-of-word optimization SLAM system, the difficulty of feature extraction in traditional SLAM in challenging environments is solved, and efficient pose estimation and map construction are achieved.

CN120543637APending Publication Date: 2025-08-26NAT INNOVATION INST OF DEFENSE TECH PLA ACAD OF MILITARY SCI
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202510522330.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-04-24
Publication Date
2025-08-26

AI Technical Summary

Technical Problem

Traditional SLAM systems are difficult to extract effective features in low-light, dynamic lighting, jitter and other scenarios, and end-to-end deep learning methods consume high computing resources and limited generalization capabilities, resulting in insufficient pose estimation accuracy and real-time performance.

Method used

Image features are extracted using deep learning models, feature points are screened through graph neural networks, feature descriptors are processed binarized, and closed-loop detection is combined with visual bags of words to optimize position and map construction.

Benefits of technology

It improves the robustness and accuracy of SLAM systems in challenging environments, and realizes real-time and accurate pose estimation and map construction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120543637A_ABST
    Figure CN120543637A_ABST
Patent Text Reader

Abstract

The invention discloses a deep learning-based synchronous positioning and mapping method, which comprises the following steps of: extracting image features by using a deep learning model to obtain a feature point probability tensor and a descriptor tensor; adaptively screening feature points in the feature point probability tensor according to the inter-frame feature relationship and the intra-frame feature relationship, and generating a screened feature point set; constructing a graph neural network model by using the screened feature point set and the descriptor tensor; obtaining a feature point matching relation according to an output result of the graph neural network; performing binarization processing on the depth feature descriptors of the floating-point number, and training a visual word bag by using the feature descriptors after binarization processing; performing closed-loop detection by using the trained visual word bag, and retrieving similar key frames; and carrying out pose optimization and map construction by utilizing the retrieved similar key frames and feature matching results. According to the invention, a more accurate scene identification effect is realized, and real-time, robust and accurate pose estimation and map construction of the hybrid SLAM system are also realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for simultaneous positioning and mapping based on deep learning. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) systems, which enable simultaneous localization and mapping, are systems that build environmental maps and determine their own position in real time. SLAM systems based on traditional feature extraction algorithms struggle to extract effective features and perform pose estimation in areas with low or dynamic lighting, severe jitter, or weak textures, as they only consider local, low-dimensional features within the image. The rapid development of deep learning has revolutionized the field of computer vision. Through extensive data training, deep learning models can incorporate high-dimensional features such as complex scene structure and semantics, thereby enhancing the feature extraction algorithm's understanding of diverse scenarios and improving feature robustness. SLAM technologies that incorporate deep learning methods generally fall into two categories. End-to-end approaches take an image as input and process it through a deep learning neural network to directly determine the camera's pose. However, these approaches excel only in specific scenarios similar to the training dataset and have limited generalization to new scenarios. Furthermore, end-to-end training requires a large number of training samples, consuming high computing resources and storage space. Furthermore, due to computing power constraints, real-time pose estimation is not possible. The second approach is a hybrid approach, which replaces certain components of traditional SLAM systems with deep learning modules, thereby balancing the robustness of deep learning in complex environments with the real-time and generalization capabilities of traditional SLAM technology. However, this approach still uses the classic visual SLAM front-end and back-end architecture of "front-end image tracking and back-end map optimization," resulting in a relatively complex overall structure. Furthermore, the information transmission between the front-end and back-end is limited to specific data representations, resulting in a certain amount of data loss.

[0003] In order to solve the above problems, the main methods currently used are visual inertial SLAM and ORB-SLAM3, among which:

[0004] Visual-Inertial SLAM (VI-SLAM) is a SLAM technology that combines vision and an inertial measurement unit (IMU). It uses cameras and inertial sensors to achieve simultaneous localization and map construction. This approach of combining visual and inertial information aims to overcome the limitations of a single sensor and improve the performance and robustness of the SLAM system. VI-SLAM is a core component of many autonomous navigation systems, including robots, self-driving cars, and augmented reality devices.

[0005] ORB-SLAM3, a highly effective visual-inertial SLAM solution, has been widely used and researched. The ORB-SLAM3 system's main architecture consists of three components: feature tracking, local mapping, and loop closure detection and optimization. In the feature tracking module, the system processes continuously received image frames and IMU observation data, computes ORB feature points and descriptors on the images, calculates IMU integrals, and initializes the system using image frames with significant parallax and their corresponding IMU observations. The system then determines the initial IMU bias, restores the map scale, constructs 3D map points, and aligns the IMU coordinate system with the inertial frame. The constructed 3D local map and the keyframes corresponding to these map points are then used to perform bundle adjustment to optimize the pose. Keyframes are then determined based on specific criteria, and the images and corresponding IMU data for keyframes identified as keyframes are passed to the local mapping module. In this module, feature points in adjacent keyframes are triangulated to create new map points, while unnecessary map points and their corresponding keyframes are selectively discarded. At the same time, bundle adjustment is performed on the map points and corresponding keyframes in the local map to optimize the map point positions and keyframe poses as a whole. The closed-loop detection module constrains and corrects the accumulated errors in the SLAM system by establishing the correct spatial topological relationship between the current frame and the historical map, thereby enhancing the accuracy of long-term pose estimation. Specifically, the visual word bag model is used to select keyframes similar to the current keyframe from the database of keyframes corresponding to the map points, and then verify the temporal consistency and spatial consistency. If the verification passes, the affine transformation of the two is calculated, and the transformation is used as a constraint to perform a global bundle adjustment optimization on all keyframe poses, and the pose estimation of each keyframe is corrected as a whole, thereby achieving consistency correction for trajectory drift. In summary, ORB-SLAM3 realizes real-time estimation and update of camera pose and three-dimensional map through the above three modules. These modules work together to enable ORB-SLAM3 to achieve excellent positioning and mapping effects.

[0006] Traditional SLAM systems typically use image geometry-based methods to calculate feature points, such as ORB, SIFT, SURF, and so on. However, these methods often fail in scenarios with weak textures and varying illumination, making it impossible to extract effective and robust feature points. Furthermore, they are sensitive to rotation and scale changes. Furthermore, end-to-end deep learning methods rely on prior knowledge from a large number of training samples to enhance the robustness of SLAM systems in challenging areas. However, there are significant problems with the collection of training samples, the consumption of training resources, and the generalization of the system in different scenarios. Furthermore, these methods lag significantly behind traditional SLAM methods in terms of pose estimation accuracy and real-time performance. Summary of the Invention

[0007] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a simultaneous positioning and mapping method and system based on deep learning.

[0008] The technical solutions of the present invention are as follows:

[0009] In a first aspect, the present invention provides a method for simultaneous positioning and mapping based on deep learning, comprising:

[0010] Use deep learning models to extract image features and obtain feature point probability tensors and descriptor tensors;

[0011] Adaptively filter feature points in the feature point probability tensor based on inter-frame feature relationships and intra-frame feature relationships to generate a filtered feature point set;

[0012] Build a graph neural network model using the filtered feature point set and descriptor tensor;

[0013] According to the output results of the graph neural network, the feature point matching relationship is obtained;

[0014] Binarize the floating-point depth feature descriptor and use the binarized feature descriptor to train the visual word bag;

[0015] Use the trained visual word bag to perform loop closure detection and retrieve similar key frames;

[0016] The retrieved similar keyframes and feature matching results are used for pose optimization and map construction.

[0017] Furthermore, in the above-mentioned simultaneous positioning and mapping method based on deep learning, the deep learning model includes:

[0018] Description head: The description head is used to extract image features and output feature descriptor tensors;

[0019] Keypoint detection head: The keypoint detection head is used to detect feature points in the image and output feature point probability tensors.

[0020] Furthermore, in the above-mentioned deep learning-based simultaneous positioning and mapping method, the deep learning model includes an XFeat model.

[0021] Furthermore, in the above-mentioned simultaneous positioning and mapping method based on deep learning,

[0022] The inter-frame feature relationship includes: screening out feature points whose response values ​​are greater than a threshold;

[0023] The intra-frame feature relationship includes: screening out feature points whose response values ​​and gradients are greater than a threshold.

[0024] Furthermore, in the above-mentioned simultaneous localization and mapping method based on deep learning, the floating-point deep feature descriptor is binarized, and the binarized feature descriptor is used to train the visual word bag, including:

[0025] The feature descriptors detected in the training image set are clustered into k sets using K-means to form the first layer of the dictionary tree. Recursive operations are performed within each set to form a final dictionary tree with a depth of L and k branches. A visual vocabulary is established, and each leaf node in the dictionary tree is regarded as a word. When executing the graph neural network model, bag-of-words vectors and feature vectors are generated online from all feature points of the current image and described using binary descriptors. At the same time, the floating-point descriptors are binary encoded to improve the query speed of visual location recognition.

[0026] Furthermore, in the above-mentioned simultaneous positioning and mapping method based on deep learning, the binary encoding method is determined by the following formula:

[0027]

[0028] Among them, v i Indicates binary encoding.

[0029] Furthermore, in the above-mentioned simultaneous localization and mapping method based on deep learning, pose optimization and map construction using the retrieved similar keyframes and feature matching results include:

[0030] Calculate the relative poses between similar keyframes using feature matching results;

[0031] Optimize the pose using the relative pose and pose transformation output by the closed-loop detection module;

[0032] Map construction is performed using the optimized pose and observation information of similar keyframes;

[0033] The map construction results are used to perform global pose optimization to obtain the final pose estimation and map.

[0034] In a second aspect, the present invention provides a simultaneous positioning and mapping system based on deep learning, comprising:

[0035] A feature tracking module is used to extract image features using a deep learning model to obtain a feature point probability tensor and a descriptor tensor, and adaptively filter feature points in the feature point probability tensor based on inter-frame feature relationships and intra-frame feature relationships to generate a filtered feature point set;

[0036] A graph neural network model construction module is used to construct a graph neural network model using the filtered feature point set and descriptor tensor;

[0037] The local map construction module is used to obtain feature point matching relationships based on the output of the graph neural network, binarize the floating-point deep feature descriptors, and use the binarized feature descriptors to train the visual word bag;

[0038] The closed loop detection module is used to perform closed loop detection using the trained visual word bag, retrieve similar key frames, and use the retrieved similar key frames and feature matching results to perform pose optimization and map construction.

[0039] The main advantages of the technical solution of the present invention are as follows:

[0040] The present invention's deep learning-based simultaneous localization and mapping method integrates a deep learning feature point extraction module into the system and uses this feature as the only feature representation in the entire system. A feature matching method based on a graph neural network replaces the feature matching method in the entire SLAM system, improving the robustness and accuracy of traditional methods. Preprocessing the deep feature point descriptors and combining them with training on the corresponding bag-of-visual-words achieves more accurate scene recognition and enables real-time, robust, and accurate pose estimation and mapping for the hybrid SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0042] Figure 1 Schematic diagram of a process for simultaneous positioning and mapping based on deep learning according to an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of image feature extraction using a deep learning model in a deep learning-based simultaneous localization and mapping method provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0046] First, in order to more clearly illustrate the technical solutions provided by the embodiments of the present invention, the technical terms involved in the embodiments of the present invention are further explained below.

[0047] Specifically, the technical terms involved in the present invention include:

[0048] Local feature points: Landmarks or features that are significant and stable in images or sensor data. These feature points are often used to match between different frames to achieve camera positioning and map construction. Feature points have obvious characteristics in the image, such as corners, edges, or textured areas. Feature points should also remain stable under different scenes and observation conditions. Secondly, each local feature point is usually associated with a descriptor, which is a numerical vector that describes the appearance and characteristics of the area around the feature point.

[0049] Feature matching: Feature point matching is a crucial task in computer vision and image processing. It is used to find corresponding feature points in two or more images for applications such as scene reconstruction, object tracking, and image registration. A common approach is to calculate the similarity between all feature points in the two images and find the most similar feature point pairs. Commonly used similarity metrics include Euclidean distance, Hamming distance, and mutual information. The matching process also typically considers the spatial relationships between feature points, such as the distance and angle between adjacent feature points.

[0050] Bag of Visual Words: The Bag of Visual Words (VisualBag of Words, VisualBoW) model is a classic method for computer vision tasks such as image classification, object detection, and image retrieval. This model is based on the concept of the bag of words model in natural language processing and represents local features in an image as a collection of visual words. The advantages of the bag of visual words model include simplicity and ease of understanding, high computational efficiency, and robustness to local transformations and partial occlusions of the image to a certain extent. However, it also has some disadvantages, such as ignoring the spatial information of feature points, failing to capture the structural information of the image, and being sensitive to the number and distribution of local features in the image. Therefore, in practical applications, the bag of visual words model is often used in combination with other methods to overcome its limitations.

[0051] In order to solve the technical problems existing in the prior art, the present invention provides a simultaneous positioning and mapping method and system based on deep learning. The following specifically describes how the present invention effectively integrates deep learning technology into the traditional SLAM system to improve the performance and robustness of traditional SLAM technology in challenging environments such as low light, dynamic lighting, and jitter, while ensuring that the system can run in real time.

[0052] As attached Figure 1As shown, an embodiment of the present invention provides a simultaneous positioning and mapping method based on deep learning, which includes steps S1 to S7:

[0053] Step S1: Use the deep learning model to extract image features and obtain feature point probability tensor and descriptor tensor.

[0054] Specifically, in the present invention, the deep learning model includes: a description head and a key point detection head, wherein:

[0055] The description head is used to extract image features and output feature descriptor tensors; the key point detection head is used to detect feature points in the image and output feature point probability tensors.

[0056] In some optional implementations of this embodiment, the deep learning model includes an XFeat model.

[0057] The network structure of the deep learning model is as follows Figure 2 After obtaining the feature point tensor and descriptor tensor output by the network, in order to improve the robustness of feature extraction, we adopt an adaptive threshold selection strategy to filter out feature points and perform post-processing operations to obtain feature points and their descriptors.

[0058] Step S2: adaptively filtering feature points in the feature point probability tensor according to inter-frame feature relationships and intra-frame feature relationships to generate a filtered feature point set;

[0059] Specifically, the relationship between inter-frame features is used to evaluate the matching quality between the current frame and the adjacent frames. This is because the interval between adjacent frames is small, and the matching quality of the previous frame can roughly reflect the matching quality of the current frame. Let m represent the number of matching features between the current frame and the adjacent frame. If the number of matching features decreases, it indicates that it has entered a weak texture area or there is significant motion. The threshold needs to be lowered to extract more feature points to support subsequent tracking; if the match is sufficient, it indicates that it is in a rich texture area. The threshold needs to be increased to reduce feature extraction to reduce computational overhead. In this way, a threshold calculation formula that is inversely proportional to m can be established to ensure that feature points with response values ​​greater than the threshold are screened out from all inter-frame features;

[0060] The intra-frame feature relationship is used to evaluate the spatial distribution of feature points within a single frame image. Feature points with response values ​​greater than a threshold and gradients greater than a threshold are selected.

[0061] It should be noted that the threshold value and the established threshold value formula are set and selected according to actual needs and are not limited to the specific value in the embodiments of the present invention.

[0062] Step S3: Build a graph neural network model using the filtered feature point set and descriptor tensor;

[0063] Exemplarily, constructing a graph neural network model based on the filtered feature point set and descriptor tensor includes:

[0064] The filtered feature points are used as nodes, and node features are initialized with descriptor tensors (such as SIFT, geometric coordinates, etc.). Edges are constructed based on spatial proximity (KNN) or feature similarity (such as cosine distance threshold), and edge features are assigned (such as relative distance, feature difference). Architectures such as GCN, GAT, or GraphSAGE are selected to aggregate neighborhood information through multi-layer message passing (for example, GAT uses the attention mechanism to dynamically weight the contributions of adjacent nodes), gradually fusing local and global features. Graph pooling (such as DiffPool) or skip connections are introduced to compress redundant nodes and retain key topological structures, alleviating the over-smoothing problem of deep networks. For node-level tasks (such as key point classification), node features are directly output, while graph-level tasks (such as overall prediction) aggregate information through global pooling. Edge-level tasks (such as matching) calculate edge probabilities based on node pair interactions. Backpropagation is combined with task loss functions (such as cross entropy, MSE, etc.), supplemented by edge dropout and random masking of node features to improve the model's robustness to noise and sparse features.

[0065] Step S4: Obtain feature point matching relationships based on the output results of the graph neural network;

[0066] For example, given M and N feature point sets extracted from images Ia and Ib, the point sets are output to a feature matching network model built based on a graph neural network, which can directly generate the matching relationship between M and N feature points end-to-end.

[0067] Step S5: binarize the floating-point depth feature descriptor and use the binarized feature descriptor to train the visual word bag;

[0068] It is known that image binarization is to set the grayscale value of the pixels on the image to 0 or 255, that is, to make the entire image appear to be only black and white.

[0069] Specifically, the floating-point depth feature descriptor is binarized, and the binarized feature descriptor is used to train the visual word bag, including:

[0070] The feature descriptors detected in the training image set are clustered into k sets using K-means to form the first layer of the dictionary tree. Recursive operations are performed within each set to form a final dictionary tree with a depth of L and k branches. A visual vocabulary is established, and each leaf node in the dictionary tree is regarded as a word. When executing the graph neural network model, bag-of-words vectors and feature vectors are generated online from all feature points of the current image and described using binary descriptors. At the same time, the floating-point descriptors are binary encoded to improve the query speed of visual location recognition.

[0071] Specifically, the binary encoding method is determined by the following formula:

[0072]

[0073] Among them, v i Indicates binary encoding.

[0074] Step S6: Use the trained visual word bag to perform loop closure detection and retrieve similar key frames;

[0075] The loop correction thread uses a keyframe database and a bag-of-words trained on XFeat descriptors to retrieve similar keyframes. Binarization of the XFeat descriptors is used to enhance retrieval. Selected keyframes are then feature-matched using LightGlue to verify common view geometry, reducing the likelihood of false matches. Finally, loop fusion and global bundle adjustment are performed to optimize the overall pose.

[0076] Step S7: Use the retrieved similar key frames and feature matching results to perform pose optimization and map construction.

[0077] Specifically, pose optimization and map construction using the retrieved similar keyframes and feature matching results include:

[0078] The relative poses between similar keyframes are calculated using feature matching results. After obtaining the feature matching results, the results are converted to the matching relationship between the current keyframe and the 3D map points. The relative poses are estimated using the PnP (Perspective-n-Point) algorithm and outliers are removed in combination with RANSAC.

[0079] Use the keyframes and their map points in the local map to optimize the local map pose. The keyframe poses in the local map are nodes of the pose graph, and the current frame and the keyframes in other local maps are used as edges of the pose graph. Finally, the pose graph is optimized by minimizing the reprojection error of all edges in the graph.

[0080] Map construction is performed using the optimized pose and observation information of similar keyframes. The same 3D points observed by multiple keyframes are fused using matching relationships. If the matching point pair contains feature points that have not yet been triangulated, new map points are generated through triangulation.

[0081] The map construction results are used to perform global pose optimization, and the poses of all key frames and the reprojection errors of map point positions are jointly optimized to obtain the final pose estimation and map.

[0082] In summary, the synchronous positioning and mapping method based on deep learning of the present invention integrates the deep learning feature point extraction module into the system, and uses the feature as the only feature representation in the entire system, maintains the consistency of feature representation, and avoids the introduction of errors due to inconsistent feature semantics between different modules. Traditional feature matching algorithms (such as brute force matching + RANSAC) are prone to failure in cases of large parallax, occlusion, etc. To this end, the present invention uses a feature matching method based on a graph neural network to replace the feature matching method in the entire SLAM system, improving the feature matching effect for large parallax, photometric transformation, and weak texture areas, thereby improving the robustness and accuracy of pose estimation. In addition, the present invention pre-processes the deep feature point descriptor and combines it with the training of the corresponding visual word bag to achieve a more accurate scene recognition effect and improve the retrieval efficiency for similar key frames. In addition, the present invention also proposes strategies such as adaptive feature quantity screening, word bag based on deep features and adaptive bundle adjustment, which can adaptively adjust the number of extracted feature points according to the complexity of the scene, thereby avoiding the computational redundancy and memory consumption brought by a large number of feature points, thereby realizing real-time, robust and accurate pose estimation and map construction of the hybrid SLAM system.

[0083] In a second aspect, the present invention further discloses a simultaneous positioning and mapping system based on deep learning, comprising:

[0084] Feature tracking module, which is used to extract image features using a deep learning model, obtain feature point probability tensors and descriptor tensors, adaptively filter feature points in the feature point probability tensors based on inter-frame feature relationships and intra-frame feature relationships, and generate a filtered feature point set;

[0085] A graph neural network model construction module is used to construct a graph neural network model using the filtered feature point set and descriptor tensor;

[0086] The local map construction module is used to obtain feature point matching relationships based on the output of the graph neural network, binarize the floating-point deep feature descriptors, and use the binarized feature descriptors to train the visual word bag;

[0087] The closed loop detection module is used to perform closed loop detection using the trained visual word bag, retrieve similar key frames, and use the retrieved similar key frames and feature matching results to perform pose optimization and map construction.

[0088] Specifically, the system of the present invention primarily features four sensor configurations: monocular, monocular inertial, binocular, and binocular inertial. Furthermore, it incorporates the aforementioned modules based on ORB-SLAM3. To integrate deep learning models into the system, the ONNX Runtime deep learning deployment framework was used, combined with XFeat and LightGlue models.

[0089] Specifically, for each incoming image, the system first inputs it into the Xfeat network to obtain the probability tensor and descriptor tensor of the feature points. The system then initializes with two frames and performs coarse tracking on each frame. The system further refines the pose estimate through local map tracking. In the event of tracking failure, the system uses the reference frame for tracking or performs relocalization to regain the pose. It should be noted that in coarse tracking, initialization, reference frame tracking, and relocalization, the system uses LightGlue for feature matching. This feature matching method based on graph neural networks ensures the accuracy and robustness of feature matching, thereby improving the effectiveness of feature tracking.

[0090] The local map thread's primary function is to dynamically construct a local map in real time, consisting of 3D map points and their corresponding keyframes. It uses the local map to perform bundle adjustment (BA) to reduce pose estimation errors and enhance consistency. Using the keyframes output by the tracking thread, the local map thread employs LightGlue-based triangulation and adaptive bundle adjustment to reconstruct accurate map points. It then identifies and removes redundant map points and keyframes.

[0091] In summary, the synchronous positioning and mapping system based on deep learning of the present invention uses deep learning feature points as the only feature representation of the system, and adaptively filters feature values ​​according to the inter-frame feature relationship and the intra-frame feature relationship, thereby ensuring that enough feature points are extracted in challenging areas. Since the feature matching technology based on deep learning has a good matching effect under conditions such as large baseline and photometric changes, it is incorporated into the entire process of the SLAM system, including tracking, local mapping, and closed-loop detection modules. The deep feature descriptors of floating-point numbers are binarized and word bags are trained to achieve faster training and retrieval speeds. At the same time, a closed-loop detection module is constructed using it and the adaptive bundle adjustment method.

[0092] In some optional implementations of this embodiment, the above-mentioned deep learning-based simultaneous positioning and mapping method and system of the present invention can be extended and applied to the following scenarios to achieve real-time, robust and accurate pose estimation and map construction.

[0093] Therefore, in combination with practical applications, the deep learning-based simultaneous positioning and mapping method and system provided by the embodiments of the present invention have at least the following technical effects:

[0094] 1. It can provide critical positioning and environmental perception capabilities for autonomous vehicles. By combining visual and inertial sensors, the system can build a real-time map of the vehicle's surroundings and achieve high-precision positioning and navigation, thereby improving the safety and stability of autonomous driving systems.

[0095] 2. It can provide high-precision positioning and tracking capabilities for augmented reality and virtual reality applications. For example, users can experience more realistic and immersive AR and VR experiences such as indoor navigation and virtual tour guides through mobile phones, head-mounted displays, or other devices.

[0096] 3. It can be applied to drone positioning, navigation, and obstacle avoidance, enabling safer and more efficient drone flight. Furthermore, the system can provide critical environmental perception capabilities for drone delivery, enabling autonomous flight and package delivery in complex environments.

[0097] In summary, the deep learning-based simultaneous positioning and mapping method and system of the present invention can also have important application value in the fields of intelligent transportation, intelligent robots, augmented reality, drones, etc., and will bring more convenience and possibilities to future intelligent life and work.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this document are all referenced to the placement states shown in the accompanying drawings.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A simultaneous positioning and mapping method based on deep learning, characterized in that: include: Use deep learning models to extract image features and obtain feature point probability tensors and descriptor tensors; Adaptively filter feature points in the feature point probability tensor based on inter-frame feature relationships and intra-frame feature relationships to generate a filtered feature point set; Build a graph neural network model using the filtered feature point set and descriptor tensor; According to the output results of the graph neural network, the feature point matching relationship is obtained; Binarize the floating-point depth feature descriptor and use the binarized feature descriptor to train the visual word bag; Use the trained visual word bag to perform loop closure detection and retrieve similar key frames; The retrieved similar keyframes and feature matching results are used for pose optimization and map construction.

2. The method for simultaneous positioning and mapping based on deep learning according to claim 1, characterized in that: The deep learning model includes: Description head: The description head is used to extract image features and output feature descriptor tensors; Keypoint detection head: The keypoint detection head is used to detect feature points in the image and output feature point probability tensors.

3. The method for simultaneous positioning and mapping based on deep learning according to claim 1, characterized in that: The deep learning model includes an XFeat model.

4. The method for simultaneous positioning and mapping based on deep learning according to claim 1, wherein: The inter-frame feature relationship includes: screening out feature points whose response values ​​are greater than a threshold; The intra-frame feature relationship includes: screening out feature points whose response values ​​and gradients are greater than a threshold.

5. The method for simultaneous positioning and mapping based on deep learning according to claim 1, wherein: Binarize the floating-point depth feature descriptor and use the binarized feature descriptor to train the visual word bag, including: The feature descriptors detected in the training image set are clustered into k sets using K-means to form the first layer of the dictionary tree. Recursive operations are performed within each set to form a final dictionary tree with a depth of L and k branches. A visual vocabulary is established, and each leaf node in the dictionary tree is regarded as a word. When executing the graph neural network model, bag-of-words vectors and feature vectors are generated online from all feature points of the current image and described using binary descriptors. At the same time, the floating-point descriptors are binary encoded to improve the query speed of visual location recognition.

6. The method for simultaneous positioning and mapping based on deep learning according to claim 5, characterized in that: The binary encoding method is determined by the following formula: Among them, v i Indicates binary encoding.

7. The method for simultaneous positioning and mapping based on deep learning according to claim 1, characterized in that: Using the retrieved similar keyframes and feature matching results for pose optimization and map construction includes: Calculate the relative poses between similar keyframes using feature matching results; Optimize the pose using the relative pose and pose transformation output by the closed-loop detection module; Map construction is performed using the optimized pose and observation information of similar keyframes; The map construction results are used to perform global pose optimization to obtain the final pose estimation and map.

8. A simultaneous positioning and mapping system based on deep learning, characterized in that: include: A feature tracking module is used to extract image features using a deep learning model to obtain a feature point probability tensor and a descriptor tensor, and adaptively filter feature points in the feature point probability tensor based on inter-frame feature relationships and intra-frame feature relationships to generate a filtered feature point set; A graph neural network model construction module is used to construct a graph neural network model using the filtered feature point set and descriptor tensor; The local map construction module is used to obtain feature point matching relationships based on the output of the graph neural network, binarize the floating-point deep feature descriptors, and use the binarized feature descriptors to train the visual word bag; The closed loop detection module is used to perform closed loop detection using the trained visual word bag, retrieve similar key frames, and use the retrieved similar key frames and feature matching results to perform pose optimization and map construction.

Citation Information

Patent Citations

  • Synchronous positioning and map construction method and apparatus

    CN108765326A

  • Loop detection method and device for visual simultaneous localization and mapping

    CN108805149A

  • Synchronous tracking and mapping method fusing deep learning feature matching

    CN116228816A