Visual SLAM optimization method and system based on lightweight depth features

Abstract

The invention belongs to the field of visual SLAM, and discloses a visual SLAM optimization method and system based on lightweight depth features, and the method comprises the steps: calling an acceleration feature module to extract feature key points, a high-dimensional feature descriptor and a confidence map from an image, screening high-confidence feature points through a feature matching module, and carrying out the recognition of the feature points, and then matching is carried out by combining nearest neighbor search and a multi-layer perceptron to generate a matching pair, then pose estimation and local map tracking are carried out based on the matching pair, and finally a local map area is updated. The method effectively deals with a complex environment, reduces tracking failures, enhances feature extraction capability, improves pose estimation precision, reduces error accumulation, reduces calculation burden, improves real-time performance, reduces error accumulation in an unknown scene or a scene with large appearance change, improves trajectory precision, improves feature matching accuracy in a complex scene, and has good application prospects. And the stability, the consistency and the tracking performance of the SLAM system are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of visual SLAM technology, and in particular relates to a visual SLAM optimization method and system based on lightweight depth features. Background Art

[0002] In the field of visual SLAM, with the continuous advancement of technology, feature-point-based, direct, and deep learning methods have become the main types of traditional visual SLAM. Feature-point-based SLAM calculates and optimizes camera motion through keypoint extraction and matching, offering strong robustness. Direct SLAM uses pixel intensity information to estimate camera motion, performing well in low-texture environments. Deep learning-based SLAM integrates deep networks to enhance scene understanding and representation. These methods play an important role in their respective application scenarios and are driving the continued development of the field.

[0003] However, current visual SLAM systems have many limitations. Methods based on feature points are prone to tracking failure in complex environments such as those with drastic lighting changes, scarce textures, or motion blur. Feature point extraction is difficult in environments with weak textures or single structures, resulting in affected pose estimation accuracy and error accumulation. Systems based on direct methods have poor adaptability to lighting changes and dynamic scenes, heavy computational burdens, and affected real-time performance when processing high-resolution images. Systems based on deep learning rely on large amounts of labeled data for training, have high computational resource requirements, limited real-time performance, and are prone to error accumulation in unknown or appearance-changing scenes, affecting trajectory accuracy. At the same time, existing feature matching methods, such as nearest neighbor search based on Hamming distance, are prone to mismatching in complex scenes, and have low feature point matching rates in environments with multi-path reflections or severe occlusion, which seriously affects the stability, consistency, and tracking performance of the SLAM system. Summary of the Invention

[0004] The present invention provides a visual SLAM optimization method and system based on lightweight deep features. This method uses the deep learning model XFeat (AcceleratedFeatures) to automatically extract features from images, combines them with confidence maps for feature screening, and optimizes matching results through a multi-layer perceptron. It can effectively cope with challenges such as ambient lighting changes, dynamic objects, and texture scarcity, and has strong matching accuracy and robustness. At the same time, the introduction of a loop detection mechanism based on a bag-of-words model helps to improve the consistency and stability of the system in long-term operation. The deep network model is designed as a lightweight structure. While maintaining high accuracy, it significantly reduces the consumption of computing resources, enabling it to better adapt to application scenarios such as resource-constrained embedded devices and drone platforms.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A visual SLAM optimization method based on lightweight depth features, comprising: The accelerated feature module in the pre-built visual map construction model is called to perform feature extraction on the collected image information to obtain feature key points, as well as high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; Call the feature matching module in the visual map construction model to filter out feature points with high confidence among the feature key points based on the confidence map; Based on high-dimensional feature descriptors, the nearest neighbor search and multi-layer perceptron are combined to match the filtered feature points with high confidence to generate feature point matching pairs; Estimating the pose of the current frame based on feature point matching, performing local map tracking based on the pose estimation result of the current frame, and generating local map tracking data; Update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0006] Furthermore, the process of performing local map tracking based on the pose estimation result of the current frame and generating local map tracking data further includes: If it is determined that the current frame meets the key frame insertion conditions, the current frame is added as a key frame; the key frame and the observation information corresponding to the key frame are used to transmit to the local mapping module to perform the map optimization task.

[0007] Furthermore, after updating the local map area based on the local map tracking data, the method further includes: According to the analysis results of the observation redundancy between key frames, unnecessary key frames are deleted.

[0008] Furthermore, the visual SLAM optimization method further includes: Based on the analysis results of the similarity between the current frame and the historical key frames, it is determined whether there is a potential loop closure relationship; if there is a loop closure relationship, graph optimization and global bundle adjustment operations are performed to correct the accumulated drift error.

[0009] Furthermore, the method of calling the accelerated feature module in the pre-built visual map construction model to extract features from the collected image information to obtain feature key points, as well as high-dimensional feature descriptors and confidence maps corresponding to the feature key points, includes: Input the collected image information into the acceleration feature module in the pre-built visual map construction model; Perform preprocessing on the acquired images, including normalization, contrast enhancement, and denoising operations; A deep neural network is used to perform multi-level convolution and pooling operations on the preprocessed image to automatically detect potential points of interest and extract key feature points. For each feature key point, a high-dimensional feature descriptor and confidence map corresponding to the feature key point are generated.

[0010] Furthermore, the calling of the feature matching module in the visual map construction model to select feature points with high confidence among the feature key points according to the confidence map includes: According to the extracted feature key points and confidence map, the feature key points are sorted by confidence to screen out feature points with high confidence.

[0011] Furthermore, the method of matching the filtered feature points with high confidence based on the high-dimensional feature descriptor in combination with the nearest neighbor search and the multi-layer perceptron to generate feature point matching pairs includes: Based on the similarity between the high-dimensional feature descriptors corresponding to the feature points with high confidence, the nearest neighbor search is used to perform preliminary matching of feature points; A multi-layer perceptron is used to optimize the preliminary matching results of feature points to obtain feature point matching pairs.

[0012] Furthermore, the pose estimation of the current frame based on the feature point matching includes: Based on the feature point matching pairs, local bundle adjustment is used to optimize the feature points of the current frame and its adjacent frames to obtain the pose estimation result of the current frame.

[0013] Furthermore, after calling the acceleration feature module in the pre-built visual map construction model to extract features from the collected image information, the method further includes: A bag-of-words model is used to convert the feature descriptors of each clustered image into corresponding visual words, and the frequency of occurrence of each visual word is counted to generate a set of fixed-length bag-of-words vectors; wherein the bag-of-words model is pre-trained based on the acceleration feature; The bag-of-words vector of each image is stored in the image database and a corresponding index is created.

[0014] A visual SLAM optimization module based on lightweight depth features, including: A feature extraction module is used to call the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points and the high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; The screening module is used to call the feature matching module in the visual map construction model and screen out feature points with high confidence among the feature key points based on the confidence map; The matching module is used to match the filtered feature points with high confidence based on high-dimensional feature descriptors, combined with nearest neighbor search and multi-layer perceptron to generate feature point matching pairs; The tracking module is used to estimate the pose of the current frame based on the feature point matching pair, perform local map tracking based on the pose estimation result of the current frame, and generate local map tracking data; The optimization module is used to update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a visual SLAM optimization method based on lightweight deep features, calling the acceleration feature module to extract feature key points, high-dimensional feature descriptors and confidence maps from the image, and then filtering high-confidence feature points through the feature matching module, and then combining the nearest neighbor search and multi-layer perceptron to match and generate matching pairs, and then performing pose estimation and local map tracking based on the matching pairs, and finally updating the local map area. The acceleration feature module uses the deep neural network model as a basis to efficiently extract features and generate high-dimensional descriptors and confidence maps. The feature matching module filters feature points according to confidence, and combines the nearest neighbor search and multi-layer perceptron matching to improve the matching accuracy. This method effectively copes with complex environments, reduces tracking failures, enhances feature extraction capabilities, improves pose estimation accuracy, reduces error accumulation, and at the same time reduces computational burden, improves real-time performance, reduces error accumulation in unknown or large appearance change scenes, improves trajectory accuracy, and improves feature matching accuracy in complex scenes, enhancing SLAM system stability, consistency and tracking performance.

[0016] In the present invention, preferably, during the local map tracking process, it is determined whether the current frame meets the key frame insertion conditions. If so, it is added as a key frame and transmitted to the local mapping module to perform the map optimization task; key frames can provide rich information, and reasonable insertion of key frames is helpful for map optimization, improving map quality and accuracy, enriching map information, improving map accuracy and completeness, and enhancing the adaptability of visual SLAM to environmental changes.

[0017] In the present invention, preferably, after updating the local map area based on the local map tracking data, unnecessary key frames are deleted according to the observation redundancy analysis results between key frames; deleting redundant key frames can reduce the computational burden, avoid information duplication, improve map update efficiency, reduce computational complexity, optimize map storage, and improve the real-time performance and efficiency of visual SLAM.

[0018] In the present invention, preferably, based on the similarity analysis results between the current frame and the historical key frames, it is determined whether a loop closure relationship exists, and if so, graph optimization and global bundle adjustment operations are performed; loop closure can correct the accumulated drift error, and graph optimization and global bundle adjustment can globally adjust the pose and map to improve accuracy; effectively eliminate drift errors and improve the long-term stability and accuracy of visual SLAM.

[0019] In the present invention, preferably, the collected image information is input into the acceleration feature module, and after preprocessing, a deep neural network is used to perform multi-level convolution and pooling operations to extract feature key points and generate high-dimensional feature descriptors and confidence maps; preprocessing enhances image quality, and the deep neural network automatically detects points of interest and generates descriptors and confidence maps for subsequent processing; it can improve the accuracy and efficiency of feature extraction and provide a reliable basis for subsequent feature matching and pose estimation.

[0020] In the present invention, preferably, the confidence of the feature key points is sorted according to the extracted feature key points and the confidence map, and feature points with high confidence are screened; high-confidence feature points are more reliable, and after screening, the matching quality and pose estimation accuracy can be improved; it can reduce mismatching and improve the stability and accuracy of visual SLAM.

[0021] In the present invention, preferably, based on the similarity between high-dimensional feature descriptors, the nearest neighbor search is used to perform preliminary matching of feature points, and the multi-layer perceptron is used to optimize the preliminary matching results; the nearest neighbor search quickly finds similar feature points, and the multi-layer perceptron further optimizes the matching results to improve matching accuracy; the accuracy and efficiency of feature point matching are improved, and the tracking performance of visual SLAM is enhanced.

[0022] In the present invention, preferably, based on the feature point matching pairs, local cluster adjustment is used to optimize the feature points of the current frame and its adjacent frames to obtain the pose estimation result of the current frame; local cluster adjustment comprehensively considers the information of multiple frames, optimizes the position of feature points, and improves the accuracy of pose estimation; improves the accuracy of pose estimation and enhances the positioning accuracy of visual SLAM.

[0023] In the present invention, preferably, after calling the accelerated feature module to extract features, the word bag model is used to convert the feature descriptor into visual words, the frequency of occurrence is counted to generate word bag vectors, which are stored in the image database and indexed; the word bag model converts image features into fixed-length vectors, which facilitates image retrieval and matching, and improves the loop detection capability of visual SLAM; accelerates image retrieval speed, improves the accuracy and efficiency of loop detection, and enhances the long-term stability of visual SLAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A flowchart of a visual SLAM optimization method based on lightweight depth features provided by an embodiment of the present invention; Figure 2 A flowchart of a visual SLAM optimization method based on lightweight depth features provided by the present invention; Figure 3 This is a structural diagram of a visual SLAM optimization system based on lightweight depth features provided by the present invention. DETAILED DESCRIPTION

[0025] Current visual SLAM systems still have limitations in many aspects. Feature-based SLAM methods (such as ORB-SLAM) rely on the extraction and matching of image feature points. However, in complex environments (such as those with drastic lighting changes, scarce textures, or motion blur), these methods are prone to tracking failures, affecting the accuracy of pose estimation. Furthermore, feature point extraction is difficult in environments with weak textures or simple structures (such as walls and marble floors), leading to tracking failures and error accumulation. Direct SLAM systems (such as LSD-SLAM and DSO) use image pixel intensity information to directly estimate motion. This method performs well in low-texture or featureless environments, but has poor adaptability to lighting changes and dynamic scenes, which can affect system stability. Furthermore, direct methods are computationally heavy, which can lead to reduced computational efficiency when processing high-resolution images, affecting the real-time performance of the system.

[0026] Deep learning-based SLAM systems attempt to enhance system robustness by learning the environment's motion relationships and structural information through deep neural networks. While this approach can handle dynamic object interference in complex scenes, its reliance on large amounts of labeled data for training requires high computational resources, limiting its real-time performance. Furthermore, deep learning methods are prone to error accumulation when dealing with unknown or highly variable scenes, compromising trajectory accuracy.

[0027] Furthermore, feature matching accuracy is a key factor affecting the stability of SLAM systems. Existing matching methods, particularly nearest neighbor search based on Hamming distance, are prone to mismatching in scenes with varying lighting, repetitive textures, or dynamics. This leads to inaccurate pose estimation, impacting system stability and consistency. In environments with multipath reflections or severe occlusion, the matching rate of feature points is particularly low, which in turn affects the system's tracking performance.

[0028] In order to solve the above problems, Figure 2 As shown, this embodiment provides a visual SLAM optimization method based on lightweight depth features, including: The accelerated feature module in the pre-built visual map construction model is called to perform feature extraction on the collected image information to obtain feature key points, as well as high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; Call the feature matching module in the visual map construction model to filter out feature points with high confidence among the feature key points based on the confidence map; Based on high-dimensional feature descriptors, the nearest neighbor search and multi-layer perceptron are combined to match the filtered feature points with high confidence to generate feature point matching pairs; Estimating the pose of the current frame based on feature point matching, performing local map tracking based on the pose estimation result of the current frame, and generating local map tracking data; Update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0029] The following is an explanation of the technical terms involved in this embodiment: SLAM: The abbreviation of Simultaneous Localization and Mapping, which means real-time positioning and mapping. It is a technology that allows robots to simultaneously locate themselves and build maps in unknown environments. It is widely used in robot navigation, autonomous driving and other fields.

[0030] XFeat: The abbreviation of Accelerated Features, which usually refers to a type of feature-related technology designed to improve feature extraction and processing efficiency in fields such as computer vision.

[0031] BoW: Abbreviation for BagofWords, the bag-of-words model. In computer vision and natural language processing, it treats an image or text as a collection of unordered "words" (features or vocabulary), ignoring information such as the order between words, to represent the characteristics of the image or text.

[0032] CNN: Abbreviation for Convolutional Neural Network. It is a deep learning neural network designed specifically for processing grid-structured data (such as images). It automatically extracts data features through structures such as convolutional layers, pooling layers, and fully connected layers. It is widely used in fields such as image recognition and object detection.

[0033] NNS: Short for Nearest Neighbor Search. It's an algorithm for finding the data points in a dataset that are most similar (closest) to a given query. It has many applications in pattern recognition, cluster analysis, and more.

[0034] MLP: Multi-Layer Perceptron. It is a feedforward neural network consisting of an input layer, multiple hidden layers, and an output layer. It can learn complex nonlinear mapping relationships and can be used for tasks such as classification and regression.

[0035] BA: Bundle Adjustment. In computer vision, it is an optimization method used to simultaneously optimize parameters such as the camera pose and the positions of 3D points in the scene to minimize reprojection errors and improve the accuracy of 3D reconstruction and SLAM systems.

[0036] The optimization method provided by this embodiment is further described below with reference to the accompanying drawings: like Figure 1 As shown in the figure, this embodiment provides a visual SLAM optimization method based on lightweight depth features. In this method, a lightweight deep learning model is adopted and integrated into the visual SLAM system to achieve efficient and stable visual SLAM functions in resource-constrained environments. The specific implementation steps are as follows: XFeat Network Feature Extraction: The XFeat model's feature extraction process involves a series of precise steps designed to effectively extract highly stable and discriminative feature points from the input image. First, the model preprocesses the input data, including image normalization, enhancement, and denoising, to remove noise and improve data consistency, providing more reliable input for subsequent feature extraction. Next, XFeat automatically detects potential points of interest in the image using a deep convolutional neural network (CNN). This process, through multi-layer feature learning, accurately identifies structural and texture features in the image, extracting salient keypoints that are highly stable and discriminative across diverse environments. During training, the XFeat model uses supervised learning with a large amount of labeled image data, optimizing network parameters using a loss function to achieve optimal feature extraction performance in various scenarios. During training, the model uses a backpropagation algorithm to adjust the weights of the convolutional neural network layer by layer, learning to effectively extract stable and discriminative feature points. Through continuous training on the dataset, XFeat adaptively optimizes its feature extraction strategy to cope with varying viewpoints, scales, and illumination variations. For each detected key point, XFeat generates a high-dimensional feature descriptor as a visual feature representation of the local area. This descriptor can effectively capture the details of the local area of the image and maintain high consistency under different viewing angles, scales, and lighting conditions, ensuring high accuracy in the subsequent matching process. In addition, XFeat also generates a confidence map to evaluate the reliability of each feature point. After feature extraction, the confidence map can be used as a screening mechanism to prioritize stable and reliable feature points, thereby reducing the occurrence of mismatches and improving the overall accuracy and robustness of the system. As a basic component of the SLAM system, this module provides support for common feature extraction tasks and optimizes the feature matching process to ensure the stability and compatibility of basic SLAM functions.

[0037] Feature matching module: After feature extraction, XFeat generates key points, descriptors, and confidence maps. The role of the confidence map is to evaluate the reliability of each feature point and screen based on the evaluation results. In actual applications, images may be interfered with by factors such as noise, lighting changes, and dynamic backgrounds. These factors may cause some low-confidence feature points to be incorrectly extracted. By using the confidence map, XFeat can eliminate these unreliable feature points and reduce the probability of false matching. Specifically, the system sorts each feature point according to the confidence value and selects feature points with high confidence to enter the subsequent matching process. In this way, by introducing the confidence map, the system can ensure that the matched feature points have high stability and recognition in complex environments, thereby improving the accuracy and robustness of the matching process. Higher confidence values represent more reliable feature points, thereby avoiding the interference of low-quality feature points on the matching results.

[0038] After completing the confidence screening, XFeat uses Nearest Neighbor Search (NNS) for preliminary matching. The core idea of NNS is to find the most matching feature point pairs by calculating the similarity between feature point descriptors (typically using Euclidean distance or Hamming distance). Nearest neighbor search is an efficient and widely used matching method that can quickly identify potential matching pairs among a large number of feature points. However, preliminary NNS matching results are not always perfect. In particular, NNS can produce inaccurate matching results when the image undergoes significant transformations (such as viewpoint changes or scale changes) or when the environment is complex. While NNS can provide a rough match, it cannot resolve subtle differences and therefore requires further optimization. Specifically, the system ranks each feature point based on its confidence value and selects those with high confidence for subsequent matching. By introducing a confidence map, the system ensures that matched feature points are highly stable and recognizable even in complex environments, thereby improving the accuracy and robustness of the matching process. Higher confidence values represent more reliable feature points, thus preventing low-quality feature points from interfering with the matching results. To overcome the errors introduced by NNS initial matching, XFeat introduces a multi-layer perceptron (MLP) for matching optimization. MLP learns the spatial relationships and semantic features between feature points and precisely adjusts the matching offset, thereby improving matching accuracy and robustness. MLP optimization ensures that the system maintains strong stability even in dynamic environments.

[0039] Pose estimation and optimization: After completing feature matching, the system uses the matching point pairs generated by XFeat to estimate the camera pose. XFeat has high feature point matching accuracy and effectively reduces the interference of false matching on pose estimation by screening reliable feature points through confidence maps. This makes the estimation of camera pose more accurate, especially in complex environments or dynamic scenes, and can provide more stable and accurate estimation results. The accuracy of pose estimation is one of the core performance of SLAM systems. XFeat provides the system with reliable pose information through its precise feature point matching and high-confidence screening mechanism, thereby improving the accuracy of overall navigation.

[0040] To further improve the accuracy of pose estimation, the system uses local bundle adjustment (BA) for short-term optimization. Local BA optimizes the feature points of the current frame and its adjacent frames to minimize the estimation error, thereby improving the accuracy of the current frame's pose. The local BA optimization process not only reduces errors caused by rapid motion, occlusion, or lighting changes, but also effectively improves robustness in dynamic scenes. In scenes with rapid motion or sudden changes, the stability and accuracy of pose estimation are particularly important. Local BA can reduce error accumulation by adjusting the camera pose and map point positions, ensuring that the system continues to operate stably for a short period of time.

[0041] In order to cope with the accumulated errors during long-term operation, the system further optimizes the pose of the key frames through global BA during the loop detection phase. The loop detection system determines whether it has returned to an area that has been passed before. By identifying loops, the system can reduce the accumulation of errors during long-term operation and improve the stability and accuracy of the SLAM system during long-term operation. During the loop detection phase, the system can quickly match the current frame with the historical frame by searching the feature word bag to identify the loop. By comparing the features of the historical frames, the system can determine whether there is an area that has been passed before and further confirm the loop. When a loop is detected, global BA will optimize all key frames, adjust the pose and map points to minimize the global error.

[0042] The global BA optimization process involves all keyframes and their associated feature points, typically requiring high computational resources, but it significantly improves the system's accuracy and robustness. Global BA not only improves the global consistency of the map but also effectively reduces deviations caused by accumulated errors, ensuring high pose estimation and map quality over long periods of time. By combining the optimization strategies of local and global BA, XFeat enables the system to quickly adapt to changes in the short term while maintaining high accuracy over the long term.

[0043] It can be seen from this that it is necessary to ensure that the feature points and matching results generated by XFeat can be seamlessly integrated into the SLAM system, and utilize the high reliability of deep learning features to improve the stability and adaptability of the system in complex environments, while keeping the overall architecture unchanged to achieve high-precision, real-time SLAM tasks.

[0044] The method provided in this embodiment is implemented in combination with actual application scenarios. The specific implementation process is as follows: Step 1: After receiving the image input, the system selects a feature extraction method based on the user-defined parameters. If the user selects XFeat as the feature extraction method, the system invokes the XFeat network to process the input image and simultaneously outputs feature points, feature descriptors, and confidence maps. Feature points identify key areas in the image, descriptors encode the local information of feature points, and confidence maps assess the reliability of feature points, helping the subsequent matching process to select stable features. XFeat performs more stably in complex environments such as illumination changes, scale transformations, and viewpoint rotations, providing more robust feature extraction capabilities. If the user selects ORB as the feature extraction method, the system invokes the ORB module to perform feature point detection and descriptor extraction on the input image and pass the extraction results to the subsequent matching module. The ORB feature matching module calculates the similarity of feature descriptors, completes feature point matching, and outputs the matching results to the tracking module.

[0045] Step 2: Based on the feature results extracted by the XFeat network, the program automatically enters the XFeat matching processing stage. First, a region selection operation is performed, analyzing the entire image based on the confidence map, prioritizing areas with dense feature points and high confidence, thereby reducing computational complexity and improving matching accuracy. After completing region selection, the system uses a nearest neighbor search strategy based on the similarity of feature descriptors to find the most matching candidate feature point pair for each feature point. The system then enters the offset prediction phase, inferring the possible motion range of the feature points in the current frame based on the relative pose relationship of the historical frame or the previous frame, thereby limiting the matching search space and improving matching accuracy and efficiency. Finally, the program performs matching refinement on the initially matched feature point pairs to ensure that the matching results meet strict spatial consistency and motion rationality requirements. After matching refinement is completed, high-precision feature matching pairs are output, providing reliable data support for the subsequent tracking module.

[0046] Step 3: After feature matching is complete, whether using ORB features or XFeat features, the system passes the matching results to the tracking module to begin pose estimation and map tracking. The tracking module first estimates the camera pose of the current frame based on the matched feature point pairs and then completes local map tracking based on this estimate to maintain map continuity. During this process, it also determines whether a new keyframe needs to be inserted to better record the camera motion trajectory and feature observation information. If the current frame meets the keyframe insertion conditions, it is added as a keyframe, and the keyframe and its observation information are passed to the local mapping module to continue the map optimization task.

[0047] Step 4: After receiving data from the tracking module, the local mapping module incrementally updates the local map area. First, new map points are created based on the observation relationship of the new keyframe. At the same time, a local BA algorithm is introduced to jointly optimize the camera pose and map point position in the local map to minimize the reprojection error. Next, the system also performs a redundant keyframe removal operation. By analyzing the observation redundancy between keyframes, it deletes no longer necessary keyframes to reduce the computational burden, improve map sparsity, and maintain the compactness and efficiency of the map. The optimization results of the local mapping will be directly fed back to the tracking module to help maintain the stability of subsequent tracking.

[0048] Step 5: While the system is running, the loop detection module continuously monitors the historical keyframe sequence, searching for possible loop closures. Loop detection uses bag-of-words models, global feature descriptions, or other similarity metrics to determine the similarity between the current frame and historical keyframes and identify potential loops. Once a possible loop closure is detected, the system performs graph optimization and global bundle adjustment (BA) operations. Through global joint optimization, it corrects accumulated drift errors to ensure the accuracy and consistency of the overall map. The optimized map and trajectory information are ultimately provided as system output to users or other modules, laying a high-precision foundation for subsequent positioning, navigation, and other tasks.

[0049] In summary, the process designed in this method fully leverages the advantages of two different feature extraction methods: ORB features and XFeat features. It dynamically switches processing paths based on the needs of different scenarios, retaining the efficiency and lightweight computational characteristics of traditional features while fully leveraging the stability and robustness of deep features in complex environments. At the same time, through strategies such as match refinement, offset prediction, and region selection, the accuracy of feature matching is greatly improved, providing reliable support for subsequent tracking, mapping, and loop optimization. The entire process has a clear structure, rigorous logic, and strong adaptability, capable of meeting the needs of high-precision SLAM mapping and positioning in a variety of application scenarios.

[0050] To further improve system performance, a bag-of-words model based on XFeat features must be pre-trained using the DBoW3 library. First, image data from a variety of indoor and outdoor scenes is prepared to ensure data diversity and complexity. The images cover different viewpoints, lighting conditions, and environments to enhance the generalization capabilities of the trained bag-of-words model. Then, the XFeat algorithm is used to extract local feature descriptors from the images. These descriptors accurately capture image details and generate high-dimensional feature vectors, providing the necessary data support for subsequent visual dictionary training.

[0051] After feature extraction, all image feature descriptors are merged into a large feature set, and these features are clustered using the K-Means clustering algorithm. During this process, the number of clusters, K, is set to determine the size of the dictionary. Each cluster center represents a visual word, and the image features are mapped to a limited visual dictionary to ensure the discreteness and operability of the features. Subsequently, the trained visual dictionary is used to convert each image's feature descriptor into its corresponding visual word. The frequency of each visual word is counted to generate a fixed-length bag-of-words vector.

[0052] The system stores the bag-of-words vector for each image in the map data and creates corresponding indexes to improve the efficiency of image query and matching. The establishment of an image database enables rapid retrieval of historical images during the operation of the SLAM system, as well as loop detection and pose correction. For each new input image, the system uses a trained bag-of-words model to convert its features into a bag-of-words vector and performs similarity matching with historical images in the database. By calculating the distance between BoW vectors (such as Euclidean distance or cosine similarity), the system can identify the historical image most similar to the current image, thereby determining whether loop closure exists and performing corresponding optimization processing.

[0053] Through the above steps, the XFeat bag-of-words model can not only efficiently extract key information from images, but also convert this information into visual features that can be used for matching and optimization, thereby providing strong support for loop detection, trajectory optimization, and map construction in the SLAM system, ensuring the high accuracy and robustness of the system.

[0054] In summary, this embodiment provides a visual SLAM optimization method based on lightweight depth features, which has the following advantages: This method replaces the feature extraction and matching modules of ORB-SLAM3 and introduces XFeat features, which improves the system's adaptability in complex environments and solves the problem that traditional SLAM solutions are prone to failure in situations such as lighting changes, dynamic scenes, and low-texture environments. Compared with traditional methods, this method combines the advantages of deep learning, uses a lightweight XFeat model for feature extraction, and optimizes the matching strategy, enabling the system to maintain high precision and high robustness in various complex scenarios. Compared with existing technical solutions, this method has the following advantages: First, improved robustness: XFeat features are extracted using a deep learning network. Compared to traditional, manually designed descriptors such as ORB, SIFT, and SURF, they are more adaptable to complex environments such as changing lighting, insufficient texture, and motion blur. For example, in scenes switching between indoor and outdoor environments, and in bright or dim light, XFeat leverages its deep feature learning capabilities to extract more robust feature points, reducing tracking failure rates. Furthermore, the system uses confidence maps to screen reliable feature points and introduces an MLP for matching optimization, significantly reducing the false match rate and improving the accuracy and stability of feature matching. To further enhance the stability and efficiency of feature matching, we trained a bag-of-words model based on XFeat features. The bag-of-words model converts local feature descriptors in the image into visual words by performing K-means clustering on the XFeat features, making feature matching more efficient. During the matching process, the XFeat bag-of-words model reduces the computational overhead of descriptor comparison while improving matching robustness. This approach is particularly suitable for dynamic environments and low-texture scenes, effectively reducing false matches and improving the stability and reliability of the system in complex scenes.

[0055] Second, optimizing computational overhead: Traditional ORB-SLAM3 imposes a heavy computational burden on high-resolution images or high-frame-rate video streams, making it difficult to meet real-time requirements, especially on embedded platforms or low-power devices. This method uses a lightweight XFeat network combined with a bag-of-words model to optimize the feature matching process. Clustering reduces the matching range and computational complexity, enabling the system to operate efficiently on resource-constrained devices (such as mobile robots, drones, and AR devices). Furthermore, by introducing a confidence screening mechanism, only high-confidence feature points are calculated, further reducing computational redundancy, thereby reducing computing resource usage and improving the real-time performance of the SLAM system.

[0056] Third, enhanced scalability: This method adopts a modular design, so that key components such as feature extraction, matching optimization, and pose estimation can be independently replaced or upgraded, making it easy to adapt to different SLAM tasks. For example, in specific industrial applications (such as autonomous driving and intelligent security), feature extraction methods (such as XFeat or ORB) can be switched according to needs to adapt to different computing power environments. In addition, XFeat is modularly encapsulated through the PyTorch C++ API, allowing deep learning features to be integrated without major modifications to the system architecture, improving the flexibility and maintainability of the SLAM system.

[0057] Therefore, it can be seen that this method has the following promotion and application value: Improving system stability and adaptability: This method improves system robustness in complex environments by optimizing ORB-SLAM3 with XFeat features, reducing tracking failures and improving positioning and mapping accuracy. Traditional SLAM solutions are susceptible to failure in conditions such as changing lighting and motion blur. This technology improves the adaptability of visual SLAM in a variety of scenarios, making it more suitable for applications such as drones, robotics, and autonomous driving.

[0058] Reduced Computing Resource Consumption: Optimized feature extraction and matching modules reduce computational overhead, enabling SLAM systems to run on resource-constrained devices (such as embedded systems or mobile robots), helping to reduce hardware costs and improve system energy efficiency. Compared to traditional SLAM solutions, this method reduces reliance on high-performance computing devices while maintaining accuracy, allowing low-power devices to efficiently complete SLAM tasks.

[0059] Improved application versatility: This method utilizes a modular design, making it easier to adapt and scale to different SLAM tasks. It can be flexibly applied to various scenarios, including robotic navigation, augmented reality (AR), mapping, and drone control. By enhancing the versatility of SLAM systems, this technical solution can be applied to a wider range of commercial applications, increasing market competitiveness.

[0060] In summary, this method has significant commercial value in the field of visual SLAM. It not only improves the stability and adaptability of the system, but also reduces computing costs. At the same time, it enhances the flexibility of the technical solution, enabling it to meet the needs of more industries and improve overall economic benefits and market competitiveness.

[0061] like Figure 3As shown, this embodiment also provides a visual SLAM optimization system based on lightweight deep features, including: a feature extraction module, used to call the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points, and high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; a screening module, used to call the feature matching module in the visual map construction model, and screen out feature points with high confidence among the feature key points according to the confidence map; a matching module, used to match the filtered feature points with high confidence based on the high-dimensional feature descriptor, combined with the nearest neighbor search and the multi-layer perceptron, to generate feature point matching pairs; a tracking module, used to perform pose estimation of the current frame based on the feature point matching pairs, perform local map tracking according to the pose estimation result of the current frame, and generate local map tracking data; an optimization module, used to update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0062] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the visual SLAM optimization method based on lightweight depth features when executing the computer program.

[0063] When the processor executes the computer program, the above-mentioned steps of visual SLAM optimization based on lightweight deep features are implemented, for example: calling the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points, and high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; calling the feature matching module in the visual map construction model to screen out feature points with high confidence among the feature key points according to the confidence map; based on the high-dimensional feature descriptor, combined with the nearest neighbor search and the multi-layer perceptron, the filtered feature points with high confidence are matched to generate feature point matching pairs; based on the feature point matching pairs, the pose of the current frame is estimated, and the local map tracking is performed according to the pose estimation result of the current frame to generate local map tracking data; based on the local map tracking data, the local map area is updated to optimize the visual SLAM task.

[0064] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-mentioned system, for example: a feature extraction module, which is used to call the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points, and high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; a screening module, which is used to call the feature matching module in the visual map construction model to screen out feature points with high confidence among the feature key points according to the confidence map; a matching module, which is used to match the filtered feature points with high confidence based on the high-dimensional feature descriptor, combined with the nearest neighbor search and the multi-layer perceptron, to generate feature point matching pairs; a tracking module, which is used to perform pose estimation of the current frame based on the feature point matching pairs, perform local map tracking according to the pose estimation result of the current frame, and generate local map tracking data; an optimization module, which is used to update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0065] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing preset functions, and the instruction segments are used to describe the execution process of the computer program in the visual SLAM optimization device based on lightweight depth features. For example, the computer program can be divided into a feature extraction module, a screening module, a matching module, a tracking module and an optimization module; the feature extraction module is used to call the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points, and high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; the screening module is used to call the feature matching module in the visual map construction model to screen out feature points with high confidence among the feature key points according to the confidence map; the matching module is used to match the filtered feature points with high confidence based on the high-dimensional feature descriptor, combined with the nearest neighbor search and the multi-layer perceptron to generate feature point matching pairs; the tracking module is used to perform pose estimation of the current frame based on the feature point matching pairs, perform local map tracking according to the pose estimation result of the current frame, and generate local map tracking data; the optimization module is used to update the local map area based on the local map tracking data to optimize the visual SLAM task.

[0066] The visual SLAM optimization device based on lightweight depth features can be a computing device such as a desktop computer, a notebook, a PDA and a cloud server. The visual SLAM optimization device based on lightweight depth features can include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above is an example of a visual SLAM optimization device based on lightweight depth features, and does not constitute a limitation on the visual SLAM optimization device based on lightweight depth features. It can include more components than the above, or combine certain components, or different components. For example, the visual SLAM optimization device based on lightweight depth features can also include input and output devices, network access devices, buses, etc.

[0067] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor serves as the control center of the lightweight depth feature-based visual SLAM optimization, and utilizes various interfaces and lines to connect various parts of the entire lightweight depth feature-based visual SLAM optimization device.

[0068] The memory can be used to store the computer program and / or module, and the processor implements various functions of the visual SLAM optimization device based on lightweight depth features by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0069] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0070] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the visual SLAM optimization method based on lightweight depth features are implemented.

[0071] If the module / unit integrated in the lightweight depth feature-based visual SLAM optimization system is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0072] Based on this understanding, the present invention implements all or part of the processes in the above-mentioned visual SLAM optimization method based on lightweight depth features, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned visual SLAM optimization method based on lightweight depth features. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0073] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0074] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0075] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A visual SLAM optimization method based on lightweight depth features, characterized in that: include: The accelerated feature module in the pre-built visual map construction model is called to perform feature extraction on the collected image information to obtain feature key points, as well as high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; Call the feature matching module in the visual map construction model to filter out feature points with high confidence among the feature key points based on the confidence map; Based on high-dimensional feature descriptors, the nearest neighbor search and multi-layer perceptron are combined to match the filtered feature points with high confidence to generate feature point matching pairs; Estimating the pose of the current frame based on feature point matching, performing local map tracking based on the pose estimation result of the current frame, and generating local map tracking data; Update the local map area based on the local map tracking data to optimize the visual SLAM task.

2. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein The process of performing local map tracking based on the pose estimation result of the current frame and generating local map tracking data also includes: If it is determined that the current frame meets the key frame insertion conditions, the current frame is added as a key frame; the key frame and the observation information corresponding to the key frame are used to transmit to the local mapping module to perform the map optimization task.

3. The visual SLAM optimization method based on lightweight depth features according to claim 2, wherein After the local map area is updated based on the local map tracking data, the method further includes: According to the analysis results of the observation redundancy between key frames, unnecessary key frames are deleted.

4. The visual SLAM optimization method based on lightweight depth features according to claim 2, wherein The visual SLAM optimization method further comprises: Based on the analysis results of the similarity between the current frame and the historical key frames, it is determined whether there is a potential loop closure relationship; if there is a loop closure relationship, graph optimization and global bundle adjustment operations are performed to correct the accumulated drift error.

5. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein The method of calling the acceleration feature module in the pre-built visual map construction model to extract features from the collected image information to obtain feature key points, and high-dimensional feature descriptors and confidence maps corresponding to the feature key points, including: Input the collected image information into the acceleration feature module in the pre-built visual map construction model; Perform preprocessing on the acquired images, including normalization, contrast enhancement, and denoising operations; A deep neural network is used to perform multi-level convolution and pooling operations on the preprocessed image to automatically detect potential points of interest and extract key feature points. For each feature key point, a high-dimensional feature descriptor and confidence map corresponding to the feature key point are generated.

6. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein The calling of the feature matching module in the visual map construction model to select feature points with high confidence among the feature key points according to the confidence map includes: According to the extracted feature key points and confidence map, the feature key points are sorted by confidence to screen out feature points with high confidence.

7. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein The method of matching the filtered feature points with high confidence based on the high-dimensional feature descriptor in combination with the nearest neighbor search and the multi-layer perceptron to generate feature point matching pairs includes: Based on the similarity between the high-dimensional feature descriptors corresponding to the feature points with high confidence, the nearest neighbor search is used to perform preliminary matching of feature points; A multi-layer perceptron is used to optimize the preliminary matching results of feature points to obtain feature point matching pairs.

8. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein The pose estimation of the current frame based on the feature point matching includes: Based on the feature point matching pairs, local bundle adjustment is used to optimize the feature points of the current frame and its adjacent frames to obtain the pose estimation result of the current frame.

9. The visual SLAM optimization method based on lightweight depth features according to claim 1, wherein After calling the acceleration feature module in the pre-built visual map construction model to extract features from the collected image information, the method further includes: A bag-of-words model is used to convert the feature descriptors of each clustered image into corresponding visual words, and the frequency of occurrence of each visual word is counted to generate a set of fixed-length bag-of-words vectors; wherein the bag-of-words model is pre-trained based on the acceleration feature; The bag-of-words vector of each image is stored in the image database and a corresponding index is created.

10. A visual SLAM optimization module based on lightweight depth features, characterized in that: include: A feature extraction module is used to call the acceleration feature module in the pre-built visual map construction model to perform feature extraction on the collected image information to obtain feature key points and the high-dimensional feature descriptors and confidence maps corresponding to the feature key points; the high-dimensional feature descriptors are used to characterize the visual features of the local area; wherein the basic model of the visual map construction model adopts a deep neural network model; The screening module is used to call the feature matching module in the visual map construction model and screen out feature points with high confidence among the feature key points based on the confidence map; The matching module is used to match the filtered feature points with high confidence based on high-dimensional feature descriptors, combined with nearest neighbor search and multi-layer perceptron to generate feature point matching pairs; The tracking module is used to estimate the pose of the current frame based on the feature point matching pair, perform local map tracking based on the pose estimation result of the current frame, and generate local map tracking data; The optimization module is used to update the local map area based on the local map tracking data to optimize the visual SLAM task.

Images