Visual slam system with adaptive key frame selection function and construction method thereof

Abstract

The application provides a visual SLAM system with self-adaptive key frame selection function and a construction method thereof, compared with an existing visual SLAM system, the system adds a matchability prediction branch on the basis of a three-dimensional reconstruction main network, and realizes active quantitative evaluation of cross-view matching reliability at a pixel level. On this basis, a scene quality self-adaptive key frame insertion mechanism is constructed, a screening threshold is dynamically adjusted according to real-time frame-level matchable quality, and the tracking stability of a weak texture scene and the operation efficiency of a high-quality scene are considered. Meanwhile, a matchability perception weighted bundle adjustment strategy (MMBA) is designed, different weights are applied to matching pixels according to bidirectional matching quality, and the interference of low-reliability noise matching on the optimization result is inhibited. The application solves the defects of poor adaptability of a traditional fixed threshold key frame selection strategy and uniformity of optimization weights without distinction, and effectively improves the camera pose estimation accuracy and dense three-dimensional map reconstruction quality in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to the field of visual SLAM, and more specifically, to a visual SLAM system with adaptive keyframe selection function and a method for constructing the same. Background Technology

[0002] Visual SLAM is a core technology that uses camera image sequences to estimate the six-degree-of-freedom pose of the camera (including three-axis translation and three-axis rotation) in real time and simultaneously construct a 3D map of the environment. Modern visual SLAM systems typically consist of two core components: a front-end tracking module and a back-end optimization module. The front-end tracking module is responsible for estimating the camera motion increment frame by frame; the back-end optimization module uses global bundle adjustment (BA) to jointly optimize the pose and 3D point cloud of all keyframes, eliminating cumulative drift.

[0003] Keyframe selection strategy plays a crucial role in bridging front-end tracking and back-end optimization. Keyframes are representative frames carefully selected from continuous high-frame-rate video streams. They serve as constraint nodes for BA optimization, and their selection quality directly determines the positioning accuracy, operational robustness, and computational resource utilization efficiency of the SLAM system.

[0004] To select efficient and high-quality keyframes for camera pose estimation and 3D map reconstruction, academia and industry have conducted extensive research on keyframe selection strategies. Currently, three main technical approaches have emerged: Approach 1, represented by the ORB-SLAM series, uses manual feature matching-driven methods, relying on fixed thresholds based on multiple conditions such as the number of matching feature points, the proportion of tracking map points, and time intervals to determine keyframes; Approach 2, represented by DROID-SLAM, uses deep learning end-to-end methods, relying on dense optical flow and inter-frame co-location to complete tracking and basic keyframe management, using a fixed optical flow displacement threshold for insertion judgment; Approach 3, represented by MASt3R-SLAM, uses dense SLAM methods based on 3D reconstruction priors, leveraging the Transformer backbone to output dense 3D point maps and matching features to achieve tracking and reconstruction, while still using a fixed matching rate threshold for keyframe selection. Despite continuous iterative optimizations in feature representation and geometric modeling, the three types of methods generally suffer from structural limitations: the keyframe insertion threshold remains fixed throughout the process, lacking direct perception of pixel-level cross-view matching quality; some adaptive improvement methods only adjust the threshold based on external geometric / sensor signals such as disparity baseline and IMU inertial information, without fundamentally utilizing the inherent matching reliability information of the scene texture itself. In challenging scenarios such as low-texture walls, repetitive textures, and dramatic lighting changes, keyframe distribution imbalances are prone to occur, i.e., high-quality texture areas generate a large number of redundant keyframes that drag down real-time performance, while weak texture areas suffer from insufficient keyframe insertion and lack of geometric constraints, causing camera pose drift or even tracking failure.

[0005] In summary, existing visual SLAM systems cannot dynamically and adaptively adjust keyframe selection based on the feature quality of the current scene. This leads to insufficient keyframe insertion and pose constraint degradation in complex, weakly textured scenes, causing localization drift and even tracking failure. Furthermore, redundant keyframes are easily generated in high-quality scenes, resulting in high computational overhead. Additionally, existing systems lack proactive quantification capabilities for pixel-level matching reliability, relying solely on post-matching statistics to evaluate scene quality, resulting in limited evaluation accuracy. Moreover, the bundle adjustment optimization uses equal weights for all matched pixels, failing to suppress the interference of low-reliability noise matching on pose solving, ultimately limiting the localization accuracy, robustness, and mapping quality of visual SLAM systems.

[0006] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0007] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a visual SLAM system with adaptive keyframe selection function and its construction method.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] According to a first aspect of the present invention, a visual SLAM system with adaptive keyframe selection function is provided for constructing a dense point cloud map of a target environment based on an RGB image sequence acquired from the target environment. The system includes: a data preprocessing module for configuring a reference RGB image for each original RGB image in the image sequence to construct multiple RGB image pairs; a visual encoding module for encoding two RGB images in each RGB image pair constructed by the data preprocessing module into feature vectors to obtain multiple feature vector pairs; and a front-end module for performing feature decoding on the feature vectors of each RGB image in each RGB image pair to obtain a matchability score for each RGB image in each RGB image pair. The system collects pixel-level geometric information and predicts effective pixel matching pairs, camera poses, and 3D point maps of each RGB image pair based on the matchability score map and pixel-level geometric information of all RGB images in each RGB image pair. It also determines the keyframe judgment threshold of each RGB image pair based on the matchability score map. If the effective pixel matching pairs of the RGB image pair are greater than the keyframe judgment threshold, the original RGB image in the RGB image pair is selected as a keyframe. The back-end optimization module is used to construct a dense 3D map based on all keyframes selected by the front-end module, the pixel geometric information of each keyframe, the predicted camera pose, and the predicted 3D point map.

[0010] This solution can achieve at least the following beneficial technical effects:

[0011] This invention establishes a complete modular architecture for dense visual SLAM, abandoning the traditional fixed threshold frame selection method and dynamically adjusting the keyframe judgment threshold based on pixel-level matchability scores. It can adaptively adapt to different scene feature qualities, ensuring tracking stability and positioning accuracy in complex scenes while suppressing keyframe redundancy in high-quality scenes, effectively balancing system robustness and computational efficiency, and improving the quality of dense 3D mapping.

[0012] Preferably, the visual encoding unit includes two image encoders with shared weights, which respectively encode the features of the two RGB images in the RGB image pair to obtain two feature tensors.

[0013] This solution can achieve at least the following beneficial technical effects:

[0014] This invention employs a weight-shared dual encoder for feature encoding, unifying the feature space of two frames and effectively reducing cross-frame feature matching errors. Simultaneously, it simplifies network parameters and reduces computational overhead, providing stable and reliable feature support for subsequent scene quality assessment, 3D geometric decoding, and accurate inter-frame matching.

[0015] Preferably, the front-end module includes a matchable prediction head, a 3D decoder, and a front-end tracker. The matchable prediction head and the 3D decoder constitute two parallel feature decoding branches, wherein: the matchable prediction head is used to perform feature decoding on the feature vector of each RGB image in each RGB image pair to obtain a matchability score map for each RGB image in each RGB image pair, wherein the matchability score map of an RGB image contains the matchability score of each pixel in the RGB image, and the matchability score of each pixel indicates the complexity of the environmental information contained in the pixel; the 3D decoder uses a cross-attention mechanism to perform bidirectional information interaction and feature decoding on the feature vector of each RGB image in each RGB image pair to output each... The pixel-level geometric information of each RGB image in an RGB image pair includes the 3D point cloud coordinates, feature descriptor, geometric confidence, and descriptor confidence of each pixel in the RGB image. The front-end tracker is used to determine the effective pixel matching pairs, camera pose, and 3D point map of each RGB image based on the matching prediction head to obtain the matching score map and pixel-level geometric information. It also determines the keyframe judgment threshold of each RGB image pair based on the matching score map. If the effective pixel matching pairs of the RGB image pair are greater than the keyframe judgment threshold, the original RGB image in the RGB image pair is selected as the keyframe.

[0016] This solution can achieve at least the following beneficial technical effects:

[0017] This invention employs a dual-parallel decoding branch structure to simultaneously realize 3D geometric information calculation and pixel-level matchability prediction, completing the pre-quantitative evaluation of scene quality. This overcomes the shortcomings of traditional methods that rely solely on post-matching statistics to evaluate the scene, providing accurate and effective prior data for adaptive keyframe scheduling and weighted pose optimization.

[0018] Preferably, the range of the matchability score for each pixel is: Among them, the higher the matching score, the more complex the environmental information corresponding to that pixel.

[0019] This solution can achieve at least the following beneficial technical effects: The present invention limits the pixel matchability score to the range of 0 to 1, standardizes the pixel matching reliability quantification scale, adapts to subsequent frame-level quality statistics, dynamic threshold adjustment and pixel weight calculation, avoids numerical scale disorder problems, and ensures the stability and reliability of the system's adaptive control and optimization process.

[0020] Preferably, the front-end tracker determines the camera pose of the original RGB images in each image pair as follows: It constructs effective matching pixel pairs for the image pair based on the 3D point cloud coordinates and feature descriptors of each pixel in each RGB image of the image pair; based on the constructed effective matching pixel pairs and the matchability score map of each RGB image in the image pair, it evaluates the original matching quality of each effective matching pixel pair as follows:

[0021]

[0022] in, Indicates the first The original matching quality of each valid matching pixel pair. and Indicates the first Two pixels in a valid matching pixel pair For the pixels in the original RGB image, To reference the pixels in the RGB image, The matching score graph representing the original RGB image Matchability score The matching scores of the reference RGB images are shown in the following order. Matchability score The value is a preset positive number; the original matching quality of each valid matching pixel pair in the image is normalized to obtain the final matching quality of each valid matching pixel pair; the global pose of the current frame is constructed in the following manner, constrained by the final matching quality of all valid matching pixel pairs:

[0023]

[0024]

[0025] in, This represents the predicted camera pose of the original RGB image. This represents the camera pose of the reference RGB image. Indicates the first The geometric residual of a valid matching pixel pair Indicates the preset first The weight of each valid matching pixel pair This represents the weights of basic information determined based on pixel-level geometric information. and Indicates the first The 3D point cloud coordinates of the two pixels in a valid matching pixel pair and Indicates the first Geometric confidence of two pixels in a valid matching pixel pair Represents the identity matrix.

[0026] This solution can achieve at least the following beneficial technical effects:

[0027] This invention combines bidirectional matchability scores to calculate the original matching quality, integrates multi-dimensional pixel geometric confidence to construct differentiated optimization weights, and relies on weighted residual constraints to solve for camera pose. This effectively reduces the interference of low-quality matching noise, strengthens the constraint effect of high-reliability matching, and significantly improves the accuracy and stability of pose estimation in complex scenes.

[0028] Preferably, the normalization process is as follows:

[0029]

[0030] in, Indicates the first The final matching quality of a pair of valid matching pixels. Indicates the first The original matching quality of each valid matching pixel pair. and These represent the original matching quality of all valid matching pixel pairs after sorting the matching quality in ascending order, with preset low and preset high resolutions. Indicates a preset positive number. This represents the normalization function.

[0031] This solution can achieve at least the following beneficial technical effects:

[0032] This invention uses a high-low quantile normalization method to standardize the matching quality, eliminating interference from extreme outlier matching samples and normalizing the matching quality to a valid range of 0 to 1. This achieves fair quantification of matching quality across different scenarios, ensuring the rationality and stability of pixel-weighted optimization, and adapting to pose solving needs in various complex scenarios.

[0033] Preferably, the keyframe judgment threshold is determined in the following way:

[0034]

[0035]

[0036] in, The preset base threshold, As a preset reference quality level, The preset adjustment coefficient, The upper bound is a preset value. This indicates the adjusted judgment threshold. This represents frame-level quality statistics. This indicates the number of pixels. Indicates the RGB image to be evaluated The quality score.

[0037] This solution can achieve at least the following beneficial technical effects:

[0038] This invention dynamically modulates the keyframe selection threshold based on frame-level matchable quality statistics. When scene quality is poor, the threshold is relaxed, and keyframes are inserted promptly to ensure tracking constraints. When scene quality is excellent, the threshold is tightened, and redundant keyframes are suppressed. This achieves full-scene adaptive control of keyframe selection, balancing system tracking robustness with real-time computational efficiency.

[0039] According to a second aspect of the present invention, a method for constructing a visual SLAM system with adaptive keyframe selection function is proposed. The method includes: S1, acquiring a dataset, the dataset including multiple samples, wherein each sample includes a reference keyframe RGB image and an RGB image to be evaluated, as well as a true 3D point map of each RGB image, the true camera pose of each RGB image, and all valid pixel matching pairs between the reference keyframe RGB image and the RGB image to be evaluated; S2, acquiring a visual SLAM system as described in the first aspect of the present invention; S3, based on the dataset acquired in S1, taking the image pair composed of the reference keyframe RGB image and the RGB image to be evaluated as input, and taking the predicted true 3D point map of the RGB image, the true camera pose of each RGB image, and all valid pixel matching pairs between the reference keyframe RGB image and the RGB image to be evaluated as output, and training the visual SLAM system to convergence using a preset loss function, wherein, during the training process, the parameters of the data preprocessing module and the back-end optimization module are frozen, and only the parameters of the visual encoding module and the front-end module are adjusted.

[0040] Preferably, the preset loss function is:

[0041]

[0042] in, This represents the matching ability supervision loss. The loss between the matchability prediction head output matchability map and the true matchability map is denoted as . Indicates reprojection loss. The loss between the pixel coordinates obtained by projecting the 3D point cloud coordinates of a pixel and the pixel's true pixel coordinates. Indicates epipolar constraint loss. Indicates deep loss. Indicates the loss of the 3D point cloud. This represents the loss between the 3D point cloud coordinates output by the 3D decoder and the actual 3D point cloud coordinates. This represents the descriptor matching constraint loss. The loss between the pixel descriptor output by the 3D decoder and the real descriptor. For the matching distribution calibration item, , , , , , and All of these are preset loss weights.

[0043] Compared with the prior art, the advantages of the present invention are as follows:

[0044] This invention addresses the shortcomings of existing visual SLAM systems, such as fixed keyframe selection thresholds, inability to actively assess scene feature quality, and lack of pixel-level differentiated weighting in bundle optimization. It proposes a quality-aware adaptive keyframe scheduling and matchability-weighted dense SLAM scheme. This invention adds a matchability prediction branch to the 3D decoding architecture, enabling proactive quantitative assessment of pixel-level cross-view matching reliability, overcoming the problems of traditional methods relying on post-matching statistics and lagging scene evaluation. Based on frame-level feature quality, this invention dynamically adjusts the keyframe selection threshold, promptly supplementing keyframes in complex scenes such as low-texture and drastic lighting changes to ensure sufficient tracking constraints and avoid pose drift and tracking failure. In high-quality scenes, redundant keyframes are suppressed, reducing backend computational overhead and balancing system robustness and real-time performance. Furthermore, this invention combines bidirectional matchability scores and geometric confidence to construct a pixel-level weighting mechanism, unifying the quality scale through quantile normalization, suppressing low-reliability noise matching interference in bundle optimization, strengthening effective geometric constraints, and improving pose solution accuracy and stability. In summary, this invention effectively overcomes the shortcomings of traditional SLAM, such as poor adaptive capability and limited optimization accuracy, and significantly improves the tracking stability, positioning accuracy, and dense 3D mapping quality of the system in complex scenarios. Attached Figure Description

[0045] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0046] Figure 1 This is a schematic diagram of a visual SLAM system with adaptive keyframe selection function according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0048] As mentioned in the background section, existing visual SLAM systems cannot dynamically and adaptively adjust the selection of keyframes based on the feature quality of the current observation scene. This results in insufficient keyframe insertion and pose constraint degradation in complex, weakly textured scenes, leading to localization drift or even tracking failure. At the same time, redundant keyframes are easily generated in high-quality scenes, resulting in high computational overhead. Meanwhile, existing systems lack the ability to actively quantify pixel-level matching reliability, relying solely on post-matching statistics to assess scene quality, resulting in limited evaluation accuracy. Furthermore, the bundle adjustment optimization applies equal weights to all matched pixels, failing to suppress the interference of low-reliability noise matching on pose solving. Ultimately, this limits the localization accuracy, robustness, and mapping quality of visual SLAM systems. Specifically, because existing visual SLAM systems cannot dynamically and adaptively adjust keyframe selection based on the feature quality of the current observed scene, the following problems arise: Problem 1: In challenging scenes such as low texture, repetitive texture, local blur, or lighting variations, keyframe insertion is often delayed, and front-end constraints are insufficient, leading to decreased pose estimation accuracy and, in severe cases, even tracking failure. Conversely, in scenes with good feature conditions, unnecessary keyframe insertion may occur, increasing the number of back-end optimization nodes and computational burden. This problem arises because existing keyframe management strategies primarily rely on fixed judgment conditions for triggering, failing to distinguish between situations where "the current scene quality has deteriorated, making keyframe insertion necessary" and "whether the current keyframes are sufficient to support stable tracking." Therefore, it is difficult to simultaneously consider system robustness and computational efficiency. Problem 2: Most existing methods, after completing feature matching or tracking, decide whether to insert keyframes based on the number of matches, the matching ratio, or the tracking results. They lack the ability to proactively quantify and evaluate the quality of current scene features. This is because matching statistics primarily reflect the matching results themselves and are difficult to directly characterize the true matchability and geometric consistency of pixels or regions under cross-viewpoint conditions. Therefore, in scenes with medium texture, repetitive texture, local blur, or varying illumination, they struggle to accurately reflect the current scene quality. Problem 3: During the beamforming (BA) optimization stage, existing methods apply uniform weights to all matched pixels or only consider reprojection residuals, failing to explicitly distinguish the reliability of pixel-level features. This is problematic because matching in areas with weak texture, motion blur, or reflected highlights inherently carries high noise, yet these areas receive equal weight in optimization compared to observations in high-reliability regions. This noise propagates directly to the pose solution, weakening the accuracy and stability of the front-end tracking.

[0049] After in-depth research into existing visual SLAM systems, the inventors discovered four structural limitations in keyframe selection strategies: First, the threshold mechanism is decoupled from scene quality. Whether it's the classic method based on handcrafted features (ORB-SLAM, ORB-SLAM3), the end-to-end method based on deep learning (DROID-SLAM), or the state-of-the-art method based on 3D reconstruction priors (MASt3R-SLAM), the keyframe insertion decision remains constant during system operation and does not dynamically adjust with changes in scene feature quality. Although Reference 7 (PKS) introduces the concept of adaptive thresholding, its adaptive signal source is geometric angles and IMU data, rather than the quality information of the scene features themselves. This means that the keyframe selection strategies of all existing methods are decoupled from scene feature quality. Second, there is a lack of proactive evaluation capability for scene feature quality. Existing visual SLAM systems all rely on statistical results after matching (such as the number of matches, matching ratio, or optical flow displacement) to passively trigger keyframe insertion. These statistics only reflect "how many matches occurred" and cannot distinguish between two physically distinct situations: "low matching rate but reliable feature quality" and "low matching rate and poor feature quality." The lack of proactive and forward-looking scene quality assessment methods is the fundamental reason why keyframe selection strategies cannot truly perceive scene difficulty. Third, there is a lack of quantitative supervision signals for matchability. Even though predicting feature quality through deep learning networks is technically feasible, "feature matchability," as a semantic attribute describing the reliability of pixel-level cross-view matching, currently lacks readily available labeled datasets and a universally accepted quantitative definition method. Furthermore, adding a prediction branch to a pre-trained base model with 3D reconstruction capabilities requires specially designed training strategies to ensure safety. Fourth, the constraint adjustment phase treats all matching observations equally. Existing visual SLAM methods (including the latest schemes such as MASt3R-SLAM) use uniform or reprojection error-based weights for each matching pixel in the bundle adjustment (BA) optimization, lacking explicit modeling of pixel-level feature reliability. Noisy observations in low-reliability areas such as weak textures, motion blur, and reflective highlights are given equal weight to high-quality observations, causing noise to propagate along the optimization link towards pose estimation. This becomes another common bottleneck restricting the accuracy and robustness of front-end tracking. The limitations at these four levels, from the three dimensions of decision-making mechanism, perception capability, and learning method, jointly restrict the keyframe selection efficiency of existing visual SLAM systems in complex scenes, and also constitute the core technical problem that this invention needs to solve.

[0050] To address the aforementioned issues, this invention proposes a visual SLAM system with adaptive keyframe selection capabilities. This system embeds pixel-level matchability prediction as an inherent perceptual capability into its overall architecture, enabling the system to proactively assess the feature quality of the current scene and dynamically adjust the keyframe selection strategy accordingly. Furthermore, the system is designed as a unified architecture with five core components working together organically: a visual feature encoder, a 3D decoder, a matchability prediction head, a front-end tracker, and a back-end optimizer.

[0051] According to one embodiment of the present invention, a visual SLAM system with adaptive keyframe selection function is proposed, see appendix. Figure 1 In summary, the system includes a data preprocessing module, a visual encoding module, a front-end module, and a back-end optimization module. The following will describe each module in the present invention in detail with reference to specific embodiments.

[0052] (a) Data Preprocessing Module

[0053] The data preprocessing module is used to input continuously acquired target environment RGB image sequences, and matches the corresponding reference RGB image for each frame of the original RGB image to complete the temporal image pairing. During pairing, historical keyframes with high spatial overlap and appropriate time intervals with the current frame are selected as reference frames first, and images with excessive viewpoint offset or no overlap are excluded. Multiple sets of ordered RGB image pairs are generated in batches. Subsequently, each set of image pairs is synchronously sent to the visual encoding module to perform unified feature extraction, providing standardized dual-view input materials for parallel 3D geometric decoding and pixel matchability prediction.

[0054] (ii) Visual encoding module

[0055] A visual feature encoder encodes an input RGB image to obtain a high-dimensional semantic feature tensor. Its input is an RGB image, and after forward propagation, the output shape is... A high-dimensional feature tensor is generated, where B is the batch dimension, L is the token sequence length, and D is the feature dimension (B, L, and D correspond to the shape parameters of tensor F along the first, second, and third dimensions, respectively, jointly defining the third-order tensor structure of F). The two view images are independently passed through a visual feature encoder with shared weights, resulting in a pair of structurally identical but semantically different feature tensors. The two encoding branches operate independently during forward propagation, without cross-branch feature interaction during the encoding stage; weight sharing ensures that both images use consistent feature mapping rules, placing the two feature paths in a unified feature representation space and reducing the number of model parameters and computational overhead. The resulting first and second high-dimensional feature tensors are used as inputs to downstream branches: one is input to the 3D decoder, which exchanges information between the two feature paths through a cross-attention mechanism to establish a cross-view 3D geometric correspondence; the other is input to the matchability prediction head, used to predict the pixel-level matching reliability prior for a single frame.

[0056] (III) Front-end module

[0057] The front-end module includes a matchable prediction head, a 3D decoder, and a front-end tracker. The matchable prediction head and the 3D decoder form two parallel feature decoding branches. Specifically, the matchable prediction head performs feature decoding on the feature vectors of each RGB image in each RGB image pair to obtain a matchability score map for each RGB image in each RGB image pair. The matchability score map of an RGB image contains the matchability score of each pixel in that RGB image, and the matchability score of each pixel indicates the complexity of the environmental information contained in that pixel. The 3D decoder uses a cross-attention mechanism to perform bidirectional information exchange and feature decoding on the feature vectors of each RGB image in each RGB image pair to output each RGB image. The pixel-level geometric information of each RGB image in the B image pair, wherein the pixel-level geometric information of an RGB image pair includes the 3D point cloud coordinates, feature descriptor, geometric confidence, and descriptor confidence of each pixel in the RGB image; the front-end tracker is used to determine the effective pixel matching pairs, the camera pose of each RGB image, and the 3D point map of each RGB image based on the matching prediction head to obtain the matching score map and pixel-level geometric information; and to determine the keyframe judgment threshold of each RGB image pair based on the matching score map of each RGB image pair. If the effective pixel matching pairs of the RGB image pair are greater than the keyframe judgment threshold of the RGB image pair, then the original RGB image in the RGB image pair is selected as the keyframe. To facilitate understanding of the various modules in the front-end module, the following will provide a detailed description of each module.

[0058] 1. 3D decoder

[0059] The 3D decoder is used to establish the 3D geometric correspondence between two views based on dual-view features and decode the pixel-level 3D geometry and appearance representation. Its input is a pair of high-dimensional feature tensors F from the visual feature encoder (extracted by the two views independently forward through the same set of shared weight encoders).

[0060] According to one embodiment of the present invention, after the 3D decoder performs bidirectional information interaction on the dual-view features through a cross-attention mechanism, the point map prediction branch and the descriptor prediction branch decode in parallel, and output the three-dimensional point cloud coordinates of each pixel. Feature descriptors (Corresponding point location), geometric confidence level With descriptor confidence The downstream destinations of each output are as follows: X and D serve as inputs to Iterative Projective Matching, used to establish cross-view pixel correspondences and obtain the matching index. With valid matching mask C and Q are injected into the information matrix of subsequent pose solution as residual weighting factors to suppress low-quality observations; C also participates in the weighted fusion of keyframe canonical point maps as a confidence weight.

[0061] 2. Matchability prediction head

[0062] The matchability prediction head and the 3D decoder form a functionally equivalent parallel processing branch, used to predict the matching reliability prior for each pixel from shared encoder features. According to one embodiment of the invention, the input to the matchability prediction head is the high-dimensional feature tensor F output by the visual feature encoder. The matchability prediction head first reconstructs the one-dimensional token sequence into a two-dimensional spatial feature map, then upsamples and aligns it to pixel resolution, and finally performs pixel-by-pixel regression through a feature fusion layer and a matchability prediction layer to output a matchability score map. A value closer to 1 indicates that the 3D prediction of that pixel is more reliable in cross-view matching. The two views respectively obtain the current frame score map. Score map with reference keyframe The output is distributed to the front-end tracker along two independent paths: the pixel-level path combines the dual-view score map with the matching index, and obtains pixel-wise weights through geometric mean, percentile normalization, and adaptive mixing. The information matrix for pose solving is injected; the frame-level path takes the average of all pixels in the current frame's fractional map, which is used for dynamic modulation of the keyframe insertion threshold. The two paths share the same data source but are completely decoupled in computation.

[0063] 3. Front-end tracker

[0064] The front-end tracker is used to estimate camera pose frame by frame, incrementally fuse local geometry of keyframes, and adaptively schedule keyframes. Its input is the (X, C, D, Q) output from the 3D decoder and the output of the matchability prediction head. Four operations are performed sequentially: iterative projection matching, establishing a pixel-by-pixel matching index on the dual-view point map X and descriptor D. and valid matching mask Matchable Modulation Pose Optimization (MMBA): Based on Dual Views Pixel-level weights are obtained through geometric averaging, percentile normalization, and adaptive mixing. The weighted Sim(3) Gaussian-Newton solver is injected to solve the pose of the current frame. Local point map fusion weights the transformed current frame observations according to confidence level C and fuses them into the canonical point map of the keyframe. Quality-aware adaptive keyframe scheduling dynamically modulates the keyframe insertion threshold based on the full-pixel mean of the current frame to determine whether to promote the current frame to a new keyframe. Its output includes the current frame pose, the updated canonical point map of the keyframe, and the keyframe insertion decision.

[0065] According to one embodiment of the present invention, the front-end tracker performs iterative projection matching, matchable modulation pose optimization, local point map fusion, and quality-aware adaptive keyframe scheduling in the following manner: First, the system performs iterative projection matching to obtain the pixel correspondence between the current frame and the reference keyframe. Specifically, for each pixel in the current frame... According to the current frame point map output by the 3D decoder and reference keyframe point diagram The projection search problem is constructed by transforming the 3D points of the current frame to the coordinate system of the reference keyframe, and then searching for the position on the reference keyframe point map or ray map that best matches its geometric orientation; subsequently, this is combined with the descriptor. Within a local neighborhood, four-way matching refinement is performed, and occlusion, out-of-bounds, and inconsistent matching are filtered out using 3D distance, projection residuals, or effective view constraints. After several iterations, the matching index is output. and valid matching mask .in, Represents the current frame pixels The corresponding pixel in the reference keyframe is , This indicates whether the match is valid. It's important to note that iterative projection matching outputs the matching relationship and validity mask, not the matching quality directly; the matching quality is determined after obtaining the match result. Then, the matchability graph between the current frame and the reference keyframe is generated. Calculated after aligning according to the matching relationship. After obtaining the matching index, for each pair of valid matches ,in, The system constructs the initial matching-level quality based on two-sided matchability:

[0066]

[0067] in, Indicates the first The original matching quality of each valid matching pixel pair. and Indicates the first Two pixels in a valid matching pixel pair For the pixels in the original RGB image, To reference the pixels in the RGB image, The matching score graph representing the original RGB image Matchability score The matching scores of the reference RGB images are shown in the following order. Matchability score The value is a very small positive number to avoid instability caused by a value of zero. This symmetric geometric mean form ensures that a matching observation receives a high quality score only when both sides of the match have high matchability; if either side is in a blurred, weakly textured, occluded, or structurally degraded region, the matching quality will be reduced accordingly. To mitigate the scale differences in the distribution of matchability values ​​between different scenes and datasets, the system further refines the original matching-level quality of all valid matches in the current frame. Perform quantile normalization. Let... and Let these be the low-quantile and high-quantile statistics of the effective matching quality set of the current frame, such as the 10th percentile and 90th percentile, respectively. Then, the normalized matching quality is defined as:

[0068]

[0069] in, Indicates the first The final matching quality of each effective matching pixel pair (effective matching observation), Indicates the first The original matching quality of each valid matching pixel pair. and These represent the original matching quality of all valid matching pixel pairs after sorting the matching quality in ascending order, with preset low and preset high resolutions. Indicates a preset positive number. This represents the normalization function.

[0070] Subsequently, the system injects the normalized matching quality as a soft weight into the pose optimization of the current frame relative to the reference keyframe, forming a matchable modulation pose optimization (MMBA). Let the th... The geometric residuals of each valid matched observation are ,in, This represents the relative pose transformation from the current frame coordinate system to the reference keyframe coordinate system; let... Given the basic information matrix of this observation, the pose estimate of the current frame relative to the reference keyframe is:

[0071]

[0072] Among them, the final soft weight Defined as:

[0073]

[0074] in, For the first The weights for each matched observation when entering the weighted least squares optimization. These are the matchability modulation coefficients, used to control the intensity of the influence of matchability quality on pose optimization. When When all matched observations degenerate to uniform weights; when As the size increases, the system places greater emphasis on highly matchable observations. In practical applications, It can be set to a fixed value, or the matching quality can be normalized based on the current frame. The statistical distribution is adaptively determined, for example, when the variance of the quality distribution is large and the discrimination is strong, the value is increased. When the mass distribution is flat, it decreases This is to avoid over-reliance on a single quality signal. In the above formula... For the first The geometric residuals of each matched observation can take different forms depending on whether the system uses camera intrinsics. In calibration mode, the matching pixels of the current frame are set as follows: The predicted 3D points are The reference keyframe corresponds to the pixel. The camera projection function is The residual can then be written as:

[0075]

[0076] in, From camera internal parameters Determined, used to project three-dimensional points onto a two-dimensional pixel plane; Output from iterative projection matching This is obtained. In uncalibrated or generalized camera modes, the residual can also be expressed as ray direction error, which is the angular difference between the transformed 3D point orientation in the current frame and the corresponding point orientation in the reference keyframe. Basic information matrix. The base confidence level used to represent the observation in weighted least squares can be generated from the system's existing geometric confidence and descriptor confidence levels, for example:

[0077]

[0078] in, It is the identity matrix. and These are the geometric confidence scores of the corresponding pixels in the current frame and the reference keyframe, respectively. and These are the confidence scores for the two descriptors, respectively. If the system only uses the confidence score for the reference keyframe, it can also be set as follows: It is important to note that, This represents the weights of the basic information derived from the confidence scores of the original 3D decoder output. This indicates that the output is from the matchability prediction head. The matching quality weights obtained through further modulation have different origins and functions.

[0079] The optimal relative pose of the current frame with respect to the reference keyframe is obtained using MMBA. Then, the system uses the global pose of the reference keyframe. Calculate the global pose of the current frame:

[0080]

[0081] Then, local point map fusion is performed, assuming the currently maintained canonical point map of the reference keyframe is... The cumulative geometric confidence level is Current frame pose The viewpoint plot after alignment to the reference keyframe coordinate system is as follows:

[0082]

[0083] Its corresponding geometric confidence level is Then the keyframe local point map is updated by confidence weighting as follows:

[0084]

[0085]

[0086] in, Indicates pixel-by-pixel multiplication. To prevent numerically stable terms with a denominator of zero, this local point map fusion operation is used to incrementally update the local 3D representation of keyframes after front-end tracking. The fusion operation itself is primarily based on geometric confidence. Use weighted average instead of direct use ; Its main function is reflected in the previous MMBA pose optimization step, enabling the current frame to participate in fusion under a more reliable aligned pose, thereby indirectly improving the quality of local point map fusion.

[0087] According to one embodiment of the present invention, after completing online tracking and local geometry fusion, the system needs to decide whether the current observation should be promoted to a new keyframe. If the keyframe insertion is too conservative, the local constraint coverage will be insufficient, making it difficult to stably cross observation intervals involving rapid motion, sudden changes in viewpoint, and degradation; if the insertion is too frequent, redundant states will be introduced, increasing the burden of subsequent optimization. To address this, this paper proposes a Quality Aware Keyframe Selection (QAKS) mechanism, which explicitly introduces observation quality statistics into the keyframe decision-making process, transforming the keyframe triggering from a fixed threshold rule into an adaptive scheduling process driven by matchability. The core idea is that the front end should not only identify high-quality observations at the pixel and matching levels, but also adjust the triggering timing of keyframes at the frame level using the overall quality distribution.

[0088] The matchability graph (matchability score graph) for the current frame is The effective pixel set is Then the frame-level quality statistics are defined as follows:

[0089]

[0090] in, This describes the average matchability of the current observations in a global sense. A higher value indicates that the current frame can provide a stable and sufficient geometric correspondence, and the system maintains a relatively conservative keyframe insertion strategy; when... A lower value indicates a reduced reliable constraint area, and the keyframe trigger sensitivity should be increased to prevent tracking from losing stable support in the degradation range.

[0091] Based on the above statistics, this paper rewrites the keyframe determination threshold in a quality modulation form:

[0092]

[0093] in, Based on the threshold, For reference quality level, For adjustment coefficients, The upper bound is defined by the keyframes. QAKS's goal is not simply to increase the number of keyframes, but to reshape the triggering timing and temporal distribution of keyframes based on observation quality. Therefore, it optimizes the effectiveness of the keyframe set, rather than mechanically adjusting the sampling density. Thus, MMBA and QAKS complement each other at the front end: the former improves single-step registration reliability at the matching level, while the latter improves keyframe scheduling discriminability at the frame level. Together, they enable the system to prioritize observations that contribute more to geometric estimation in complex scenes, providing more stable and information-dense constraints for subsequent global optimization.

[0094] (iv) Backend Optimizer

[0095] The backend optimizer performs global pose optimization on the accumulated keyframe set to maintain long-term consistency. Its input is the keyframe set and its matching relationships progressively output by the frontend tracker. The matching relationships include all relevant data from the preceding modules: pixel-level 3D point cloud coordinates X, feature descriptor D, geometric confidence C, and descriptor confidence Q output by the dual-view 3D decoder, as well as the matching index obtained by matching the pixel-level matchability map generated by the matchability prediction head with the frontend iterative projection. With valid matching mask The backend optimizer first retrieves loop closure candidate frames from the keyframe database using ASMK visual retrieval; then, it performs 3D decoder symmetric verification on the matching relationship between the candidate loop closure frames and the current keyframe, and adds the verified loop closure matches as loop edges in the factor graph. The constructed factor graph contains edges formed by the matching relationships between all keyframes, and then performs global Sim(3) Gaussian-Newton optimization. After optimization, it outputs globally consistent keyframe pose trajectories. The dense point cloud map provides a unified and consistent global map representation for the system. This design ensures the effective use of front-end pixel-level matching and matchability scoring information in global optimization, enabling the system to maintain accurate pose estimation in long-term scenarios, while giving full play to the synergistic effect of keyframe incremental fusion, loop closure detection and quality-aware weights.

[0096] According to an embodiment of the present invention, the present invention also proposes a construction method for the above-mentioned SLAM system, which adopts a phased task adaptation strategy in training: first, the basic geometric representation is fixed and only lightweight task branches are learned; then, small-step joint fine-tuning is performed so that the high-level representation gradually adapts to the SLAM data distribution, while maintaining the original 3D reconstruction capability as much as possible.

[0097] Specifically, this invention employs a phased task adaptation strategy during the training phase. The training dataset consists of RGB images with temporal or spatial overlap, camera poses, camera intrinsics, and corresponding 3D geometric annotations. For example, a training sample includes two or more frames of RGB images from the same scene, such as a dual-view image pair consisting of the current frame and a reference frame. Each frame is provided with a corresponding depth map or 3D point map label, an effective pixel mask, cross-view pixel correspondence, camera pose, and camera intrinsics. The 3D point map label constrains the pixel-level 3D coordinates output by the decoder. The cross-view pixel correspondence and effective pixel mask determine which pixels can form a reliable match. The matchability label can be generated based on the error between the predicted 3D point and the real 3D point, the cross-view geometric consistency error, or the effective matching relationship, and is used to supervise the matching reliability score of each pixel output by the matchability prediction head.

[0098] The training process consists of two phases: The first phase is used to fix the basic geometric representation and learn only the lightweight task branch. Specifically, the parameters of the shared weighted visual feature encoder and 3D decoder are frozen, and only the parameters of the newly added matchability prediction head and its feature fusion layer are optimized. This allows the model to learn to predict pixel-level matching reliability from the high-dimensional features output by the encoder without changing the original 3D geometric representation capability. The second phase is used for small-step joint fine-tuning. Specifically, the parameters of the bottom-level visual feature encoder are kept frozen, or only the high-level representation module of the visual feature encoder, the high-level decoding module of the 3D decoder, and the matchability prediction head are unfrozen. Joint optimization is performed at a learning rate lower than that of the first phase, so that the high-level semantic representation, 3D geometric decoding results, and matchability prediction results gradually adapt to the SLAM scene data distribution. At the same time, the model is constrained by the 3D reconstruction loss, the matching reliability prediction loss, and the geometric consistency loss to prevent the newly added matchability branch from destroying the original basic 3D reconstruction capability. Thus, the first stage primarily completes the stable initialization of the newly added lightweight branch, while the second stage completes the overall adaptation related to the SLAM data distribution. This allows for pixel-level matchability priors that can be used for front-end matching weight adjustment and keyframe selection while maintaining the original 3D geometric capabilities. The overall optimization objective (loss function) is:

[0099]

[0100] The constraints and monitoring methods for each loss item are as follows:

[0101] The matching reliability supervision loss is constrained by the pixel-level matching reliability map output by the matching reliability prediction head. With supervision label Predict 3D points from the current frame With GT 3D points Generate. Specifically, generate exponentially decaying soft tags:

[0102]

[0103] For reprojection loss, constrain the prediction of 3D points Camera projection function Get pixel coordinates , and GT pixel coordinates Geometric errors between:

[0104]

[0105] in, For camera internal references relative pose The combined projection function, GT pixel coordinates are obtained by backprojection from the depth map.

[0106] For epipolar constraint loss, for cross-view matching point pairs Satisfying polar geometry constraints Minimize the vertical distance from the matching point to the corresponding epipolar line:

[0107]

[0108] Where F is the fundamental matrix calculated from the keyframe and the pose of the current frame, and the matching point pairs are obtained through... Automatically generated, no additional tags required.

[0109] The depth loss constraint constrains the network's depth prediction. With GT depth Error between:

[0110]

[0111] GT depth is obtained by a depth sensor or by a 3D point cloud projection.

[0112] This refers to the 3D point cloud loss, corresponding to the 3D point cloud error constraint mentioned in this paper, which constrains the prediction of 3D points. With GT 3D points Euclidean distance:

[0113]

[0114] in, Obtained from the point map output by the 3D decoder. It is generated by ground truth depth backprojection. This loss is the core of the supervision signal in this paper, directly driving the adaptive learning of pixel-level matching reliability.

[0115] For descriptor matching constraints, the cosine similarity of the descriptors of correctly matched point pairs is higher than that of incorrectly matched pairs. A contrastive loss in the form of InfoNCE is used:

[0116]

[0117] Among them, the positive samples are Corresponding points, negative samples represent other unmatched points. This is the temperature coefficient.

[0118] As a matchability distribution calibration term, it constrains the mean of the matchability prediction M distribution. With variance Maintain within a reasonable range to prevent premature saturation:

[0119]

[0120]

[0121]

[0122] in, , The target mean and variance are preset.

[0123] To demonstrate the beneficial effects of this invention, the proposed method was comprehensively evaluated on three benchmark datasets: TUM RGB-D, 7-Scenes, and EuRoC, covering camera pose estimation, ablation analysis, etc., and visual verification was performed on the DTU and Replica datasets. The system runs at approximately 15 FPS, and the maximum dimension of the MASt3R output resolution is 512. Pose evaluation uses scale-aligned absolute trajectory error (ATE, RMSE, in meters).

[0124] To separate the independent contributions of MMBA and QAKS, 2×2 factorial ablation experiments were conducted on four point map fusion strategies (Recent, First, Median, and Weighted). Experimental results show that the contribution of MMBA exhibits a significant conditional dependence. Under the Weighted+Calibration configuration, its gain is close to zero, indicating that multi-frame weighted fusion effectively smooths point map noise. The Weighted+Uncalibration configuration shows a better performance improvement. Under the First strategy, it brings error reductions of 16.0 mm (calibrated) and 19.0 mm (uncalibrated), respectively. The fundamental reason is that First relies only on a single point map observation, lacking multi-frame redundancy, making prediction errors in low-reliability regions more easily propagated directly to pose estimation. MMBA effectively suppresses noise propagation through pixel-wise weight modulation. Even in the uncalibrated Weighted configuration, MMBA still brings an improvement of 2.0 mm, indicating its stable value even when the observation model is incomplete. In calibration mode, QAKS consistently contributes approximately 1.0 mm of ATE improvement to each fusion strategy, which is the main source of accuracy improvement under the default Weighted+Calibration configuration. In uncalibrated Recent fusion, QAKS contributes up to 6.0 mm. Recent fusion only retains the most recently observed pointmap, amplifying the impact of keyframe quality: once a low-quality keyframe is selected, its pointmap error directly determines the pose estimation of all subsequent non-keyframes. QAKS avoids the selection of such harmful frames by dynamically increasing the threshold. The contribution of QAKS remains stable across all fusion strategies and calibration modes, indicating that frame-level keyframe quality optimization has a universal effect on improving system performance.

[0125] In the loop closure ablation experiment, after enabling loop closure, the ATE of EuRoC in calibration mode decreased significantly from 0.443m to 0.040m (an improvement of 90.9%), and TUM and 7-Scenes also showed significant improvements. The significant improvement on EuRoC is due to its long sequence characteristics; the MH series trajectory length exceeds 80m, and accumulated drift is unavoidable without loop closure. Loop closure detection was achieved by visually retrieving spatial revisits using ASMK, and global bundle adjustment eliminated drift to achieve trajectory closure. This result confirms that backend global optimization is indispensable for long sequence scenarios and also shows that the frontend improvements (QAKS and MMBA) and backend optimizations in this paper are positively complementary: the former suppresses frontend error accumulation, and the latter eliminates global drift.

[0126] In summary, compared with the prior art, the present invention has the following six beneficial effects: (1) Adaptability: For the first time, scene perception key frame selection based on deep learning prediction is realized in a visual SLAM system. The system can dynamically adjust the key frame insertion density according to the feature quality of the current observed scene, thereby selecting high-quality key frames, automatically increasing key frame constraints in difficult scenes (low texture, high occlusion), and avoiding redundancy in simple scenes. (2) Enhanced robustness: Relying on the active perception of scene feature quality and the dynamic intervention of key frame insertion strategy, the system defects of traditional fixed thresholds that easily lead to a sudden reduction in constraints and tracking loss under extreme conditions (such as feature-scarce environments) are overcome, ensuring the zero-fault stable operation of the SLAM system in the entire test sequence, and greatly expanding the environmental adaptation boundary of the system. (3) Significantly improved accuracy: The dynamic key frame selection mechanism based on feature matching perception replaces the traditional globally fixed frame building method. This mechanism effectively compensates for the lack of pose constraints in low-texture areas by adaptively optimizing the spatial distribution density of key frames, greatly reducing the cumulative drift of the system, thereby significantly reducing the global absolute trajectory error of the system in various complex scenes. (4) Efficiency optimization: The unidirectional adjustment design of the quality-aware keyframe selection mechanism does not introduce any additional operations in normal scenarios, while the fast forward path of the newly trained branch only introduces very low additional latency, ensuring that the acquisition of quality-aware capabilities does not affect the real-time performance of the system. (5) Resource conservation: By avoiding the insertion of redundant keyframes in normal scenarios, the node scale of the back-end global optimization is effectively controlled, reducing the computational complexity and memory consumption of the beam adjustment, and ensuring the real-time operation of the system. (6) Robust weighting of the optimization layer: The MMBA mechanism for the first time injects pixel-level feature matching as an explicit weight into the beam adjustment process, effectively suppressing the noise propagation of low-reliability observations such as weak textures, motion blur, and reflection specular highlights into the pose solution. It significantly improves the accuracy and stability of front-end tracking without discarding any observations, and automatically degenerates to uniform weighting when scene differences are not obvious, ensuring numerical consistency with the baseline method.

[0127] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0128] This invention can be a system, method, electronic device, computing device, or computer program product. A computer program product mainly refers to a software product that implements this solution through a computer program.

[0129] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual SLAM system with adaptive keyframe selection capability, used to construct a dense point cloud map of a target environment based on RGB image sequences acquired from the target environment, characterized in that, The system includes: The data preprocessing module is used to configure a reference RGB image for each raw RGB image in the image sequence to construct multiple RGB image pairs; The visual encoding module is used to encode the two RGB images in each RGB image pair constructed by the data preprocessing module into feature vectors to obtain multiple feature vector pairs; The front-end module is used to perform feature decoding on the feature vector of each RGB image in each RGB image pair to obtain the matchability score map and pixel-level geometric information of each RGB image in each RGB image pair. Based on the matchability score map and pixel-level geometric information of all RGB images in each RGB image pair, it predicts the effective pixel matching pairs, the camera pose of each RGB image, and the 3D point map of each RGB image. Based on the matchability score map of each RGB image pair, it determines the keyframe judgment threshold of the RGB image pair. If the effective pixel matching pairs of the RGB image pair are greater than the keyframe judgment threshold of the RGB image pair, the original RGB image in the RGB image pair is selected as the keyframe. The backend optimization module is used to construct a dense 3D map based on all keyframes selected by the frontend module, the pixel geometry information of each keyframe, the predicted camera pose, and the predicted 3D point map.

2. The system according to claim 1, characterized in that, The visual encoding unit includes two image encoders with shared weights. The two image encoders encode the features of the two RGB images in the RGB image pair to obtain two feature tensors.

3. The system according to claim 1, characterized in that, The front-end module includes a matchable prediction head, a 3D decoder, and a front-end tracker. The matchable prediction head and the 3D decoder constitute two parallel feature decoding branches, wherein: The matchable prediction head is used to perform feature decoding on the feature vector of each RGB image in each RGB image pair to obtain a matchability score map for each RGB image in each RGB image pair. The matchability score map of an RGB image contains the matchability score of each pixel in the RGB image, and the matchability score of each pixel indicates the complexity of the environmental information contained in the pixel. The 3D decoder uses a cross-attention mechanism to perform bidirectional information exchange and feature decoding on the feature vectors of each RGB image in each RGB image pair to output the pixel-level geometric information of each RGB image in each RGB image pair. The pixel-level geometric information of an RGB image pair includes the three-dimensional point cloud coordinates, feature descriptor, geometric confidence and descriptor confidence of each pixel in the RGB image. The front-end tracker is used to determine the effective pixel matching pairs, camera pose of each RGB image, and 3D point map of each RGB image based on the matching score map and pixel-level geometric information obtained by the matching prediction head. It also determines the keyframe judgment threshold of each RGB image pair based on the matching score map. If the effective pixel matching pairs of the RGB image pair are greater than the keyframe judgment threshold of the RGB image pair, the original RGB image in the RGB image pair is selected as the keyframe.

4. The system according to claim 1, characterized in that, The range of the matchability score for each pixel is: Among them, the higher the matching score, the more complex the environmental information corresponding to that pixel.

5. The system according to claim 1, characterized in that, The front-end tracker determines the camera pose of the original RGB image in each image pair in the following manner: Based on the 3D point cloud coordinates and feature descriptors of each pixel in each RGB image of the image pair, construct an effective matching pixel pair for the image pair; Based on the constructed effective matching pixel pairs and the matchability score map of each RGB image in the image pair, the original matching quality of each effective matching pixel pair is evaluated in the following way: in, Indicates the first The original matching quality of each valid matching pixel pair. and Indicates the first Two pixels in a valid matching pixel pair For the pixels in the original RGB image, To reference the pixels in the RGB image, The matching score graph representing the original RGB image Matchability score The matching scores of the reference RGB images are shown in the following order. Matchability score The preset positive number; The original matching quality of each valid matching pixel pair in the image is normalized to obtain the final matching quality of each valid matching pixel pair. Using the final matching quality of all valid matching pixel pairs as constraints, the global pose of the current frame is constructed as follows: in, This represents the predicted camera pose of the original RGB image. This represents the camera pose of the reference RGB image. Indicates the first The geometric residual of a valid matching pixel pair Indicates the preset first The weight of each valid matching pixel pair This represents the weights of basic information determined based on pixel-level geometric information. and Indicates the first The 3D point cloud coordinates of the two pixels in a valid matching pixel pair and Indicates the first Geometric confidence of two pixels in a valid matching pixel pair Represents the identity matrix.

6. The system according to claim 5, characterized in that, The normalization process is as follows: in, Indicates the first The final matching quality of a pair of valid matching pixels. Indicates the first The original matching quality of each valid matching pixel pair. and These represent the original matching quality of all valid matching pixel pairs after sorting the matching quality in ascending order, with preset low and preset high resolutions. Indicates a preset positive number. This represents the normalization function.

7. The system according to claim 1, characterized in that, The keyframe judgment threshold is determined as follows: in, The preset base threshold, As a preset reference quality level, The preset adjustment coefficient, The upper bound is a preset value. This indicates the adjusted judgment threshold. This represents frame-level quality statistics. This indicates the number of pixels. Indicates the RGB image to be evaluated The quality score.

8. A method for constructing a visual SLAM system with adaptive keyframe selection function, characterized in that, The method includes: S1. Obtain the dataset, which includes multiple samples, wherein each sample includes a reference keyframe RGB image and an RGB image to be evaluated, as well as the true 3D point map of each RGB image, the true camera pose of each RGB image, and all valid pixel matching pairs between the reference keyframe RGB image and the RGB image to be evaluated; S2. Obtain the visual SLAM system as described in any one of claims 1-7; S3. Based on the dataset obtained in S1, the image pair consisting of the reference keyframe RGB image and the RGB image to be evaluated is used as input. The real 3D point map of the predicted RGB image, the real camera pose of each RGB image, and all valid pixel matching pairs between the reference keyframe RGB image and the RGB image to be evaluated are used as output. The visual SLAM system is trained to convergence using a preset loss function. During the training process, the parameters of the data preprocessing module and the back-end optimization module are frozen, and only the parameters of the visual encoding module and the front-end module are adjusted.

9. The method according to claim 8, characterized in that, The preset loss function is: in, This represents the matching ability supervision loss. The loss between the matchability prediction head output matchability map and the true matchability map is denoted as . Indicates reprojection loss. The loss between the pixel coordinates obtained by projecting the 3D point cloud coordinates of a pixel and the pixel's true pixel coordinates. Indicates epipolar constraint loss. Indicates deep loss. Indicates the loss of the 3D point cloud. This represents the loss between the 3D point cloud coordinates output by the 3D decoder and the actual 3D point cloud coordinates. This represents the descriptor matching constraint loss. The loss between the pixel descriptor output by the 3D decoder and the real descriptor. For the matching distribution calibration item, , , , , , and All of these are preset loss weights.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 8 and 9.

11. A computer program product, comprising a computer program / instructions, characterized in that, The computer program / instructions are executed by a processor to implement the steps of the method according to any one of claims 8 and 9.

Images