A binocular vision-inertial tightly coupled localization method based on a learning-based sparse vision front end

Abstract

This application proposes a binocular vision-inertial tightly coupled localization method and device based on a learning-based sparse vision front-end, relating to the fields of artificial intelligence and data processing technology. The method acquires multi-scale feature maps through a learning feature encoding network, followed by semantic-guided modulation, keypoint detection, global correlation initialization, and multi-channel adaptive damped sparse Lucas-Kanade iterative refinement. Combined with geometric filtering and binocular matching, the final result, along with IMU pre-integrated data, is fed into a sliding window factor map optimization, outputting a six-DOF pose estimate of the carrier. This application significantly improves the robustness of the vision front-end in challenging scenarios while maintaining the real-time performance of the embedded platform, and can be seamlessly integrated with a classic tightly coupled VIO back-end.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, specifically to a binocular vision inertial tight coupling positioning method and device based on a learning-based sparse visual front end. Background Technology

[0002] Visual inertial odometry (VIO) systems are a core positioning technology in fields such as autonomous robots and micro-drones. Classic tightly coupled VIO systems, such as VINS-Mono / Fusion, employ traditional feature point detection (GFTT) and optical flow (Lucas-Kanade) tracking as the visual front end, offering the advantage of stable performance in textured scenes. However, traditional LK optical flow tracking assumes that the brightness gradient of the LK optical flow is a single-channel image. This assumption leads to a significant loss of feature points in scenes with weak textures, motion blur, or abrupt changes in illumination, resulting in a marked decrease in robustness and a degradation or even divergence in the positioning accuracy of the VIO system.

[0003] In recent years, deep learning optical flow methods (such as RAFT and GMFlow) have made significant progress in general optical flow estimation. However, these methods target dense optical flow, resulting in large-scale models that are difficult to run in real time on computationally limited embedded platforms. They also lack optimization for the sparse correspondence quality required by VIO. Existing solutions for integrating learning front-ends into VIO either rely on computationally expensive feature descriptor matching (such as SuperPoint+SuperGlue), directly embed dense optical flow networks into VIO while neglecting real-time performance, or lack robust mechanisms to handle tracking failures in weakly textured scenes. Summary of the Invention

[0004] The main purpose of this application is to provide a binocular vision inertial tight-coupled localization method and device based on a learning sparse vision front-end. It aims to propose a unified learning sparse correspondence generation framework based on a lightweight backbone network, which significantly improves the robustness of the vision front-end in challenging scenarios while maintaining the real-time performance of the embedded platform, and seamlessly integrates with the classic tight-coupled VIO back-end through a standardized interface.

[0005] To achieve the above objectives, the first aspect of this application provides a binocular vision-inertial tightly coupled localization method based on a learned sparse visual front end, comprising: Input the current frame's binocular grayscale image into a learning feature encoding network to obtain multi-scale feature maps; The feature maps of multiple scales are input into a pre-constructed semantic-guided modulation module, which outputs a multi-scale matching feature map. The semantic-guided modulation module is used to adaptively modulate and fuse the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map. The binocular grayscale image is applied to the shallow scale feature map of the backbone network through the geometric key point detection head, and the key point confidence map is output. Candidate key points are obtained after screening. The global normalized correlation matrix of candidate key points between the reference frame and the current frame is calculated on the deep scale feature map. After probability truncation and renormalization, the coarse corresponding displacement estimate of sparse key points is obtained by sampling. Using the coarse corresponding displacement estimate as the initial value, multi-channel adaptive damped sparse Lucas-Kanade iterative refinement is performed step by step on the multi-scale feature map to obtain sparse key points with accurate sub-pixel correspondence. The left-eye temporal tracking key points corresponding to the sub-pixel level are sequentially subjected to boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control. When the number of surviving points is insufficient, new detection key points are added to obtain the left-eye retained key points of the current frame. The right-eye matching is performed with the left-eye retained key points of the current frame as input, and the boundary validity checks, disparity range constraints, epipolar threshold constraints under rectified binocular conditions, bidirectional consistency error checks, and RANSAC outlier removal are sequentially performed to obtain the matching feature points of the left-eye previous and next frames and the matching feature points of the left and right eyes of the current frame. Pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance; The matching feature points of the left and right frames and the matching feature points of the left and right targets in the current frame, along with the relative pose increment and its covariance, are fed into the sliding window factor map for optimization. The marginalization prior residual, visual reprojection residual and IMU pre-integration residual are jointly minimized to output the six-degree-of-freedom pose estimate of the carrier.

[0006] Optionally, the adaptive modulation and fusion of the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map includes: The deep feature map is bicubic upsampled to the size of the shallow feature map, and then subjected to depthwise separable smooth convolution to eliminate the grid aliasing introduced by upsampling, resulting in the upsampled deep feature map. The upsampled deep feature map is used to generate channel-gated weights through global average pooling and 1×1 convolution, and spatial-gated weights are generated through 1×1 convolution. After multiplying the shallow geometric features element-wise with the channel gating weights and spatial gating weights, the deep semantic projection values ​​are superimposed to complete the adaptive modulation and fusion of semantic information.

[0007] Optionally, the learning-based feature encoding network is the backbone network, and its output multi-scale features include four scale feature maps: 1 / 2, 1 / 4, 1 / 8, and 1 / 16. The four scale feature maps of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 are the outputs of the backbone network stage1 to stage4, respectively. The geometric key point detection head includes a 3×3 convolutional neighborhood integration layer, a batch normalization layer, a ReLU activation layer, a 1×1 output convolutional layer, a 3×3 average pooling layer, and a sigmoid mapping layer connected in sequence. The geometric keypoint detection head is directly applied to the 1 / 4 scale feature map output by stage2 to output a keypoint confidence map.

[0008] Optionally, the step of calculating the global normalized correlation matrix of candidate keypoints between the reference frame and the current frame on the deep-scale feature map, and obtaining a coarse corresponding displacement estimate of sparse keypoints through sampling after probability truncation and renormalization includes: Calculate the normalized feature vectors of the 1 / 16 scale feature maps of two adjacent frames, and calculate the scaling cosine correlation of any two positions in the two adjacent frames based on the two normalized feature vectors. The truncation probability is calculated based on the normalized scaled cosine correlation, and the truncation probability is normalized to obtain a dense coarse displacement field. Sampling is performed in a dense displacement field to obtain coarse correspondence estimates of sparse key points.

[0009] Optionally, the step of performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on multi-scale feature maps to obtain sub-pixel-level precisely corresponding sparse keypoints includes: When performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement: Multi-channel patch alignment is performed on multi-scale feature maps to reduce the refinement loss rate in weak texture regions. An adaptive damping term based on the matrix trace is introduced when inverting the Hessian matrix. The damping coefficient is proportional to the trace of the Hessian matrix to prevent numerical collapse in weak texture regions. The region outside the feature map boundary is sampled using the border extrapolation mode to avoid the failure of corresponding fine-tuning of key points at the image edge.

[0010] Optionally, the order of performing the complete time-series tracking of multi-channel adaptive damped sparse Lucas-Kanade iterative refinement step by step is 1 / 8 scale feature map, 1 / 4 scale feature map, and 1 / 2 scale feature map. The stereo matching sequence for progressively refining the multi-channel adaptive damped sparse Lucas-Kanade iterative process is 1 / 4 scale feature map and 1 / 2 scale feature map.

[0011] Optionally, the feature point dispersion control includes: Using a predetermined radius as the neighborhood range, when the number of keypoints in the neighborhood exceeds the upper limit, stable keypoints with more tracking frames are retained first, while the remaining keypoints are suppressed.

[0012] Optionally, it also includes: For rectified stereo image pairs, retain matching points where the x-coordinate of the key point in the left image is greater than the x-coordinate of the key point in the right image, the Y-direction deviation is within the threshold, and the disparity is within the effective range. For non-rectified raw inputs, no hard threshold constraint is applied in the Y direction; instead, geometric consistency screening is performed using the fundamental matrix RANSAC.

[0013] Optionally, the state variables to be optimized in the sliding window factor graph include the position, velocity, attitude, IMU bias, and camera extrinsic parameters of each keyframe.

[0014] To achieve the above objectives, a second aspect of this application also provides a binocular vision-inertial tightly coupled positioning device based on a learning-based sparse vision front end, comprising: The multi-scale feature encoding module is used to input the current frame's binocular grayscale image into the learning feature encoding network to obtain multi-scale feature maps; The semantic-guided modulation module is used to input feature maps of multiple scales into the pre-constructed semantic-guided modulation module and output multi-scale matching feature maps. The semantic-guided modulation module is used to adaptively modulate and fuse the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map. The key point detection module is used to apply the binocular grayscale image to the shallow scale feature map of the backbone network through the geometric key point detection head, output the key point confidence map, and obtain candidate key points through screening. The global correlation initialization module is used to calculate the global normalized correlation matrix of candidate key points between the reference frame and the current frame on the deep scale feature map. After probability truncation and renormalization, the coarse corresponding displacement estimate of sparse key points is obtained by sampling. The multi-channel LK refinement module is used to perform multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on multi-scale feature maps step by step, using the coarse corresponding displacement estimate as the initial value, to obtain sparse key points with accurate sub-pixel correspondence. The geometric filtering and binocular matching module is used to sequentially perform boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control on the sub-pixel level precisely corresponding left-eye temporal tracking key points. When the number of surviving points is insufficient, new detection key points are added to obtain the left-eye retained key points of the current frame. The right-eye matching is performed with the left-eye retained key points of the current frame as input, and boundary validity checks, disparity range constraints, epipolar threshold constraints under rectified binocular conditions, bidirectional consistency error checks, and RANSAC outlier removal are sequentially performed to obtain the matching feature points of the previous and next left-eye frames and the matching feature points of the left and right eyes of the current frame. The IMU pre-integration module is used to pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance. The tightly coupled optimization module is used to feed the matching feature points of the left and right frames and the matching feature points of the left and right targets in the current frame, along with the relative pose increments and their covariances, into the sliding window factor map for optimization. It jointly minimizes the marginalized prior residual, the visual reprojection residual, and the IMU pre-integration residual, and outputs the six-degree-of-freedom pose estimate of the carrier.

[0015] This application discloses a binocular vision-inertial tightly coupled localization method and apparatus based on a learning-based sparse visual front-end. The method includes: inputting the current frame binocular grayscale image into a learning feature encoding network to obtain multi-scale feature maps; inputting the multi-scale feature maps into a pre-constructed semantic-guided modulation module to output a multi-scale matching feature map; applying the binocular grayscale image to the shallow-scale feature map of the backbone network through a geometric keypoint detection head to output a keypoint confidence map, and obtaining candidate keypoints after screening; calculating the global normalized correlation matrix of candidate keypoints between the reference frame and the current frame on the deep-scale feature map, and obtaining a coarse corresponding displacement estimate of the sparse keypoints through sampling after probability truncation and renormalization; using the coarse corresponding displacement estimate as the initial value, performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on the multi-scale feature map step by step to obtain sparse keypoints with sub-pixel-level accurate correspondence; and sequentially performing temporal tracking of the sub-pixel-level accurate correspondence of the left eye. Keypoint detection performs boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control. New keypoints are added when the number of surviving points is insufficient, resulting in the left-eye retained keypoints for the current frame. Right-eye matching is then performed using these left-eye retained keypoints as input, sequentially executing boundary validity checks, disparity range constraints, epipolar threshold constraints under rectified binocular conditions, bidirectional consistency error checks, and RANSAC outlier removal to obtain matching feature points for the left-eye preceding and following frames, and matching feature points for the left and right eyes in the current frame. IMU measurement data between binocular image frames is pre-integrated to calculate the relative pose increment and its covariance. The matching feature points for the left-eye preceding and following frames, and the matching feature points for the left and right eyes in the current frame, along with the relative pose increment and its covariance, are fed into a sliding window factor map optimization. This jointly minimizes the marginalization prior residual, visual reprojection residual, and IMU pre-integration residual, outputting a six-DOF pose estimate of the vehicle. The technical advantages of this application are as follows: A semantically guided modulation module adaptively injects deep semantic features into shallow geometric features, providing high-quality multi-scale matching features with both semantic context and local geometric details for subsequent sparse LK refinement; a global correlation initialization module establishes global matching priors at the deep scale, providing accurate coarse correspondence initialization for large displacement scenes; a multi-channel adaptive damped sparse LK refinement module achieves robust sub-pixel-level correspondence refinement on the learned feature map, preventing numerical collapse in weak texture regions; a geometric filtering pipeline (bidirectional error checking, CrowdSuppression dispersion control, epipolar constraints, RANSAC outlier removal) further ensures observation quality; and finally, the data is fed into a classic tightly coupled VIO backend to complete six-degree-of-freedom pose estimation, exhibiting superior robustness compared to traditional optical flow front-ends in challenging scenarios such as insufficient texture and varying illumination. Attached Figure Description

[0016] Figure 1A flowchart illustrating a binocular vision-inertial tightly coupled localization method based on a learning sparse visual front end, provided for an embodiment of this application; Figure 2 A schematic diagram of a binocular vision-inertial tightly coupled localization method based on a learning sparse visual front end provided in an embodiment of this application; Figure 3 A schematic diagram of a learning-based sparse correspondence generation network (MNv4-GTS) structure provided for an embodiment of this application; Figure 4 This is a schematic diagram of a semantically guided modulation (SGM) module structure provided in an embodiment of this application.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0019] Explanation of key terms: Multi-scale feature maps: These refer to feature representations with different spatial resolutions output by learned feature coding networks, such as 1 / 2, 1 / 4, 1 / 8, and 1 / 16 scale feature maps.

[0020] Semantic-guided modulation module: refers to a neural network module used to adaptively modulate and fuse the geometric features of a shallow-scale feature map with the semantic features of the deep-scale feature map.

[0021] Geometric keypoint detection head: refers to a lightweight neural network structure that directly acts on the shallow-scale feature map of the backbone network and outputs a keypoint confidence map.

[0022] Global normalized correlation matrix: refers to the global matching probability distribution matrix of candidate key points between the reference frame and the current frame calculated on the deep-scale feature map.

[0023] Coarse corresponding displacement estimation: refers to the initial displacement vector of sparse keypoints between the reference frame and the current frame, calculated using the expected coordinates.

[0024] Multi-channel adaptive damping sparse Lucas-Kanade iterative refinement: refers to a sparse optical flow refinement algorithm that is performed step-by-step on multi-scale feature maps and introduces an adaptive damping term based on the Hessian matrix trace.

[0025] Binocular keypoints in geometric filtering: These are high-quality sparse keypoints obtained after boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal from the fundamental matrix, and feature point dispersion control.

[0026] Relative pose increment and its covariance: refers to the relative motion state of the carrier and its uncertainty obtained by pre-integrating the IMU measurement data between binocular image frames.

[0027] Sliding window factor graph optimization: refers to a nonlinear optimization process based on the VINS-Fusion framework that jointly minimizes the marginalized prior residual, visual reprojection residual, and IMU pre-integration residual.

[0028] Visual inertial odometry (VIO) is a core positioning technology in fields such as autonomous robots and micro-drones. Classic tightly coupled VIO systems, such as VINS-Mono / Fusion, employ traditional feature point detection (GFTT) and optical flow (Lucas-Kanade) tracking as the visual front end, exhibiting stable performance in textured scenes. However, traditional Lucas-Kanade optical flow, based on the assumption of a single-channel image brightness gradient, shows significantly reduced robustness in scenes with weak textures, motion blur, and abrupt changes in illumination, leading to the loss of numerous feature points and deterioration or even divergence in the positioning accuracy of the VIO system. In recent years, deep learning optical flow methods (such as RAFT and GMFlow) have made significant progress in general optical flow estimation; however, these methods target dense optical flow, resulting in large model sizes that are difficult to run in real-time on computationally limited embedded platforms, and they are not optimized for the sparse correspondence quality required by VIO. Existing solutions for integrating learning-based front-ends into VIO either rely on computationally expensive feature descriptor matching (such as SuperPoint+SuperGlue), or directly embed dense optical flow networks into VIO while ignoring real-time performance, or lack robust mechanisms to handle tracking failures in weak texture scenarios.

[0029] It should be noted that the core contribution of this invention is the MNv4-GTS network (Mobile NetV4 with Geometric Tracking Suite). This network takes a grayscale single-channel image as input and uniformly completes three tasks: key point detection, temporal tracking initialization, and subpixel refinement, sharing the same backbone computing.

[0030] Figure 1 The complete pipeline is described, which inputs stereo images and raw IMU data into a learning front-end (feature encoding → key point detection → global correlation initialization → sparse LK refinement), passes through a geometric filtering pipeline, is tightly coupled to a VIO back-end, and finally outputs the pose.

[0031] refer to Figure 1 and Figure 2The first embodiment of this application provides a binocular vision inertial tight-coupled localization method based on a learning-based sparse vision front-end to solve the technical problems mentioned in the background art, such as insufficient robustness of traditional LK optical flow in scenarios with weak texture, lighting changes, and motion blur, and frequent failures in feature point tracking. This method can be executed by a processor, which can be located in a terminal or a server. The execution process of this method is as follows: Step S101: Input the current frame binocular grayscale image into the learning feature encoding network to obtain multi-scale feature maps.

[0032] It should be noted that this application runs on the ROS2 framework and uses the VINS-Fusion framework as a tightly coupled VIO backend. This application utilizes a novel learning-based sparse vision frontend to replace the original GFTT+LK traditional frontend of VINS-Fusion. The new learning-based sparse vision frontend is decoupled from the backend through a standardized Feature Frame interface (including feature ID, left eye normalized coordinates, right eye normalized coordinates, and feature velocity), so the backend does not need to be aware of the frontend implementation details.

[0033] In one embodiment of this application, the learning-based feature encoding network is a backbone network, and its output multi-scale features include four scale feature maps: 1 / 2, 1 / 4, 1 / 8, and 1 / 16. In other words, the four scale feature maps of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 are the outputs of the backbone network stages 1 to 4, respectively.

[0034] The backbone network uses MobileNetV4 (loaded from the timm library and pre-trained on ImageNet for grayscale input adaptation). MobileNetV4's multi-scale projection layers linearly project features at each scale to a compact dimension (1 / 16→32 channels, 1 / 8→16 channels, 1 / 4→8 channels, 1 / 2→8 channels). In this embodiment, the processor introduces a Bottleneck Transformer at the 1 / 16 scale for global context modeling.

[0035] refer to Figure 3 , Figure 3 This is a diagram of the Learning Sparse Correspondence Generating Network (MNv4-GTS). Figure 3 The data flow relationships of MobileNetV4 backbone, multi-scale projection layer, Bottleneck Transformer, three-layer semantic guided modulation, geometric keypoint detection head, global correlation initialization, and multi-channel sparse LK refinement are described.

[0036] Step S102: Input feature maps of multiple scales into the pre-constructed semantic-guided modulation module and output multi-scale matching feature maps. The semantic-guided modulation module is used to adaptively modulate and fuse the geometric features of the shallow scale feature maps with the semantic features of the deep scale feature maps in the multi-scale feature maps.

[0037] In one embodiment of this application, the process of adaptively modulating and fusing the geometric features of a shallow scale feature map with the semantic features of a deep scale feature map in a multi-scale feature map may include the following: The deep feature map is bicubic upsampled to the size of the shallow feature map, and then subjected to depthwise separable smooth convolution to eliminate the grid aliasing introduced by upsampling, resulting in the upsampled deep feature map.

[0038] The upsampled deep feature map is used to generate channel-gated weights through global average pooling and 1×1 convolution, and spatial-gated weights are generated through 1×1 convolution.

[0039] After multiplying the shallow geometric features element-wise with the channel gating weights and spatial gating weights, the deep semantic projection values ​​are superimposed to complete the adaptive modulation and fusion of semantic information.

[0040] In this embodiment, shallow features retain rich local geometric details (edges, corners), but lack semantic robustness to occlusion, lighting changes, etc. Deep features are semantically robust, but lose detailed geometric information. Traditional feature pyramids simply add or concatenate features, which cannot adaptively control the fusion ratio of the two types of information. Therefore, this application designs a Semantic Guided Modulation (SGM) module to solve the above problems.

[0041] For example, the following describes the processing flow of the Semantic Guided Modulation (SGM) module using a modulation process that propagates from deep semantics to shallow semantics as an example: During this modulation process, the processor can perform bicubic upsampling on the 1 / 16 scale deep feature map to a 1 / 8 scale spatial resolution, and then eliminate the grid aliasing introduced by bicubic upsampling through depthwise separable smooth convolution (3×3 depthwiseconv).

[0042] Specifically, refer to Figure 4The processor can perform global average pooling on the upsampled deep features, followed by 1×1 convolution and sigmoid activation to obtain channel-dimensional gated weight vectors. Then, it performs another 1×1 convolution and sigmoid activation to obtain a pixel-wise spatial gated weight map. Finally, the processor can element-wise multiply the 1 / 8 scale shallow geometric features with the channel gated weights and spatial gated weights, and then superimpose the 1×1 semantic projection values ​​of the deep features to obtain the modulated fused features. The expression for the above process can be as follows:

[0043]

[0044] in, This indicates the current shallow scale. This represents the result of deep semantic features after bicubic interpolation upsampling and depthwise separable smooth convolution. This indicates shallow geometric features. and These represent the channel gating weight and the spatial gating weight, respectively. This represents a 1×1 semantically injected convolution. (Symbol) This indicates element-wise multiplication.

[0045] In other words, the multi-layer semantic-guided modulation module can sequentially complete modulation from deep to shallow layers.

[0046] Step S103: Apply the binocular grayscale image to the shallow scale feature map of the backbone network through the geometric key point detection head, output the key point confidence map, and obtain candidate key points through screening.

[0047] In one embodiment of this application, the geometric key point detection head includes a 3×3 convolutional neighborhood integration layer, a batch normalization layer, a ReLU activation layer, a 1×1 output convolutional layer, a 3×3 average pooling layer, and a Sigmoid mapping layer connected in sequence.

[0048] The processor can directly apply the geometric keypoint detection head to the 1 / 4 scale feature map output by stage2, and output a keypoint confidence map.

[0049] In other words, the geometric keypoint detection head can directly reuse the shallow features at the 1 / 4 scale of the backbone network to output a keypoint confidence map. The structure of the geometric keypoint detection head is a 3×3 convolution, which is used for neighborhood integration to prevent isolated noise points. Then, it outputs a single-channel confidence map through BatchNorm→ReLU→1×1 convolution→3×3 average pooling→Sigmoid activation.

[0050] The key computational features of the geometric key point detection head are: the detection head and the backbone share forward computation, and no independent heavy detector branches are introduced, so the computational overhead is only a small number of convolutional layers.

[0051] In actual computation, the processor needs to perform local maximum filtering and threshold filtering on the confidence map on the GPU, and retain the top Nmax candidate points by score. When the number of surviving points in time-series tracking is insufficient, the processor further filters candidates that are too close to existing tracking points and adds new keypoints to fill the maximum number of points.

[0052]

[0053]

[0054]

[0055] in, This represents a score chart for key points. Indicates position Candidate keypoint masks that satisfy local maxima, threshold, and effective region constraints. Indicates the effective detection area. This indicates the scoring threshold. Indicates The set of key points after retention. This indicates the maximum number of key points to retain.

[0056] Step S104: Calculate the global normalized correlation matrix of candidate key points between the reference frame and the current frame on the deep scale feature map. After probability truncation and renormalization, obtain the coarse corresponding displacement estimate of sparse key points by sampling.

[0057] In one embodiment of this application, the process of calculating the global normalized correlation matrix of candidate keypoints between the reference frame and the current frame on the deep-scale feature map, and obtaining the coarse corresponding displacement estimate of sparse keypoints by sampling after probability truncation and renormalization can be represented as follows: Calculate the normalized feature vectors of the 1 / 16 scale feature maps of two adjacent frames, and calculate the scaling cosine correlation of any two positions in the two adjacent frames based on the two normalized feature vectors.

[0058] The truncation probability is calculated based on the normalized scaled cosine correlation.

[0059] The truncation probability is calculated based on the normalized scaled cosine correlation, and the truncation probability is normalized to obtain a dense coarse displacement field. Sampling is performed in a dense displacement field to obtain coarse correspondence estimates of sparse key points.

[0060] Traditional sparse LK algorithms use the current position as the initial value, which can easily lead to local optima in scenarios with large displacements or large parallax. This application designs a global correlation initialization module to solve the above-mentioned technical problems.

[0061] Specifically, the processor can establish a global matching prior between the reference frame and the current frame based on the 1 / 16 scale features of the backbone network. In actual execution, the processor can project and normalize the feature maps of the two frames using a 1×1 convolution, and then calculate the global cosine similarity matrix to obtain a dense coarse displacement field. Sampling is then performed in the dense displacement field to obtain a coarse correspondence estimate of sparse keypoints.

[0062] Then, the processor maps the coarse displacement estimate of each keypoint to the current image space, serving as the initial values ​​for subsequent multi-scale LK refinement. Therefore, the global correlation initialization module enables the front-end to have global search capabilities, significantly reducing the convergence difficulty of LK refinement, especially in scenarios with large binocular parallax or rapid camera movement, thus significantly improving tracking success rate. The mathematical expression for the global correlation initialization module can be:

[0063]

[0064]

[0065]

[0066]

[0067] Among them, C ij Indicates the reference frame number i The position is related to the current frame. j Scaling cosine correlation between locations. f i ref and f j cur These represent the normalized feature vectors of the reference frame and the current frame, respectively. p cut,ij This represents the probability after truncation at a threshold of 0.01. p norm,ij This represents the truncation probability after renormalization. Δx i 0 and Δy i 0 This represents the coarse initialization displacement, where I represents the cutoff factor, and x represents the initial displacement. i x j These represent the x-coordinate and y-coordinate of a point in the reference frame and the current frame, respectively. i yj These represent a y-coordinate point in the reference frame and the current frame, respectively.

[0068] Step S105: Using the coarse corresponding displacement estimate as the initial value, perform multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on the multi-scale feature map step by step to obtain sparse key points with accurate sub-pixel correspondence.

[0069] In one embodiment of this application, the process of performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on multi-scale feature maps to obtain sub-pixel-level precisely corresponding sparse keypoints can be represented as follows: When performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement: Multi-channel patch alignment is performed on multi-scale feature maps to reduce the refinement failure rate of weak texture regions.

[0070] An adaptive damping term based on the matrix trace is introduced when inverting the Hessian matrix. The damping coefficient is proportional to the trace of the Hessian matrix to prevent numerical collapse in weak texture regions.

[0071] The region outside the feature map boundary is sampled using the border extrapolation mode to avoid the failure of corresponding fine-tuning of key points at the image edge.

[0072] In this embodiment, the present application provides a multi-channel adaptive damped sparse LK module that performs sparse Lucas-Kanade iterative refinement on a learned multi-channel feature map.

[0073] Unlike traditional LK modules, the channel adaptive damped sparse LK module has the following three key improvements: Multi-channel feature alignment: The processor can use this module to construct patches on multi-scale feature maps, and reduce the failure rate of fine-tuning in weak texture regions by leveraging the robustness of multi-channel semantic features.

[0074] Adaptive damping based on the Hessian matrix trace: The processor can use this module to add a damping term λ=max(0.01×tr(H),10^-3) proportional to its trace to the Hessian matrix when calculating the LK update step size, so that the inversion value in the weak texture region (where Hessian is close to singular) is stable, avoiding the step size explosion of the traditional LK in the low gradient region.

[0075]

[0076]

[0077] in, This indicates the local window sampling area. Indicates the feature channel index. This represents the two-dimensional gradient Jacobian at the current position. This indicates the error between the current feature block and the reference feature block at the corresponding channel and pixel. Representation matrix The traces. This represents the two-dimensional displacement increment of the current iteration.

[0078] The processor can use this module to extrapolate the coordinates of sampling points that exceed the boundary of the feature map from the nearest boundary pixel, thereby ensuring the physical consistency of feature sampling of key points at the image edge during multi-scale refinement.

[0079] The complete refinement chain of the channel adaptive damped sparse LK module can be based on a coarse-to-fine strategy. Specifically, the processor can use this module to execute a complete refinement chain of global correlation initialization (1 / 16 scale) → multi-channel sparse LK (1 / 8 scale) → multi-channel sparse LK (1 / 4 scale) → multi-channel sparse LK (1 / 2 scale). In this way, due to computational resource constraints, the 1 / 8 scale of stereo matching is omitted, and the stereo matching process can be represented as: currently global correlation initialization → multi-channel sparse LK (1 / 4 scale) → multi-channel sparse LK (1 / 2 scale).

[0080] In one embodiment of this application, the order of performing the complete time-series tracking of multi-channel adaptive damped sparse Lucas-Kanade iterative refinement is 1 / 8 scale feature map, 1 / 4 scale feature map, and 1 / 2 scale feature map.

[0081] The processor can execute multi-channel adaptive damped sparse Lucas-Kanade iterative refinement of binocular stereo matching in the order of 1 / 4 scale feature map and 1 / 2 scale feature map.

[0082] Step S106: For sparse keypoints with precise sub-pixel correspondence, perform boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control sequentially. If the number of surviving points is insufficient, supplement with new detected keypoints to obtain geometrically filtered binocular keypoints. Perform right-eye matching on the left-eye keypoints in the binocular keypoints, and perform boundary validity checks, disparity range constraints, epipolar threshold constraints, bidirectional consistency error checks, and RANSAC outlier removal sequentially to obtain matching feature points for the left-eye and previous frames, and matching feature points for the left and right eyes in the current frame.

[0083] In other words, the processor can use a predetermined radius as the neighborhood range. When the number of keypoints in the neighborhood exceeds the upper limit, it will prioritize retaining stable keypoints that have been tracked for more frames and suppress the remaining keypoints.

[0084] Step S107: Pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance.

[0085] Step S108: The matching feature points of the left and right frames and the matching feature points of the left and right targets in the current frame, along with the relative pose increment and its covariance, are fed into the sliding window factor map optimization. The marginalization prior residual, visual reprojection residual and IMU pre-integration residual are jointly minimized to output the six-degree-of-freedom pose estimate of the carrier.

[0086] Among them, the state variables to be optimized in the sliding window factor graph include the position, velocity, attitude, IMU bias, and camera extrinsic parameters of each keyframe.

[0087] Specifically, this application first designs a systematic geometric filtering strategy for VIO application scenarios to filter out erroneous matches contained in matching pairs. The systematic geometric filtering strategy may include the following: The processor can perform bidirectional consistency error checks, running forward and backward tracking simultaneously. When the difference between the forward point and the backward projection point exceeds a threshold (1.0 pixel for timing tracking, 1.5 pixels for binocular matching), the corresponding point is discarded.

[0088] The processor can perform CrowdSuppression feature point dispersion control. Specifically, with a radius of 25 pixels as the neighborhood, when the number of keypoints in the neighborhood exceeds 4, the stable point with the most tracking frames is retained first, and the remaining keypoints are suppressed. This ensures that the feature points in the whole image have good spatial dispersion and prevents the observability of the VIO backend from being degraded due to feature clustering.

[0089] In one embodiment of this application, the binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end further includes: For rectified stereo image pairs, retain matching points where the x-coordinate of the key point in the left image is greater than the x-coordinate of the key point in the right image, the Y-direction deviation is within the threshold, and the disparity is within the effective range.

[0090] For non-rectified raw inputs, no hard threshold constraint is applied in the Y direction; instead, geometric consistency screening is performed using the fundamental matrix RANSAC.

[0091] The processor can retain only the binocular matching points in the rectified binocular image pair where the left image's x-coordinate is greater than the right image's x-coordinate, the Y-direction deviation is within 1.5 pixels, and the disparity is within the effective range (0~128 pixels). For the non-rectified original binocular input, the processor does not apply the aforementioned hard constraint in the Y-direction, but instead uses the fundamental matrix RANSAC for geometric consistency screening.

[0092] The processor can eliminate tracking points with globally geometrically inconsistent characteristics by performing RANSAC (using the existing VINS-Fusion framework) on the timing tracking points. It should be noted that the processor also performs RANSAC on stereo matching points. The output of the above process is the output of the front end.

[0093] It is understandable that, since this application is aimed at stereo cameras, their control points are more dispersed and satisfy epipolar constraints (the stereo cameras are fixed and both are facing forward, so there will be no situation where a point below one camera matches the point above another camera). RANSAC is to remove mismatched points, and bidirectional consistency is to track from the left eye to the right eye, and then from the right eye to the left eye. If the error is less than a certain threshold, it is retained.

[0094] In summary, this application implements a tightly coupled VIO backend based on a sliding window factor graph using the VINS-Fusion framework. The VINS-Fusion framework includes modules for IMU pre-integration, initialization, nonlinear optimization, and edge detection. This application encapsulates the front-end output into a FeatureFrame data structure compatible with the VINS-Fusion interface through a processor, seamlessly integrating the aforementioned learning-based frontend into an existing backend optimization framework. The state variables to be optimized within the backend sliding window include the position, velocity, pose, IMU bias, and camera extrinsic parameters of each keyframe. The optimization jointly minimizes the IMU pre-integration residual and the visual reprojection residual, using a Doggleg trust region strategy for iterative solution. This invention does not modify the backend optimization algorithm itself; the backend implementation is based on the VINS-Fusion open-source framework.

[0095] The state variables to be optimized within the back-end sliding window include the position, velocity, pose, IMU bias, and camera extrinsic parameters of each keyframe. The expression for jointly minimizing the IMU pre-integration residual and the visual reprojection residual is as follows:

[0096] in, x This indicates the state of the sliding window to be optimized, including parameters such as pose, velocity, bias, extrinsic parameters, and time delay. x * This represents the optimized state estimate. r prior This represents the marginalized prior residual. r IMU This represents the IMU pre-integration residual. r rep This represents the visual reprojection residual. ρ(·) This represents a robust kernel function.

[0097] It is understandable that the front-end and back-end optimizations adopt CeresSolver's Doglg trust region strategy and DENSE_SCHUR linear solver. Therefore, this paper will not list the fixed-form incremental formula to avoid mistakenly writing the internal steps of the solver as the explicit analytical update of this invention.

[0098] To address the aforementioned technical problems, the second embodiment of this application provides a binocular vision-inertial tightly coupled positioning device based on a learning-based sparse visual front-end, solving the same technical problems as the method embodiment. This device may include the following modules: a multi-scale feature encoding module, a semantic-guided modulation module, a keypoint detection module, a global correlation initialization module, a multi-channel LK refinement module, a geometric filtering and binocular matching module, an IMU pre-integration module, and a tightly coupled optimization module. The multi-scale feature encoding module inputs the current frame binocular grayscale image into a learning-based feature encoding network to obtain multi-scale feature maps. The semantic-guided modulation module inputs feature maps of multiple scales into a pre-constructed semantic-guided modulation module and outputs a multi-scale matching feature map. The semantic-guided modulation module adaptively modulates and fuses the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map. The keypoint detection module applies the binocular grayscale image to the shallow scale feature map of the backbone network through a geometric keypoint detection head, outputs a keypoint confidence map, and obtains candidate keypoints through screening. The global correlation initialization module calculates the global normalized correlation matrix of candidate keypoints between the reference frame and the current frame on the deep-scale feature map. After probability truncation and renormalization, it obtains a coarse corresponding displacement estimate of the sparse keypoints through sampling. The multi-channel LK refinement module uses the coarse corresponding displacement estimate as the initial value and performs multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on the multi-scale feature map step by step to obtain sparse keypoints with sub-pixel-level accurate correspondence. The geometric filtering and binocular matching module performs boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control on the sparse keypoints with sub-pixel-level accurate correspondence in sequence. When the number of surviving points is insufficient, it supplements the detected keypoints to obtain geometrically filtered binocular keypoints. Right-eye matching is performed on the left-eye keypoints in the binocular keypoints, and boundary validity checks, disparity range constraints, epipolar threshold constraints, bidirectional consistency error checks, and RANSAC outlier removal are performed in sequence to obtain the matching feature points of the left-eye previous and next frames and the matching feature points of the left and right eyes in the current frame. The IMU pre-integration module is used to pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance. The tightly coupled optimization module uses the matching feature points of the left and right targets in the previous and following frames, along with the matching feature points of the left and right targets in the current frame, to feed the relative pose increment and its covariance into a sliding window factor map for optimization. This jointly minimizes the marginalization prior residual, the visual reprojection residual, and the IMU pre-integration residual, outputting a six-DOF pose estimate of the carrier. It is understood that this embodiment is a device / system weight corresponding to the method weight; therefore, all the technical effects of the method embodiment are present in this embodiment and will not be repeated here.

[0099] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end, characterized in that, include: Input the current frame's binocular grayscale image into a learning feature encoding network to obtain multi-scale feature maps; The feature maps of multiple scales are input into a pre-constructed semantic-guided modulation module, which outputs a multi-scale matching feature map. The semantic-guided modulation module is used to adaptively modulate and fuse the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map. The binocular grayscale image is applied to the shallow scale feature map of the backbone network through the geometric key point detection head, and the key point confidence map is output. Candidate key points are obtained after screening. The global normalized correlation matrix of candidate key points between the reference frame and the current frame is calculated on the deep scale feature map. After probability truncation and renormalization, the coarse corresponding displacement estimate of sparse key points is obtained by sampling. Using the coarse corresponding displacement estimate as the initial value, multi-channel adaptive damped sparse Lucas-Kanade iterative refinement is performed step by step on the multi-scale feature map to obtain sparse key points with accurate sub-pixel correspondence. The left-eye temporal tracking key points corresponding to the sub-pixel level are sequentially subjected to boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control. When the number of surviving points is insufficient, new detection key points are added to obtain the left-eye retained key points of the current frame. The right-eye matching is performed with the left-eye retained key points of the current frame as input, and the boundary validity checks, disparity range constraints, epipolar threshold constraints under rectified binocular conditions, bidirectional consistency error checks, and RANSAC outlier removal are sequentially performed to obtain the matching feature points of the left-eye previous and next frames and the matching feature points of the left and right eyes of the current frame. Pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance; The matching feature points of the left and right frames and the matching feature points of the left and right targets in the current frame, along with the relative pose increment and its covariance, are fed into the sliding window factor map for optimization. The marginalization prior residual, visual reprojection residual and IMU pre-integration residual are jointly minimized to output the six-degree-of-freedom pose estimate of the carrier.

2. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 1, characterized in that, The adaptive modulation and fusion of the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map includes: The deep feature map is bicubic upsampled to the size of the shallow feature map, and then subjected to depthwise separable smooth convolution to eliminate the grid aliasing introduced by upsampling, resulting in the upsampled deep feature map. The upsampled deep feature map is used to generate channel-gated weights through global average pooling and 1×1 convolution, and spatial-gated weights are generated through 1×1 convolution. After multiplying the shallow geometric features element-wise with the channel gating weights and spatial gating weights, the deep semantic projection values ​​are superimposed to complete the adaptive modulation and fusion of semantic information.

3. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 1, characterized in that, The learning-based feature encoding network is the backbone network, and its output multi-scale features include four scale feature maps: 1 / 2, 1 / 4, 1 / 8, and 1 / 16. The four scale feature maps of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 are the outputs of the backbone network stage 1 to stage 4, respectively. The geometric key point detection head includes a 3×3 convolutional neighborhood integration layer, a batch normalization layer, a ReLU activation layer, a 1×1 output convolutional layer, a 3×3 average pooling layer, and a sigmoid mapping layer connected in sequence. The geometric keypoint detection head is directly applied to the 1 / 4 scale feature map output by stage2 to output a keypoint confidence map.

4. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 3, characterized in that, The step of calculating the global normalized correlation matrix of candidate keypoints between the reference frame and the current frame on the deep-scale feature map, and obtaining a coarse corresponding displacement estimate of sparse keypoints through sampling after probability truncation and renormalization includes: Calculate the normalized feature vectors of the 1 / 16 scale feature maps of two adjacent frames, and calculate the scaling cosine correlation of any two positions in the two adjacent frames based on the two normalized feature vectors. The truncation probability is calculated based on the normalized scaled cosine correlation, and the truncation probability is normalized to obtain a dense coarse displacement field. Sampling is performed in a dense displacement field to obtain coarse correspondence estimates of sparse key points.

5. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 3, characterized in that, The process of performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on multi-scale feature maps to obtain sub-pixel-level precisely corresponding sparse keypoints includes: When performing multi-channel adaptive damped sparse Lucas-Kanade iterative refinement: Multi-channel patch alignment is performed on multi-scale feature maps to reduce the refinement loss rate in weak texture regions. An adaptive damping term based on the matrix trace is introduced when inverting the Hessian matrix. The damping coefficient is proportional to the trace of the Hessian matrix to prevent numerical collapse in weak texture regions. The region outside the feature map boundary is sampled using the border extrapolation mode to avoid failure of corresponding fine-tuning of key points at the image edge.

6. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 5, characterized in that, The complete time-series tracking of the step-by-step execution of multi-channel adaptive damped sparse Lucas-Kanade iterative refinement is performed in the order of 1 / 8 scale feature map, 1 / 4 scale feature map, and 1 / 2 scale feature map. The stereo matching sequence for progressively refining the multi-channel adaptive damped sparse Lucas-Kanade iterative process is 1 / 4 scale feature map and 1 / 2 scale feature map.

7. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 1, characterized in that, The feature point dispersion control includes: Using a predetermined radius as the neighborhood range, when the number of keypoints in the neighborhood exceeds the upper limit, stable keypoints with more tracking frames are retained first, while the remaining keypoints are suppressed.

8. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 1, characterized in that, Also includes: For rectified stereo image pairs, retain matching points where the x-coordinate of the key point in the left image is greater than the x-coordinate of the key point in the right image, the Y-direction deviation is within the threshold, and the disparity is within the effective range. For non-rectified raw inputs, no hard threshold constraint is applied in the Y direction; instead, geometric consistency screening is performed using the fundamental matrix RANSAC.

9. The binocular vision-inertial tightly coupled localization method based on a learning-based sparse visual front end according to claim 1, characterized in that, The state variables to be optimized in the sliding window factor graph include the position, velocity, attitude, IMU bias, and camera extrinsic parameters of each keyframe.

10. A binocular vision-inertial tightly coupled positioning device based on a learning-based sparse visual front end, characterized in that, include: The multi-scale feature encoding module is used to input the current frame's binocular grayscale image into the learning feature encoding network to obtain multi-scale feature maps; The semantic-guided modulation module is used to input feature maps of multiple scales into the pre-constructed semantic-guided modulation module and output multi-scale matching feature maps. The semantic-guided modulation module is used to adaptively modulate and fuse the geometric features of the shallow scale feature map with the semantic features of the deep scale feature map in the multi-scale feature map. The key point detection module is used to apply the binocular grayscale image to the shallow scale feature map of the backbone network through the geometric key point detection head, output the key point confidence map, and obtain candidate key points through screening. The global correlation initialization module is used to calculate the global normalized correlation matrix of candidate key points between the reference frame and the current frame on the deep scale feature map. After probability truncation and renormalization, the coarse corresponding displacement estimate of sparse key points is obtained by sampling. The multi-channel LK refinement module is used to perform multi-channel adaptive damped sparse Lucas-Kanade iterative refinement on multi-scale feature maps step by step, using the coarse corresponding displacement estimate as the initial value, to obtain sparse key points with accurate sub-pixel correspondence. The geometric filtering and binocular matching module is used to sequentially perform boundary validity checks, bidirectional consistency error checks, RANSAC outlier removal, and feature point dispersion control on the sub-pixel level precisely corresponding left-eye temporal tracking key points. When the number of surviving points is insufficient, new detection key points are added to obtain the left-eye retained key points of the current frame. The right-eye matching is performed with the left-eye retained key points of the current frame as input, and boundary validity checks, disparity range constraints, epipolar threshold constraints under rectified binocular conditions, bidirectional consistency error checks, and RANSAC outlier removal are sequentially performed to obtain the matching feature points of the previous and next left-eye frames and the matching feature points of the left and right eyes of the current frame. The IMU pre-integration module is used to pre-integrate the IMU measurement data between binocular image frames to calculate the relative pose increment and its covariance. The tightly coupled optimization module is used to feed the matching feature points of the left and right frames and the matching feature points of the left and right targets in the current frame, along with the relative pose increment and its covariance, into the sliding window factor map optimization. It jointly minimizes the marginalized prior residual, the visual reprojection residual, and the IMU pre-integration residual, and outputs the six-degree-of-freedom pose estimate of the carrier.

Images