Vision-inertial odometry method and system based on motion constraints and uncertainty constraints

Abstract

The application provides a visual inertial odometer method and system based on motion constraint and uncertainty constraint. The method performs pre-integration and feature coding on synchronous images and inertial data, aligns multiple frames of visual features by using inertial motion prior, and aggregates them according to observation reliability. A motion adaptive sparse graph is constructed, visual and inertial selective state propagation is performed on the graph, and the graph topology and cross-modal fusion are adjusted by local observability score. The inertial pre-integrated pose is used as a reference to decode the Lie algebra pose residual, the relative pose and covariance are obtained through SE(3) exponential mapping, and the observability score and pose covariance are fed back to subsequent feature propagation and fusion. The method can reduce the pose estimation error caused by large motion, visual degradation and inertial noise, and improve the stability and robustness of long-time visual inertial odometer.

Description

Technical Field

[0001] This application belongs to the field of visual positioning technology, and in particular relates to a visual inertial odometry method and system based on motion constraints and uncertainty constraints. Background Technology

[0002] Accurate and robust real-time pose estimation is a core prerequisite for applications such as autonomous driving, drones, mobile robots, and augmented reality. Visual-Inertial Odometry (VIO) technology, by integrating the rich environmental information from visual sensors with the high-frequency motion data from inertial measurement units (IMUs), can achieve complementary advantages and has become one of the mainstream positioning solutions.

[0003] Existing deep learning-based visual inertial odometry typically extracts visual and inertial features separately, then fuses them through feature concatenation, attention weighting, or gating mechanisms, and regresses the pose of adjacent frames based on recurrent or fully connected networks. While this approach can achieve pose estimation when sensor observation quality is relatively stable, significant changes occur when large translations, rapid rotations, image blurring, local occlusion, low-texture regions, or IMU noise and bias variations arise. The proportion of effective information from different modalities changes markedly, and fixed fusion paths or fusion weights relying solely on the features themselves fail to adequately reflect the reliability of the current observations.

[0004] Furthermore, fixed convolutional meshes, fixed spatiotemporal receptive fields, or discrete-time recurrent networks typically cannot directly alter the information propagation path based on motion direction, amplitude, and sampling interval. This can easily lead to visual feature misalignment and temporal error accumulation during long-term operation and in non-uniform motion scenarios. While some methods output pose uncertainty or sensor noise covariance, this uncertainty is often used for backend weighting or result evaluation, without further inverse constraints on feature propagation, graph structure, and cross-modal fusion in subsequent frames. Summary of the Invention

[0005] In view of this, this application aims to propose a visual inertial odometry method and system based on motion constraints and uncertainty constraints, in order to solve the problems in existing deep learning visual inertial odometry, such as the difficulty in adaptively fusing visual and inertial information with the reliability of observation, insufficient long-term motion dependence modeling ability, and lack of geometric constraints and uncertainty closed-loop feedback in pose regression.

[0006] To achieve the above objectives, the technical solution of this application is implemented as follows: In a first aspect, this application provides a visual inertial odometry method based on motion constraints and uncertainty constraints, including: Continuous image frames and synchronous inertial measurement unit (IMU) data are acquired. By pre-integrating the IMU data, the inertial relative motion prior and its covariance are obtained. Based on the inertial relative motion prior, the visual features of multiple frames within a local time window are projected and aligned. The aggregation weights are determined according to the visual residuals and inertial uncertainties to obtain the visual features. By performing continuous-time selective state-space encoding on the IMU data, the zero bias, observation confidence, and covariance are estimated to obtain the inertial features. Based on the visual features, inertial features, and inertial relative motion priors, a motion-adaptive sparse graph is constructed. The graph neighborhood and propagation weights are determined based on spatial relationships, motion directions, visual feature similarities, and inertial uncertainties. Visual state and inertial state propagation are performed on the motion-adaptive sparse graph. The fusion features are obtained based on the local observational scoring adjustment graph topology and cross-modal fusion gating determined by the Jacobian matrix and joint covariance of the visual reprojection residual and the inertial prediction residual. Using the inertial pre-integrated pose as the geometric reference, the fusion features are decoded to obtain the six-dimensional Lie algebraic pose residual and pose uncertainty. The pose residual is applied to the geometric reference through exponential mapping to obtain the SE(3) relative pose and pose covariance. The local observation score and pose covariance are then fed back to the visual feature aggregation, graph neighborhood selection and fusion gating update of subsequent frames.

[0007] Secondly, based on the same inventive concept, this application also provides a visual inertial odometry system based on motion constraints and uncertainty constraints, comprising: The inertial pre-integration module is used to obtain the prior inertial relative motion and its covariance based on the data from the synchronous inertial measurement unit. The visual feature propagation module is used to perform motion compensation on multiple frames of visual features based on the inertial relative motion prior and aggregate them according to observation reliability. The inertial state coding module is used to perform continuous-time selective state-space coding on inertial measurement unit data and output inertial characteristics, observation confidence, and covariance. The graph fusion module is used to construct motion-adaptive sparse graphs, perform visual and inertial state propagation, and utilize local observational scoring for graph topology filtering and cross-modal fusion. The residual pose decoding module is used to output the relative pose and pose covariance of SE(3) based on the inertial pre-integrated pose as the geometric reference. The feedback module is used to feed back the local observational scores and pose covariance to the visual feature propagation and graph fusion process of subsequent frames.

[0008] Thirdly, based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0009] Fourthly, based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in the first aspect.

[0010] Compared with existing technologies, the visual inertial odometry method and system based on motion constraints and uncertainty constraints described in this application have the following advantages: This application integrates inertial pre-integration motion priors and their uncertainties into the visual feature propagation, inertial state encoding, cross-modal graph fusion, and SE(3) pose decoding processes. It also dynamically adjusts feature transmission, graph topology, and fusion weights using local observational scoring, while feeding the pose covariance back to subsequent frame processing. This improves the reliability of visual and inertial information fusion in scenarios with large-scale motion, image blurring, local occlusion, and sensor noise changes, reduces feature misalignment and long-term error accumulation, and ensures geometric consistency of pose updates. Ultimately, this enhances the pose estimation accuracy, stability, and adaptability of visual inertial odometry in complex motion environments. Attached Figure Description

[0011] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a visual inertial odometry method based on motion constraints and uncertainty constraints as described in an embodiment of this application; Figure 2 This is a diagram of the overall network framework as described in the embodiments of this application; Figure 3 This is a schematic diagram of the motion adaptive sparse graph fusion module described in an embodiment of this application; Figure 4 This is a schematic diagram of a visual inertial odometry system based on motion constraints and uncertainty constraints, as described in an embodiment of this application. Figure 5 This is a schematic diagram of the hardware structure of the electronic device described in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0013] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0014] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0015] Please see Figure 1 As shown, a visual inertial odometry method based on motion constraints and uncertainty constraints specifically includes the following steps: Step S10: Acquire continuous image frames and synchronous inertial measurement unit (IMU) data. By pre-integrating the IMU data, obtain the inertial relative motion prior and its covariance. Based on the inertial relative motion prior, project and align the visual features of multiple frames within the local time window, and determine the aggregation weights based on the visual residuals and inertial uncertainties to obtain the visual features. By performing continuous-time selective state-space encoding on the IMU data, estimate the zero bias, observation confidence, and covariance to obtain the inertial features. Step S20: Construct a motion-adaptive sparse graph based on visual features, inertial features, and inertial relative motion priors. Determine the graph neighborhood and propagation weights based on spatial relationships, motion directions, visual feature similarities, and inertial uncertainties. Perform visual state and inertial state propagation on the motion-adaptive sparse graph. Obtain fusion features based on the local observational scoring adjustment graph topology and cross-modal fusion gating determined by the Jacobian matrix and joint covariance of the visual reprojection residual and the inertial prediction residual. Step S30: Using the inertial pre-integration pose as the geometric reference, decode the fusion features to obtain the six-dimensional Lie algebra pose residual and pose uncertainty. Apply the pose residual to the geometric reference through exponential mapping to obtain the SE(3) relative pose and pose covariance. Feedback the local observation score and pose covariance to the visual feature aggregation, graph neighborhood selection and fusion gating update of subsequent frames.

[0016] The method described in this embodiment injects the motion priors and uncertainties obtained from inertial pre-integration into three main modules: visual feature propagation, cross-modal graph state-space fusion, and pose decoding, thereby constructing a closed-loop feedback mechanism from observation reliability to pose estimation. The overall network framework diagram is shown below. Figure 2 As shown, the process includes a feature extraction stage, a feature fusion stage, and a pose regression stage. The feature extraction stage processes the RGB image sequence and the IMU sequence respectively. The IMU pre-integration result serves as both the motion prior for visual feature propagation and the geometric reference for pose decoding. The feature fusion stage performs visual state propagation, inertial state propagation, and asymmetric fusion on the motion-adaptive sparse graph and calculates the local observation score. The pose regression stage outputs the poses of adjacent frames SE(3) and their covariances, and feeds back the uncertainty to subsequent time steps.

[0017] This method integrates the inertial pre-integration motion prior and its uncertainty into the visual feature propagation, inertial state encoding, cross-modal graph fusion, and SE(3) pose decoding processes. It also dynamically adjusts feature transmission, graph topology, and fusion weights using local observational scoring, while feeding the pose covariance back to subsequent frame processing. This improves the reliability of visual and inertial information fusion in scenarios with large motion, image blurring, local occlusion, and sensor noise changes, reduces feature misalignment and long-term error accumulation, and ensures the geometric consistency of pose updates. Ultimately, it enhances the pose estimation accuracy, stability, and adaptability of visual inertial odometry in complex motion environments.

[0018] In some implementations, obtaining visual features includes: A local time window is established with the current image frame as a reference, and the inertial measurement unit data corresponding to adjacent image frames within the window are pre-integrated; The features of historical frames are projected onto the coordinate system of the current frame by the inverse depth and inertial relative motion prior of the current image frame. The photometric residual, gradient residual and inertial uncertainty penalty term determined by the pre-integrated covariance are calculated after projection alignment. The position-related normalized aggregation weights are determined based on the photometric residual, gradient residual and inertial uncertainty penalty term. The aligned multi-frame features are weighted and aggregated according to the normalized aggregation weights, and then combined with the residual term of the current frame features to obtain the visual features.

[0019] Specifically, in this embodiment, the camera acquires a continuous sequence of images. The inertial measurement unit (IMU) acquires triaxial acceleration and triaxial angular velocity sequences corresponding to the time range of the image sequence. Using the current image frame... For reference, a preset number of image frames are selected before and after the current frame to form a local time window. ,in, This represents the time index relative to the current frame.

[0020] By pre-integrating the IMU data corresponding to the local time window. Let the... The measurement of each IMU sampling point is In the formula, It is a triaxial acceleration. For the three-axis angular velocity; the zero bias estimate is In the formula, and If the accelerometer and gyroscope have zero bias respectively, then the measurement after bias correction can be expressed as: ; By pre-integrating the bias-corrected IMU measurements on the Lie group SE(3), the results are obtained from time [time]. At the time inertial relative motion prior In the formula, , To translate the number of Li Dai, Let be the number of rotational Lie substitutions. The corresponding pre-integral covariance can be obtained from the propagation of noise and zero-biased random walk: ; In the formula, The Jacobian matrix of the pre-integration result with respect to measurement noise. The Jacobian matrix of the pre-integral result with respect to zero bias. To measure the noise covariance, The covariance of the zero-biased random walk is... Let be the initial covariance at the start of the pre-integration.

[0021] Each image frame within a local time window is input into the visual backbone network to obtain multi-scale visual features. And estimate the current frame in the 1st epoch by inverse depth branching. Reference inverse depth at each scale For feature map location By utilizing inertial motion priors to project historical frame features onto the current reference frame, the alignment position can be represented as: ; In the formula, For camera projection function, Let be the camera back projection function. This indicates that the six-dimensional Lie algebra vector is converted into the matrix representation corresponding to SE(3).

[0022] Calculate the aligned photometric residual for each historical frame. Gradient residual and inertial uncertainty penalty term .For example: ; ; ; In the formula, Represents the trace of a matrix. This is the penalty coefficient for changes in motion.

[0023] Based on the above residual calculation of the location-related normalized aggregate weights, these weights jointly characterize the visual matching quality, local texture richness, and confidence level of the inertial prior: ; In the formula, , and These are the weighting coefficients for the photometric residual, gradient residual, and inertial uncertainty penalty term, respectively. The aligned historical frame features are then processed according to... Weighted and combined with the residual features of the current frame: ; In the formula, For network units that perform residual mapping on the features of the current frame, The output visual features.

[0024] When the visual residual between the historical frame and the current frame increases, or the corresponding IMU pre-integration covariance increases, the aggregation weight reduces the contribution of the historical frame features, thereby effectively suppressing the influence of large motion, image blurring and local occlusion on feature aggregation; therefore, the effective temporal neighborhood of visual feature propagation is jointly determined by motion prior and observation uncertainty.

[0025] In terms of visual feature transmission, this embodiment uses visual features obtained by inertial pre-integration for alignment, and combines local observational scoring to adaptively adjust feature aggregation weights, thereby improving the accuracy of feature correspondence in scenarios with large translation, rapid rotation, image blurring, and partial occlusion, and reducing feature misalignment and loss of effective information.

[0026] In some implementations, continuous-time selective state-space coding includes: The inertial input is constructed based on the triaxial acceleration, triaxial angular velocity, corresponding first-order difference components, and the coupling amount between acceleration and angular velocity of adjacent inertial sampling points. The state transition parameters, input parameters, and time step are adaptively generated based on inertial input, and the inertial hidden state is recursively derived in a continuous time state update manner. The accelerometer zero-bias increment, gyroscope zero-bias increment, inertial observation confidence, and inertial observation covariance are predicted from the inertial hidden state. The inertial hidden state is then subjected to time interval pooling to obtain the inertial features corresponding to the visual feature grid.

[0027] Specifically, in this embodiment, an inertial input is constructed for adjacent IMU sampling points. Among them, inertial input At least include the current acceleration Current angular velocity First-order difference of acceleration First-order difference of angular velocity And reflecting the coupling relationship between translational excitation and rotational excitation Its formula is: By including raw quantities, changing quantities, and coupled quantities, state-space encoders are able to distinguish between different motion states such as stable motion, rapid rotation, and sudden acceleration changes.

[0028] The inertial input sequence is fed into a continuous-time selective state-space unit. For the first... Each sampling point, the state space unit according to The state transition matrix is ​​generated through a learnable mapping. Input matrix and effective time step The inertial hidden state is recursively derived according to the continuous-time state update relationship: ; in, ; In the formula, For the first The inertial hidden state of each sampling point; Controlling state decay and inter-state coupling; Injecting control inputs into the hidden state; This represents the valid state update step size determined by the current input; and They respectively represent the current number of... Input features of each inertial sampling point Extracted step size control features and state transition control features; The learnable weight vector for step size control; is the learnable mapping matrix for the state transition parameters.

[0029] The accelerometer zero-bias increment, gyroscope zero-bias increment, inertial observation confidence, and inertial observation covariance are predicted from the inertial hidden state. The zero-bias increment is superimposed onto the current zero-bias estimate after bounded activation. The inertial observation confidence reflects the reliability of the current inertial segment, and the inertial observation covariance is used for subsequent graph fusion and uncertainty propagation. The formula can be expressed as: , ; , ; In the formula, For the observation confidence level of the inertial channel, The observed covariance estimated by the network. , , This indicates the corresponding preset mapping parameters.

[0030] Temporal interval pooling and spatial mapping are performed on the inertial hidden states within the time interval of a corresponding image frame to obtain the inertial features corresponding to the current visual feature grid. And retain the inertial observation covariance for that time interval. and inertial observation confidence This serves as an additional state variable for subsequent graph fusion.

[0031] In terms of inertial time-series modeling, this embodiment adopts continuous-time selective state-space units, which adaptively generate state transition matrices, input matrices and time steps based on inertial input. This can adapt to different sampling rates, enhance the ability to model long-term motion dependence, and reduce inertial pre-integration errors and long-term drift.

[0032] In some implementations, the motion-adaptive sparse graph uses the spatial location of visual features as nodes, and the candidate neighborhood includes spatially adjacent nodes, time-corresponding nodes obtained by inertial relative motion prior mapping, and cross-scale parent-child nodes. The propagation weights of candidate edges are determined based on the spatial relationship between nodes, consistency of motion direction, similarity of visual features, and inertial observation covariance. The candidate edges are then TopK filtered based on local observation scores and inertial observation covariance to obtain an effective propagation neighborhood that is updated as the observation quality changes.

[0033] Specifically, in this embodiment, such as Figure 3 As shown, visual features Divided into There are spatial nodes, and the set of nodes is denoted as . . No. The center position of each node is denoted as The corresponding visual node features are denoted as The corresponding inertial state characteristics are denoted as .

[0034] The current motion direction description is constructed based on the translation and rotation components pre-integrated by inertia. For each node... Establish candidate neighborhoods The candidate neighborhood includes: spatially adjacent nodes at the current scale; temporally corresponding nodes obtained by alignment with motion prior features; and parent-child nodes between adjacent scales.

[0035] For nodes and its candidate neighbors The edge weights of the dynamic graph are calculated based on spatial distance, consistency of motion direction, similarity of visual features, and the covariance of corresponding inertial observations of nodes. ; In the formula, , and These are the weight coefficients for the motion direction constraint, visual feature constraint, and inertial uncertainty constraint, respectively. d t This is the motion direction vector obtained from the inertial pre-integration result at the current moment; The square of the spatial distance scale parameter is used to adjust the influence of spatial distance on edge weights; For the first q Visual features of each node; For the first q The inertial observation covariance matrix corresponding to each node, Represents a node For nodes The propagation weights are influenced by the combined effects of the current motion state, visual content, and inertial uncertainty, resulting in a time-varying graph topology rather than a fixed, regular grid.

[0036] Perform visual state propagation and inertial state propagation separately on the dynamic graph. (The sentence is incomplete and requires further context.) Taking layer state propagation as an example, visual nodes and inertial nodes respectively aggregate the states of candidate neighbors and update the node states through input mapping, state mapping, and residual mapping obtained by selective state space parameterization. The selective parameters can be adjusted by the local observation quality of the node, so that state propagation changes with the node reliability.

[0037] ; ; ; In the formula, and Representing nodes respectively In the The propagation of visual state and inertial state received by the layer; Indicates visual state; Indicates an inertial state; , and It is obtained by selective state-space parameterization and adaptively adjusted by the local observation quality of the nodes.

[0038] Calculate the cross-modal innovation vector for the updated visual and inertial states. Cross-modal innovation characterizes the two modes at the node The prediction differences between the two modes are analyzed. Node-level cross-modal gating coefficients are generated based on visual state, inertial state, cross-modal innovation, and inertial observation covariance. Asymmetric fusion is then performed using these gating coefficients. When the inertial covariance increases or the prediction differences between the two modes increase, the direct impact of the inertial state on the fusion result is reduced. When visual texture is insufficient but the inertial state is stable, the fusion contribution of the inertial state is increased.

[0039] ; ; ; In the formula, where, and The projection matrix is ​​for visual and inertial features, used to map the two modalities to the same feature dimension; For nodes p The visual-inertial characteristics differ; This is the cross-modal gating vector, also known as the gating coefficient; and These are the learnable parameters of the gated network; For nodes p The inertial observation covariance.

[0040] To quantify the effective constraint of a node on the current pose estimation, a local observation score is calculated. Let... Let be the Jacobian matrix of the visual reprojection residuals with respect to node features. This is the Jacobian matrix of the inertial prediction residuals with respect to the node states. Given the joint covariance matrix consisting of the visual observation covariance and the inertial observation covariance, the local observational score... It can be represented as: ; In the formula, To prevent singular positive numbers in the matrix, It is an identity matrix. The normalized representation of the node The degree of effective constraint on the current pose increment. The observational score is not only used to output confidence, but also serves as a common control variable for subsequent visual transmission, dynamic graph construction, cross-modal gating, and pose covariance prediction.

[0041] Based on the local observational score of the current frame and the pose covariance output from the previous time step, adjust the visual feature transfer weights for the next time step: ; In the formula, , and These are the weighting coefficients for the photometric residual, gradient residual, and inertial observation covariance term, respectively. and They represent the first Frame and current frame in position The photometric residual and gradient residual at the location; Indicates the first The inertial observation covariance corresponding to the frame, Indicates the position of the historical frame Local observational scoring at the location; This represents the pose covariance output for the current frame. Therefore, the estimation uncertainty of the current frame is fed back into the visual feature aggregation process of the next frame, forming a time-series uncertainty loop.

[0042] Topological filtering of candidate neighborhoods is performed using local observational scoring. For nodes... The candidate nodes can be selected using the following formula: ; In the formula, For observational scoring gain coefficients, This is the inertial uncertainty suppression coefficient. Let be the inertial observation covariance corresponding to the candidate node. Through the TopK operation, effective propagation nodes are dynamically selected from three types of candidate nodes: spatial nearest neighbors, motion prior correspondences, and cross-scale nodes. This allows the graph topology to switch discretely with local observation quality and inertial confidence, rather than being continuously weighted and adjusted.

[0043] The same local observational score is also used to correct the cross-modal gating and pose covariance decoding scale. The node-level fusion gating formula and pose covariance decoding formula can be expressed as: ; ; In the formula, For learnable gating mapping parameters, For the Sigmoid function, and These represent the node states after visual state propagation and inertial state propagation, respectively, with the semicolon indicating feature concatenation. This represents the log-variance of the six-dimensional pose predicted by the network. Let be any constant greater than 0. Represents a node The local observational score is used for feature propagation, TopK graph neighborhood selection, cross-modal fusion gating, and pose uncertainty output, thereby forming a unified observation reliability control path among the modules.

[0044] All fusion nodes are restored to spatial feature maps, and fusion features are obtained through residual convolution. Its formula is ; in, ; In the formula, and These are the visual and inertial feature projection matrices, used to map the visual and inertial states to a unified feature space.

[0045] In terms of multimodal fusion, a motion-adaptive sparse graph is constructed. Node neighborhoods and propagation weights are determined based on motion direction, spatial location, visual content, and inertial uncertainty. Local observational scoring is used to control cross-modal fusion gating. This reduces the impact of unreliable modalities when visual observation quality deteriorates or inertial covariance increases. Simultaneously, information propagation based on local sparse neighborhoods reduces invalid feature interactions, thus improving fusion efficiency and real-time processing capabilities.

[0046] In some implementations, decoding the fused features includes: By performing global pooling on the fused features, and inputting the pooling result, the Lie algebraic motion quantity corresponding to the inertial pre-integration, and the inertial pre-integration covariance into the residual decoder, the six-dimensional Lie algebraic pose residual and the corresponding logarithmic variance are output. The six-dimensional Lie algebraic pose residual is applied to the inertial pre-integration pose through exponential mapping to obtain the SE(3) relative pose between adjacent frames, and the pose covariance is determined by the logarithmic variance. Based on visual reprojection residuals, inertial consistency residuals, and geometric consistency residuals of adjacent frames, a preset number of geometric corrections are performed on the six-dimensional Lie algebra pose residuals in the Lie algebra tangent space.

[0047] Specifically, in this embodiment, the reference relative pose of the current adjacent frame is constructed based on the aforementioned inertial pre-integration result. , for fusion features Perform global pooling and combine the pooling result with the inertial Lie algebra motion. The diagonal term of the inertial pre-integral covariance is input into the residual multilayer perceptron, which outputs a six-dimensional Lie algebraic pose residual. and the corresponding logarithmic variance : ; In the formula, Indicates global pooling. This represents a multilayer perceptron. This represents the diagonal element of the extracted inertial pre-integral covariance.

[0048] Applying the six-dimensional Lie algebraic pose residual to the inertial reference pose yields the SE(3) relative pose of the current frame with respect to the previous frame: , ; In the formula, , To estimate the translation vector, To estimate the rotated Lie algebra vector, Let SE(3) be the logarithmic mapping to its Lie algebra. Further, according to... Obtain the pose covariance , is used to represent the confidence level of the current pose output.

[0049] After the current frame pose decoding is completed, Together with the inertial pre-integral covariance of the next time interval, they form the feedback covariance. And used for weight update in step S1 of the next frame: ; in, Belongs to [0, 1]. When the uncertainty of pose estimation in the current frame increases, the dependence on long-range visual features is reduced in the next frame. When the observability of the current frame is high, the utilization of features is expanded in the next frame, thereby realizing the reverse constraint of pose decoding results on the feature extraction stage.

[0050] Repeat the above steps to recursively update the observational score, dynamic graph topology, pose covariance, and visual feature transfer weights for the next time step between consecutive frames.

[0051] ; in, For the number of corrections, Step size, Let be the Jacobian matrix of the residuals with respect to the Lie algebraic residuals. Determined by both visual and inertial covariance. Recalculated after correction. .

[0052] Compared to the method of directly regressing pose using LSTM hidden states, this step restricts pose estimation to... On the manifold, and using inertial pre-integration as an interpretable geometric reference, only the residual correction of the inertial reference by visual features is learned, thereby reducing the impact of rotation parameterization inconsistency and long sequence error accumulation.

[0053] In some implementations, during the training phase, translation and rotation errors are constructed based on the true relative pose, inertial consistency errors are constructed based on the inertial pre-integrated pose and corresponding covariance, closed-loop geometric consistency errors are constructed based on the relative pose at three consecutive time points, and uncertainty calibration errors are constructed based on the predicted log-variance of the pose. The translation, rotation, inertial consistency, closed-loop geometric consistency, and uncertainty calibration errors are weighted to optimize the network parameters for visual feature propagation, inertial state-space encoding, graph fusion, and residual decoding.

[0054] Specifically, in this embodiment, the training phase is based on the true relative pose. Construct pose loss. Let the translation and rotation of the true relative pose be respectively... and The prediction result is and Then the pose loss can be achieved by: ; In the formula, and Let represent the relative translation vector and relative rotation matrix predicted by the network, respectively; Log(·) represents the logarithmic mapping from the rotation group to the Lie algebra. This is the balance coefficient between translation error and rotation error.

[0055] Construct an inertial consistency loss and use the estimated covariance to apply a weighted constraint to the residuals between the network output and the inertial reference: ; In the formula, This corresponds to the covariance of the prior uncertainty of inertial motion.

[0056] Construct a closed-loop geometric consistency error for three consecutive time points to ensure that the composite result of two adjacent relative poses is consistent with the true relative pose across two frames: ; In the formula, From time At the time The true relative pose; Network predictions from arrive The relative pose; The uncertainty calibration error is constructed based on the log-variance predicted by the residual decoder: ; In the formula, Traverse the translation and rotation residual components, For the first The predicted log-variance of each component, This corresponds to the pose residual.

[0057] The overall training objective of the network can be expressed as: ; In the formula, , , , and These are the weighting coefficients for each loss term. The parameters to be trained on the network are defined by this joint objective. This objective enables visual feature propagation, inertial state encoding, motion adaptive graph fusion, and SE(3) residual decoding to share consistent pose geometry and uncertainty constraints.

[0058] In terms of pose decoding and error feedback, the inertial pre-integrated pose is used as the geometric reference. Only the Lie algebra pose residual is corrected, and the residual is applied to the SE(3) pose manifold through exponential mapping, thereby ensuring the geometric consistency of rotation and translation updates. At the same time, the pose covariance is output, and the uncertainty of the current frame is fed back to the subsequent visual feature transmission, graph structure construction and fusion weight update process, forming an uncertainty closed loop between observation credibility, feature fusion and pose decoding, improving the stability and generalization ability of the system under sensor noise changes and unseen motion modes.

[0059] In this embodiment, "continuous-time selective state-space unit" refers to a state-space network unit whose state update parameters or effective time step can change according to the current inertial input; "motion-adaptive sparse graph" refers to a graph structure in which the candidate neighborhood and effective propagation neighborhood are jointly determined by the current motion prior, visual content, and uncertainty and updated over time; "local observational scoring" is used to quantify the degree of effective constraint of local observation on pose increment; "SE(3) residual decoding" refers to updating the pose on a special Euclidean group SE(3) based on the inertial pre-integrated pose, through Lie algebra residuals and exponential mapping. The above terms are used to describe the computational relationships in this embodiment and do not limit the specific number of neural network layers or hardware structure.

[0060] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] Based on the same inventive concept, and corresponding to the methods of any of the above embodiments, the embodiments of this application also provide a visual inertial odometry system based on motion constraints and uncertainty constraints.

[0062] like Figure 4 As shown, the visual inertial odometry system based on motion constraints and uncertainty constraints includes: The inertial pre-integration module 11 is used to obtain the prior inertial relative motion and its covariance based on the data from the synchronous inertial measurement unit. The visual feature propagation module 12 is used to perform motion compensation on multiple frames of visual features based on inertial relative motion priors and aggregate them according to observation reliability. The inertial state coding module 13 is used to perform continuous-time selective state-space coding on the inertial measurement unit data and output inertial characteristics, observation confidence and covariance; Graph fusion module 14 is used to construct motion-adaptive sparse graphs, perform visual and inertial state propagation, and utilize local observational scoring for graph topology filtering and cross-modal fusion. The residual pose decoding module 15 is used to output the relative pose and pose covariance of SE(3) based on the inertial pre-integrated pose as the geometric reference. Feedback module 16 is used to feed back local observational scores and pose covariance to the visual feature propagation and graph fusion process of subsequent frames.

[0063] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0064] The system described in the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0065] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0066] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0067] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0068] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0069] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0070] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0071] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0072] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0073] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0074] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0075] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0076] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0077] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0078] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A visual inertial odometry method based on motion constraints and uncertainty constraints, characterized in that, include: Continuous image frames and synchronous inertial measurement unit (IMU) data are acquired. By pre-integrating the IMU data, the inertial relative motion prior and its covariance are obtained. Based on the inertial relative motion prior, the visual features of multiple frames within a local time window are projected and aligned. The aggregation weights are determined according to the visual residuals and inertial uncertainties to obtain the visual features. By performing continuous-time selective state-space encoding on the IMU data, the zero bias, observation confidence, and covariance are estimated to obtain the inertial features. Based on the visual features, inertial features, and inertial relative motion priors, a motion-adaptive sparse graph is constructed. The graph neighborhood and propagation weights are determined based on spatial relationships, motion directions, visual feature similarities, and inertial uncertainties. Visual state and inertial state propagation are performed on the motion-adaptive sparse graph. The fusion features are obtained based on the local observational scoring adjustment graph topology and cross-modal fusion gating determined by the Jacobian matrix and joint covariance of the visual reprojection residual and the inertial prediction residual. Using the inertial pre-integrated pose as the geometric reference, the fusion features are decoded to obtain the six-dimensional Lie algebraic pose residual and pose uncertainty. The pose residual is applied to the geometric reference through exponential mapping to obtain the SE(3) relative pose and pose covariance. The local observation score and pose covariance are then fed back to the visual feature aggregation, graph neighborhood selection and fusion gating update of subsequent frames.

2. The method according to claim 1, characterized in that, The obtained visual features include: A local time window is established with the current image frame as a reference, and the inertial measurement unit data corresponding to adjacent image frames within the window are pre-integrated; The historical frame features are projected onto the current frame coordinate system by the inverse depth of the current image frame and the inertial relative motion prior. The photometric residual, gradient residual and inertial uncertainty penalty term determined by the pre-integrated covariance are calculated after projection alignment. The position-related normalized aggregation weights are determined based on the photometric residual, gradient residual and inertial uncertainty penalty term. The aligned multi-frame features are weighted and aggregated according to the normalized aggregation weights, and then combined with the residual term of the current frame features to obtain the visual features.

3. The method according to claim 1, characterized in that, The continuous-time selective state-space coding includes: The inertial input is constructed based on the triaxial acceleration, triaxial angular velocity, corresponding first-order difference components, and the coupling amount between acceleration and angular velocity of adjacent inertial sampling points. Based on the inertial input, state transition parameters, input parameters, and time step are adaptively generated, and the inertial hidden state is recursively derived in a continuous time state update manner. The inertial hidden state is used to predict the accelerometer zero bias increment, gyroscope zero bias increment, inertial observation confidence, and inertial observation covariance. The inertial hidden state is then subjected to time interval pooling to obtain the inertial features corresponding to the visual feature grid.

4. The method according to claim 1, characterized in that: The motion-adaptive sparse graph uses the spatial location of visual features as nodes, and the candidate neighborhood includes spatially adjacent nodes, time-corresponding nodes obtained through inertial relative motion prior mapping, and cross-scale parent-child nodes. The propagation weights of candidate edges are determined based on the spatial relationship between nodes, consistency of motion direction, similarity of visual features, and inertial observation covariance. The candidate edges are then TopK filtered based on the local observation score and inertial observation covariance to obtain an effective propagation neighborhood that is updated as the observation quality changes.

5. The method according to claim 4, characterized in that: The local observational score is determined by the Jacobian matrix of the visual reprojection residual to the node features, the Jacobian matrix of the inertial prediction residual to the node state, and the joint covariance matrix composed of the visual observation covariance and the inertial observation covariance, and is used to characterize the degree of effective constraint of the corresponding node on the current pose increment. The same local observational score is used to adjust the decoding scale of visual feature transmission weights, effective propagation neighborhood, cross-modal fusion gating, and pose covariance for subsequent frames.

6. The method according to claim 1, characterized in that, The decoding of the fused features includes: By performing global pooling on the fused features, and inputting the pooling result, the Lie algebraic motion quantity corresponding to the inertial pre-integration, and the inertial pre-integration covariance into the residual decoder, the six-dimensional Lie algebraic pose residual and the corresponding logarithmic variance are output. The six-dimensional Lie algebraic pose residual is applied to the inertial pre-integration pose through exponential mapping to obtain the SE(3) relative pose between adjacent frames, and the pose covariance is determined by the logarithmic variance. Based on visual reprojection residuals, inertial consistency residuals, and adjacent frame geometric consistency residuals, the six-dimensional Lie algebra pose residuals are geometrically corrected a preset number of times in the Lie algebra tangent space.

7. The method according to claim 1, characterized in that: During the training phase, translation and rotation errors are constructed based on the true relative pose, inertial consistency error is constructed based on the inertial pre-integrated pose and its corresponding covariance, closed-loop geometric consistency error is constructed based on the relative pose at three consecutive time points, and uncertainty calibration error is constructed based on the predicted logarithmic variance of the pose. The translation, rotation, inertial consistency, closed-loop geometric consistency, and uncertainty calibration errors are weighted to optimize the network parameters for visual feature propagation, inertial state-space encoding, graph fusion, and residual decoding.

8. A visual inertial odometry system based on motion constraints and uncertainty constraints, characterized in that, include: The inertial pre-integration module is used to obtain the prior inertial relative motion and its covariance based on the data from the synchronous inertial measurement unit. The visual feature propagation module is used to perform motion compensation on multiple frames of visual features based on the inertial relative motion prior and aggregate them according to observation reliability. The inertial state coding module is used to perform continuous-time selective state-space coding on inertial measurement unit data and output inertial characteristics, observation confidence, and covariance. The graph fusion module is used to construct motion-adaptive sparse graphs, perform visual and inertial state propagation, and utilize local observational scoring for graph topology filtering and cross-modal fusion. The residual pose decoding module is used to output the relative pose and pose covariance of SE(3) based on the inertial pre-integrated pose as the geometric reference. The feedback module is used to feed back the local observational scores and pose covariance to the visual feature propagation and graph fusion process of subsequent frames.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, in, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1-7.

Images