Visual inertial odometer method and system based on adaptive decision

Abstract

The invention discloses a visual inertial odometer method and system based on adaptive decision, and relates to the technical field of visual inertial odometers, the specific steps are as follows: obtaining original image data and original IMU data, and constructing a data set containing synchronous image data and IMU data, the original image data being collected based on a visual sensor, and the original IMU data being collected based on a visual sensor; the original IMU data is collected based on an IMU sensor; inputting the image data and the IMU data into a trained visual odometer network model to respectively extract optical flow features and inertial features, performing modal selection by using a dynamic modal selection mechanism, and performing selection or fusion processing on the optical flow features and the inertial features according to the selected modal to generate output features; and decoding the output feature to obtain a relative pose. According to the method, the robustness of the visual inertial odometer system in a complex environment is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual inertial odometry, and more particularly to a visual inertial odometry method and system based on adaptive decision-making. Background Art

[0002] Visual-inertial odometry plays a vital role in autonomous driving and robotic positioning and navigation by combining visual information with data from an inertial measurement unit (IMU). Existing visual-inertial odometry methods typically fuse camera images and IMU data in a tightly coupled manner, maintaining good positioning accuracy across multiple sensor modes. However, sensor noise is a significant factor affecting the accuracy of visual-inertial odometry systems, especially in real-world environments where IMU and camera data are often subject to noise and interference, leading to error accumulation and positioning drift.

[0003] Traditional visual-inertial odometry methods typically assume accurate or low-noise sensor data. However, in real-world applications, sensor noise and data loss are unavoidable. Existing visual-inertial odometry methods often fail to effectively address severe sensor noise or failures, resulting in large positioning errors. To address this issue, several deep learning-based visual-inertial odometry methods have been proposed in recent years. These methods utilize deep neural network architectures such as convolutional neural networks and long short-term memory networks for feature extraction and data fusion. However, their robustness and adaptability in noisy environments still require significant improvement.

[0004] Therefore, how to solve the vulnerability of traditional deep learning models to unknown noise patterns is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a visual inertial odometry method and system based on adaptive decision-making, which overcomes the above technical defects.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A visual inertial odometry method based on adaptive decision making, the specific steps are as follows:

[0008] Acquire raw image data and raw IMU data, and construct a data set containing synchronized image data and IMU data, wherein the raw image data is acquired based on a visual sensor, and the raw IMU data is acquired based on an IMU sensor;

[0009] Inputting the image data and the IMU data into a trained visual odometry network model to extract optical flow features and inertial features respectively, and performing modality selection using a dynamic modality selection mechanism, selecting or fusing the optical flow features and the inertial features according to the selected modality to generate output features;

[0010] The output features are decoded to obtain relative pose.

[0011] Furthermore, the steps of obtaining the data set are:

[0012] Establishing a sliding time window covering a preset time threshold using the image frame timestamp of the original image data as a reference anchor point;

[0013] Extracting continuous data blocks corresponding to each time window from the raw IMU data;

[0014] Performing time domain alignment processing on the continuous data blocks, using an interpolation algorithm to compensate for the time deviation between the image frame timestamp and the IMU sampling point, and generating an inertial measurement sequence that is consistent with the time of the original image data;

[0015] The frame overlap ratio between adjacent data blocks is controlled by a sliding window strategy to generate a dataset containing synchronized image-inertial data pairs.

[0016] Furthermore, the visual odometry network model includes a feature extraction module, an intuitive bias selection module and a pose decoding module;

[0017] The feature extraction module is configured to extract the optical flow features of the image data using a FlowNet-S architecture and to extract the inertial features of the IMU data using a Conv1D architecture;

[0018] The intuitive bias selection module includes two sub-networks and a feature output module. The first sub-network is a decision network based on a closed continuous-time neural network CfC, which inputs the optical flow features and the inertial features and outputs a fusion intention weight vector; the second sub-network is a trust network composed of multiple linear layers, which inputs the optical flow features and the inertial features and outputs a modality preference probability; the feature output module generates a decision mask and a trust mask based on the Gumbel-Softmax function for the fusion intention weight vector and the modality preference probability, and performs modality selection or feature fusion based on the decision mask and the trust mask, and outputs the output feature;

[0019] The posture decoding module is used to decode the output features to obtain relative posture.

[0020] Furthermore, the FlowNet-S architecture includes a dual-branch convolutional encoder, a downsampling convolutional layer and an upsampling decoder, and the spatial information is maintained between the downsampling convolutional layer and the upsampling decoder through a jump connection.

[0021] Furthermore, the expression of the inertial features of the IMU data extracted using the Conv1D architecture is:

[0022]

[0023] Where [A,Ω] t→t+1 The linear acceleration and rotation rate during the current time period.

[0024] Furthermore, when training the visual odometry network model, it is necessary to preprocess the constructed training data set, simulate the degradation scenarios of the visual sensor and the IMU sensor, and set the probability of degradation occurrence.

[0025] Furthermore, the preprocessing is noise preprocessing, which processes the image data by adding Gaussian noise and zero bias compensation, random occlusion, Gaussian blur, and salt and pepper noise, and applies zero-value filling, rotation matrix perturbation, and random zero-space noise to the IMU data to simulate the degradation of the IMU sensor.

[0026] Furthermore, when training the visual odometry network model, a composite loss function including motion loss and regularization term is used to optimize the model, and its expression is:

[0027]

[0028] Where, is the motion loss function; and Both are regularized loss functions.

[0029] Furthermore, the expression of motion loss is:

[0030]

[0031] Where N is the time step of the dataset; V t is the predicted translation vector at time t; v t is the translation vector at time t; α is a hyperparameter that balances the rotation and displacement losses; Φ t is the predicted rotation vector at time t; φ t is the rotation vector at time t.

[0032] A visual inertial odometry system based on adaptive decision making, comprising:

[0033] Data is collected in a preprocessing module, which is used to obtain raw image data and raw IMU data, and to construct a data set containing synchronized image data and IMU data. The raw image data is collected based on the visual sensor, and the raw IMU data is collected based on the IMU sensor.

[0034] a feature extraction and processing module, configured to input the image data and the IMU data into a trained visual odometry network model to extract optical flow features and inertial features respectively, and to select a modality using a dynamic modality selection mechanism, and to select or fuse the optical flow features and the inertial features according to the selected modality to generate output features;

[0035] The feature decoding module is used to decode the output features to obtain relative pose.

[0036] It can be seen from the above technical solution that compared with the existing technology, the present invention provides a visual inertial odometry method and system based on adaptive decision-making. The intuitive selection module simulates the light tendency and obstacle avoidance behavior of insects to construct a modal decision-making mechanism, and adopts a dynamic modal selection mechanism to realize the dynamic fusion of visual and inertial measurement units. When the sensor degrades, it can autonomously switch to the optimal mode, effectively overcoming the problems of noise interference and data loss; and the biological-like decision-making process of the neural network gives the algorithm a transparent feature, making the decision logic explainable, which not only ensures the optimal fusion of multi-source information in complex environments, but also significantly improves the robustness of the visual inertial odometry system in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A schematic flow chart of the method provided by the present invention;

[0039] Figure 2 A schematic diagram of the working principle of the intuitive selection module provided by the present invention;

[0040] Figure 3 Schematic diagram of different sensor degradation conditions provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the visual inertial odometry system architecture provided by the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] On the one hand, the embodiment of the present invention discloses a visual inertial odometry method based on adaptive decision making, such as Figure 1 As shown, the specific steps are:

[0044] Step 1: Obtain raw image data and raw IMU data, and construct a dataset containing synchronized image data and IMU data. The raw image data is collected based on the visual sensor, and the raw IMU data is collected based on the IMU sensor.

[0045] Step 2: Input the image data and IMU data into the trained visual odometry network model to extract optical flow features and inertial features respectively, and use the dynamic mode selection mechanism to select the mode. According to the selected mode, the optical flow features and inertial features are selected or fused to generate output features;

[0046] Step 3: Decode the output features to obtain the relative pose.

[0047] In one embodiment, the steps of obtaining the data set are:

[0048] Using the image frame timestamp of the original image data as a reference anchor point, a sliding time window covering a preset time threshold is established;

[0049] Extract continuous data blocks corresponding to each time window from the original IMU data;

[0050] Perform time domain alignment on continuous data blocks, use interpolation algorithms to compensate for the time deviation between the image frame timestamp and the IMU sampling point, and generate an inertial measurement sequence that is consistent with the time of the original image data;

[0051] The frame overlap ratio between adjacent data blocks is controlled by a sliding window strategy to generate a dataset containing synchronized image-inertial data pairs.

[0052] Furthermore, when constructing the dataset, we used image frames as anchor points based on timestamp information, extracting continuous data blocks within the corresponding time range from the high-frequency IMU sequence. We used linear or spline interpolation to compensate for time differences between sampling points, obtaining inertial measurements that were consistent with the image timing. Furthermore, a sliding window strategy was used to control the overlap between image frames, thereby constructing a training dataset containing synchronized image and inertial information.

[0053] In one embodiment, the visual odometry network model includes a feature extraction module, an intuitive bias selection module, and a pose decoding module;

[0054] Feature extraction module, which uses the FlowNet-S architecture to extract optical flow features from image data and the Conv1D architecture to extract inertial features from IMU data;

[0055] The intuitive bias selection module includes two sub-networks and a feature output module. The first sub-network is a decision network based on a closed continuous-time neural network (CfC), which inputs optical flow features and inertial features and outputs a fusion intention weight vector. The second sub-network is a trust network composed of multiple linear layers, which inputs optical flow features and inertial features and outputs modality preference probabilities. The feature output module uses the Gumbel-Softmax function to generate a decision mask and a trust mask from the fusion intention weight vector and modality preference probability, respectively. It then performs modality selection or feature fusion based on the decision mask and trust mask, and outputs the output features.

[0056] The pose decoding module is used to decode the output features to obtain the relative pose.

[0057] Furthermore, this embodiment designs a visual inertial odometry framework, such as Figure 2 As shown in the figure, a convolutional neural network is used to extract high-dimensional features of image data and IMU data, and a recurrent neural network is used to decode pose information; the Gumbel-Softmax trick and neural circuit strategy NCPs are introduced into the intuitive bias selection module to construct an interpretable modality selection mechanism; the core of this module consists of two sub-networks: one is a decision network based on a closed continuous-time neural network CfC, whose structure consists of 32 sparsely connected neural units and has memory dynamics controlled by a time constant τ, which is used to model long-term dependencies and determine whether to perform modal fusion; the other is a Confidence Network composed of multiple linear layers, which is used to evaluate the quality of visual and inertial modalities and assign modal weights. These two networks output decision weights and modal trust weights respectively, and after Gumbel-Softmax activation, a binary mask is obtained to perform differentiable modal selection and fusion control. This structure realizes the "fusion-selection" two-level modal adaptability through neural dynamics and mask mechanisms, enhancing the system's robustness to multi-sensor degradation.

[0058] Furthermore, the Gumbel-Softmax trick and NCPs introduce an intuitive bias selection module to construct an interpretable and learnable two-stage dynamic modality selection mechanism. Specifically, the intuitive bias selection module first receives the optical flow features from the visual modality. and the acceleration and angular velocity characteristics from the inertial mode The first is a decision network based on the CfC structure, which has 32 sparsely connected neurons, simulating the neural topology of Caenorhabditis elegans, and performs state updates in the continuous time domain. Its dynamic behavior is controlled by the time constant τ, thereby estimating whether modal fusion should be performed at the current moment and outputting the fusion intention weight vector. The second path is a confidence network composed of linear transformation layers, which is used to evaluate the confidence of the two modalities and generate the modality preference probability. Afterwards, respectively and Apply the Gumbel-Softmax trick to obtain two sets of binary masks by continuously relaxing the approximate discrete sampling method. and The mask discreteness is controlled by the temperature parameter τ and annealed down during training, encouraging exploration in the early stages of training and stabilizing decisions in the later stages. The decision mask in the first stage controls whether to perform modal fusion, while the selection mask in the second stage selects the best modality to retain. Finally, based on these two levels of masks, the following operations are performed:

[0059] If you choose to merge, the output If single mode is selected, then according to The activation state of or One of them is used for subsequent LSTM pose regression. In this way, the module not only achieves dynamic switching between fusion and selection, but also maintains the temporal consistency and interpretability of the modal switching trajectory through the pulse coding dynamics of NCPs, making the model more adaptable and decision-making robust in the face of multimodal degradation.

[0060] In one embodiment, the FlowNet-S architecture includes a dual-branch convolutional encoder, a downsampling convolutional layer, and an upsampling decoder, where the spatial information is maintained between the downsampling convolutional layer and the upsampling decoder via a skip connection.

[0061] In one embodiment, a visual inertial odometry framework is constructed. For visual input (i.e., image data), a 2D convolutional neural network is used to extract the optical flow features of the image. For inertial input such as acceleration (i.e., IMU data), a 1D convolutional neural network is sampled to extract high-dimensional features, and an LSTM long short-term memory network is used as a pose decoder.

[0062] In one embodiment, when training the visual odometry network model, it is necessary to preprocess the constructed training data set, simulate the degradation scenarios of the visual sensor and the IMU sensor, and set the degradation probability.

[0063] In one embodiment, the preprocessing is noise preprocessing, which processes the image data by adding Gaussian noise and zero bias compensation, random occlusion, Gaussian blur, and salt and pepper noise, and applies zero-value filling, rotation matrix perturbation, and random zero-setting spatiotemporal noise to the IMU data to simulate the degradation of the IMU sensor.

[0064] Furthermore, a visual inertial odometry dataset is constructed. Since the sampling frequencies of camera sensors and inertial measurement units are different (the IMU sampling frequency is often 10 times that of the camera), it is necessary to first align the image data and the IMU data. Continuous data blocks are intercepted according to IMU_FREQ and bound into synchronous units after compensating for time deviations through interpolation. At the same time, a sliding window is used to control the data overlap rate to generate training samples, and then the training dataset is constructed.

[0065] Furthermore, noise preprocessing is specifically performed by adding Gaussian noise and zero-bias compensation, random occlusion (covering a 128x128 area), Gaussian blur (15x15 convolution kernel), and salt and pepper noise (4% density) to process the visual data. At the same time, zero-value padding, rotation matrix perturbation (random angle rotation of 5°-10°), and random zeroing (probability 50%) are applied to the inertial data to simulate sensor degradation and enhance the robustness of the model in complex scenarios such as occlusion, motion blur, and signal loss.

[0066] In one embodiment, the training steps of the visual odometry network model are:

[0067] The KITTI dataset is obtained. The dataset provides synchronized RGB images and IMU data for training and testing. This example selects sequences 00, 01, 02, 04, 06, 08, and 09 from the KITTI dataset for training and evaluates sequences 05, 07, and 10. (Sequence 03 is not suitable for this task due to the lack of IMU data.) To account for the time difference between the camera and IMU sampling frequencies, linear interpolation is performed on the raw IMU data to ensure alignment of the IMU data and image frame timestamps.

[0068] Then, we simulated various sensor degradation scenarios and conducted experiments on visual degradation (including occlusion, blur and loss) and IMU degradation (including noise, loss, spatial and temporal alignment errors). Figure 3 Specifically, the probability of the visual degradation type is set to 10%, and the probability of the IMU degradation type is set to 7.5%. Under all degradation scenarios, the probability of sensor degradation occurring at each time step is 25-35%, while in the remaining 65-75% of time steps, the data remains normal.

[0069] This embodiment designs a visual odometry network model for training and testing, such as Figure 4The model input is image data (monocular image sequence) and raw IMU data. The image and IMU data features are extracted by using the visual feature encoder and inertial feature encoder respectively. For the visual modality, the FlowNet-S architecture is used to extract optical flow features. The architecture consists of a dual-branch convolutional encoder that extracts features from consecutive image frames. The extracted feature maps are concatenated in the channel dimension and input into multiple downsampling convolutional layers. The spatial resolution is then restored layer by layer through a symmetrical upsampling decoder, supplemented by skip connections to enhance the preservation of low-level information. The final output is a two-dimensional optical flow estimate for each pixel, thereby obtaining inter-frame motion information. The processing formula is:

[0070]

[0071] For IMU modality, Conv1D architecture is used to extract inertial features The formula is:

[0072]

[0073] Where, I t→t+1 is the image frame of consecutive time steps, [A,Ω] t→t+1 By the acceleration vector and the angular velocity vector It is composed of a three-axis accelerometer and a three-axis gyroscope, respectively, which represent the linear acceleration and rotation rate in the current time period.

[0074] The core of the algorithm in this embodiment is the intuitive bias selection module, which uses the Gumbel-Softmax sampling mechanism to achieve adaptive selection of visual and inertial sensor data. When the data is normal, the algorithm will fuse the features of the two modalities; when a modality is degraded, the algorithm will select the most reliable modality based on the degradation type. In the intuitive bias selection module, the features of the two modalities are used to select the most reliable modality. and A hard fusion strategy based on Gumbel-Softmax is used to make modality selection. Specifically, the decision weight is first calculated and Represent the confidence of modal fusion and selection respectively:

[0075]

[0076] Where, F CfC and F conf For the decision network, by learning the features of different modalities, the corresponding selection weights are obtained. Then, the Gumbel-Softmax function is used to generate a binary mask and

[0077]

[0078] Through these masks, it is decided whether to perform modality fusion or select a single modality:

[0079]

[0080] Where ⊙ represents the element-by-element multiplication operation, Act on each modal feature to control its selectivity. If you choose to fuse, combine the two modal features to get the fused feature representation and

[0081]

[0082] Where, Represents the concatenation operation of features. The final output feature Y t The decision and feature fusion results of the above two stages are given as:

[0083]

[0084] After the above process, the final features are decoded by the LSTM network and the relative pose is output:

[0085] φ t ,v t =F LSTM (Y t ) (8);

[0086] Among them, φ t Represents the rotation vector, v t Represents the translation vector.

[0087] In this embodiment, the mean square error (MSE) loss is used to optimize the motion error of the model. The formula is:

[0088]

[0089] Where N represents the time step of the dataset, and α is a hyperparameter that balances the rotation and translation losses. In addition, to prevent the model from over-relying on fusion features or a single modality, two penalty factors λ and γ are added during training to regularize the decision process:

[0090]

[0091] The final total loss function is:

[0092]

[0093] The insect-inspired visual-inertial odometry method provides strong robustness while maintaining high accuracy, ensuring continuous and stable operation in various sensor degradation environments. It is of certain significance to promote the application of artificial intelligence models in real environments in the field of autonomous driving.

[0094] An embodiment of the present invention discloses, on one hand, a visual inertial odometry system based on adaptive decision-making, comprising:

[0095] Data is collected in the preprocessing module, which is used to obtain raw image data and raw IMU data, and construct a data set containing synchronized image data and IMU data. The raw image data is collected based on the visual sensor, and the raw IMU data is collected based on the IMU sensor.

[0096] The feature extraction and processing module is used to input image data and IMU data into the trained visual odometry network model to extract optical flow features and inertial features respectively, and use the dynamic mode selection mechanism to select the mode. According to the selected mode, the optical flow features and inertial features are selected or fused to generate output features;

[0097] The feature decoding module is used to decode the output features and obtain the relative pose.

[0098] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0099] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual inertial odometry method based on adaptive decision making, characterized in that: The specific steps are: Acquire raw image data and raw IMU data, and construct a data set containing synchronized image data and IMU data, wherein the raw image data is acquired based on a visual sensor, and the raw IMU data is acquired based on an IMU sensor; Inputting the image data and the IMU data into a trained visual odometry network model to extract optical flow features and inertial features respectively, and performing modality selection using a dynamic modality selection mechanism, selecting or fusing the optical flow features and the inertial features according to the selected modality to generate output features; The output features are decoded to obtain relative pose.

2. The method of visual inertial odometry based on adaptive decision making according to claim 1, characterized in that: The steps for obtaining the dataset are: Establishing a sliding time window covering a preset time threshold using the image frame timestamp of the original image data as a reference anchor point; Extracting continuous data blocks corresponding to each time window from the raw IMU data; Performing time domain alignment processing on the continuous data blocks, using an interpolation algorithm to compensate for the time deviation between the image frame timestamp and the IMU sampling point, and generating an inertial measurement sequence that is consistent with the time of the original image data; The frame overlap ratio between adjacent data blocks is controlled by a sliding window strategy to generate a dataset containing synchronized image-inertial data pairs.

3. The method of visual inertial odometry based on adaptive decision making according to claim 1, characterized in that: The visual odometry network model includes a feature extraction module, an intuitive bias selection module and a pose decoding module; The feature extraction module is configured to extract the optical flow features of the image data using a FlowNet-S architecture and to extract the inertial features of the IMU data using a Conv1D architecture; The intuitive bias selection module includes two sub-networks and a feature output module. The first sub-network is a decision network based on a closed continuous-time neural network CfC, which inputs the optical flow features and the inertial features and outputs a fusion intention weight vector; the second sub-network is a trust network composed of multiple linear layers, which inputs the optical flow features and the inertial features and outputs the modality preference probability; the feature The output module generates a decision mask and a trust mask based on the Gumbel-Softmax function for the fusion intention weight vector and the modality preference probability, performs modality selection or feature fusion based on the decision mask and the trust mask, and outputs output features; The posture decoding module is used to decode the output features to obtain relative posture.

4. The method of visual inertial odometry based on adaptive decision making according to claim 3, characterized in that: The FlowNet-S architecture includes a dual-branch convolutional encoder, a downsampling convolutional layer, and an upsampling decoder, wherein the downsampling convolutional layer and the upsampling decoder maintain spatial information through a skip connection.

5. The method of visual inertial odometry based on adaptive decision making according to claim 3, characterized in that: The expression of the inertial features of the IMU data extracted using the Conv1D architecture is: Where [A,Ω] t→t+1 The linear acceleration and rotation rate during the current time period.

6. The method of visual inertial odometry based on adaptive decision making according to claim 1, characterized in that: When training the visual odometry network model, it is necessary to preprocess the constructed training data set, simulate the degradation scenarios of the visual sensor and the IMU sensor, and set the probability of degradation occurrence.

7. The method of visual inertial odometry based on adaptive decision making according to claim 6, characterized in that: The preprocessing is noise preprocessing, which processes the image data by adding Gaussian noise and zero bias compensation, random occlusion, Gaussian blur, and salt and pepper noise, and applies zero-value filling, rotation matrix perturbation, and random zero-space noise to the IMU data to simulate the degradation of the IMU sensor.

8. The method of visual inertial odometry based on adaptive decision making according to claim 1, characterized in that: When training the visual odometry network model, a composite loss function including motion loss and regularization term is used for model optimization, and its expression is: Where, is the motion loss function; and Both are regularized loss functions.

9. The method of visual inertial odometry based on adaptive decision making according to claim 8, characterized in that: The expression of motion loss is: Where N is the time step of the dataset; V t is the predicted translation vector at time t; v t is the translation vector at time t; α is a hyperparameter that balances the rotation and displacement losses; Φ t is the predicted rotation vector at time t; φ t is the rotation vector at time t.

10. A visual inertial odometry system based on adaptive decision making, characterized in that: include: Data is collected in a preprocessing module, which is used to obtain raw image data and raw IMU data, and to construct a data set containing synchronized image data and IMU data. The raw image data is collected based on the visual sensor, and the raw IMU data is collected based on the IMU sensor. a feature extraction and processing module, configured to input the image data and the IMU data into a trained visual odometry network model to extract optical flow features and inertial features respectively, and to select a modality using a dynamic modality selection mechanism, and to select or fuse the optical flow features and the inertial features according to the selected modality to generate output features; The feature decoding module is used to decode the output features to obtain relative pose.

Images