A visual inertial odometry based on deep learning and its use method

Abstract

The present application belongs to the field of UAV visual positioning technology. The present application provides a visual inertial odometry based on deep learning and a method for using the same. The disclosed embodiments can. An efficient dynamic perception network is introduced to replace ordinary convolution, and a feature fusion module with enhanced cross-modal consistency is constructed, which can capture long-distance spatial information, while eliminating the difference between the two modalities and enhancing the consistency between the two. At the same time, a high real-time processing speed can be achieved under the premise of low memory usage, making it very suitable for deployment on resource-constrained embedded platforms. The multimodal information fusion strategy efficiently integrates visual information with inertial data to ensure that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as cities, rural areas, and high-speed driving, showing excellent robustness and wide applicability.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of drone visual positioning technology, and more particularly to a deep learning-based visual inertial odometry and a method for using the same. Background Art

[0002] Visual-Inertial Odometry (VIO) is a technology that estimates the trajectory and pose of a vehicle in three-dimensional space by fusing data from visual sensors and inertial measurement units (IMUs). By integrating complementary information from visual sensors (such as cameras) and IMUs, VIO overcomes the limitations of single sensors: visual data provides rich environmental features but is susceptible to interference from lighting changes and rapid motion, while IMU data provides high-frequency motion information but suffers from cumulative drift. Existing VIO technologies fall into two main categories: traditional geometry-based VIO methods and deep learning-based VIO methods.

[0003] Traditional VIO methods calculate the pose transformation by calculating feature points and matching relationships between image frames. They also pre-integrate the IMU to obtain additional information to address issues such as scale ambiguity and improve overall system accuracy. Based on the different sensor fusion methods, traditional VIO methods can be categorized as loosely coupled and tightly coupled. Research has shown that tightly coupled methods, by uniformly modeling the correlation between visual and inertial data, can more fully leverage the complementary properties of multimodal sensors, significantly improving pose estimation accuracy and becoming a research hotspot in the VIO field. From an implementation perspective, tightly coupled VIO algorithms are primarily based on two frameworks: filtering-based methods (such as multi-state constrained Kalman filtering) estimate the system state recursively. Classic filtering methods include MSCKF and ROVIO. Nonlinear optimization-based methods (such as sliding window graph optimization) achieve a global optimal solution by minimizing a joint objective function of reprojection error and inertial measurement error. Classic optimization methods include VINS-Mono and ORB-SLAM3. However, all traditional VIO methods have the following problems: they rely on manual initialization parameters (such as the number of feature points, keyframe selection threshold, and IMU deviation calibration), resulting in complex deployment and poor real-time performance; feature matching and nonlinear optimization require large computational loads, making it difficult to meet the real-time requirements of embedded devices; and they are prone to scale blur and trajectory drift in scenarios with rapid motion or texture loss.

[0004] With the rapid development of artificial intelligence in the field of deep learning, deep learning-based VIO methods have begun to attract the attention of researchers. This end-to-end VIO method demonstrates the potential of integrating sensor information through direct learning. It is also much faster than VIO based on geometric methods in actual deployment and inference, to some extent alleviating the difficulties and pain points of traditional VIO methods. End-to-end VIO methods utilize deep neural networks (DNNs) for training on large datasets and use feature encoders to encode image data and inertial data. Compared to geometric methods, this data-driven approach goes beyond simply modeling 3D / 2D correspondences between image frames and performing simple preprocessing on the IMU. Instead, it learns the underlying connections between image frames and the long-term dependencies of the IMU to address accumulated drift. However, some existing deep learning-based VIO methods, such as VINet and ATVIO, also have some problems: visual feature extraction relies on traditional modules such as FlowNet, only focusing on local optical flow information and ignoring global spatial correlation; the visual and inertial modal features differ significantly, and simple splicing or weighted fusion leads to feature distribution mismatch and low information utilization; the existing attention mechanism (such as EMA-VIO) does not solve the problem of cross-modal temporal alignment, which limits the fusion effect.

[0005] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0007] The purpose of the embodiments of the present disclosure is to provide a deep learning-based visual inertial odometry and a method for using the same, thereby overcoming, at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0008] According to a first aspect of an embodiment of the present disclosure, a deep learning-based visual inertial odometry is provided, comprising:

[0009] Visual encoder, inertial encoder, feature fusion module and multiple regression module. The outputs of the visual encoder and inertial encoder are input to the feature fusion module, and the output of the feature fusion module is input to the multiple regression module.

[0010] The visual encoder includes a first dynamic perception network, a second dynamic perception network, a third dynamic perception network, a fourth dynamic perception network, a fifth dynamic perception network, a sixth dynamic perception network, a seventh dynamic perception network, an eighth dynamic perception network, a ninth dynamic perception network and a first fully connected layer in series;

[0011] The inertial encoder consists of three one-dimensional convolutional neural networks and a second fully connected layer in series;

[0012] The feature fusion module includes a shared feature extractor and a multi-layer perceptron;

[0013] The multivariate regression module includes a long short-term memory network and a third fully connected layer.

[0014] Furthermore, each dynamic perception network includes a dual-stream feature generation unit and a space-channel collaborative gating mechanism unit.

[0015] According to a second aspect of an embodiment of the present disclosure, a method for using a deep learning-based visual inertial odometry is provided, the method comprising:

[0016] Acquire an image pair and its corresponding IMU data; wherein the image pair is two consecutive image frames, and the IMU data is the IMU measurement data between the two consecutive image frames, and the IMU data includes accelerometer data and gyroscope data;

[0017] Utilize visual encoders to perform cheap convolution-based operations and cross-channel attention weighting on image pairs to generate visual features;

[0018] Use the inertial encoder to extract features from IMU data to obtain inertial features;

[0019] The feature fusion module is used to perform cross-modal consistency fusion of visual features and inertial features to obtain fused features;

[0020] The multivariate regression module is used to construct the temporal dependency relationship across time steps based on the fused features to obtain the relative pose between two consecutive image frames.

[0021] Furthermore, the step of using the visual encoder to perform cheap convolution-based operations and cross-channel attention weighting on the image pairs to generate visual features includes:

[0022] The two-stream feature generation unit extracts the basic features of the image pair through the main feature extractor and combines it with deep convolution to generate cheap auxiliary features;

[0023] The spatial-channel collaborative gating mechanism unit performs decomposition and full-connection operations on the image pair in the horizontal and vertical directions respectively to calculate the attention score of the image pair in the horizontal direction and the vertical direction;

[0024] Based on the horizontal attention score and the vertical attention score, the attention weight is normalized using the Sigmoid function;

[0025] Visual features are generated by concatenating basic features and cheap auxiliary features and multiplying them with the normalized attention weights.

[0026] Furthermore, the step of extracting features from the IMU data using the inertial encoder to obtain inertial features includes:

[0027] The inertial encoder uses a one-dimensional convolutional neural network to extract high-frequency dynamic motion features from multiple sets of IMU data between two frames of images to generate inertial features.

[0028] Furthermore, the step of using the feature fusion module to perform cross-modal consistency fusion on the visual features and the inertial features to obtain fused features includes:

[0029] The shared feature extractor is used to concatenate the visual features and the inertial features and then input them into the collaborative attention mechanism to generate the first shared feature;

[0030] Dynamically weighting and adjusting the first shared feature and the inertia feature to obtain a second shared feature;

[0031] The second shared feature is fused with the visual feature, and a fused feature is generated through a multi-layer perceptron.

[0032] Furthermore, the step of using a multivariate regression module to construct a temporal dependency relationship across time steps based on the fusion features to obtain the relative pose between two consecutive image frames includes:

[0033] The long short-term memory network of the multivariate regression module constructs temporal dependencies across time steps based on the fused features;

[0034] The third fully connected layer of the multivariate regression module projects the temporal dependencies across time steps into a 6-DOF pose to obtain the relative pose between two consecutive image frames.

[0035] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0036] In the embodiments of the present disclosure, through the above-mentioned deep learning-based visual inertial odometry and its use method, on the one hand, an efficient dynamic perception network is introduced to replace ordinary convolution, and a feature fusion module with enhanced cross-modal consistency is constructed, which can capture long-distance spatial information, and at the same time eliminate the differences between the two modalities and enhance the consistency between the two. At the same time, a high real-time processing speed can be achieved under the premise of low memory usage, making it very suitable for deployment on resource-constrained embedded platforms. On the other hand, the multimodal information fusion strategy efficiently integrates visual information with inertial data to ensure that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as urban, rural and high-speed driving, showing excellent robustness and wide applicability. In addition, the end-to-end network architecture effectively simplifies the system design, avoids heavy initialization and optimization steps, thereby reducing the computational complexity and implementation difficulty, and facilitating promotion and deployment in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0038] Figure 1 A framework diagram of a deep learning-based visual inertial odometry system in an exemplary embodiment of the present disclosure is shown;

[0039] Figure 2 A schematic diagram showing the structure of a visual encoder in an exemplary embodiment of the present disclosure is shown;

[0040] Figure 3 A diagram showing a method for using a deep learning-based visual inertial odometry in an exemplary embodiment of the present disclosure;

[0041] Figure 4 A specific flow chart showing a method for using a deep learning-based visual inertial odometry in an exemplary embodiment of the present disclosure;

[0042] Figure 5 Showing the comparison visualization results between the method of the present application and Ground Truth in the exemplary embodiment of the present disclosure;

[0043] Figure 6 The GPU memory usage result of the method of the present application in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0045] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0046] This example embodiment first provides a deep learning-based visual inertial odometry. Figure 1 As shown in , the deep learning-based visual inertial odometry may include:

[0047] Visual encoder, inertial encoder, feature fusion module and multiple regression module. The outputs of the visual encoder and inertial encoder are input to the feature fusion module, and the output of the feature fusion module is input to the multiple regression module.

[0048] The visual encoder includes a first dynamic perception network, a second dynamic perception network, a third dynamic perception network, a fourth dynamic perception network, a fifth dynamic perception network, a sixth dynamic perception network, a seventh dynamic perception network, an eighth dynamic perception network, a ninth dynamic perception network and a first fully connected layer in series;

[0049] The inertial encoder consists of three one-dimensional convolutional neural networks and a second fully connected layer in series;

[0050] The feature fusion module includes a shared feature extractor and a multi-layer perceptron;

[0051] The multivariate regression module includes a long short-term memory network and a third fully connected layer.

[0052] Through the above-mentioned deep learning-based visual-inertial odometry, on the one hand, an efficient dynamic perception network is introduced to replace ordinary convolution, and a feature fusion module with enhanced cross-modal consistency is constructed, which can capture long-distance spatial information while eliminating the differences between the two modalities and enhancing their consistency. At the same time, it can achieve a high real-time processing speed with only a small memory usage, making it very suitable for deployment on resource-constrained embedded platforms. On the other hand, the multimodal information fusion strategy efficiently integrates visual information with inertial data, ensuring that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as urban, rural and high-speed driving, demonstrating excellent robustness and wide applicability. In addition, the end-to-end network architecture effectively simplifies the system design, avoids heavy initialization and optimization steps, thereby reducing computational complexity and implementation difficulty, and facilitating promotion and deployment in practical applications.

[0053] Below, we will refer to Figures 1 to 2 Each component of the deep learning-based visual-inertial odometry in this example embodiment is described in more detail.

[0054] In one embodiment, the visual encoder includes a first dynamic perceptual network, a second dynamic perceptual network, a third dynamic perceptual network, a fourth dynamic perceptual network, a fifth dynamic perceptual network, a sixth dynamic perceptual network, a seventh dynamic perceptual network, an eighth dynamic perceptual network, a ninth dynamic perceptual network and a first fully connected layer, which are arranged in series.

[0055] Specifically, such as Figure 2 As shown in Figure 1, the framework of the visual encoder is based on FlowNet-S, replacing the standard convolution with a dynamic perception network, and adding a fully connected layer at the end of the network to output a visual feature vector with a dimension of 512. The framework details are shown in Table 1:

[0056] Table 1

[0057]

[0058] More specifically, each dynamic perception network includes a dual-stream feature generation unit and a space-channel collaborative gating mechanism unit.

[0059] In one embodiment, the inertial encoder includes three one-dimensional convolutional neural networks and a second fully connected layer connected in series.

[0060] Specifically, the three-layer 1D CNN (i.e., one-dimensional convolutional neural network) in the inertial encoder has a kernel size of 3 and a stride of 1. Each layer is followed by LeakyReLU activation and batch normalization, and outputs 256-dimensional features. The details are shown in Table 2:

[0061] Table 2

[0062]

[0063] In one embodiment, the feature fusion module includes a shared feature extractor and a multi-layer perceptron.

[0064] Specifically, the feature fusion module consists of collaborative attention (i.e., shared feature extractor) and two MLPs (i.e., multi-layer perceptrons). The first part is used to generate shared features, with the dimension changing from 768 → 256 → 256, using LeakyReLU as the activation function; the second part is used for dynamic fusion, also using LeakyReLU as the activation function. The details are shown in Table 3:

[0065] Table 3

[0066]

[0067] In one embodiment, a long short-term memory network and a third fully connected layer.

[0068] Specifically, the long short-term memory network (LSTM) in the multivariate regression module is 2 layers with 1024 hidden units and two fully connected layers (feature dimensions 1024→128→6) outputting the translation vector and the rotation vector .

[0069] Furthermore, this example embodiment also provides a method for using a deep learning-based visual inertial odometry. Figure 3 As shown in , the deep learning-based visual inertial odometry may include:

[0070] Acquire an image pair and its corresponding IMU data; wherein the image pair is two consecutive image frames, and the IMU data is the IMU measurement data between the two consecutive image frames, and the IMU data includes accelerometer data and gyroscope data;

[0071] Utilize visual encoders to perform cheap convolution-based operations and cross-channel attention weighting on image pairs to generate visual features;

[0072] Use the inertial encoder to extract features from IMU data to obtain inertial features;

[0073] The feature fusion module is used to perform cross-modal consistency fusion of visual features and inertial features to obtain fused features;

[0074] The multivariate regression module is used to construct the temporal dependency relationship across time steps based on the fused features to obtain the relative pose between two consecutive image frames.

[0075] In one embodiment, the step of using a visual encoder to perform convolution-based cheap operations and cross-channel attention weighting on image pairs to generate visual features includes: a dual-stream feature generation unit extracts basic features of the image pair through a main feature extractor, and generates cheap auxiliary features in combination with deep convolution; a spatial-channel collaborative gating mechanism unit decomposes and fully connects the image pair in the horizontal and vertical directions respectively to calculate the attention score of the image pair in the horizontal direction and the attention score in the vertical direction; based on the horizontal attention score and the vertical attention score, the attention weight is normalized using the Sigmoid function; after splicing the basic features and the cheap auxiliary features, the visual features are multiplied by the normalized attention weight.

[0076] Specifically, the visual encoder and inertial encoder adopt discrete structural designs to adapt to the heterogeneous modal characteristics of visual and inertial data respectively; for the overall architecture of the visual encoder, the first nine layers of the convolutional optical flow network FlowNet-S are used and an additional fully connected layer is added as its feature encoder, such as Figure 2 As shown in the figure, the structure of the visual encoder and its working principle are demonstrated.

[0077] The optical flow technology in the visual encoder is used to directly characterize the pixel displacement between consecutive image frames and improve the robustness and scene adaptability of motion estimation by fusion of spatiotemporal features. The visual feature is represented as : , represents the visual encoder, and They are and Image frames per time step.

[0078] More specifically, the improved visual encoder uses an efficient dynamic perception network to replace the standard convolution in the optical flow network FlowNet-S to extract global image features. The dynamic perception network consists of a dual-stream feature generation unit and a spatial-channel collaborative gating mechanism.

[0079] Dual-stream feature generation unit: main feature extraction and auxiliary feature expansion, controlling the computational complexity of the entire module.

[0080] Space-channel collaborative gating mechanism unit: Enhances the global spatial information modeling capability.

[0081] The dual-stream feature generation unit generates basic features through the main feature extractor and combines deep convolution to generate cheap auxiliary features. The formula is:

[0082]

[0083] in, is the input feature, represents the convolution operation, is the depthwise convolution kernel.

[0084] The spatial-channel collaborative gating mechanism unit decomposes the fully connected operation along the horizontal and vertical directions, and the compressed sensing path is used to enhance the spatial information modeling capability. In general, the gating weight has a great influence on the low-cost auxiliary features. Select to have nonlinear modulation effect:

[0085]

[0086] in, Represents the Sigmoid function, which is used to normalize the attention value to the range of [0,1]. The input features are generated by the dual-stream feature generation module The result after transformation. For a given input feature , which can be considered as channel vector The traditional fully connected layer calculates the attention score in the following way:

[0087]

[0088] in, represents element-wise multiplication, is a learnable parameter. However, the computational complexity of this method is ,In order to reduce the amount of calculation, the space-channel collaborative gating mechanism unit calculates along the horizontal and vertical directions respectively:

[0089]

[0090] in, and are two independent transformation matrices. By this decomposition method, the computational complexity is reduced from the traditional downgraded to , significantly reducing the amount of computation while retaining the modeling capability of global information.

[0091] In one embodiment, the step of extracting features from IMU data using an inertial encoder to obtain inertial features includes: the inertial encoder uses a one-dimensional convolutional neural network to extract high-frequency dynamic motion features from multiple sets of IMU data between two frames of images to generate inertial features.

[0092] Specifically, 1D CNN contains multiple layers of one-dimensional convolution kernels, each layer is followed by an activation function and batch normalization operation, and outputs an inertial feature vector with a dimension of 256 to extract high-frequency dynamic motion features. The inertial feature is expressed as , represents the visual encoder, and They are the accelerometer data and gyroscope data between two frames of images.

[0093] In one embodiment, the step of using a feature fusion module to perform cross-modal consistency fusion of visual features and inertial features to obtain fused features includes: using a shared feature extractor to splice the visual features and inertial features and then inputting them into a collaborative attention mechanism to generate a first shared feature; dynamically weighting and adjusting the first shared feature and the inertial feature to obtain a second shared feature; fusing the second shared feature with the visual feature, and generating a fused feature through a multi-layer perceptron.

[0094] Specifically, shared features are generated through a shared feature extractor, and the inertial features are dynamically weighted and adjusted before being fused with the visual features.

[0095] The shared feature generation process of the cross-modal consistency enhancement module (i.e., feature fusion module) is as follows:

[0096]

[0097] The adjusted inertial feature (i.e., the second shared feature) is , the final fusion feature is:

[0098]

[0099] in, is the first shared feature, For coordinated attention.

[0100] In one embodiment, the step of using a multivariate regression module to construct a temporal dependency across time steps based on fused features to obtain the relative pose between two consecutive image frames includes: a long short-term memory network of the multivariate regression module to construct a temporal dependency across time steps based on the fused features; and a third fully connected layer of the multivariate regression module to project the temporal dependency across time steps to a 6-degree-of-freedom pose to obtain the relative pose between two consecutive image frames.

[0101] Specifically, the 6-DOF pose estimation is output based on the long short-term memory network (LSTM) and the fully connected layer.

[0102] In a specific embodiment, Figure 4 The following is a flowchart of a method for using a deep learning-based visual inertial odometry. The specific steps include:

[0103] 1. Data preprocessing:

[0104] Sequences 00, 01, 02, 04, 06, 08, and 09 of the KITTI dataset were selected as the training set, and sequences 05, 07, and 10 were selected as the test set. Sequence 03 lacked the original IMU data and was not used. The training set contains 17,337 images, and the test set contains 5,063 images. Images and ground truth poses are recorded at 10 Hz, while IMU data is recorded at 100 Hz. The image data comes from the left camera in the KITTI odometry dataset.

[0105] Image data: Input continuous monocular image frames (resolution 256), the length of the training subsequence is set to 11.

[0106] IMU data: collects 11 sets of accelerometer and gyroscope data between two frames (dimension 11), linear interpolation is used to achieve strict synchronization with the image and ground truth.

[0107] 2. Build a deep learning-based visual inertial odometry:

[0108] Visual Encoder: The framework is based on FlowNet-S, replacing the standard convolution with a dynamic perception network. A fully connected layer is added to the end of the network to output a visual feature vector of dimension 512. The framework details are shown in Table 4 below:

[0109] Table 4

[0110]

[0111] Inertial encoder: 3-layer 1D CNN (kernel size 3, stride 1), each layer is followed by LeakyReLU activation and batch normalization, outputting 256-dimensional features. Details are shown in Table 5:

[0112] Table 5

[0113]

[0114] Feature fusion module: It consists of collaborative attention and two MLP parts. The first part is used to generate shared features with a dimension change of 768 → 256 → 256, using LeakyReLU as the activation function; the second part is used for dynamic fusion, also using LeakyReLU as the activation function. The details are shown in Table 6:

[0115] Table 6

[0116]

[0117] Multivariate regression module: LSTM (2 layers, 1024 hidden units), two fully connected layers (feature dimensions 1024→128→6) outputting translation vectors and the rotation vector .

[0118] 3. Training Strategy

[0119] Loss function: Mean Squared Error (MSE) is used as the core metric to effectively reduce the error of pose estimation:

[0120]

[0121] Where T represents the number of time series frames, and Represent the true value and estimated value of the rotation vector of the t-th frame, respectively. and Represent the true value and estimated value of the translation vector of the t-th frame, respectively. is a weight coefficient used to balance the loss of rotation vector and translation vector. According to the experience of supervised learning VIO method with prior knowledge, Set to 100.

[0122] Optimizer: AdamW( ), batch_size is set to 14, and training epochs is 100. The learning rate decay strategy is adopted, and the initial learning rate is set to , which decays to , the 80th cycle further decreased to .

[0123] Regularization: Dropout (with a probability ranging from 0.2 to 0.5) is added during training to prevent overfitting.

[0124] 4. Experimental results:

[0125] Accuracy comparison: On the KITTI test set (sequence 05 / 07 / 10), the average The accuracy is 1.87%, which is 87.3% lower than that of VINS-Mono (12.7%) and 28.6% lower than that of the best supervised method CMIF-VIO (2.62%). The specific results are shown in Table 7 below.

[0126] Table 7 Sequence test results

[0127]

[0128] As can be seen from Table 7, the method of this application achieved the SOTA results on sequence 05 and sequence 07. Although it was slightly inferior to the best result on sequence 10, it also maintained an excellent level, verifying the superiority and robustness of the method of this application, as follows Figure 5 Shown is the visualization result of the method of the present application, which is very close to the ground truth.

[0129] Ablation experiment: using only dynamic perception network, average is 2.86%; using only the cross-modal consistency enhancement module, the average When used together, the average error is further optimized to 1.87%, proving that the modules have a significant synergistic effect. The specific results are shown in Table 8 below.

[0130] Table 8 Comparison results of ablation experiments

[0131]

[0132] As can be seen in Table 8, the effectiveness of the two modules in this application has been verified through independent experiments. Furthermore, the combined effect is even more significant. Neither the dynamic perception network nor the cross-modal consistency enhancement module, when used alone, achieves the expected optimal performance. Only by combining the two modules can their synergistic effect be fully utilized, resulting in a significant performance improvement.

[0133] Finally, the GPU memory usage of the overall model of this application (i.e., the visual inertial odometry based on deep learning) was tested during runtime, as shown in the following example: Figure 6 shown.

[0134] As can be seen from the figure, the overall model of this application only takes up 1.3GB of memory during runtime, which is lower than many other existing end-to-end visual odometry / visual-inertial odometry methods. This proves that this application not only has advantages in positioning accuracy, but also meets the actual needs of real-time applications and resource-constrained environments.

[0135] Through the above-mentioned deep learning-based visual-inertial odometry and its use method, on the one hand, an efficient dynamic perception network is introduced to replace ordinary convolution, and a feature fusion module with enhanced cross-modal consistency is constructed, which can capture long-distance spatial information while eliminating the differences between the two modalities and enhancing their consistency. At the same time, it can achieve a high real-time processing speed while maintaining a low memory usage, making it very suitable for deployment on resource-constrained embedded platforms. On the other hand, the multimodal information fusion strategy efficiently integrates visual information with inertial data, ensuring that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as urban, rural and high-speed driving, demonstrating excellent robustness and wide applicability. In addition, the end-to-end network architecture effectively simplifies the system design, avoids heavy initialization and optimization steps, thereby reducing computational complexity and implementation difficulty, and facilitating promotion and deployment in practical applications.

[0136] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0137] In the embodiments of the present disclosure, unless otherwise expressly specified or limited, the terms "installed," "connected," "connected," "fixed," and the like should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present disclosure based on specific circumstances.

[0138] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0139] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for using a visual inertial odometry based on deep learning, characterized in that: The method includes: Acquire an image pair and its corresponding IMU data; wherein the image pair is two consecutive image frames, and the IMU data is the IMU measurement data between the two consecutive image frames, and the IMU data includes accelerometer data and gyroscope data; The visual encoder is used to perform convolution-based cheap operations and cross-channel attention weighting on image pairs to generate visual features. Specifically, the dual-stream feature generation unit extracts the basic features of the image pair through the main feature extractor and generates cheap auxiliary features in combination with deep convolution. The spatial-channel collaborative gating mechanism unit decomposes the image pair into fully connected operations in the horizontal and vertical directions to calculate the horizontal and vertical attention scores of the image pair. Based on the horizontal and vertical attention scores, the attention weights are normalized using the sigmoid function. The basic features and cheap auxiliary features are concatenated and multiplied by the normalized attention weights to generate visual features. Use the inertial encoder to extract features from IMU data to obtain inertial features; The feature fusion module is used to perform cross-modal consistency fusion of visual features and inertial features to obtain fused features; The multivariate regression module is used to construct the temporal dependency relationship across time steps based on the fusion features to obtain the relative pose between two consecutive image frames; The deep learning-based visual-inertial odometry includes a visual encoder, an inertial encoder, a feature fusion module, and a multivariate regression module. The outputs of the visual encoder and the inertial encoder are both input into the feature fusion module, and the output of the feature fusion module is input into the multivariate regression module. The visual encoder includes a first dynamic perception network, a second dynamic perception network, a third dynamic perception network, a fourth dynamic perception network, a fifth dynamic perception network, a sixth dynamic perception network, a seventh dynamic perception network, an eighth dynamic perception network, a ninth dynamic perception network and a first fully connected layer in series; The inertial encoder consists of three one-dimensional convolutional neural networks and a second fully connected layer in series; The feature fusion module includes a shared feature extractor and a multi-layer perceptron; The multivariate regression module includes a long short-term memory network and a third fully connected layer; Each dynamic perception network includes a dual-stream feature generation unit and a space-channel collaborative gating mechanism unit.

2. The method for using the deep learning-based visual inertial odometry according to claim 1, characterized in that: The steps of extracting features from IMU data using an inertial encoder to obtain inertial features include: The inertial encoder uses a one-dimensional convolutional neural network to extract high-frequency dynamic motion features from multiple sets of IMU data between two frames of images to generate inertial features.

3. The method for using the deep learning-based visual inertial odometry according to claim 2, characterized in that: The steps of using the feature fusion module to perform cross-modal consistency fusion of visual features and inertial features to obtain fused features include: The shared feature extractor is used to concatenate the visual features and the inertial features and then input them into the collaborative attention mechanism to generate the first shared feature; Dynamically weighting and adjusting the first shared feature and the inertia feature to obtain a second shared feature; The second shared feature is fused with the visual feature, and a fused feature is generated through a multi-layer perceptron.

4. The method for using the deep learning-based visual inertial odometry according to claim 3, characterized in that: The step of using a multivariate regression module to construct a temporal dependency relationship across time steps based on fusion features to obtain the relative pose between two consecutive image frames includes: The long short-term memory network of the multivariate regression module constructs temporal dependencies across time steps based on the fused features; The third fully connected layer of the multivariate regression module projects the temporal dependencies across time steps into a 6-DOF pose to obtain the relative pose between two consecutive image frames.

Images