A monocular visual odometry positioning method based on end-to-end deep learning

Abstract

The present application belongs to the field of computer vision and robot navigation technology. The present application provides a monocular visual odometry positioning method based on end-to-end deep learning. The disclosed embodiment adopts an explicit scale correction method and performs deep modeling on the complete time series, which can accurately capture subtle motion changes between consecutive frames, thereby significantly improving the accuracy of the monocular visual odometry in translation and rotation estimation, and effectively solving the problems of error accumulation and scale ambiguity in traditional methods. The fusion module efficiently integrates the initial pose estimation, global depth information and temporal pose information to ensure that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as cities, rural areas and high-speed driving, showing excellent robustness and wide applicability.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the fields of computer vision and robot navigation technology, and in particular to a monocular visual odometry positioning method based on end-to-end deep learning. Background Art

[0002] At the forefront of robotics and intelligent perception, visual odometry (VO) is becoming a crucial enabling technology for autonomous robot navigation and environmental perception. While traditional positioning systems often rely on multi-sensor fusion and pre-built maps, VO directly infers the robot's trajectory and pose in space using continuous video frames captured by a camera through real-time image processing and motion estimation. The core challenge of VO lies in extracting robust and reliable motion features from raw visual data and accurately calculating the six-degree-of-freedom (6-DOF) pose transformation of the robot or camera between adjacent frames—information consisting of both translation and rotation. In particular, monocular VO relies solely on image sequences captured by a single camera, lacking direct depth information. This makes pose estimation more complex and unstable. In recent years, advancements in image processing algorithms, feature matching techniques, and deep learning methods have significantly improved the accuracy, robustness, and real-time performance of monocular VO, leading to its widespread application in fields such as autonomous driving, robotic navigation, and augmented reality.

[0003] Traditional visual odometry mainly relies on geometric methods, which are generally divided into two categories: indirect and direct methods:

[0004] (1) The indirect method is also called the feature point method. Its main idea is to first detect a set of representative feature points (such as SIFT, ORB, SURF, etc.) in the image, and then achieve matching between images by extracting the descriptors of these feature points. After the matching is completed, geometric constraints (such as the essential matrix or the fundamental matrix) and robust estimation methods such as RANSAC are used to eliminate false matches, thereby calculating the relative motion between adjacent frames. The indirect method usually estimates the pose based on sparse feature points, so it is relatively efficient in calculation and has good robustness to illumination changes. However, it relies on sufficient texture information in the image and is easily affected by motion blur and occlusion during feature point detection and matching. Large errors may accumulate over long periods of time.

[0005] (2) The direct method estimates camera motion by directly utilizing image pixel intensity information without explicitly extracting and matching feature points. This method assumes that the image satisfies brightness invariance between adjacent frames, that is, the same physical point has similar grayscale values in consecutive frames. The direct method solves the relative pose of the camera by minimizing the photometric error between consecutive frames using iterative optimization (such as Gauss-Newton or Levenberg-Marquardt algorithm). This method can utilize all pixel information in the image, so it can obtain a relatively stable estimate even in low-texture areas. However, the direct method is highly sensitive to illumination changes, motion blur, and large displacements, and is prone to falling into local optimality when initialized poorly. It has a large computational load and is sometimes difficult to guarantee real-time performance.

[0006] In recent years, deep learning technology has made significant progress in the field of visual odometry. End-to-end network models can learn pose mapping directly from image sequences, reducing engineering complexity and improving robustness to a certain extent.

[0007] However, existing end-to-end visual odometry systems based on deep learning have the following shortcomings:

[0008] (1) Scale ambiguity: Monocular systems lack absolute depth information, which often leads to scale uncertainty in translation estimation.

[0009] (2) Insufficient utilization of time series: Some models only use local time information and fail to fully capture the temporal dependencies in the complete series;

[0010] (3) High computing resource requirements: Although some hybrid methods improve accuracy, steps such as back-end bundling optimization bring about large computing overhead, which is not conducive to real-time applications.

[0011] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0012] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0013] The purpose of the embodiments of the present disclosure is to provide a monocular visual odometry positioning method based on end-to-end deep learning, thereby overcoming one or more problems caused by the limitations and defects of related technologies, at least to a certain extent.

[0014] According to an embodiment of the present disclosure, a monocular visual odometry positioning method based on end-to-end deep learning is provided, the method comprising:

[0015] Acquire continuous video frames, synchronously record the timestamp of each video frame, and preprocess each video frame to obtain an input image;

[0016] The timestamp is position-encoded using the sine-cosine harmonic function to obtain a high-dimensional mapping;

[0017] Implicitly transform the high-dimensional mapping to obtain temporal pose information; wherein the temporal pose information includes translation vectors and rotation vectors;

[0018] Initialize the input image to obtain an initial pose estimate; wherein the initial pose estimate includes an initial translation estimate and an initial rotation estimate;

[0019] Perform feature extraction on the input image to obtain a depth map of the input image;

[0020] Use wavelet convolution operations and convolutional layers to extract global attention of the depth map to obtain global depth information;

[0021] The initial pose estimation, global depth information and temporal pose information are multimodally fused to obtain a 6-DOF pose to complete positioning.

[0022] Furthermore, the method further comprises:

[0023] Construct a monocular visual odometry, which includes a preprocessing module, a pose initialization module, a temporal pose network, a depth estimation network, a depth information extraction module and a fusion module.

[0024] Furthermore, the steps of acquiring continuous video frames, synchronously recording the timestamp of each video frame, and preprocessing each video frame to obtain an input image include:

[0025] Continuously capture video frames at a predetermined frame rate and synchronously record the timestamp of each video frame;

[0026] The video frame is input into the preprocessing module for size normalization, color correction and noise suppression to obtain the input image to improve image quality and data consistency.

[0027] Furthermore, the step of initializing the input image to obtain an initial pose estimate includes:

[0028] The input image is input into the pose initialization module for initialization to obtain the initial translation estimate and the initial rotation estimate.

[0029] Furthermore, the step of position encoding the timestamp using the sine-cosine harmonic function to obtain a high-dimensional mapping includes:

[0030] Input the timestamp into the temporal pose network, use the sine-cosine harmonic function to generate 2L sine and cosine values of different frequencies according to the timestamp, and embed the timestamp into the high-dimensional space to obtain the corresponding high-dimensional mapping;

[0031] Among them, the expression of high-dimensional mapping is:

[0032]

[0033] Where L is the sequence length, is the sine-cosine harmonic function, is the timestamp.

[0034] Furthermore, the step of implicitly transforming the high-dimensional mapping to obtain temporal pose information includes:

[0035] The high-dimensional mapping is input into T-Net and R-Net in the temporal pose network respectively. T-Net implicitly transforms the high-dimensional mapping and uses MLP combined with ReLU activation function and tanh activation function to adjust the output to 3 dimensions and limit its value to the range of [-1, 1] to output the translation vector.

[0036] R-Net implicitly transforms the high-dimensional mapping, adjusts the output to 4 dimensions, represents the rotation in the form of a quaternion [qw,qx,qy,qz], performs tanh activation on the [qx,qy,qz] component, performs 1-tanh transformation on the [qw] component, and performs L2 normalization on the whole to construct a unit quaternion to output the rotation vector.

[0037] Furthermore, the step of extracting features from the input image to obtain a depth map of the input image includes:

[0038] The input image is input into the pre-trained depth estimation network for feature extraction to generate a depth map corresponding to the input image.

[0039] Furthermore, the step of extracting the global attention of the depth map using the wavelet convolution operation and the convolution layer to obtain the global depth information includes:

[0040] The depth map is input into the depth information extraction module, and the depth map is downsampled using wavelet convolution. The depth map is resized to H / 2 × W / 2, and all frequency bands are summed along the channel dimension to quadruple the number of channels to obtain the low-frequency sub-band, the first high-frequency sub-band, the second high-frequency sub-band, and the third high-frequency sub-band. Where H is the height of the depth map and W is the width of the depth map.

[0041] The convolutional layer is used to extract and reconstruct the low-frequency sub-band, the first high-frequency sub-band, the second high-frequency sub-band, and the third high-frequency sub-band to obtain a depth map feature of size H / 2×W / 2×D; where D is the number of channels;

[0042] Construct the Query, Key, and Value of the attention module, and calculate the attention score based on the depth map features to obtain global depth information;

[0043] Among them, the expression of global depth information is:

[0044]

[0045] in, is the global depth information, is the first weight matrix, is the second weight matrix, is the third weight matrix, represents the wavelet convolution operation, represents Sigmoid activation, is the depth map feature.

[0046] Furthermore, the step of performing multimodal fusion on the initial pose estimation, global depth information, and temporal pose information to obtain a 6-DOF pose includes:

[0047] The fusion module is used to perform linear weighted fusion of the initial translation estimate, global depth information and translation vector to obtain the translation fusion feature;

[0048] The fusion module is used to perform linear weighted fusion on the initial rotation estimate and the rotation vector to obtain the rotation fusion feature;

[0049] The preset pose regression multilayer perceptron in the fusion module is used to perform nonlinear mapping on the translation fusion features and rotation fusion features to obtain the 6-DOF pose.

[0050] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0051] In the embodiments disclosed herein, the monocular visual odometry positioning method based on end-to-end deep learning, on the one hand, by adopting an explicit scale correction method and performing deep modeling of the complete time series, can accurately capture subtle motion changes between consecutive frames, thereby significantly improving the accuracy of the monocular visual odometry in translation and rotation estimation, and effectively solving the problems of error accumulation and scale ambiguity in traditional methods. On the other hand, the fusion module efficiently integrates the initial pose estimation, global depth information, and temporal pose information, ensuring that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as urban, rural, and high-speed driving, demonstrating excellent robustness and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0053] Figure 1 A diagram showing the steps of a monocular visual odometry positioning method based on end-to-end deep learning in an exemplary embodiment of the present disclosure is shown;

[0054] Figure 2 A network structure diagram of a monocular visual odometry in an exemplary embodiment of the present disclosure is shown;

[0055] Figure 3 A schematic diagram illustrating a wavelet convolution operation in an exemplary embodiment of the present disclosure is shown;

[0056] Figure 4 1. A diagram showing the test results of sequence 09 in an exemplary embodiment of the present disclosure;

[0057] Figure 5 A diagram showing test results of sequence 10 in an exemplary embodiment of the present disclosure is shown;

[0058] Figure 6 The figure shows the GPU memory usage result in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0060] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0061] This example implementation provides a monocular visual odometry positioning method based on end-to-end deep learning. Figure 1 As shown in , the monocular visual odometry positioning method based on end-to-end deep learning may include:

[0062] Step S101: Acquire continuous video frames, synchronously record the timestamp of each video frame, and pre-process each video frame to obtain an input image;

[0063] Step S102: Position encoding the timestamp using a sine-cosine harmonic function to obtain a high-dimensional mapping;

[0064] Step S103: implicitly transforming the high-dimensional mapping to obtain temporal pose information; wherein the temporal pose information includes a translation vector and a rotation vector;

[0065] Step S104: Initializing the input image to obtain an initial pose estimation; wherein the initial pose estimation includes an initial translation estimation and an initial rotation estimation;

[0066] Step S105: performing feature extraction on the input image to obtain a depth map of the input image;

[0067] Step S106: extracting the global attention of the depth map using wavelet convolution operation and convolution layer to obtain global depth information;

[0068] Step S107: Perform multimodal fusion on the initial pose estimation, global depth information and temporal pose information to obtain a 6-DOF pose to complete positioning.

[0069] This end-to-end deep learning-based monocular visual odometry positioning method, on the one hand, utilizes an explicit scale correction method and performs deep modeling of the complete time series, accurately capturing subtle motion changes between consecutive frames. This significantly improves the accuracy of monocular visual odometry in translation and rotation estimation, effectively addressing the error accumulation and scale ambiguity issues inherent in traditional methods. Furthermore, the fusion module efficiently integrates initial pose estimation, global depth information, and temporal pose information, ensuring stable and reliable positioning performance in a variety of complex scenarios, including urban and rural environments, and high-speed driving, demonstrating exceptional robustness and broad applicability.

[0070] Below, we will refer to Figures 1 to 6 Each step of the above-mentioned monocular visual odometry positioning method based on end-to-end deep learning in this example embodiment is described in more detail.

[0071] In step S101 , continuous video frames are acquired, a timestamp of each video frame is synchronously recorded, and each video frame is pre-processed to obtain an input image.

[0072] Specifically, in the initial stage, the monocular camera continuously captures video frames at a predetermined frame rate (10Hz). Each frame is accompanied by a precise timestamp to ensure accurate timing information during subsequent processing. The captured video frames first pass through a preprocessing module, which performs size normalization, color correction, and noise suppression on each frame to improve image quality and data consistency, thereby ensuring high reliability of the input data for the depth estimation network.

[0073] In step S102 and step S103, the timestamp is position-encoded using a sine-cosine harmonic function to obtain a high-dimensional mapping; the high-dimensional mapping is implicitly transformed to obtain temporal pose information; wherein the temporal pose information includes a translation vector and a rotation vector.

[0074] Specifically, for each timestamp t, the position is first encoded using the sine-cosine harmonic function to obtain a high-dimensional mapping t*. By using the sin and cos functions to transform the timestamp at different frequencies, it is mapped into a d-dimensional vector containing multi-level frequency domain information. This method not only captures periodic changes and smooth continuity in time, but also reflects subtle differences between adjacent frames, providing rich and stable temporal features for subsequent networks, thereby achieving accurate and continuous mapping from time to position.

[0075] Input t* into the time pose network TimePoseNet (including T_Net and R_Net), T_Net outputs the translation vector v∈R³, and R_Net outputs the rotation quaternion (i.e., rotation vector) q∈R 4 (satisfy ).

[0076] Among them, T-Net consists of 8 layers of MLP. The output dimension of the last layer is fixed to 3, corresponding to the translation vector of three degrees of freedom. The middle layer uses the ReLU activation function, and the last layer uses the tanh activation function to limit the translation value to the [-1, 1] interval.

[0077] R-Net also consists of 8 MLP layers. The final layer outputs a fixed dimension of 4, corresponding to the quaternion [qw, qx, qy, qz], which represents rotation. The intermediate layers use the Reluctant Unit (ReLU) activation function, and the final layer performs a tanh activation on the [qx, qy, qz] components and a 1-tanh transformation on the qw component. The entire quaternion is then L2-normalized to ensure the output is a unit quaternion.

[0078] In step S104, the input image is initialized to obtain an initial pose estimate, wherein the initial pose estimate includes an initial translation estimate and an initial rotation estimate. Obtaining the initial pose estimate is a prior art and will not be described in detail here.

[0079] In step S105 and step S106, feature extraction is performed on the input image to obtain a depth map of the input image; and global attention of the depth map is extracted using a wavelet convolution operation and a convolution layer to obtain global depth information.

[0080] Specifically, the preprocessed input image is then fed into a pre-trained depth estimation network, which uses an advanced convolutional neural network structure to quickly generate a depth map corresponding to the current frame, providing a solid foundation for subsequent scale correction and feature extraction.

[0081] Wavelet convolution is used to losslessly downsample the depth map. Leveraging the simplicity and efficiency of Haar wavelets, the depth map is decomposed into four subbands (low-frequency subband, first high-frequency subband, second high-frequency subband, and third high-frequency subband). Appropriate combinations of convolution kernels are then used to extract and reconstruct the image's low-frequency information. The downsampled depth map is reduced to half its original size, with 64 channels. This reduces computational complexity while providing stable and accurate global depth information for subsequent feature extraction and attention mechanisms.

[0082] The depth features are extracted through the subsequent convolutional layer, and the Query, Key, and Value of the attention module are constructed. After calculating the attention score, the global depth feature vector F_d (i.e., global depth information) is obtained. This vector fully reflects the depth distribution of the entire image and provides reliable data information and guidance for subsequent translation and scale correction.

[0083] In step S107, the initial pose estimate (derived from the initial pose network) and the temporal pose information obtained by TimePoseNet are each expanded to the same dimension as F_d through a fully connected layer to ensure effective integration of features from different modalities. Using the fusion module, the initial translation features are first linearly weighted fused with F_d to produce the fused features F_fuse (i.e., translational fusion features). Simultaneously, the rotational features are similarly processed to produce rotational fusion features. Finally, the fused high-dimensional features of the translational and rotational fusion features are input into a pre-defined pose regression multilayer perceptron, which outputs the final 6-DOF pose P_final, completing the transformation from multi-source information to an accurate pose estimate.

[0084] In a specific embodiment, Figure 2 As shown in the figure, the network structure diagram of the monocular visual odometry, its specific processing flow is as follows:

[0085] Preprocess the input adjacent frame images, crop them to the appropriate size and convert them into the corresponding format, and obtain the initial pose estimation through the pose initialization module, where the initial pose estimation includes the initial translation estimation and the initial rotation estimation.

[0086] To learn the time-to-pose mapping, we designed an MLP network called TimePoseNet (i.e., temporal pose information). This achieves an accurate and continuous mapping from time to pose, yielding the temporal pose information fθ(t). TimePoseNet consists of two independent sub-networks, T_Net and R_Net, which learn the mapping of temporal input to translation and rotation, respectively. Each network consists of an 8-layer MLP with ReLU activation functions and 256-dimensional hidden units.

[0087] Before inputting the network, the timestamp t is first position-encoded using the sine-cosine harmonic function. For each timestamp t, the γ function (i.e., the sine-cosine harmonic function) generates 2L values, which are different frequencies (given by Decision, among which i From 0 to L-1) sine and cosine values, thus embedding the time variable t into the high-dimensional space and obtaining its high-dimensional mapping , the formula is as follows:

[0088] (1)

[0089] Where L represents the sequence length.

[0090] Will Input into the temporal pose network (including T_Net and R_Net), T_Net outputs the translation vector , R_Net outputs the rotation quaternion (i.e., the rotation vector) (satisfy ).

[0091] The current frame image (i.e., video frame) is input into the depth estimation network to obtain the depth map of the image. Combined with the wavelet transform to extract global attention, global depth information is obtained. The input wavelet convolution is downsampled and its shape is adjusted to H / 2×W / 2. Then all frequency bands are added along the channel dimension to quadruple the number of channels. Next, the convolution layer is used to extract features and increase the dimension. Finally, the depth map feature with a shape of H / 2×W / 2×D is obtained. , where the number of channels D is set to 64.

[0092] The depth map features after wavelet convolution Input the linear layer, construct Query, Key and Value, perform attention calculation according to formula (2), obtain global depth information, and finally convert it into a depth feature vector suitable for subsequent fusion .

[0093] (2)

[0094] in, is the first weight matrix, is the second weight matrix, is the third weight matrix, represents the wavelet convolution operation, represents Sigmoid activation, is the depth map feature.

[0095] Finally, after obtaining the global depth information After obtaining the time pose information fθ(t), the fusion module is used to perform effective multimodal fusion of the initial pose, depth map information and time information, and finally mapped into the required 6-DOF pose through the fully connected layer.

[0096] In a specific embodiment, Figure 3 The figure below shows the principle diagram of the wavelet convolution operation. The wavelet transform (i.e., wavelet convolution operation) divides the image into a low-frequency and three high-frequency parts, which are then convolved together across channels. This allows a larger receptive field to be obtained using only a small convolution kernel, saving computational resources while improving feature extraction. It also provides additional frequency domain information for subsequent features, facilitating a better focus on image details. At the same time, based on the characteristics of the wavelet transform, the image is also downsampled during the frequency domain transform (a single wavelet transform can convert an H×W×D feature map into H / 2×W / 2×4D). Since data from all frequency bands is subsequently used, it can be considered that the wavelet transform achieves lossless downsampling, reducing information loss during the entire feature extraction process.

[0097] In a specific embodiment, the present application is implemented on a 12th Gen Intel(R) Core(TM) i5-12400F 2.50 GHz CPU, 64G memory, NVIDIA RTX 3090 graphics card, Ubuntu 20.04 operating system, based on Pytorch 1.7.1 and Python 3.9 language environment.

[0098] The KITTI odometry dataset is a widely used benchmark in autonomous driving and computer vision research, jointly released by KIT and TTI Chicago. It contains 22 stereo sequences saved in lossless PNG format: 11 sequences (00-10) contain ground truth trajectories, and 10 sequences (11-21) do not, and are commonly used for testing.

[0099] The KITTI odometry evaluation metric examines subsequences ranging in length from 100 to 800 meters, reporting the average translation and rotation errors. Absolute trajectory error (ATE) measures the root mean squared error between the predicted camera pose and the ground truth, while relative pose error (RPE) captures the relative pose error between frames. Lower values indicate closer results to the ground truth.

[0100] 1. Training Settings

[0101] During system training, this application used the 00-08 sequences of the KITTI dataset as supervised data and the AdamW optimizer to train the entire network. During training, the system used mean squared error (MSE) as the loss function for both translation and rotation, and applied an additional weight (multiplied by 100) to the rotation loss to ensure that rotation errors were fully accounted for. The entire network was trained in 30 epochs, with each batch processing 32 frames. A linear learning rate decay strategy was used to ensure smooth convergence during training.

[0102] 2. Experimental Results

[0103] First, we tested and compared the proposed method with existing monocular visual odometry methods on sequences 09 and 10, which were not seen during training. As shown in Table 1, the experimental results demonstrate that the proposed method achieves optimal performance across most metrics on both unknown sequences. This demonstrates that the proposed method exhibits high overall accuracy and stability. The best results are shown in bold, while suboptimal results are underlined.

[0104] Table 1 Sequence test results

[0105]

[0106] At the same time, a visual comparison of the ground truth of each comparison method and the system comparison of this application is also carried out. The results are as follows Figure 4 and Figure 5 As shown. Among them, Figure 4 This is the test result diagram of sequence 09; Figure 5 This is the test result diagram for sequence 10.

[0107] from Figure 4 and Figure 5 It can be seen that the method proposed in this application is closest to the ground truth results in both sequences and is more suitable for monocular visual odometry tasks in a wide range of scenarios.

[0108] Finally, the GPU memory usage during runtime was tested for the overall system of this application, such as Figure 6 shown.

[0109] from Figure 6 It can be seen that when this application is running, the memory usage is only 2.3G, which is lower than many other existing end-to-end visual odometry methods. This proves that this application not only has advantages in positioning accuracy, but also meets the actual needs of real-time applications and resource-constrained environments.

[0110] Through the above-mentioned monocular visual odometry positioning method based on end-to-end deep learning, on the one hand, by adopting an explicit scale correction method and performing deep modeling of the complete time series, it is possible to accurately capture subtle motion changes between consecutive frames, thereby significantly improving the accuracy of the monocular visual odometry in translation and rotation estimation, and effectively solving the problems of error accumulation and scale ambiguity in traditional methods. On the other hand, the fusion module efficiently integrates the initial pose estimation, global depth information, and temporal pose information to ensure that the system can maintain stable and reliable positioning performance in a variety of complex scenarios such as urban, rural, and high-speed driving, demonstrating excellent robustness and wide applicability. At the same time, the end-to-end network architecture effectively simplifies the system design, avoids the heavy back-end bundling optimization steps in traditional methods, thereby reducing the computational complexity and implementation difficulty, and facilitating promotion and deployment in practical applications.

[0111] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0112] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0113] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A monocular visual odometry positioning method based on end-to-end deep learning, characterized in that: The method includes: Construct a monocular visual odometry, which includes a preprocessing module, a pose initialization module, a temporal pose network, a depth estimation network, a depth information extraction module, and a fusion module. Acquire continuous video frames, synchronously record the timestamp of each video frame, and preprocess each video frame to obtain an input image; The timestamp is positionally encoded using the sine-cosine harmonic function to obtain a high-dimensional mapping. Specifically, the timestamp is input into the temporal pose network, 2L sine and cosine values of different frequencies are generated according to the timestamp using the sine-cosine harmonic function, and the timestamp is embedded in the high-dimensional space to obtain the corresponding high-dimensional mapping. The expression of the high-dimensional mapping is: Where L is the sequence length, is the sine-cosine harmonic function, is the timestamp; Implicitly transform the high-dimensional mapping to obtain temporal pose information; wherein the temporal pose information includes translation vectors and rotation vectors; Initialize the input image to obtain an initial pose estimate; wherein the initial pose estimate includes an initial translation estimate and an initial rotation estimate; Perform feature extraction on the input image to obtain a depth map of the input image; The global attention of the depth map is extracted using wavelet convolution operations and convolution layers to obtain global depth information. The expression of global depth information is: in, is the global depth information, is the first weight matrix, is the second weight matrix, is the third weight matrix, represents the wavelet convolution operation, represents Sigmoid activation, is the depth map feature; The initial pose estimation, global depth information and temporal pose information are multimodally fused to obtain a 6-DOF pose to complete positioning.

2. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 1 is characterized in that: The steps of acquiring continuous video frames, synchronously recording the timestamp of each video frame, and preprocessing each video frame to obtain an input image include: Continuously capture video frames at a predetermined frame rate and synchronously record the timestamp of each video frame; The video frame is input into the preprocessing module for size normalization, color correction and noise suppression to obtain the input image to improve image quality and data consistency.

3. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 2 is characterized in that: The steps of initializing the input image to obtain the initial pose estimate include: The input image is input into the pose initialization module for initialization to obtain the initial translation estimate and the initial rotation estimate.

4. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 3 is characterized in that: The steps of implicitly transforming the high-dimensional mapping to obtain temporal pose information include: The high-dimensional mapping is input into T-Net and R-Net in the temporal pose network respectively. T-Net implicitly transforms the high-dimensional mapping and uses MLP combined with ReLU activation function and tanh activation function to adjust the output to 3 dimensions and limit its value to the interval [-1, 1] to output the translation vector. R-Net implicitly transforms the high-dimensional mapping, adjusts the output to 4 dimensions, represents the rotation in the form of a quaternion [qw,qx,qy,qz], performs tanh activation on the [qx,qy,qz] component, performs 1-tanh transformation on the [qw] component, and performs L2 normalization on the whole to construct a unit quaternion to output the rotation vector.

5. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 4 is characterized in that: The step of extracting features from an input image to obtain a depth map of the input image includes: The input image is input into the pre-trained depth estimation network for feature extraction to generate a depth map corresponding to the input image.

6. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 5 is characterized in that: The steps of extracting the global attention of the depth map using wavelet convolution operations and convolution layers to obtain global depth information include: The depth map is input into the depth information extraction module, and the depth map is downsampled using wavelet convolution. The depth map is resized to H / 2 × W / 2, and all frequency bands are summed along the channel dimension to quadruple the number of channels to obtain the low-frequency sub-band, the first high-frequency sub-band, the second high-frequency sub-band, and the third high-frequency sub-band. Where H is the height of the depth map and W is the width of the depth map. The convolutional layer is used to extract and reconstruct the low-frequency sub-band, the first high-frequency sub-band, the second high-frequency sub-band, and the third high-frequency sub-band to obtain a depth map feature of size H / 2×W / 2×D; where D is the number of channels; Construct the Query, Key, and Value of the attention module, and calculate the attention score based on the depth map features to obtain global depth information.

7. The monocular visual odometry positioning method based on end-to-end deep learning according to claim 6 is characterized in that: The steps of performing multimodal fusion of the initial pose estimate, global depth information, and temporal pose information to obtain a 6-DOF pose include: The fusion module is used to perform linear weighted fusion of the initial translation estimate, global depth information and translation vector to obtain the translation fusion feature; The fusion module is used to perform linear weighted fusion on the initial rotation estimate and the rotation vector to obtain the rotation fusion feature; The preset pose regression multilayer perceptron in the fusion module is used to perform nonlinear mapping on the translation fusion features and rotation fusion features to obtain the 6-DOF pose.

Images