Monocular visual odometry system and method based on bird's eye view representation

Abstract

The application discloses a monocular visual odometer system and method based on bird's-eye view representation, comprising: a visual BEV encoder for extracting image features from monocular images and predicting the depth distribution probability of the features, then fusing the image features with their depth distribution probability and mapping the image features to BEV to obtain bird's-eye view representation, and then performing camera coordinate system to ontology coordinate system conversion and height direction compression to obtain compressed bird's-eye view representation; a correlation feature extraction module for calculating the correlation between the compressed bird's-eye view representations of adjacent two key frame monocular images according to the compressed bird's-eye view representation and generating a correlation volume; and a pose prediction decoder for performing pose prediction of a target object according to the correlation volume. The system and method not only perform well in reducing scale drift and improving motion estimation accuracy, but also can be widely applied to automatic driving, robot navigation and other fields, and help to improve the navigation ability and reliability of autonomous systems in complex environments.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, robot navigation and automatic driving, and particularly relates to a monocular visual odometry system based on bird's eye view representation and a method thereof. BACKGROUND

[0002] Monocular visual odometry (MVO) has a wide range of applications in autonomous navigation and robotics due to its high cost-effectiveness and ease of deployment. MVO can serve as an economical and flexible complement to global positioning systems (GPS) and inertial navigation systems, providing real-time motion tracking capabilities. However, despite its many advantages, the widespread application of MVO is still limited by a key problem, namely scale ambiguity. Due to the lack of depth information in monocular vision systems, motion is usually estimated in relative scale, leading to accumulated errors over time, a phenomenon known as scale drift.

[0003] Traditional MVO methods, such as feature-based methods, semi-direct methods and direct methods, usually determine the scale in the initialization phase and use it as a global reference. This method closely links scale estimation with initial motion, making motion tracking performance highly dependent on the speed of motion at startup. In addition, these methods are heavily dependent on the initial scale setting, leading to increasingly serious scale drift problems over time.

[0004] In recent years, learning-based MVO methods have utilized the powerful fitting capabilities of deep learning models to attempt to solve this problem by modeling prior distributions from training data. These methods use convolutional neural networks (CNNs) to automatically extract features from images and perform pose regression based on temporal modeling methods. For example, some studies combine the interpretability of traditional methods with the powerful data fitting capabilities of deep learning, attempting to improve the accuracy of scale estimation by introducing auxiliary tasks such as depth supervision or optical flow supervision. However, these methods often require the introduction of additional supervised data, such as depth estimation or optical flow estimation, which not only increases computational complexity but also brings higher data acquisition costs.

[0005] With the progress of Bird's Eye View (BEV) conversion technology and the excellent performance of BEV representation in 3D detection and scene segmentation, some studies begin to use BEV representation to realize visual odometry. However, these methods still often rely on scene segmentation results to estimate the pose, which raises a question: whether the inherent scale consistency of BEV representation is derived from BEV representation itself or relies on the supervision of scene segmentation. In addition, these methods have high cost when obtaining labeled data, and do not fully utilize the information directly provided by BEV representation.

[0006] Although the traditional MVO method shows high robustness in various environments, it is difficult to avoid the scale drift problem due to the inherent limitations of monocular vision. Although the deep learning method can improve the accuracy of motion estimation through supervised learning, it also faces challenges such as difficulty in obtaining supervised data and high computational complexity. In addition, although the method based on BEV representation can simplify the odometry problem in some cases, the accuracy and robustness of the scene segmentation supervision information it relies on is still a problem to be solved. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide a monocular visual odometry system and method based on bird's eye view representation, which reduces or eliminates the scale drift phenomenon in the MVO system by utilizing the inherent scale consistency of BEV representation, thereby achieving more accurate and stable motion estimation. By optimizing the image feature and depth distribution feature matching process, the dependence on explicit depth estimation, optical flow estimation and other tasks that require additional supervision is avoided, thereby simplifying the structure of the MVO system and reducing the cost of calculation and data acquisition.

[0008] To achieve the above-mentioned purpose of the application, an embodiment provides a monocular visual odometry system based on bird's eye view representation, comprising a visual BEV encoder, a related feature extraction module, and a pose prediction decoder.

[0009] The visual BEV encoder is used to extract image features from monocular images and predict the depth distribution probability of the features, then fuse the image features with their depth distribution probability and map them to BEV to obtain bird's eye view representation, and then convert the camera coordinate system to the body coordinate system and compress the height direction to obtain compressed bird's eye view representation.

[0010] The related feature extraction module is used to calculate the correlation between the compressed bird's eye view representations of adjacent two key frame monocular images according to the compressed bird's eye view representation, and generate a correlation volume.

[0011] The pose prediction decoder is used to predict the pose of the target object according to the correlation volume, wherein the pose includes a translation vector and a rotation matrix.

[0012] Optionally, the visual BEV encoder comprises a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule, and a feature mapping submodule.

[0013] The feature extraction submodule is configured to extract multi-scale perspective view feature maps from monocular images.

[0014] The camera parameter auxiliary encoding submodule is configured to encode camera parameters and obtain weighted feature maps by channel attention weighting of the camera parameter encoding and the perspective view feature maps.

[0015] The depth distribution prediction submodule is configured to predict the depth distribution probability of features based on the weighted feature maps through a deep network.

[0016] The feature mapping submodule is configured to fuse the image features after convolution of the weighted feature maps and the depth distribution probability of features, and obtain the bird's eye view angle representation after mapping to the BEV, and obtain the compressed bird's eye view angle representation by converting the bird's eye view angle representation from the camera coordinate system to the body coordinate system and compressing in the height direction.

[0017] Optionally, the feature extraction submodule adopts the structure of ResNet-50+FPN to extract multi-scale perspective view feature maps.

[0018] Optionally, the camera parameter auxiliary encoding submodule adopts MLP to encode the camera parameters.

[0019] Optionally, the deep network adopted by the depth distribution prediction submodule comprises Res Block, ASPP, and DCN connected in sequence, wherein the Res Block reduces the dimension of the weighted feature, the ASPP expands the receptive field of the reduced feature and captures depth features of different scales, and the DCN further extracts the depth features to obtain the depth distribution features.

[0020] Optionally, in the feature mapping submodule, the process of fusing the image features after convolution of the weighted feature maps and the depth distribution probability of features is as follows:

[0021] The size of the image features is changed from the original CxH feature xW feature to Cx1xH feature xW feature , wherein C represents the number of channels, H feature represents the height of the feature map, W feature represents the width of the feature map, and 1 is an introduced dimension.

[0022] The size of the depth distribution probability is changed from the original DxH feature xW featureInserting one dimension becomes 1 x D x H feature x W feature , wherein D represents the dimension number of the depth of the depth distribution probability;

[0023] The image feature with a size of C x 1 x H feature x W feature is multiplied by the depth distribution probability with a size of 1 x D x H feature x W feature , to obtain the depth distribution probability of each feature, forming a matrix of C x D x H feature x W feature .

[0024] Optionally, the related feature extraction module comprises a feature translation sub-module, a correlation calculation sub-module, and a related volume generation sub-module;

[0025] The feature translation sub-module is configured to perform translation operations with different step lengths on the compressed bird's eye view angle representation of the next frame monocular image and record an offset matrix with a size of D x x D y , to obtain a series of translated bird's eye view angle representations.

[0026] The correlation calculation sub-module is configured to calculate a correlation score matrix with a size of H BEV x W BEV from the series of translated bird's eye view angle representations, wherein the correlation score captures the relative motion information between the adjacent two frame monocular images.

[0027] The related volume generation sub-module is configured to generate a four-dimensional related volume with a size of D x x D y x H BEV x W BEV from the correlation score matrix and the offset matrix.

[0028] Optionally, the cosine similarity is used to calculate the correlation score of the adjacent two frame monocular images.

[0029] Optionally, the pose prediction decoder comprises a feature fusion and dimension reduction sub-module, a fully connected sub-module, and a post-processing sub-module.

[0030] The feature fusion and dimension reduction sub-module is configured to combine the offset dimension in the four-dimensional related volume and then reduce the dimension through a convolution layer, to output a three-dimensional correlation feature map.

[0031] The fully connected sub-module is configured to expand the three-dimensional correlation feature map into a one-dimensional vector and reduce the dimension through a series of fully connected layers, and then predict two outputs based on the one-dimensional vector through two branches, respectively, to obtain a total of four outputs.

[0032] The post-processing submodule is used for scaling tanh activation of two outputs of one branch to obtain translation vectors in x and y directions, and sequentially performing tanh activation and L2 regularization on two outputs of another branch to obtain cos and sin values of a yaw angle, and further transforming to form a rotation matrix.

[0033] Optionally, the visual BEV encoder, the correlation feature extraction module and the pose prediction decoder need to be optimized in parameters before being applied, and a loss function used in the parameter optimization is constructed based on translation error and rotation error of predicted translation vectors, rotation matrices and corresponding labels to update parameters.

[0034] To achieve the above-mentioned purposes, the embodiment of the present application further provides a monocular visual odometry method based on bird's eye view representation, which adopts the system and comprises the following steps:

[0035] Image features are extracted from monocular images and the depth distribution probability of the features is predicted by using the visual BEV encoder, then the image features and the depth distribution probability thereof are fused and mapped to the BEV to obtain bird's eye view representation, and then the conversion from the camera coordinate system to the body coordinate system and the compression in the height direction are performed to obtain compressed bird's eye view representation.

[0036] The correlation between the compressed bird's eye view representations of adjacent two key frame monocular images is calculated according to the compressed bird's eye view representation by using the correlation feature extraction module, and a correlation volume is generated.

[0037] The pose prediction of the target object is performed according to the correlation volume by using the pose prediction decoder, wherein the pose includes a translation vector and a rotation matrix.

[0038] Compared with the prior art, the present application has at least the following beneficial effects:

[0039] In the method of the present application, the bird's eye view representation is used to reduce the scale drift of monocular visual odometry in long sequence application, thereby improving the accuracy and robustness of monocular visual odometry. The advantage is that the plane assumption commonly used in plane automatic driving tasks is used to simplify the 6 degree of freedom odometry estimation task to 3 degree of freedom; although the information related to depth needs to be predicted when the BEV feature map is projected, the present application uses the method of predicting the depth distribution probability of high-dimensional feature points to replace the method of predicting the absolute depth of each pixel in the original monocular visual odometry, so that the network has better generalization and is easier to train.

[0040] In the method of the present application, the complex binding adjustment and loop detection steps in the traditional method are eliminated, and the MVO can be realized only by using the visual BEV encoder, the correlation feature extraction module and the pose prediction decoder, thereby simplifying the traditional MVO process and improving the real-time performance of the system.

[0041] In the method, dependence on additional supervision data (such as depth information and optical flow information) is avoided, and training is performed only through a loss function obtained by predicting an error of a real relative pose, so that the cost of data acquisition and processing is reduced, and applicability in various application scenarios is improved.

[0042] In summary, the method provides a technical solution capable of significantly improving the accuracy and robustness of a monocular visual odometry system without increasing the cost of additional hardware and computation, and is particularly suitable for application scenarios such as robot navigation and automatic driving that require long-term reliable motion estimation. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0044] Figure 1 is a structural schematic diagram of a monocular visual odometry system based on bird's-eye view representation provided by the embodiment;

[0045] Figure 2 is a flowchart of a monocular visual odometry method based on bird's-eye view representation provided by the embodiment;

[0046] Figure 3 is an intermediate result visualization diagram and a final trajectory diagram provided by the embodiment;

[0047] Figure 4 is a trajectory comparison result under different data sets provided by the embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0049] The technical concept of the present application is that in the existing MVO technology, due to the lack of absolute depth information of the monocular vision system, the motion can usually be estimated only in the relative scale, which leads to the accumulated error in long-time operation, seriously affecting the accuracy of motion estimation and the robustness of the system. The existing traditional MVO technology based on feature matching and geometric method, and the end-to-end MVO method relying on deep learning, are difficult to reliably solve the scale drift problem without additional supervision data. To solve the scale drift problem commonly existing in the monocular visual odometry (MVO) system, the embodiment of the present application provides a monocular visual odometry system based on bird's eye view representation and a method thereof, which utilizes the inherent scale consistency of BEV representation, significantly reduces the scale drift problem in long-time operation, and does not need to rely on depth supervision or complex optimization technology. The experimental results of this method on multiple data sets show that compared with the existing MVO method, the present application has significant advantages in improving the motion estimation accuracy and reducing the scale drift.

[0050] As shown in Figure 1 The embodiment provides a monocular visual odometry system based on bird's eye view representation, which comprises a visual BEV encoder, a related feature extraction module and a pose prediction decoder.

[0051] In the embodiment, the visual BEV encoder is used to extract image features from monocular images and predict the depth distribution probability of the features, then fuse the image features with their depth distribution probability and map them to BEV to obtain the bird's eye view representation, and then convert the camera coordinate system to the body coordinate system and compress the height direction to obtain the compressed bird's eye view representation.

[0052] Specifically, the visual BEV encoder comprises a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule and a feature mapping submodule. The feature extraction submodule is used to extract multi-scale perspective view feature maps from monocular images. In this submodule, the structure of ResNet-50+FPN is adopted, and specifically, ResNet-50 is used as the feature extraction backbone network to extract multi-scale features from monocular images (input images). In order to enhance the expressiveness of the features, these features are fused through a feature pyramid network (FPN) to generate multi-scale perspective view (PV) feature maps Fpv.

[0053] The camera parameter auxiliary coding submodule is used for encoding camera parameters (camera intrinsic parameters and extrinsic parameters of the camera and the body), and fusing the camera parameter coding and the perspective view feature map Fpv through an SE layer to obtain a weighted feature map. In this submodule, the camera parameters are encoded through an MLP, and in the channel attention weighting, the camera parameter coding is processed through the convolution layer and the activation function of the SE layer to generate a weighted map, and then the perspective view feature map Fpv is subjected to element-level multiplication operation to obtain the final weighted feature map.

[0054] The depth distribution prediction submodule is used for predicting the depth distribution probability of the feature based on the weighted feature map through a deep network. It is emphasized that the generation of the depth distribution probability does not depend on depth supervision, but is optimized through pose supervision, so that the depth distribution prediction process is closely combined with pose estimation. Specifically, the deep network adopted by the depth distribution prediction submodule includes Res Block, ASPP and DCN connected in turn, wherein the Res Block reduces the dimension of the weighted feature to reduce the calculation complexity, the ASPP (Atrous Spatial Pyramid Pooling) expands the receptive field of the reduced dimension feature and captures depth features of different scales, and the DCN further extracts the depth features to obtain the depth distribution probability of each pixel point, which provides key depth information Fdepth for subsequent feature mapping to the BEV space.

[0055] The feature mapping submodule is used for mapping the weighted feature map to the BEV space based on the depth distribution feature through Frustum Projection to generate a BEV representation, and then compressing the information in the Z-axis direction through voxel pooling technology to finally form a BEV representation with multi-dimensional features. Specifically, the image feature Fpv1 after convolution of the weighted feature map is fused with the depth distribution probability of the feature, and the fusion method is:

[0056] The size of the image feature is changed from the original CxH feature xW feature to Cx1xH feature xW feature , wherein C represents the number of channels, H feature represents the height of the feature map, W feature represents the width of the feature map, and 1 is an introduced dimension; the size of the depth distribution probability is changed from the original DxH feature xW feature to 1xDxH feature xW feature , wherein D represents the number of dimensions of the depth of the depth distribution probability; the size of the weighted feature map is changed from the original Cx1xH feature× W feature The image features are multiplied by the depth distribution probability of size 1 x D x H feature × W feature to obtain the depth distribution probability of each feature, forming a C x D x H feature × W feature matrix.

[0057] The weighted fusion not only realizes the perspective projection, but also maps the image features into the bird's eye view perspective of the three-dimensional space. Specifically, each pixel point in the two-dimensional image is projected into the three-dimensional space according to its depth information, thereby generating a BEV representation. The BEV representation indicates that the feature is in the BEV space of the camera.

[0058] Next, the bird's eye view representation also needs to be converted from the camera coordinate system to the body coordinate system and compressed in the height direction, where the height direction compression is completed by voxel pooling to obtain the compressed bird's eye view representation. Specifically, by applying the camera intrinsic matrix, the transformation matrix of the sensor to the target object (such as a vehicle) coordinate system, etc., the feature points in the three-dimensional space are converted from the camera coordinate system to the body coordinate system. This process ensures that image features of different cameras and different perspectives can be uniformly mapped into the BEV space of the body coordinate system. After being converted to the BEV space of the body coordinate system, voxel pooling operations are also performed on these three-dimensional features. Voxel pooling converts three-dimensional features into two-dimensional BEV representations with multi-dimensional features through information compression in the Z-axis direction, i.e., voxel-compressed bird's eye view representation. Voxel pooling reduces the computational complexity while retaining key spatial information.

[0059] In the embodiment, the relevant feature extraction module is configured to calculate the correlation between the compressed bird's eye view representations of the adjacent two frames of monocular images according to the compressed bird's eye view representation and generate a relevant volume, which is used to estimate the pose prediction of the target object.

[0060] Specifically, the relevant feature extraction module includes a feature translation sub-module, a correlation calculation sub-module, and a relevant volume generation sub-module. In the feature translation sub-module, in order to capture the relative motion information between the two frames, first, the compressed bird's eye view representation of the latter frame of monocular images is subjected to a translation operation with different step lengths. Specifically, the feature map is moved pixel by pixel in the x-axis and y-axis directions with a fixed step length, and an offset matrix of size D x x D y is recorded to obtain a series of translated bird's eye view representations. These translation operations include positive and negative movements, so that the feature map can cover the possible relative movement range.

[0061] The correlation calculation sub-module is configured to calculate a correlation matrix of size H BEV x W BEVThe correlation score matrix between the current frame bird's eye view feature and the next frame shifted bird's eye view feature is calculated for each shifted bird's eye view representation, and the correlation score matrix measures the matching degree of the features at different spatial positions. The correlation between the features of adjacent frames is calculated by the cosine similarity, which captures the relative motion information between the adjacent two monocular images. It should be noted that H BEV ×W BEV Actually, it is the resolution of the BEV feature map (excluding the channel dimension). Wherein, the cosine similarity is defined as:

[0062]

[0063] Wherein, A and B represent the compressed bird's eye view representation feature vectors of the adjacent two monocular images, the symbol · represents the vector dot product, and ∥A∥ and ∥B∥ represent the L2 norm of the feature vector. This calculation captures the relative motion information between the adjacent two monocular images, and the input is the feature map under the bird's eye view of the previous frame and the shifted bird's eye view feature map of the next frame, and the output is the correlation score under different shifts of the two frames.

[0064] The correlation volume generation submodule is used to generate a four-dimensional correlation volume with a size of D x ×D y ×H BEV ×W BEV According to the correlation score matrix and the shift matrix. Specifically, the input is the bird's eye view feature correlation score under different shifts of the previous frame and the next frame, and the output is the correlation volume matrix arranged as a four-dimensional matrix, that is, after completing all shift traversals, a four-dimensional correlation volume is generated, which contains the correlation information between the x-axis, y-axis shift and the feature map. This volume effectively describes the feature space of the inter-frame relative motion and provides basic data for subsequent pose prediction.

[0065] Through the above process, the four-dimensional correlation volume generated by the correlation feature extraction module accurately captures the relative motion between adjacent frames, especially the subtle displacement in the BEV plane. This process ensures the accuracy and stability of subsequent pose estimation.

[0066] In the embodiment, the pose prediction decoder is used to perform pose prediction of the target object according to the correlation volume, wherein the pose includes a translation vector and a rotation matrix.

[0067] Specifically, the pose prediction decoder includes a feature fusion and dimension reduction submodule, a full connection submodule, and a post-processing submodule. The feature fusion and dimension reduction submodule is configured to combine the offset dimension in the four-dimensional correlation volume and reduce the dimension through a convolution layer to output a three-dimensional correlation feature map. The dimension reduction operation not only reduces the complexity of the data, but also extracts the most critical motion information, ensuring that the relative motion between adjacent frames is efficiently described.

[0068] The full connection submodule is configured to expand the three-dimensional correlation feature map into a one-dimensional vector and reduce the dimension through a series of full connection layers, and then predict two outputs based on the one-dimensional vector through two branches, for a total of four outputs. Specifically, based on the reduced features, the one-dimensional vector is first expanded, and then a set of full connection layers and a nonlinear activation function are used, followed by two branches: one branch is used to predict two outputs in the x and y directions of the target object for subsequent prediction of the translation vector, and the other branch also predicts two outputs for subsequent rotation matrix. The two branches are also implemented through full connection layers. This method avoids the discontinuity problem caused by directly predicting the rotation matrix, thereby improving the stability and accuracy of pose estimation.

[0069] The post-processing submodule is configured to predict the translation vector and the rotation matrix based on the two branches. Specifically, the two outputs of one branch are activated using a tanh activation function to obtain the translation vector in the x and y directions, and the two outputs of the other branch are also activated using a tanh activation function, and then the L2 regularization is performed to obtain the cos and sin values of the yaw angle, and further transformation is performed to form the rotation matrix. The tanh activation function scales the output of the branch to a reasonable range, thereby reducing the influence of outliers and ensuring that the predicted pose parameters (including translation and rotation) are reasonable in the actual scenario. In actual operation, the cosine and sine values of the angle are scaled to 0-1, and L2 regularization is performed to ensure that their squares sum to 1, which can correspond to a unique yaw angle, and the translation vector is scaled to 0-4 (unit: meters).

[0070] Through the processing of the pose prediction decoder, the pose information can be effectively extracted from the correlation volume, and the translation vector and the rotation matrix of the target object can be predicted with high accuracy. The combination of the pose prediction decoding and the correlation feature extraction module provides an efficient and accurate pose estimation method, which is helpful to realize a robust monocular visual odometry system.

[0071] The visual BEV encoder, the related feature extraction module, and the pose prediction decoder described above need to be optimized in parameters before being applied. Specifically, pose supervision is adopted to supervise the training by calculating the L1 loss of the rotation matrix and the displacement vector. Specifically, the prediction of the rotation matrix adopts cosine and sine functions to avoid discontinuity in angle prediction, thereby improving the stability of pose estimation. The final loss function is the weighted sum of the translation error and the rotation error constructed based on the predicted translation vector, the rotation matrix, and the corresponding label, which is used to update the network parameters. This supervision method effectively reduces the scale drift problem in traditional methods and improves the accuracy and robustness of overall motion estimation.

[0072] The monocular visual odometry system based on the aerial perspective representation described above includes a plurality of carefully designed neural network modules to effectively reduce scale drift and improve motion estimation accuracy in monocular visual odometry. The entire system includes image feature extraction, feature mapping, correlation calculation, pose prediction, and supervised learning, which work together to optimize network parameters using only pose supervision, without introducing complex auxiliary tasks such as depth estimation or optical flow estimation, thereby simplifying the implementation process of the system and maintaining high-precision motion estimation while reducing computational complexity, making it particularly suitable for applications such as robot autonomous navigation and autonomous driving that require high real-time performance and accuracy.

[0073] As shown in Figure 2 The embodiment also provides a monocular visual odometry method based on an aerial perspective representation, which adopts the system described above and includes the following steps:

[0074] S210, image features are extracted from monocular images using a visual BEV encoder and the depth distribution probability of the features is predicted, then the image features and their depth distribution probability are fused and mapped to BEV to obtain an aerial perspective representation, and then the camera coordinate system is converted to the body coordinate system and compressed in the height direction to obtain a compressed aerial perspective representation;

[0075] S220, a related feature extraction module is used to calculate the correlation between the compressed aerial perspective representations of adjacent two key frame monocular images according to the compressed aerial perspective representation and generate a correlation volume;

[0076] S230, a pose prediction decoder is used to perform pose prediction of the target object according to the correlation volume, wherein the pose includes a translation vector and a rotation matrix.

[0077] The monocular visual odometry system and method are mainly used in the fields of automatic driving and planar motion robot navigation, and are especially suitable for long-time sequence scenarios that require high-precision motion estimation, which helps to improve the navigation ability and reliability of autonomous systems in complex environments. The core advantage of the present application is that it greatly reduces the scale drift problem in long-time sequences by optimizing the bird's eye view representation under the configuration of a monocular camera, thereby improving the stability and precision of motion estimation. By simplifying the pipeline, improving computational efficiency and robustness, it shows great potential in practical applications. The effects of the system of the present application will be described in detail below according to the experimental results:

[0078] Significant reduction of scale drift: In the monocular visual odometry system, scale drift is one of the main problems affecting accuracy. The present application effectively reduces the scale drift in long-time sequences by utilizing the inherent scale consistency of the bird's eye view (BEV) representation. As shown in Table 1, the experimental results show that the method of the present application exhibits superior scale consistency on multiple datasets (such as NCLT, Oxford and KITTI).

[0079] Table 1

[0080]

[0081] * Scale adjustment according to the true trajectory of the first 10 meters, and SE(3) alignment

[0082] ** SE(3) alignment

[0083] Scale drift table explanation: First, the trajectories of all methods are scaled according to the true trajectory (GT) of the first 10 meters, except for the method of the present application and DF-VO (tested on the KITTI dataset). Then, the scale drift is calculated using the following formula:

[0084]

[0085] where Dscal e represents the average scale drift of all segments, N is the total number of segments, d i is the estimated displacement distance of the i-th segment, is the true displacement distance of the i-th segment. The reason for using the logarithmic function and absolute value to measure the scale drift is that the scale drift usually manifests as a proportional difference from the true value rather than a linear deviation. The logarithmic function can normalize these proportional differences, ensuring that overestimation and underestimation of scale are equally penalized.

[0086] Improving motion estimation accuracy: Through testing on different complexity datasets, the test results are shown in Table 2. The method of the present application shows higher accuracy in terms of absolute trajectory error (ATE), relative translation error (RTE) and relative rotation error (RRE) relative to other existing methods (such as ORB-SLAM3, DF-VO, DROID-SLAM). Especially on challenging datasets such as NCLT and Oxford, the performance of the method of the present application is significantly better than other methods.

[0087] Table 2

[0088]

[0089] In Table 2, (w / LC) uses closed-loop optimization; (w / o LC) does not use closed-loop optimization; (w / 0 GBA) does not use final global bundle adjustment optimization; (F. Mode I) uses the base model for bidirectional optical flow prediction and monocular depth estimation; (Stereo Trained) uses a model trained using stereo camera data; (w / o DS) does not use depth supervision during training; * scale adjustment according to the true trajectory of the first 10 meters, and SE(3) alignment is used; ** SE(3) alignment is used; ↑ Sim(3) alignment is used.

[0090] Excellent performance without depth supervision: As verified by the ablation experiment shown in Table 3, the method of the present application can still achieve performance comparable to traditional methods using depth supervision without introducing depth supervision. This is due to the use of pose supervision to optimize the feature depth distribution estimation of the visual BEV encoder in the present application, without being limited to capturing more effective key points in the perspective view. In addition, the use of pose supervision also avoids the inaccuracy caused by the problems such as synchronization error, extrinsic calibration error and sparse point cloud introduced by LiDAR data.

[0091] Table 3

[0092]

[0093] Simplify MVO pipeline and improve computing efficiency: By eliminating the complex steps (such as bundle adjustment, pose graph optimization and depth supervision) in traditional MVO methods, the present application greatly simplifies the entire pipeline, improves the computing efficiency and robustness of the method. Compared with methods such as DROID-SLAM and DF-VO, the present application achieves a running speed of more than 60 frames per second (fps) on an RTX4090 graphics card, and has low memory consumption. This high efficiency and resource saving makes the method of the present application more suitable for practical application scenarios.

[0094] Strong robustness and adaptability: Through the analysis of complete trajectories and selected trajectory segments, the invented method exhibits robustness over long time and complex environments, as well as excellent generalization ability in new environments. Even in severely jittered NCLT datasets, the invented method can maintain scale accuracy throughout, demonstrating its stability in handling complex scenarios.

[0095] As Figure 3 shown, the embodiments also provide intermediate variables and final output results of the invented system and method during operation, including the following key parts:

[0096] 1. Figure 3 The trajectory comparison chart shown in the upper left part shows the comparison between the trajectory predicted by the invented system and the actual reference trajectory (GT). Figure 3 The A, B, C, and D positions marked in the middle correspond to straight road segments, left turn scenarios, left turn scenarios, and right turn scenarios, respectively. These positions are used to further demonstrate the system's running results in different situations.

[0097] 2. Figure 3 The camera perspective examples shown in the upper right part show the camera images at times t and t+1 at positions A, B, C, and D, which are used to compare the visual input of the system under different turning scenarios.

[0098] 3. Figure 3 The BEV feature maps and optical flow information shown in the bottom part of the figure show the BEV spatial feature maps encoded by the visual BEV encoder and the BEV optical flow information extracted from the relevant feature extraction module. During straight, left, and right turns, specific flow direction patterns can be observed in these feature maps, which are more pronounced in different turning scenarios, reflecting the depth distribution prediction ability of the visual BEV encoder in converting perspective view features to BEV representations.

[0099] Figure 3 The figure illustrates the robustness and accuracy of the invented system in handling complex turning scenarios, demonstrating how the system effectively reduces scale drift and improves the accuracy of pose estimation using BEV representation.

[0100] As Figure 4 shown, the embodiments also provide trajectory comparison results under different datasets. Figure 4 The figure shows the trajectory prediction results of the invented system on NCLT, Oxford, and KITTI datasets, and compares them with the results of other comparative methods (such as ORB-SLAM3, DF-VO, and DROID-SLAM). Figure 4 The contents of the figure include:

[0101] 1.Figure 4 The upper part of FIG. 1 shows the trajectory results of the NCLT dataset: the left image shows the test results of the complete path, and the right part shows the results of the partial path. It can be seen that the trajectory prediction of the system of the present application is closer to the reference trajectory (GT), showing strong scale consistency and trajectory accuracy.

[0102] 2. Figure 4 The middle part of FIG. 1 shows the trajectory results of the Oxford dataset: the left image also shows the test results of the complete path, and the right image shows the results of the partial path. The system of the present application performs better than other methods in complex paths, especially in terms of scale consistency.

[0103] 3. Figure 4 The lower part of FIG. 1 shows the trajectory results of the KITTI dataset: the left image is the test result of seq9, and the right image is the test result of seq10. Compared with other methods, the system of the present application shows higher accuracy in predicting the overall trajectory, especially when dealing with long-distance paths, the trajectory of the system of the present application is more stable and close to the true trajectory.

[0104] Figure 4 Further verification of the robustness and adaptability of the system of the present application in different complex scenarios, showing its advantages in maintaining scale consistency and improving motion estimation accuracy.

[0105] The following describes the application examples of the system and method of the present application:

[0106] Application Example 1: Application in autonomous driving

[0107] In the application of autonomous driving, vehicles need to continuously perform high-precision motion estimation in complex road environments to ensure safe driving and accurate path planning. Traditional monocular vision systems are prone to error accumulation over long periods of driving, leading to scale drift and affecting the accuracy of driving trajectories. The system of the present application plays a role in such scenarios in the following ways:

[0108] 1. Stable motion estimation in long-distance driving: When autonomous vehicles drive for a long time on highways or urban roads, they need to maintain accurate pose estimation. The system of the present application uses the scale consistency of the bird's eye view to provide high-precision pose data even after long-distance driving, avoiding the error accumulation problem in traditional monocular vision systems.

[0109] 2. Real-time path adjustment and dynamic response: In the face of dynamic changes in traffic environment (such as sudden obstacles, complex urban road conditions, etc.), the system of the present invention can adjust the driving path of the vehicle in real time through accurate pose estimation. Its advantage is that it can make quick and accurate decisions in dynamic environments without relying on additional depth information, thereby ensuring the safety and efficiency of the vehicle.

[0110] Application Example 2: Application in planar motion robots

[0111] In the field of industrial automation or service robots, robots usually perform various tasks in a planar environment, such as cargo handling in warehouses or automatic inspection in factories. These robots need to maintain stable path and motion estimation over a long period of time to ensure accurate task completion. The application of the system of the present invention in such scenarios is as follows:

[0112] 1. Continuous accurate positioning in complex environments: In environments such as large warehouses or factories, robots need to move for a long time and maintain the accuracy of the path. Traditional monocular vision systems may cause the robot to deviate from the predetermined trajectory due to the easy occurrence of scale drift, while the system of the present invention can provide stable pose estimation in long-term task execution by optimizing the scale consistency of image features, ensuring that the robot accurately completes the predetermined task.

[0113] 2. Efficient use of monocular cameras: Due to cost and equipment limitations, many robots can only be equipped with monocular cameras. The system of the present invention can achieve efficient motion estimation using only monocular cameras, meeting the accuracy requirements of robots in planar motion. Especially in dynamic environments, the system can adapt to environmental changes to ensure the continuous accuracy of the robot's path.

[0114] Application Example 3: Application in planar navigation unmanned vehicles

[0115] In the application of unmanned vehicles, such as indoor delivery vehicles or automatic guided vehicles in airports, they usually run for a long time in a planar environment and need to maintain high-precision motion estimation. The advantage of the system of the present invention is its low scale drift, making it particularly suitable for such application scenarios:

[0116] 1. Efficient path planning and execution: Unmanned vehicles need to run on predetermined paths and navigate in complex planar environments. With the system of the present invention, unmanned vehicles can maintain accurate pose estimation over a long period of time, even if they encounter environmental changes during path planning and execution, ensuring the accuracy of navigation.

[0117] 2. Response and adjustment in dynamic environment: In busy scenes such as airports or shopping malls, unmanned vehicles need to deal with a large number of pedestrians or the emergence of temporary obstacles. The system of the present application provides real-time high-precision pose data, enabling the unmanned vehicle to quickly adjust the driving path, avoid obstacles and continue to perform tasks.

[0118] In summary, the system of the present application is particularly suitable for applications such as autonomous driving and planar motion robots that require accurate pose estimation, through long-time stable low-scale drift motion estimation under monocular camera configuration. Its advantage is that it can maintain high-precision, low-drift motion estimation in long-time sequences, and adapt to high-precision motion requirements in various complex dynamic environments

[0119] The specific embodiments described above have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A monocular visual odometry system based on aerial view representation, characterized in that, The visual BEV encoder, the related feature extraction module, and the pose prediction decoder; The visual BEV encoder is used for extracting image features from monocular images and predicting the depth distribution probability of the features, then fusing the image features with the depth distribution probability thereof and mapping the image features to BEV to obtain a bird's eye view representation, and then performing conversion from a camera coordinate system to a body coordinate system and height direction compression to obtain a compressed bird's eye view representation. The related feature extraction module is used for calculating the correlation between compressed bird's eye view representations of adjacent two key frame monocular images according to the compressed bird's eye view representation and generating a correlation volume. The pose prediction decoder is used for performing pose prediction of a target object according to the correlation volume, wherein the pose includes a translation vector and a rotation matrix.

2. The bird's eye view perspective based monocular visual odometry system of claim 1, wherein, The visual BEV encoder includes a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule, and a feature mapping submodule. The feature extraction submodule is used for extracting multi-scale perspective view feature maps from monocular images. The camera parameter auxiliary encoding submodule is used for encoding camera parameters and obtaining a weighted feature map after weighting the camera parameter encoding and the perspective view feature map through channel attention. The depth distribution prediction submodule is used for predicting the depth distribution probability of the features based on the weighted feature map through a deep network. The feature mapping submodule is used for fusing the image features after convolution of the weighted feature map with the depth distribution probability of the features, mapping to BEV to obtain a bird's eye view representation, performing conversion from a camera coordinate system to a body coordinate system on the bird's eye view representation, and performing height direction compression to obtain a voxel compressed bird's eye view representation.

3. The bird's eye view perspective based monocular visual odometry system of claim 2, wherein, The feature extraction submodule extracts multi-scale perspective view feature maps using the structure of ResNet-50+FPN. The camera parameter auxiliary encoding submodule encodes the camera parameters using MLP.

4. The bird's eye view perspective based monocular visual odometry system of claim 2, wherein, The deep network used by the depth distribution prediction submodule includes sequentially connected Res Blocks, ASPP, and DCN, wherein the Res Blocks perform dimension reduction on the weighted features, the ASPP performs receptive field expansion on the reduced features and captures depth features of different scales, and the DCN further extracts depth features to obtain depth distribution features.

5. The monocular visual odometry system based on bird's-eye view representation according to claim 2, characterized in that: In the feature mapping submodule, the process of fusing the image features after convolution of the weighted feature map with the depth distribution probability of the features is as follows: The size of the image feature is changed from CxH feature xW feature to Cx1xH feature xW feature , wherein C represents a channel number, H feature represents a feature map height, W feature represents a feature map width, and 1 is an introduced dimension; The size of the depth distribution probability is changed from D x H feature x W feature by inserting a dimension to 1 x D x H feature x W feature where D represents the dimension number of the depth of the depth distribution probability; The image features with a size of Cx1xH feature xW feature are multiplied by the depth distribution probability with a size of 1xDxH feature xW feature to obtain the depth distribution probability of each feature, forming a matrix with a size of CxDxH feature xW feature .

6. The bird's eye view perspective based characterization monocular visual odometry system of claim 1, wherein, The related feature extraction module includes a feature translation submodule, a correlation calculation submodule, and a correlation volume generation submodule. The feature translation sub-module is configured to perform translation operations with different step lengths on the compressed bird's eye view angle representations of the next frame of monocular images and record an offset matrix with a size of D x ×D y to obtain a series of translated bird's eye view angle representations; The correlation calculation sub-module is configured to calculate a correlation score matrix with a size of H BEV ×W BEV from a series of translated bird's eye view features, wherein the correlation score captures the relative motion information between two adjacent monocular images. The correlation volume generating submodule is configured to generate a four-dimensional correlation volume with a size of D x x D y x H BEV x W BEV according to the correlation score matrix and the offset matrix.

7. The bird's eye view perspective based monocular visual odometry system of claim 6, wherein, The correlation between adjacent two frames of monocular images is calculated using cosine similarity to obtain a correlation score.

8. The bird's eye view perspective based characterization monocular visual odometry system of claim 1, wherein, The pose prediction decoder includes a feature fusion and dimension reduction submodule, a fully connected submodule, and a post-processing submodule. The feature fusion and dimension reduction submodule is used for merging the offset dimension in the four-dimensional correlation volume and performing dimension reduction through a convolution layer to output a three-dimensional correlation feature map. The fully connected submodule is used for expanding the three-dimensional correlation feature map into a one-dimensional vector and performing dimension reduction through a series of fully connected layers, and then predicting two outputs based on the one-dimensional vector through two branches, for a total of four outputs. The post-processing submodule is configured to scale the tanh activation of the two outputs of one branch to obtain the translation vectors in the x and y directions, and to sequentially perform tanh activation and L2 regularization on the two outputs of another branch to obtain the cos and sin values of the yaw angle, and further transform to form a rotation matrix.

9. The bird's eye view perspective based characterization monocular visual odometry system of claim 1, wherein, The visual BEV encoder, the correlation feature extraction module, and the pose prediction decoder need to be optimized in parameters before being applied, and the loss function used in the parameter optimization is the translation error and rotation error constructed based on the predicted translation vector, rotation matrix, and corresponding label.

10. A monocular visual odometry method based on bird's eye view representation, characterized in that, The method employs the system of any one of claims 1-9, and comprises the following steps: An image feature is extracted from a monocular image by using the visual BEV encoder, and a depth distribution probability of the feature is predicted, then the image feature and the depth distribution probability thereof are fused and mapped to a bird's eye view to obtain a bird's eye view representation, and a conversion from a camera coordinate system to a body coordinate system and a compression in a height direction are performed to obtain a compressed bird's eye view representation; The correlation feature extraction module is used to calculate the correlation between the compressed bird's eye view representations of adjacent two key frame monocular images according to the compressed bird's eye view representation, and a correlation volume is generated; The pose prediction decoder is used to perform pose prediction of a target object according to the correlation volume, wherein the pose includes a translation vector and a rotation matrix.

Images