Monocular vision odometer system based on aerial view representation and method thereof

Abstract

The invention discloses a monocular visual odometer system and method based on aerial view representation, and the system comprises a visual BEV encoder which is used for extracting image features from a monocular image and predicting the depth distribution probability of the features, fusing the image features and the depth distribution probability of the image features, mapping the image features and the depth distribution probability of the image features to BEV to obtain bird's-eye view representation, and performing conversion from a camera coordinate system to a body coordinate system and compression in the height direction to obtain compressed bird's-eye view representation; the correlation feature extraction module is used for calculating the correlation between the compressed bird's-eye view representation of the two adjacent key frame monocular images according to the compressed bird's-eye view representation and generating a correlation volume; and the attitude prediction decoder is used for performing attitude prediction on the target object according to the related volume. The system and the method are excellent in the aspects of reducing the scale drift and improving the motion estimation precision, can be widely applied to the fields of automatic driving, robot navigation and the like, and are beneficial to improving the navigation capability and the reliability of an autonomous system in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of the combination of computer vision, robot navigation and automatic driving, and specifically relates to a monocular visual odometer system and method based on bird's-eye view representation. Background Art

[0002] Monocular Visual Odometry (MVO) has been widely used in autonomous navigation and robotics due to its cost-effectiveness and ease of deployment. MVO can provide real-time motion tracking capabilities as an economical and flexible supplement to global positioning systems (GPS) and inertial navigation systems. However, despite its many advantages, its widespread application is still limited by a key problem, namely scale ambiguity. Due to the lack of depth information, monocular vision systems can usually only estimate motion at relative scales, resulting in errors that accumulate over time, a phenomenon known as scale drift.

[0003] Traditional MVO methods, such as feature-based methods, semi-direct methods, and direct methods, usually determine the scale at the initialization stage and use this scale as a global reference. This approach closely links the scale estimation with the initial motion, making the motion tracking performance highly dependent on the motion speed at startup. In addition, these methods heavily rely on the initial scale setting, resulting in the scale drift problem becoming more serious over time.

[0004] In recent years, learning-based MVO methods have attempted to address this problem by modeling prior distributions from training data using the powerful fitting capabilities of deep learning models. These methods use Convolutional Neural Networks (CNN) to automatically extract features from images and perform pose regression based on temporal modeling methods. For example, some studies have combined the interpretability of traditional methods with the powerful data fitting capabilities of deep learning, attempting to improve the accuracy of scale estimation by introducing auxiliary tasks such as depth supervision or optical flow supervision. However, these methods usually require the introduction of additional supervised data, such as depth estimation or optical flow estimation, which not only increases computational complexity but also brings higher data acquisition costs.

[0005] With the advancement of Bird's Eye View (BEV) conversion technology and the excellent performance of BEV representation in 3D detection and scene segmentation, some studies have begun to use BEV representation to implement visual odometry. However, these methods often still rely on scene segmentation results to estimate pose, which raises the question of whether the intrinsic scale consistency of BEV representation originates from the BEV representation itself or relies on the supervision of scene segmentation. In addition, these methods have a high cost in acquiring labeled data and do not fully utilize the information directly provided by BEV representation.

[0006] Although the traditional MVO method shows high robustness in a variety of environments, it is difficult to avoid the scale drift problem due to the inherent limitations of monocular vision. Although deep learning methods can improve the accuracy of motion estimation through supervised learning, they also face challenges such as difficulty in obtaining supervised data and high computational complexity. In addition, although the method based on BEV representation can simplify the odometry problem in some cases, the accuracy and robustness of the scene segmentation supervision information it relies on is still an issue to be solved. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a monocular visual odometer system and method based on bird's-eye view representation, which reduces or eliminates the scale drift phenomenon in the MVO system by utilizing the inherent scale consistency of BEV representation, thereby achieving more accurate and stable motion estimation, and avoids dependence on explicit depth estimation, optical flow estimation and other tasks that require additional supervision by optimizing the image feature and depth distribution feature matching process, thereby simplifying the structure of the MVO system and reducing the cost of calculation and data acquisition.

[0008] To achieve the above-mentioned purpose of the invention, the embodiment provides a monocular visual odometer system based on bird's-eye view representation, including a visual BEV encoder, a relevant feature extraction module, and a posture prediction decoder;

[0009] The visual BEV encoder is used to extract image features from a monocular image and predict the depth distribution probability of the features, then fuse the image features with their depth distribution probability and map them to BEV to obtain a bird's-eye view representation, then perform a conversion from the camera coordinate system to the body coordinate system and perform height compression to obtain a compressed bird's-eye view representation;

[0010] The relevant feature extraction module is used to calculate the correlation between the compressed bird's-eye view representations of two adjacent key frame monocular images according to the compressed bird's-eye view representation and generate a relevant volume;

[0011] The posture prediction decoder is used to predict the posture of the target object based on the relevant volume, wherein the posture includes a translation vector and a rotation matrix.

[0012] Optionally, the visual BEV encoder includes a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule, and a feature mapping submodule;

[0013] The feature extraction submodule is used to extract a multi-scale perspective feature map from a monocular image;

[0014] The camera parameter auxiliary encoding submodule is used to encode the camera parameters and obtain a weighted feature map by weighting the camera parameter encoding and the perspective view feature map through channel attention;

[0015] The depth distribution prediction submodule is used to predict the depth distribution probability of features through a deep network based on the weighted feature map;

[0016] The feature mapping submodule is used to fuse the image features after convolution of the weighted feature map with the depth distribution probability of the features and map them to BEV to obtain a bird's-eye view representation, convert the bird's-eye view representation from the camera coordinate system to the body coordinate system and compress it in the height direction to obtain a compressed bird's-eye view representation.

[0017] Optionally, the feature extraction submodule uses a ResNet-50+FPN structure to extract a multi-scale perspective feature map.

[0018] Optionally, the camera parameter auxiliary encoding submodule adopts MLP encoding camera parameters.

[0019] Optionally, the deep network adopted by the depth distribution prediction submodule includes Res Block, ASPP, and DCN connected in sequence, wherein Res Block performs dimensionality reduction of weighted features, ASPP expands the receptive field of features after dimensionality reduction and captures deep features of different scales, and DCN further extracts deep features to obtain deep distribution features.

[0020] Optionally, in the feature mapping submodule, the process of fusing the image features after convolution of the weighted feature map with the depth distribution probability of the features is:

[0021] The size of the image feature is changed from the original C×H feature ×W feature Insert a dimension to become C×1×H feature ×W feature , where C represents the number of channels, H feature Represents the feature map height, W feature Represents the width of the feature map, 1 is an introduced dimension;

[0022] The size of the depth distribution probability is changed from the original D×H feature ×W featureInsert a dimension to become 1×D×H feature ×W feature , where D represents the depth dimension of the depth distribution probability;

[0023] The size is C×1×H feature ×W feature The image features are of size 1×D×H feature ×W feature The depth distribution probability of each feature is multiplied by the depth distribution probability of each feature to form C×D×H feature ×W feature The matrix of .

[0024] Optionally, the relevant feature extraction module includes a feature translation submodule, a correlation calculation submodule, and a relevant volume generation submodule;

[0025] The feature translation submodule is used to perform translation operations of different step lengths on the compressed bird's-eye view representation of the next frame of monocular image and record the size D x ×D y The offset matrix is ​​used to obtain a series of bird's-eye view representations after translation;

[0026] The correlation calculation submodule is used to calculate the H with the same size as the bird's-eye view feature map according to a series of translated bird's-eye view representations. BEV ×W BEV The correlation score matrix captures the relative motion information between two adjacent monocular images.

[0027] The correlation volume generation submodule is used to generate a volume with a size of D according to the correlation score matrix and the offset matrix. x ×D y ×H BEV ×W BEV The four-dimensional correlation volume.

[0028] Optionally, cosine similarity is used to calculate the correlation scores of two adjacent monocular image frames.

[0029] Optionally, the posture prediction decoder includes a feature fusion and dimension reduction submodule, a fully connected submodule, and a post-processing submodule;

[0030] The feature fusion and dimension reduction submodule is used to merge the offset dimensions in the four-dimensional correlation volume and then reduce the dimension through the convolution layer to output a three-dimensional correlation feature map;

[0031] The fully connected submodule is used to expand the three-dimensional correlation feature map into a one-dimensional vector and reduce the dimension through a series of fully connected layers, and then predict two outputs based on the one-dimensional vector through two branches, for a total of four outputs;

[0032] The post-processing submodule is used to perform tanh activation and scaling on the two outputs of one branch to obtain the translation vectors in the x and y directions, and to perform tanh activation and L2 regularization on the two outputs of the other branch in turn to obtain the cos and sin values ​​of the yaw angle, which are further transformed to form a rotation matrix.

[0033] Optionally, the visual BEV encoder, related feature extraction module, and posture prediction decoder need to undergo parameter optimization before being applied. The loss function used in parameter optimization is to update the parameters based on the translation error and rotation error constructed based on the predicted translation vector, rotation matrix and corresponding label.

[0034] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a monocular visual odometer method based on bird's-eye view representation, wherein the method adopts the above-mentioned system and comprises the following steps:

[0035] The visual BEV encoder is used to extract image features from the monocular image and predict the depth distribution probability of the features. The image features and their depth distribution probability are then fused and mapped to the BEV to obtain a bird's-eye view representation. The camera coordinate system is then converted to the body coordinate system and compressed in the height direction to obtain a compressed bird's-eye view representation.

[0036] The correlation feature extraction module is used to calculate the correlation between the compressed bird's-eye view representations of two adjacent key-frame monocular images according to the compressed bird's-eye view representation and generate a correlation volume;

[0037] The pose prediction decoder is used to predict the pose of the target object based on the relevant volume, where the pose includes the translation vector and the rotation matrix.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] In the method of the present invention, the bird's-eye view table is used to reduce the scale drift of the monocular visual odometer in long sequence applications, thereby improving the accuracy and robustness of the monocular visual odometer. Its advantage is that the common plane assumption in the plane autonomous driving task is used to simplify the 6-degree-of-freedom mileage estimation task to 3 degrees of freedom; although it is also necessary to predict information related to depth when projecting the BEV feature map, this patent uses the method of predicting the depth distribution probability of high-dimensional feature points to replace the original monocular visual odometer The method of predicting absolute depth pixel by pixel makes the network have better generalization and is easier to train.

[0040] In the method of the present invention, by eliminating the complicated bundling adjustment and loop detection steps in the traditional method, MVO can be realized only through the visual BEV encoder, the relevant feature extraction module, and the posture prediction decoder, which simplifies the traditional MVO process and improves the real-time performance of the system.

[0041] In the method of the present invention, dependence on additional supervisory data (such as depth information and optical flow information) is avoided, and training is performed only by using a loss function obtained by the error between the predicted relative pose and the true relative pose, thereby reducing the cost of data collection and processing and improving applicability in various application scenarios.

[0042] In summary, the method of the present invention provides a technical solution that can significantly improve the accuracy and robustness of a monocular visual odometer system without adding additional hardware and computing costs. It is particularly suitable for application scenarios such as robot navigation and autonomous driving that require long-term reliable motion estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a structural schematic diagram of a monocular visual odometer system based on bird's-eye view representation provided in an embodiment;

[0045] Figure 2 is a flow chart of a monocular visual odometer method based on bird's-eye view representation provided by an embodiment;

[0046] Figure 3 is an intermediate result visualization diagram and a final trajectory diagram provided by the embodiment;

[0047] Figure 4 It is the trajectory comparison result under different data sets provided by the embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0049] The technical concept of the present invention is: in the existing MVO technology, due to the lack of absolute depth information of the monocular vision system, the motion can usually only be estimated on a relative scale, which leads to accumulated errors in long-term operation, seriously affecting the accuracy of motion estimation and the robustness of the system. The existing traditional MVO technology based on feature matching and geometric methods, as well as the end-to-end MVO method relying on deep learning, are difficult to reliably solve the scale drift problem without the need for additional supervised data. In order to solve the scale drift problem that is prevalent in monocular visual odometer (MVO) systems, an embodiment of the present invention provides a monocular visual odometer system and method based on bird's-eye view representation, which utilizes the inherent scale consistency of BEV representation to significantly reduce the scale drift problem in long-term operation, and does not need to rely on deep supervision or complex optimization techniques. The experimental results of this method on multiple data sets show that compared with the existing MVO method, the scheme of the present invention has significant advantages in improving the accuracy of motion estimation and reducing scale drift.

[0050] like Figure 1 As shown, an embodiment provides a monocular visual odometer system based on bird's-eye view representation, including a visual BEV encoder, a relevant feature extraction module, and a posture prediction decoder.

[0051] In an embodiment, the visual BEV encoder is used to extract image features from a monocular image and predict the depth distribution probability of the features, and then fuse the image features with their depth distribution probability and map them to BEV to obtain a bird's-eye view representation, and then perform a conversion from the camera coordinate system to the body coordinate system and compress it in the height direction to obtain a compressed bird's-eye view representation.

[0052] Specifically, the visual BEV encoder includes a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule, and a feature mapping submodule. Among them, the feature extraction submodule is used to extract multi-scale perspective view feature maps from monocular images. This submodule adopts the ResNet-50+FPN structure, specifically using ResNet-50 as the feature extraction backbone network to extract multi-scale features from the monocular image (input image). To enhance the expressiveness of the features, these features are fused through the Feature Pyramid Network (FPN) to generate a multi-scale perspective view (PV) feature map Fpv.

[0053] The camera parameter auxiliary encoding submodule is used to encode the camera parameters (camera intrinsic parameters and camera and body extrinsic parameters), and implement channel attention weighting through the SE layer, fuse the camera parameter encoding with the perspective view feature map Fpv, and obtain the weighted feature map. In this submodule, the camera parameters are encoded through MLP, and when the channel attention is weighted, the camera parameter encoding is processed through the convolution layer and activation function of the SE layer to generate a weighted map, and then the perspective view feature map Fpv is multiplied at the element level to obtain the final weighted feature map.

[0054] The depth distribution prediction submodule is used to predict the depth distribution probability of the feature through a deep network based on the weighted feature map. It should be emphasized that the generation of the depth distribution probability does not rely on depth supervision, but is optimized through posture supervision, so that the depth distribution prediction process is closely integrated with the posture estimation. Specifically, the deep network used by the depth distribution prediction submodule includes Res Block, ASPP, and DCN connected in sequence, among which Res Block performs dimensionality reduction of weighted features to reduce computational complexity, ASPP (Atrous Spatial Pyramid Pooling) expands the feature receptive field after dimensionality reduction and captures deep features of different scales, and DCN further extracts the deep features to obtain the depth distribution probability of each pixel point, which provides key depth information Fdepth for subsequent feature mapping to BEV space.

[0055] The feature mapping submodule is used to map the weighted feature map to the BEV space based on the depth distribution feature through frustum projection to generate a BEV representation, and then compress the information in the Z-axis direction through voxel pooling technology to finally form a BEV representation with multi-dimensional features. Specifically, the image feature Fpv1 after convolution of the weighted feature map is fused with the depth distribution probability of the feature, and the fusion method is:

[0056] The size of the image feature is changed from the original C×H feature ×W feature Insert a dimension to become C×1×H feature ×W feature , where C represents the number of channels, H feature Represents the feature map height, W feature Represents the width of the feature map, 1 is an introduced dimension; the size of the depth distribution probability is changed from the original D×H feature ×W feature Insert a dimension to become 1×D×H feature ×W feature , where D represents the depth dimension of the depth distribution probability; the size is C×1×H feature×W feature The image features are of size 1×D×H feature ×W feature The depth distribution probability of each feature is multiplied by the depth distribution probability of each feature to form C×D×H feature ×W feature The matrix of .

[0057] Weighted fusion realizes the cone projection, which maps the image features into the bird's-eye view of the three-dimensional space. Specifically, each pixel in the two-dimensional image is projected into the three-dimensional space according to its depth information, thereby generating a BEV representation, which represents the feature in the BEV space of the camera.

[0058] Next, the bird's-eye view representation needs to be converted from the camera coordinate system to the body coordinate system and compressed in the height direction, where the height compression is completed through voxel pooling to obtain the compressed bird's-eye view representation. Specifically, the feature points in the three-dimensional space are converted from the camera coordinate system to the body coordinate system by applying the camera intrinsic parameter matrix, the transformation matrix from the sensor to the target object (such as a vehicle) coordinate system, etc. This process ensures that the image features of different cameras and different perspectives can be uniformly mapped to the BEV space of the body coordinate system. After conversion to the BEV space of the body coordinate system, these three-dimensional features are also voxel pooled. Voxel pooling converts the three-dimensional features into a two-dimensional BEV representation with multi-dimensional features by compressing the information in the Z-axis direction, that is, the bird's-eye view representation after voxel compression. Voxel pooling reduces computational complexity while retaining key spatial information.

[0059] In an embodiment, the correlation feature extraction module is used to calculate the correlation between the compressed bird's-eye view representations of two adjacent monocular image frames based on the compressed bird's-eye view representation and generate a correlation volume, which is used to estimate the posture prediction of the target object.

[0060] Specifically, the relevant feature extraction module includes a feature translation submodule, a correlation calculation submodule, and a relevant volume generation submodule. In the feature translation submodule, in order to capture the relative motion information between two frames, the compressed bird's-eye view representation of the next frame of monocular image is first translated with different step lengths. Specifically, the feature map is moved pixel by pixel in the x-axis and y-axis directions with a fixed step length, and the size is recorded as D x ×D y The offset matrix is ​​used to obtain a series of translated bird's-eye view representations. These translation operations include positive and negative movements, so that the feature map can cover the possible relative movement range.

[0061] The correlation calculation submodule is used to calculate the H of the same size as the bird's-eye view feature map based on a series of translated bird's-eye view representations. BEV ×W BEVSpecifically, for each translated bird's-eye view representation, the correlation score matrix between the current frame's bird's-eye view representation and the next frame's translated bird's-eye view representation is calculated. The correlation score matrix measures the matching degree of features at different spatial positions. The correlation of features between adjacent frames is calculated by cosine similarity, capturing the relative motion information between two adjacent monocular images. It should be noted that H BEV ×W BEV In fact, it is the resolution of the BEV feature map (excluding the channel dimension). Among them, the cosine similarity is defined as:

[0062]

[0063] Among them, A and B represent the compressed bird's-eye view feature vectors of two adjacent monocular images, the symbol · represents the vector dot product, and ∥A∥ and ∥B∥ represent the L2 norm of the feature vector. This calculation captures the relative motion information between two adjacent monocular images. The input is the feature map of the previous frame under the bird's-eye view and the feature map of the next frame after the offset. The output is the correlation score of the two frames under different offset conditions.

[0064] The correlation volume generation submodule is used to generate a volume with a size of D according to the correlation score matrix and the offset matrix. x ×D y ×H BEV ×W BEV Specifically, the input is the correlation score of the bird's-eye view feature map under different offset conditions of the previous frame and the next frame, and the output is the correlation volume matrix arranged as a four-dimensional matrix, that is, after completing the traversal of all offsets, a four-dimensional correlation volume is generated. The correlation volume contains the correlation information between the offsets of the x-axis and y-axis and the feature map. This volume effectively describes the feature space of relative motion between frames and provides basic data for subsequent posture prediction.

[0065] Through the above process, the four-dimensional correlation volume generated by the correlation feature extraction module accurately captures the relative motion between adjacent frames, especially the subtle displacement in the BEV plane. This process ensures the accuracy and stability of subsequent posture estimation.

[0066] In an embodiment, a pose prediction decoder is used to predict the pose of the target object based on the relevant volume, wherein the pose includes a translation vector and a rotation matrix.

[0067] Specifically, the posture prediction decoder includes a feature fusion and dimensionality reduction submodule, a fully connected submodule, and a post-processing submodule. Among them, the feature fusion and dimensionality reduction submodule is used to merge the offset dimensions in the four-dimensional correlation volume and then reduce the dimension through the convolution layer to output a three-dimensional correlation feature map. The dimensionality reduction operation not only reduces the complexity of the data, but also can extract the most critical motion information, ensuring that the efficient description of the relative motion between adjacent frames is retained.

[0068] The fully connected submodule is used to expand the three-dimensional correlation feature map into a one-dimensional vector and reduce the dimension through a series of fully connected layers, and then predict two outputs based on the one-dimensional vector through two branches, for a total of four outputs. Specifically, based on the reduced-dimensional features, it is first flattened into a one-dimensional vector, and then passed through a set of fully connected layers and nonlinear activation functions, followed by two branches: one branch is used to predict the two outputs of the target in the x and y directions for subsequent prediction of the translation vector, and the other branch also predicts two outputs for subsequent rotation matrices. These two branches are also implemented through fully connected layers. This method avoids the discontinuity problem caused by directly predicting the rotation matrix, thereby improving the stability and accuracy of posture estimation.

[0069] The post-processing submodule is used to predict the translation vector and rotation matrix based on the branch of the branch. Specifically, the two outputs of one branch are activated using the tanh activation function to obtain the translation vectors in the x and y directions, and the two outputs of the other branch are also activated using the tanh activation function, and then the cos and sin values ​​of the yaw angle are obtained after L2 regularization, which are further transformed to form a rotation matrix. The tanh activation function scales the output of the branch to a reasonable range, thereby reducing the impact of outliers and ensuring that the predicted attitude parameters (including translation and rotation) are reasonable in actual scenarios. In actual operation, the sine and cosine values ​​of the angle are scaled to 0-1, and L2 regularization is performed to ensure that the sum of the squares is 1, which can correspond to a unique yaw angle, and the translation vector is scaled to 0-4 (unit: meter).

[0070] Through the processing of the attitude prediction decoder, the attitude information can be effectively extracted from the correlation volume, and the translation vector and rotation matrix of the target object can be predicted with high accuracy. The combination of the attitude prediction decoding and the correlation feature extraction module provides an efficient and accurate attitude estimation method, which helps to realize a robust monocular visual odometry system.

[0071] The above-mentioned visual BEV encoder, related feature extraction module, and posture prediction decoder need to be parameter optimized before being applied. Specifically, posture supervision is adopted to supervise the training by calculating the L1 loss of the rotation matrix and displacement vector. Specifically, the prediction of the rotation matrix uses cosine and sine functions to avoid discontinuities in angle prediction, thereby improving the stability of posture estimation. The final loss function is the weighted sum of the translation error and rotation error constructed based on the predicted translation vector, rotation matrix and corresponding labels, which is used to update the network parameters. This supervision method effectively reduces the scale drift problem in traditional methods and improves the accuracy and robustness of overall motion estimation.

[0072] The above-mentioned monocular visual odometer system based on bird's-eye view representation includes multiple carefully designed neural network modules to ensure that scale drift is effectively reduced and motion estimation accuracy is improved in monocular visual odometer. The entire system includes image feature extraction, feature mapping, correlation calculation, posture prediction and supervised learning. These steps work together to optimize network parameters by using only posture supervision, without introducing complex auxiliary tasks such as depth estimation or optical flow estimation, thereby simplifying the implementation process of the system and maintaining high-precision motion estimation while reducing computational complexity. It is particularly suitable for application scenarios with high real-time and precision requirements such as robot autonomous navigation and unmanned driving.

[0073] like Figure 2 As shown, the embodiment also provides a monocular visual odometer method based on bird's-eye view representation, the method adopts the above system, and includes the following steps:

[0074] S210, using a visual BEV encoder to extract image features from a monocular image and predict the depth distribution probability of the features, then fusing the image features with their depth distribution probability and mapping them to BEV to obtain a bird's-eye view representation, then performing a conversion from a camera coordinate system to a body coordinate system and compressing it in the height direction to obtain a compressed bird's-eye view representation;

[0075] S220, using a relevant feature extraction module to calculate the correlation between the compressed bird's-eye view representations of two adjacent key-frame monocular images according to the compressed bird's-eye view representation and generate a relevant volume;

[0076] S230 , using a posture prediction decoder to predict the posture of the target object according to the relevant volume, wherein the posture includes a translation vector and a rotation matrix.

[0077] The above-mentioned monocular visual odometer system and method are mainly used in fields such as autonomous driving and planar motion robot navigation, and are particularly suitable for long time series scenarios that require high-precision motion estimation, which helps to improve the navigation capability and reliability of autonomous systems in complex environments. The core advantage of the present invention is that it greatly reduces the scale drift problem in long time series by optimizing the bird's-eye view representation under the configuration of a monocular camera, thereby improving the stability and accuracy of motion estimation, and also demonstrates its great potential in practical applications by simplifying the pipeline and improving computational efficiency and robustness. The following is a detailed description of the effect of the system of the present invention based on experimental results:

[0078] Significantly reduce scale drift: In monocular visual odometry systems, scale drift is one of the main issues affecting accuracy. The present invention effectively reduces scale drift in long time series by utilizing the inherent scale consistency of the bird's-eye view (BEV) representation. The experimental results shown in Table 1 show that the method of the present invention exhibits excellent scale consistency on multiple datasets (such as NCLT, Oxford, and KITTI).

[0079] Table 1

[0080]

[0081] * Scaled according to the ground truth trajectory of the first 10 meters and aligned using SE(3)

[0082] **Aligned using SE(3)

[0083] Scale drift table description: First, the trajectories of all methods are scaled according to the ground truth trajectory (GT) of the first 10 meters, except for the proposed method and DF-VO (tested on the KITTI dataset). Then, the scale drift is calculated using the following formula:

[0084]

[0085] Among them, Dscal e represents the average scale drift of all segments, N is the total number of segments, and d i is the estimated displacement distance of the ith segment, is the true displacement distance of the i-th segment. The reason for using logarithmic function and absolute value to measure scale drift is that scale drift usually manifests as proportional differences from the true value rather than linear deviations. The logarithmic function can normalize these proportional differences to ensure that overestimation and underestimation of the scale are punished equally.

[0086] Improve motion estimation accuracy: By testing on data sets of different complexity, the test results are shown in Table 2. The proposed method shows higher accuracy in absolute trajectory error (ATE), relative translation error (RTE) and relative rotation error (RRE) compared with other existing methods (such as ORB-SLAM3, DF-VO, DROID-SLAM). Especially on challenging data sets such as NCLT and Oxford, the performance of the proposed method is significantly better than other methods.

[0087] Table 2

[0088]

[0089] In Table 2, (w / LC) uses closed-loop optimization; (w / o LC) does not use closed-loop optimization; (w / 0GBA) does not use the final global bundle adjustment optimization; (F.ModeI) uses the basic model for bidirectional optical flow prediction and monocular depth estimation; (StereoTrained) uses a model trained with stereo camera data; (w / oDS) does not use depth supervision during training; * scales according to the real trajectory of the first 10 meters and uses SE(3) alignment; ** uses SE(3) alignment; ↑ uses Sim(3) alignment.

[0090] Excellent performance without deep supervision: The ablation experiments shown in Table 3 show that the method of the present invention can achieve comparable performance to traditional methods using deep supervision without introducing deep supervision. This is due to the use of posture supervision to optimize the feature depth distribution estimation of the visual BEV encoder without being restricted to capturing more effective key points in the perspective view. In addition, the use of posture supervision also avoids the inaccuracies caused by synchronization errors, external parameter calibration errors, and sparse point clouds introduced by LiDAR data.

[0091] Table 3

[0092]

[0093] Simplify the MVO pipeline and improve computational efficiency: This invention greatly simplifies the entire pipeline and improves the computational efficiency and robustness of the method by eliminating the complex steps in the traditional MVO method (such as bundle adjustment, posture graph optimization, and deep supervision, etc.). Compared with methods such as DROID-SLAM and DF-VO, this invention achieves a running speed of more than 60 frames per second (fps) on an RTX4090 graphics card with low memory consumption. This high efficiency and resource conservation make the method of the present invention more suitable for practical application scenarios.

[0094] Strong robustness and adaptability: Through the analysis of complete trajectories and selected trajectory segments, the proposed method demonstrates robustness in long-term and complex environments and excellent generalization in new environments. Even in the severely jittered NCLT dataset, the proposed method can still maintain scale accuracy from beginning to end, proving its stability in handling complex scenarios.

[0095] like Figure 3 As shown, the embodiment also provides intermediate variables and final output results of the system and method of the present invention during operation, including the following key parts:

[0096] 1. Figure 3 The trajectory comparison diagram shown in the upper left part shows the comparison between the trajectory predicted by the system of the present invention and the actual reference trajectory (GT). Figure 3 The A, B, C, and D positions marked in the figure correspond to the straight section, left turn scenario, left turn scenario, and right turn scenario, respectively. These positions are used to further demonstrate the operating results of the system in different situations.

[0097] 2. Figure 3 An example of a camera view shown in the upper right part of the figure shows the camera images at positions A, B, C, and D at time t and t+1, respectively. These camera images are used to compare the visual input of the system in different turning scenarios.

[0098] 3. Figure 3 The BEV feature maps and optical flow information shown in the bottom part of the figure show the BEV spatial feature maps encoded by the visual BEV encoder and the BEV optical flow information extracted from the relevant feature extraction module. Specific flow direction patterns can be observed in these feature maps during straight driving, left turning, and right turning. These patterns are more obvious in different turning scenarios, reflecting the deep distribution prediction ability of the visual BEV encoder when converting perspective view features to BEV representation.

[0099] Figure 3 The robustness and accuracy of the proposed system in handling complex turning scenarios are demonstrated, showing how the system can effectively reduce scale drift and improve the accuracy of pose estimation using BEV representation.

[0100] like Figure 4 As shown, the embodiment also provides trajectory comparison results under different data sets. Figure 4 The trajectory prediction results of the system of the present invention on the NCLT, Oxford and KITTI datasets are presented, and compared with the results of other comparison methods (such as ORB-SLAM3, DF-VO and DROID-SLAM). Figure 4 The content includes:

[0101] 1. Figure 4 The upper part of the figure shows the trajectory results of the NCLT dataset: the left image shows the test results of the complete path. Since other methods cannot complete the test on the entire path, the right image shows the results of the partial path. It can be seen that the trajectory prediction of the system of the present invention is closer to the reference trajectory (GT), showing strong scale consistency and trajectory accuracy.

[0102] 2. Figure 4 The middle part of the figure shows the trajectory results of the Oxford dataset: the left image also shows the test results of the complete path, and the right image shows the results of the partial path. The system of the present invention performs better than other methods in complex paths, especially in terms of scale consistency.

[0103] 3. Figure 4 The lower part shows the trajectory results of the KITTI dataset: the left figure is the test result of seq9, and the right figure is the test result of seq10. Compared with other methods, the system of the present invention shows higher accuracy in predicting the overall trajectory, especially when processing long-distance paths, the trajectory of the system of the present invention is more stable and close to the real trajectory.

[0104] Figure 4 The robustness and adaptability of the system of the present invention in different complex scenarios are further verified, demonstrating its advantages in maintaining scale consistency and improving motion estimation accuracy.

[0105] The following is an application example of the system and method of the present invention:

[0106] Application Example 1: Application in Autonomous Driving

[0107] In autonomous driving applications, vehicles need to continuously perform high-precision motion estimation in complex road environments to ensure safe driving and accurate path planning. Traditional monocular vision systems tend to accumulate errors over long periods of driving, resulting in scale drift, which affects the accuracy of the driving trajectory. The system of the present invention works in such scenarios in the following ways:

[0108] 1. Stable motion estimation in long-distance driving: When an autonomous vehicle is driving for a long time on a highway or urban road, it is necessary to maintain accurate pose estimation. The system of the present invention uses the scale consistency of the bird's-eye view to provide high-precision pose data even after long-distance driving, avoiding the error accumulation problem in traditional monocular vision systems.

[0109] 2. Real-time path adjustment and dynamic response: When facing dynamically changing traffic environments (such as sudden obstacles, complex urban road conditions, etc.), the system of the present invention can adjust the vehicle's driving path in real time through accurate posture estimation. Its advantage is that it can make fast and accurate decisions in dynamic environments without relying on additional depth information, thereby ensuring the safety and driving efficiency of the vehicle.

[0110] Application Example 2: Application in Planar Motion Robot

[0111] In fields such as industrial automation or service robots, robots usually perform various tasks in a planar environment, such as cargo handling in a warehouse or automatic inspection in a factory. These robots need to maintain stable paths and motion estimates during long-term operation to ensure accurate completion of tasks. The application of the system of the present invention in such scenarios is as follows:

[0112] 1. Continuous and accurate positioning in complex environments: In large warehouses or factories, robots need to move for a long time and maintain the accuracy of the path. Traditional monocular vision systems are prone to scale drift, which may cause the robot to deviate from the predetermined trajectory. The system of the present invention can provide stable posture estimation during long-term task execution by optimizing the scale consistency of image features, ensuring that the robot accurately completes the predetermined task.

[0113] 2. Efficient use of monocular cameras: Due to cost and equipment limitations, many robots can only be equipped with monocular cameras. The system of the present invention can achieve efficient motion estimation using only a monocular camera, meeting the accuracy requirements of the robot in planar motion. Especially in dynamic environments, the system can adapt to environmental changes and ensure the continuous accuracy of the robot's path.

[0114] Application Example 3: Application in Plane Navigation Unmanned Vehicle

[0115] In the application of unmanned vehicles, such as indoor delivery vehicles or automatic guided vehicles at airports, they usually run for a long time in a flat environment and need to maintain high-precision motion estimation. The advantage of the system of the present invention is its low-scale drift characteristics, making it particularly suitable for such application scenarios:

[0116] 1. Efficient path planning and execution: Unmanned vehicles need to run on a predetermined path and navigate in a complex planar environment. Through the system of the present invention, the unmanned vehicle can maintain accurate posture estimation during long-term operation, and even if the environment changes during path planning and execution, the system can still ensure the accuracy of navigation.

[0117] 2. Response and adjustment in dynamic environments: In busy scenes (such as airports or shopping malls), unmanned vehicles need to deal with the presence of a large number of pedestrians or temporary obstacles. The system of the present invention provides real-time high-precision posture data, enabling unmanned vehicles to quickly adjust their driving paths, avoid obstacles and continue to perform their tasks.

[0118] In summary, the system of the present invention, under the configuration of a monocular camera, is particularly suitable for application scenarios that require accurate pose estimation, such as autonomous driving and planar motion robots, through long-term stable low-scale drift motion estimation. Its advantage is that it can maintain high-precision, low-drift motion estimation in a long time series, and adapt to the high-precision motion requirements in various complex dynamic environments.

[0119] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A monocular visual odometer system based on bird's-eye view representation, characterized in that: It includes a visual BEV encoder, a relevant feature extraction module, and a posture prediction decoder; The visual BEV encoder is used to extract image features from a monocular image and predict the depth distribution probability of the features, then fuse the image features with their depth distribution probability and map them to BEV to obtain a bird's-eye view representation, then perform a conversion from the camera coordinate system to the body coordinate system and perform height compression to obtain a compressed bird's-eye view representation; The relevant feature extraction module is used to calculate the correlation between the compressed bird's-eye view representations of two adjacent key frame monocular images according to the compressed bird's-eye view representation and generate a relevant volume; The posture prediction decoder is used to predict the posture of the target object based on the relevant volume, wherein the posture includes a translation vector and a rotation matrix.

2. The monocular visual odometer system based on bird's-eye view representation according to claim 1, characterized in that: The visual BEV encoder includes a feature extraction submodule, a camera parameter auxiliary encoding submodule, a depth distribution prediction submodule, and a feature mapping submodule; The feature extraction submodule is used to extract a multi-scale perspective feature map from a monocular image; The camera parameter auxiliary encoding submodule is used to encode the camera parameters and obtain a weighted feature map by weighting the camera parameter encoding and the perspective view feature map through channel attention; The depth distribution prediction submodule is used to predict the depth distribution probability of features through a deep network based on the weighted feature map; The feature mapping submodule is used to fuse the image features after convolution of the weighted feature map with the depth distribution probability of the features and map them to BEV to obtain a bird's-eye view representation, convert the bird's-eye view representation from the camera coordinate system to the body coordinate system and compress it in the height direction to obtain a bird's-eye view representation after voxel compression.

3. The monocular visual odometer system based on bird's-eye view representation according to claim 2 is characterized in that: The feature extraction submodule uses the ResNet-50+FPN structure to extract multi-scale perspective feature maps; The camera parameter auxiliary encoding submodule adopts MLP encoding camera parameters.

4. The monocular visual odometer system based on bird's-eye view representation according to claim 2, characterized in that: The deep network used by the depth distribution prediction submodule includes Res Block, ASPP, and DCN connected in sequence, wherein Res Block performs dimensionality reduction of weighted features, ASPP expands the receptive field of features after dimensionality reduction and captures deep features of different scales, and DCN further extracts deep features to obtain deep distribution features.

5. The monocular visual odometer system based on bird's-eye view representation according to claim 2, characterized in that: In the feature mapping submodule, the process of fusing the image features after convolution of the weighted feature map with the depth distribution probability of the features is as follows: The size of the image feature is changed from the original C×H feature ×W feature Insert a dimension to become C×1×H feature ×W feature , where C represents the number of channels, H feature Represents the feature map height, W feature Represents the width of the feature map, 1 is an introduced dimension; The size of the depth distribution probability is changed from the original D×H feature ×W feature Insert a dimension to become 1×D×H feature ×W feature , where D represents the depth dimension of the depth distribution probability; The size is C×1×H feature ×W feature The image features and size are 1×D×H feature ×W feature The depth distribution probability of each feature is multiplied by the depth distribution probability of each feature to form C×D×H feature ×W feature The matrix of .

6. The monocular visual odometer system based on bird's-eye view representation according to claim 1, characterized in that: The relevant feature extraction module includes a feature translation submodule, a correlation calculation submodule, and a relevant volume generation submodule; The feature translation submodule is used to perform translation operations of different step lengths on the compressed bird's-eye view representation of the next frame of monocular image and record the size D x ×D y The offset matrix is ​​used to obtain a series of bird's-eye view representations after translation; The correlation calculation submodule is used to calculate the H with the same size as the bird's-eye view feature map according to a series of translated bird's-eye view representations. BEV ×W BEV The correlation score matrix captures the relative motion information between two adjacent monocular images. The correlation volume generation submodule is used to generate a volume with a size of D according to the correlation score matrix and the offset matrix. x ×D y ×H BEV ×W BEV The four-dimensional correlation volume.

7. The monocular visual odometer system based on bird's-eye view representation according to claim 6, characterized in that: The cosine similarity is used to calculate the correlation score between two adjacent monocular images.

8. The monocular visual odometer system based on bird's-eye view representation according to claim 1, characterized in that: The posture prediction decoder includes a feature fusion and dimension reduction submodule, a fully connected submodule, and a post-processing submodule; The feature fusion and dimension reduction submodule is used to merge the offset dimensions in the four-dimensional correlation volume and then reduce the dimension through the convolution layer to output a three-dimensional correlation feature map; The fully connected submodule is used to expand the three-dimensional correlation feature map into a one-dimensional vector and reduce the dimension through a series of fully connected layers, and then predict two outputs based on the one-dimensional vector through two branches, for a total of four outputs; The post-processing submodule is used to perform tanh activation and scaling on the two outputs of one branch to obtain the translation vectors in the x and y directions, and to perform tanh activation and L2 regularization on the two outputs of the other branch in turn to obtain the cos and sin values ​​of the yaw angle, which are further transformed to form a rotation matrix.

9. The monocular visual odometer system based on bird's-eye view representation according to claim 1, characterized in that: The visual BEV encoder, related feature extraction module, and posture prediction decoder need to undergo parameter optimization before being applied. The loss function used in parameter optimization is to update the parameters based on the translation error and rotation error constructed based on the predicted translation vector, rotation matrix and corresponding labels.

10. A monocular visual odometer method based on bird's-eye view representation, characterized in that: The method adopts the system according to any one of claims 1 to 9, comprising the following steps: The visual BEV encoder is used to extract image features from the monocular image and predict the depth distribution probability of the features. The image features and their depth distribution probability are then fused and mapped to the BEV to obtain a bird's-eye view representation. The camera coordinate system is then converted to the body coordinate system and compressed in the height direction to obtain a compressed bird's-eye view representation. The correlation feature extraction module is used to calculate the correlation between the compressed bird's-eye view representations of two adjacent key-frame monocular images according to the compressed bird's-eye view representation and generate a correlation volume; The pose prediction decoder is used to predict the pose of the target object based on the relevant volume, where the pose includes the translation vector and the rotation matrix.

Images