Monocular visual odometer system and method based on PV-BEV fusion and dense flow supervision

Abstract

The invention discloses a monocular visual odometer system and method based on PV-BEV fusion and dense flow supervision, through a PV-BEV double-branch architecture, correlation volume is calculated in a PE space, 5-degree-of-freedom motion information is reserved, and then shared LSS parameters are projected to a BEV space to be fused with BEV native correlation, so that information loss of a traditional BEV method is effectively compensated. Meanwhile, a dense BEV optical flow field is directly constructed from a three-degree-of-freedom pose truth value by using the measurement scale characteristic represented by BEV, a pixel-level supervision signal is provided, and the limitation of sparse pose supervision is overcome. In cooperation with an enhanced rotation sampling strategy, the system achieves the first advanced performance on a plurality of data sets, the relative translation error is reduced by more than 40% compared with an existing BEV method, and extra labeling such as depth and optical flow is not needed. The method is particularly suitable for low-cost and high-precision visual navigation of intelligent traffic scenes such as automatic driving and logistics robots.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and intelligent transportation technology, specifically relating to a monocular visual odometry (MVO) technology based on deep learning, and particularly to a monocular visual odometry system and method that utilizes a perspective view to bird's eye view (PV-BEV) dual-branch fusion architecture and dense BEV optical flow supervision. Background Technology

[0002] Visual odometry (MVO) is one of the core technologies for autonomous navigation of mobile robots. Its goal is to estimate the motion trajectory of a camera (i.e., the mobile platform) through a continuous sequence of images. Compared to binocular or RGB-D vision systems, monocular visual odometry (MVO) requires only a single camera, offering advantages such as lower cost and more flexible deployment, making it of significant application value in intelligent transportation systems.

[0003] Traditional monocular visual odometry methods, such as ORB-SLAM and DSO, achieve pose estimation through feature extraction and matching or direct optimization, but they have the following inherent limitations: (1) Scale ambiguity: Monocular cameras cannot directly obtain the absolute scale information of the scene and need to establish a scale reference through several initial frames. During long-distance operation, due to factors such as feature matching error, motion blur, and calibration error, the scale estimation will continue to drift, resulting in accumulated errors that seriously affect the positioning accuracy. (2) Feature dependence: Feature-based methods are prone to failure in degraded scenarios such as texture loss, drastic changes in lighting, and interference from dynamic objects, and lack robustness.

[0004] In recent years, deep learning-based MVO methods have overcome some of the limitations of traditional methods through end-to-end training, but still face the following technical challenges: (1) Auxiliary supervision dependency problem: Many methods (such as TartanVO, DF-VO, DROID-SLAM) require auxiliary supervision signals such as depth maps, optical flow fields, and semantic segmentation to anchor the scale or enhance geometric constraints. These additional supervision data require expensive sensor equipment (such as LiDAR, binocular cameras) or large-scale manual annotation, which significantly increases the data acquisition cost and system complexity, limiting the practicality and scalability of the methods. (2) Perspective view scale consistency problem: Even if perspective view-based methods introduce depth or optical flow supervision, it is still difficult to maintain stable scale consistency during long-term operation, especially in complex dynamic scenes where scale abrupt changes or drifts are prone to occur.

[0005] Bird's Eye View (BEV) representation has made significant progress in autonomous driving perception tasks in recent years. Given that the movement of ground mobile robots mainly occurs on the ground plane, BEV representation has inherent advantages such as implicit scale anchoring and spatial consistency. Existing BEV-based MVO methods (such as BEV-ODOM and BEV-DWPVO) have demonstrated that excellent scale consistency can be achieved using only 3-DoF pose supervision. However, these methods still have two key technical bottlenecks: (1) Sparsity of supervision signals: Existing BEV methods use only a single 3-DoF pose vector (three scalars: ) as a supervision signal. For a typical 128×128 pixel BEV feature map, there are 16,384 pixels, but the supervision signal has only 3 values. This extremely sparse supervision cannot provide the network with pixel-level correspondence guidance, which limits the network's ability to learn fine motion patterns. Recent studies (such as DUSt3R) show that dense, pixel-level supervision is crucial for geometric understanding. (2) Information loss from PV to BEV projection: Existing methods usually use the Lift-Splat-Shoot (LSS) architecture to project perspective view features onto the BEV plane. This projection process is essentially compressing 3D information into a 2D plane, which inevitably loses geometric cues related to non-primary motion axes. Specifically, it includes: ① Loss of pitch angle change information: When a vehicle travels over a slope or uneven road surface, changes in pitch angle cause vertical displacement of the same scene point in the image, but this information is erased after projection onto the BEV. ② Loss of roll angle change information: The lateral roll of a vehicle during cornering generates radial displacement patterns of feature points, which cannot be distinguished after BEV projection. ③ Loss of vertical displacement (z-axis translation) information: Although the z-axis motion of a vehicle on the ground is small, it still exists under bumpy road conditions, and this motion information is not visible in the BEV plane. These information losses cause the network to learn inconsistent motion patterns during training, reducing estimation accuracy and robustness in complex scenes. For example, when the road slope changes, the feature displacement patterns in the perspective view differ from those of pure planar motion, and the BEV representation cannot capture this difference, potentially leading to incorrect motion estimation.

[0006] Furthermore, existing MVO training datasets (such as KITTI) exhibit significant motion pattern bias: most sequences primarily involve straight-line driving, with a low proportion of samples showing large rotational movements such as turning. This leads to overfitting of trained models to straight-line motions, while their generalization ability to maneuver maneuvers such as turning and U-turns is insufficient, limiting their application in real-world complex traffic scenarios.

[0007] In summary, the existing technology has the following pressing technical problems that need to be solved: (1) How to construct richer signals from existing pose ground truth without introducing additional sensors or manual annotation, so as to provide dense, pixel-level supervision for BEV-based MVO; (2) How to compensate for the 6-DoF motion information lost during the projection from PV to BEV while maintaining the implicit scale anchoring advantage of BEV; (3) How to balance the sample distribution of different motion modes at the data level and enhance the model’s adaptability to diverse scenarios. Summary of the Invention

[0008] To address the shortcomings of the existing technologies, the present invention aims to provide a monocular visual odometry system and method based on PV-BEV fusion and dense flow supervision, applicable to the autonomous localization of ground mobile robots in visual navigation systems of ground mobile platforms such as autonomous vehicles, logistics delivery robots, and intelligent inspection vehicles.

[0009] To achieve the above-mentioned objectives, an embodiment provides a monocular visual odometry method based on PV-BEV fusion and dense flow supervision, comprising the following steps: Collect consecutive frames of images as sample data; A monocular visual odometry model is constructed, comprising: a feature extraction module for extracting multi-scale image features from continuous images and fusing them with the encoded features corresponding to camera parameters to obtain PV features; a PV processing module for calculating the PV correlation volume based on the PV features in PV space and then decoding and predicting 6-DoF motion information; a BEV projection module for projecting the PV features and PV correlation volume onto BEV space to obtain BEV features and projected PV correlation volume, respectively; a BEV processing module for calculating the BEV correlation volume based on the BEV features in BEV space and fusing it with the projected PV correlation volume to obtain a fused correlation volume; and a decoding processing module for decoding the fused correlation volume to predict 3-DoF motion information and BEV optical flow field. The model is constructed and supervised training is conducted, which includes: directly constructing the dense flow ground truth based on the 3-DoF pose ground truth; constructing the dense flow loss based on the dense flow ground truth and the BEV optical flow field; constructing the 6-DoF PV pose loss based on 6-DoF motion information; constructing the 3-DoF BEV pose loss based on 3-DoF motion information; and combining all the losses and augmenting the sample data to supervise the training of the model to optimize the model parameters. Monocular visual odometry is performed using a parameter-optimized model.

[0010] To achieve the above-mentioned objectives, embodiments of the present invention also provide a monocular visual odometry system based on PV-BEV fusion and dense flow supervision, comprising: The data augmentation unit is used to acquire consecutive frames of images as sample data. The model building unit, used to construct a monocular visual odometry model, includes: a feature extraction module for extracting multi-scale image features from continuous images and fusing them with the encoded features corresponding to camera parameters to obtain PV features; a PV processing module for calculating the PV correlation volume based on the PV features in PV space and then decoding and predicting 6-DoF motion information; a BEV projection module for projecting the PV features and PV correlation volume onto BEV space to obtain BEV features and projected PV correlation volume, respectively; a BEV processing module for calculating the BEV correlation volume based on the BEV features in BEV space and fusing it with the projected PV correlation volume to obtain a fused correlation volume; and a decoding processing module for decoding the fused correlation volume to predict 3-DoF motion information and BEV optical flow field. The training unit, which is used to construct the loss and supervise training, includes: directly constructing the dense flow ground truth based on the 3-DoF pose ground truth; constructing the dense flow loss based on the dense flow ground truth and the BEV optical flow field; constructing the 6-DoF pose loss based on 6-DoF motion information; constructing the 3-DoF BEV pose loss based on 3-DoF motion information; and supervising the training of the model by combining all the losses and augmenting the sample data to achieve model parameter optimization. The inference unit is used for monocular visual odometry using a parameter-optimized model.

[0011] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention employs a PV-BEV dual-branch architecture. It calculates the correlation volume in the PE space while preserving 5-DOF motion information, and then uses shared LSS parameters to project it into the BEV space, fusing it with the native BEV correlation. This effectively compensates for the information loss of traditional BEV methods. Simultaneously, leveraging the metric scaling characteristics of BEV representation, it directly constructs a dense BEV optical flow field from the ground truth 3-DOF pose, providing pixel-level supervision signals and overcoming the limitations of sparse pose supervision. Combined with an enhanced rotation sampling strategy, the system achieves state-of-the-art performance on multiple datasets, reducing relative translation error by more than 40% compared to existing BEV methods, without requiring additional annotations for depth, optical flow, etc. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of the monocular vision odometry method based on PV-BEV fusion and dense flow supervision provided in the embodiment; Figure 2 This is a schematic diagram of the structure of the monocular vision odometry model provided in the embodiment; Figure 3 This is a schematic diagram of the method for constructing dense BEV optical flow monitoring signals provided in the embodiment; Figures 4-7 This is a comprehensive performance case study diagram of BEV-ODOM2 on four datasets provided in the implementation examples; Figure 8 This is a schematic diagram illustrating the comprehensive description of the ZJH-VO multi-scale dataset provided in the embodiment; Figure 9 This is a table showing the performance comparison results on the NCLT and Oxford datasets provided in the example; Figure 10 This is a table showing the performance comparison results from the KITTI odometer dataset provided in the example. Figure 11 This is a table showing the performance comparison results on the ZJH-VO multi-scale odometer dataset provided in the example; Figure 12 These are tables of ablation experiment results from three datasets provided in the example; Figure 13 This is a schematic diagram of the structure of a monocular vision odometry system based on PV-BEV fusion and dense flow supervision, provided in the embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0015] The inventive concept of this invention is to provide a monocular visual odometry scheme based on PV-BEV fusion and dense flow supervision, in order to solve the following technical problems: (1) Construct a unified, end-to-end trainable framework. All supervision signals are derived from the pose ground truth. No additional annotations such as depth and optical flow are required, but it can provide dense, pixel-level supervision signals, overcoming the limitations of supervision using only sparse pose vectors. (2) While maintaining the metric advantages and implicit scale anchoring capabilities of BEV representation, it effectively compensates for the loss of 6-DoF motion information during the PV to BEV projection process. (3) Provide a data augmentation strategy to balance the distribution of different motion modes in the training data. Improve the odometry accuracy, robustness and scale consistency of ground mobile robots in complex motion modes (especially large-angle turns) and diverse scenarios.

[0016] like Figure 1 As shown in the embodiment, a monocular visual odometry method based on PV-BEV fusion and dense flow supervision includes the following steps: S1, acquire consecutive frame images as sample data.

[0017] In this embodiment, monocular camera images at consecutive times t and t+1 are acquired. , The camera's intrinsic and extrinsic parameters K and E are used as sample data for training the model.

[0018] S2, Construct a monocular visual odometry model.

[0019] In the embodiments, such as Figure 2 As shown, the constructed monocular visual odometry model includes a feature extraction module, a PV processing module, a BEV projection module, a BEV processing module, and a decoding processing module. These modules are combined to predict 5-DOF (6-DoF) motion information, 3-DoF motion information, and BEV optical flow field.

[0020] In this embodiment, the feature extraction module is used to extract multi-scale image features from continuous images and then fuse them with the coded features corresponding to camera parameters to obtain perspective view features (PV features). Specifically, this includes: The image feature extraction submodule uses a ResNet-50 backbone network combined with a Feature Pyramid Network (FPN) to extract features from consecutive time points. and Monocular images , Perform feature extraction and output multi-scale image features. ,in The number of feature channels, and The height and width of the PV feature map; The camera parameter encoding submodule is used to encode the camera intrinsic parameter matrix. and extrinsic parameter matrix Encode the features into feature vectors using a multilayer perceptron (MLP) and combine them with multiscale image features. Element-wise multiplication and fusion are performed to obtain the encoded PV features. .

[0021] In this embodiment, the PV processing module is used to decode and predict 6-DoF motion information after calculating the PV correlation volume based on PV features in the PV space. Specifically, it includes: The PV correlation calculation submodule is used to calculate the correlation between consecutive time points. and PV characteristics and Calculate feature correlation within a local neighborhood to generate PV correlation volume. : in As coordinates, , For the search radius, For the current channel, This represents the total number of channels for the PV feature.

[0022] The PV correlation calculation submodule calculates correlations in PV space before LSS projection, thus preserving complete 6-DoF motion features, including pitch angle, roll angle, and z-axis translational non-planar motion information. Specifically, this includes vertical parallax caused by pitch angle changes, radial displacement patterns caused by roll angle changes, and scale changes caused by z-axis translation.

[0023] The PV pose decoding submodule is used to decode the pose using a convolutional neural network (PNN). ) and fully connected layers ( ) for PV-related volume Dimensionality reduction and regression are performed to predict 6-DoF motion information. , For the complete rotation matrix, The normalized translation vector (unit vector) is calculated using the following formula: In this embodiment, the BEV projection module is used to project the PV features and PV-related volumes onto the BEV space to obtain the BEV features and the projected PV-related volumes, thereby projecting the PV features and PV-related volumes onto a unified BEV space. Specifically, this includes: The deep-context prediction submodule is used for PV features. Predicting deep features using two convolutional networks respectively and context features ,in This represents the discretized depth resolution.

[0024] The feature fusion submodule is used to fuse contextual features. and depth features The fused features are obtained by performing dimensional expansion and element-wise multiplication. : in , ⊙ represents broadcast multiplication, resulting in .

[0025] The LSS projection submodule is used for projection using an LSS network, specifically for fusing features. The BEV features are generated by mapping the image to a 3D voxel space using frustum projection, transforming it to the vehicle coordinate system, and finally accumulating the features onto the BEV plane using voxel pooling. ,in , and The size of the BEV feature map; The PV correlation projection submodule is used to project the PV correlation volume using the exact same LSS network parameters (including the depth prediction network) as the feature projection. Projecting into the BEV space, specifically the depth features Volume related to PV After fusion, the data is projected onto a 3D voxel space via view cone projection, then transformed to the vehicle coordinate system, and finally accumulated onto the BEV plane via voxel pooling to obtain the projected PV-related volume. ， The projection formula is: In the PV correlation projection submodule, the correlation volume encodes the relative motion pattern at the feature level, rather than the absolute spatial coordinates, and therefore can be treated as a feature channel. The shared LSS parameters ensure the geometric consistency between the motion features derived from PV and the native features of BEV. In this way, 6-DoF motion information enters the BEV space in the form of features, while maintaining compatibility with the BEV metric scale.

[0026] Correlation is treated as a feature channel: each channel of the correlation volume encodes the motion pattern under a specific displacement (Δx, Δy), and the projection process transforms these motion patterns from PV geometry to BEV geometry; Shared depth prediction: Ensure that PV correlation and PV features use consistent geometric mapping relationships to guarantee the alignment of the fused feature space; Information compensation mechanism: The 6-DoF motion cues exist in the form of implicit features, and the network can learn to extract information that is beneficial to the 3-DoF pose in subsequent processing.

[0027] In this embodiment, the BEV processing module is used to calculate the BEV correlation volume based on BEV features in the BEV space and then fuse it with the projected PV correlation volume to obtain a fused correlation volume. Specifically, this includes: The BEV correlation calculation submodule is used to calculate the BEV features at consecutive projected time points. and Calculate feature correlations within a local neighborhood to generate the BEV correlation volume. : in As coordinates, , For the search radius, For the current channel, The total number of channels for BEV features. The multimodal correlation fusion submodule is used to condense the BEV correlation volume. Correlation volume with projected PV By stitching along the channel dimension, the fused correlation volume is obtained. : in, In the multimodal correlation fusion submodule, It encodes the direct planar motion correspondences in BEV space, exhibiting scale consistency, but only contains two-dimensional planar motion information. Carrying the 6-DoF motion information before projection, the network maintains the same spatial resolution and scale as C_BEV after LSS projection, compensating for pitch / roll / z-axis changes. Through channel-level stitching rather than additive fusion, the network can adaptively learn the weights of the two motion cues.

[0028] In this embodiment, the decoding processing module is used to decode and predict 3-DoF motion information and BEV optical flow field from the fused correlation volume. Specifically, it includes: UNet encoder, used for fusion correlation volume Perform multi-layer convolutional downsampling to extract global motion context. : Dense BEV optical flow prediction branch: Reconstructs a dense two-channel BEV optical flow field using a UNet decoder (with skip connections). This optical flow field represents the displacement vector of each pixel in the BEV space between consecutive frames. Skip connections preserve local correspondences in the encoder, preventing the loss of fine-grained motion information during downsampling; Dense prediction: Predicts a two-dimensional displacement vector for each BEV pixel to form a complete motion field.

[0029] The 3-DoF pose prediction branch is used to extract features from the penultimate layer of the UNet decoder. Through convolution Dimensionality reduction and fully connected layers Returning to the final 3-DoF motion information: Where θ is the yaw angle. and This is a translation of the BEV plane.

[0030] The penultimate layer of features has been optimized with dense flow supervision and contains accurate pixel-level correspondences. Compared to predicting pose directly from the encoder, this design ensures that pose estimation benefits from the learning process of flow prediction.

[0031] The technical considerations in designing the decoding processing module are: pose prediction directly utilizes features optimized by dense flow supervision to ensure that pose estimation is based on accurate pixel-level correspondences.

[0032] S3, construct the loss and supervise training.

[0033] In this embodiment, a multi-level supervision signal is constructed and a loss function is calculated. Specifically, the dense flow ground value is directly constructed based on the 3-DoF pose ground value, the dense flow loss is constructed based on the dense flow ground value and the BEV optical flow field, the 6-DoF PV pose loss is constructed based on the 6-DoF motion information, and the 3-DoF BEV pose loss is constructed based on the 3-DoF motion information. The model parameters are optimized by combining all the loss supervision training models.

[0034] Specifically, by utilizing the metric scaling characteristics of BEV representation, the true value of 3-DoF pose is obtained. The ground truth of dense flow can be constructed directly without the need for external optical flow sensors or annotations, including the 3-DoF relative pose ground truth. The transformation is constructed as a 4×4 homogeneous transformation matrix: For each pixel position (u, v) in the BEV feature map, the calculation steps are as follows: (1) Establish the homogeneous coordinates of each pixel position (u, v) in the vehicle coordinate system. : in BEV resolution (meters per pixel). The pixel coordinates of the origin of the vehicle coordinate system in the BEV image; (2) Using 3-DoF pose truth values Perform relative pose transformation: (3) Transform the coordinates Project back into pixel space and calculate the optical flow vector. : in This is the reverse mapping from vehicle coordinates to pixel coordinates, i.e.: at this time, .

[0035] Figure 3 This is a schematic diagram of the construction method for dense BEV optical flow monitoring signals. Figure 3 This paper describes how to directly construct a dense BEV optical flow field from the ground truth of relative pose. The left side shows the BEV mesh space, where the origin of the vehicle coordinate system is located at the center of the mesh, and the mesh resolution is r meters / pixel. The motion parameters experienced by the vehicle are labeled (Yaw = 15°, Δx = 3 pixels). The position of each mesh point in the next frame is calculated through homogeneous coordinate transformation and 2D rigid transformation. The right side shows the visualization result of the constructed dense optical flow field. The flow vectors exhibit an arc pattern consistent with the vehicle's steering motion, and the flow field density has a corresponding two-dimensional displacement vector for each BEV pixel. This construction method utilizes the uniform metric scale characteristics of BEV representation, eliminating the need for additional optical flow sensors or annotations, and generating pixel-level dense supervision signals solely from the ground truth of pose.

[0036] When constructing the ground truth of dense flow, the two-dimensional rigid body transformation property of BEV is utilized, eliminating the need for complex depth-dependent calculations; it is directly constructed from the pose, meaning that the flow and pose originate from the same ground truth, ensuring geometric consistency with pose supervision; it provides dense supervision signals for 16,384 pixels (128×128 grid), increasing the supervision density by 5000 times compared to 3 pose scalars, and eliminating the need for optical flow sensors, stereo matching, or manual annotation.

[0037] When constructing the dense flow loss based on the dense flow ground truth and BEV optical flow field, the following is adopted: Loss calculation of the difference between the predicted flow and the true value: in, For dense flow loss, For BEV optical flow field, For the truth value of dense flow, It is an L1 norm.

[0038] A 6-DoF PV pose loss is constructed based on 6-DoF motion information to supervise the prediction of the PV branch. Specifically, the Frobenius norm of the normalized translation and rotation matrices is used. in, For 6-DoF PV pose loss, and These are the predicted and true values ​​for the normalized translation direction, respectively. and These are the predicted and true values ​​of the rotation matrix, respectively. It is an L1 norm. It is the Frobenius norm. The balance coefficients, translation directions, and rotation matrices constitute 6-DoF motion information. Indicates the normalized translation direction.

[0039] The rationale for normalized translation is to maintain consistency with the scale ambiguity of monocular vision, with scale information determined entirely by depth prediction and BEV projection, ensuring compatibility of feature representations for PV and BEV branches.

[0040] A 3-DoF BEV pose loss is constructed based on 3-DoF motion information to supervise the final output 3-DoF pose: in, For 3-DoF BEV pose loss, and These are the BEV planes along x Predicted and true values ​​of directional translation. and These are the BEV planes along y Predicted and true values ​​of directional translation. and respectively These are the predicted and true values ​​of the yaw angle. Dimensions of error used to balance translation and rotation.

[0041] In this embodiment, the model parameters are optimized by combining all loss-supervised training models, specifically by weighting and combining the three loss terms: in, For the total loss, and This is a hyperparameter, set to [value] in this embodiment of the invention. The total loss is monitored at multiple levels, and its synergistic effects include: Dense flow supervision: provides pixel-level guidance and strengthens correspondence learning; 6-DoF PV supervision: ensures that the PV branch extracts effective 6-DoF motion features (without depth, depth is introduced through the projection process); 3-DoF BEV supervision: directly optimizes the final output and ensures task relevance; the three work together to form a hierarchical constraint system from local to global and from multiple views to a unified view.

[0042] In this embodiment, sample data augmentation is used to balance the sampling distribution of different motion modes during training, including: Each training sequence undergoes offline preprocessing to construct a motion-aware frame pair database, specifically including: for each frame in the sequence Establish a time window Enumerate all frame pairs within the window. Calculate relative pose Based on relative pose Extracting yaw angle difference Translational distance Based on filtering criteria Meters will classify frame pairs that meet the following conditions: if Store in a high-rotation list ,like Store in a standard rotation list For each frame Maintain the corresponding and ; The technical considerations for this design include: the 60-second time window ensures the continuity of scene content and lighting conditions, while being long enough to capture back-and-forth motion; the 4-meter distance constraint ensures sufficient feature overlap for correspondence learning; and for angle segmentation, The following are regular exercises. For challenging turns, The above actions are too drastic and may result in feature mismatch.

[0043] During training, when a certain frame is selected... At that time, frame pairs are sampled with non-uniform probability, specifically 70% probability from Random sampling, 0% probability from Random sampling in the dataset effectively mitigates the bias caused by the dominance of linear motion in the dataset, ensuring that the network fully learns motion patterns such as turning.

[0044] In summary, the technical effects of data augmentation include: Mitigating data bias: In datasets such as KITTI, linear motion accounts for a large proportion. This strategy increases the effective proportion of large rotation samples to 70%; Maintaining diversity: 30% of regular motion samples are still retained to avoid overfitting to turning motions; Temporal consistency: A 60-second window ensures the similarity of scene content and simulates real-world scenarios such as two-way driving.

[0045] S4 utilizes a parameter-optimized model for monocular visual odometry.

[0046] In this embodiment, during the inference calculation using the parameter-optimized model, consecutive image pairs are input, and the 3-DoF pose is obtained through forward propagation. Simultaneously, the relative pose is accumulated into the global trajectory: .

[0047] Figures 4-7 This diagram presents a comprehensive overview of the performance of the method of this invention on four datasets. Figures 4-7 Representative trajectories and detailed analyses for four datasets (NCLT, Oxford, KITTI, and ZJH-VO) are presented. Each dataset includes: a comparison of the complete predicted trajectory (red dashed line) with the ground truth trajectory (black solid line), highlighting representative right-turn (R), straight-ahead (S), and left-turn (L) motion segments; a visualization of dense BEV optical flow field predictions for each representative segment, with flow vectors represented by colored arrows, clearly showing the arc-shaped flow field during turns and the parallel flow field during straight-ahead movements; a corresponding optical flow prediction error heatmap, using a purple-blue tone to represent error magnitude, with most pixel errors below 0.4 pixels; and violin plots showing the translational and rotational error distributions for 25 frames (51 frames in total) centered on a selected segment, illustrating the statistical characteristics of errors under the three motion modes of straight-ahead, left-turn, and right-turn. This comprehensive visualization validates the effectiveness of the enhanced rotational sampling and dense supervision strategies, maintaining a consistently low error level across different motion modes.

[0048] Figure 8 This is a schematic diagram illustrating the comprehensive features of the ZJH-VO multiscale dataset. Figure 5 The dataset is divided into three parts: The left side shows the data acquisition platform, including a four-wheeled mobile robot equipped with a four-camera system, with annotations illustrating the camera coordinate system settings and the transformation relationship from perspective view to bird's-eye view. The front-view stereo camera pair and the left and right front cameras have approximately 30 degrees of field of view overlap. The middle section shows representative trajectory path diagrams for four floors (9th, 4th, 1st, and 0th floors), with each floor containing three trajectories with unique coverage areas (only one trajectories per floor are shown in the diagrams). The right side shows real-world image sample pairs (non-adjacent frames) for each environment, from top to bottom: dense office area (dense furniture, large-angle rotation, white-walled corridor), conference corridor (variable lighting, glass curtain wall, narrow passage), outdoor open space (sparse features at long distances, ground-dominated vision), and underground parking garage (dynamic vehicles, dim lighting). The dataset contains a total of 12 trajectories, 12,666 frames, and 3,054 meters of motion data, with a sampling interval of 8 frames. The pose ground truth was obtained through a pre-built 2D LiDAR map and positioning system.

[0049] Figure 9 This table shows the performance comparison results on the NCLT and Oxford datasets. Figure 9The relative translation error (RTE) and relative rotation error (RRE) performance of the proposed method were compared with those of traditional methods (ORB-SLAM3), perspective view learning methods (DeepVO, TartanVO, DF-VO, DROID-SLAM, DPVO), and bird's-eye view learning methods (BEV-DWPVO, BEV-ODOM) on the NCLT dataset (sequences 02-02, 02-19, 03-17, 08-20) and the Oxford dataset (sequences 11-12, 15-13, 16-14, 17-12). The table uses the "Ex. Supv." column to indicate whether the method requires additional supervision signals (depth, optical flow, semantic segmentation), where ✓ indicates that it is required, and ✗ indicates that only pose supervision is required; under the "Metric" column, * indicates that SE(3) alignment is used, and ** indicates that SE(3) alignment is used after scaling with the ground truth of the first 10 meters. Experimental results show that the method of this invention achieves the best average performance on both datasets. The average RTE is 5.01% and RRE is 2.14° / 100m on the NCLT dataset, and the average RTE is 4.60% and RRE is 1.02° / 100m on the Oxford dataset, which are 42.21% and 33.53% lower than the previous best BEV method, respectively.

[0050] Figure 10 This is a table showing the performance comparison results on the KITTI odometer dataset. Figure 10 The relative translation error (RTE), relative rotation error (RRE), and absolute trajectory error (ATE) of each method are presented on the KITTI-09 and KITTI-10 sequences, where ATE is obtained using both SE(3) and Sim(3) alignment methods. The table is labeled "Original Paper" to indicate the results reported in the original paper, and "Pretrained Model" to indicate the results obtained using pre-trained weights. The method of this invention achieves the best RTE (1.37%) and ATE performance on the loop sequence 09. While its accuracy is slightly lower than DF-VO (using stereo data and depth / optical flow supervised training) on ​​sequence 10, which contains significant elevation changes, it still exhibits excellent scale consistency, with an average ATE of 4.49m under SE(3) alignment. This performance surpasses most baseline methods using Sim(3) alignment (including truth scale correction) without any ground truth scale information. The average RTE for both sequences is 2.42%, and the RRE is 0.44° / 100m.

[0051] Figure 11 This table presents the performance comparison results on the ZJH-VO multi-scale odometry dataset. Figure 11The RTE, RRE, and ATE performance of each method were compared on test sequences in four floor environments of ZJH-VO (F-09 dense office area, F-04 conference corridor, F-01 outdoor open space, and F-00 underground parking garage). ATE also provides SE(3) and Sim(3) aligned results. The table is labeled "Partial" to indicate that the method did not complete the full sequence evaluation, "Retrained" to indicate that a new model was obtained by retraining on this dataset, "Pretrained Model" to indicate that pretrained weights were used, and "F. Model" to indicate that the base model was used to provide optical flow and depth estimation. The method of this invention achieves the best performance on average metrics, demonstrating the generalization ability across multiple scenarios, with an average RTE of 3.57%, RRE of 4.69° / 100m, ATE (SE(3)) of 2.11m, and ATE (Sim(3)) of 2.04m. Compared to BEV-ODOM, which only uses sparse pose supervision, dense BEV optical flow supervision significantly improves the network's ability to learn clear mappings of objects at different distances in BEV representations, enhances the quality of feature representations, improves pose estimation accuracy, and maintains excellent scale consistency.

[0052] Figure 12 This is a table showing the ablation experiment results from three datasets. Figure 12 The effectiveness of the core components of BEV-ODOM2 was systematically verified. The performance of RTE, RRE, and ATE with different configurations was compared on the NCLT, Oxford, and ZJH-VO datasets. RTE and RRE were aligned using SE(3), and ATE was aligned using Sim(3). Ablation variables included: Enhanced Rotation Sampling strategy, used to balance diverse motion patterns and address dataset bias; Dense Flow Supervision, pixel-level BEV optical flow field supervision constructed from pose ground truth, used for dense correspondence learning; and PV-BEV Fusion, a dual-branch fusion strategy that retains 6-DOF motion information while maintaining the advantages of BEV representation. Experimental results show that the baseline model (with all components off) achieved average RTEs of 9.95%, 8.67%, and 7.28% on the three datasets, respectively. Introducing enhanced rotational sampling significantly improved performance. After adding dense flow supervision, the RTE of all datasets improved by over 20%, with the ZJH-VO dataset showing an improvement of nearly 30%, and ATE also showing significant improvement. Finally, the addition of PV-BEV fusion achieved optimal performance, reducing the RTEs of the three datasets to 5.01%, 4.60%, and 3.57%, respectively. This demonstrates the effectiveness of the collaborative efforts of each component and that the PV branch successfully compensated for the design goal of information loss in non-primary degrees of freedom during BEV projection.

[0053] like Figure 13 As shown, the embodiment also provides a monocular visual odometry system 60 based on PV-BEV fusion and dense flow supervision, including: a data augmentation unit 61, a model building unit 62, a training unit 63, and an inference unit 64. The data augmentation unit 61 is used to acquire continuous frame images as sample data; the model building unit 62 is used to build a monocular visual odometry model, which includes: a feature extraction module for extracting multi-scale image features from continuous images and fusing them with encoded features corresponding to camera parameters to obtain PV features; a PV processing module for calculating the PV correlation volume based on the PV features in PV space and then decoding and predicting 6-DoF motion information; and a BEV projection module. The block is used to project PV features and PV-related volumes into the BEV space to obtain BEV features and projected PV-related volumes, respectively; the BEV processing module is used to calculate the BEV-related volume based on BEV features in the BEV space and then fuse it with the projected PV-related volume to obtain the fused related volume; the decoding processing module is used to decode the fused related volume to predict 3-DoF motion information and BEV optical flow field; the training unit 63 is used to construct the loss and supervise training, which includes: directly constructing the dense flow ground truth based on the 3-DoF pose ground truth, constructing the dense flow loss based on the dense flow ground truth and BEV optical flow field, constructing the 6-DoF PV pose loss based on 6-DoF motion information, constructing the 3-DoF BEV pose loss based on 3-DoF motion information, and supervising the training of the model after combining all losses and augmenting the sample data to achieve model parameter optimization; the inference unit 64 is used to perform monocular visual odometry using the parameter-optimized model.

[0054] It should be noted that the monocular visual odometry system based on PV-BEV fusion and dense flow supervision provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing monocular visual odometry. The functions described above can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the monocular visual odometry system based on PV-BEV fusion and dense flow supervision provided in the above embodiments belongs to the same concept as the monocular visual odometry method embodiment based on PV-BEV fusion and dense flow supervision. For details of its implementation process, please refer to the monocular visual odometry method embodiment based on PV-BEV fusion and dense flow supervision, which will not be repeated here.

[0055] The technical effects of the monocular visual odometry method and system provided in the embodiments include: 1. Improved monitoring density This invention improves the supervisory signal density by constructing dense BEV optical flow from the pose ground truth, increasing it from 3 scalars (sparse pose) to 32,768 scalars (128×128×2 flow vectors), an improvement of over 10,000 times. Experimental results show that: On the NCLT dataset, compared to the BEV-ODOM method which only uses pose supervision, the present invention reduces the relative translation error (RTE) by 49.7% (from 9.95% to 5.01%) and the relative rotation error (RRE) by 50.3% (from 4.31° / 100m to 2.14° / 100m).

[0056] On the Oxford dataset, this invention also demonstrates significant performance improvements, with a 46.9% reduction in relative translation error (RTE) (from 8.67% to 4.60%) and a 41.4% reduction in relative rotation error (RRE) (from 1.74° / 100m to 1.02° / 100m). The Oxford dataset is known for its complex urban environments, numerous dynamic objects, and varying lighting conditions. The experimental results validate the effectiveness and robustness of dense BEV optical flow supervision in addressing the challenges of urban scenes.

[0057] 2. Information retention and integration effect The PV-BEV dual-branch fusion architecture of this invention effectively compensates for the information loss of LSS projection. On KITTI sequence 9, which contains significant slope changes, compared with BEV-ODOM, the RTE of this invention is reduced by 11.3%, and the Absolute Trajectory Error (ATE) is reduced by 55.5%. Ablation experiments confirm that PV-BEV fusion alone contributes approximately 6.4-18.4% to the accuracy improvement (across all metrics on all datasets), with more significant effects in non-planar motion scenarios.

[0058] 3. Advantage of scale consistency This invention maintains the inherent scale consistency advantage of the BEV method. On long-distance test sequences on the NCLT and Oxford datasets, this invention, along with other BEV methods (BEV-DWPVO, BEV-ODOM), exhibits superior scale stability, with the logarithmic scale factor curve remaining close to zero and its fluctuation significantly lower than that of perspective view methods. This performance verifies the effectiveness of the implicit scale anchoring strategy implemented through BEV mesh representation, which can be achieved using only pose supervision signals without requiring additional supervision information such as depth and optical flow.

[0059] Despite the introduction of a more complex PV-BEV fusion architecture in this invention, this characteristic is successfully maintained. This is attributed to the unified processing flow used in both motion feature extraction paths: both BEV motion features and PV motion features use the same depth estimation network and BEV projection mechanism, ensuring that all motion features are anchored to a consistent metric scale defined by the BEV mesh structure, thereby maintaining scale consistency throughout the framework.

[0060] 4. No additional advantages required All supervision signals (dense flow, 5-DOF pose, 3-DOF pose) in this invention are derived from the pose ground truth, requiring no depth sensor, optical flow sensor, or manual annotation. Compared to DF-VO, which requires depth and optical flow supervision, this invention significantly reduces data acquisition costs (eliminating the need for LiDAR or binocular cameras) and avoids dependencies on base models (such as ZoeDepth) and error propagation issues.

[0061] 5. Generalization effect of motion patterns The enhanced rotation sampling strategy significantly improves the model's generalization ability to diverse motion patterns. This strategy effectively mitigates dataset bias by balancing the distribution of motion patterns in the training data, enhancing the model's ability to estimate infrequent but critical actions such as turning. Ablation experiments show that on the NCLT, Oxford, and ZJH-VO datasets, the introduction of the enhanced rotation sampling strategy improves RTE, RRE, and ATE metrics. Particularly on the ZJH-VO multi-scale scene dataset (containing various environments such as underground parking garages, outdoor plazas, corridors, and offices), this strategy, combined with dense BEV optical flow supervision, reduces RTE from 7.28% to 3.57% (a 51.0% improvement) compared to the BEV-ODOM baseline using only sparse pose supervision; RRE from 7.98° / 100m to 4.69° / 100m (a 41.2% improvement); and ATE from 3.68 to 2.04 (a 44.6% improvement).

[0062] 6. Robustness across multiple scenarios This invention achieves state-of-the-art performance on four datasets with different characteristics: NCLT (High-Frequency Lighting Change): RTE 5.01%, RRE 2.14° / 100m; Oxford (complex urban environment, dynamic objects): RTE 4.60%, RRE 1.02° / 100m; KITTI (Standard Scene): RTE 2.42%, RRE 0.44° / 100m; ZJH-VO (Multi-scale Scene): RTE 3.57%, RRE 4.69° / 100m; The average performance is improved by approximately 35.5% compared to the best baseline (BEV-DWPVO). In the ZJH-VO multi-scenario test, the absolute trajectory error (ATE) of this invention under SE(3) alignment (without scale correction) is better than most comparative methods using Sim(3) alignment (with global scale correction), demonstrating robustness across a variety of operating conditions.

[0063] 7. Industrial Application Value This invention is particularly applicable to the fields of intelligent transportation and industrial mobile robots: Low-cost deployment: Only a single-view camera sensor is required, and the hardware cost is significantly lower than that of multi-sensor fusion solutions; Simplified integration: End-to-end deep learning architecture, eliminating the need for complex feature engineering, backend optimization, or loop closure detection modules; Strong scene adaptability: It maintains robust performance in multi-scale scenes (underground parking garage, outdoor plaza, corridor, office) verified on the ZJH-VO dataset and high-jitter scenes verified on the NCLT dataset; Scale consistency: Implicit scale anchoring through BEV mesh representation maintains excellent scale stability during long-distance operation without the need for external scale correction; Scalability: Supports multi-camera configuration expansion and application expansion such as integration with other sensors (IMU, wheel speedometer).

[0064] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A monocular visual odometry method based on PV-BEV fusion and dense flow supervision, characterized in that, Includes the following steps: Collect consecutive frames of images as sample data; A monocular visual odometry model is constructed, comprising: a feature extraction module for extracting multi-scale image features from continuous images and fusing them with the encoded features corresponding to camera parameters to obtain PV features; a PV processing module for calculating the PV correlation volume based on the PV features in PV space and then decoding and predicting 6-DoF motion information; a BEV projection module for projecting the PV features and PV correlation volume onto BEV space to obtain BEV features and projected PV correlation volume, respectively; a BEV processing module for calculating the BEV correlation volume based on the BEV features in BEV space and fusing it with the projected PV correlation volume to obtain a fused correlation volume; and a decoding processing module for decoding the fused correlation volume to predict 3-DoF motion information and BEV optical flow field. The model is constructed and supervised training is performed, which includes: directly constructing the dense flow ground truth based on the 3-DoF pose ground truth, constructing the dense flow loss based on the dense flow ground truth and the BEV optical flow field, constructing the 6-DoF PV pose loss based on 6-DoF motion information, constructing the 3-DoF BEV pose loss based on 3-DoF motion information, and supervising the training of the model by combining all the losses and augmenting the sample data to achieve model parameter optimization. Monocular visual odometry is performed using a parameter-optimized model.

2. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, The PV processing module is used to decode and predict 6-DoF motion information after calculating the PV correlation volume based on PV features in the PV space, including: The PV correlation calculation submodule is used to calculate the correlation between consecutive time points. and PV characteristics and Calculate feature correlation within a local neighborhood to generate PV correlation volume. : in As coordinates, , For the search radius, For the current channel, The total number of channels for the PV feature. The PV pose decoding submodule is used to decode the PV-related volume using a convolutional neural network and fully connected layers. Dimensionality reduction and regression are performed to predict 6-DoF motion information. , For the complete rotation matrix, It is a normalized translation vector.

3. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, The BEV projection module is used to project PV features and PV-related volumes onto the BEV space to obtain BEV features and projected PV-related volumes, including: The deep-context prediction submodule is used for PV features. Predicting deep features using two convolutional networks respectively and context features ; The feature fusion submodule is used to fuse contextual features. and depth features The fused features are obtained by performing dimensional expansion and element-wise multiplication. ; The LSS projection submodule is used for projection using an LSS network, specifically for fusing features. The BEV features are generated by mapping the view cone onto a 3D voxel space, transforming it to the vehicle coordinate system, and finally accumulating the voxel features onto the BEV plane through voxel pooling. ; The PV correlation projection submodule is used to project the PV correlation volume using the exact same LSS network parameters as the feature projection. Projecting onto the BEV space, we obtain the projected PV-related volume. .

4. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, The BEV processing module is used to calculate the BEV correlation volume based on BEV features in the BEV space and then fuse it with the projected PV correlation volume to obtain the fused correlation volume, including: The BEV correlation calculation submodule is used to calculate the BEV features at consecutive projected time points. and Calculate feature correlations within a local neighborhood to generate the BEV correlation volume. : in As coordinates, , For the search radius, For the current channel, The total number of channels for BEV features. The multimodal correlation fusion submodule is used to condense the BEV correlation volume. Correlation volume with projected PV By stitching along the channel dimension, the fused correlation volume is obtained. .

5. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, The decoding processing module is used to decode the fused correlation volume to predict 3-DoF motion information and BEV optical flow field, including: The UNet encoder is used to perform multi-layer convolutional downsampling on the fusion-related volume to extract the global motion context. Dense BEV optical flow prediction branch: Reconstructing a dense two-channel BEV optical flow field using the UNet decoder; The 3-DoF pose prediction branch is used to extract features from the penultimate layer of the UNet decoder. The final 3-DoF motion information is obtained by convolutional dimensionality reduction and regression using fully connected layers.

6. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, Dense flow ground truth is directly constructed based on 3-DoF pose ground truth, including: For each pixel position (u, v) in the BEV feature map, the calculation steps are as follows: (1) Establish the homogeneous coordinates of each pixel position (u, v) in the vehicle coordinate system. : in BEV resolution (meters per pixel). The pixel coordinates of the origin of the vehicle coordinate system in the BEV image; (2) Using 3-DoF pose truth values Perform relative pose transformation: (3) Transform the coordinates Project back into pixel space and calculate the optical flow vector. : in This is a reverse mapping from vehicle coordinates to pixel coordinates.

7. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, Dense flow loss is constructed based on the dense flow ground truth and BEV optical flow field, including: in, For dense flow loss, For BEV optical flow field, For the truth value of dense flow, It is an L1 norm.

8. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, A 6-DoF PV pose loss is constructed based on 6-DoF motion information, including: in, For 6-DoF PV pose loss, and These are the predicted and true values ​​for the normalized translation direction, respectively. and These are the predicted and true values ​​of the rotation matrix, respectively. It is an L1 norm. It is the Frobenius norm. The balance coefficients, translation directions, and rotation matrices constitute 6-DoF motion information; The 3-DoF BEV pose loss is constructed based on 3-DoF motion information, including: in, For 3-DoF BEV pose loss, and These are the BEV planes along x Predicted and true values ​​of directional translation. and These are the BEV planes along y Predicted and true values ​​of directional translation. and respectively These are the predicted and true values ​​of the yaw angle. Dimensions of error used to balance translation and rotation.

9. The monocular visual odometry method based on PV-BEV fusion and dense flow supervision according to claim 1, characterized in that, Data augmentation of sample data includes: Each training sequence undergoes offline preprocessing to construct a motion-aware frame pair database, specifically including: for each frame in the sequence Establish a time window and enumerate all frame pairs within the window. Calculate relative pose Based on relative pose Extracting yaw angle difference Translational distance Based on filtering criteria Meters will classify frame pairs that meet the following conditions: if Store in a high-rotation list ,like Store in a standard rotation list For each frame Maintain the corresponding and ; During training, when a certain frame is selected... At that time, frame pairs are sampled with non-uniform probability.

10. A monocular visual odometry system based on PV-BEV fusion and dense flow supervision, characterized in that, include: The data augmentation unit is used to acquire consecutive frames of images as sample data. The model building unit, used to construct a monocular visual odometry model, includes: a feature extraction module for extracting multi-scale image features from continuous images and fusing them with the encoded features corresponding to camera parameters to obtain PV features; a PV processing module for calculating the PV correlation volume based on the PV features in PV space and then decoding and predicting 6-DoF motion information; a BEV projection module for projecting the PV features and PV correlation volume onto BEV space to obtain BEV features and projected PV correlation volume, respectively; a BEV processing module for calculating the BEV correlation volume based on the BEV features in BEV space and fusing it with the projected PV correlation volume to obtain a fused correlation volume; and a decoding processing module for decoding the fused correlation volume to predict 3-DoF motion information and BEV optical flow field. The training unit, used to construct the loss and supervise training, includes: directly constructing the dense flow ground truth based on the 3-DoF pose ground truth; constructing the dense flow loss based on the dense flow ground truth and the BEV optical flow field; constructing the 6-DoF PV pose loss based on 6-DoF motion information; constructing the 3-DoF BEV pose loss based on 3-DoF motion information; and supervising the training of the model by combining all losses and augmenting the sample data to achieve model parameter optimization. The inference unit is used for monocular visual odometry using a parameter-optimized model.

Images