An end-to-end monocular visual odometry method integrating spatiotemporal semantic information

Abstract

The present invention discloses an end-to-end monocular visual odometry method that integrates spatiotemporal semantic information. This method collects continuous image sequence frames through a color monocular camera and constructs an end-to-end deep learning framework for multi-information fusion; it also adopts a heterogeneous training domain to set multiple data sets to share parameter fusion training, and by inputting a continuous image sequence, the end-to-end deep learning framework dynamically couples the hidden state feature vectors of the historical output to form a time-series-aware feature mapping relationship, thereby achieving interpretable feature decoupling of static background elements and dynamic entity objects in the scene. After iterative feature fusion, the system outputs sparse depth and camera motion pose that meet the geometric constraints of the scene, thereby constructing a camera trajectory estimation model with high robustness and strong generalization ability in complex environments. This method significantly improves the positioning accuracy and stability of the monocular visual odometry.

Description

Technical Field

[0001] The present invention relates to fields such as robot autonomous positioning, drone navigation and autonomous driving, and is specifically an end-to-end monocular visual odometry method that integrates spatiotemporal semantic information. Background Art

[0002] Visual Simultaneous Localization and Mapping (V-SLAM), a core technology for environmental perception in modern unmanned autonomous systems, has significant applications in areas such as robotic navigation and augmented reality. Visual odometry (VO), the front-end pose estimation module of a V-SLAM system, uses a continuous image sequence captured by a monocular camera to initialize a projection structure using geometric correlations between frames. It then estimates the camera's internal parameters through self-calibration, recovers the camera's motion, and infers its six-degree-of-freedom (6-DoF) trajectory. However, existing VO methods face two challenges in practical applications: first, the static scene assumption fails due to interference from dynamic objects; second, monocular scale ambiguity and complex illumination variations affect the robustness of pose estimation. Therefore, developing a monocular visual odometry framework that can adapt to dynamic environments and possess strong generalization capabilities has become a key technical bottleneck for improving the reliability of environmental perception in autonomous systems.

[0003] Traditional visual odometry methods are mainly divided into two categories: geometric methods based on feature points and direct methods. Both methods have significant limitations. Feature point-based methods construct a reprojection error function through feature extraction and matching using SIFT and ORB methods, and use bundle adjustment for pose optimization. However, their feature matching process has a high failure rate in dynamic scenes and environments with repetitive textures. The highly non-convex optimization problem is prone to falling into local optimality. The error accumulation caused by initial pose deviations causes trajectory drift to grow quadratically. Direct methods bypass feature extraction and directly construct pixel intensity consistency constraints. Although this improves computational efficiency, it is extremely sensitive to lighting changes. When the signal-to-noise ratio of the photometric error decreases significantly, the pose estimation will fail. Although this method has the advantage of eliminating feature extraction time, it requires the use of all information in the image, resulting in a full-image optimization problem of a much larger scale than when using feature points. Therefore, the magnitude of computational complexity brought by direct visual odometry (VO) requires GPU acceleration, making it difficult to achieve low-power real-time processing on embedded devices. Both methods have essential defects in dynamic adaptability, algorithm robustness and hardware deployment efficiency, which restrict the application reliability of visual odometry in actual complex scenarios.

[0004] On the other hand, with the development of deep learning, learning-based methods have demonstrated excellent performance in many visual tasks, including image classification (ResNet), object detection (YOLO), and depth estimation (MiDaS). However, in the field of learning-based visual odometry, existing learning-based VO models exhibit significant generalization issues, with significant performance degradation in cross-domain testing. Currently, most VO models use a "closed-set training" paradigm, meaning they are trained and tested on the same dataset. Furthermore, some multi-task learning frameworks only test their generalization capabilities on depth prediction, rather than camera pose estimation. Therefore, a more effective strategy is still needed to estimate camera pose and improve the generalization capabilities of models with large parameters. Summary of the Invention

[0005] This paper proposes an end-to-end monocular visual odometry method that integrates spatiotemporal semantic information. It realizes the dynamic fusion of geometric and semantic features through the optical flow-semantic-depth trimodal cross-attention mechanism. It aims to solve the problems of traditional methods such as the lack of object-level matching, the inability to initialize the camera pose in low-texture and high-exposure environments, and the increase of cumulative error. At the same time, it introduces monocular depth estimation information and large data volume to train the Transformer model to increase its dynamic scene adaptability and cross-domain generalization ability.

[0006] In order to achieve the above-mentioned purpose of the invention, the technical solution to be adopted by the present invention is as follows:

[0007] An end-to-end monocular visual odometry method integrating spatiotemporal semantic information includes the following steps:

[0008] (1) Collect continuous image sequences through a color monocular camera to obtain image frame sequences 、 、 … ,in Indicates the frame number of the current frame;

[0009] (2) Construct an end-to-end deep learning positioning Transformer framework, which includes: an optical flow estimation network, a multi-scale optical flow feature extraction network, an optical flow intrinsic parameter encoder, a deep semantic feature encoder, a multimodal spatiotemporal feature fusion decoder, and a camera pose prediction layer;

[0010] (3) Utilize multi-source heterogeneous datasets and self-constructed dynamic fuzzy datasets to learn course shared parameters, using an adaptive moment estimation optimizer with weight decay combined with a learning rate adjuster based on Lambda function and a learning rate segmented decay strategy to learn and train network weights;

[0011] (4) Continuous image sequence acquired by color monocular camera and Input into the constructed end-to-end deep learning positioning Transformer framework to obtain sparse depth and camera relative pose transformation information between image frames.

[0012] Specifically, the optical flow estimation network in step (2) adopts the SEA-RAFT network architecture, and realizes motion information representation by constructing a dense optical flow field estimated between adjacent frame images.

[0013] Specifically, the multi-scale optical flow feature extraction network of step (2) adopts the architecture of ResNet-50 as the feature backbone, and constructs a core feature extraction module by introducing residual jump connection and cross-level feature fusion mechanism; the multi-scale optical flow feature extraction network includes a five-level downsampling architecture, uses the camera intrinsic parameter matrix to construct a learnable parameter embedding layer, and fuses it with the optical flow feature at the channel level as input, and uses convolution to compress the spatial dimension of the optical flow field and the intrinsic parameter fusion information. Finally, a composite optical flow feature map containing shallow geometric information and deep semantic information is generated.

[0014] Specifically, the optical flow intrinsic parameter encoder in step (2) uses a deformable attention mechanism to aggregate the shallow geometric information, deep semantic information, and camera intrinsic parameter layer generated by the multi-scale optical flow feature extraction network using a three-layer deformable attention mechanism through sparsely sampled dynamic features to generate a multi-scale optical flow intrinsic parameter encoding feature vector.

[0015] Furthermore, the deep semantic feature encoder in step (2) is implemented in the following manner:

[0016] (5.1) Use the encoder of the Depth Anything v2 model to extract multi-level semantic features from the input image frame;

[0017] (5.2) Input the extracted semantic features into the multi-layer perceptron MLP for feature dimension encoding;

[0018] (5.3) Simultaneously use the depth decoder branch of Depth Anything v2 to perform scene depth estimation;

[0019] (5.4) The semantic features encoded by MLP in step (5.2) are fused with the positional encoding to generate the key-value pair features required by the attention mechanism and input into the cross-attention module of the feature fusion decoder.

[0020] Furthermore, the feature fusion decoder in step (2) implements temporal information fusion in the following manner:

[0021] (1) A three-level cascaded Transformer decoder architecture is used to achieve iterative three-time estimation of pose and depth. Each decoder layer contains a composite attention mechanism module, which is integrated in the following order: a temporal self-attention module based on the BEVFormer architecture, a deformable attention module, a standard self-attention module, and a cross-attention module;

[0022] (2) A reference point adjustment module consisting of a fully connected neural network is deployed between adjacent decoder layers. The execution of this module is as follows: based on the feature output of the previous layer, the reference point coordinates input to the next layer are dynamically updated, and the depth value and confidence weight of the corresponding reference point in the current frame are simultaneously predicted;

[0023] (3) During the reference point coordinate update process, a gradient separation operation is performed on the adjusted reference point coordinate tensor to block the gradient transfer of the coordinate parameters during the back propagation process.

[0024] Furthermore, the camera pose prediction layer of step (2) is used to predict the relative motion of the camera between two frames of images. This motion is a relative pose transformation matrix with six degrees of freedom components. Its architecture consists of a layer of 1D convolution and a parallel two-branch fully connected network structure, in which the translation prediction branch consists of a three-layer fully connected network that outputs a three-dimensional translation vector to predict the relative amount of camera translation. ,in is the 3D translation, and the other rotation prediction branch consists of a three-layer fully connected network that outputs the rotation matrix The Lie algebra parameters represent the predicted 3D rotation of the camera.

[0025] Furthermore, the step (3) utilizes a multi-source heterogeneous dataset and a self-constructed dynamic fuzzy dataset to perform course shared parameter learning, and progressively trains the model in the following manner: the heterogeneous training data are progressively divided from low to high difficulty according to the source, a multi-course learning sequence is constructed based on the scene complexity, and the training batches are sorted in order from low to high data difficulty; in each training cycle, four types of data are synchronously extracted at a sampling rate proportional to the scale of the multi-source heterogeneous dataset and the self-constructed dynamic fuzzy dataset to form a mixed batch, and input into the shared parameter Transformer framework for joint training; through the course learning mechanism with increasing difficulty, the model is gradually adapted from simple geometric scenes to complex dynamic fuzzy environments, and the generalization ability of motion estimation and the training convergence efficiency are simultaneously improved.

[0026] Furthermore, the sparse depth and camera relative pose transformation information between image frames obtained in step (4) is used to estimate the camera motion trajectory through the following iterative spatiotemporal feature fusion mechanism:

[0027] (9.1) Optical flow feature generation stage: continuous input frames and Input SEA-RAFT optical flow estimation network and output pixel-level optical flow field ;

[0028] (9.2) Multimodal feature encoding stage: Optical flow field and camera intrinsic layer Splice the input optical flow intrinsic parameter encoder to generate a multi-scale optical flow intrinsic parameter encoding feature vector ; Then, the current frame Input DepthAnything v2 model to extract multi-level semantic features simultaneously and deep feature maps , and through the multi-layer perceptron MLP and Dimension transformation is consistently weighted and summed to generate enhanced deep semantic features ;

[0029] (9.3) Cross-modal attention fusion stage: In the feature fusion decoder, 、 Hidden features with history Input, temporal information is fused through temporal self-attention, the deformable attention module realizes adaptive optical flow spatial feature aggregation, and the cross attention module realizes deep semantic information fusion;

[0030] (9.4) Temporal recursive optimization stage: the output features of the last layer of the current decoder Input the camera pose prediction layer and output the six-degree-of-freedom relative pose of the camera between the current frames; at the same time, a depth map of sparse reference points is generated through a fully connected neural network filter; and in the next time step and When As part of the input of the temporal self-attention module, it builds cross-frame feature associations. As time goes by, each time frame incorporates the potential features of the previous frame, similar to the recurrent neural network (RNN) that continuously optimizes the camera's motion trajectory in the temporal dimension.

[0031] (9.5) Through the progressive curriculum learning mechanism, the recursive feature transfer process can maintain stable convergence in complex scenarios, and finally form a trajectory optimization result with temporal chain dependence.

[0032] Compared with the prior art, the present invention has the following beneficial effects: the present invention constructs an end-to-end monocular visual odometry model based on the Transformer codec architecture, and significantly improves the generalization ability and robustness of the model through a multi-course joint training strategy of multi-source heterogeneous data sets. The present invention uses a Deformable Transformer encoder to extract multi-scale features of the optical flow, forming a pyramid feature representation, and inputs it into the decoder to fuse coarse and fine granularity information, thereby constructing an efficient model for predicting the relative motion of the camera. At the same time, in order to improve the adaptability and generalization ability in low-texture and high-exposure scenes, the decoder incorporates the semantic features and depth features provided by the deep semantic encoder through the Cross Attention mechanism, so that the feature matching is extended from a single pixel difference to an associated matching based on object-level semantics and relative depth, effectively reducing the matching error caused by brightness changes. In addition to spatial information, the present invention introduces a temporal self-attention module, which dynamically represents the motion trend of objects in the environment by fusing historical temporal features, thereby improving the perception ability in dynamic scenes. This paper uses a network decoder consisting of three layers of decoder blocks. Each layer outputs an intermediate prediction result, which serves as the input for the next layer. Through three levels of iteration, the system gradually achieves coarse-to-fine camera relative pose prediction. Compared with traditional single-shot prediction methods, this structure significantly improves the accuracy and stability of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall architecture diagram of the Transformer of the present invention;

[0034] Figure 2 is a flow chart of the application of the present invention to time series information;

[0035] Figure 3 This is a specific structural diagram of the camera pose prediction layer of the present invention. DETAILED DESCRIPTION

[0036] In order to more clearly illustrate the technical objectives, solution configuration and beneficial effects of the present invention, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0037] The embodiment of the present invention provides an end-to-end monocular visual odometry method that integrates spatiotemporal semantic information. The specific implementation process is as follows:

[0038] (1) Collect continuous image sequences through a color monocular camera to obtain image frame sequences 、 、 … ,in Indicates the frame number of the current frame.

[0039] (2) Construct an end-to-end deep learning positioning Transformer framework, which includes: optical flow estimation network, multi-scale optical flow feature extraction network, optical flow intrinsic parameter encoder, deep semantic feature encoder, multimodal spatiotemporal feature fusion decoder and camera pose prediction layer (such as Figure 3 shown).

[0040] Optical flow is a basic task in computer vision, which is used to estimate the two-dimensional motion information of each pixel between video frames. It is of great significance for downstream tasks such as camera motion prediction, three-dimensional reconstruction and action recognition. Figure 1 As shown in the figure, the present invention first inputs two frames of images into the optical flow estimation network to generate a pixel-level optical flow field. The optical flow estimation network adopts the SEA-RAFT architecture, which has the advantage of balancing efficient reasoning and high-precision results.

[0041] Subsequently, considering the intrinsic parameter differences of cameras in different data sets, the camera intrinsic parameter layer is spliced ​​with the optical flow field and inputted into the optical flow intrinsic parameter encoder together. The optical flow intrinsic parameter encoding process is as follows: the multi-scale features of the optical flow intrinsic parameter fusion are extracted through the multi-scale feature extraction network and the corresponding position codes are generated; the multi-scale features are spliced ​​and added to the position codes and inputted into the three-level attention module of the encoder. In order to optimize the use of GPU memory, the present invention introduces a deformable attention mechanism (Deformable Attention) in the cross-attention layer of the optical flow intrinsic parameter encoder and the subsequent feature fusion decoder. Each attention module contains a Deformable Self-Attention layer and a feedforward network (FFN) layer, which finally outputs the multi-scale optical flow intrinsic parameter encoding features. At the same time, the image The deep semantic feature encoder is used to extract depth and semantic information. The deep semantic feature encoding framework uses the Depth Anything v2 depth estimation model architecture (Yang L, Kang B, Huang Z, et al. Depth Anything v2[J]. Advances in Neural InformationProcessing Systems, 2024, 37: 21875-21911). It obtains semantic information based on the large visual domain model (DINOv2) and depth information based on the DPT depth estimation head. After unifying the dimensions of the semantic and depth information through a linear layer, the depth information is used as the position encoding, and the semantic information is combined as the feature to generate deep semantic encoding features.

[0042] After obtaining multi-scale optical flow intrinsic parameter encoding features and depth semantic encoding features, the present invention uses a feature fusion decoder to fuse the two features. The decoder is composed of three levels of attention modules, each of which contains temporal self-attention, deformable attention, self-attention, and cross-attention. These modules respectively realize the functions of cross-frame temporal feature fusion, adaptive aggregation of optical flow intrinsic parameter features, feature quality improvement, and incorporation of deep semantic features to assist in camera relative pose and depth estimation. Figure 2 As shown in the temporal attention module, the model and The output of two frames, and and The randomly initialized inputs of the two-frame inference model are concatenated and fed into the temporal self-attention of the feature fusion decoder; then, the output This input is fed into the next deformable attention module, which serves as the optical flow intrinsic parameter feature fusion module. Another input to this module is the multi-scale optical flow intrinsic parameter encoding feature, which aims to integrate the optical flow into the decoder. Subsequently, the inherent features are weighted by the self-attention module. Finally, cross-attention is used to integrate the deep semantic encoding features into the decoder, assisting in supervising the decoder's calculation of the relative camera pose and depth estimation. A reference point filtering module is provided between each module level to dynamically update the reference point coordinates and predict depth and confidence. Gradient blocking operations are also used to adjust the gradient propagation of the reference points to optimize the training process.

[0043] Finally, the output features of the feature fusion decoder are passed to the camera pose prediction layer for processing to predict the relative pose transformation matrix of the camera between the two frames. This prediction layer contains a 1D convolution layer and a parallel two-branch fully connected network (each branch consists of three fully connected layers), which outputs the three-dimensional translation of the camera respectively. and the three-dimensional rotation , together forming a relative pose transformation matrix with six degrees of freedom components .

[0044] During the model training process, the setting of the loss function directly affects the optimization effect of the network model parameters. The present invention designs corresponding loss functions for camera pose prediction and depth estimation tasks. Used to optimize camera translation and rotation parameters, depth loss Used to measure the accuracy of depth estimation. The total loss function of the model is defined as (where , ): ;

[0045] Since the scale of the monocular camera motion itself cannot be observed from the monocular image sequence, the loss function of the camera pose is , scale ambiguity only affects translation , rotate The loss remains unchanged, so the camera pose loss function is defined according to the TartanVO loss function : ;in, and is the camera motion prediction value, and is the true value of the camera motion, Avoid miscalculations caused by dividing by zero.

[0046] For depth prediction, the present invention uses relative depth for supervision. is to consider unknown or variable scales using scale-invariant loss in logarithmic depth space, while Multi-scale, scale-invariant gradient matching is adapted to the disparity space. The purpose of this gradient matching is to keep the discontinuities sharp and consistent with the discontinuities in the ground truth. The overall depth prediction loss function for a frame of image ,in is 0.5, ; ;in ;in, ;

[0047] At the same time, the confidence value of the area where the specific reference point is located is evaluated in combination with the confidence level, and the prediction loss function of the depth of the specific area of ​​the reference point is set using heteroscedastic arbitrary uncertainty : ;in, and are depth prediction value and true value respectively; For the The reference point set of the frame image, is the pixel position of a single reference point, is the confidence level of the corresponding reference point, where Predicted by a fully connected layer of filters.

[0048] The final depth total loss function is : .

[0049] (3) Multi-source heterogeneous datasets are used for course shared parameter learning, in which AdamW (adaptive moment estimation optimizer with weight decay) is combined with LambdaLR (learning rate regulator based on Lambda function) learning rate segmented decay strategy to learn and train network weights.

[0050] In view of the significant domain differences between different data sets, the present invention adopts a curriculum learning method to construct training batches to enhance the generalization ability of the model. Specifically, the data set is first divided into multiple parts according to the source. Then, based on the multi-course learning strategy, the difficulty of the training data is graded according to the complexity of the scene (such as static scenes vs. dynamic scenes, texture richness), and the training batches are sampled in order from low to high. In each batch, all data sets are sampled synchronously according to their scale ratio to ensure data diversity. Subsequently, the sampled data is input into the parameter-sharing model for training. Through this progressive learning method, the model can first master the characteristics of simple scenes and gradually adapt to complex scenes, thereby further improving the generalization ability and convergence effect of the model.

[0051] For the hyperparameters in the training process, this paper uses 8 heads for all attention modules and sets the number of queries to is set to 100, and the query is a learnable embedding with predicted 2D reference points. The latent feature dimension of all MLPs is set to 256. This model can be trained on a single RTX 4090 GPU. The epoch of training is 200, the batch size is 24, and the learning rate is 2×10 4 The optimizer uses a weight decay of 10 4 We used AdamW and reduced the learning rate by a factor of 0.1 at 100 and 150 epochs.

[0052] (4) Continuous image sequence acquired by color monocular camera and Input into this Transformer framework, and finally, obtain the sparse depth and camera relative pose transformation information between image frames.

[0053] The overall process of the present invention is as follows: First, the continuous images acquired by the color monocular camera are used as the initialization sequence and Input optical flow estimation network (SEA-RAFT) to estimate optical flow and generate pixel-level optical flow Then, the optical flow and camera intrinsics Splicing, input to the optical flow intrinsic parameter encoder, generate multi-scale optical flow intrinsic parameter encoding feature vector At the same time, Input to the depth estimation network (Depth Anything v2) to simultaneously extract multi-level semantic features and deep feature maps , through the multi-layer perceptron (MLP) and Perform dimension alignment and weighted summation to generate enhanced deep semantic features .

[0054] In the feature fusion decoder, for the initial time frame, since there is no output feature of the previous frame in the time dimension, a randomly initialized feature vector is used as the decoder input query. Subsequently, the optical flow intrinsic parameter encoding feature is aggregated through the Deformable Attention mechanism. and fuse deep semantic features through the Cross Attention mechanism The queries output by each decoder layer are fed into the camera pose prediction layer and the fully connected neural network filter, which output the camera relative pose and sparse depth information respectively. At the same time, the output features of the last layer of the current decoder are retained as the time series input for the next time frame.

[0055] In subsequent time frames (such as processing and The retained output features from the previous time frame are concatenated with a randomly initialized query as the input to the current decoder. The temporal self-attention module fuses temporal information, and the feature fusion and prediction steps are repeated. This temporal recursive mechanism continuously incorporates latent features from past frames, similar to a recurrent neural network, to continuously optimize the camera trajectory estimation accuracy over time.

[0056] In summary, the present invention proposes a monocular visual odometry method based on deep learning, which innovatively applies the Transformer network encoding and decoding architecture to the camera positioning task. The cross-attention module effectively fuses depth and semantic information, thereby enhancing the robustness of the model to environmental changes; at the same time, the temporal self-attention mechanism dynamically couples historical features to achieve feature decoupling of static background and dynamic objects in the scene, significantly improving the simulation capability of long-term dependencies. The model predicts the relative pose and sparse depth of the camera respectively through parallel modules, further optimizing the positioning accuracy. In the training stage, a progressive curriculum learning mechanism is adopted, combined with multi-source heterogeneous data sets, so that the model gradually adapts from simple scenes to complex environments, greatly improving the generalization ability and convergence effect. Compared with traditional monocular visual odometry, the positioning performance of the present invention in dynamic and complex scenes is better.

[0057] The above embodiments fully illustrate the core scheme and innovative benefits of the present invention through specific technical implementation paths and comparative experimental data. It should be noted that the above embodiments are only optional practical examples of the technical solution of the present invention and do not constitute a limitation on the scope of protection of the present invention. Any adaptive modifications based on the essence of the technical solution of the present invention (such as alternative implementations of the same algorithm), functional expansions (including but not limited to adding new training constraints), and technical adjustments that have equivalent replacement effects with existing technical features (such as hyperparameter modifications or using other optimizers to replace the AdamW optimizer, etc.) should fall within the scope of protection defined by the claims of the present invention. When implementing the present invention, those skilled in the art shall also regard the adjustment of non-core parameters such as data matching parameters and the specific functional form of the learning rate decay curve as a reasonable application scope of the technical solution of the present invention.

Claims

1. An end-to-end monocular visual odometry method integrating spatiotemporal semantic information, characterized in that: The following steps are involved: (1) Collect continuous image sequences through a color monocular camera to obtain image frame sequence I t , I t+1 , I t+2 …I t+n , where t represents the frame number of the current frame; (2) Construct an end-to-end deep learning positioning Transformer framework, which includes: an optical flow estimation network, a multi-scale optical flow feature extraction network, an optical flow intrinsic parameter encoder, a deep semantic feature encoder, a multimodal spatiotemporal feature fusion decoder, and a camera pose prediction layer; (3) Using multi-source heterogeneous datasets and self-constructed dynamic fuzzy datasets to learn course shared parameters, an adaptive moment estimation optimizer with weight decay is used in combination with a learning rate adjuster based on Lambda function and a learning rate segmented decay strategy to learn and train network weights; (4) The continuous image sequence I acquired by the color monocular camera t with I t+1 This is input into the constructed end-to-end deep learning positioning Transformer framework to obtain sparse depth and camera relative pose transformation information between image frames. Specifically, the camera motion trajectory is estimated through the following iterative spatiotemporal feature fusion mechanism: (4.1) Optical flow feature generation stage: continuous input frame I t with I t+1 Input SEA-RAFT optical flow estimation network, output pixel-level optical flow field F t→t+1 ; (4.2) Multimodal feature encoding stage: Optical flow field F t→t+1 The optical flow intrinsic parameter encoder is spliced ​​with the camera intrinsic parameter layer K to generate a multi-scale optical flow intrinsic parameter encoding feature vector E flow_intrin ; Then, the current frame I t Input Depth Anything v2 model and extract multi-level semantic features E simultaneously sem and deep feature map D i , and transform E through the multi-layer perceptron MLP sem With D i The dimension transformation is consistent with the weighted summation to generate enhanced deep semantic features E depth_sem ; (4.3) Cross-modal attention fusion stage: In the multimodal spatiotemporal feature fusion decoder, E flow_intrin 、E depth_sem With the historical hidden feature H i-1 Input, temporal information is fused through temporal self-attention, the deformable attention module realizes adaptive optical flow spatial feature aggregation, and the cross attention module realizes deep semantic information fusion; (4.4) Temporal recursive optimization stage: The current multimodal spatiotemporal features are fused with the decoder’s final output feature H i Input the camera pose prediction layer and output the six-degree-of-freedom relative pose of the camera between the current frames; at the same time, a depth map of sparse reference points is generated through a fully connected neural network filter; and in the next sequential step, the depth map is processed. t+1 with I t+2 When H i As part of the input to the temporal self-attention module, it builds cross-frame feature associations so that over time, each time frame incorporates the latent features of the previous frame; (4.5) Through the progressive curriculum learning mechanism, the recursive feature transfer process maintains stable convergence in complex scenarios, and finally forms a trajectory optimization result with temporal chain dependence.

2. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1 is characterized in that: The optical flow estimation network in step (2) adopts the SEA-RAFT network architecture, and realizes motion information representation by constructing a dense optical flow field estimated between adjacent frame images.

3. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1 is characterized in that: The multi-scale optical flow feature extraction network of step (2) adopts the ResNet-50 architecture as the feature backbone, and constructs a core feature extraction module by introducing residual jump connection and cross-level feature fusion mechanism; the multi-scale optical flow feature extraction network includes a five-level downsampling architecture, uses the camera intrinsic parameter matrix to construct a learnable parameter embedding layer, and fuses it with the optical flow feature at the channel level as input, and uses convolution to compress the spatial dimension of the optical flow field and the intrinsic parameter fusion information. Finally, a composite optical flow feature map containing shallow geometric information and deep semantic information is generated.

4. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1 is characterized in that: The optical flow intrinsic parameter encoder in step (2) uses a deformable attention mechanism to aggregate the shallow geometric information, deep semantic information, and camera intrinsic parameter layer generated by the multi-scale optical flow feature extraction network using a three-layer deformable attention mechanism through sparsely sampled dynamic features to encode the features. Generate multi-scale optical flow intrinsic parameter encoding feature vectors.

5. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1, characterized in that: The deep semantic feature encoder in step (2) is implemented in the following way: (5.1) Use the encoder of the Depth Anything v2 model to extract multi-level semantic features from the input image frame; (5.2) Input the extracted semantic features into the multi-layer perceptron MLP for feature dimension encoding; (5.3) Simultaneously use the depth decoder branch of Depth Anything v2 to perform scene depth estimation; (5.4) The semantic features encoded by MLP in step (5.2) are fused with the positional encoding to generate the key-value pair features required by the attention mechanism, and input into the cross-attention module of the multimodal spatiotemporal feature fusion decoder.

6. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1, characterized in that: The multimodal spatiotemporal feature fusion decoder of step (2) realizes temporal information fusion in the following manner: (1) A three-level cascaded Transformer decoder architecture is used to implement iterative three-time pose and depth estimation. Each decoder layer contains a composite attention mechanism module, which is integrated in the following order: a temporal self-attention module based on the BEVFormer architecture, a deformable attention module, a standard self-attention module, and a cross-attention module. (2) A reference point adjustment module consisting of a fully connected neural network is deployed between adjacent decoder layers. The module is specifically implemented as follows: based on the feature output of the previous layer, it dynamically updates the reference point coordinates input to the next layer and simultaneously predicts the depth value and confidence weight of the corresponding reference point in the current frame; (3) During the reference point coordinate update process, a gradient separation operation is performed on the adjusted reference point coordinate tensor to block the gradient transfer of the coordinate parameters during the back propagation process.

7. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1, characterized in that: The camera pose prediction layer of step (2) is used to predict the relative motion of the camera between two frames of images. This motion is a relative pose transformation matrix with six degrees of freedom components. Its architecture consists of a layer of 1D convolution and a parallel two-branch fully connected network structure, in which the translation prediction branch consists of a three-layer fully connected network that outputs a three-dimensional translation vector to predict the relative amount of camera translation T = (t x ,t y ,t z ),in is the 3D translation, and the other rotation prediction branch consists of a three-layer fully connected network, and the Lie algebra parameters of the output rotation matrix R∈so(3) represent the predicted 3D rotation of the camera.

8. The end-to-end monocular visual odometry method integrating spatiotemporal semantic information according to claim 1, characterized in that: The step (3) utilizes a multi-source heterogeneous dataset and a self-constructed dynamic fuzzy dataset to perform course shared parameter learning, and progressively trains the model in the following manner: progressively divides the heterogeneous training data from low to high difficulty according to the source, constructs a multi-course learning sequence based on the scene complexity, and sorts the training batches in order from low to high data difficulty; In each training cycle, various types of data are synchronously extracted at a sampling rate proportional to the scale of multi-source heterogeneous datasets and independently constructed dynamic fuzzy datasets to form mixed batches, and input into the Transformer framework with shared parameters for joint training; through a curriculum learning mechanism with increasing difficulty, the model gradually adapts from simple geometric scenes to complex dynamic fuzzy environments, while simultaneously improving the generalization ability of motion estimation and the training convergence efficiency.

Images