Self-supervised monocular visual odometry method based on optical flow guidance and dynamic mask

Abstract

The application provides a self-supervised monocular visual odometer method based on optical flow guidance and a dynamic mask, and relates to the technical field of computer vision. The method comprises the following steps: S1, constructing a double-branch pose estimation network, a depth estimation network and an optical flow estimation network; S2, inputting all images in a video image sequence into the constructed double-branch pose estimation network, the depth estimation network and the optical flow estimation network, and training the constructed double-branch pose estimation network and the depth estimation network; and S3, estimating the relative camera pose corresponding to each image in an image sequence to be estimated by using the trained double-branch pose estimation network. By using the application, the accuracy and robustness of the estimation result of the relative camera pose can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking. Background Technology

[0002] Visual odometry is a crucial component of visual simultaneous localization and mapping (SMR), aiming to recover the camera's ego motion from a continuous sequence of images. It has been widely applied in fields such as robot navigation, autonomous driving, and augmented reality. Monocular visual odometry, relying solely on a monocular camera, offers advantages such as low cost and easy deployment, thus attracting significant attention from researchers in recent years.

[0003] Visual odometry methods can be divided into traditional methods and deep learning-based methods. Traditional methods are mainly divided into feature-based methods and direct methods. Feature-based methods extract and match feature points from adjacent frames and estimate camera pose using geometric constraints, but they tend to degrade in scenes with sparse or repetitive textures. Direct methods, based on the photometric consistency assumption, directly minimize inter-frame photometric errors to solve for pose, offering high computational efficiency, but are sensitive to changes in illumination, exposure inconsistencies, and sensor noise.

[0004] With the development of deep learning technology, researchers have begun to use end-to-end networks to study visual odometry. Among them, visual odometry based on self-supervised learning (hereinafter referred to as self-supervised visual odometry) does not require pose and depth annotation. It is trained through constraints such as image reprojection and photometric consistency, which effectively reduces the cost of data acquisition and annotation and has stronger generalization performance.

[0005] However, existing self-supervised methods still have shortcomings in relative pose estimation accuracy and robustness, especially in non-static scenes such as illumination changes, dynamic objects, and occlusion. Furthermore, many methods use network backbones designed for single-frame tasks, failing to fully utilize temporal and motion information in the sequence, thus affecting the model's generalization ability and stability. Overall, how to more fully extract temporal cues under unlabeled conditions, improve adaptability to illumination changes, and effectively handle dynamic scenes remains a key problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To improve the accuracy and robustness of relative camera pose estimation results, this invention provides a self-supervised monocular vision odometry method based on optical flow guidance and dynamic masking. The technical solution is as follows:

[0007] On the one hand, a self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking is provided, including: S1. Construct a bi-branch pose estimation network, a depth estimation network, and an optical flow estimation network; S2. Input all images from the video image sequence into the constructed bi-branch pose estimation network, depth estimation network, and optical flow estimation network, and train the constructed bi-branch pose estimation network and depth estimation network; among them, During the training phase, the input image is reprojected based on the relative camera pose and brightness alignment parameters between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network. Inter-frame predicted optical flow is generated, and the photometric error map between the reprojected image and the real image is calculated. The optical flow prediction error map is calculated based on the obtained inter-frame predicted optical flow and the inter-frame optical flow pseudo-label estimated by the optical flow estimation network. Based on the relative camera pose between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network, the predicted depth of the input image is generated, and the depth consistency error mask between the predicted depth and the depth image of the input frame output by the depth estimation network is calculated. Based on statistical and morphological analysis, the photometric error map and optical flow prediction error map are analyzed to generate corresponding error masks, which are then fused with the depth consistency error mask to generate a motion region self-discovery mask. The photometric loss and optical flow prediction loss are obtained by using a dynamic region mask weighted photometric error map and an optical flow prediction error map. The constructed bi-branch pose estimation network and depth estimation network are trained based on the obtained photometric loss function and optical flow prediction loss function. S3. Use the trained bi-branch pose estimation network to estimate the relative camera pose of each frame in the image sequence to be estimated.

[0008] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described self-supervised monocular visual odometry methods based on optical flow guidance and dynamic masking.

[0009] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) In view of the current situation of insufficient motion modeling, a feature modeling strategy for inter-frame motion is adopted. The feature extraction of adjacent frame images is performed by a dual-branch pose estimation network. Among them, the wavelet attention module is used to adaptively enhance the input, which effectively improves the ability of relative pose estimation to distinguish sparse texture, repetitive texture and detail region. The dense camera coordinates generated by the camera intrinsic parameters are used together with the enhanced image for subsequent feature extraction, which alleviates the performance degradation caused by the change of camera intrinsic parameters in the generalization performance experiment of the network in cross-domain datasets. The inter-frame motion information is learned by differential fusion to improve the accuracy and stability of relative camera pose estimation.

[0010] (2) To address the issue that photometric supervision is susceptible to changes in illumination, optical flow prediction constraints are introduced as an additional supervision signal. By aligning the pixel displacement obtained from reprojection with the results of a mature optical flow network, geometric constraints that are insensitive to brightness inconsistencies are obtained, thereby accelerating convergence in the early stages of training and improving the robustness and accuracy of the algorithm in scenarios with strong illumination changes and inconsistent exposure.

[0011] (3) To address the supervision error caused by dynamic objects and occlusions violating the static scene assumption, a motion region self-discovery mask generation strategy based on multi-error statistics is proposed. Based on statistical and morphological analysis, photometric error maps and optical flow prediction error maps are analyzed to generate corresponding error masks, which are then fused with depth consistency error masks to generate motion region self-discovery masks used to mask relatively moving objects and occluded areas. This can automatically identify and mask relatively moving or occluded areas, reducing the interference of dynamic regions on training and inference.

[0012] (4) In order to improve the accuracy and robustness of the dual-branch motion modeling visual odometry network, a motion region self-discovery mask is used to weight the photometric error map and the optical flow prediction error map to obtain the photometric loss and optical flow prediction loss for supervising network training. Based on the obtained photometric loss function and optical flow prediction loss function, the constructed dual-branch pose estimation network and depth estimation network are trained. Among them, applying the dynamic mask to the photometric loss and optical flow prediction loss can effectively suppress the interference of dynamic regions on network training. The dynamic mask can effectively handle the problem of violating the static assumption caused by occlusion and moving objects, thereby improving the accuracy and robustness of the relative camera pose estimation results. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking provided by an embodiment of the present invention; Figure 2 A schematic diagram of the framework of the self-supervised monocular vision odometry method based on optical flow guidance and dynamic masking provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dual-branch pose estimation network structure provided in an embodiment of the present invention; Figures 4(a)-4(b) are schematic diagrams of the trajectories estimated on sequences 09 and 10 in the KITTI odometer dataset by the method provided in the embodiments of the present invention. Figures 5(a)-5(d) are schematic diagrams of the trajectories estimated by the method provided in the embodiments of the present invention on sequences 12, 13, 14 and 16 in the KITTI odometer dataset; Figures 6(a)-6(d) are schematic diagrams of the trajectories estimated by the method provided in the embodiments of the present invention on sequences 01, 02, 18 and 20 in the vKITTI2 odometer dataset. Detailed Implementation

[0015] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0016] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0017] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0018] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0019] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0020] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides a self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking. The processing flow of this method may include the following steps: S1. Construct a two-branch pose estimation network (DB-PoseNet), a depth estimation network (DispNet), and an optical flow estimation network (FlowNet) for self-supervised visual odometry; In this embodiment, the constructed bi-branch pose estimation network, depth estimation network, and optical flow estimation network together constitute a bi-branch motion modeling visual odometry network; wherein... Depth estimation networks are used to estimate the depth of monocular images. The high-precision depth estimation network framework Lite-Mono-S can be selected for training and fine-tuning.

[0021] The optical flow estimation network is used to estimate the pseudo-labels of optical flow between frames of the input image. You can choose the high-precision optical flow estimation network Flow Former++ as the optical flow network, freeze the network parameters, and do not perform training fine-tuning.

[0022] A two-branch pose estimation network is used to output the relative camera pose and brightness alignment parameters between each pair of input images.

[0023] like Figure 3 As shown, the dual-branch pose estimation network mainly consists of three stages: feature extraction, motion feature analysis, and parameter regression. The dual-branch pose estimation network adopts a dual-branch encoder-decoder structure. The encoder structure (including the two stages of feature extraction and motion feature analysis) is carefully designed for modeling inter-frame motion features, while the decoder structure is used to implement parameter regression. 1) In the feature extraction stage, a dual-branch network structure with shared weights is used to extract features from two adjacent frames of images independently, ensuring that the extracted features of the two frames of images are in the same feature space and are comparable.

[0024] In the feature extraction stage, to reduce the impact of illumination changes on the stability of network feature representation, this embodiment designs a wavelet attention module in each branch to adaptively enhance the input image, and its structure is as follows: Figure 3 As shown in Part (B), the wavelet attention module can transform the input image from the time domain to the frequency domain and use the attention mechanism to adaptively enhance low-frequency information, thereby guiding the bi-branch pose estimation network to better focus on key content such as texture during feature extraction, while reducing its sensitivity to brightness changes.

[0025] Specifically, the wavelet attention module first performs wavelet decomposition on the input image using Haar wavelet transform, converting it to the frequency domain space, which includes high-frequency and low-frequency features of different frequency components. Then, it generates corresponding attention weights through pooling layers, fully connected layers, and nonlinear transformation functions, and multiplies the generated attention weights with the corresponding high- and low-frequency features to achieve adaptive enhancement of different frequency domain features. Finally, it uses Haar wavelet inverse transform to convert the enhanced frequency domain features back to the spatial domain, obtaining the image features after adaptive frequency domain enhancement. This provides a more robust input for subsequent motion feature extraction.

[0026] In the feature extraction stage, to alleviate the performance degradation of the bi-branch pose estimation network due to inconsistent camera intrinsic parameters during cross-dataset testing, the camera coordinate generation module in the encoder uses the camera's intrinsic parameter matrix to transform the image's pixel coordinates into the camera coordinate system, generating a normalized dense camera coordinate system pixel grid tensor, and then combining it with the wavelet-enhanced image features. After concatenation, the data is input into a shared-weight convolutional layer and a residual convolutional layer for feature extraction; the specific steps include: Assume the camera's intrinsic parameter matrix is ​​as follows:

[0027] in, Let be the intrinsic parameter matrix of the camera. and Cameras described using pixels shaft and On-axis focal length; and Indicates the location of the principal point in the image; For each pixel coordinate in the image The coordinates can be transformed to the normalized camera coordinate system using the camera's intrinsic parameters. :

[0028] The above formula can then be used to generate a dense normalized coordinate tensor with the same size as the input image. :

[0029] in, and These represent the normalized coordinates in the camera coordinate system. and This indicates the height and width of the input image.

[0030] Will Compared with the enhanced image After concatenation along the channel dimension, the input to the subsequent feature extraction network (i.e., convolutional layers with shared weights and residual convolutional layers) is used for further feature extraction. The input to the subsequent feature encoding network is then:

[0031] in This represents the image after wavelet enhancement. This indicates splicing along the channel dimension.

[0032] 2) In the motion feature analysis stage, this embodiment uses a feature difference fusion module to fuse the two branches of image features extracted in the feature extraction stage to effectively model the motion information between adjacent frames.

[0033] The structure of the feature difference fusion module is as follows: Figure 3 As shown in section (C), firstly, the two branches of image features extracted in the input feature extraction stage are enhanced using a Convolutional Block Attention Module (CBAM); then, three branches—a concatenation branch, a difference branch, and a correlation branch—are used to model inter-frame motion information; finally, the output features of the three branches are concatenated by channels, and further feature fusion is achieved through a multi-layer convolutional network. The fused features are then fed into multiple residual convolutional layers for further feature extraction to refine the motion features. These refined motion features are used in the subsequent parameter regression stage to regress the inter-frame camera pose and inter-frame brightness alignment parameters; among them, The splicing branch directly splices the features of two adjacent frames in the channel dimension, and then extracts the fused features through a convolutional layer; The difference branch calculates the difference between two feature maps by subtracting them element by element to obtain information on inter-frame feature changes, and uses convolutional layers to perform non-linear transformations on these features. The correlation branch borrows from the cost volume construction method in optical flow estimation networks to calculate the correlation between corresponding feature points and their neighborhoods in two feature maps, so as to characterize the motion relationship of features between frames.

[0034] 3) In the parameter regression stage, this embodiment adopts a parallel dual-branch fully convolutional architecture, which is used for pose decoding and inter-frame brightness alignment parameter regression, respectively.

[0035] These two branches share the same network structure, using a series of convolutional layers and nonlinear activation functions to map high-dimensional motion features to target parameters. Specifically, the pose estimation branch regresses the camera's 6-DOF pose transformation, i.e., the 3D rotation and translation parameters; while the brightness alignment branch outputs brightness-aligned affine transformation parameters for calculating photometric consistency loss. These brightness alignment parameters effectively compensate for brightness changes caused by differences in lighting conditions between frames when subsequently calculating the photometric consistency loss of the reprojected image, thereby significantly improving the overall accuracy and robustness of pose estimation.

[0036] In this embodiment, the inputs to the bi-branch pose estimation network, depth estimation network, and optical flow estimation network are all... RGB image.

[0037] S2. Input all images from the video image sequence into the constructed bi-branch pose estimation network, depth estimation network, and optical flow estimation network, and train the constructed bi-branch pose estimation network and depth estimation network; among them, During the training phase, the input image is reprojected based on the relative camera pose and brightness alignment parameters between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network. Inter-frame predicted optical flow is generated, and the photometric error map between the reprojected image and the real image is calculated. The optical flow prediction error map is calculated based on the obtained inter-frame predicted optical flow and the inter-frame optical flow pseudo-label estimated by the optical flow estimation network. In this embodiment, for and A pair of images with relative motion acquired at each time step A two-branch pose estimation network is used to estimate the inter-frame relative camera pose. Inter-frame linear brightness alignment parameters and A depth estimation network is used to estimate the depth image of the input frame. Using an optical flow estimation network to estimate inter-frame optical flow pseudo-labels ,Will Reprojection Inter-frame predicted optical flow is generated simultaneously during this process. ; Calculate the reprojected image With real images Photometric error diagram between Inter-frame predicted optical flow And optical flow pseudo-labels Optical flow prediction error diagram Specifically, this may include the following steps: A1, in At any given moment, adjacent frames with relative motion relationships will be... Input a two-branch pose estimation network, and the two-branch pose estimation network outputs a 6-DOF relative camera pose. And brightness alignment parameters and ;in, and These represent the camera's translation and rotation, respectively, with subscripts. Indicates the sampling time of the current image. Figure 2 and Figure 3 In yes The abbreviated form; and These represent the weights and biases of the linear brightness transformation, respectively. A2, Depth Estimation Network Output Corresponding depth image ; A3, the optical flow estimation network estimates inter-frame optical flow pseudo-labels. ; A4, use brightness alignment parameters to Adjust the brightness to match Similar, obtained :

[0038] A5, for images pixels on and pixels on The following projection correspondence exists between them:

[0039] in, The camera intrinsic parameter matrix, for Image depth estimation at time step; A6, using the pixel projection correspondence described above, apply the following formula to the image. Sampling, obtaining Reprojection image of the time-lapse image :

[0040] A7, during the reprojection process, generates inter-frame predictive optical flow based on the pixel correspondence. : Assumption pixel coordinates for:

[0041] in Representing the pixel image coordinates, calculated through the above reprojection transformation. In the target image The corresponding projection position :

[0042] according to and Inter-frame predicted optical flow is obtained. :

[0043] A8, obtain the reprojected image. Afterwards, combined Loss and structural similarity metrics: photometric error map between reprojected and ground truth images :

[0044] in, Represents a measure of structural similarity. , To control the hyperparameters of the proportions of each part, express Norm; A9, Calculate inter-frame predicted optical flow and inter-frame optical flow pseudo-tags Optical flow prediction error diagram : .

[0045] Based on the relative camera pose between each pair of adjacent frames output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network, the predicted depth of the input image is generated, and the depth consistency error mask between the predicted depth and the depth image of the input frame output by the depth estimation network is calculated; specifically, this may include the following steps: In this embodiment, the output of the depth estimation network is utilized. Corresponding depth image Relative camera pose between frames predict Image depth at time Generate a deep consistency error mask : .

[0046] in, Essentially, it is the image depth estimated by a depth estimation network. Based on statistical and morphological analysis, the photometric error map and optical flow prediction error map are analyzed to generate corresponding error masks, which are then fused with the depth consistency error mask to generate a motion region self-discovery mask;

[0047] In this embodiment, the photometric error and optical flow prediction error at the corresponding positions of moving objects in the scene are usually significantly greater than those in other static areas. This characteristic can be used to perform statistical analysis on the photometric error map and optical flow prediction error map, and thereby generate a photometric error mask and an optical flow prediction error mask; specifically, this can include the following steps:

[0048] B1, for optical flow prediction error map The distribution characteristics of flow prediction errors were calculated using statistical methods:

[0049] in, The mean of the flow prediction error plot is... Represents the set of effective projected pixels. Represents the pixel coordinates of the image; Based on the distribution characteristics of the obtained flow prediction error, an adaptive threshold is determined. :

[0050]

[0051] in, This is an adaptive factor used to control the stringency of the threshold. The 95th percentile of the flow prediction error plot. This indicates taking the 95th percentile; Based on adaptive threshold Generate the initial flow prediction error motion mask. :

[0052] To eliminate noise and ensure the connectivity of the mask, a morphological closing operation is performed on the initial flow prediction error motion mask:

[0053] in, This represents the intermediate state mask after morphological transformation. express Structural elements Represents the morphological closing operation; right Perform connectivity component analysis and obtain the results. :

[0054] in, For the connected component labeling results, each connected region is assigned a unique label value; Removal area smaller than threshold The noise region is used to obtain the optical flow prediction error mask. for:

[0055] in, express Middle One connection component, Indicates the area of ​​the connecting component. This represents the minimum area of ​​the manually defined interconnecting component. Similarly, a photometric error mask based on photometric error analysis can be generated using B2. In this process, only statistical analysis methods are used, without using image morphology methods; B3 generates a motion region self-discovery mask through element-wise multiplication. :

[0056] in, This represents the depth consistency error mask. This indicates element-wise multiplication.

[0057] The photometric loss and optical flow prediction loss are obtained using a dynamically masked weighted photometric error map and an optical flow prediction error map; among them... Photometric loss Represented as:

[0058] Optical flow prediction loss Represented as:

[0059] in, This represents a photometric error diagram.

[0060] The constructed bi-branch pose estimation network and depth estimation network are trained based on the obtained photometric loss function and optical flow prediction loss function; specifically, this may include the following steps: In this embodiment, in addition to the loss function mentioned above, a deep smoothing loss function is also used. and geometric consistency constraint loss function pose flip loss Photometric loss in non-adjacent frames and pose continuity loss The network is trained. Therefore, in this embodiment, the overall loss function used is... for:

[0061]

[0062] in These represent the weighting coefficients for different loss functions.

[0063] In this embodiment, the obtained overall loss function is used to train the bi-branch pose estimation network and the depth estimation network.

[0064] It should be noted that the depth smoothing loss function and geometric consistency constraint loss function pose flip loss Photometric loss in non-adjacent frames and pose continuity loss These are all usage methods from other literature, and their relevant information is as follows: A self-supervised visual odometry algorithm jointly estimates the relative camera pose and monocular image depth, using photometric loss to supervise network training. However, photometric loss fails to provide effective supervision in low-texture scenes and regions with uniform depth, resulting in insufficient constraints on the depth estimation network. Therefore, an edge-aware smoothing loss is introduced as a depth regularization term. This loss aims to ensure that the predicted depth map is structurally consistent with its corresponding original image; that is, it should align with image edges where depth changes drastically, while remaining smooth in smooth regions. The loss is defined as follows:

[0065]

[0066] in and Representing pixels Image gradient and depth gradient, This indicates element-wise multiplication.

[0067] Self-supervised visual odometry lacks scale constraints, which can lead to scale inconsistencies in the predictions made by depth estimation networks for different image frames within the same time series. To address this issue, a scale consistency constraint is imposed on the monocular depth estimation results across consecutive frames in the sequence:

[0068]

[0069] The pose flip loss leverages the inherent properties of the camera pose transformation matrix, namely that the forward pose transformation and the backward pose transformation should be inverse transformations of each other. Specifically, the two-branch pose estimation network predicts the Euler angle representation vector of the inter-frame relative motion with 6 degrees of freedom. By converting Euler angle representation to The homogeneous transformation matrix can be used to obtain the corresponding pose transformation matrix. For a pair of completely opposite relative camera motions, their corresponding pose transformation matrices are inverse matrices. Based on this property, two consecutive frames are input into a bi-branch pose estimation network in different orders to obtain the forward and backward camera pose transformation matrices, respectively. and Then the pose flipping loss can be defined as:

[0070]

[0071] in Let be the identity matrix, here , where represents the dimension of the pose transformation matrix.

[0072] Non-adjacent frame photometric loss can provide additional supervision signals for the above scenarios by establishing inter-frame photometric consistency constraints with different time spans:

[0073] in Indicates the length of the input image sequence. Indicates the first image in the input image sequence. Frame image and the first The photometric loss calculated between frames. These are hyperparameters that are manually set.

[0074] Pose continuity loss can constrain the difference between directly estimated cross-frame poses and the chain propagation results of poses in adjacent frames, thereby suppressing trajectory drift over longer spans:

[0075] in, The first branch is directly estimated by the two-branch pose estimation network. Frame and the The inter-frame relative camera pose transformation matrix of the frame image; , is the cumulative pose obtained through chain propagation of pose transformations in adjacent frames; These are hyperparameters that are manually set.

[0076] In this embodiment, during training, all images from a batch of video image sequences are input into the bi-branch pose estimation network, depth estimation network, and optical flow estimation network. The optical flow estimation network has fixed parameters and is not trained; only the bi-branch pose estimation network and depth estimation network are trained. S3: The trained bi-branch pose estimation network is used to estimate the relative camera pose corresponding to each frame in the image sequence whose pose is to be estimated.

[0077] In this embodiment, after training is completed, the trained bi-branch pose estimation network is used to estimate the camera pose corresponding to each frame in the image sequence to be estimated. The depth estimation network and optical flow estimation network do not participate in this process.

[0078] The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking provided in this embodiment of the invention has at least the following advantages: (1) In view of the current situation of insufficient motion modeling, a feature modeling strategy for inter-frame motion is adopted. The feature extraction of adjacent frame images is performed by a dual-branch pose estimation network. Among them, the wavelet attention module is used to adaptively enhance the input, which effectively improves the ability of relative pose estimation to distinguish sparse texture, repetitive texture and detail region. The dense camera coordinates generated by the camera intrinsic parameters are used together with the enhanced image for subsequent feature extraction, which alleviates the performance degradation caused by the change of camera intrinsic parameters in the generalization performance experiment of the network in cross-domain datasets. The inter-frame motion information is learned by differential fusion to improve the accuracy and stability of relative camera pose estimation.

[0079] (2) To address the issue that photometric supervision is susceptible to changes in illumination, an optical flow prediction constraint is introduced. As an additional monitoring signal. Through Aligning the pixel displacement obtained by reprojection with the results of a mature optical flow network, we obtain geometric constraints that are insensitive to brightness inconsistencies, thereby accelerating convergence in the early stages of training and improving the robustness and accuracy of the algorithm in scenarios with strong lighting changes and inconsistent exposure.

[0080] (3) To address the supervision error caused by dynamic objects and occlusions violating the static scene assumption, a motion region self-discovery mask generation strategy based on multi-error statistics is proposed. Based on statistical and morphological analysis, photometric error maps and optical flow prediction error maps are analyzed to generate corresponding error masks, which are then fused with depth consistency error masks to generate motion region self-discovery masks used to mask relatively moving objects and occluded areas. This can automatically identify and mask relatively moving or occluded areas, reducing the interference of dynamic regions on training and inference.

[0081] (4) In order to improve the accuracy and robustness of the dual-branch motion modeling visual odometry network, a motion region self-discovery mask is used to weight the photometric error map and the optical flow prediction error map to obtain the photometric loss and optical flow prediction loss for supervising network training. Based on the obtained photometric loss function and optical flow prediction loss function, the constructed dual-branch pose estimation network and depth estimation network are trained. Among them, applying the dynamic mask to the photometric loss and optical flow prediction loss can effectively suppress the interference of dynamic regions on network training. The dynamic mask can effectively handle the problem of violating the static assumption caused by occlusion and moving objects, thereby improving the accuracy and robustness of the relative camera pose estimation results.

[0082] To verify the effectiveness of the self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking provided in this embodiment of the invention, its performance was tested using the evaluation metrics provided in the KITTI odometry dataset: (1) Relative displacement mean square error ( ): The average displacement RMSE (Root Mean Square Error) of all subsequences of length 100, 200, ..., 800 meters in a sequence, measured as a percentage, that is, the number of meters of deviation per 100 meters. The smaller the value, the better.

[0083] (2) Relative rotation mean square error ( ): The average rotation RMSE of all subsequences of length 100, 200, ..., 800 meters in a sequence, measured in deg / m, with a smaller value being better.

[0084] (3) Absolute trajectory error (ATE): The offset of the overall sequence predicted trajectory, measured in m, and the smaller the value, the better.

[0085] In this embodiment, the eight sequences 00-08 from the KITTI odometry dataset were used as the training and validation sets to train the two-branch pose estimation network, depth estimation network, and pose optimization network, and the two sequences 09-10 were used to test the performance of the method.

[0086] The KITTI odometer dataset consists of binocular images, radar points, and actual trajectories of urban road environments collected by devices such as vehicle-mounted cameras.

[0087] During implementation, steps S1-S3 are performed, including addressing photometric errors. , In the overall loss function , , , , , , During the self-discovery mask generation process of the motion region , Set to 5% of the total number of image pixels; photometric loss in non-adjacent frames and pose continuity loss. Set to 4. In these two loss functions... and Both are related to the sequence length Related, all were set to The initial learning rate of the two-branch pose estimation network is set to... Since deep estimation networks only need fine-tuning based on pre-trained network parameters, their initial learning rate is set to... Furthermore, during training, the learning rate is reduced to 0.8 times its previous value every 10 epochs. A total of 50 epochs are trained, with the batch size set to 4.

[0088] To verify the performance of the method described in this embodiment, this embodiment compares it with self-supervised monocular visual odometry methods based on deep learning from recent years. The experimental results are shown in Table 1. The trajectories generated in this embodiment are shown in Figures 4(a) and 4(b).

[0089] Table 1. Comparison of the method in this embodiment with other methods.

[0090] To verify the effectiveness of each proposed component, ablation experiments were conducted on the KITTI dataset in this embodiment. The results of the ablation experiments are shown in Table 2. "DB" indicates the use of a redesigned bi-branch pose estimation network, "FLOW" indicates the addition of optical flow prediction loss during training, and "MASK" indicates the use of a motion region self-discovery mask to shield dynamic region interference during training. It can be seen that introducing the various modules proposed in this embodiment into the basic method improves the algorithm's accuracy in estimating relative camera pose. The algorithm's performance gradually increases with the addition of each component, demonstrating the effectiveness of each component in the method of this embodiment.

[0091] Table 2 Ablation Experiment Results

[0092] In addition, this embodiment also conducted additional test experiments on some other sequences in the KITTI dataset (sequences 11-21, without real camera poses). Since these sequences did not provide real camera poses, the results of ORB-SLAM2 with loop closure detection were used as a reference. Among these sequences, four sequences (12, 13, 14, and 16) were selected for additional experiments. The results of the additional experiments are shown in Figures 5(a)-(d).

[0093] In addition, this embodiment also selected four sequences (01, 02, 18, and 20) from the vKITTI2 dataset to conduct generalization performance testing experiments on the algorithm. Additional experimental results are shown in Figures 6(a)-(d). Quantitative results of the experiments are shown in Table 3.

[0094] Table 3. Performance comparison of the method in this embodiment with other methods on the vKITTI2 dataset.

[0095] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0096] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0097] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0098] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0101] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0104] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking, characterized in that, The method includes: S1. Construct a bi-branch pose estimation network, a depth estimation network, and an optical flow estimation network; S2. Input all images from the video image sequence into the constructed bi-branch pose estimation network, depth estimation network, and optical flow estimation network, and train the constructed bi-branch pose estimation network and depth estimation network; among them, During the training phase, the input image is reprojected based on the relative camera pose and brightness alignment parameters between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network. Inter-frame predicted optical flow is generated, and the photometric error map between the reprojected image and the real image is calculated. The optical flow prediction error map is calculated based on the obtained inter-frame predicted optical flow and the inter-frame optical flow pseudo-label estimated by the optical flow estimation network. Based on the relative camera pose between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network, the predicted depth of the input image is generated, and the depth consistency error mask between the predicted depth and the depth image of the input frame output by the depth estimation network is calculated. Based on statistical and morphological analysis, photometric error maps and optical flow prediction error maps are analyzed to generate corresponding error masks. These masks are then fused with depth consistency error masks to generate motion region self-discovery masks. Specifically, the motion region self-discovery mask is generated through element-wise multiplication. : in, This represents the depth consistency error mask. This represents the optical flow prediction error diagram. This represents the photometric error mask. This indicates element-wise multiplication; The photometric loss and optical flow prediction loss are obtained by using a dynamic region mask weighted photometric error map and an optical flow prediction error map. The constructed bi-branch pose estimation network and depth estimation network are trained based on the obtained photometric loss function and optical flow prediction loss function. S3. Use the trained bi-branch pose estimation network to estimate the relative camera pose of each frame in the image sequence whose pose is to be estimated. The bi-branch pose estimation network consists of three stages: feature extraction, motion feature analysis, and parameter regression. In the feature extraction stage, a dual-branch network structure with shared weights is used to extract features from two adjacent frames of images independently, ensuring that the extracted features of the two frames of images are in the same feature space. In the motion feature analysis stage, the two branches of image features extracted in the feature extraction stage are fused by the feature difference fusion module to model the motion information between adjacent frames; In the parameter regression stage, a parallel dual-branch fully convolutional architecture is adopted, which is used for pose decoding and inter-frame brightness alignment parameter regression, respectively. Specifically, in the feature extraction stage, a dual-branch network structure with shared weights is used to independently extract features from two adjacent frames of images, ensuring that the extracted features of the two frames of images are in the same feature space. In the feature extraction stage, each branch transforms the input image to the frequency domain through a wavelet attention module, uses an attention mechanism to adaptively enhance high- and low-frequency features, and then transforms it back to the spatial domain to obtain image features that have undergone frequency domain adaptive enhancement. ; The camera coordinate generation module uses the camera intrinsic parameter matrix to convert the pixel coordinates of the image into a dense camera coordinate vector. ; Will Compared with enhanced image features After concatenating the channels, features are extracted from the convolutional layers with shared weights and the residual convolutional layers.

2. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 1, characterized in that, The camera's intrinsic parameter matrix is: in, Let be the intrinsic parameter matrix of the camera. and Cameras described using pixels shaft and On-axis focal length; and Indicates the location of the principal point in the image; The camera coordinate generation module uses the camera intrinsic parameter matrix to convert the pixel coordinates of the image into a dense camera coordinate vector. include: For each pixel coordinate in the image The coordinates are transformed to the normalized camera coordinate system using the camera's intrinsic parameters. : The above transformation formula generates a dense normalized coordinate tensor with the same size as the input image. : in, and These represent the normalized coordinates in the camera coordinate system.

3. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 1, characterized in that, The feature difference fusion module uses a convolutional unit attention module to enhance the two branches of image features extracted in the feature extraction stage. The enhanced features are then used in three branches: a concatenation branch, a difference branch, and a correlation branch, to model inter-frame motion information. The output features of the three branches are concatenated and fused. The fused features are then fed into a residual convolutional layer for further feature extraction to refine the motion features. These refined motion features are used in the parameter regression stage to regress inter-frame camera pose and inter-frame brightness alignment parameters. The splicing branch directly splices the features of two adjacent frames along the channel dimension; The difference branch calculates the difference between two feature maps by subtracting each element one by one. The correlation branch calculates the correlation between corresponding feature points and their neighborhoods in two feature maps to characterize the motion relationship of features between frames.

4. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 1, characterized in that, The process of reprojecting the input image based on the relative camera pose and brightness alignment parameters between each pair of adjacent frames output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network, generating inter-frame predicted optical flow, calculating the photometric error map between the reprojected image and the real image, and calculating the optical flow prediction error map based on the obtained inter-frame predicted optical flow and the inter-frame optical flow pseudo-label estimated by the optical flow estimation network includes: A1, in At any given moment, adjacent frames with relative motion relationships will be... Input a two-branch pose estimation network, and the two-branch pose estimation network outputs a 6-DOF relative camera pose. And brightness alignment parameters and ;in, and These represent the camera's translation and rotation, respectively, with subscripts. Indicates the sampling time of the current image; and These represent the weights and biases of the linear brightness transformation, respectively. A2, Depth Estimation Network Output Corresponding depth image ; A3, the optical flow estimation network estimates inter-frame optical flow pseudo-labels. ; A4, use brightness alignment parameters to Adjust the brightness to match Similar, obtained : A5, for images pixels on and pixels on The following projection correspondence exists between them: in, The camera intrinsic parameter matrix, for Image depth estimation at time step; A6, using the pixel projection correspondence described above, apply the following formula to the image. Sampling, obtaining Reprojection image of the time-lapse image : A7, during the reprojection process, generates inter-frame predictive optical flow based on the pixel correspondence. : Assumption pixel coordinates for: in Representing the pixel image coordinates, calculated through the above reprojection transformation. In the target image The corresponding projection position : according to and Inter-frame predicted optical flow is obtained. : A8, obtain the reprojected image. Afterwards, combined Loss and structural similarity metrics: photometric error map between reprojected and ground truth images : in, Represents a measure of structural similarity. , To control the hyperparameters of the proportions of each part, express Norm; A9, Calculate inter-frame predicted optical flow and inter-frame optical flow pseudo-tags Optical flow prediction error diagram : 。 5. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 1, characterized in that, The step of generating a predicted depth of the input image based on the relative camera pose between each pair of adjacent frame images output by the dual-branch pose estimation network and the depth image of the input frame output by the depth estimation network, and calculating the depth consistency error mask between the predicted depth and the depth image of the input frame output by the depth estimation network includes: Using the output of a depth estimation network Corresponding depth image Relative camera pose between frames predict Image depth at time Generate a deep consistency error mask : 。 6. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 1, characterized in that, The analysis of photometric error maps and optical flow prediction error maps based on statistics and morphology to generate corresponding error masks includes: B1, for optical flow prediction error map The distribution characteristics of flow prediction errors were calculated using statistical methods: in, The mean of the flow prediction error plot is... Represents the set of effective projected pixels. Represents the pixel coordinates of the image; Based on the distribution characteristics of the obtained flow prediction error, an adaptive threshold is determined. : in, This is an adaptive factor used to control the stringency of the threshold. The 95th percentile of the flow prediction error plot. ( () indicates taking the 95th percentile; Based on adaptive threshold Generate the initial flow prediction error motion mask. : Perform morphological closing operation on the motion mask of the initial flow prediction error: in, This represents the intermediate state mask after morphological transformation. express Structural elements ( ) represents the morphological closing operation; right Perform connectivity component analysis and obtain the results. : in, ; Removal area smaller than threshold The noise region is used to obtain the optical flow prediction error mask. for: in, express Middle One connection component, ( () indicates the area of ​​the connecting component. This represents the minimum area of ​​the manually defined interconnecting component. B2, using statistical methods, generates a photometric error mask based on photometric error analysis. .

7. The self-supervised monocular visual odometry method based on optical flow guidance and dynamic masking according to claim 6, characterized in that, Photometric loss Represented as: Optical flow prediction loss Represented as: in, This represents a photometric error diagram. This represents the optical flow prediction error diagram. Represents the pixel coordinates of the image.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images