Method for improving robustness of visual odometry in dynamic scene based on image block trajectory

Abstract

This invention discloses a method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories, relating to the field of visual motion estimation technology. Firstly, for each image I... t A set of image patches is formed by randomly sampling M image patches of width p. The trajectory τ of the image patch is obtained based on the image patch graph composed of all image patches and frames. k Then based on the trajectory τ k Estimate the motion label M for each block k Finally, the sports tag M will be added. k block trajectory τ k Update camera pose G t This invention samples several image patches in an image and does not perform dense optical flow estimation on the entire image, but only estimates the optical flow information of the image patches, thus reducing the inference speed; it stores information from multiple frames in the form of trajectories, and by analyzing the trajectories of image patches between frames, it can more accurately predict the motion state of image patches, and the deweighting of dynamic image patches improves the performance of visual odometry in dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of visual motion estimation technology, and specifically to a method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories. Background Technology

[0002] Visual odometry (VO) is an indispensable component of SLAM systems, aiming to provide crucial localization capabilities for applications such as robotics and autonomous driving by estimating the camera's position and orientation from visual measurements. VO is generally based on static assumptions, but real-world environments contain many dynamic objects. Because these methods ignore the influence of dynamic objects, they perform poorly in practical applications. To address this issue, current approaches primarily utilize semantic or motion information to identify dynamic object regions in videos, and then reduce the contribution weight of dynamic object regions in pose updates to improve the robustness of VO in dynamic scenes. These two methods are described below.

[0003] (1) Semantic information discrimination method

[0004] Semantic information-based discrimination methods utilize the semantic information of images to segment moving objects. Through real semantic segmentation networks or object detection networks, such as Mask R-CNN and YOLO, objects of specific semantic categories that may move around in the video, such as people and vehicles, are filtered out. However, in practice, many ordinary objects may move in a scene (e.g., a chair carried by a person). Moreover, those "special" objects, such as humans and vehicles, may not necessarily move in the video, thus limiting these semantic-based methods. Therefore, judgments should be based on motion information rather than semantic information.

[0005] (2) Motion information discrimination method

[0006] Current motion information discrimination methods utilize motion information between consecutive frames in a video for motion segmentation. This approach typically requires estimating global optical flow using a learning-based dense optical flow estimation method, then estimating the initial camera pose to calculate the optical flow information generated by camera motion. The difference between these two estimates yields the optical flow caused by the moving object, which is then used to determine the motion state of each pixel in the image. However, because this method only considers motion information between two frames, it has a very low tolerance for errors in optical flow estimation. Furthermore, this method generally requires multiple iterations, inevitably increasing the computational resources required exponentially. Summary of the Invention

[0007] To address the aforementioned technical problems of high computational cost and low tolerance for optical flow estimation errors, this application proposes a method to improve the robustness of visual odometry in dynamic scenes based on image patch trajectories, building upon the motion information discrimination method. This method samples several image patches from the image, estimating only the optical flow information of the image patches instead of performing dense optical flow estimation on the entire image, thus reducing inference speed. Furthermore, it stores information from multiple frames in the form of trajectories. By analyzing the trajectories of image patches between frames, the motion state of the image patches can be predicted more accurately. The deweighting of dynamic image patches further improves the performance of visual odometry in dynamic environments.

[0008] The technical solution of the present invention is as follows.

[0009] A method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories includes:

[0010] The image patch trajectory estimation step extracts image features from each video frame in the input image sequence, obtains image patches for each video frame using random sampling, and extracts image patch features; an image patch-graph representation is used to represent the relationship between image patches and frames. For frame I... i Each image block P k Calculate P for each edge (k, j) of the image patch graph. k In frame I j Projection block P on kj Using projection block P kj Center pixel coordinates Use bilinear interpolation to index frame I. j Zhongyu Image features of regions within a preset distance are used to calculate the correlation tensor; the correlation tensor is input into the optical flow estimation network, which outputs confidence and optical flow correction; the optical flow correction is used to correct the projection block P. kj The center pixel coordinates are used to obtain the corrected image block P. k In all with frame I i Frame I of adjacent edges j Center pixel coordinates of the projected block Combine the center pixel coordinates of the projection blocks to form the image block trajectory. Finally, trajectory data of k image blocks are obtained; where i∈[0,N), j∈[0,N), i≠j, N represents the number of video frame images; k∈[0,N×M), M represents the random sampling width;

[0011] The image patch trajectory motion estimation step is based on obtaining the image patch trajectory τ. k For each image block P k Assign a dynamic label M k ;

[0012] The pose update step is based on each image block P.k Dynamic tag M k The confidence level is updated; based on the updated confidence level, the beam adjustment (BA) technique is used in conjunction with the loss function to optimize the camera pose and the depth values ​​of the image patch.

[0013] Preferably, the image features of each video frame in the input image sequence are extracted, and image patches of each video frame are obtained by random sampling, and the image patch features are extracted, specifically including:

[0014] Obtain the input image sequence Use a feature encoder to extract image features. Image patch coordinates are obtained through random sampling. Based on these coordinates, bilinear interpolation is used to apply the image features f output by the feature encoder. θ (I t Extracting image patch features in, H represents a real number tensor, H×W×D represents the tensor size, H represents the image height, W represents the image width, and D represents the number of channels.

[0015] Preferred methods for calculating correlation tensors include:

[0016] A patch-graph is used to represent the relationship between patch and frame. The edge (k, j) ∈ ε between patch and frame represents patch P. k Source frame I i and frame I j There are overlapping regions; construct the correlation tensor C on each edge. kj To represent image block P k and frame I j The similarity; ε represents the set of all edges;

[0017] Specifically, in each iteration, the current pose and depth are used to search for the corresponding relationship. For frame I... i Each image block Calculate P for each edge (k, j) ∈ ε in the image patch-graph. k In frame I j Projection block P on kj ,as follows:

[0018]

[0019] Among them, Π c This represents a camera model that projects 3D coordinate points onto the image plane. G represents the inverse projection function that maps the pixel coordinates and inverse depth of an image plane to 3D coordinate points; j This represents the camera pose in the j-th frame; G represents the inverse of the camera pose in the i-th frame;ij Indicates the relative pose between frame i and frame j; d k Represents image block P k The depth; p represents the composite mapping; p represents the image I t Width of randomly sampled image blocks;

[0020] Using projection block P kj Center pixel coordinates Use bilinear interpolation to index frame I. j Zhongyu Image features of regions with a distance not exceeding r are used to calculate the correlation tensor C. kj ,as follows:

[0021]

[0022]

[0023] Where dx represents all coordinates in a neighborhood centered at zero; ||dx||1 represents the L1 norm of dx; r represents the radius of this neighborhood; ||dx||1≤r means that the value of dx is within the range where the L1 norm is not greater than r.

[0024] Preferably, the image block trajectory τ k The generation methods specifically include:

[0025] The acquired correlation tensor is fed into a GRU that includes an optical flow estimation network, and then passes through a fully connected layer to output the confidence score. and optical flow correction

[0026] Using optical flow correction r kj Correction projection block P kj The center pixel coordinates are as follows:

[0027]

[0028] Obtain the corrected image patch P k In all with frame I i Frame I of adjacent edges j Center pixel coordinates of the projected block Combined into a trajectory Finally, we obtain the trajectory data of k image blocks with shape (N, 2).

[0029] Preferably, the image patch trajectory motion estimation step specifically includes:

[0030] The Transformer model is used to process each block trajectory τ. kFeature encoding is performed, the trajectory length is N, and the shape of the encoded feature is (N, C), where C represents the number of feature channels after encoding by the Transformer model;

[0031] Max pooling is performed in the time dimension to obtain the feature vector of the entire trajectory with shape (1, C), so that the trajectory of image blocks of different trajectory lengths can be represented by a feature of one shape;

[0032] The binary mask is obtained after using an OANet-based decoder, and then applied to each image patch P. k Assign a dynamic label M k The dynamic tag M k Used to identify whether a trajectory is dynamic or static.

[0033] Preferably, the pose update step specifically includes:

[0034] Based on each image block P k Dynamic tag M k The previously estimated confidence level is then updated to have a certain dynamic representation, as follows:

[0035]

[0036] Where σ represents the dynamic weight residual control parameter, and its value range is [0, +∞);

[0037] Beam adjustment (BA) is used to increase the loss function for updating the pose G and image patch depth d on the image patch. The loss function E(G′, d′) is defined as follows:

[0038]

[0039] Among them, ||·|| ∑ The distance to Maharanobis is represented by a weighted confidence level that incorporates the motion labels. Weight the error term; G′ ij d′ represents the relative pose between the i-th and j-th frames after the update. k This represents the depth of the updated image patch k.

[0040] Preferably, the loss function is decomposed using the Schur complement method, and the gradient is backpropagated through Gauss-Newton iteration to optimize the camera pose and the depth value of the image patch.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The method of the present invention does not target every pixel on the image, but only operates on the image block to estimate the optical flow information of the image block under multiple frames of images, thereby reducing the inference speed and memory usage;

[0043] (2) The method of the present invention uses the optical flow information of image blocks in multiple frames to form a trajectory as the basis for motion state estimation, which makes full use of the time factor, making the motion state estimation more reasonable and greatly improving the error tolerance of optical flow estimation.

[0044] (3) The method of the present invention uses the Transformer model to encode the features of image block trajectories, which solves the problem that it is difficult to extract features when the image block trajectories are of different lengths. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram illustrating the implementation framework of a method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories, as described in an embodiment of the invention.

[0047] Figure 2 This is a flowchart of a method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories, according to an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of image blocks from an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of image block trajectory motion estimation according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of pose update according to an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] It is essential to effectively improve the robustness of visual odometry in dynamic scenes using fewer resources. This invention includes the following: (1) using a neural network to extract features of image patches and estimate their optical flow in other video frames; (2) concatenating the estimated optical flow into the form of trajectories; (3) using a Transformer model to estimate the motion state of each image patch trajectory; and (4) reducing the contribution weight of dynamic image patches in pose estimation to improve the robustness of visual odometry in dynamic scenes.

[0053] The main framework of this invention is as follows Figure 1 As shown.

[0054] Figure 1 In an ordered sequence of images The operation is performed on (N represents N images) to maintain the camera pose of each keyframe. With each image block P k depth As new frames are added and processed, these variables are continuously updated iteratively. The framework consists of three parts, first for each image I... t A set of image patches is formed by randomly sampling M image patches of width p. Image patch composed of all image patches and frames - diagram (see figure) Figure 3 As shown, the trajectory τ of the block is obtained by using it as input for image block trajectory estimation. k Then the trajectory τ k As input for image patch trajectory motion estimation, estimate the motion label M for each patch. k Finally, the sports tag M will be added. k block trajectory τ k Input pose estimation and update camera pose G t .

[0055] like Figure 2 As shown, a method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories includes:

[0056] Image patch trajectory estimation step S1, for each image I t A set of image patches is formed by randomly sampling M image patches of width p. The trajectory τ of an image patch is obtained from an image patch-graph composed of all image patches and frames. k ;

[0057] Image patch trajectory motion estimation step S2, based on obtaining the image patch trajectory τ k For each image block P k Assign a dynamic label M k ;

[0058] Pose update step S3, based on each image block Pk Dynamic tag M k The confidence level is updated; based on the updated confidence level, the beam adjustment (BA) technique is used in conjunction with the loss function to optimize the camera pose and the depth values ​​of the image patch.

[0059] Specifically, the image patch trajectory estimation step S1 can be implemented by the image patch trajectory estimation module, which extracts image patches from the image sequence and uses a neural network to estimate the trajectories of these image patches on other frames in the image sequence, in order to prepare for subsequent trajectory motion estimation and pose optimization. This includes the following three steps.

[0060] S11, Image patch feature extraction.

[0061] Input image sequence Use a feature encoder to extract multi-scale features. Image patch coordinates are obtained through random sampling. Based on these coordinates, bilinear interpolation is used to apply the image features f output by the feature encoder. θ (I t Extracting image patch features in, H represents a real number tensor, H×W×D represents the tensor size, H represents the image height, W represents the image width, and D represents the number of channels.

[0062] S12, Correlation Tensor Calculation.

[0063] like Figure 3 As shown, a patch-graph is used to represent the relationship between image patches and frames. For example, edge (k, j) ∈ ε represents image patch P. k Source frame I i and frame I j There are overlapping regions. This is addressed by constructing the correlation tensor C on each edge. kj To represent image block P k and frame I j The similarity. Here, ε represents the set of all edges. For example, if frames 1 and 2 have a large overlap, then image blocks 1, 2, and 3 sampled from frame 1 will all have an edge with frame 2, namely (1, 2), (2, 2), and (3, 2), respectively.

[0064] First, in each iteration, the current pose and depth are used to search for the corresponding relationship. For frame I... i Each image block P can be computed for each edge (k, j) ∈ ε in the image patch-graph. k In frame I j Projection block P on kj :

[0065]

[0066] Among them, Π c This represents a camera model that projects 3D coordinate points onto the image plane. G represents the inverse projection function that maps the pixel coordinates and inverse depth of an image plane to 3D coordinate points; j This represents the camera pose in the j-th frame; G represents the inverse of the camera pose in the i-th frame; ij Indicates the relative pose between frame i and frame j; d k Represents image block P k The depth; p represents the composite mapping; p represents the image I t Width of randomly sampled image blocks.

[0067] Using projection block P kj Center pixel coordinates Use bilinear interpolation to index frame I. j Zhongyu Image features of regions with a distance not exceeding r are used to calculate the correlation tensor C. kj :

[0068]

[0069]

[0070] Where dx represents all coordinates in a neighborhood centered at zero; ||dx||1 represents the L1 norm of dx; r represents the radius of the L1 norm of this neighborhood, which is an adjustable hyperparameter, and in this embodiment it can be taken as 3; ||dx||1≤r means that the value of dx is within the range where the L1 norm is not greater than r.

[0071] S13, Image patch trajectory generation.

[0072] The acquired relevance tensor and context features obtained by the context encoder are fed into the optical flow estimation network in the GRU, and then the results are output as confidence scores after passing through a fully connected layer. and optical flow correction Then the optical flow correction factor r can be used. kj Correction projection block P kj The center pixel coordinates are as follows:

[0073]

[0074] Obtain the corrected image patch P k In all with frame I i Frame I of adjacent edges j Center pixel coordinates of the projected block Combined into a trajectory Finally, k image patch trajectory data with shape (N, 2) are obtained. Here, 2 in (N, 2) represents the XY coordinate system, that is, the final result is a combination of N (X, Y) coordinates.

[0075] Specifically, the image patch trajectory motion estimation step S2 can be implemented through the image patch trajectory motion estimation module, that is, by using the obtained image patch trajectory τ k To estimate image patch P k The motion state, its overall process is as follows: Figure 4 As shown.

[0076] The overall process can be divided into three steps:

[0077] (1) Firstly, since the length of the extracted trajectory varies each time, it is difficult to perform feature extraction using conventional convolutional neural networks. Therefore, this invention uses the Transformer model to extract features from each block trajectory τ. k Feature encoding is performed, and the shape of the encoded feature obtained with a trajectory length of N is (N, C), where C represents the number of feature channels.

[0078] (2) Then perform max pooling operation in the time dimension to obtain the feature vector of the entire trajectory with shape (1, C). In this way, the trajectory of image blocks with different trajectory lengths can be represented by a feature of one shape.

[0079] (3) Finally, a binary mask is obtained through an OANet-based decoder, thus assigning a dynamic label M to each image block Pk. k . Figure 3 In the diagram, red represents tracks with dynamic labels, while blue represents tracks with static labels. Specifically, the binary mask is either 0 or 1, with an output of 0 indicating static and 1 indicating dynamic.

[0080] Specifically, such as Figure 5 As shown, pose update step S3 can update the pose and depth of the image block by combining the optical flow correction amount and confidence level output by the GRU and the dynamic label output by the image block trajectory motion estimation module. The confidence level output by the GRU only represents the reliability of the estimated optical flow correction amount and does not well represent the dynamic and static aspects of the optical flow. Based on each image block P... k Dynamic tag M k The previously estimated confidence level is updated to have a certain dynamic representation:

[0081]

[0082] Where σ represents the dynamic weight residual control parameter, and its value range is [0, +∞).

[0083] Differentiable beamforming (DBA) is used to update the pose G and image patch depth d on the patch image. The loss function is defined as follows:

[0084]

[0085] Among them, ||·|| ∑ The distance to Maharanobis is represented by a weighted confidence level that incorporates the motion labels. Weight the error term; G′ ij d′ represents the relative pose between the i-th and j-th frames after the update. k This represents the depth of the updated image patch k.

[0086] Differentiable bundle adjustment using a loss function effectively improves the accuracy of BA (Bundle Adjustment, a pose optimization step in SLAM) in dynamic scenes. The loss function employs the Schur complement method for efficient decomposition and optimizes camera pose and image patch depth values ​​through Gaussian-Newton iterative backpropagation of gradients.

[0087] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories, characterized in that, include: The image patch trajectory estimation step extracts image features from each video frame in the input image sequence, obtains image patches for each video frame using random sampling, and extracts image patch features; an image patch-graph representation is used to represent the relationship between image patches and frames. For frame I... i Each image block P k Calculate P for each edge (k, j) of the image patch graph. k In frame I j Projection block P on kj Using projection block P kj Center pixel coordinates Use bilinear interpolation to index frame I. j Zhongyu Image features of regions within a preset distance are used to calculate the correlation tensor; the correlation tensor is input into the optical flow estimation network, which outputs confidence and optical flow correction; the optical flow correction is used to correct the projection block P. kj The center pixel coordinates are used to obtain the corrected image block P. k In all with frame I i Frame I of adjacent edges j Center pixel coordinates of the projected block Combine the center pixel coordinates of the projection blocks to form the image block trajectory. Finally, trajectory data of k image blocks are obtained; where i∈[0,N), j∈[0,N), i≠j, N represents the number of video frame images; k∈[0,N×M), M represents the random sampling width; a step of estimating the trajectory of the image block based on the obtained trajectory of the image block τ k for each image block P k a dynamic label M is assigned k ; a pose updating step, based on each image patch P k a dynamic tag M k updating the confidence; based on the updated confidence, using bundle adjustment BA techniques, combined with a loss function, to optimize the camera poses and the depth values of the image patches.

2. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 1, characterized in that, Extracting image features from each video frame in the input image sequence, obtaining image patches from each video frame using random sampling, and extracting image patch features, specifically including: Obtain the input image sequence Use a feature encoder to extract image features. Image patch coordinates are obtained through random sampling. Based on these coordinates, bilinear interpolation is used to apply the image features f output by the feature encoder. θ (I t Extracting image patch features in, H represents a real number tensor, H×W×D represents the tensor size, H represents the image height, W represents the image width, and D represents the number of channels.

3. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 2, characterized in that, Methods for calculating correlation tensors include: A patch-graph is used to represent the relationship between patch and frame. The edge (k, j) ∈ ε between patch and frame represents patch P. k Source frame I i and frame I j There are overlapping regions; construct the correlation tensor C on each edge. kj To represent image block P k and frame I j The similarity; ε represents the set of all edges; Specifically, in each iteration, the current pose and depth are used to search for the corresponding relationship. For frame I... i Each image block Calculate P for each edge (k, j) ∈ ε in the image patch-graph. k In frame I j Projection block P on kj ,as follows: Among them, Π c This represents a camera model that projects 3D coordinate points onto the image plane. G represents the inverse projection function that maps the pixel coordinates and inverse depth of an image plane to 3D coordinate points; j This represents the camera pose in the j-th frame; G represents the inverse of the camera pose in the i-th frame; ij Indicates the relative pose between frame i and frame j; d k Represents image block P k The depth; p represents the composite mapping; p represents the image I t Width of randomly sampled image blocks; Using projection block P kj Center pixel coordinates Use bilinear interpolation to index frame I. j Zhongyu Image features of regions with a distance not exceeding r are used to calculate the correlation tensor C. kj ,as follows: Where dx represents all coordinates in a neighborhood centered at zero; ||dx||1 represents the L1 norm of dx; r represents the radius of this neighborhood; ||dx||1≤r means that the value of dx is within the range where the L1 norm is not greater than r.

4. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 3, characterized in that, Image Patch Trajectory τ k The generation methods specifically include: The acquired correlation tensor is fed into a GRU that includes an optical flow estimation network, and then passes through a fully connected layer to output the confidence score. and optical flow correction Using optical flow correction r kj Correction projection block P kj The center pixel coordinates are as follows: Obtain the corrected image patch P k In all with frame I i Frame I of adjacent edges j Center pixel coordinates of the projected block Combined into a trajectory Finally, we obtain the trajectory data of k image blocks with shape (N, 2).

5. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 4, characterized in that, The image patch trajectory motion estimation step specifically includes: The Transformer model is used to process each block trajectory τ. k Feature encoding is performed, the trajectory length is N, and the shape of the encoded feature is (N, C), where C represents the number of feature channels after encoding by the Transformer model; Max pooling is performed in the time dimension to obtain the feature vector of the entire trajectory with shape (1, C), so that the trajectory of image blocks of different trajectory lengths can be represented by a feature of one shape. The binary mask is obtained after using an OANet-based decoder, and then applied to each image patch P. k Assign a dynamic label M k The dynamic tag M k Used to identify whether a trajectory is dynamic or static.

6. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 5, characterized in that, The pose update step specifically includes: Based on each image block P k Dynamic tag M k The previously estimated confidence level is then updated to have a certain dynamic representation, as follows: Where σ represents the dynamic weight residual control parameter, and its value range is [0, +∞); Beam adjustment (BA) is used to increase the loss function for updating the pose G and image patch depth d on the image patch. The loss function E(G′, d′) is defined as follows: Among them, ||·|| ∑ The distance to Maharanobis is represented by a weighted confidence level that incorporates the motion labels. Weight the error term; G′ ij d′ represents the relative pose between the i-th and j-th frames after the update. k This represents the depth of the updated image patch k.

7. The method for improving the robustness of visual odometry in dynamic scenes based on image patch trajectories according to claim 6, characterized in that, The loss function is decomposed using the Schur complement method, and the gradient is backpropagated through Gauss-Newton iterations to optimize the camera pose and the depth value of the image patch.

Images