Robust initialization method and system for non-overlapping multi-camera visual odometry

Abstract

The application provides a robust initialization method and system for non-overlapping multi-camera visual odometry, which comprises using a feature tracking method based on deep learning to extract and track feature points from multiple continuous images input from a camera; selecting key frames based on the feature points, calculating the pose transformation relationship between two frames and three-dimensional feature point information, and completing the initialization of a single camera; estimating the pose of each camera motion, obtaining the pose of the main body through the camera external parameters, establishing a correction vector of the camera pose estimation scale, and taking the error between the main body poses obtained by different cameras as a loss function to construct an optimization problem based on a nonlinear least squares method; deriving the optimal solution of the optimization problem, multiplying the original pose estimation scale of each camera by the correction vector to obtain the final real pose estimation scale, and realizing robust initialization. The application does not require the cameras to have overlapping fields of view, and can be flexibly applied to any camera model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a robust initialization method and system for non-overlapping multi-view visual odometry. Background Technology

[0002] The initialization process of visual odometry mainly includes two parts: first, establishing the relative coordinate transformation relationship of the local visual odometry; second, estimating the scale transformation relationship between the local pose motion coordinate system and the real world. The initialization step is extremely important in visual odometry, as its accuracy directly affects the performance of the entire system. Stable and accurate initialization provides a reliable reference for subsequent pose estimation. Currently, traditional multi-view visual odometry initialization still heavily relies on other modal sensors or specific stereo camera models, especially for estimating motion scale; multi-modal sensors or stereo camera models with large overlapping fields of view are indispensable in these cases.

[0003] However, data transmission between different modal sensors requires time alignment and data coupling, which complicates system initialization and reduces overall robustness. Overlapping field-of-view camera models acquire excessive repetitive image data, reducing the breadth of information obtained. Compared to the initialization methods described above, system initialization using only non-overlapping image data avoids dependence on other sensors by processing only single-modal data, resulting in a lower overall CPU load and significantly improved image acquisition efficiency and system robustness. However, accurately estimating the true scale of motion using only multi-view camera image data has not yet been achieved. Furthermore, the increased image processing requirements of multi-view cameras would place an excessive CPU load on traditional visual odometry systems, reducing real-time performance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, such as insufficient robustness and over-reliance on other modal sensors for scale estimation, this invention provides an initialization technique for robust non-overlapping multi-view visual odometry, achieving true scale estimation of camera motion entirely based on multi-view camera images. By employing a deep learning-based feature tracking algorithm, the overall CPU usage during system initialization is reduced, improving system real-time performance. The subject pose is estimated using camera pose and intrinsic parameters, and an optimization problem related to the true scale is established based on the subject pose error, thus achieving the estimation of the true scale of camera motion. Using only image data, this avoids dependence on multimodal data, reduces system load, and improves overall robustness.

[0005] This invention provides a robust initialization method for non-overlapping multi-view visual odometry, comprising the following steps:

[0006] Step 1: Use a deep learning-based feature tracking method to extract and track feature points from multiple consecutive frames of images input from the camera, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part;

[0007] Step 2: Based on the feature points obtained in Step 1, select key frames. If the disparity between the key frame and the current frame is large enough, calculate the pose transformation relationship and 3D feature point information between the two frames according to the epipolar constraint, realize the transformation from the camera coordinate system to the real world coordinate system, and complete the initialization of a single camera.

[0008] Step 3: Based on the initialization and feature point tracking results of each camera in Step 2, estimate the pose of each camera motion, obtain the pose of the subject through camera extrinsic parameters, establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained by different cameras as the loss function to construct an optimization problem based on nonlinear least squares method.

[0009] Step 4: Derive the Jacobian matrix of the scale correction vector in the optimization problem obtained in Step 3, and obtain the optimal solution of the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0; multiply the original pose estimation scale of each camera by the correction vector to obtain the final unified true pose estimation scale, thus achieving robust initialization.

[0010] Furthermore, the initialization of a single camera in step 2 is implemented as follows:

[0011] Reference frames are extracted from a sliding window of several consecutive frames. Epipolar geometry is used to calculate the 3D feature point information and the pose transformation relationship between the two frames by matching the feature points of the reference frame and the current frame. Then, the pose of a certain frame in the middle is obtained by the 2D-3D PnP algorithm. Finally, the pose of the remaining frames in the sliding window is estimated by the pose relationship of the above three frames, thus completing the initialization of a single camera.

[0012] Furthermore, the following processing is performed in step 3 to achieve true scale estimation of camera motion.

[0013] A scale correction vector is established to correct the scale estimated by the camera. The subject pose is estimated by using the camera's extrinsic parameters and the camera's pose estimation results. Then, an optimization problem based on nonlinear least squares is established by using the error between the subject poses estimated by multiple cameras as the loss function, and the Jacobin matrix with respect to the scale correction vector is obtained. Based on the scale correction vector when the Jacobin matrix is ​​0, the true scale of the final camera motion is obtained.

[0014] Furthermore, the deep learning-based feature tracking in step 1 is implemented as follows:

[0015] The image is processed using the SuperPoint network feature point detection method to extract feature points from the image, and the feature points are then distorted using camera intrinsic parameters. LK-optical flow tracing is then used to track feature points between adjacent images, and outlier removal is performed using RANSAC. If the number of removed feature points is less than a corresponding preset threshold, the SuperPoint network feature point detection method is used to extract feature points to replenish the number to the corresponding preset threshold.

[0016] On the other hand, the present invention also provides a robust initialization system for non-overlapping multi-view visual odometry, for implementing a robust initialization method for non-overlapping multi-view visual odometry as described above.

[0017] Moreover, it includes the following modules,

[0018] The first module is used to extract and track feature points from multiple consecutive images input from the camera based on a deep learning-based feature tracking method, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part.

[0019] The second module is used to select key frames based on the feature points obtained by the first module. If the disparity between the key frame and the current frame is large enough, the pose transformation relationship and three-dimensional feature point information between the two frames are calculated according to the epipolar constraint, so as to realize the transformation from the camera coordinate system to the real world coordinate system and complete the initialization of a single camera.

[0020] The third module is used to estimate the pose of each camera based on the initialization and feature point tracking results of each camera in the second module, obtain the pose of the subject through the camera extrinsic parameters, establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained by different cameras as the loss function to construct an optimization problem based on nonlinear least squares method.

[0021] The fourth module is used to derive the Jacobian matrix of the scale correction vector in the optimization problem obtained in the third module, and obtain the optimal solution of the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0; after multiplying the original pose estimation scale of each camera by the correction vector, the final unified true pose estimation scale is obtained, realizing robust initialization.

[0022] Alternatively, it may include a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the stored instructions in the memory to execute a robust initialization method for non-overlapping multi-view visual odometry as described above.

[0023] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a robust initialization method for non-overlapping multi-view visual odometry as described above.

[0024] This invention considers that the scale obtained during the initialization of a single camera is not uniform. Therefore, to estimate the true scale of the camera and subject motion, a scale correction term needs to be added to the pose estimation result of a single camera to unify it to the scale of the real world. The scale correction terms of multiple cameras constitute a scale correction vector, and the estimation of the scale correction vector is achieved by constructing and solving an optimization problem. Since each camera can calculate the pose change relationship with the subject through extrinsic parameters, the subject pose can be obtained from the camera pose. Theoretically, the subject pose calculated from any camera should be the same. However, due to the scale error of the camera pose, there will also be errors between the obtained subject poses. By using the error between subject poses as a loss function, an optimization problem based on nonlinear least squares can be constructed. Solving this optimization problem yields the scale correction term of the camera pose. Since this scale correction term is obtained from the camera extrinsic parameters at the true scale, multiplying the scale correction term by the camera pose will also result in the final camera pose at the true scale.

[0025] Compared with the relevant literature in the prior art, the present invention also has significant differences and progress:

[0026] 1) Patent document CN113012197A

[0027] 1. This patent focuses on binocular visual odometry and uses only a binocular camera as an image sensor, while this invention uses a multi-view camera as an image sensor, which can obtain a wider field of view.

[0028] 2. This patent primarily addresses visual odometry in dynamic traffic scenarios, improving estimation accuracy through feature point selection. In contrast, this invention addresses the unresolved scale estimation problem for non-overlapping multi-view cameras, relying entirely on image information and independent of specific camera settings (such as field-of-view overlap) and other sensors.

[0029] 3. This patent does not involve deep learning methods, while this invention utilizes deep learning methods to improve the accuracy of feature point extraction and tracking, while also transferring computing power to the GPU, reducing the system burden.

[0030] 2) Patent document CN112179373A

[0031] 1. The patent relates to the field of visual-inertial odometry, while the present invention is a visual odometry that does not rely on other sensors.

[0032] 2. This patent uses an RGB-D camera and an IMU sensor to perform scale estimation using image depth information. In contrast, this invention can perform accurate scale estimation of pose and motion using only images acquired by a multi-view camera, achieving the same or even better results than the aforementioned method without using other sensor information.

[0033] 3. This patent does not offer significant innovation in feature point extraction and tracking, while this invention utilizes deep learning methods to improve the accuracy of feature point extraction and tracking, and also transfers computing power to the GPU, reducing the system burden.

[0034] 3) Patent document CN 115830116A

[0035] 1. The patent is a monocular visual odometry, which uses only a monocular camera as an image sensor, while the present invention uses a non-overlapping multi-view camera as an image sensor, which can obtain a wider field of view.

[0036] 2. This patent only mentions how to obtain the pose estimation values ​​R and t, but cannot guarantee that R and t are the real-world scale; that is, it does not address how to recover the true scale of the pose estimation. In contrast, this invention can achieve accurate scale estimation of pose motion using only images acquired by multiple cameras.

[0037] 3. This patent does not involve knowledge related to deep learning, while this invention uses the SuperPoint deep learning algorithm to extract and track feature points. While improving accuracy, it also transfers computing power to the GPU, reducing the system burden. Attached Figure Description

[0038] Figure 1 This is an overall structural block diagram of an embodiment of the present invention;

[0039] Figure 2 This is a comparison diagram of the estimated trajectory and the Ground Truth trajectory in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the scale estimation offset value in an embodiment of the present invention. Detailed Implementation

[0041] This invention provides a robust initialization method for non-overlapping multi-view visual odometry. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Traditional visual odometry relies heavily on other sensors or stereo camera models with large overlapping fields of view for estimating the true scale of motion. This makes scale estimation difficult in image-based, non-overlapping multi-view visual odometry. Furthermore, the increased number of images processed by multi-view cameras leads to excessive CPU load on traditional visual odometry systems. Current visual odometry methods experience high CPU utilization and significantly reduced real-time performance when processing large numbers of images acquired by multi-view cameras. Therefore, this invention proposes a front-end image feature point extraction and matching method to reduce CPU burden. Current traditional multi-view visual odometry still heavily relies on IMUs, LiDAR, or overlapping field-of-view camera models for scale estimation. However, data transmission between different modal sensors requires time alignment and data coupling, which complicates system initialization and reduces overall robustness. Overlapping field-of-view camera models, on the other hand, acquire duplicate image data, reducing the breadth of information obtained. Therefore, this invention proposes a scale estimation method entirely based on multi-camera systems. The subject pose is obtained by combining the pose estimation results from multiple cameras with camera extrinsic parameters. An optimization problem based on nonlinear least squares is then established based on the subject pose error. This problem is solved by taking partial derivatives using the Jacobin matrix to obtain the true pose estimation scale. This method does not require overlapping fields of view between cameras and can be flexibly applied to any camera model.

[0043] As attached Figure 1 As shown in the figure, an embodiment of the present invention provides a robust initialization method for non-overlapping multi-view visual odometry, which includes the following steps:

[0044] Step 1: Use a deep learning-based feature tracking method to efficiently extract and track feature points from multiple consecutive frames of images input from the camera, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part.

[0045] This invention further proposes that the feature point detection part uses the SuperPoint network's feature point detection method to process the image, extracting suitable feature points, and then performing distortion correction on the feature points using camera intrinsic parameters. Subsequently, LK-optical flow tracing is used to track feature points between adjacent images, and outlier removal is performed using the RANSAC method. If the number of removed feature points is less than a corresponding preset threshold, feature points are extracted again using the SuperPoint network's feature point detection method to replenish the number to the threshold.

[0046] In this embodiment, the specific steps are as follows: Feature points are extracted from a single frame of the original image *i* input from the camera using the SuperPoint deep learning feature detection network, and distortion correction is performed on the feature points using camera intrinsic parameters. Then, all extracted feature points are sorted according to their scores, and only the top 200 feature points are selected for subsequent tracking processing. LK-optical flow is used to achieve feature tracking between the *i*-th and *i+1*-th frames, and the RANSAC method is used for outlier removal to obtain more accurate feature matching results. If the number of tracked feature points in the *i+1*-th frame after outlier removal is less than 200, feature points are extracted in the *i+1*-th frame using the SuperPoint feature point detection method to replenish the number to a threshold before feature tracking in subsequent frames.

[0047] In specific implementation, the RANSAC method is existing technology and will not be described in detail in this invention.

[0048] Step 2: Based on the feature point matching results obtained in Step 1, select a reference frame. If the disparity between the reference frame and the current frame is large enough, the pose transformation relationship and 3D feature point information between the two frames can be calculated according to the epipolar constraint, realizing the transformation from the camera coordinate system to the real-world coordinate system and completing the initialization of a single camera.

[0049] This invention further proposes the initialization of a single camera, implemented as follows:

[0050] Reference frames are extracted from a sliding window spanning several consecutive frames (e.g., 10 frames). Epipolar geometry is used to calculate the 3D feature point information and pose transformation relationship between the two frames by matching the feature points of the reference frame and the current frame. Then, the pose of a certain intermediate frame is obtained using a 2D-3D PnP algorithm. Finally, the pose of the remaining frames in the sliding window is estimated based on the pose relationship of the above three frames, completing the initialization of a single camera.

[0051] The specific implementation method is as follows: For each frame in the sliding window, calculate the average disparity with the current frame, and the average disparity p between the i-th frame and the j-th frame. ij The calculation formula is as follows:

[0052]

[0053] Where N represents the number of feature points successfully matched between two frames, and x and y represent the coordinates of the feature points in the image. This represents the coordinates of the nth feature point in the i-th frame. This represents the coordinates of the nth feature point in the j-th frame.

[0054] If the average disparity between the two frames exceeds a preset threshold, the pose transformation relationship between them is calculated using epipolar constraints (the implementation method is existing technology and will not be elaborated upon in this invention). If the solution is successful, the frame (named frame i) is selected as a candidate reference frame. If all frames i from all cameras are candidate reference frames, then frame i is selected as the final reference frame; if not all frames i from all cameras are candidate reference frames, then the frame selected as a candidate reference frame from as many cameras as possible is selected as the final reference frame. Then, based on the position of the reference frame in the sliding window, another frame j is selected, making the selected frame j, the final reference frame, and the current frame more evenly distributed in the sliding window.

[0055] Then, based on the epipolar constraints of the reference frame and the current frame, the pose transformation relationship between the two frames is calculated, and feature point triangulation is performed to obtain the pose and 3D feature point information of the current frame relative to the reference frame. Based on the matching relationship between the feature points of the selected frame j and the reference frame, the matching relationship between the feature points and 3D feature points in the selected frame j is obtained. Then, the pose of the selected frame j relative to the reference frame is obtained through a 2D-3D PnP algorithm. BundleAdjustment (BA) is used to minimize the reprojection error of the above three frames to obtain the optimized camera pose and 3D feature points. Finally, the pose of the remaining frames in the sliding window is obtained through the pose relationship of the above three frames and the timestamp of each frame, realizing the initialization of a single camera. The 2D-3D PnP algorithm and BundleAdjustment are existing technologies and will not be elaborated upon in this invention.

[0056] Step 3: Based on the initialization and feature point tracking results of each camera in Step 2, estimate the pose of each camera's motion, and obtain the subject's pose through camera extrinsic parameters. Establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained from different cameras as the loss function to construct an optimization problem based on nonlinear least squares.

[0057] This invention further proposes a method for estimating the true scale of camera motion, implemented as follows:

[0058] In step 2, during the initialization of a single camera, the scale estimation is inaccurate, requiring the establishment of a scale correction vector to correct the estimated scale. Since multiple cameras are bound together by the same subject, the estimated subject pose can be obtained using the camera's extrinsic parameters (the transformation relationship between the camera coordinate system and the subject coordinate system) and the camera's pose estimation results. Theoretically, the subject poses obtained by different cameras should be identical. Subsequently, an optimization problem based on nonlinear least squares is established using the error between the estimated subject poses of multiple cameras as the loss function. The Jacobin matrix with respect to the scale correction vector is then calculated. Based on the scale correction vector when the Jacobin matrix is ​​0, the final true scale of the camera motion is obtained.

[0059] The specific implementation is as follows: In the initialization of a single camera in step 2 above, the pose R,T of each camera for 10 consecutive frames in the sliding window can be obtained. However, the estimation of the pose scale is inaccurate, and a scale correction vector S needs to be established to correct the scale of the camera pose estimation, where S is a vector composed of the scale correction values ​​of all cameras. Wherein:

[0060]

[0061]

[0062] s={s c}

[0063] Among them, R c Let represent the set of rotation matrices representing the pose transformations of camera c over 10 consecutive frames. Let T be the rotation matrix representing the pose transformation of camera c in frame t. c This represents the set of translation matrices representing the pose transformations of camera c over 10 consecutive frames. Let s represent the translation matrix of the pose transformation of camera c in frame t. c This represents the correction value for the motion scale of camera c.

[0064] Since the i-th frame has been established as the reference frame in step 2, the pose of the reference frame relative to the subject (i.e., the camera extrinsic parameters) is set as follows:

[0065]

[0066]

[0067] in, This represents the rotation matrix representing the pose transformation of camera c in the i-th frame. Let r be the translation matrix representing the pose transformation of camera c in the i-th frame. c The rotation matrix (i.e., the extrinsic parameter of camera c) represents the transformation of camera c relative to the subject's pose.c This represents the translation matrix (i.e., the extrinsic parameters of camera c) relative to the pose transformation of the subject.

[0068] Since multiple cameras are bound together by the same subject, the estimated subject pose can be obtained using the camera's extrinsic parameters (the transformation relationship between the camera coordinate system and the subject coordinate system) and the camera's pose estimation results. That is, by using the pose transformation relationship between each frame of different cameras and the reference frame, the pose of each frame of different cameras is obtained, and then combined with the camera extrinsic parameters, the subject pose of each frame is obtained.

[0069]

[0070] in, This represents the rotation matrix of the subject's pose transformation in frame t, obtained based on camera c. This represents the displacement matrix of the subject in frame t, obtained based on the pose transformation of camera c.

[0071] In theory, the subject pose obtained from different cameras should be exactly the same. However, due to scale differences in the initialization of a single camera, it can be deduced that the subject pose from multiple cameras will not be exactly the same. Therefore, we can compare the subject poses obtained from different cameras and obtain the difference between them.

[0072]

[0073]

[0074] Where e() represents the difference between the two values ​​within the parentheses. This represents the displacement matrix of the subject in frame t, obtained based on the pose transformation of camera m. This represents the displacement matrix of the subject's pose transformation in frame t, obtained based on camera n. Let r represent the rotation matrix of the subject's pose transformation in frame t, obtained from camera m. m Let t represent the rotation matrix (i.e., the extrinsic parameter of camera m) representing the pose transformation of camera m relative to the subject. m Let s represent the translation matrix (i.e., the extrinsic parameters of camera m) relative to the pose transformation of the subject. m This represents the correction value for the camera's motion scale m. This represents the displacement matrix of the subject in frame t, obtained based on the pose transformation of camera m. Let r represent the rotation matrix of the subject's pose transformation in frame t, obtained from camera n. n Let tn represent the rotation matrix of camera n relative to the subject's pose transformation (i.e., the extrinsic parameter of camera n), and s represent the translation matrix of camera n relative to the subject's pose transformation (i.e., the extrinsic parameter of camera n). n This represents the correction value for the camera's motion scale n. Let θ represent the displacement matrix of the subject's pose transformation in frame t, obtained from camera n, where θ represents the expression related to s in the above equation. n s m Irrelevant items.

[0075] And it can be written more generally as follows:

[0076]

[0077] Where e() represents the difference between the two values ​​within the parentheses, 0 3×(m-1) 0 3×(n-m-1) 0 3×(c-n) Indicates the vector Complete up to term c, where F represents the abbreviation of the above vector.

[0078] After obtaining the error between the subject poses obtained from any two cameras, the error between the subject poses is used as the loss function. After summing their squares, an optimization problem based on nonlinear least squares method is established, as shown in the following equation:

[0079]

[0080] in, It represents the square of the magnitude of a vector.

[0081] For example, Figure 1 In the process, after cameras 1 to n are initialized, their poses are obtained respectively, and then corresponding scale correction terms s are established. 1 ~s n Then, the pose error is obtained according to the corresponding subject pose 1 to n.

[0082] Step 4: Derive the Jacobian matrix for the scale correction vector in the optimization problem of Step 3, and obtain the optimal solution to the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0. Multiplying the original pose estimation scale of each camera by the correction vector yields the final unified true pose estimation scale.

[0083] The specific approach is as follows: Expand the above optimization problem into the following formula:

[0084]

[0085] Then, by using the definition of the Jacobin matrix, the Jacobin matrix related to S in the above function can be obtained as follows:

[0086]

[0087] Among them, A t express b t express Represents ∑ t A t , Represents ∑ t b t ,

[0088] According to the general optimization problem-solving methods, when the Jacobin matrix is ​​0, the value of S is the optimal solution to the optimization problem. Since... so It is symmetric and diagonalizable, i.e., invertible. Therefore, the final scale correction vector S is:

[0089]

[0090] Since the scale correction term is optimized based on the camera extrinsic parameters, and the camera extrinsic parameters are pose data at the real scale, the camera pose scale after the scale correction term will become the real scale, thereby achieving the task of unifying the camera pose scale to the real pose scale.

[0091] This invention conducted multiple experiments on the KITTI360 dataset and verified the results using RPE(rpg_trajectory).

[0092] The _evaluation) was compared with Ground Truth. The experimental results are attached. Figure 2 As shown, its pose estimation results almost perfectly match GroundTruth's, which is sufficient to demonstrate that the proposed method achieves the expected results in pose estimation accuracy. In terms of quantitative indicators, the scale estimation offset of this invention during trajectory estimation was experimentally measured, as shown in the attached figure. Figure 3 As shown, the scale estimation offset was controlled within 0.4% throughout the process, which well demonstrates that the method of the present invention is extremely effective in estimating the true scale of motion.

[0093] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0094] In some possible embodiments, a robust initialization system for non-overlapping multi-view visual odometry is provided, comprising the following modules:

[0095] The first module is used to extract and track feature points from multiple consecutive images input from the camera based on a deep learning-based feature tracking method, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part.

[0096] The second module is used to select key frames based on the feature points obtained by the first module. If the disparity between the key frame and the current frame is large enough, the pose transformation relationship and three-dimensional feature point information between the two frames are calculated according to the epipolar constraint, so as to realize the transformation from the camera coordinate system to the real world coordinate system and complete the initialization of a single camera.

[0097] The third module is used to estimate the pose of each camera based on the initialization and feature point tracking results of each camera in the second module, obtain the pose of the subject through the camera extrinsic parameters, establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained by different cameras as the loss function to construct an optimization problem based on nonlinear least squares method.

[0098] The fourth module is used to derive the Jacobian matrix of the scale correction vector in the optimization problem obtained in the third module, and obtain the optimal solution of the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0; after multiplying the original pose estimation scale of each camera by the correction vector, the final unified true pose estimation scale is obtained, realizing robust initialization.

[0099] In some possible embodiments, a robust initialization system for non-overlapping multi-view visual odometry is provided, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a robust initialization method for non-overlapping multi-view visual odometry as described above.

[0100] In some possible embodiments, a robust initialization system for non-overlapping multi-view visual odometry is provided, including a readable storage medium storing a computer program that, when executed, implements a robust initialization method for non-overlapping multi-view visual odometry as described above.

[0101] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A robust initialization method for non-overlapping multi-view visual odometry, characterized in that, Includes the following steps: Step 1: Use a deep learning-based feature tracking method to extract and track feature points from multiple consecutive frames of images input from the camera, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part; Step 2: Based on the feature points obtained in Step 1, select key frames. If the disparity between the key frame and the current frame is large enough, calculate the pose transformation relationship and 3D feature point information between the two frames according to the epipolar constraint, realize the transformation from the camera coordinate system to the real world coordinate system, and complete the initialization of a single camera. Step 3: Based on the initialization and feature point tracking results of each camera in Step 2, estimate the pose of each camera motion, obtain the pose of the subject through camera extrinsic parameters, establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained by different cameras as the loss function to construct an optimization problem based on nonlinear least squares method. This includes performing the following processing to achieve true scale estimation of camera motion. A scale correction vector is established to correct the scale estimated by the camera. The estimated subject pose is obtained by using the camera's extrinsic parameters and the camera's pose estimation results. Then, an optimization problem based on nonlinear least squares is established by using the error between the subject poses estimated by multiple cameras as the loss function, and the Jacobin matrix with respect to the scale correction vector is obtained. Based on the scale correction vector when the Jacobin matrix is ​​0, the true scale of the final camera motion is obtained. Step 4: Derive the Jacobian matrix of the scale correction vector in the optimization problem obtained in Step 3, and obtain the optimal solution of the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0; multiply the original pose estimation scale of each camera by the correction vector to obtain the final unified true pose estimation scale, thus achieving robust initialization.

2. The robust initialization method for non-overlapping multi-view visual odometry according to claim 1, characterized in that: The initialization of a single camera in step 2 is implemented as follows: Reference frames are extracted from a sliding window of several consecutive frames. Epipolar geometry is used to calculate the 3D feature point information and the pose transformation relationship between the two frames by matching the feature points of the reference frame and the current frame. Then, the pose of a certain frame in the middle is obtained by the 2D-3D PnP algorithm. Finally, the pose of the remaining frames in the sliding window is estimated by the pose relationship of the above three frames, thus completing the initialization of a single camera.

3. The robust initialization method for non-overlapping multi-view visual odometry according to claim 1, characterized in that: The deep learning-based feature tracking in step 1 is implemented as follows: The image is processed using the SuperPoint network feature point detection method to extract feature points from the image, and the feature points are then distorted using camera intrinsic parameters. LK-optical flow tracing is then used to track feature points between adjacent images, and outlier removal is performed using RANSAC. If the number of removed feature points is less than a corresponding preset threshold, the SuperPoint network feature point detection method is used to extract feature points to replenish the number to the corresponding preset threshold.

4. A robust initialization system for non-overlapping multi-view visual odometry, characterized in that: This method is used to implement a robust initialization method for non-overlapping multi-view visual odometry as described in any one of claims 1-3.

5. The robust initialization system for non-overlapping multi-view visual odometry according to claim 4, characterized in that: Includes the following modules, The first module is used to extract and track feature points from multiple consecutive images input from the camera based on a deep learning-based feature tracking method, and input the obtained continuously tracked feature points into the multi-view visual odometry initialization part. The second module is used to select key frames based on the feature points obtained by the first module. If the disparity between the key frame and the current frame is large enough, the pose transformation relationship and three-dimensional feature point information between the two frames are calculated according to the epipolar constraint, so as to realize the transformation from the camera coordinate system to the real world coordinate system and complete the initialization of a single camera. The third module is used to estimate the pose of each camera based on the initialization and feature point tracking results of each camera in the second module, obtain the pose of the subject through the camera extrinsic parameters, establish a correction vector for the camera pose estimation scale, and use the error between the subject poses obtained by different cameras as the loss function to construct an optimization problem based on nonlinear least squares method. The fourth module is used to derive the Jacobian matrix of the scale correction vector in the optimization problem obtained in the third module, and obtain the optimal solution of the optimization problem based on the value of the scale correction vector when the Jacobian matrix is ​​0; after multiplying the original pose estimation scale of each camera by the correction vector, the final unified true pose estimation scale is obtained, realizing robust initialization.

6. The robust initialization system for non-overlapping multi-view visual odometry according to claim 4, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a robust initialization method for non-overlapping multi-view visual odometry as described in any one of claims 1-3.

7. The robust initialization system for non-overlapping multi-view visual odometry according to claim 4, characterized in that: It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements a robust initialization method for non-overlapping multi-view visual odometry as described in any one of claims 1-3.

Images