A three-dimensional tracking method based on direct optimization of depth data

Abstract

The application relates to a three-dimensional tracking method based on direct optimization of depth data and belongs to the technical field of three-dimensional tracking. The three-dimensional tracking method comprises the following steps: (1) reading in a camera parameter configuration file; (2) initializing a camera intrinsic parameter matrix; (3) giving an initial pose, a three-dimensional model and a depth map; (4) estimating the pose of a next frame through a pose perturbation model based on depth data; and (5) repeating step (4) to continue tracking. The application introduces a perturbation model for depth data, does not need to find the corresponding relationship of three-dimensional points, and improves tracking accuracy.

Description

Technical Field

[0001] This invention relates to a 3D tracking method based on direct optimization of depth data, belonging to the field of 3D tracking technology. Background Technology

[0002] Predicting object pose in 3D space is a fundamental task in computer vision, with wide applications in augmented reality, robotics, and medicine. The goal of 3D tracking is to predict camera pose from a continuous sequence of image frames and an object model. Traditional 3D tracking methods include region-based methods, which assume different statistical probability distributions for the object and background regions and model them based on color information, aiming to find the optimal segmentation contour. However, these methods are prone to failure when the background and object have high similarity. Feature-point-based methods match salient features in the 2D image, such as gradient-based features, but this requires the object to have good texture to display sufficiently strong local features in the 2D image. Edge-based methods match the object model edges with edges in the image using gradient information from the 2D image, but are susceptible to motion blur and cluttered backgrounds. Furthermore, deep learning-based methods are also used for 3D tracking, but due to the need for large amounts of training data and lower accuracy, they often fall short of traditional methods.

[0003] With the advent of low-cost, high-reliability RGB-D vision sensors, depth-based methods have begun to be used for 3D tracking. Depth-based methods use depth information to try to minimize the distance between the 3D model surface and the depth camera measurement, and usually use the Iterative Closest Point (ICP) framework. In this case, the selection strategy of the 3D point correspondence greatly affects the tracking effect. Therefore, this invention is proposed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a 3D tracking method based on direct optimization of depth data. This method introduces a perturbation model into the depth data, eliminating the need to find the correspondence between 3D points and improving tracking accuracy.

[0005] The technical solution of the present invention is as follows:

[0006] A 3D tracking method based on direct optimization of depth data, comprising the following steps:

[0007] (1) Read in the camera parameter configuration file;

[0008] (2) Initialize the camera intrinsic parameter matrix;

[0009] (3) Given the initial pose, 3D model and depth map;

[0010] (4) Estimate the pose of the next frame using a pose perturbation model based on depth data;

[0011] (5) Repeat step (4) to continue tracking.

[0012] According to a preferred embodiment of the present invention, the construction steps of the pose perturbation model based on depth data in step (4) are as follows:

[0013] Step 1: First, define the rotation matrix for the small rotation of the object in the object coordinate system;

[0014] Step 2: Determine the pose perturbation of the object in the object coordinate system;

[0015] Step 3: Based on the pose perturbation defined in Step 2, define the coordinates of the transformed 3D points in the object coordinate system;

[0016] Step 4: Based on the perturbed 3D coordinates obtained in Step 3, obtain the coordinates of the corresponding points in the 2D image;

[0017] Step 5: In the above steps, the image coordinates and depth value changes after pose change are obtained. The depth values ​​on the depth map are obtained from the two-dimensional image coordinates, and the relationship between the depth value changes of corresponding points in the two-dimensional image and pose changes is obtained. The energy function for the depth data is determined.

[0018] Step 6: Use the energy function obtained in Step 5 to differentiate the pose perturbation and derive the pose perturbation model based on depth data.

[0019] According to a preferred embodiment of the present invention, step one: First, for the minute rotation of the object in the object coordinate system, the axis angle is defined, and using an exponential mapping method, the rotation matrix is ​​defined as:

[0020]

[0021] Where Φ^ is the skew-symmetric matrix of Φ, Φ∈R 3 Let I be the element in the Lie algebra so(3), and let I be the identity matrix.

[0022] According to a preferred embodiment of the present invention, step two involves perturbing the pose of the object in the object coordinate system Δξ. o ,

[0023] Where ΔΦ o For the rotational component, Δρ o For translation components, the subscript O represents the object coordinate system.

[0024] According to a preferred embodiment of the present invention, step three: based on the pose perturbation defined in step two, the coordinates of the transformed three-dimensional point in the object coordinate system are defined as follows:

[0025]

[0026] Among them, X o The coordinates of a 3D point in the object's coordinate system. Let X be the homogeneous coordinates of a 3D point in the object coordinate system. o ′ is a disturbance The three-dimensional coordinates are then obtained using the above formula, and the three-dimensional coordinates under pose perturbation Δξ are then calculated. o The changes below.

[0027] According to a preferred embodiment of the present invention, step four: based on the perturbed three-dimensional coordinates obtained in step three, the coordinates of the corresponding points in the two-dimensional image are obtained:

[0028]

[0029]

[0030]

[0031]

[0032] in, o T c This represents the transformation matrix from the camera coordinate system to the object coordinate system, and its inverse represents the inverse transformation of the corresponding transformation, X. c Here, [x, y, z] represents the 3D coordinates in the camera coordinate system, K is the camera intrinsic parameter matrix, [x, y, z] represents the homogeneous coordinates of the 2D image points before perturbation, and [Δx, Δy, Δz] represent the perturbations in each dimension, respectively. These are the non-homogeneous coordinates of the corresponding point in the two-dimensional image after perturbation.

[0033] According to a preferred embodiment of the present invention, step five involves obtaining the image coordinates after pose change by considering the relationships between Δx, Δy, and Δz in the above steps. And the depth value change Δz, the depth value on the depth map is obtained from the two-dimensional image coordinates, and the relationship between the depth value change and pose change of the corresponding point in the two-dimensional image is as follows:

[0034]

[0035] Where D represents the depth map of the current frame, This represents the rendering depth map obtained based on the current frame pose ξ, where x is the coordinate in the 2D image to be optimized. The region to be optimized should ensure the validity of the points taken. Δx is added to D because, through perturbation, the new point x′ obtained by x is on D.

[0036] According to a preferred embodiment of the present invention, step six: using the energy function obtained in step five to adjust Δξ o Take the derivative: Let the derivative at point x be:

[0037]

[0038] Simplify

[0039]

[0040] Where h(x) is a function of the coordinates of the image to be optimized, and J is the Jacobian matrix of the energy function, let We can obtain:

[0041]

[0042] Thus, a pose perturbation model based on depth data is derived. This model directly optimizes the pose from the depth data, improving the accuracy of 3D tracking. In addition, this method can also be used in the field of 3D reconstruction.

[0043] The beneficial effects of this invention are as follows:

[0044] When estimating pose, finding the relationship between corresponding points in two consecutive frames is very sensitive to optimization. This invention proposes a direct optimization method based on depth data, which does not require finding the relationship between corresponding 3D points and improves tracking accuracy. Attached Figure Description

[0045] Figure 1 This is a flowchart of the depth map-based tracking process of the present invention;

[0046] Figure 2 This is a schematic diagram of depth map-based tracking according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of depth map-based reconstruction according to an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0049] Example 1:

[0050] A 3D tracking method based on direct optimization of depth data, comprising the following steps:

[0051] (1) Read in the camera parameter configuration file;

[0052] (2) Initialize the camera intrinsic parameter matrix;

[0053] (3) Given the initial pose, 3D model and depth map;

[0054] (4) Estimate the pose of the next frame using a pose perturbation model based on depth data;

[0055] (5) Repeat step (4) to continue tracking.

[0056] The steps for constructing the pose perturbation model based on depth data in step (4) are as follows:

[0057] Step 1: First, for the small rotation of the object in the object coordinate system, define it using axis-angles. Using an exponential mapping method, the rotation matrix is ​​defined as follows:

[0058]

[0059] Where Φ^ is the skew-symmetric matrix of Φ, Φ∈R 3 Let I be the element in the Lie algebra so(3), and let I be the identity matrix;

[0060] Step 2: Object pose perturbation Δξ in the object coordinate system o , Where ΔΦ o For the rotational component, Δρ o For translation components, the subscript O represents the object coordinate system;

[0061] Step 3: Based on the pose perturbation defined in Step 2, the coordinates of the transformed 3D points in the object coordinate system are defined as follows:

[0062]

[0063] Among them, X o The coordinates of a 3D point in the object's coordinate system. Let X be the homogeneous coordinates of a 3D point in the object coordinate system. o ′ is a disturbance The three-dimensional coordinates are then obtained using the above formula, and the three-dimensional coordinates under pose perturbation Δξ are then calculated. o The following changes;

[0064] Step 4: Based on the perturbed 3D coordinates obtained in Step 3, obtain the coordinates of the corresponding points in the 2D image:

[0065]

[0066]

[0067]

[0068]

[0069] in, o T c This represents the transformation matrix from the camera coordinate system to the object coordinate system, and its inverse represents the inverse transformation of the corresponding transformation, X. cHere, [x, y, z] represents the 3D coordinates in the camera coordinate system, K is the camera intrinsic parameter matrix, [x, y, z] represents the homogeneous coordinates of the 2D image points before perturbation, and [Δx, Δy, Δz] represent the perturbations in each dimension, respectively. These are the non-homogeneous coordinates of the corresponding points in the two-dimensional image after perturbation.

[0070] Step 5: In the above steps, Δx, Δy, and Δz are related to Δξ, thus obtaining the image coordinates after the pose change. And the depth value change Δz, the depth value on the depth map is obtained from the two-dimensional image coordinates, and the relationship between the depth value change and pose change of the corresponding point in the two-dimensional image is as follows:

[0071]

[0072] Where D represents the depth map of the current frame, This represents the rendering depth map obtained based on the current frame pose ξ, where x is the coordinate in the 2D image to be optimized. The region to be optimized should ensure the validity of the points taken. Δx is added to D because, through perturbation, the new point x′ obtained by x is on D;

[0073] Step Six: Use the energy function obtained in Step Five to apply to Δξ o Take the derivative: Let the derivative at point x be:

[0074]

[0075] Simplify

[0076]

[0077] Where h(x) is a function of the coordinates of the image to be optimized, and J is the Jacobian matrix of the energy function, let We can obtain:

[0078]

[0079] Thus, a pose perturbation model based on depth data is derived. This model directly optimizes the pose from the depth data, improving the accuracy of 3D tracking. In addition, this method can also be used in the field of 3D reconstruction.

[0080] Experiments were conducted using the 3D tracking method of this embodiment. A 3D object model, a sequence of RGB image frames captured by a camera, the corresponding depth image, and the pose of the first frame image were input into a computer. Pose optimization was performed using the pose perturbation model constructed in this embodiment, based on the depth data relationship of corresponding feature points. This embodiment optimizes the tracking pose and conducted experiments on the OPT dataset, which includes 6 objects and consists of 552 real-world sequences with significant motion blur. For evaluation, the area under the accuracy threshold curve (ADD) score was used, calculated as follows:

[0081]

[0082] Where m is the number of frames, and the threshold e t =0.2d, where d is the maximum distance between model vertices and e is the average distance error. ADD The definition is as follows:

[0083]

[0084] in This represents the difference between the estimated pose and the true pose (the estimated pose is the pose optimized by the perturbation model in this embodiment). is the model vertex, and n is the number of vertices.

[0085] The final evaluation value of this invention scales the ADD score between 0 and 20 and is denoted as the AUC score.

[0086] The AUC scores are shown in Table 1. The proposed algorithm is compared with the current best tracking algorithms SRT3D and ICG on the Soda, Chest, Ironman, House, Bike, and Jet objects on the OPT dataset. It can be seen that this embodiment achieves better results than the other methods, whether using RGBD data or depth data alone.

[0087] Table 1: Comparison of AUC values ​​between Example 1 and other methods during tracking

[0088] algorithm data Soda Chest Ironman House Bike Jet average score SRT3D RGB 15.64 16.30 17.41 16.36 13.02 15.64 15.73 ICG RGBD 15.32 15.85 17.86 17.92 16.36 15.90 16.54 ICG D —— —— —— —— —— —— 8.94 This embodiment RGBD 15.66 16.65 17.68 17.60 15.94 17.20 16.79 This embodiment D 10.13 11.35 10.20 12.90 10.65 8.99 10.70

[0089] Table 2: Comparison of ATE values ​​during reconstruction between this embodiment and other methods

[0090] RoseFusion 0.037 Example 1 0.030

[0091] In addition, the pose perturbation model proposed in this embodiment can also be used for 3D reconstruction. During the experiment, the absolute trajectory (ATE) of the root mean square error (RMSE) of the camera tracking error was recorded. The test results on the TUM dataset sequence fr2 / xyz are shown in Table 2. It can be seen that the error of this embodiment is 0.007 lower than that of RoseFusion.

Claims

1. A method of three-dimensional tracking based on direct optimization of depth data, characterized in that, The steps are as follows: (1) read in the camera parameter configuration file; (2) initialize the camera intrinsic matrix; (3) give the initial pose, three-dimensional model and depth map; (4) estimate the pose of the next frame through the pose perturbation model based on depth data, specifically: Step one: first, for the small rotation of the object in the object coordinate system, define the rotation matrix; Step two: determine the pose perturbation of the object in the object coordinate system; Step three: according to the pose perturbation defined in step two, define the coordinates of the changed three-dimensional points in the object coordinate system; Step four: according to the perturbed three-dimensional coordinates obtained in step three, obtain the coordinates of the corresponding points in the two-dimensional image; Step five: [Δx, Δy, Δz] represent the perturbation of each dimension, respectively, to get the image coordinates after the pose change and the depth value change amount Δz, the depth value on the depth map is obtained from the two-dimensional image coordinates, and the relationship between the depth value change and the pose change of the corresponding point in the two-dimensional image. The energy function about the depth data is as follows: wherein D represents a depth map of the current frame, represents a rendered depth map obtained by rendering according to the current frame pose ξ, x is a coordinate in the two-dimensional image to be optimized. Step six: Use the energy function from step five to perturb the pose Δξ o Take the derivative: Denote the derivative at point x: simplify where h(x) is a function of the coordinates of the image to be optimized, J is the Jacobian matrix of the energy function, and let It follows that Thus, the pose perturbation model based on depth data is derived; (5) repeat step (4) to continue tracking.

2. The direct optimization based on depth data three-dimensional tracking method of claim 1, wherein, Step one: first, for the small rotation of the object in the object coordinate system, define the rotation matrix using the exponential mapping method: where Φ^ is the skew-symmetric matrix of Φ, Φ ∈ R 3 are elements of the Lie algebra so(3) and I is the identity matrix.

3. The direct optimization based on depth data three-dimensional tracking method of claim 2, wherein, Step two: object pose perturbation in object coordinate system Δξ o , where ΔΦ o is the rotational component, and Δρ o is the translational component, with subscript O representing the object coordinate system.

4. The direct optimization based on depth data three-dimensional tracking method of claim 3, wherein, Step three: according to the pose perturbation defined in step two, the coordinates of the changed three-dimensional points in the object coordinate system are defined as follows: where X o is the three-dimensional point coordinate in the object coordinate system, is the homogeneous coordinate of the three-dimensional point in the object coordinate system, X o ′ is the three-dimensional coordinate after the disturbance According to the above formula, the change of the three-dimensional coordinate under the pose disturbance Δξ o is obtained.

5. The direct optimization based on depth data three-dimensional tracking method of claim 4, wherein, Step four: according to the perturbed three-dimensional coordinates obtained in step three, obtain the coordinates of the corresponding points in the two-dimensional image: wherein, o T c represents the transformation matrix from the camera coordinate system to the object coordinate system, and its inverse represents the inverse transformation of the corresponding transformation, X c is the three-dimensional coordinate in the camera coordinate system, K is the camera intrinsic matrix, [x, y, z] is the homogeneous coordinate of the two-dimensional image point before perturbation, and [Δx, Δy, Δz] respectively represents the perturbation of each dimension, is the non-homogeneous coordinate value of the corresponding point in the two-dimensional image after perturbation.

Images