A 3D object tracking method based on multi-source data fusion

Abstract

The present invention discloses a 3D target tracking method based on multi-source data fusion. The method comprises the following steps: first, obtaining point cloud information and image information of a target through a laser radar and a camera; then inputting the 2D image information into a 2D detection network, which outputs the pixel frame position and category vector of the target in the 2D image; then, filtering the 3D point cloud data using the pixel frame boundary condition and the camera projection matrix; selecting the point cloud within the pixel frame after projection as the point cloud of interest; inputting the point cloud of interest and the category vector into a point cloud segmentation network to obtain the target point cloud; then, inputting the target point cloud into a T‑Net rotation network to transfer the target point cloud to a new coordinate system; then, inputting the target point cloud in the new coordinate system into a target box regression network to obtain the 3D box parameters of the target; finally, modeling the motion state of the target, constructing the target motion matrix and observation matrix, taking the output 3D box parameters of the network as observation variables, and tracking the target using a fading factor Kalman filter.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision tracking, and more specifically, relates to a 3D target tracking method based on multi-source data fusion. Background Art

[0002] Target tracking technology is a very important research direction in the field of computer vision. It has been widely used in fields such as autonomous driving, autonomous navigation, robotics, and virtual reality. With the rapid development of artificial intelligence and sensor technologies, and the gradual reduction in the cost of three-dimensional point cloud acquisition equipment, visual tasks have gradually shifted from 2D to 3D. However, a single sensor is difficult to cope with complex and changing scenes. Multi-sensor fusion has become the mainstream research solution. It is hoped that by fusing the characteristics of different sensor data and complementing each other's advantages, the requirements of safety and reliability can be met.

[0003] Over the past decade, image-based deep network technology has developed rapidly, capable of maturely describing features such as an object's shape, texture, and high-level semantics. However, because camera data is two-dimensional, it cannot accurately capture information such as the target's precise location and size. Furthermore, data from traditional cameras is easily affected by weather and lighting. LiDAR, however, due to its inherent sensor characteristics, can accurately describe real-world spatial information in the surrounding 3D space, such as distance, depth, and surface shape, and is unaffected by lighting. However, due to its device characteristics, LiDAR lacks the native resolution and efficient array structure of images, and does not contain extensive semantic information, making direct target identification and tracking difficult. Therefore, target tracking methods that combine the characteristics of both sensors can complement each other's shortcomings and achieve accurate target identification and tracking. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a 3D target tracking method based on multi-source data fusion to achieve accurate position and posture tracking of the target.

[0005] To achieve the above-mentioned object of the invention, the present invention provides a 3D target tracking method based on multi-source data fusion, characterized by comprising the following steps:

[0006] (1) Collect 3D point cloud data of the tracking target through LiDAR at time t Use a monocular camera to capture a 2D image of the tracked target

[0007] (2) The 2D image at time t Input to the 2D target detection network and output the 2D target detection box R t and target type vector I t ;

[0008] (3) Construct the projection matrix K according to the factory parameters of the monocular camera;

[0009] (4) Use the projection matrix K to transform the 3D point cloud data Projecting to a 2D image If the projected 3D point cloud coordinates are located in the 2D target detection frame R t If it is inside, it is considered to be an interest point cloud, recorded as P1 t ; Otherwise it is not considered as a point cloud of interest;

[0010] (5) The point of interest cloud P1 t and target type vector I t Input to the point cloud segmentation network to output the probability value P I ; According to the probability value P I Determine whether the point cloud of interest is the target point cloud. If P I > 0.5, then the interest point cloud is judged to be the target point cloud, recorded as Otherwise, it is judged as background point cloud;

[0011] (6) The target point cloud Input to the coordinate rotation network T-Net, and calculate the target point cloud through T-Net The optimal coordinate orientation corresponds to the rotation matrix K';

[0012] (7) The target point cloud Perform coordinate transformation to obtain point cloud data in the new coordinate system

[0013]

[0014] (8) Point cloud data after coordinate transformation Input into the target frame regression network to obtain the observed 3D frame of the tracked target at time t in, Indicates the position coordinates of the observed 3D frame from the laser radar. Indicates the width, length and height of the tracking target. Indicates the yaw angle of the tracking target;

[0015] (9) According to the motion state of the tracking target, construct the motion model matrix M and the observation model matrix H;

[0016] (10) Assume that the target state Q of the tracking target at the initial time t = 0 0 , calculate the predicted target state of the tracking target at time t

[0017]

[0018] Among them, Q t-1 Indicates the final target state of the tracking target at time t-1;

[0019] (11) Observe the 3D target frame and predicted target state Input into the fading factor Kalman filter to obtain the final target state Q of the tracking target at time t t ;

[0020]

[0021] Among them, F t is the Kalman gain matrix at time t, and H is the observation model matrix;

[0022] (12) Determine whether time t is the final time of tracking the target. If so, end the tracking; otherwise, jump to (1).

[0023] The object of the invention of the present invention is achieved like this:

[0024] The present invention is based on a 3D target tracking method based on multi-source data fusion. First, the point cloud information and image information of the target are obtained through a laser radar and a camera. Then, the 2D image information is input into a 2D detection network, and the detection network outputs the pixel frame position and category vector of the target in the 2D image. Subsequently, the 3D point cloud data is screened using the pixel frame boundary conditions and the camera's projection matrix, and the point cloud within the pixel frame after projection is selected as the point cloud of interest. Afterwards, the point cloud of interest and the category vector are input into the point cloud segmentation network to obtain the target point cloud. Then, the target point cloud is input into the T-Net rotation network to transfer the target point cloud to a new coordinate system. The target point cloud in the new coordinate system is then input into the target frame regression network to obtain the 3D frame parameters of the target. Finally, the motion state of the target is modeled, the target motion matrix and the observation matrix are constructed, the output 3D frame parameters of the network are used as observation variables, and the target is tracked using a fading factor Kalman filter.

[0025] At the same time, the 3D target tracking method based on multi-source data fusion of the present invention also has the following beneficial effects:

[0026] (1) The present invention uses a 2D detection network to determine the target area point cloud and target type, making full use of the rich semantic information of the 2D image, reducing the number of point clouds subsequently input into the point cloud network, and greatly improving the network's operating efficiency.

[0027] (2) The point cloud segmentation network and 2D detection network are used to perform independent feature extraction on two types of heterogeneous data, lidar point cloud data and monocular images, respectively, making full use of the data advantages of a single sensor to achieve accurate position estimation, size estimation and type judgment.

[0028] (3) The present invention improves the accuracy of obtaining the fading factor through the new information covariance estimate, so that the algorithm can effectively improve the target tracking accuracy in the case of model mismatch and noise mismatch, prevent filtering divergence, and further improve the robustness of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the 3D target tracking method based on multi-source data fusion of the present invention;

[0030] Figure 2 It is a schematic diagram of the structure of the point cloud segmentation network in the present invention;

[0031] Figure 3 It is a structural diagram of the target box regression network in the present invention.

[0032] Figure 4 It is a schematic diagram of the structure of the T-Net network in the present invention. DETAILED DESCRIPTION

[0033] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0034] Example

[0035] Figure 1 This is a flow chart of the 3D target tracking method based on multi-source data fusion of the present invention.

[0036] In this embodiment, if Figure 1 As shown, the present invention provides a 3D target tracking method based on multi-source data fusion, comprising the following steps:

[0037] S1, collect 3D point cloud data of the tracking target through LiDAR at time t Use a monocular camera to capture a 2D image of the tracked target

[0038] S2, the 2D image at time t Input to the 2D target detection network and output the 2D target detection box R t and target type vector I t ;

[0039] In this embodiment, the 2D object detection network can use SSD, YOLO, FCOS, etc.

[0040] S3. Construct the projection matrix K according to the factory parameters of the monocular camera;

[0041] In this embodiment, the model of the monocular camera selected is Canon 200D, and the following projection matrix K is constructed according to the factory parameters of the camera:

[0042]

[0043] S4, use the projection matrix K to transform the 3D point cloud data Projecting to a 2D image If the projected 3D point cloud coordinates are located in the 2D target detection frame R t If it is inside, it is considered to be an interest point cloud, recorded as P1 t ; Otherwise it is not considered as a point cloud of interest;

[0044] S5, the interest point cloud P1 t and target type vector I t Input to the point cloud segmentation network to output the probability value P I ; According to the probability value P I Determine whether the point cloud of interest is the target point cloud. If P I > 0.5, then the interest point cloud is judged to be the target point cloud, recorded as Otherwise, it is judged as background point cloud;

[0045] In this embodiment, the point cloud segmentation network uses PointNet as the basic network. The specific structure of the network is as follows: Figure 2 Assume that the number of input target point clouds is n. First, the input point cloud is passed through a multi-layer 1D convolutional network to obtain intermediate feature 1, which is of size (64×n). Then, it is passed through multiple layers of 1D convolution to obtain intermediate feature 2, which is of size (1024×n). Then, the category vector, intermediate feature 1, and intermediate feature 2 are concatenated to obtain the global feature. The global feature is subjected to multiple 1D convolutions to obtain a category vector matrix of size (n×2).

[0046] S6. Target point cloud Input to the coordinate rotation network T-Net, and calculate the target point cloud P2 through T-Net t The optimal coordinate orientation corresponds to the rotation matrix K';

[0047] The specific structure of the T-Net network in this embodiment is as follows Figure 3 Assume that the number of point clouds input to the network is m. First, the input point cloud data is subjected to multiple layers of 1D convolution to obtain intermediate features of size m×512. Then, the intermediate features are max-pooled to obtain global features of size 1×512. After that, three layers of fully connected layers are passed to obtain the final rotation matrix K' of size 3×3.

[0048] S7, target point cloud Perform coordinate transformation to obtain point cloud data P3 in the new coordinate system t ;

[0049]

[0050] S8. Point cloud data after coordinate transformation Input into the target frame regression network to obtain the observed 3D frame of the tracked target at time t in, Indicates the position coordinates of the observed 3D frame from the laser radar. Indicates the width, length and height of the tracking target. Indicates the yaw angle of the tracking target;

[0051] In this embodiment, the 3D frame is observed The calculation method of each parameter is:

[0052] S8.1, the point cloud data at time t and target type vector I t Input to the target box regression network;

[0053] S8.2. The classification network in the target frame regression network divides the yaw angle into η categories, and outputs the category C of the yaw angle through the classification network. θ and the size category C of the tracking target (w,h,l) , of which category C θ The corresponding yaw angle is θ, category C (w,h,l) The length, width and height of the corresponding target are (w, h, l);

[0054] S8.3. The regression network in the target box regression network predicts the yaw angle The yaw margin Δθ relative to θ, the size of the tracking target The size margin (Δw, Δh, Δl) relative to (w, h, l), and the position coordinates of the observed 3D frame from the lidar

[0055] S8.4. Calculate the yaw angle of the tracked target and size

[0056]

[0057]

[0058] Among them, size(C (w,h,l) ) represents each category C (w,h,l) The length, width and height of the corresponding target are the mean of (w, h, l).

[0059] The specific structure of the target frame regression network in this embodiment is as follows Figure 4 As shown in the figure. First, the coordinate-transformed point cloud data is subjected to multiple layers of 1D convolution to obtain intermediate feature variables of size m×512. Maximum pooling is then performed to obtain global features of size 1×512. The category vector and the global feature vector are concatenated, and the concatenated vector is passed through three fully connected layers, with sizes of 512+k, 256, and 3+4NS+2NH, respectively. Finally, the 3D bounding box parameters of the target are output. k is the length of the category vector, NS is the number of target size categories set by the network, and NH is the number of yaw angle categories set by the network.

[0060] S9. Construct a motion model matrix M, an observation model matrix H, and a state covariance noise matrix U according to the motion state of the tracking target;

[0061] In this embodiment, it is assumed that the motion state of the tracking target is uniform motion, and the following motion model matrix M is established:

[0062]

[0063] Among them, (w,h,l) represents the width, length and height of the target, (x,y,z) represents the coordinates of the 3D frame from the lidar, θ represents the yaw angle of the 3D frame, () (v x ,vy,vz) represents the velocity of the target in the x,y,z directions, and ω represents the angular velocity of the target.

[0064] According to the 3D frame parameter type, the observation matrix H of this embodiment is:

[0065]

[0066] In this embodiment, the state covariance matrix U is an 11×11 identity matrix.

[0067] S10, assume that the target state Q of the tracking target at the initial time t = 0 0 , Q 0 The value of needs to be initialized based on the observed 3D box parameters and initial motion state assumptions at time 0. Calculate the predicted target state of the tracked target at time t

[0068]

[0069] Among them, Q t-1 Indicates the final target state of the tracking target at time t-1;

[0070] S11, observe the 3D target frame and predicted target state Input into the fading factor Kalman filter to obtain the final target state Q of the tracking target at time t t ;

[0071]

[0072] Among them, F t is the Kalman gain matrix at time t, and H is the observation model matrix;

[0073] In this embodiment, the Kalman gain matrix F t The calculation formula is:

[0074] F t =P t pre H T (HP t pre H T +R) -1

[0075] Among them, P t pre is the prediction covariance matrix at time t, R is the preset observation noise matrix, H represents the observation model matrix, and the superscript T represents transpose.

[0076] The observation noise matrix R in this embodiment is:

[0077]

[0078] Among them, the prediction covariance matrix P t pre The calculation method is:

[0079] P t pre =λ t MP t-1 M T +U

[0080] Among them, λ t is the fading factor, M is the motion model matrix, P t-1 is the state covariance matrix at time t-1, and the specific calculation formula is:

[0081]

[0082] In this embodiment, the state covariance matrix P0 at time 0 is:

[0083]

[0084] Among them, the fading factor λ t The calculation method is:

[0085] 1) Take the N moments before the current moment t, and then calculate the new information variable ε of the tracking target at each moment in these N moments g ;

[0086]

[0087] Where g = 1, 2, ..., N, Represents the observed 3D frame of the tracking target at time g, represents the predicted target state of the tracking target at time g;

[0088] 2) Calculate the initial weight W of the fading factor Kalman filter at each moment g ;

[0089]

[0090] Among them, the mean μ=(1+N) / 2, the standard deviation

[0091] 3) Initial weight W at each moment g Perform normalization processing;

[0092]

[0093] in, Represents the normalized weight value at the g-th moment;

[0094] 4) Calculate the innovation covariance estimator

[0095]

[0096] 5) Calculate the fading factor λ t ;

[0097]

[0098] S12: Determine whether the moment t is the final moment of tracking the target. If so, end the tracking; otherwise, jump to S1.

[0099] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A 3D target tracking method based on multi-source data fusion, characterized in that: The following steps are involved: (1) Collect 3D point cloud data of the tracking target through LiDAR at time t Use a monocular camera to capture a 2D image of the tracked target (2) The 2D image at time t Input to the 2D target detection network and output the 2D target detection box R t and target type vector I t ; (3) Construct the projection matrix K according to the factory parameters of the monocular camera; (4) Use the projection matrix K to transform the 3D point cloud data Projecting to a 2D image If the projected 3D point cloud coordinates are located in the 2D target detection frame R t If it is inside, it is considered to be an interest point cloud, recorded as P1 t ; Otherwise it is not considered as a point cloud of interest; (5) The point of interest cloud P1 t and target type vector I t Input to the point cloud segmentation network to output the probability value P I ; According to the probability value P I Determine whether the point cloud of interest is the target point cloud. If P I > 0.5, then the interest point cloud is judged to be the target point cloud, recorded as Otherwise, it is judged as background point cloud; (6) The target point cloud Input to the coordinate rotation network T-Net, and calculate the target point cloud through T-Net The optimal coordinate orientation corresponds to the rotation matrix K'; (7) The target point cloud Perform coordinate transformation to obtain point cloud data in the new coordinate system (8) Point cloud data after coordinate transformation Input into the target frame regression network to obtain the observed 3D frame of the tracked target at time t in, Indicates the position coordinates of the observed 3D frame from the laser radar. Indicates the width, length and height of the tracking target. Indicates the yaw angle of the tracking target; (9) According to the motion state of the tracking target, construct the motion model matrix M and the observation model matrix H; (10) Assume that the target state Q of the tracking target at the initial time t = 0 0 , calculate the predicted target state of the tracking target at time t Among them, Q t-1 Indicates the final target state of the tracking target at time t-1; (11) Observe the 3D target frame and predicted target state Input into the fading factor Kalman filter to obtain the final target state Q of the tracking target at time t t ; Among them, F t is the Kalman gain matrix at time t, and H is the observation model matrix; (12) Determine whether time t is the final time of tracking the target. If so, end the tracking; otherwise, jump to (1).

2. The 3D target tracking method based on multi-source data fusion according to claim 1, characterized in that: The observation 3D frame The calculation method of each parameter is: (2.1) The point cloud data at time t and target type vector I t Input to the target box regression network; (2.2) The classification network in the target frame regression network divides the yaw angle into η categories, and the classification network outputs the category C of the yaw angle θ and the size category C of the tracking target (w,h,l) , of which category C θ The corresponding yaw angle is θ, category C (w,h,l) The length, width and height of the corresponding target are (w, h, l); (2.3) The regression network in the target frame regression network predicts the yaw angle The yaw margin Δθ relative to θ, the size of the tracking target The size margin (Δw, Δh, Δl) relative to (w, h, l), and the position coordinates of the observed 3D frame from the lidar (2.4) Calculate the yaw angle of the tracking target and size Among them, size(C (w,h,l) ) represents each category C (w,h,l) The length, width and height of the corresponding target are the mean of (w, h, l).

3. The 3D target tracking method based on multi-source data fusion according to claim 1, characterized in that: The Kalman gain matrix F t The calculation formula is: F t =P t pre H T (HP t pre H T +R) -1 Among them, P t pre is the prediction covariance matrix at time t, R is the preset observation noise, H represents the observation model matrix, and the superscript T represents transpose.

4. The 3D target tracking method based on multi-source data fusion according to claim 3, characterized in that: The prediction covariance matrix P t pre The calculation method is: (4.1) According to the motion state of the tracking target, construct the state covariance noise matrix U; (4.2), calculate the prediction covariance matrix P t pre ; P t pre =λ t MP t-1 M T +U Among them, λ t is the fading factor, M is the motion model matrix, P t-1 is the state covariance matrix at time t-1, and the specific calculation formula is:

5. The 3D target tracking method based on multi-source data fusion according to claim 4, characterized in that: The fading factor λ t The calculation method is: (5.1) Take the N moments before the current moment t, and then calculate the new information variable ε of the tracking target at each moment in these N moments g ; Where g = 1, 2, ..., N, Represents the observed 3D frame of the tracking target at time g, represents the predicted target state of the tracking target at time g; (5.2) Calculate the initial weight W of the fading factor Kalman filter at each moment g ; Among them, the mean μ=(1+N) / 2, the standard deviation (5.3), the initial weight W at each moment g Perform normalization processing; in, Represents the normalized weight value at the g-th moment; (5.4), calculate the innovation covariance estimator (5.5), calculate the fading factor λ t ;

Images