A multi-lens adaptive tracking method based on vehicle wheelbase

Abstract

The application discloses a kind of self-adapting vehicle cross lens tracking method, steps include: step S1: obtaining vehicle 3D detection frame and vehicle attitude;Step S2: obtaining the extrinsic parameter focal length of camera, height, tilt angle f, h, θ;Step S3: obtaining the bird's-eye view of vehicle 2D detection frame;Step S4: according to the bird's-eye view, vehicle trajectory is handled, and the transformation matrix between each camera coordinate system is obtained;Step S5: obtaining the bird's-eye view position of detection target from one camera to another camera;Step S6: the same target in two cameras is associated with each other, and cross lens tracking is completed.The application has the advantages of simple principle, wide application range, easy implementation, higher tracking precision and the like.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of vehicle intelligent tracking technology, and particularly relates to a multi-lens adaptive tracking method based on vehicle wheelbase. BACKGROUND

[0002] Video object tracking refers to giving the initial position of a target in a video and then outputting the position of the target at each moment in the video. Object tracking is an important problem in computer vision and is usually the first step of video analysis processing. Therefore, a large number of scholars are engaged in the research of object tracking, and a large number of effective object tracking algorithms have been proposed.

[0003] In some monitoring scenes, it is necessary to track multiple objects in a complex scene at the same time. The mutual occlusion between multiple objects increases the difficulty of object tracking, which often occurs in vehicle tracking. When a large number of vehicles appear in the camera device picture at the same time, the mutual overlap between each vehicle makes it impossible to accurately obtain the actual position.

[0004] At present, the multi-target tracking method mainly includes two types: multi-target tracking based on a single camera and multi-target tracking based on multiple cameras.

[0005] The multi-target tracking method based on multiple cameras mainly focuses on how to perform data fusion of multiple cameras, mainly including a method based on camera device calibration and a method based on feature matching. The method based on camera device calibration mainly uses the projection matrix of the camera device to project different camera device pictures onto the same picture. For the method based on feature matching, efficient apparent features and spatial-temporal information are mainly used to improve the matching result. The tracking problem of multiple camera devices has greater challenges compared with the tracking problem of a single camera device due to the large light and view angle difference between different lenses. Due to the complexity of the problem, the existing cross-camera tracking method has considerable error due to various reasons.

[0006] However, for the tracking problem of multiple objects in a complex scene, one effective way is to use a multi-camera monitoring system. In the monitoring area overlapped by multiple camera devices, the position of the object can be more accurately obtained with the help of the information of multiple camera devices. With the decline in the prices of sensors and processors, the use of multiple cameras in many scenes has become more and more common.

[0007] The real-time tracking problem of multiple cameras mainly includes two parts: tracking inside the camera and cross-camera tracking. The repeated coverage area in the cross-camera tracking problem and the processing method of the non-coverage area are very meaningful for the multi-target tracking based on multiple cameras, but at the same time, due to the complexity of the problem, this work also has great challenges.

[0008] In summary, based on the multi-camera multi-target tracking method, at present, in the data fusion of multiple cameras, the feature matching based method has large calculation amount and low efficiency, cannot meet the real-time requirement, the device calibration based method has large artificial workload, and has weak anti-interference ability and poor robustness. SUMMARY

[0009] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides a self-adaptive vehicle cross-lens tracking method which is simple in principle, wide in application range, easy to implement and high in tracking accuracy.

[0010] To solve the above technical problems, the present application adopts the following technical solutions:

[0011] A self-adaptive vehicle cross-lens tracking method, comprising the following steps:

[0012] Step S1: obtaining a vehicle 3D detection frame and a vehicle attitude;

[0013] Step S2: obtaining the extrinsic focal length, height and tilt angle f, h, theta of the camera;

[0014] Step S3: obtaining an overhead view of the vehicle 2D detection frame;

[0015] Step S4: processing the vehicle trajectory according to the overhead view to obtain a transformation matrix between the coordinate systems of each camera;

[0016] Step S5: obtaining the overhead view position of the detected target when it moves from one camera to another camera;

[0017] Step S6: realizing the correlation of the same target in two cameras and completing the cross-lens tracking.

[0018] As a further improvement of the method of the present application: in step S1, the camera collects road video stream data, the road driving vehicle is taken as the target in vision, a deep neural network is used for attitude estimation of the vehicle to obtain the vehicle 3D detection frame and the vehicle attitude.

[0019] As a further improvement of the method of the present application: in step S3, according to f, h, theta of the camera, the 3D detection frame is projected in the Cartesian coordinate system with the camera as the origin and the ground as the coordinate plane, so that the 3D detection frame is presented in the 2D detection frame in the overhead view, and the overhead view of the vehicle 2D detection frame is visualized.

[0020] As a further improvement of the method of the present application: the processing of the vehicle trajectory in step S3 comprises:

[0021] Gaussian denoising, kinematic filtering and Kalman smoothing;

[0022] The vehicle trajectory is fitted by using a Bezier curve;

[0023] The transformation matrix between the camera coordinate systems is solved according to the trajectory.

[0024] As a further improvement of the method of the present application: in step S5, the tracking target detection result is projected to the bird's eye view of the camera plane through f, h, theta, and then the transformation matrix of the camera coordinate system is used to calculate the bird's eye view position of the detection target from one camera to another camera.

[0025] As a further improvement of the method of the present application: the flow of step S2 includes:

[0026] Step S201: identify the detection vehicle model, query the wheelbase and track of the vehicle model;

[0027] Step S202: detect the tire center position in the image, and use the attitude of the detection vehicle to obtain the geometric relationship between the tires;

[0028] Step S203: calculate the camera extrinsic parameters of focal length, height, and tilt angle f, h, theta.

[0029] As a further improvement of the method of the present application: in step S203, the specific calculation method of f, h, theta is:

[0030]

[0031]

[0032]

[0033] Wherein: argminf(x) is the value of x that makes the function take the minimum value; the symbol "^" represents the estimated value of the variable, the superscript "T" represents the transpose matrix of the matrix, the symbol "||" represents the modulus value of the variable, and N is the number of detected tires in a vehicle; i and j represent the i-th and j-th tires; is the direction vector of the i-th tire to the j-th tire in the world coordinate system; d is the wheelbase or track distance between the i-th tire and the j-th tire; and lambda is an optimized hyperparameter.

[0034] As a further improvement of the method of the present application: in step S4, the transformation matrix between the camera coordinate systems is calculated in the following manner:

[0035] It is known that the relationship between camera A and camera B is represented by:

[0036]

[0037] Wherein: h function is the Hungarian algorithm, matching A, B camera at the same time corresponding point, the result is the total cost; A is all points in A lens matrix, B is all points in B lens matrix, and η is a hyperparameter.

[0038] As a further improvement of the method of the application: in step S5, the calculation formula for detecting the target projection to the bird's eye view is:

[0039]

[0040]

[0041] Wherein: X i , Z i is the coordinate of the bird's eye plane, x i , y i is the pixel coordinate in the image, is the estimated value of the focal length, height and tilt angle.

[0042] As a further improvement of the method of the application: the flow of step S6 includes:

[0043] According to the calculated target position, the target detected by the camera itself at the position is associated and fused;

[0044] The same target in the two cameras is associated with each other and assigned the same number;

[0045] The same target is tracked by multiple lenses.

[0046] Compared with the prior art, the application has the advantages of:

[0047] 1. The adaptive vehicle cross-lens tracking method of the application has simple principle, wide application range and easy implementation, can reduce the workload of manual calibration parameters, enhance the robustness of multi-camera multi-target tracking and improve the multi-target tracking accuracy, and solve the problem of number switching in single-lens tracking when occlusion and overlap occur.

[0048] 2. The adaptive vehicle cross-lens tracking method of the application can calculate the focal length, height and tilt angle of the camera according to the wheelbase and track pitch of the vehicle, so as to greatly improve the tracking accuracy and efficiency.

[0049] 3. The adaptive vehicle cross-lens tracking method of the application calculates the coordinate system conversion matrix of each camera according to the optimized trajectory of the bird's eye view, which reduces the hardware cost, has better real-time performance and higher tracking accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of the method of the application. DETAILED DESCRIPTION

[0051] The application will be further described in detail below in combination with the accompanying drawings and specific embodiments.

[0052] As shown in the figure, the adaptive vehicle cross-lens tracking method of the application includes the following steps: Figure 1

[0053] Step S1: Obtain a vehicle 3D detection frame and a vehicle pose;

[0054] Obtain a vehicle 3D detection frame and a vehicle pose by collecting road video stream data through a camera, taking a road driving vehicle as a target in vision, and using a deep neural network to estimate the pose of the vehicle.

[0055] Step S2: Obtain the extrinsic focal length, height, and tilt angle f, h, θ of the camera;

[0056] Step S3: Obtain an aerial view of the vehicle 2D detection frame;

[0057] According to f, h, and θ of the camera, the 3D detection frame is projected in a Cartesian coordinate system with the camera as the origin and the ground as the coordinate plane, so that the 3D detection frame is converted into a 2D detection frame with a top-down perspective, and the aerial view of the vehicle 2D detection frame is visualized.

[0058] Step S4: Process the vehicle trajectory according to the aerial view to obtain the transformation matrix between the camera coordinate systems;

[0059] Step S5: Obtain the aerial view position of the detection target when it moves from one camera to another;

[0060] Project the tracking target detection result to the aerial view perspective of the camera plane through f, h, and θ, and then calculate the aerial view position of the detection target when it moves from one camera to another through the transformation matrix of the camera coordinate system.

[0061] Step S6: Realize the correlation between the same target in two cameras and complete cross-lens tracking.

[0062] In a specific application example, the process of step S2 includes:

[0063] Step S201: Identify the detection vehicle model and query the wheelbase and track of the vehicle model;

[0064] Step S202: Detect the tire center position in the image, and use the pose of the detection vehicle to deduce the geometric relationship between the tires.

[0065] Step S203: Calculate the extrinsic focal length, height, and tilt angle f, h, θ of the camera.

[0066] ​In a specific application example, the specific calculation method of f, h, θ in step S203 is as follows:

[0067]

[0068]

[0069]

[0070] Wherein: argminf(x) is the value of x that makes the function take the minimum value; the symbol "^" represents the estimated value of the variable, the superscript "T" represents the transpose matrix of the matrix, the symbol "||" represents the modulus value of the variable, N is the number of tires detected in a vehicle; i, j represent the i, j th tire; is the direction vector of the i th tire pointing to the j th tire in the world coordinate system; d is the wheelbase or track of the i th tire and the j th tire; λ is an optimized hyperparameter.

[0071] In a specific application example, the processing of the vehicle trajectory in step S3 includes:

[0072] Gaussian denoising, kinematic filtering, Kalman smoothing;

[0073] fitting the vehicle trajectory by using a Bessel curve;

[0074] synchronizing the time of each camera, and solving the transformation matrix between each camera coordinate system according to the trajectory.

[0075] In a specific application example, the transformation matrix between the camera coordinate systems in step S4 is calculated as follows:

[0076] The relationship between camera A and camera B can be represented as follows:

[0077]

[0078] Wherein: h function is a Hungarian algorithm, matching the corresponding points of the same time of cameras A and B, and the result is the total cost; A is a matrix composed of all points in the A lens, B is a matrix composed of all points in the B lens, and η is a hyperparameter.

[0079] In a specific application example, the calculation formula of the projection of the detection target to the bird's eye view in step S5 is as follows:

[0080]

[0081]

[0082] Wherein: X i , Z iFor the coordinates of the bird's eye view plane, x i ,y i For the pixel coordinates in the image, For the estimates of the focal length, height, and tilt angle.

[0083] In a specific application example, the step S6 includes:

[0084] According to the calculated target position, the target detected by the camera itself at the position is associated and fused;

[0085] The same target in the two cameras is associated with each other and given the same number;

[0086] The same target is tracked by multiple lenses.

[0087] The above is only a preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments only. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall be considered as falling within the protection scope of the present application.

Claims

1. An adaptive vehicle cross-camera tracking method, characterized in that, The steps include: Step S1: Obtain the vehicle's 3D bounding box and vehicle pose; Step S2: Obtain the camera's extrinsic parameters: focal length, altitude, and tilt angle. ; Step S3: Obtain a bird's-eye view of the 2D detection frame of the vehicle; Step S4: Process the vehicle trajectory based on the bird's-eye view to obtain the transformation matrix between the coordinate systems of each camera; Step S5: Obtain the bird's-eye view of the detected target's location as it moves from one camera to another; Step S6: Establish correlation between the same target in the two cameras to complete cross-camera tracking; The process of step S2 includes: Step S201: Identify and detect the vehicle model, and query the wheelbase and track width of the vehicle model; Step S202: Detect the center position of the tires in the image, and use the detected vehicle posture to obtain the geometric relationship between the tires; Step S203: Calculate the camera's extrinsic parameters: focal length, altitude, and tilt angle. ; In step S203 The specific calculation method is as follows: in: To find the value of x that makes the function reach its minimum; the symbol " " indicates the estimated value of the variable, the superscript "T" indicates the transpose of the matrix, the symbol "| |" indicates the magnitude of the variable, x and y represent the pixel coordinates in the camera image, N is the number of tires detected in a car; i and j represent the i-th and j-th tires, respectively. , Represents the pixel coordinates of the image of the i-th tire; , Represents the pixel coordinates of the image of the j-th tire; d is the direction vector from the i-th tire to the j-th tire in the world coordinate system; d is the wheelbase or track width between the i-th tire and the j-th tire. For optimization of hyperparameters; when When the minimum is obtained, The corresponding value can be obtained; In step S5, the formula for calculating the projection of the detected target onto the bird's-eye view is: in: , The coordinates of the bird's-eye view plane. Let be the coordinates of a pixel on the image along the horizontal and vertical axes, where i represents the i-th tire. The estimated values ​​for the focal length, height, and tilt angle are... "" indicates an estimate of the parameter variable.

2. The adaptive vehicle cross-camera tracking method according to claim 1, characterized in that, In step S1, road video stream data is collected by a camera. Taking vehicles traveling on the road in the vision as targets, a deep neural network is used to estimate the vehicle's pose in order to obtain the vehicle's 3D detection box and vehicle pose.

3. The adaptive vehicle cross-camera tracking method according to claim 1, characterized in that, In step S3, according to the camera's... By projecting the 3D detection frame onto a Cartesian coordinate system with the camera as the origin and the ground as the coordinate plane, the 3D detection frame will be converted into a 2D detection frame, presenting a top-down view, and then a bird's-eye view of the vehicle's 2D detection frame will be visualized.

4. The adaptive vehicle cross-camera tracking method according to claim 3, characterized in that, The vehicle trajectory processing in step S3 includes: Gaussian denoising, kinematic filtering, and Kalman smoothing; Bézier curves were used to fit the vehicle trajectory; The time of each camera is synchronized, and the transformation matrix between the coordinate systems of each camera is solved based on the trajectory.

5. The adaptive vehicle cross-camera tracking method according to claim 1, characterized in that, In step S5, the target detection result is transmitted through... The bird's-eye view projected onto the camera plane is used to calculate the position of the detected target as it moves from one camera to another using the transformation matrix of the camera coordinate system.

6. The adaptive vehicle cross-camera tracking method according to any one of claims 1-5, characterized in that, In step S4, the transformation matrix between the camera coordinate systems is calculated as follows: The relationship between camera A and camera B is represented by the following: , Where: h is the function of the Hungarian algorithm, which matches the corresponding points of cameras A and B at the same moment; the result is the total cost, i.e., minimizing the error. Let A be the matrix consisting of all points in shot A. Let B be the matrix consisting of all the points in shot B. H is a hyperparameter, and H is the transformation matrix of matrix A and matrix B formed by the sampling points in cameras A and B. The optimal transformation matrix H is estimated by estimating a large number of sampling points of matrix A and matrix B.

7. The adaptive vehicle cross-camera tracking method according to any one of claims 1-5, characterized in that, The process of step S6 includes: Based on the calculated target location, it is correlated and fused with the target detected by the camera itself at that location; Associate the same target in two cameras and assign them the same number; To achieve multi-camera tracking of targets with the same serial number.

Images