An improved multi-target tracking method based on Gaussian process

Abstract

This invention discloses an improved multi-target tracking method based on Gaussian processes. First, it relies on laser radar point cloud information and camera image information for multi-target recognition. Secondly, based on the target uniform motion model assumption, the prior positions of multiple targets at different times and the current recognition positions are obtained according to the laser radar multi-target recognition frequency and the camera multi-target recognition frequency. The IOU is calculated and used to form a matching cost matrix. The Hungarian algorithm is then used to associate and assign multiple detected targets and motion trajectories. Simultaneously, the detected targets are uniformly converted to the world coordinate system using the camera's intrinsic parameters and the extrinsic parameters between the laser radar, the camera, and the world coordinate system. Finally, Gaussian interpolation is used to integrate the target detection results from the camera and laser radar with the target prior positions to construct residuals. This method completes batch optimization of each target position sequence in the world coordinate system, thereby achieving multi-target tracking. This scheme can improve the accuracy and robustness of multi-target tracking.

Description

Technical Field

[0001] The invention relates to an improved multi-target tracking method based on Gaussian process, and belongs to the technical field of robot multi-target tracking. Background Art

[0002] With the further development of science and technology, more and more functional robots have been invented and applied to various fields including military, industrial and civilian use. At present, the multi-target tracking solutions of robots are usually based on the design framework of the Sort solution and the DeepSort solution. The Sort solution combines the Hungarian algorithm and the Kalman filter algorithm for target tracking, while the DeepSort solution adds a cascade matching process on the basis of the Sort solution, and adds state classification of deterministic and non-deterministic states, thereby improving the accuracy of multi-target tracking.

[0003] Target information in multi-target tracking is obtained from data from various sensors, such as camera images or LiDAR point cloud modeling. Deep learning methods are used to detect these different sensor signals, thereby detecting dynamic targets and achieving multi-target detection. Commonly used deep learning methods for multi-target detection include Faster R-CNN, YOLOv5, and SSD. These methods train their own models by learning from existing data, thereby improving the detection performance of the model algorithm.

[0004] The state estimation of multi-target tracking is achieved through different filters. For example, in the Sort scheme, the main framework is Kalman filtering. However, for discrete-time filter methods such as Kalman filtering, it is impossible to accurately fuse the results of multi-target recognition between different sensors when the operating frequencies of different sensors are different, and thus it is impossible to accurately track the target. Therefore, it is necessary to propose a new fusion architecture from the back-end architecture to utilize the results of target detection by different sensors at the front end to achieve more accurate multi-target tracking effects. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to use a continuous Gaussian process to model the target trajectory of multi-target tracking in the background framework of multi-sensor fusion, thereby improving the algorithm structure of multi-target tracking under the classic Sort framework, and at the same time using a nonlinear optimization method to optimize the overall motion posture of the target and generate a trajectory, thereby improving the accuracy of trajectory prediction and trajectory tracking, and enhancing the overall effect of multi-target tracking.

[0006] To solve the above technical problems, the present invention provides an improved multi-target tracking method based on Gaussian process, which comprises the following steps:

[0007] S1: Run YOLOv5 and Point RCNN target detectors to detect and identify dynamic objects in each frame of camera image and lidar point cloud, and obtain the target detection box and the center position of the target detection box of the dynamic object;

[0008] S2: Based on the LiDAR multi-target recognition frequency, the camera multi-target recognition frequency and the target uniform motion model prediction, the prior position of each target in the trajectory manager in the world coordinate system is obtained and the coordinates are converted to the camera coordinate system and the LiDAR coordinate system respectively. The IOU (intersection-over-union) matching is performed with the center position of the current target detection box obtained in S1, and a similarity matrix is ​​generated. The Hungarian algorithm is used to associate the data of multiple detected targets with the existing trajectories.

[0009] S3: For the targets associated in S2, the target position is first converted from the sensor coordinate system to the unified world coordinate system using the external parameter value between the sensor coordinate system and the world coordinate system. Then, the target prior position calculated in S2 and the current target observation position are combined to construct the residual to optimize the position and generate the trajectory. For the newly detected targets in S2, a new ID value is assigned, and the target position is converted from the sensor coordinate system to the unified world coordinate system using the external parameter value between the sensor coordinate system and the world coordinate system. Then, the state modeling of its trajectory is performed using the Gaussian process. For the targets that cannot be associated in S2, if the number of times they cannot be associated is greater than the mismatch threshold, their trajectory is deleted. If it is less than the mismatch threshold, their trajectory is retained. The present invention takes into account the inconsistent timing in the fusion of multiple sensors, uses the Gaussian process to model the motion trajectory of each target, and uses the Gaussian interpolation method to integrate the detection results of different sensors at different times, and finally achieves batch optimization of the position sequence of each target, thereby obtaining accurate motion trajectories of multiple targets, which can improve the accuracy and robustness of multi-target tracking.

[0010] The target detection frame and the target detection frame center position model of the dynamic object in step S1 are as follows:

[0011] S1.1 uses the YOLOv5 algorithm to identify dynamic objects in each frame of the image, and uses the Point RCNN algorithm to identify dynamic objects in each frame of the lidar point cloud information. This method obtains the target detection box of the dynamic object in the image information and point cloud information respectively, and models the center position of the target detection box:

[0012] The dynamic object target detection box in the image information can be modeled as:

[0013] x=[u,v] T

[0014] Where [u, v] is the two-dimensional coordinate of the center of the target detection box in the image coordinate system;

[0015] The dynamic object target detection box in the lidar point cloud information can be modeled as:

[0016] x=[x,y,z] T

[0017] Where [x, y, z] are the three-dimensional coordinates of the center of the target detection frame in the laser radar coordinate system;

[0018] The method for obtaining the target trajectory prior position, IOU matching method and Hungarian algorithm in step S2 are as follows:

[0019] S2.1 assumes that the target moves at a constant speed. According to the laser radar multi-target recognition frequency, the prior position of each target trajectory in the world coordinate system is calculated. Assume that the laser radar target recognition time of the next frame is t i , the last optimization time is t0, from which the current target prior position and velocity and prior covariance are obtained as follows:

[0020] x(t i )=Φ(t i ,t0)x(t0)

[0021] where x(t i ) is the prior position p(t i ) and speed Φ(t i ,t0) is the state transition function:

[0022]

[0023]

[0024] Target prior covariance κ(t i )for:

[0025]

[0026] Where Φ(t i ,t0),Φ(t i ,s) is the state transfer function, Q0 is the initial covariance at time t0, Q c is the power spectrum density matrix, and F(s) is the noise control matrix:

[0027]

[0028] According to the camera multi-target recognition frequency, the prior position x(τ) of each target in the world coordinate system at the time of camera multi-target detection is calculated, t0<τ <t i :

[0029]

[0030] in t i The target position and velocity at the moment, Φ(τ,t i ) is the state transfer equation:

[0031]

[0032] S2.2 uses the external parameters between the world coordinate system and different sensor coordinate systems to convert the target trajectory prior position in the world coordinate system at different times to the image coordinate system and the lidar coordinate system respectively:

[0033] The prior position of the center of the target detection box in the image coordinate system is x=[u,v] T It can be calculated by the following formula:

[0034]

[0035] Where λ is the depth information of the center of the target detection frame, f x ,f y ,c x ,c y All are internal parameters of the camera, and [X, Y, Z] is the coordinate of the center of the target detection frame in the camera coordinate system.

[0036] The prior position of the center of the target detection box in the lidar coordinate system is x = [x, y, z] T , which can be directly obtained through the external parameters between the lidar coordinate system and the world coordinate system.

[0037] S2.3 performs IOU matching on each target prior detection frame and the current target detection frame. The IOU value can be calculated as follows:

[0038]

[0039] Where A is the area / volume of the target trajectory prior detection box, and B is the area / volume of the current target detection box;

[0040] According to the IOU value, the similarity matrix C can be generated as follows:

[0041]

[0042] where c ij is the IOU value between the i-th target trajectory prior detection box and the j-th current target detection box.

[0043] S2.4 uses the similarity matrix obtained in S2.3 and the Hungarian algorithm to match multiple detection targets, and the following three matching results can be obtained:

[0044] The first one: the target detection result of the current frame matches a specific trajectory.

[0045] The second type: the target detection result of the current frame cannot be associated with any trajectory.

[0046] The third type: The current trajectory cannot find any matching target detection results.

[0047] The Gaussian process modeling trajectory optimization method in step S3 is as follows:

[0048] S3.1 For the target associated with S2, first use the external parameter between the sensor coordinate system and the world coordinate system to convert the target position from the sensor coordinate system to the unified world coordinate system:

[0049] For the target in the image coordinate system, the target position is converted from the image coordinate system to the camera coordinate system through the camera intrinsic parameter value:

[0050]

[0051] Where [u, v] is the coordinate of the center of the target detection frame in the image coordinate system, λ is the depth information of the center of the dynamic target detection frame, and f x ,f y ,c x ,c y All are internal parameters of the camera, and [X, y, Z] are the center coordinates of the target detection box in the camera coordinate system.

[0052] According to the external parameter value between the camera coordinate system and the world coordinate system, the detected target position is converted from the camera coordinate system to the world coordinate system;

[0053] For the target in the LiDAR point cloud coordinate system, the external parameter value between the LiDAR coordinate system and the world coordinate system is used to convert the detected target position from the LiDAR coordinate system to the world coordinate system;

[0054] S3.2 For the newly identified targets by the lidar and camera, the target detection results of both are combined to form the observation quantity M of the current system front-end sensor t =[X,Y,Z] T :

[0055]

[0056] where [X c ,Y c ,Z c ] T is the measurement result of the camera, [X l ,Y l ,Z l ]T is the measurement result of the lidar, β is the reliability, β = 0.3.

[0057] The Gaussian process based on the linear time-varying stochastic difference equation (LTV_SDE) is used to describe the uniform motion state of the target:

[0058]

[0059] Where A(t) is the state transfer matrix, u(t) is the input matrix, F(t) is the noise control matrix, and w(t) is the motion noise matrix:

[0060]

[0061] where p(t) = [X, Y, Z] T , Q c is the power spectral density matrix, and δ(tt′) is the Dirkat function.

[0062] S3.3 For the case of a target that cannot be associated, if the number of times it cannot be associated is greater than the mismatch threshold, its trajectory is deleted; if it is less than the mismatch threshold, its trajectory is retained.

[0063] S3.4 For the case where the identified target and trajectory match, taking into account the difference in target detection frequencies between the lidar and camera, the lidar target detection frequency is first selected as the reference frequency for back-end state fusion:

[0064] Comprehensive target detection results from lidar The constructed residual is:

[0065]

[0066] Among them is Noise covariance matrix for lidar target detection.

[0067] Comprehensive target detection position z from the camera τ , construct t0 and t i The residual between the laser radar target detection positions at the moment is:

[0068]

[0069] in, and are t0 and t i LiDAR target detection position at time z τ is the current camera target detection position, x τ is the camera target position, Σ τ Noise covariance matrix for camera detection.

[0070] S3.5 combines the residuals from S3.4 and finds the least squares solution for each target position:

[0071]

[0072] The Gauss-Newton method is used to solve the problem, thereby obtaining the precise position of the target and generating a trajectory.

[0073] Compared with the prior art, the advantages of the present invention are as follows: the present invention uses Gaussian process to perform state modeling on dynamic targets, and for the situation where the detection frequencies of laser radar and camera targets are inconsistent, Gaussian interpolation is used to fuse the camera target detection results with higher frequency into the laser radar target detection results with lower frequency, thereby making it possible to efficiently utilize the detection results of different sensors, and at the same time, a back-end processing method based on nonlinear optimization is used to optimize the trajectories of different targets, thereby obtaining a more accurate target trajectory. The present invention uses Gaussian process to perform state modeling on dynamic targets, and for the situation where the detection frequencies of laser radar and camera targets are inconsistent, Gaussian interpolation is used to fuse the camera target detection results with higher frequency into the laser radar target detection results with lower frequency, thereby making it possible to efficiently utilize the detection results of different sensors, and at the same time, a back-end processing method based on nonlinear optimization is used to optimize the trajectories of different targets, thereby obtaining a more accurate target trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is the overall system block diagram of the present invention.

[0075] Figure 2 Insert a schematic diagram of the Gaussian process observation state of the present invention. DETAILED DESCRIPTION

[0076] In order to deepen the knowledge and understanding of the present invention, the solution is described in detail below with reference to the accompanying drawings and embodiments.

[0077] Example: Figure 1 As shown, an improved multi-target tracking method based on Gaussian process includes the following steps:

[0078] S1: Run YOLOv5 and Point RCNN target detectors to detect and identify dynamic objects in each frame of camera image and lidar point cloud, and obtain the target detection box and the center position of the target detection box of the dynamic object;

[0079] S2: According to the LiDAR multi-target recognition frequency, the camera multi-target recognition frequency and the target uniform motion model prediction, the prior position of each target in the trajectory manager in the world coordinate system is obtained and the coordinates are converted to the camera coordinate system and the LiDAR coordinate system respectively. Combined with the center position of the current target detection frame obtained in S1, IOU (intersection-over-union) matching is performed to generate a similarity matrix and use the Hungarian algorithm to associate data between multiple detected targets and existing trajectories.

[0080] S3: For the targets associated in S2, the target position is first converted from the sensor coordinate system to the unified world coordinate system using the extrinsic parameter value between the sensor coordinate system and the world coordinate system. Secondly, the target prior position inferred in S2 and the current target observation position are combined to construct the residual to optimize the position and generate the trajectory. For the newly detected targets in S2, a new ID value is assigned, and the target position is converted from the sensor coordinate system to the unified world coordinate system using the extrinsic parameter value between the sensor coordinate system and the world coordinate system. Then, the Gaussian process is used to model the state of its trajectory. For the targets that cannot be associated in S2, if the number of times it cannot be associated is greater than the mismatch threshold, its trajectory is deleted. If it is less than the mismatch threshold, its trajectory is retained.

[0081] The target detection frame and the target detection frame center position model of the dynamic object in step S1 are as follows:

[0082] S1.1 uses the YOLOv5 algorithm to identify dynamic objects in each frame of the image, and uses the Point RCNN algorithm to identify dynamic objects in each frame of the lidar point cloud information. This method obtains the target detection box of the dynamic object in the image information and point cloud information respectively, and models the center position of the target detection box:

[0083] The dynamic object target detection box in the image information can be modeled as:

[0084] x=[u,v] T

[0085] Where [u, v] is the two-dimensional coordinate of the center of the target detection box in the image coordinate system;

[0086] The dynamic object target detection box in the lidar point cloud information can be modeled as:

[0087] x=[x,y,z] T

[0088] Where [x, y, z] are the three-dimensional coordinates of the center of the target detection frame in the laser radar coordinate system;

[0089] The method for obtaining the target trajectory prior position, IOU matching method and Hungarian algorithm in step S2 are as follows:

[0090] S2.1 assumes that the target moves at a constant speed. According to the laser radar multi-target recognition frequency, the prior position of each target trajectory in the world coordinate system is calculated. Assume that the laser radar target recognition time of the next frame is t i , the last position optimization time is t0, from which the current target prior position and velocity and prior covariance are obtained as follows:

[0091] x(t i )=Φ(t i ,t0)x(t0)

[0092] where x(t i ) is the prior position p(t i ) and speed Φ(t i ,t0) is the state transition function:

[0093]

[0094] Target prior covariance k(t i )for:

[0095]

[0096] Where Φ(t i ,t0),Φ(t i ,s) is the state transfer function, Q0 is the initial covariance of the position at time t0, Q c is the power spectrum density matrix, and F(s) is the noise control matrix:

[0097]

[0098] According to the camera multi-target recognition frequency, the prior position x(τ) of each target in the world coordinate system at the time of camera multi-target detection is calculated, t0<τ <t i :

[0099]

[0100] in t i The target position and velocity at the moment, Φ(τ,t i ) is the state transfer equation:

[0101]

[0102] S2.2 uses the external parameters between the world coordinate system and different sensor coordinate systems to convert the target trajectory prior position in the world coordinate system at different times to the image coordinate system and the lidar coordinate system respectively:

[0103] The prior position of the center of the target detection box in the image coordinate system is x=[u,v] T It can be calculated by the following formula:

[0104]

[0105] Where λ is the depth information of the center of the target detection frame, f x ,f y ,c x ,c y All are internal parameters of the camera, and [X, Y, Z] is the coordinate of the center of the target detection frame in the camera coordinate system.

[0106] The prior position of the center of the target detection box in the lidar coordinate system is x = [x, y, z] T , which can be directly obtained through the external parameters between the lidar coordinate system and the world coordinate system.

[0107] S2.3 performs IOU matching on each target prior detection frame and the current target detection frame. The IOU value can be calculated as follows:

[0108]

[0109] Where A is the area / volume of the target trajectory prior detection box, and B is the area / volume of the current target detection box;

[0110] According to the IOU value, the similarity matrix C can be generated as follows:

[0111]

[0112] where c ij is the IOU value between the i-th target trajectory prior detection box and the j-th current target detection box.

[0113] S2.4 uses the similarity matrix obtained in S2.3 and the Hungarian algorithm to match multiple detection targets, and the following three matching results can be obtained:

[0114] The first one: the target detection result of the current frame matches a specific trajectory.

[0115] The second type: the target detection result of the current frame cannot be associated with any trajectory.

[0116] The third type: The current trajectory cannot find any matching target detection results.

[0117] The Gaussian process modeling trajectory optimization method in step S3 is as follows:

[0118] S3.1 For the target associated with S2, first use the external parameter between the sensor coordinate system and the world coordinate system to convert the target position from the sensor coordinate system to the unified world coordinate system:

[0119] For the target in the image coordinate system, the target position is converted from the image coordinate system to the camera coordinate system through the camera intrinsic parameter value:

[0120]

[0121] Where [u, v] is the coordinate of the center of the target detection frame in the image coordinate system, λ is the depth information of the center of the dynamic target detection frame, and f x ,f y ,c x ,c y All are internal parameters of the camera, and [X, Y, Z] are the center coordinates of the target detection box in the camera coordinate system.

[0122] According to the external parameter value between the camera coordinate system and the world coordinate system, the detected target position is converted from the camera coordinate system to the world coordinate system;

[0123] For the target in the LiDAR point cloud coordinate system, the external parameter value between the LiDAR coordinate system and the world coordinate system is used to convert the detected target position from the LiDAR coordinate system to the world coordinate system;

[0124] S3.2 For the newly identified targets by the lidar and camera, the target detection results of both are combined to form the observation quantity M of the current system sensor t =[X,Y,Z] T :

[0125]

[0126] where [X c ,Y c ,Z c ] T is the measurement result of the camera, [X l ,Y l ,Z l ] T is the measurement result of the lidar, β is the reliability, β = 0.3.

[0127] The Gaussian process based on the linear time-varying stochastic difference equation (LTV_SDE) is used to describe the uniform motion state of the target:

[0128]

[0129] Where A(t) is the state transfer matrix, u(t) is the input matrix, F(t) is the noise control matrix, and w(t) is the motion noise matrix:

[0130]

[0131] w(t)~GP(0,Q c δ(tt ′ ))

[0132] where p(t) = [X, Y, Z] T , Q c is the power spectral density matrix, and δ(tt′) is the Dirkat function.

[0133] S3.3 For the case of a target that cannot be associated, if the number of times it cannot be associated is greater than the mismatch threshold, its trajectory is deleted; if it is less than the mismatch threshold, its trajectory is retained.

[0134] S3.4 For the case where the identified target and trajectory match, taking into account the difference in target detection frequencies between the lidar and camera, the lidar target detection frequency is first selected as the reference frequency for back-end state fusion:

[0135] Target detection results from lidar The constructed residual is:

[0136]

[0137] Among them is Noise covariance matrix for lidar target detection.

[0138] Detect the target position z from the camera τ ,like Figure 2 As shown, construct t0 and t i The residual between the laser radar target detection positions at the moment is:

[0139]

[0140] in, and are t0 and t i LiDAR target detection position at time z τ is the current camera target detection position, x τ is the camera target position, Σ τ Noise covariance matrix for camera detection.

[0141] S3.5 combines the residuals from S3.4 and finds the least squares solution for each target position:

[0142]

[0143] The Gauss-Newton method is used to solve the problem, thereby obtaining the precise position of the target and generating a trajectory.

[0144] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent replacements or substitutions made on the basis of the above technical solutions all fall within the scope of protection of the present invention.

Claims

1. An improved multi-target tracking method based on Gaussian process, characterized in that: The following steps are involved: S1: Run YOLOv5 and Point RCNN target detectors to detect and identify dynamic objects in each frame of camera image and lidar point cloud, and obtain the target detection box and the center position of the target detection box of the dynamic object; S2: According to the LiDAR multi-target recognition frequency, the camera multi-target recognition frequency and the target uniform motion model prediction, the prior position of each target in the trajectory manager in the world coordinate system is obtained and the coordinates are converted to the camera coordinate system and the LiDAR coordinate system respectively. The IOU matching is performed based on the center position of the current target detection frame obtained in S1, and a similarity matrix is ​​generated. The Hungarian algorithm is used to associate the data of multiple detected targets with the existing trajectories. S3: For the targets associated in S2, the target position is first converted from the sensor coordinate system to the unified world coordinate system using the external parameter value between the sensor coordinate system and the world coordinate system. Secondly, the residual is constructed by combining the target prior position inferred in S2 and the current target observation position to optimize the position and generate a trajectory. For the newly detected targets in S2, a new ID value is assigned, and the target position is converted from the sensor coordinate system to the unified world coordinate system using the external parameter value between the sensor coordinate system and the world coordinate system. Then, the Gaussian process is used to model the state of its trajectory. For the targets that cannot be associated in S2, if the number of times they cannot be associated is greater than the mismatch threshold, their trajectory is deleted. If it is less than the mismatch threshold, their trajectory is retained. The Gaussian process modeling and trajectory optimization method in step S3 is as follows: S3.1 For the target associated with S2, first use the external parameter between the sensor coordinate system and the world coordinate system to convert the target position from the sensor coordinate system to the unified world coordinate system: For the target in the image coordinate system, the target position is converted from the image coordinate system to the camera coordinate system through the camera intrinsic parameter value: Where [u, v] is the two-dimensional coordinate of the center of the target detection frame in the image coordinate system, λ is the depth information of the center of the dynamic target detection frame, and f x ,f y ,c x ,c y All are internal parameters of the camera, [X, Y, Z] is the center coordinate of the target detection frame in the camera coordinate system, According to the external parameter value between the camera coordinate system and the world coordinate system, the detected target position is converted from the camera coordinate system to the world coordinate system; For the target in the LiDAR point cloud coordinate system, the external parameter value between the LiDAR coordinate system and the world coordinate system is used to convert the detected target position from the LiDAR coordinate system to the world coordinate system; S3.2 For the newly identified targets by the lidar and camera, the target detection results of both are combined to form the observation quantity M of the current system front-end sensor t =[X,Y,Z] T : where [X c ,Y c ,Z c ] T is the measurement result of the camera, [X l ,Y l ,Z l ] T is the measurement result of the laser radar, β is the reliability, β=0.3, The Gaussian process based on the linear time-varying stochastic difference equation (LTV_SDE) is used to describe the uniform motion state of the target: Where A(t) is the state transfer matrix, F(t) is the noise control matrix, and w(t) is the motion noise matrix: w(t)~GP(0,Q c δ(t-t′)) where p(t) = [X, Y, Z] T , Q c is the power spectrum density matrix, δ(tt′) is the Dirkat function, S3.3 For the case of a target that cannot be associated, if the number of times it cannot be associated is greater than the mismatch threshold, its track is deleted; if it is less than the mismatch threshold, its track is retained. S3.4 For the case where the identified target and trajectory match, taking into account the difference in target detection frequencies between the lidar and camera, the lidar target detection frequency is first selected as the reference frequency for back-end state fusion: Target detection results from lidar The constructed residual is: Among them is is the noise covariance matrix of lidar target detection, Detect the target position z from the camera τ , construct t0 and t using Gaussian interpolation i The residual between the laser radar target detection positions at the moment is: in, and are t0 and t i LiDAR target detection position at time z τ is the current camera target detection position, x τ is the camera target position, Σ τ is the camera detection noise covariance matrix, S3.5 combines the residuals from S3.4 and finds the least squares solution for each target position: The Gauss-Newton method is used to solve the problem, thereby obtaining the precise position of the target and generating a trajectory.

2. The improved multi-target tracking method based on Gaussian process according to claim 1, characterized in that: The target detection frame and target detection frame center position model of the dynamic object in step S1 are as follows: S1.1 uses the YOLOv5 algorithm to identify dynamic objects in each frame of the image, and uses the Point RCNN algorithm to identify dynamic objects in each frame of the lidar point cloud information. This method obtains the target detection box of the dynamic object in the image information and point cloud information respectively, and models the center position of the target detection box: The dynamic object target detection box in the image information can be modeled as: x=[u,v] T Where [u,v] is the two-dimensional coordinate of the center of the target detection frame in the image coordinate system, The dynamic object target detection box in the lidar point cloud information can be modeled as: x=[x,y,z] T Where [x, y, z] is the three-dimensional coordinate of the center of the target detection box in the lidar coordinate system.

3. The improved multi-target tracking method based on Gaussian process according to claim 1, characterized in that: The method for obtaining the target trajectory prior position, IOU matching method and Hungarian algorithm in step S2 are as follows: S2.1 assumes that the target moves at a constant speed. According to the laser radar multi-target recognition frequency, the prior position of each target trajectory in the world coordinate system is calculated. Assume that the laser radar target recognition time of the next frame is t i , the last position optimization time is t0, from which the current target prior position, speed and prior covariance are obtained as follows: x(t i )=Φ(t i ,t0)x(t0) where x(t i ) is determined by the prior position p(t i ) and speed The target state composed of i ,t0) is the state transition function: Target prior covariance k(t i )for: Where Φ(t i ,t0),Φ(t i ,s) is the state transfer function, Q0 is the initial covariance at time t0, Q c is the power spectrum density matrix, and F(s) is the noise control matrix: According to the camera multi-target recognition frequency, the prior position x(τ) of each target in the world coordinate system at the time of camera multi-target detection is calculated, t0<τ <t i : in t i The target position and velocity at the moment, Φ(τ,t i ) is the state transfer equation: S2.2 uses the external parameters between the world coordinate system and different sensor coordinate systems to convert the target trajectory prior position in the world coordinate system at different times to the image coordinate system and the lidar coordinate system respectively: The prior position of the center of the target detection box in the image coordinate system is x=[u,v] T It can be calculated by the following formula: Where λ is the depth information of the center of the target detection frame, f x ,f y ,c x ,c y All are internal parameters of the camera, [X, Y, Z] is the coordinate of the center of the target detection frame in the camera coordinate system, The prior position of the center of the target detection box in the lidar coordinate system is x = [x, y, z] T , which can be directly obtained through the external parameters between the lidar coordinate system and the world coordinate system, S2.3 performs IOU matching on each target prior detection frame and the current target detection frame. The IOU value can be calculated as follows: Where A is the area / volume of the target trajectory prior detection box, and B is the area / volume of the current target detection box; According to the IOU value, the similarity matrix C can be generated as follows: where c ij is the IOU value between the i-th target trajectory prior detection box and the j-th current target detection box, S2.4 uses the similarity matrix obtained in S2.3 and the Hungarian algorithm to match multiple detection targets, and the following three matching results can be obtained: The first one: the target detection result of the current frame matches a specific trajectory. The second type: The target detection result of the current frame cannot be associated with any trajectory. The third type: The current trajectory cannot find any matching target detection results.

Images