Three-dimensional target tracking method, equipment, medium and product for nighttime driving scene

Abstract

The present invention provides a three-dimensional target tracking method for nighttime driving scenes, specifically a three-dimensional target tracking method of an interactive multi-model (IMM) combined with a lossless Kalman filter (UKF). The present invention effectively improves the continuity of target detection and tracking by estimating and tracking the results of a purely visual three-dimensional target detection method at night, and has strong robustness for tracking highly maneuverable targets, and has better effects for missed detections, false detections, occlusions, and estimation of motion speed and direction. The nonlinear performance of state estimation is effectively enhanced by using a lossless Kalman filter, and the problem of rapid changes in the target's motion direction and speed is overcome by using an interactive multi-model. The present invention can serve as the basis for advanced driver assistance functions of automobiles to achieve a nighttime driving driver assistance system with better performance and safety.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional space target tracking, and in particular to a three-dimensional target tracking method, device, medium and product for nighttime driving scenes. Background Art

[0002] With the widespread application of automotive ADAS systems (Advanced Driving Assistance System), their working hours have gradually developed from daytime hours to all-day hours. Since pure visual ADAS systems may encounter problems such as weak light, complex light refraction, and blurred image motion at night, they pose challenges to target detection and tracking. In this scenario, pure visual three-dimensional target detection will have more missed detections and false detections than during the day, and will then lose target information. These problems may cause ADAS systems to face severe challenges when driving at night.

[0003] In order to make full use of the information of the previous and next frame data to achieve better detection performance, and to track and estimate the speed of the target in the three-dimensional scene, the three-dimensional target tracking method can be used to improve the continuity of the target, supplement the missed detection and false detection of pure visual three-dimensional target detection in night driving scenes, and then perform state prediction.

[0004] At present, Kalman filters are generally used to update and predict target states. However, Kalman filters are linear and have poor adaptability to nonlinear motion models. They are difficult to track targets with variable motion and are frequently missed by target detectors at night. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a three-dimensional target tracking method for night driving scenes, which selects a probabilistic motion model based on an interactive multiple model (IMM) algorithm, uses an unscented Kalman filter (UKF) for nonlinear state estimation and updating, and adopts a constant velocity model (CV) and a constant turn rate and velocity amplitude model (CTRV) to describe the target motion, so as to realize three-dimensional target tracking and enhance the perception capability of a pure vision ADAS system in night scenes.

[0006] The present invention provides a three-dimensional target tracking method for nighttime driving scenes, comprising the following steps:

[0007] Obtain detection results. For the ADAS perception system, use the pure camera 3D target detection algorithm to obtain the detection results of the 3D target;

[0008] Expanding the target state, at the current time t, expanding the detection result and speed of the three-dimensional target as the target state to the three-dimensional space, and obtaining a uniform speed model and a constant turning rate and speed amplitude model;

[0009] Build an observation model, and use the motion model of the target at time t+1 to build the observation model corresponding to each three-dimensional target;

[0010] Estimating the motion model: for each three-dimensional target, using a lossless Kalman filter to estimate each motion model of the target, and obtaining the state variable estimation result of each motion model;

[0011] Dynamic model switching: Dynamic model switching is performed for the state variable estimation results of the same target using two motion models;

[0012] The weighted mixed estimation result is the target tracking result obtained by weighted mixing of the estimation results of each filter based on the interactive multi-model algorithm.

[0013] Furthermore, after the weighted mixed estimation result step, a filter state updating step is also included, and the lossless Kalman filter updates the filter state according to the estimation result of the probability mixing based on the interactive multi-model algorithm.

[0014] Furthermore, in the step of obtaining the detection result, the pure camera includes a monocular camera and a binocular camera; the three-dimensional target detection algorithm includes three-dimensional target detection based on monocular images and three-dimensional target detection based on multi-eye stereo vision; the detection result of the three-dimensional target is (x, y, z, ry, l, w, h, s), wherein (x, y, z) is the target coordinates of a single target in the self-vehicle coordinate system, ry is the angle between the target orientation of the top view of the single target and the x-axis of the self-vehicle coordinate system, (l, w, h) are the length, width and height of the single target, respectively, and s is the confidence score for classifying the single target.

[0015] Furthermore, in the step of extending the target state, the motion model equation is:

[0016]

[0017] in, is the state of a single target under the CV model at time t, It is the state of a single target under the CV model at time t, which consists of the detection result and state of the target;

[0018] where v x ,v y ,v z Indicates the speed of the three-dimensional target along the xyz direction in the vehicle coordinate system; Where v, ω represents the forward speed and yaw rate of the three-dimensional target in the vehicle coordinate system;

[0019] is the state of each model of the target at time t+1, F CV ,F CTRV is the state transfer matrix of each model, w t is a process noise with a mean of 0 at time t, a covariance matrix of Q, and a normal distribution.

[0020] Furthermore, in the step of building the observation model, the observation model equation is:

[0021] z t+1 =Hx t+1 +v t+1

[0022] Among them, z t+1 is the state measurement value of each target at time t+1, H is the measurement matrix, including the CV model and the CTRV model, v t+1 The measurement noise has a mean of 0 at time t+1, a covariance matrix of R, and follows a normal distribution.

[0023] Furthermore, in the model dynamic switching step, a Markov chain is used to perform model dynamic switching, and the Markov matrix is:

[0024]

[0025] Among them, π ij represents the probability of switching from model i to model j.

[0026] Furthermore, in the weighted mixed estimation result step, the formula for weighted mixing of the estimation results of each filter based on the interactive multi-model algorithm is:

[0027]

[0028] in, is the probability of the ith model at time t.

[0029] An electronic device, comprising: a processor;

[0030] A memory; and a program, wherein the program is stored in the memory and is configured to be executed by a processor, the program including a method for executing a three-dimensional target tracking method for a nighttime driving scene.

[0031] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a three-dimensional target tracking method for nighttime driving scenes.

[0032] A computer program product includes a computer program / instruction, which implements a three-dimensional target tracking method for nighttime driving scenes when executed by a processor.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention uses historical data before time t to predict and track time t+1, and uses IMM to fuse multiple motion models, solving the problem that targets with high mobility are difficult to track. It improves the continuity of the target in time series, has strong robustness to sudden changes in target movement direction and speed, and can predict the three-dimensional speed of the target.

[0035] The present invention utilizes a lossless Kalman filter to effectively improve the nonlinear performance of target state prediction, effectively improves the accuracy of target state estimation, and has a higher accuracy rate for position and speed estimation, which can be further used for other advanced vehicle driver assistance functions.

[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 It is a flow chart of the three-dimensional target tracking method for nighttime driving scene of the present invention;

[0039] Figure 2 Schematic diagram of the tracking results of the three-dimensional target tracking method of the present invention on the KITTI dataset. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0041] Three-dimensional target tracking methods for nighttime driving scenes, such as Figure 1 As shown, the following steps are included:

[0042] Obtain the detection results. For the ADAS perception system, the detection results of the three-dimensional target obtained by using the pure camera three-dimensional target detection algorithm are (x, y, z, ry, l, w, h, s), where (x, y, z) is the target coordinates of a single target in the ego-vehicle coordinate system, ry is the angle between the target orientation of the top view of a single target and the x-axis of the ego-vehicle coordinate system, (l, w, h) are the length, width and height of a single target, respectively, and s is the confidence score for the classification of a single target. Pure cameras include monocular cameras, binocular cameras, etc.; three-dimensional target detection algorithms include three-dimensional target detection based on monocular images, three-dimensional target detection based on multi-eye stereo vision, etc.

[0043] Expand the target state. At the current time t, the detection result and speed of the three-dimensional target are expanded into the three-dimensional space as the target state to obtain the uniform speed model and the constant turning rate and speed amplitude model. The motion model equation is:

[0044]

[0045] in, is the state of a single target under the CV model at time t, It is the state of a single target under the CV model at time t, which consists of the detection result and state of the target.

[0046] Right now where v x ,v y ,v z Indicates the speed of the three-dimensional target along the xyz direction in the vehicle coordinate system; Where v, ω represents the forward speed and yaw rate of the three-dimensional target in the vehicle coordinate system;

[0047] is the state of each model of the target at time t+1, F CV ,F CTRV is the state transfer matrix of each model, w t is a process noise with a mean of 0 at time t, a covariance matrix of Q, and a normal distribution.

[0048] Among them, the state transfer equation of each model is:

[0049]

[0050] The process noise covariance matrix Q is:

[0051]

[0052] The prior estimated covariance matrix P is:

[0053]

[0054] Build an observation model, and use the motion model of the target at time t+1 to build the observation model corresponding to each three-dimensional target; the observation model equation is:

[0055] z t+1 =Hx t+1 +v t+1

[0056] Among them, z t+1 is the state measurement value of each target at time t+1, H is the measurement matrix, including the CV model and the CTRV model, v t+1 The measurement noise has a mean of 0 at time t+1, a covariance matrix of R, and follows a normal distribution.

[0057] Estimating the motion model: for each three-dimensional target, using a lossless Kalman filter to estimate each motion model of the target, and obtaining the state variable estimation result of each motion model;

[0058] Model dynamic switching, for the state variable estimation results of the two motion models for the same target, the Markov chain is used to perform model dynamic switching, and the Markov matrix is:

[0059]

[0060] Among them, π ij represents the probability of model i switching to model j. In this embodiment, 3DIoU is used to form a probability matrix.

[0061] The weighted mixed estimation result, based on the interactive multi-model algorithm, performs weighted mixing of the estimation results of each filter according to the following formula to obtain the target tracking result.

[0062]

[0063] in, is the probability of the ith model at time t; since there are two models, r = 2.

[0064] Then the filter mixture estimation result is:

[0065]

[0066] The probability Updated by:

[0067]

[0068] is the likelihood function that model i follows a normal distribution:

[0069]

[0070] and is the model i at time t+1, is the covariance. The mean is 0. Then the state result is:

[0071]

[0072] The Interacting Multiple Model (IMM) algorithm can adaptively match various motion models and adjust the probability of each model in real time. It can effectively judge the motion model of highly maneuverable targets, thereby improving the target tracking performance in night driving scenes.

[0073] The filter state is updated. The lossless Kalman filter then updates the filter state according to the estimation results of probability mixing based on the interactive multi-model algorithm. Figure 2 This is an example of the result of the present invention, and it can be seen that the tracking effect of the target frame is good.

[0074] The final target tracking results can serve as the basis for other vehicle assisted driving functions, including but not limited to pedestrian detection, vehicle tracking, adaptive cruise control, emergency braking and other functions.

[0075] The three-dimensional target tracking method for nighttime driving scenes provided by the present invention first establishes multiple target motion models, then uses multiple UKFs to estimate the target state, and uses IMMs to perform probability weighting on multiple estimation results to obtain a unified prediction estimation result. Finally, the UKF is updated using the unified estimation result, and the result is output to other functional modules to realize advanced driver assistance functions.

[0076] The present invention predicts and tracks the results of pure visual 3D target detection at night through IMM and UKF, solves the fragmentation problem of pure visual 3D target detection at night, and performs effective motion estimation for highly maneuverable targets. In addition, the detection and tracking results can be used as the basis for other advanced driver assistance functions of automobiles to build a more advanced functional framework.

[0077] An electronic device, comprising: a processor;

[0078] A memory; and a program, wherein the program is stored in the memory and is configured to be executed by the processor, the program including a method for executing a three-dimensional target tracking method for a nighttime driving scene.

[0079] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a three-dimensional target tracking method for nighttime driving scenes.

[0080] A computer program product includes a computer program / instruction, which implements a three-dimensional target tracking method for nighttime driving scenes when executed by a processor.

[0081] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the drawings and above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A three-dimensional target tracking method for nighttime driving scenes, characterized in that: The following steps are involved: Obtain detection results. For the ADAS perception system, use the pure camera 3D target detection algorithm to obtain the detection results of the 3D target; In the step of obtaining the detection result, the pure camera includes a monocular camera and a binocular camera; the 3D target detection algorithm includes a 3D target detection based on a monocular image and a 3D target detection based on multi-eye stereo vision; the detection result of the 3D target is (x, y, z, ry, l, w, h, s), where (x, y, z) is the target coordinates of a single target in the self-vehicle coordinate system, ry is the angle between the target orientation of the top view of the single target and the x-axis of the self-vehicle coordinate system, (l, w, h) are the length, width and height of the single target respectively, and s is the confidence score for the classification of the single target; Expanding the target state, at the current time t, expanding the detection result and speed of the three-dimensional target as the target state to the three-dimensional space, and obtaining a uniform speed model and a constant turning rate and speed amplitude model; In the extended target state step, the motion model equation is: in, is the state of a single target under the CV model at time t, It is the state of a single target under the CTRV model at time t, which consists of the detection result and state of the target; is the state of each model of the target at time t+1, F CV ,F CTRV is the state transfer matrix of each model, w t is the process noise with mean 0 at time t, covariance matrix Q, and normal distribution; Build an observation model, and use the motion model of the target at time t+1 to build the observation model corresponding to each three-dimensional target; In the step of building the observation model, the observation model equation is: z t+1 =Hx t+1 +v t+1 Among them, z t+1 is the state measurement value of each target at time t+1, H is the measurement matrix, including the CV model and the CTRV model, v t+1 is the measurement noise with a mean of 0 at time t+1, a covariance matrix of R, and a normal distribution. Xt+1 is the state of the target under the CV model at time t+1 or the state of the target under the CTRV model at time t+1. Estimating the motion model: for each three-dimensional target, using a lossless Kalman filter to estimate each motion model of the target, and obtaining the state variable estimation result of each motion model; Dynamic model switching: Dynamic model switching is performed for the state variable estimation results of the same target using two motion models; In the model dynamic switching step, a Markov chain is used to perform model dynamic switching, and the Markov matrix is: Among them, π ij represents the probability of model i switching to model j; Weighted mixed estimation result: the target tracking result is obtained by weighted mixing of the estimation results of each filter based on the interactive multi-model algorithm; In the weighted mixed estimation result step, the formula for weighted mixing of the estimation results of each filter based on the interactive multi-model algorithm is: in, is the probability of the ith model at time t.

2. The three-dimensional target tracking method for nighttime driving scenes according to claim 1, characterized in that: The weighted mixed estimation result step also includes a filter state update step, and the lossless Kalman filter updates the filter state according to the estimation result of probability mixing based on the interactive multi-model algorithm.

3. The three-dimensional target tracking method for nighttime driving scenes according to claim 1, characterized in that: In the step of extending the target state, where v x ,v y ,v z Indicates the speed of the three-dimensional target along the xyz direction in the vehicle coordinate system; Where v, ω represents the forward speed and yaw rate of the three-dimensional target in the vehicle coordinate system; 4. An electronic device, characterized in that include: processor; Memory; and a program, wherein the program is stored in the memory and configured to be executed by a processor, the program comprising instructions for executing the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to perform the method according to any one of claims 1 to 3.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images