A binocular camera-based radar and vision fusion event detection method

Abstract

The application provides a kind of event detection method based on binocular camera's radar vision fusion, comprising the following steps: target detection, utilize binocular camera and a radar to collect road target information;Data preprocessing, binocular camera is mapped into radar coordinate system according to the measurement value of calibration coordinate system;Track prediction, obtain the position and motion state of target future time, form target track;Target association, within a certain time difference, the radar and binocular camera target that meet threshold, lane number condition are paired;Track update, real-time update track to data preprocessing, track prediction, target association step repeatedly;Track start, based on the point trail obtained by screening generates new track, completes track initialization;Attribute calculation, further processing obtains smooth, stable, safe track;Track output, output processed track and feature information result, to realize target vehicle dynamic tracking query and visualization;For driving trajectory and vehicle feature information visualization provides faster data support, realizes highway road surface target vehicle real-time tracking and information query service efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent environment perception, and particularly relates to a radar and vision fusion event detection method based on a binocular camera. BACKGROUND

[0002] Traffic monitoring system is one of the most important parts of intelligent transportation system, which provides data support for intelligent transportation system traffic flow statistics, traffic flow detection, accident detection, speed measurement, vehicle information query and positioning functions. Intelligent roadside device as the basic component module of traffic monitoring system, has a decisive role on the detection accuracy, efficiency and stability of traffic monitoring system.

[0003] In the highway road vehicle monitoring scene, the reliability of using only a single sensor for environment perception is poor, and is limited by the characteristics of the sensor itself. In complex scenes, the application cannot achieve the expected detection accuracy. The camera in the roadside device can visualize the road information and classify the target, but it is easy to be affected by target occlusion and external environmental interference such as weather, and the performance indicators such as camera resolution, viewing angle, effective recognition distance have become the bottleneck restricting its development. Millimeter wave radar is not easy to be affected by weather and light when applied, but it is difficult to capture the feature information of the vehicle. If the roadside device is equipped with different sensing devices such as monitoring camera, radar, microwave vehicle detector at the same time, the installation cost is high, the maintenance is difficult, and the data is seriously isolated.

[0004] Nowadays, using multi-sensor fusion technology to realize the performance complementation of different sensors has become a common solution to improve the performance of road vehicle target detection and tracking. Among them, millimeter wave radar is used for target detection, vehicle flow statistics, vehicle speed detection, camera is used to obtain the license plate, model, body color of the vehicle, and the information collected by the two is fused to provide stable and efficient data support for all-weather, super-vision, multi-dimensional real-time monitoring of vehicles.

[0005] Most of the radar and vision fusion detection methods have good detection effects in congestion, traffic flow statistics, occupancy, vehicle position, etc., but the detection effect in other aspects such as traffic events is not ideal. Improving the detection effect of traffic events requires that the roadside equipment has higher precision and efficient data processing methods, but at present, in terms of hardware, the traditional monocular camera has the problems of being unable to determine the real size of the target, having scale uncertainty; needing a large sample feature database, otherwise it will lead to inaccurate identification of vehicle types, objects and obstacles; having weak ranging ability for dynamically changing targets; being unable to collect data in the case of lens contamination, having poor robustness, high maintenance cost in the later period; the detection range of the monocular camera cannot match the detection range of the radar, etc. In terms of data processing, most detection methods process the data of the camera and the radar separately, concentrate the display, and only achieve the effect of "combination" of data, resulting in low efficiency and poor precision of vehicle information query and event detection.

[0006] A road target detection method based on binocular camera and laser radar fusion is disclosed in Chinese patent application with publication number CN114463303A, which uses two left and right cameras and a laser radar to collect front road target information; obtains binocular disparity through a binocular stereo matching algorithm; uses a neural network based on monocular vision to obtain image target category and two-dimensional position information; combines binocular disparity and monocular vision detection information to obtain front target visual three-dimensional detection results; this method can output the corrected road target information, but cannot realize dynamic tracking and visualization of target vehicles through track prediction, target association, track update, track initiation, attribute calculation and track output. SUMMARY

[0007] The purpose of the present application is to provide a radar and vision fusion event detection method based on a binocular camera, which uses a binocular camera to match the detection range of vision and radar, improves the dynamic target ranging ability of the radar and vision integrated machine, completes target-level fusion of the vehicle dynamic motion information collected by the radar and the vehicle feature information collected by the camera, and realizes dynamic tracking and information query of vehicles on the highway.

[0008] To achieve the above technical purposes and effects, the present application realizes the following technical solutions:

[0009] The present application provides a radar and vision fusion event detection method based on a binocular camera, which includes the following steps:

[0010] Step S1: target detection, using a binocular camera and a radar to collect road target information, obtaining three-dimensional coordinate values through a binocular stereo matching algorithm, i.e. depth information; using a visual target detection neural network to obtain the category and two-dimensional position information of the target in the left image as left camera visual detection information; combining the depth information and the left camera visual detection information to obtain a three-dimensional visual detection result of the target; the radar obtains vehicle position coordinates and motion state information;

[0011] Step S2: data preprocessing, mapping the coordinate system measurement values of the collected target to the radar coordinate system through a transformation model according to the internal parameters K, rotation matrix R and translation matrix t of the binocular camera, realizing spatial synchronization; incrementally sorting the target observation data collected by the binocular camera and the radar according to the measurement accuracy within the same time slice, interpolating and extrapolating the data at the highest accuracy observation time point to the lowest accuracy time point according to the motion state, realizing time synchronization of the binocular camera and the radar; performing effective target consistency test according to the certain range limit principle of the change of the target motion state within the adjacent collection period of the radar, eliminating the interference of false targets and empty targets; the transformation model is as follows:

[0012]

[0013] In the formula, K is the internal parameter, R is the rotation matrix, and t is the translation matrix;

[0014] Step S3: track prediction, based on the constant acceleration model of the filter module, calculating the corresponding feature prediction value at the next time according to the x / y coordinates, x / y direction speed and x / y direction acceleration value at the historical time; based on the constant turning rate and speed model of the filter module, calculating the corresponding feature prediction value at the next time according to the speed, yaw angle and angular velocity at the historical time; counting the prediction value to obtain the position and motion state of the target at the future time, forming the target track;

[0015] Step S4: target association, setting the matching threshold according to the different longitudinal distances of the measured targets, matching the time and lane number information contained in the radar track with the binocular camera, and pairing the radar and binocular camera targets that meet the threshold and lane number conditions within a certain time difference;

[0016] Step S5: track updating, updating the track coordinates and motion state prediction value to the corresponding vehicle according to the target association result, further processing the vehicle track, and repeatedly performing the data preprocessing, track prediction and target association steps according to the radar and binocular camera collection information of the next time slice to update the track in real time;

[0017] Step S6: track initiation, according to the appearance and all track irrelevant point, based on the established correlation gate, it is judged whether the point meets the space-time constraints of the track, the generated measurement point is screened, the real measurement points and noise clutter of other targets are filtered, through the DBSCAN clustering algorithm, the new track is generated based on the screened point, and the track initialization is completed;

[0018] Step S7: attribute calculation, according to the track initialization result and target track, the starting and future time reference points are generated, the priority calculation, MotionStatus, Smooth and Confidence modules are calculated and optimized to further process to obtain smooth, stable and safe track;

[0019] Step S8: track output, based on binocular camera data, the region of interest is calculated, the corresponding target is selected according to the candidate region, the processed track and feature information result are output, so as to realize target vehicle dynamic tracking query and visualization.

[0020] As a further improvement of the application, the track initiation in step S6 further includes a constraint rule, when the maximum speed and the minimum speed satisfy v max ≥v min , and the measured or calculated speed is between the maximum speed and the minimum speed, that is, for a track initiation, the constraint condition is:

[0021]

[0022] In the formula, r k is the position vector of the kth scanning target period, t k is the sampling time;

[0023] The acceleration is less than the maximum acceleration a max , when more than one echo satisfies the acceleration constraint, the echo with the minimum acceleration is used to form the target track, and the constraint condition is:

[0024]

[0025] In the formula, r k is the position vector of the kth scanning target period, t k is the sampling time.

[0026] As a further improvement of the application, the attribute calculation in step S7 further includes a reference line calculation module, the reference line cost function is divided into a similar cost of the original path point, a smooth cost and a compact cost, there are three point tracks, the ith point p i (x i ,y i ) is an unknown point, p ir (xir ,y ir ) is a known point, and the similarity cost function is:

[0027]

[0028] The smoothing cost function is: f2 = [(x1+x3-2x2) 2 +(y1+y3-2y2) 2 ];

[0029] The compact cost function is:

[0030] The total cost function is: f = w1f1+w2f2+w3f3, wherein w i is the weight of different sub-functions.

[0031] As a further improvement of the present application, it further comprises a track management step, which comprises sensor data table establishment, sensor data table termination, sensor data table updating, fused track table establishment, fused track table termination, association relationship establishment, association relationship release and ID pool management.

[0032] Advantages of the present application:

[0033] 1. The radar and vision fusion event detection method based on a binocular camera provided by the present application is based on binocular camera video data acquisition, so that the visual detection range and radar matching are realized, the dynamic target detection precision is higher, the response is faster, the highway pavement vehicle feature information can be efficiently collected, and the method has the advantages of intuitiveness, simplicity, rapidity, accuracy and the like.

[0034] 2. The radar and vision fusion event detection method based on a binocular camera provided by the present application performs data fusion on the pavement vehicle information collected by the radar and the camera, provides faster data support for vehicle trajectory and vehicle feature information visualization, and realizes efficient highway pavement target vehicle real-time tracking and information query service.

[0035] 3. The present application screens the radar data monitoring point and the filter module vehicle motion prediction value, eliminates the clutter interference in radar detection, and performs smoothing processing and attribute calculation on the to-be-output trajectory, so that the final track display result is more accurate and reliable.

[0036] 4. The present application has a track management module, which stores key information such as sensor data, fused tracks and association relationships, provides data query services for management personnel, facilitates later maintenance, and improves data management efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 Fig. 1 is a flow chart of the radar and vision fusion event detection method based on a binocular camera according to the present application.

[0038] Figure 2 Flow chart of target detection step described in the present application;

[0039] Figure 3 Schematic diagram of target association step described in the present application;

[0040] Figure 4 Schematic diagram of track management function module of the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0042] An event detection method based on binocular camera and radar is provided in the embodiments of the present application, as shown in Figure 1 which includes the following steps:

[0043] Step S1: target detection, as shown in Figure 2 a binocular camera and a radar are used to collect road target information, a three-dimensional coordinate value is obtained through a binocular stereo matching algorithm, i.e. depth information; a visual target detection neural network is used to obtain the category and two-dimensional position information of the target in the left image, as left camera visual detection information; the target three-dimensional visual detection result is obtained by combining the depth information and the left camera visual detection information; the radar obtains the vehicle position coordinate and motion state information;

[0044] Step S2: data preprocessing, the binocular camera is used to map the coordinate system measurement value of the collected target into the radar coordinate system according to the calibrated internal parameter K, rotation matrix R and translation matrix t through a conversion model, to realize spatial synchronization; the target observation data collected by the binocular camera and the radar in the same time slice is incrementally sorted according to the measurement accuracy, the data at the highest accuracy observation time is respectively interpolated and extrapolated to the lowest accuracy time point according to the motion state, to realize time synchronization of the binocular camera and the radar; the effective target consistency test is performed according to the certain range limit principle of the target motion state change in the adjacent collection period of the radar, to eliminate the interference of false targets and empty targets; the conversion model is as follows:

[0045]

[0046] In the formula, K is the internal parameter, R is the rotation matrix, and t is the translation matrix.

[0047] Step S3: Track prediction, according to the vehicle historical track data collected by the radar, based on the filter module constant acceleration (CA) model, the x / y coordinates, x / y direction speed, x / y direction acceleration value at the historical time are used to calculate the corresponding feature prediction value at the next time; based on the filter module constant turning rate and velocity (CTRV) model, the speed, yaw angle and angular velocity at the historical time are used to calculate the corresponding feature prediction value at the next time; the prediction value is counted to obtain the position and motion state of the target at the future time, and the target track is formed.

[0048] Step S4: Target association, the matching threshold is set according to the different measured target longitudinal distance, the time and lane number information contained in the radar track are matched with the camera, the radar and camera targets within a certain time difference and meeting the threshold and lane number conditions are paired, and the matched targets are associated according to the pairing result. For example, as shown in Figure 3 , the radar target and camera target are calculated using cosine similarity, assuming that the number of measured radar targets is M, each radar target has p attributes, the number of camera measured targets is N, and each camera measured target has q attributes, then M*N times of matching are required, each time of matching, the cosine similarity of p-dimensional vector and q-dimensional vector is calculated, the similarity cos(theta) is obtained, the closer the similarity is to 1, the greater the association of the two targets is, when the association parameter exceeds the set threshold (such as 0.9995), the target is considered to be matched.

[0049] Step S5: Track update, according to the target association result, the track coordinates and motion state prediction value are updated to the corresponding vehicle, the track of the vehicle is further processed, and the track is updated in real time according to the radar camera collection information of the next time slice.

[0050] Step S6: Track initiation, if a point track that is not related to all tracks appears, whether the point track meets the space-time constraints of the track is judged according to the established correlation gate, the generated measurement point track is screened, and the real measurement points and noise clutter of other targets are filtered; a new track is generated based on the screened point track through the DBSCAN clustering algorithm, and the track initialization is completed.

[0051] Specifically, the track initiation constraint rule is: ① when the maximum speed and the minimum speed meet v max ≥v min ≥0, and the measured or calculated speed is between the maximum speed and the minimum speed, that is, for a track initiation, the constraint is where r k is the position vector of the kth scanning target period, tk is the sampling time. 2. The acceleration should be less than the maximum acceleration a max If there are more than one echo satisfying the acceleration constraint, the echo with the minimum acceleration is used to form the target track, and the constraint is

[0052] Step S7: attribute calculation, according to the track initialization result and the target track, a starting and future time reference point is generated, and a smooth, stable and safe track is obtained through calculation optimization further processing by priority calculation, MotionStatus, Smooth and Confidence modules;

[0053] Specifically, the reference line cost function is divided into a similarity cost with the original path point, a smooth cost and a compact cost, assuming that there are three point tracks, the i-th point p i (x i ,y i ) is an unknown point, and p ir (x ir ,y ir ) is a known point, and the similarity cost function is The smooth cost function is f2=[(x1+x3-2x2) 2 +(y1+y3-2y2) 2 ], and the compact cost function is The total cost function is f=w1f1+w2f2+w3f3, wherein w i is the weight of different sub-functions.

[0054] Step S8: track output, based on camera data, a Region of Interest (ROI) is calculated, a corresponding target is selected according to the candidate region, and a processed track and feature information result are output, so as to realize target vehicle dynamic tracking query and visualization.

[0055] In one embodiment, the track starting in step S6 further includes a constraint rule, when the maximum speed and the minimum speed satisfy v max ≥v min ≥0, and the measured or calculated speed is between the maximum speed and the minimum speed, that is, for a track starting, the constraint condition is:

[0056]

[0057] In the formula, r k is the position vector of the k-th scanning target period, and t k is the sampling time;

[0058] The acceleration is less than the maximum acceleration a maxWhen there are more than one echo satisfying the acceleration constraint, the echo with the minimum acceleration is used to form the target track, and the constraint condition is:

[0059]

[0060] wherein r k is the position vector of the kth scan target period, t k is the sampling time.

[0061] In one embodiment, the attribute calculation in step S7 further comprises a reference line calculation module, and the reference line cost function is divided into a similarity cost with the original path point, a smooth cost and a compact cost, and there are three point tracks, the i th point p i (x i ,y i ) is an unknown point, and p ir (x ir ,y ir ) is a known point, and the similarity cost function is:

[0062]

[0063] The smooth cost function is f2 = [(x1+x3-2x2) 2 +(y1+y3-2y2) 2 ];

[0064] The compact cost function is:

[0065] The total cost function is f = w1f1+w2f2+w3f3, wherein w i is the weight of different sub-functions.

[0066] In one embodiment, as shown in Figure 4 , it further comprises a track management module, and the track management module comprises a sensor data table establishment, a sensor data table termination, a sensor data table update, a fused track table establishment, a fused track table termination, a correlation relationship establishment, a correlation relationship release and an ID pool management.

[0067] Thus, several aspects of at least one embodiment of the present application are described, and it is understood that various changes, modifications and improvements can be made by those skilled in the art. Such changes, modifications and improvements are intended to be within the spirit and scope of the present application.

Claims

1. A binocular camera based LEO-EO fusion event detection method, characterized in that, Comprising the following steps: Step S1: target detection, using a binocular camera and a radar to collect road target information, obtaining three-dimensional coordinate values through a binocular stereo matching algorithm, i.e. depth information; Using a visual target detection neural network to obtain the category and two-dimensional position information of the target in the left image as left camera visual detection information; combining the depth information and the left camera visual detection information to obtain a three-dimensional visual detection result of the target; The radar obtains vehicle position coordinates and motion state information; Step S2: data preprocessing, mapping the coordinate system measurement values of the collected target to the radar coordinate system through a transformation model according to the internal parameters K, rotation matrix R and translation matrix t of the binocular camera, realizing spatial synchronization; in the same time slice, the target observation data collected by the binocular camera and the radar are incrementally sorted according to the measurement accuracy, and the data at the highest accuracy time point is respectively interpolated and extrapolated according to the motion state to realize time synchronization of the binocular camera and the radar; according to the principle of a certain range limit of target motion state change within the adjacent collection period of the radar, effective target consistency test is carried out to eliminate the interference of false targets and empty targets; the transformation model is as follows: In the formula, K is the internal parameter, R is the rotation matrix, and t is the translation matrix; Step S3: track prediction, based on the constant acceleration model of the filter module, the x / y coordinates, x / y direction speed and x / y direction acceleration value at the historical time are used to calculate the corresponding feature prediction value at the next time; Based on the constant turning rate and speed model of the filter module, the speed, yaw angle and angular velocity at the historical time are used to calculate the corresponding feature prediction value at the next time; Statistical prediction value, obtain the position and motion state of the target at the future time, form the target track; Step S4: target association, according to the different measurement target longitudinal distance, the matching threshold is set, according to the time and lane number information contained in the radar track and the binocular camera, the radar and binocular camera targets that meet the threshold and lane number conditions within a certain time difference are matched; Step S5: track update, according to the target association result, the track coordinates and motion state prediction value are updated to the corresponding vehicle, the track of the vehicle is further processed, and the data preprocessing, track prediction and target association steps are repeated according to the radar and binocular camera collection information of the next time slice to update the track in real time; Step S6: track initiation, according to the point track that is not related to all tracks, based on the established correlation gate, it is judged whether the point track meets the space-time constraints of the track, the generated measurement point track is screened, the real measurement points and noise clutter of other targets are filtered, through the DBSCAN clustering algorithm, a new track is generated based on the screened point track, and the track initialization is completed; Step S7: attribute calculation, according to the track initialization result and the target track, the starting and future time reference points are generated, and the smooth, stable and safe track is further processed through calculation optimization. ​ Step S8: track output, based on binocular camera data to calculate the region of interest, according to the candidate region selection corresponding target, output processed track and feature information results, so as to realize target vehicle dynamic tracking query and visualization.

2. The binocular camera based laser-radar fusion event detection method of claim 1, wherein: The track initiation in step S6 also comprises a constraint rule, when the maximum speed and the minimum speed satisfy v max ≥ v min ≥ 0, and the measured or calculated speed is between the maximum speed and the minimum speed, i.e. for a track initiation, the constraint condition is: wherein r k is the position vector of the kth scan target period, t k is the sampling time; acceleration is less than the maximum acceleration a max When more than one echo satisfies the acceleration constraint, the echo with the smallest acceleration is used to form the target track, with the constraint that where r k is the position vector of the kth scan target period, t k is the sampling time.

3. The binocular camera based laser-radar fusion event detection method of claim 1, wherein: The attribute calculation in step S7 further comprises a reference line calculation module, the reference line cost function is divided into a similarity cost with the original path point, a smooth cost and a compact cost, there are three point traces, the i-th point p i (x i ,y i ) is an unknown point, p ir (x ir ,y ir ) is a known point, and the similarity cost function is: The smooth cost function is: f2 = [(x1+x3-2x2) 2 +(y1+y3-2y2) 2 ] The compact cost function is: The total cost function is: f = w1f1 + w2f2 + w3f3, where w i are the weights of different sub-functions.

4. The binocular camera based laser-radar fusion event detection method of claim 1, wherein: Also includes track management steps, the track management steps include sensor data table establishment, sensor data table termination, sensor data table update, fusion track table establishment, fusion track table termination, association relationship establishment, association relationship release and ID pool management.

Images