Vehicle detection method based on radar data motion analysis

Abstract

The application provides a vehicle detection method based on millimeter wave radar and video data motion analysis, and aims to solve the problem that a single sensor cannot accurately detect vehicles in multiple scenes and all-weather conditions. The method comprises the following steps: collecting and calibrating video and millimeter wave radar data; generating a lane hotspot map based on multiple millimeter wave radar data; a radar and video fusion strategy based on target motion state analysis; calculation and transmission of traffic statistics; and pushing of real-time video stream. The application uses a more reasonable processing method, which improves the accuracy and robustness of target detection through motion analysis and adaptive fusion of millimeter wave radar and video data, is suitable for low-power edge computing devices, and reduces the cost of the device. At the same time, the traffic flow, queue length, parking frequency, coil existence and other statistics are balanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and proposes a vehicle detection method based on radar and video data motion analysis based on video sensors and millimeter wave radars. BACKGROUND

[0002] With the development of science and technology and the progress of society, vehicle detection technology in China has developed rapidly, urban traffic construction has gradually become intelligent, and excellent results have been achieved. Today, a variety of vehicle detection equipment is applied to urban road sections, such as ground magnetic coils, cameras, millimeter wave radars, laser radars, infrared sensors, ultrasonic waves, etc., for urban traffic signal control.

[0003] Using a single sensor for traffic area detection has its own instability; video sensors are greatly affected by weather and scene, and their detection effect is limited in rainy, snowy and foggy weather and at night; ground magnetic coil sensors are less affected by the climate, but the installation process will damage the road surface; millimeter wave radar sensors have short wavelengths and can accurately obtain the position information and speed information of the target, but they will miss detection of stationary targets and low-speed targets; laser radars have good position information detection effect, but lack speed detection, and have high maintenance costs. Therefore, using multi-sensor fusion for vehicle detection has become a major trend.

[0004] The results of vehicle detection include traffic flow, pulse, queue length, stop count, time occupancy, space occupancy, etc. Therefore, this technology not only depends on target detection, but also depends on target tracking methods; traditional single video tracking methods are divided into methods that depend on target detection and methods that do not depend on target detection, and the former has high accuracy, but in complex scenes and bad weather, the detection results are often unsatisfactory, and the real-time and accuracy of the target detection results are greatly dependent on the robustness of the neural network model to the scene and the computing power of the device; traditional single radar tracking methods are forced to rely on the characteristics of the radar, and the dynamic and static switching process is easy to cause a large amount of retention. SUMMARY

[0005] The technical problem to be solved by the present application is the defects of single sensor vehicle detector, and a vehicle detection method based on radar and video data motion analysis combining the advantages of video sensors and millimeter wave radar sensors is proposed.

[0006] The technical problem to be solved by the present application is solved by the following technical scheme, and the present application is a vehicle detection method based on radar and video data motion analysis, which is characterized by comprising the following steps:

[0007] S1, video and millimeter wave radar data acquisition and calibration;

[0008] S2, generating a lane hotspot map based on multi-frame millimeter wave radar data;

[0009] S3, a radar and vision fusion strategy based on target motion state analysis;

[0010] S4, calculation and transmission of traffic statistics;

[0011] S5, pushing of real-time video stream.

[0012] The vehicle detection method based on radar and vision data motion analysis provided by the application further preferably has the following steps in the video and millimeter wave radar data acquisition and calibration of S1:

[0013] S1.1, acquisition of video data and millimeter wave radar data;

[0014] S1.2, using a target detection model to solve the original video frame, and using a sparse optical flow method to correct the result;

[0015] S1.3, synchronization of video data and millimeter wave radar data;

[0016] S1.4, saving and pushing the video data, millimeter wave radar data, and video target detection frame to a data bus;

[0017] S1.5, calibration processing based on target speed priority.

[0018] The vehicle detection method based on radar and vision data motion analysis provided by the application further preferably has the following method for realizing S1.2: t1 and I t2 , the center coordinates of the target detection frame are used as the corner points CornerSet, and the pixel offset δ u and δ v of each corner point in the two frames is calculated by using a sparse optical flow method;

[0019] δ u = OpticalFlowDiff u (I t2 ,I t1 , CornerSet)

[0020] δ v = OpticalFlowDiff v (I t2 ,I t1 , CornerSet)

[0021] The target speed priority-based calibration processing method described in S1.5 is as follows: screening targets with a longitudinal distance less than a certain threshold, a lateral speed less than a certain threshold, and a longitudinal speed greater than a certain threshold, and specially marking these targets as priority targets for corresponding point selection.

[0022] The further preferred technical solution of the vehicle detection method based on radar and vision data motion analysis is that the lane hotspot map is generated based on multiple frames of millimeter wave radar data in S2, including the following steps:

[0023] S2.1, constructing a frequency graph based on multiple frames of millimeter wave radar data;

[0024] S2.2, performing a specific transformation on the frequency graph based on the proportion of the maximum value of the frequency graph, associating a color graph with the calculation result, and generating a lane hotspot map;

[0025] S2.3, outlining a virtual lane based on the generated hotspot map.

[0026] The further preferred technical solution of the vehicle detection method based on radar and vision data motion analysis is that the method of S2.1 is as follows: a spatial frequency graph I of multiple frames of millimeter wave radar observations is constructed through linear transformation L freq ; specifically, the lateral coordinate of the millimeter wave radar data is transformed into the value range of (0, 640), and the longitudinal coordinate is transformed into the value range of (0, 480);

[0027] I freq(x,y) =L·Radar (x,y)

[0028] The method of S2.2 is as follows: if the proportion of the frequency maximum value is greater than a certain proportion δ p , then directly perform frequency normalization processing, if the proportion of the frequency maximum value is less than a certain proportion δ p , then perform nonlinear smoothing processing to obtain the result I' freq , and through color graph association, obtain the target hotspot map I map ;

[0029]

[0030]

[0031] I map =ColorMap(I' freq )。

[0032] The further preferred technical solution of the vehicle detection method based on the radar and video data motion analysis is that the radar and video fusion strategy based on the target motion state analysis in the S3 step comprises the following steps:

[0033] S3.1 virtual lane state calculation;

[0034] S3.2 millimeter wave radar tracking target motion state analysis;

[0035] S3.3 using the corresponding fusion strategy for the tracked target in different motion states;

[0036] S3.4 in the case of dense vehicles, performing start-stop processing on the millimeter wave radar data;

[0037] S3.5 in the case of low speed and static, using the video target detection frame to supplement the points;

[0038] The further preferred technical solution of the vehicle detection method based on the radar and video data motion analysis is that the method for realizing the S3.1 step is: according to the speed, acceleration, number of unmatched frames and distance from the stop line of the first radar target in the lane, the lane set LaneSet is divided into the red light lane set LaneSet red and the green light lane set LaneSet green ; the green light lane does not retain the long-time unmatched target, and the red light lane performs vehicle target supplement and queue length maintenance;

[0039] LaneSet={Lane red ,Lane green}

[0040] The method for realizing the S3.2 step is: using the Hungarian matching method to associate the millimeter wave radar set and the video observation set, combining the state of the lane where the millimeter wave target is located, the speed of the millimeter wave radar target, the position of the millimeter wave radar and other information, when the lane where the target is located is the red light lane, the distance from the previous stopped vehicle is less than a certain threshold, and the millimeter wave radar observation cannot be matched, but the video observation can be matched, it is determined that the vehicle is stopped; thus, the millimeter wave radar target set ObjSet is divided into the moving target set Obj move and the static target set Obj static ;

[0041] ObjSet={Obj move ,Obj static}

[0042] The method for realizing the S3.3 step is: for the static target Obj redStatic in the red light lane, the moving target ObjredMove , stationary target Obj in green light lane greenStatic , moving target Obj in green light lane greenMove , respectively, different fusion processing strategies φ redStatic , φ redMove , φ greenStatic , φ greenMove , maintain the queue length in the red light lane, while reducing the existence of stagnant targets after the lane changes from red to green;

[0043] The method for realizing the step S3.4 is: when the lane state is switched from red light to green light, if the target is not matched with the radar observation and the video observation, the Y direction is moved forward / backward according to the speed of the front vehicle in the lane, until the distance with the front vehicle is less than a certain threshold.

[0044] The vehicle detection method based on radar and video data motion analysis provided by the application further preferably has the following technical scheme: the processing method of the stationary target in the red light lane is that the stationary target in the red light lane is marked as Obj redStatic , the processing logic is φ redStatic , the tracking target is mapped to the image, the Hungarian matching is used to associate the video target detection data, if the tracking target is not matched with the video detection frame within a certain time, the target is removed from the tracker, and the processing result is marked as Obj redStaticRes .

[0045] The processing method of the moving target in the red light lane is: the moving target in the red light lane is marked as Obj redMove , the processing logic is φ redMove , if the tracking target is not matched with the video detection frame within a certain time, the target is removed from the tracker, and the processing result is marked as Obj redMoveRes .

[0046] The processing method of the stationary target in the green light lane is: the stationary target in the green light lane is marked as Obj greenStatic , the processing logic is φ greenStatic , if the target is not matched with the millimeter wave radar observation for a long time, the target is removed from the tracker, and the processing result is marked as Obj greenStaticRes .

[0047] The processing method of the moving target in the green light lane is: the moving target in the green light lane is marked as Obj greenMove , the processing logic is φ greenMove , if the target is not matched with the radar observation, the video observation associated with the target is used to associate with the Kalman filter prediction value, and the processing result is marked as Obj greenMoveRes .

[0048] The application synchronizes radar and video data mainly based on millimeter wave radar data, and proposes a radar and video data calibration method based on target speed priority; the application analyzes the motion state of the millimeter wave radar target, different motion states adopt different radar and video fusion strategies, and different processing methods are used for vehicle dense scenes and severe weather scenes; the application performs statistics and nonlinear transformation on multiple frames of radar observation frequency number graphs, and uses ColorMap for visualization to generate lane hotspot graphs to assist engineers; traffic statistics calculation and transmission, the application combines virtual lanes and virtual coils to count the fusion targets, and uses the transmission mode of TCP to send traffic flow, queue length, parking frequency, coil existence and other statistics to the signal machine; real-time video stream pushing, the application can push rtsp, rtmp, flv and other video streams to users through the construction of nginx streaming media server.

[0049] Compared with the prior art, the vehicle detection method based on radar and video data motion analysis provided by the application at least has the following beneficial effects:

[0050] 1、The application uses a more reasonable processing method, which improves the accuracy and robustness of target detection through motion analysis and adaptive fusion of millimeter wave radar and video data, is suitable for low-power edge computing devices, and reduces device cost.

[0051] 2、The application balances traffic flow, queue length, parking frequency, coil existence and other statistics.

[0052] 3、The application introduces the running light state of each lane in the algorithm, analyzes the motion state of the millimeter wave radar observation data, and adopts corresponding processing strategies for different motion states, effectively solving the problems of false detection of vehicle targets in green light lanes and missing detection of vehicle targets in red light lanes.

[0053] 4、Compared with the prior art, the application can better provide accurate and diverse traffic data for signal control systems.

[0054] 5、The application combines the characteristics of video sensors and millimeter wave radar sensors, considers the real-time nature of video target detection, and adapts to low-power edge computing devices by mainly observing radar and supplementing video observation, and makes decisions based on motion state analysis. DETAILED DESCRIPTION

[0055] Figure 1 is a flowchart of the method of the application;

[0056] Figure 2 is a schematic diagram of an application scenario of the method of the application;

[0057] Figure 3A schematic diagram of an overall architecture of the method of the present application;

[0058] Figure 4 A schematic diagram of a collection and calibration technical process of the method of the present application;

[0059] Figure 5 A schematic diagram of a radar calibration operation of the method of the present application;

[0060] Figure 6 A schematic diagram of a virtual lane, stop line and coil configuration technical process of the method of the present application;

[0061] Figure 7 A schematic diagram of a virtual lane, stop line and coil configuration operation of the method of the present application;

[0062] Figure 8 A schematic diagram of a radar and vision fusion technical process of the method of the present application;

[0063] Figure 9 A schematic diagram of a radar and vision fusion effect of the method of the present application;

[0064] Figure 10 A schematic diagram of a traffic statistical quantity calculation technical process of the method of the present application;

[0065] Figure 11 A schematic diagram of a traffic statistical quantity related display of the method of the present application;

[0066] Figure 12 A schematic diagram of a real-time video stream pushing technical process of the method of the present application;

[0067] Figure 13 A schematic diagram of a real-time video stream pushing effect of the method of the present application. DETAILED DESCRIPTION

[0068] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application, and after reading the present application, various equivalent modifications of the present application made by those skilled in the art fall within the scope defined by the claims of the present application.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein is only for the purpose of describing specific embodiments and is not intended to limit the present application.

[0070] REFERENCE Figure 1As shown in the flowchart of the vehicle detection method based on the motion analysis of the radar and video data provided by the embodiment of the present disclosure, it can be seen that the method mainly includes the following processes: video and millimeter wave radar data acquisition and calibration; virtual lane, stop line and coil configuration based on the spatial frequency of millimeter wave radar observation; video target detection result and millimeter wave radar data motion analysis and fusion processing; traffic statistics calculation and transmission; real-time video stream pushing based on nginx streaming server.

[0071] The application scenario diagram of the embodiment is referred to Figure 2 .

[0072] Referring to Figure 3 As shown in the architecture diagram of the vehicle detection method based on the motion analysis of the radar and video data provided by the embodiment of the present disclosure, a total of five modules are included in the above-mentioned processes, a data bus combined with mqtt and shared memory is established, and each module publishes and subscribes data through the bus.

[0073] (1) The video and millimeter wave radar data acquisition and calibration, which pre-synchronizes the millimeter wave radar data and video data according to the data timestamp, uses a radar video calibration method based on target speed priority, and specially marks the radar observation that meets certain conditions in terms of transverse speed and longitudinal speed, so that the engineering personnel can preferentially select points containing these special marks to find the corresponding points of the radar and video coordinate system.

[0074] (2) The virtual lane, stop line and coil configuration based on the spatial frequency of millimeter wave radar observation, which constructs a frequency graph of multiple frames of millimeter wave radar observation values in space, counts the maximum value proportion in the frequency graph, respectively performs linear transformation and nonlinear transformation on the frequency graph, uses a color graph to visualize the transformation results, and respectively visualizes the inbound data and outbound data for the engineering personnel to configure traffic information such as virtual lanes, stop lines and coils.

[0075] (3) The video target detection result and millimeter wave radar data motion analysis and fusion processing, which mainly uses radar data, tracks the millimeter wave radar data based on Kalman filtering (KF) and Hungarian matching method (Hungarian Algrithom), introduces the running state of the lane, combines the lane state to analyze the motion of the tracked target, and fuses the tracked target with the video target detection result according to different states; for low-speed and stationary scenes, video data is used for compensation; for scenes with multiple target dense occlusions, a robust start-stop processing strategy is proposed.

[0076] (4) The calculation and transmission of the traffic statistics, based on the configured virtual lane, stop line, virtual coil and other traffic configuration information, the target is counted, and the traffic flow, queue length, parking frequency, coil existence and other statistics are sent to the signal machine using the transmission mode of TCP.

[0077] (5) Real-time video stream pushing based on nginx streaming media server, through building a lightweight streaming media server on the edge computing device, the edge computing device can push multiple video streams, including rtsp video stream, rtmp video stream and flv video stream.

[0078] The application uses 80GHz millimeter wave radar and 2 million pixel monocular camera as sensors, can simultaneously obtain the relative ground road target position coordinate data, target speed data, target and scene pixel information, the edge computing processor is selected as 4-core ARM Cortex-A7, and the selected GPU computing power is 0.5TFLOPS.

[0079] Step 1: video and millimeter wave radar data acquisition and calibration

[0080] The application uses a 2 million pixel monocular camera to collect video data, the frame rate is 30fps, and the resolution of the collected image is 640x480; 80GHz millimeter wave radar is used, and the radar data analysis frequency is 10fps. The technical process of step 1 is as shown in Figure 4 The processing of sensor data is mainly divided into a data pushing branch and a radar and video calibration branch, the data pushing branch pre-processes the collected video data, infers the target detection model, compensates the inference delay by the sparse optical flow method, and then publishes the video target detection result, the real-time collected video frame and the millimeter wave radar data to the data bus; the radar and video calibration branch synchronizes, saves and calibrates the real-time collected video data and millimeter wave radar data.

[0081] In the data pushing branch, the resolution of the original video frame is scaled to 416x416, the mean and standard deviation of the ImageNet data set are used to standardize the image data, yolov3 trained by traffic scene data is used as the target detection model, the inference BatchSize of which is configured to be 1, and after post-processing such as non-maximum suppression, the target detection result of the video frame is obtained; as shown in the following formula, the real-time video frames I t1 and I t2 The center coordinates of the target detection frame are used as the corner point CornerSet, and the pixel offset of each corner point in the two frames is calculated by the sparse optical flow method u and δ v, according to the u, v direction offset of the center point, the video frame target detection result is corrected; finally, the video frame target detection result, the data collected by the millimeter wave radar in real time, and the data collected by the video in real time are pushed to the mqtt data bus.

[0082] delta u = OpticalFlowDiff u (I t2 , I t1 , CornerSet)

[0083] delta v = OpticalFlowDiff v (I t2 , I t1 , CornerSet)

[0084] In the radar-visual calibration branch, first, the video frame data and the millimeter wave radar data are pre-synchronized, and the millimeter wave radar data is used as the main data. According to the time stamps of the two, comparison is carried out through threshold setting, so as to ensure that one frame of radar corresponds to one frame of image, and the corresponding mode is used for saving. Then, the multi-frame offline data is labeled in a target speed priority based mode. Specifically, targets with longitudinal distance less than a certain threshold, transverse speed less than a certain threshold, and longitudinal speed greater than a certain threshold are selected and specially marked for engineers to select. Figure 5 As shown in the figure, the engineers select the target corresponding points in the video interface and the radar interface. In order to prevent the calibration error caused by the mouse click in the radar interface, step 1 searches for the nearest radar target point in the range of the mouse click in the radar coordinate system. According to the artificial selection of the radar-visual corresponding point set CorresPointSet, the perspective transformation matrix HomoMatrix representing the mapping transformation relationship from the radar coordinate system to the pixel coordinate can be solved.

[0085] HomoMatrix = findHomography(CorresPointSet)

[0086] I (x,y) = HomoMatrix * Radar (x,y)

[0087] Step 2: Configuration of virtual lane, stop line, virtual coil and other traffic information

[0088] Existing methods for configuring virtual lanes based on video sensors mostly involve directly selecting lane areas from the video feed. However, real-world applications present a variety of intersections, often encountering issues such as blurred lane lines, road glare, and uncertain lane boundaries. Therefore, this invention primarily utilizes millimeter-wave radar signals for traffic information configuration. Figure 6 As shown, the configuration of traffic information such as virtual lanes, stop lines, and virtual loops includes: generating a heat map based on multi-frame millimeter-wave radar observations, manually drawing virtual lanes based on target heat maps, manually determining the stop lines for each lane, and manually configuring virtual loops in the video footage.

[0089] The traffic information configuration in step 2 depends on the perspective transformation matrix solved in step 1, so that the configuration information in the two coordinate systems can be mapped to each other.

[0090] In step 2, a spatial frequency map I of multi-frame millimeter-wave radar observations was constructed through linear transformation L. freq The frequency plot size is set to 640x480, and the maximum value (Max) in the frequency plot is calculated. freq Proportion P freq As shown in the following formula, if the proportion of frequency maxima is greater than a certain proportion δ p If the proportion of frequency maxima is less than a certain proportion δ, then frequency normalization is performed directly. p Then, nonlinear smoothing is performed to obtain the result I'. freq By associating color maps, the target heat map I is obtained. map .

[0091] I freq(x,y) =L·Radar (x,y)

[0092]

[0093]

[0094] I map =ColorMap(I' freq )

[0095] In step 2, based on the target heat map I map Virtual lanes are outlined manually on the heat map, such as... Figure 7 As shown, engineers only need to draw the lane boundary lines to automatically generate virtual lanes in both the radar and video coordinate systems. In step 2, the stop lines and virtual coils for each lane are manually configured; engineers only need to outline the stop lines and virtual coils for each lane in the video frame.

[0096] Step 3: Motion analysis and fusion processing of video target detection results and millimeter wave radar data

[0097] Step 3 realizes a decision layer radar and vision data adaptive fusion method based on target motion analysis. The overall technical process is as shown in Figure 8 The millimeter wave radar data processing is the main line, Kalman filtering is used to predict the current tracking target, and Hungarian matching based on Mahalanobis distance is used to associate millimeter wave radar observations and predicted values. Motion state analysis is performed on each tracking target, and lane green / red light state is introduced. Different fusion processing strategies are used for stationary targets in red light lanes, moving targets in red light lanes, stationary targets in green light lanes, and moving targets in green light lanes. For the case that millimeter wave radar has poor detection effect on low-speed targets / stationary targets, a strategy of using video to supplement points is proposed and implemented. For the case of dense occlusion of multiple targets during peak hours, a robust start-stop processing strategy is proposed and implemented.

[0098] In step 3, Kalman filtering is used to predict millimeter wave radar targets. The tracking target at time t-1 is a 6-dimensional vector [X t-1 ,Y t-1 ,Vx t-1 ,Vy t-1 ,ax t-1 ,ay t-1 ], where X t-1 represents the horizontal coordinate at time t-1, Y t-1 represents the vertical coordinate at time t-1, Vx t-1 represents the horizontal velocity at time t-1, Vy t-1 represents the vertical velocity at time t-1, ax t-1 represents the horizontal acceleration at time t-1, and ay t-1 represents the vertical acceleration at time t-1. The prediction is as shown below, which predicts the target state at time t based on the target information at time t-1.

[0099]

[0100] In step 3, Hungarian matching method is used to associate the prediction results and millimeter wave radar observations, as shown below. The covariance of the tracking target historical data is calculated, and the Mahalanobis distance between the prediction and all observations is constructed. The maximum matching between the prediction and the observation is found by the Hungarian matching method.

[0101]

[0102]

[0103] Step 3 introduces the green light state / red light state of the lane, each lane is in the green light state by default, and the data of the first vehicle in the lane is judged to determine whether the lane is switched to the red light state; when the following conditions are met, the lane is switched to the red light lane; if the lane is a coming lane, the first vehicle in the lane is within 30 meters from the stop line, the speed of the first vehicle is less than 5 m / s, the first vehicle in the lane is not matched to the millimeter wave radar observation value for 5 consecutive frames, the first vehicle in the lane has a deceleration trend within 2 seconds, and the lane has been in the red light state for more than 10 seconds since the last time.

[0104] LaneSet = {Lane red , Lane green}

[0105] Step 3 analyzes the motion state of each tracking target. There is a certain error in the processing of the millimeter wave radar on the target parking speed, and there is a problem of excessive longitudinal speed measurement when the vehicle is parked. Combined with the configured traffic information, the motion and static state of the tracking target is determined. The state of the tracking target is initially in motion, and when the following conditions are met, it is determined to be in a static state: calculate the longitudinal order of the tracking target in the lane, if the target is the first vehicle in the lane, determine whether the lane is a coming lane, the target distance from the stop line is less than a certain threshold, the speed of the target is less than 5 m / s, there is a deceleration trend within 2 seconds, the millimeter wave radar observation is continuously mismatched, and the target can be matched to the video observation. If all the above conditions are met, it is determined to be in a static state. If the tracking target is not the first vehicle in the lane, determine whether the lane is a red light lane, the distance to the nearest stopped target in front is less than 50 meters, the speed of the target is less than 5 m / s, there is a deceleration trend within 2 seconds, the millimeter wave radar observation is continuously mismatched, and the target can be matched to the video observation. If all the above conditions are met, it is determined to be in a static state. The state of the static target determined is not allowed to modify the motion state within a certain period.

[0106] ObjSet = {Obj move , Obj static}

[0107] Step 3 respectively for the static target in the red light lane, the moving target in the red light lane, the static target in the green light lane, and the moving target in the green light lane, each adopts a different fusion processing strategy; the static target in the red light lane is denoted as Obj redStatic , its processing logic is denoted as φ redStatic , specifically, the tracking target is mapped to the image, and the Hungarian matching is used to associate the video target detection data. If the tracking target is not matched to the video detection frame within 3 seconds, the target is removed from the tracker. The processing result is denoted as Obj redStaticResAt the same time, the distance of the target before and after the target in the lane is calculated, if there is a stationary target in front, it is determined whether there is a video detection frame between the targets, and the video detection frame is used to supplement the points between the two; the moving target in the red light lane is marked as Obj redMove , and the processing logic is φ redMove . Specifically, similar to the first case, if the tracking target is not matched to the video detection frame within 3s, the target is removed from the tracker, and the processing result is marked as Obj redMoveRes ; the stationary target in the green light lane is marked as Obj greenStatic , and the processing logic is φ greenStatic . Specifically, if the target has not been matched to the millimeter wave radar observation for a long time, the target is removed from the tracker, and the processing result is marked as Obj greenStaticRes ; the moving target in the green light lane is marked as Obj greenMove , and the processing logic is φ greenMove . Specifically, if the target is not matched to the radar observation, the video observation associated with the target is associated with the Kalman filter prediction value, and the processing result is marked as Obj greenMoveRes .

[0108] ObjSet = {Obj redStatic , Obj redMove , Obj greenStatic , Obj greenMove}

[0109] Obj redStaticRes = φ redStatic (Obj redStatic )

[0110] Obj redMoveRes = φ redMove (Obj redMove )

[0111] Obj greenStaticRes = φ greenStatic (Obj greenStatic )

[0112] Obj greenMoveRes = φ greenMove (Obj greenMove )

[0113] ObjSet Res = {Obj redStaticRes , Obj redMoveRes , Obj greenStaticRes , Obj greenMoveRes}

[0114] In step 3, a strategy of supplementing points by using video target detection boxes is proposed and used for the case of poor detection effect of low-speed targets / stationary targets by millimeter wave radar; in step 3, by matching the video detection boxes and millimeter wave radar observations, a set of mismatched video target detection boxes is counted, and new tracking targets are created according to these target detection boxes.

[0115] In step 3, a robust start-stop processing strategy is proposed and implemented for the case of dense occlusion of multiple targets during peak hours; according to the red light state and green light state of the lane, the time period when the red light state switches to the green light state is found, and the targets still in the stationary state are pushed forward until the distance from the front vehicle is less than a certain threshold.

[0116] Step 4: Calculation and transmission of traffic statistics

[0117] Step 4: Based on virtual lanes and virtual coils, relevant statistics are performed, and data update statistics are performed after each target fusion update; the statistical effect is as shown in Figure 11 The calculated traffic statistics include pulse data, traditional data, etc.

[0118] Pulse data refers to the state information of the target entering and leaving the coil, and the calculation process is as shown in Figure 10 For each coil, a vector for storing pulse data is established, when the target enters the coil, the state is stored as 1, and when the target leaves the coil, the state is stored as 0.

[0119] The traditional data includes queue length, traffic flow (total traffic flow or traffic flow classified by type), time occupancy, average speed, headway, etc.; the queue length is divided into static queue length and dynamic queue length; the dynamic queue length is first counted at the target point in the lane, then sorted according to the position, and the distance between the first vehicle and the last vehicle is recorded as the queue length. When the first vehicle passes the lane line, the queue length is the distance between the lane line and the last vehicle; when the first vehicle does not pass the lane line, the queue length is the distance between the head of the first vehicle and the tail of the last vehicle; the calculation of the static queue length is first counted at the target point in the lane, then filtered by the speed threshold, and then sorted; when the adjacent targets after sorting are greater than a certain threshold, the previous target is recorded as a, and the distance between the first target and the target a is recorded as the static queue length; the calculation of the time occupancy is based on the statistical coil, and the parameter is periodically counted; from the beginning of the statistical period, the time when each point enters and leaves the coil is recorded, and the difference is accumulated to the coil occupancy time; at the end of the period, the ratio of the occupancy time in the period to the statistical period is calculated, that is, the time occupancy; the calculation of the average speed is based on the statistical coil, and the speed of the target passing through the coil in the period is recorded and accumulated; at the end of the period, the ratio of the target to the speed is calculated, and the corresponding transmission is performed; the calculation of the headway needs to record the time when the first and last vehicles enter the coil in the period, and the difference is calculated; the ratio of the number of vehicles passing through in the period is calculated, that is, the headway.

[0120] Step 4: transmitting traffic statistics to the signal machine based on TCP protocol, initializing the IP of the edge computing device as a TCP server and the signal machine as a TCP client; the edge computing device establishes communication connection with the signal machine by listening to the client connection event; the edge computing device and the signal machine customize corresponding communication protocol, mainly including traffic flow, pulse transmission, dynamic queue length, fault information, road condition information, parking times, delay time, queue overvalue alarm, passing state, interval dynamic equivalent vehicle number, interval average headway, real-time speed upload, lane data, etc.

[0121] Step 5: pushing real-time video stream

[0122] Step 5: based on ffmpeg library, using H264 format for encoding and decoding, pushing raw video frames / video frames with superimposed information to rtsp video stream; by building nginx streaming media server on the edge computing device, converting rtsp video stream to rtmp and flv, and can be pushed in the local area network; the overall framework is as follows Figure 12As shown, nginx provides rtmp service through rtmp module, ffmpeg pushes an rtmp stream to nginx, and then real-time rtmp video stream is watched by accessing nginx, and similarly, when HLS is configured, the client can access flv video stream through HTTP protocol; video stream pushing effect is as follows Figure 13 As shown, the video stream effect meets real-time performance.

[0123] The overall architecture of the vehicle detection method based on radar and video data motion analysis in this embodiment is referred to Figure 3 , the calibration operation of video and millimeter wave radar data is referred to Figure 5 , the configuration of virtual lane, stop line and coil is referred to Figure 7 , the fusion tracking effect of millimeter wave radar and video data is referred to Figure 9 , the related display of traffic statistics is referred to Figure 11 , the real-time video pushing stream is referred to Figure 13 .

[0124] To sum up, the vehicle detection method based on radar and video data motion analysis is proposed, the motion analysis and adaptive fusion of millimeter wave radar and video data are carried out, the accuracy and robustness of target detection are improved, the method is suitable for low-power edge computing device, and the device cost is reduced; at the same time, the traffic flow, queue length, parking times, coil existence and other statistics are balanced, the running light state of each lane is introduced in the algorithm, the motion state of millimeter wave radar observation data is analyzed, different motion states adopt corresponding processing strategies, and the problems of green light lane vehicle target false detection and red light lane vehicle target missing detection are effectively solved. Compared with the existing vehicle detection technology, the present application can better provide accurate and diverse traffic data for signal control system.

[0125] Each of the above embodiments in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. The above is only several embodiments of the present application, although the disclosed embodiments are as above, but the content is only for understanding the technical solutions of the present application, and is not used to define the present application. Any person skilled in the art can make any modification and change in the form and details of the embodiments without departing from the spirit and scope of the present application, but the patent protection scope of the present application shall be subject to the scope defined by the appended claims.

Claims

1. A vehicle detection method based on radar data motion analysis, characterized by, It comprises the following steps: S1, video and millimeter wave radar data acquisition and calibration; S2, generating lane hotspot map based on multiple frames of millimeter wave radar data; S3, radar and video fusion strategy based on target motion state analysis; S4, calculation and transmission of traffic statistics; S5, real-time video stream pushing; The lane hotspot map generated based on multiple frames of millimeter wave radar data in S2 step comprises the following steps: S2.1, constructing frequency graph based on multiple frames of millimeter wave radar data; S2.2, based on the maximum value proportion statistics of the frequency graph, performing specific transformation on the frequency graph, associating the calculation results with the color graph, and generating the lane hotspot map; S2.3, based on the generated hotspot map, outlining the virtual lane; The method for realizing the step S2.1 is: constructing the spatial frequency diagram I of the multi-frame millimeter wave radar observation through a linear transformation L freq ; specifically, the horizontal coordinate of the millimeter wave radar data is transformed into the value range of (0, 640), and the vertical coordinate is transformed into the value range of (0, 480). I freq(x,y) = L · Radar (x,y) The method for realizing the step S2.2 is as follows: if the proportion of the frequency maximum value is greater than a certain proportion δ p , then directly performing frequency normalization processing, if the proportion of the frequency maximum value is less than or equal to a certain proportion δ p , then performing nonlinear smoothing processing to obtain a result I' freq , and obtaining a target hotspot map I map through color chart association. I map = ColorMap(I freq ).

2. The vehicle detection method based on the motion analysis of the radar data according to claim 1, wherein, The video and millimeter wave radar data acquisition and calibration in S1 step comprises the following steps: S1.1, video data and millimeter wave radar data acquisition; S1.2, using a target detection model to solve the original video frame, and using a sparse optical flow method to correct the result; S1.3, video data and millimeter wave radar data synchronization; S1.4, video data, millimeter wave radar data, and video target detection frame, saving and pushing to the data bus; S1.5, target speed priority based calibration processing.

3. The vehicle detection method based on the motion analysis of the radar data according to claim 2, wherein, The method for realizing the step S1.2 is: capturing real-time video frames I before and after model reasoning t1 With the center coordinates of the target detection frame as the corner point CornerSet, the pixel offset δ of each corner point in the two frames before and after is calculated by the sparse optical flow method t2 With the center coordinates of the target detection frame as the corner point CornerSet, the pixel offset δ of each corner point in the two frames before and after is calculated by the sparse optical flow method u With the center coordinates of the target detection frame as the corner point CornerSet, the pixel offset δ of each corner point in the two frames before and after is calculated by the sparse optical flow method v According to the u and v direction offset of the center point, the target detection result of the video frame is corrected; delta u = OpticalFlowDiff u (I t2 ,I t1 ,CornerSet) delta v = OpticalFlowDiff v (I t2 , I t1 , CornerSet) The target speed priority based calibration processing method in S1.5 step is to select targets with longitudinal distance less than a certain threshold, transverse speed less than a certain threshold, and longitudinal speed greater than a certain threshold, and specially mark these targets as priority targets for corresponding point selection.

4. The vehicle detection method based on the motion analysis of the radar data according to claim 1, wherein, The radar and video fusion strategy based on target motion state analysis in S3 step comprises the following steps: S3.1 virtual lane state calculation; S3.2 millimeter wave radar tracking target motion state analysis; S3.3 using corresponding fusion strategy for tracking targets in different motion states; S3.4, in dense vehicle conditions, performing start-stop processing on millimeter wave radar data; S3.5, in low-speed and stationary conditions, using video target detection frame for point supplement.

5. The vehicle detection method based on the motion analysis of the radar data according to claim 4, wherein, The method for realizing the step S3.1 is: according to the speed, acceleration, unmatched frame number and distance from the stop line of the first radar target in the lane, the lane set LaneSet is divided into a red light lane set LaneSet red and a green light lane set LaneSet green ; The green light lane does not retain long-mismatched targets, and the red light lane maintains vehicle target supplement and queue length; LaneSet = { Lane red ,Lane green} The method for realizing the step S3.2 is: using the Hungarian matching method to associate the millimeter wave radar set with the video observation set, combining the state of the lane where the millimeter wave target is located, the speed of the millimeter wave radar target, and the position information of the millimeter wave radar, when the lane where the target is located is a red light lane, the distance to the previous stopped vehicle is less than a certain threshold, and no millimeter wave radar observation can be matched but video observation can be matched, it is determined that the vehicle is stopped; thus, the millimeter wave radar target set ObjSet is divided into a moving target set Obj move and a static target set Obj static ; ObjSet = { Obj move , Obj static} The method for realizing the step S3.3 is: for the static target Obj redStatic , the moving target Obj redMove , the static target Obj greenStatic , the moving target Obj greenMove in the red light lane, different fusion processing strategies φ redStatic , φ redMove , φ greenStatic , φ greenMove are respectively adopted, the queuing length in the red light lane is maintained, and the existence of the stranded target after the lane changes from red to green is reduced. The method for implementing S3.4 step is: when the lane state is switched from red light to green light, if the target is not matched with radar observation and video observation, then move forward / backward in Y direction according to the speed of the preceding vehicle in the lane until the distance to the preceding vehicle is less than a certain threshold.

6. The vehicle detection method based on the motion analysis of the radar data according to claim 5, wherein, The processing method of the static target in the red light lane is: the static target in the red light lane is recorded as Obj redStatic , the processing logic of which is φ redStatic , the tracking target is mapped to the image, the Hungarian matching is used to associate the video target detection data, if the tracking target is not matched to the video detection frame within a certain time, the target is removed from the tracker, and the processing result is recorded as Obj redStaticRes ; The processing method of the moving target in the red light lane is: the moving target in the red light lane is recorded as Obj redMove , the processing logic is recorded as φ redMove , if the tracking target is not matched to the video detection frame in a continuous period of time, the target is removed from the tracker, and the processing result is recorded as Obj redMoveRes ; The processing method of the static target in the green light lane is: the static target in the green light lane is marked as Obj greenStatic , and the processing logic is marked as φ greenStatic . If the target is not matched to the millimeter wave radar observation for a long time, the target is removed from the tracker, and the processing result is marked as Obj greenStaticRes . The processing method of the moving target in the green light lane is: the moving target in the green light lane is recorded as Obj greenMove , and the processing logic is recorded as φ greenMove . If the target is not matched with the radar observation, the video observation associated with the target is used to associate with the Kalman filter prediction value, and the processing result is recorded as Obj greenMoveRes .

Images