Early warning method based on fusion of video target and radar target

Abstract

The present invention relates to the field of transportation, and specifically to a video target and radar target fusion warning method. The purpose is to overcome the deficiencies in the above-mentioned background technology and provide a video target and radar target fusion warning method, which should have the characteristics of detection accuracy and reliability. The technical solution of the present invention is: a video target and radar target fusion warning method, comprising the following steps: S1: data acquisition and preprocessing; S2: YOLOv5 model processing video data; S3: U-Net model processing radar data; S4: feature fusion module fusing video data and radar data; S5: post-processing module processing fused data; S6: vehicle warning.

Description

Technical Field

[0001] The invention relates to the field of traffic, and in particular to a video target and radar target fusion early warning method. Background Art

[0002] In tunnel traffic systems, the identification and warning of traffic accidents and the identification of target trajectories have always been a challenging problem. The existing technologies are mainly divided into two categories: macro data analysis and individual vehicle behavior analysis.

[0003] Macro data analysis methods (a highway tunnel driving safety risk identification and prevention system CN217008371U and a traffic accident risk assessment method and system CN106897826A) predict risks by analyzing macro data such as traffic flow parameters and risk factors. Although these methods can assess the overall situation, it is difficult to provide accurate warnings for individual vehicles.

[0004] The individual vehicle behavior analysis method (shown in a tunnel driving traffic accident risk intelligent early warning method CN116740986A) predicts the accident risk by analyzing the driving status of the vehicle in the monitoring video. This method has a short response time and is difficult to respond to emergencies in a timely manner.

[0005] In order to solve the above problems, the industry began to explore the technical route of video and radar fusion, such as: a tunnel safety warning system based on directional millimeter-wave radar technology CN115472017A, an event detection method, device and system based on tunnel radar data fusion CN113850995A, a traffic event monitoring equipment and method based on video and radar CN108648456A, and a traffic event monitoring equipment based on video and radar CN208335464U.

[0006] The core idea of ​​these solutions is to use the data fusion of video and millimeter-wave radar to complete event monitoring. Specifically: 1. The video system is responsible for target detection and tracking, and can identify traffic participants such as vehicles and pedestrians; 2. Millimeter-wave radar provides accurate distance and speed information, especially in tunnel environments with low visibility; 3. Through the data fusion algorithm, the information of video and radar is combined to achieve more accurate target positioning and trajectory prediction; 4. Based on the fused data, a risk assessment model is established to provide real-time risk warnings for individual vehicles.

[0007] However, the existing video and radar target fusion technology still has the following shortcomings: 1. High equipment cost: It is necessary to deploy high-quality video surveillance systems and millimeter-wave radar systems at the same time, which increases the cost of the overall solution; 2. Complex fusion algorithm: The fusion process of video and radar data has high computational complexity, which may lead to a decrease in real-time performance; 3. Limited environmental adaptability: Under extreme weather conditions (such as heavy fog, heavy rain, etc.), the performance of both video and radar may be affected, thereby affecting the fusion effect; 4. Calibration difficulties: The video system and radar system require precise spatial and temporal synchronization, and the calibration process is complex and prone to errors. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and to provide a video target and radar target fusion early warning method, which should have the characteristics of detection accuracy and reliability.

[0009] The technical solution of the present invention is:

[0010] The video target and radar target fusion early warning method includes the following steps:

[0011] S1: Data collection and preprocessing;

[0012] S2: YOLOv5 model processes video data;

[0013] S3: U-Net model processes radar data;

[0014] S4: Feature fusion module fuses video data with radar data;

[0015] S5: post-processing module processes the fused data;

[0016] S6: Vehicle warning.

[0017] The step S1 comprises:

[0018] S1.1, the surveillance camera collects the video data of the vehicle, and then performs denoising, contrast enhancement and geometric correction on the video data;

[0019] S1.2. The millimeter-wave radar collects the radar data of the vehicle, and then performs clutter elimination, Doppler processing, and angle estimation on the radar data.

[0020] In step S2, the YOLOv5 model includes an input layer, a feature extraction layer, a feature pyramid network, and a detection head connected in sequence.

[0021] In step S3, the U-Net model includes an input layer, an encoder, a decoder, and an output layer connected in sequence.

[0022] In step S4, the feature fusion module includes feature alignment, attention mechanism, and multi-scale fusion which are connected in sequence.

[0023] In step S5, the post-processing module includes non-maximum suppression, threshold filtering, and trajectory smoothing connected in sequence.

[0024] In step S6, the vehicle warning includes: risk assessment of abnormal events, calculation of subsequent vehicle arrival time, warning determination, and warning issuance.

[0025] The beneficial effects of the present invention are:

[0026] The present invention makes full use of the advantages of visual detection and radar detection. By combining the visual detection capabilities of YOLOv5 and the advantages of U-Net in processing radar data, it can achieve accurate tracking and early warning of individual vehicles, detect potential risks earlier, improve the accuracy and reliability of vehicle detection in complex environments such as tunnels, and achieve a good balance between cost and performance. It is more practical than high-end equipment such as lidar. The present invention represents a major advancement in traffic safety monitoring technology, providing an efficient, accurate and reliable target detection and accident warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the present invention.

[0028] Figure 2 It is a schematic diagram of the YOLOv5 model of the present invention.

[0029] Figure 3 Schematic diagram of the U-Net model of the present invention.

[0030] Figure 4 It is a schematic diagram of the feature fusion module of the present invention.

[0031] Figure 5 Schematic diagram of the post-processing module of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] like Figure 1 As shown, the video target and radar target fusion early warning method includes the following steps:

[0034] S1 Data Collection and Preprocessing

[0035] 1.1) Video data acquisition and preprocessing

[0036] 1.1.1) Collect video data

[0037] The surveillance camera collects video data of the vehicle;

[0038] Video data has the following characteristics:

[0039] ① Resolution: at least 1080p (1920x1080);

[0040] ②Frame rate: no less than 25fps;

[0041] ③The field of view can cover the entire lane and part of the shoulder;

[0042] ④ Light adaptation: It has a wide dynamic range (WDR) to adapt to changes in light in the tunnel.

[0043] 1.1.2) Preprocessing video data

[0044] Denoise, contrast enhancement, and geometric correction of video data:

[0045] ① Denoising: Use Gaussian filtering or median filtering to reduce image noise;

[0046] ②Contrast enhancement: Use adaptive histogram equalization (CLAHE) to improve image contrast;

[0047] ③ Geometric correction: Perform perspective transformation through camera calibration parameters to eliminate perspective distortion;

[0048] 1.2) Collect radar data

[0049] Millimeter wave radar collects radar data of the vehicle; radar data includes the distance, speed and angle of the collected target.

[0050] 1.2.1 Perform clutter removal, Doppler processing, and angle estimation on radar data:

[0051] ①Clutter elimination: Use constant false alarm rate (CFAR) technology to eliminate environmental clutter;

[0052] ② Doppler processing: extract target velocity information through fast Fourier transform (FFT);

[0053] ③Angle estimation: Use beamforming technology to improve angular resolution.

[0054] Millimeter-wave radars and surveillance cameras are arranged at the tunnel entrance; millimeter-wave radars and surveillance cameras are arranged inside the tunnel; millimeter-wave radars and surveillance cameras are arranged at the tunnel exit.

[0055] S2. YOLOv5 model processes video data

[0056] The video data is input into the YOLOv5 model, and the YOLOv5 model outputs the target detection results (including bounding box, category, and confidence).

[0057] The YOLOv5 model is used to detect the location of the suspected accident vehicle. YOLOv5 includes an input layer, a feature extraction layer, a feature pyramid network, and a detection head connected in sequence.

[0058] When training the YOLOv5 model, the video data needs to be labeled. The labeling process is as follows:

[0059] ① Use labeling tools (such as LabelImg) to label the preprocessed video data frames; the labeling categories include: small cars, medium-sized cars, large cars, pedestrians, obstacles; the number of labeled pictures is greater than 10,000

[0060] ② Draw a bounding box for each target and ensure the consistency and accuracy of the annotation;

[0061] ③ Use data enhancement techniques (such as rotation, scaling, color transformation, etc.) to expand the training data set.

[0062] Detection accuracy of the YOLOv5 model: mAP ≥ 0.85.

[0063] Detection speed of YOLOv5 model: ≥25fps.

[0064] S3 and U-Net models process radar data

[0065] The radar data is input into the U-Net model, and the U-Net model outputs the processed radar data.

[0066] The U-Net model is used to improve the accuracy and robustness of target detection. The U-Net model consists of an input layer (256×256×1), an encoder (4 layers of convolutional pooling), a decoder (4 layers of upsampling convolution), and an output layer (256×256×1) connected in sequence.

[0067] S4, feature fusion module fuses video data and radar data

[0068] The video data and radar data are input into the feature fusion module, and the feature fusion module outputs the fused features.

[0069] The feature fusion module uses a fusion algorithm to combine the advantages of two data sources to improve the accuracy of target detection and tracking and achieve more accurate target positioning and trajectory prediction. The fusion algorithm adopts a multi-sensor data fusion method based on Kalman filtering.

[0070] The principle of the fusion algorithm is as follows:

[0071] P融合 =w 视频 ×P 视频 +w 雷达 ×P 雷达

[0072] P 融合 is the target position after fusion, P 视频 is the target position detected in the video, P 雷达 is the target position detected by the radar, w 视频 and w 雷达 is the corresponding weight coefficient, w 视频 +w 雷达 =1.

[0073] The feature fusion module includes feature alignment, attention mechanism, and multi-scale fusion.

[0074] Multi-scale fusion includes the following steps:

[0075] ① State prediction: Use the state estimate of the previous moment to predict the state of the current moment;

[0076] ② Measurement update: Combine video and radar measurement data to update state estimates;

[0077] ③ Covariance update: Dynamically adjust the uncertainty of state estimation based on measurement noise and process noise.

[0078] S5, post-processing module processes fusion data

[0079] The fused features are input into the post-processing module, which outputs the vehicle detection results (position, speed, trajectory, license plate, distance, camera pile number, event behavior)

[0080] The post-processing module includes non-maximum suppression, threshold filtering, and trajectory smoothing (all existing technologies) connected in sequence.

[0081] Event behaviors include:

[0082] ①Normal events;

[0083] ② Abnormal events: abnormal parking, abnormal deceleration; abnormal trajectory (sharp turn, sudden lane change); abnormal objects on the road; abnormal reduction in the distance between vehicles.

[0084] S5. Vehicle warning

[0085] 5.1) Conduct risk assessment on abnormal events

[0086] Risk level = α × abnormality level + β × impact range + γ × traffic flow

[0087] α, β, and γ are empirical values.

[0088] Abnormality degree: The degree to which the detected abnormal event deviates from the normal state is scored, ranging from 0 to 1, depending on the actual situation.

[0089] Impact range: Quantified by the number and length of affected lanes, converted into values ​​within a certain range, and determined based on actual conditions.

[0090] Traffic flow: vehicle density in the current area; classified according to the current vehicle density and normalized according to the density.

[0091] The risk level ranges from 0 to 1, and the larger the value, the higher the risk level.

[0092] 5.2) Calculate the arrival time of subsequent vehicles

[0093] For the following vehicles, calculate the estimated time t of their arrival at the abnormal event point;

[0094] t = d / v, d is the distance between the vehicle and the abnormal event point, v is the vehicle speed (obtained from the radar);

[0095] 5.3) Early warning determination

[0096] Emergency area (directly affected area, 0-100m from the accident point): T 阈值 = 10 seconds,

[0097] Warning area (potential impact area, 100-300m away from the accident point): T 阈值 =20 seconds

[0098] Warning area (preventive warning area, 300-500m away from the accident point): T 阈值 =30 seconds

[0099] The warning judgment is directly related to the risk level. The calculation result of the risk level can be used to dynamically adjust the warning threshold T 阈值 For high risk level: If the risk level is high, that is, the abnormality is serious, the impact range is wide, and the traffic volume is large, the warning needs to be triggered in advance. For low risk level: If the risk level is low, the conditions for triggering the warning will be relaxed.

[0100] This connection makes the early warning system more flexible and can be adjusted dynamically according to the actual risk situation, thereby improving the accuracy and timeliness of early warnings and avoiding too many or too few warnings.

[0101] When the risk level is greater than or equal to 0.7, if t<T 阈值 ×(1-risk level), triggering an early warning;

[0102] When the risk level is greater than or equal to 0.3 and less than 0.7, if t<T 阈值, triggering an early warning;

[0103] When the risk level is greater than 0.3, if t<T 阈值 (1+risk level), triggering an early warning.

[0104] 5.4) Warning Release

[0105] The early warning information is sent to the manager, who then issues control measures.

[0106] The early warning information includes vehicle detection results, vehicle information in the emergency area, vehicle information in the warning area, and vehicle information in the reminder area.

[0107] Control measures include:

[0108] ① Emergency area: Send emergency braking warnings, suggest immediate deceleration, and provide recommended avoidance routes;

[0109] ② Warning area: send deceleration warnings, provide information on road conditions ahead, and suggest lane changes in advance;

[0110] ③ Reminder area: Send reminder messages, advise to stay vigilant, and provide alternative routes.

[0111] The release methods of control measures include at least one of information boards, directional loudspeakers, flashing lights, vehicle-mounted terminals, and mobile phone APPs.

[0112] The performance of the present invention is as follows:

[0113] Detection performance: target detection accuracy is greater than or equal to 95%, false alarm rate is less than or equal to 1%, and missed alarm rate is less than or equal to 0.5%.

[0114] Early warning performance: Early warning lead time is greater than or equal to 10 seconds, early warning accuracy is greater than or equal to 90%, and system response time is less than or equal to 100ms.

[0115] The advantages of the present invention are:

[0116] 1. Compared with the traditional solution based on radar-visual fusion, the present invention does not rely on radar for position positioning and radar-visual fusion technology, which can reduce the equipment used and greatly reduce the cost;

[0117] 2. Compared with the traditional solution based on traffic parameters, the present invention can perform individual calculations and early warnings for each vehicle from macro to micro perspectives;

[0118] 3. The present invention can provide early warning for potential traffic threats, greatly improving the driving safety of vehicles in tunnels;

[0119] 4. Strong environmental adaptability: By integrating video and radar data, the system can maintain high detection accuracy in all weather and lighting conditions;

[0120] 5. Excellent real-time performance: The lightweight YOLOv5 model and efficient data fusion algorithm are used to ensure that the system can process large amounts of data in real time;

[0121] 6. The surveillance camera is responsible for target detection and tracking, and can identify traffic participants such as vehicles and pedestrians. The millimeter-wave radar provides accurate distance and speed information, especially in tunnel environments with low visibility. Through the data fusion algorithm, the video and radar information are combined to achieve more accurate target positioning and trajectory prediction.

[0122] The preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

Claims

1. The video target and radar target fusion early warning method includes the following steps: S1: Data collection and preprocessing; S2: YOLOv5 model processes video data; S3: U-Net model processes radar data; S4: Feature fusion module fuses video data with radar data; S5: post-processing module processes the fused data; S6: Vehicle warning.

2. The video target and radar target fusion early warning method according to claim 1 is characterized in that: The step S1 comprises: S1.1, the surveillance camera collects the video data of the vehicle, and then performs denoising, contrast enhancement and geometric correction on the video data; S1.

2. The millimeter-wave radar collects the radar data of the vehicle, and then performs clutter elimination, Doppler processing, and angle estimation on the radar data.

3. The video target and radar target fusion early warning method according to claim 1 is characterized in that: In step S2, the YOLOv5 model includes an input layer, a feature extraction layer, a feature pyramid network, and a detection head connected in sequence.

4. The video target and radar target fusion early warning method according to claim 1 is characterized in that: In step S3, the U-Net model includes an input layer, an encoder, a decoder, and an output layer connected in sequence.

5. The video target and radar target fusion early warning method according to claim 1 is characterized in that: In step S4, the feature fusion module includes feature alignment, attention mechanism, and multi-scale fusion which are connected in sequence.

6. The video target and radar target fusion early warning method according to claim 1 is characterized in that: In step S5, the post-processing module includes non-maximum suppression, threshold filtering, and trajectory smoothing connected in sequence.

7. The video target and radar target fusion early warning method according to claim 1 is characterized in that: In step S6, the vehicle warning includes: risk assessment of abnormal events, calculation of subsequent vehicle arrival time, warning determination, and warning issuance.

Images