Rapid detection and early warning system for special vehicle

Abstract

The invention discloses a rapid detection and early warning system for a special vehicle, and particularly relates to the field of intelligent traffic supervision, which comprises a monitoring front end, a rear end server and an information center, wherein the monitoring front end comprises three cameras and one laser radar which are connected with the data processing unit of the rear-end server through the switch, and the data processing unit performs algorithm reasoning on road image information and 3D point cloud information collected by the front end to obtain vehicle target detection and behavior early warning information. With the gradual maturity of technologies such as computer vision, image processing, 3D target detection and the like, a mode of acquiring road vehicle information through a monitoring video and laser radar fused detection system is deeply studied. The two sensors are utilized to design the rapid detection and early warning system for the special transport vehicle for intelligent traffic to carry out real-time online supervision on a road target, so that the road safety can be effectively improved, and reliable data support is provided for global scheduling of the road.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic supervision, and more specifically, to a special vehicle rapid detection and early warning system. Background Art

[0002] In recent years, accidents involving special transport vehicles have been on the rise. The occurrence of accidents is shocking, with strong hazards and great social impact. It has become a hot issue of concern to governments at all levels and the people. Therefore, it is particularly important to curb the occurrence of special transport vehicle accidents. Special transport vehicles, including hazardous chemical transport vehicles, large muck trucks, and engineering mixer trucks, have extremely high requirements for safe transportation. Traffic accidents involving such vehicles are characterized by suddenness, complexity, and serious hazards. In addition, such accidents often cause secondary injuries and are very likely to cause a series of social problems such as economic losses, environmental pollution, ecological damage, and casualties. At this stage, road safety hazards are mainly composed of the following aspects: (1) vehicle safety hazards; (2) irregular driving behavior of drivers, including making phone calls, smoking, and driving while fatigued; (3) abnormal vehicle driving status, including abnormal lane changes and irregular changes in movement trajectories; (4) illegal and irregular vehicle behaviors, such as occupying emergency lanes and illegal parking. Such behaviors seriously affect road driving safety and road traffic conditions.

[0003] With the rapid development of Internet of Things technology, artificial intelligence, and big data analysis technology, online traffic monitoring mode has been well applied in current urban roads. Such as California PeMS system, Florida 511 system, Japan VICS system and Beijing traffic flow detection system. Such online traffic monitoring system provides technical ideas for solving the supervision of special transport vehicles. Intelligent transportation system organically integrates multiple technologies such as sensing, communication, computer, control, etc. and applies them to the entire management system to complete the construction of intelligent, efficient, accurate and real-time traffic management system, which can effectively solve the supervision and management problems of road traffic. In the current intelligent transportation system, vehicle feature information can be automatically collected, and vehicle traffic information such as vehicle speed and vehicle type can be obtained. However, there is a lack of online supervision of high-risk targets such as special transport vehicles. In addition, the existing system focuses on the extraction of global information on traffic roads such as road traffic statistics and traffic control, and the supervision of dangerous driving behaviors and illegal and irregular behaviors of the above-mentioned vehicle targets themselves is not strong enough. In addition, the earlier traffic detection methods lack efficiency and are costly. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiment of the present invention provides a special vehicle rapid detection and warning system. With the increasing maturity of computer vision, image processing, 3D target detection and other technologies, the way to obtain road vehicle information through a detection system that integrates monitoring video and laser radar has been deeply studied. Using these two sensors to design a special transport vehicle rapid detection and warning system for intelligent transportation to conduct real-time online supervision of road targets can effectively improve road safety and provide reliable data support for global road scheduling.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: A special vehicle rapid detection and warning system, comprising: a monitoring front end, a back end server and an information center;

[0006] The monitoring front end includes three cameras and one laser radar, which are connected to the data processing unit of the back-end server through a switch. The data processing unit performs algorithm reasoning on the road image information and 3D point cloud information collected by the front end to obtain vehicle target detection and behavior warning information. The output information is stored in the database and sent to the information center for warning.

[0007] The monitoring front end is used to perceive road information and obtain road traffic videos and global point cloud information of the road;

[0008] The backend server is used to deploy functional models, including algorithm models such as vehicle target detection, target tracking, license plate recognition, smoke detection, and driver behavior judgment, and perform secondary detection on observed targets;

[0009] The information center is used to display detection results, real-time road monitoring images, road segmentation results, daily traffic data display and historical data analysis.

[0010] Furthermore, the camera is arranged on a road gantry, and its visual range covers 50-100 meters, including all lanes of the road; the laser radar is arranged in the center of the road at a height of 3.5-4 meters, covering a road length of 150-200 meters.

[0011] Furthermore, the three cameras perform real-time video monitoring of the road from three directions: front view, side view and rear view.

[0012] Furthermore, the workflow of the special transport vehicle rapid detection and early warning system is as follows:

[0013] Step 1: The three video devices are set up simultaneously, and the video streams are transcoded and integrated into a new URL for publication;

[0014] Step 2: Draw the roi area and divide the visible area into counting area, no-parking area and lane line, which are represented by R1, R2 and L respectively. R1 is used to trigger the counting module. The targets in this area are counted according to the target ID. At the same time, this area can be used to identify and trigger the capture time and target position; R2, L and vehicle trajectory are used for behavior judgment; L is used for joint calibration in the next step;

[0015] Step 3: Joint calibration of the camera and lidar. The lidar segments the lane lines and performs spatial registration with L.

[0016] Step 4: Fireworks detection module. This module uses the vector f[f1] to record and detect the processed video image frame using the trained firework model. The detection result is directly transmitted to the subsequent information warning. If firework is detected, the state vector becomes f[1], and the information is directly transmitted to the information center for alarm. If there is no such target, the state vector becomes f[0], and the next frame of image is detected.

[0017] Step 5: Driver behavior detection module. This module classifies driver behaviors into three categories: whether or not the driver is making or receiving phone calls, smoking, or wearing a seat belt. The module records the behaviors using vector h[h1,h2,h3]. The initial state target vector is h[0,0,0]. If any behavior is detected, the corresponding state value becomes 1, otherwise it becomes 0.

[0018] Step 6: Vehicle type recognition and tracking module. This module first detects the target in the video image, and then tracks the detected target using the set tracker. Each object is assigned a unique ID, and a dictionary is established for this ID to store its historical position value, thereby tracking its driving trajectory. The maximum length of the dictionary can be set to limit the trajectory length, and the disappeared target can be cleared in real time.

[0019] Step 7: Point cloud target detection module, first filter the point cloud data, remove the ground and background segmentation, extract the target point cloud, and put it into the model for detection to obtain the target category, spatial position, and size information;

[0020] Step 8: Fusion of the image target information obtained in step 6 and the point cloud detection results obtained in step 7, and then conditional limitation with the roi area (R2, L) calibrated in step 2, by judging the trajectory curvature, comparing with L, and extracting the relationship features with R2, the corresponding behavior judgment is obtained: the continuous mutation of the own trajectory curvature, the overlap of different detection target bbox frames further judge whether the corresponding point cloud positions overlap with each other, whether the trajectory intersects with L, and whether the trajectory appears in the R2 area, which respectively correspond to trajectory abnormality, collision, compaction line, and occupation of emergency lane; the judgment method uses the detected bbox coordinates and the pre-calibrated area boundary in the image coordinate system to perform mathematical spatial position judgment. If it is judged as abnormal behavior, the capture and license plate recognition module is triggered to capture and collect evidence of the abnormal behavior, and record the vehicle risk information; the state vector of this part is g[g1, g2, g3, g4], which respectively corresponds to trajectory abnormality, collision, abnormal lane change, and illegal lane occupation;

[0021] Step 9: Video switching module, splices the above state vectors into G[f,h,g], i.e., G[f,h1,h2,h3,g1,g2,g3,g4]. This module is mainly stimulated by the state vector G. If the state value is 0, the auxiliary view is not triggered. If the state value is 1, the rear view and side view are triggered to perform secondary detection and tracking of the corresponding functions, and the abnormal behaviors and states are continuously tracked and captured. If it is a false detection, the main view detection result is updated and corrected.

[0022] Step 10: The counting module counts the target IDs set in step 6 by category in the R1 area set in step 2 to obtain the traffic flow information;

[0023] Step 11: Information summary module, which records all detection and behavior warning information, stores such information in the database on the one hand, and transmits the data to the information center through the communication module on the other hand for real-time warning.

[0024] Technical effects and advantages of the present invention:

[0025] 1. In terms of the functions of the present invention, the system sets a driver status detection model for the supervision of driver driving behavior, which can detect the driver's phone calls and smoking; by setting the ROI area, the vehicle trajectory can be tracked and the behavior can be judged, including abnormal lane changes, collision detection, illegal parking, occupying the emergency lane, etc.; and the smoke and fire detection warning function is added;

[0026] 2. The present invention adopts the method of integrating camera and laser radar to detect targets, which effectively solves the problem of reduced detection accuracy caused by insufficient camera imaging under adverse environmental conditions such as foggy days, rainy days, and nighttime, thereby ensuring the detection accuracy of the system;

[0027] 3. The present invention uses three cameras to monitor the road environment and adopts a cross-video joint matching method to reduce the problem of false detection and missed detection caused by monitoring angle and target overlap;

[0028] 4. The camera detection area of ​​the present invention is 50-100 meters, and the laser radar detection area is 150-200 meters, which can realize the detection of long sections in a large range. The detection frame rate of the system is about 36fps, which can meet the requirements of real-time rapid detection;

[0029] 5. The present invention can stably, accurately, efficiently and all-weather real-time monitor road targets. Road historical data is stored in the database on a daily basis, including traffic flow information, behavior statistics, captured images, and license plate information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the hardware organization architecture of the special transport vehicle rapid detection and early warning system of the present invention.

[0031] Figure 2 It is a schematic diagram of the structure of each module of the system of the present invention.

[0032] Figure 3 This is a flow chart of the detection and early warning algorithm of the present invention.

[0033] Figure 4 It is a schematic diagram of expressway traffic detection of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] according to Figure 1-4 A special vehicle rapid detection and warning system as shown, comprising: Figure 1 As shown, the system consists of the following three parts:

[0036] (1) Monitoring front end

[0037] like Figure 1 As shown in the figure, the monitoring front end is the data collection part of the system and the global road environment perception part. It mainly includes three hardware devices: high-definition traffic card camera, fill light, and laser radar. Among them, three high-definition card cameras are used to conduct real-time video monitoring of the road from the front, side and rear directions. The hardware installation location is shown in Figure 4The main camera and the test camera are installed on the gantry in the direction of the oncoming vehicle, the main camera faces the lane, the test camera maintains a 30-60° viewing angle with the lane, and the viewing area covers a road length of 100-150 meters; the rear-view camera is installed in the direction of the vehicle, facing the road, and covers a road length of 100-150 meters; the laser radar is set in the center of the road at a height of 3.5-4 meters, covering a road length of 150-200 meters, and the installation position should avoid trees and other obstacles affecting radar imaging; the installation position of the fill light is consistent with the camera shooting direction, and the specific installation shall prevail with the actual on-site position.

[0038] (2) Backend Server

[0039] The backend server is mainly used to run the detection algorithm model. To ensure the stability of the algorithm and the detection speed, the GPU computing power required is 6.1 or above, and the equivalent alternative products, and the storage device requires a 500G or above disk device. The model running environment requires cuda\cudnn, torch, tensorflow2.0, and opencv support. The specific required library version is subject to the actual program algorithm development, but all require GPU computing power support.

[0040] (3) Information Center

[0041] The information center includes visual display equipment, communication equipment and hardware storage equipment. It receives algorithm reasoning warning information and displays the warning synchronously in real time.

[0042] The hardware organization structure diagram of the special transport vehicle rapid detection and early warning system of the present invention is shown in the attached figure. Figure 1 Attached Figure 2 The structure of each module of the whole system is shown. Figure 3 Show the specific algorithm flow chart, attached Figure 4 It is the installation diagram and the detection diagram.

[0043] like Figure 2 As shown, the special transport vehicle rapid detection and warning system of the present invention includes a multi-sensor joint calibration module, a ROI area calibration module, a video detection module, a point cloud detection segmentation module, a data communication module, a database, an information release module, and an information center visualization module. The specific functional modules can also be divided into: a fireworks detection module, a driver behavior judgment module, a multi-target tracking module, a behavior judgment module, a multi-view switching detection module, a traffic counting module, and a capture and license plate recognition module. The above modules are according to Figure 2The methods shown are integrated with each other, and each functional layer is directly connected in sequence. Specifically, the hardware technology layer initializes the hardware equipment, including the time synchronization setting between the three cameras, NTP time calibration; camera lidar joint calibration, using standard samples (such as lane lines) for calibration. The algorithm function layer is used to detect targets and judge and warn events with the received image data and point cloud data. The communication service layer is used to ensure the real-time and stable transmission of data, the visualization of warning results, and the storage of historical data.

[0044] like Figure 3 As shown in the figure, the working process of the special transport vehicle rapid detection and early warning system is as follows:

[0045] Step 1: The three video devices are set up simultaneously, and the video streams are transcoded and integrated into a new URL for publication. The processed video has the characteristics of low latency, high resolution, no frame stuck, and no frame dropped.

[0046] Step 2: Draw the roi area and divide the visible area into counting area, no-parking area, and lane lines, represented by R1, R2, and L. R1 is used to trigger the counting module. The targets in this area are counted according to the target ID. At the same time, this area can be used to identify and trigger the capture time and target position; R2, L and vehicle trajectory are used for behavior judgment; L is used for the next step of joint calibration.

[0047] Step 3: Joint calibration of the camera and lidar. The lidar segments the lane lines and performs spatial registration with L.

[0048] Step 4, fireworks detection module, this module uses the vector f[f1] record to detect the processed video image frame using the trained fireworks model, and the detection result is directly transmitted to the subsequent information warning. If fireworks are detected, the state vector becomes f[1], and the information is directly transmitted to the information center for alarm; if there is no such target, the state vector becomes f[0], and the next frame of the image is detected.

[0049] Step 5, driver behavior detection module, this module divides the driver's behavior into three categories, whether or not the driver makes or receives a phone call, smokes, and wears a seat belt, and records them using vector h[h1,h2,h3]. The initial state target vector is h[0,0,0]. If any behavior is detected, its corresponding state value becomes 1, otherwise it is 0.

[0050] Step 6, vehicle type recognition and tracking module, this module first detects the target in the video image, and tracks the detected target using the set tracker. Each object is assigned a unique ID, and a dictionary is established for this ID to store its historical position value, thereby tracking its driving trajectory; setting the maximum length of the dictionary can limit the trajectory length and clear the disappeared target in real time.

[0051] Step 7, point cloud target detection module, first filter the point cloud data, remove the ground and background segmentation, extract the target point cloud, and input it into the model for detection to obtain the target category, spatial position, and size information.

[0052] Step 8, the image target information obtained in step 6 and the point cloud detection result obtained in step 7 are fused, and then the roi area (R2, L) calibrated in step 2 is conditionally limited. By judging the trajectory curvature, comparing with L, and extracting the relationship features with R2, the corresponding behavior judgment is obtained: the curvature of the own trajectory undergoes continuous mutations, the bbox frames of different detection targets overlap, and further judge whether the corresponding point cloud positions overlap with each other, whether the trajectory intersects with L, and whether the trajectory appears in the R2 area. The above correspond to trajectory abnormality, whether a collision occurs, compaction line, and occupation of the emergency lane. The judgment method uses the detected bbox coordinates and the pre-calibrated regional boundaries in the image coordinate system to perform mathematical spatial position judgment. If it is judged as abnormal behavior, the capture and license plate recognition module is triggered to capture and collect evidence of the abnormal behavior, and record the vehicle risk information. The state vector of this part is g[g1, g2, g3, g4], which corresponds to trajectory abnormality, collision, abnormal lane change, and illegal lane occupation.

[0053] Step 9, video switching module, splices the above state vectors into G[f,h,g], i.e. G[f,h1,h2,h3,g1,g2,g3,g4]. This module is mainly stimulated by the state vector G. If the state value is 0, the auxiliary view is not triggered. If the state value is 1, the rear view and side view are triggered to perform secondary detection and tracking of the corresponding functions, and continue to track and capture abnormal behaviors and states. If it is a false detection, the main view detection result is updated and corrected.

[0054] Step 10, the counting module counts the target ID set in step 6 by category within the R1 area set in step 2 to obtain the traffic flow information.

[0055] Step 11, information summary module, this module records all detection and behavior warning information, on the one hand stores such information in the database, on the other hand transmits the data to the information center through the communication module for real-time warning.

[0056] Specific example description, the following introduces a specific scenario usage example, different scenarios depend on actual conditions. Figure 4In the application scenario shown, the end line of the front camera field of view is 4. Anyone entering this area will trigger the video detection function. The set roi counting area is 2-3. The targets are counted in this area. Setting the counting roi area can reduce the increase in computing power brought by the global counting. Generally speaking, the tracking state is relatively stable when reaching this area, making the counting more accurate, thereby reducing the repeated counting phenomenon caused by the instability of the detection frame at the beginning of the detection and the accumulation of ID jumps caused by long-distance target tracking errors. The vehicle continues to drive, and the historical trajectory information of the vehicle is obtained by tracking. The bbox coordinates of the trajectory and the pre-calibrated lane lines (which can be 6 / 7 in the figure, and the specific calibration depends on the actual scene) are used for geometric judgment to determine whether there is compaction line behavior; the current coordinates and the coordinates of the previous 5 frames of the own trajectory are intercepted in time sequence, and the slope of the straight line formed by these two points is calculated. By comparing the slope changes at each moment, if the slope is greater than a certain threshold, it is determined that the trajectory is abnormal. The threshold is determined by the actual installation angle of the camera, and the approximate range of the angle change △ is 60° and above; in the no-parking area (emergency lane), if there is a vehicle accident occupying the emergency lane (area 5 in the figure, the specific area is actually calibrated), the position is mathematically judged by judging the geometric shape formed by the detected target bbox coordinates and the area boundary; if there are two target detection intersection and union ratios that reach a certain threshold and there is a collision, the matched point cloud depth information is used to further judge the spatial positions of the two targets to determine whether there is a collision. Whenever the above abnormal behaviors or abnormal behaviors of the driver are detected, the snapshot and license plate recognition warning will be triggered. At the same time, the state vector G will turn on the auxiliary view detection module, continue to detect and identify from the side view and rear view, correct the front view recognition results, and send the final warning information to the information center through the communication module to wait for the staff to control. The stop line 1 in the figure is the end line of the rear view, that is, all video detection areas are between 1-4. It has been verified that the solution can achieve a detection speed of more than 36 frames per second on the Geforce RTX2080 device, and the counting accuracy can be improved to more than 99.6% through roi area counting. The three-way video auxiliary detection and the fusion of lidar and camera for target recognition and tracking can improve the detection accuracy by more than 97.3%, and can achieve real-time online monitoring around the clock and in all seasons.

Claims

1. A special vehicle rapid detection and warning system, characterized in that: include: Monitor front-end, back-end servers and information centers; The monitoring front end includes three cameras and one laser radar, which are connected to the data processing unit of the back-end server through a switch. The data processing unit performs algorithm reasoning on the road image information and 3D point cloud information collected by the front end to obtain vehicle target detection and behavior warning information. The output information is stored in the database and sent to the information center for warning. The monitoring front end is used to perceive road information and obtain road traffic videos and global point cloud information of the road; The backend server is used to deploy functional models, including algorithm models such as vehicle target detection, target tracking, license plate recognition, smoke detection, and driver behavior judgment, and perform secondary detection on observed targets; The information center is used to display detection results, real-time road monitoring images, road segmentation results, daily traffic data display and historical data analysis.

2. A special vehicle rapid detection and warning system according to claim 1, characterized in that: The camera is arranged on the road gantry, and its visual range covers 50-100 meters, including all lanes of the road; the laser radar is arranged in the center of the road at a height of 3.5-4 meters, covering a road length of 150-200 meters.

3. A special vehicle rapid detection and warning system according to claim 1, characterized in that: The three cameras respectively carry out real-time video monitoring of the road from three directions: front view, side view and rear view.

4. A special vehicle rapid detection and warning system according to claim 1, characterized in that: The workflow of the special transport vehicle rapid detection and early warning system is as follows: Step 1: The three video devices are set up simultaneously, and the video streams are transcoded and integrated into a new URL for publication; Step 2: Draw the roi area and divide the visible area into counting area, no-parking area and lane line, which are represented by R1, R2 and L respectively. R1 is used to trigger the counting module. The targets in this area are counted according to the target ID. At the same time, this area can be used to identify and trigger the capture time and target position; R2, L and vehicle trajectory are used for behavior judgment; L is used for joint calibration in the next step; Step 3: Joint calibration of the camera and lidar. The lidar segments the lane lines and performs spatial registration with L. Step 4: Fireworks detection module. This module uses the vector f[f1] to record and detect the processed video image frame using the trained firework model. The detection result is directly transmitted to the subsequent information warning. If firework is detected, the state vector becomes f[1], and the information is directly transmitted to the information center for alarm. If there is no such target, the state vector becomes f[0], and the next frame of image is detected. Step 5: Driver behavior detection module. This module classifies driver behaviors into three categories: whether or not the driver is making or receiving phone calls, smoking, or wearing a seat belt. The module records the behaviors using vector h[h1,h2,h3]. The initial state target vector is h[0,0,0]. If any behavior is detected, the corresponding state value becomes 1, otherwise it becomes 0. Step 6: Vehicle type recognition and tracking module. This module first detects the target in the video image, and then tracks the detected target using the set tracker. Each object is assigned a unique ID, and a dictionary is established for this ID to store its historical position value, thereby tracking its driving trajectory. The maximum length of the dictionary can be set to limit the trajectory length, and the disappeared target can be cleared in real time. Step 7: Point cloud target detection module, first filter the point cloud data, remove the ground and background segmentation, extract the target point cloud, and put it into the model for detection to obtain the target category, spatial position, and size information; Step 8: Fusion of the image target information obtained in step 6 and the point cloud detection results obtained in step 7, and then conditional limitation with the roi area (R2, L) calibrated in step 2, by judging the trajectory curvature, comparing with L, and extracting the relationship features with R2, the corresponding behavior judgment is obtained: the continuous mutation of the own trajectory curvature, the overlap of different detection target bbox frames further judge whether the corresponding point cloud positions overlap with each other, whether the trajectory intersects with L, and whether the trajectory appears in the R2 area, which respectively correspond to trajectory abnormality, collision, compaction line, and occupation of emergency lane; the judgment method uses the detected bbox coordinates and the pre-calibrated area boundary in the image coordinate system to perform mathematical spatial position judgment. If it is judged as abnormal behavior, the capture and license plate recognition module is triggered to capture and collect evidence of the abnormal behavior, and record the vehicle risk information; the state vector of this part is g[g1, g2, g3, g4], which respectively corresponds to trajectory abnormality, collision, abnormal lane change, and illegal lane occupation; Step 9: Video switching module, splices the above state vectors into G[f,h,g], i.e., G[f,h1,h2,h3,g1,g2,g3,g4]. This module is mainly stimulated by the state vector G. If the state value is 0, the auxiliary view is not triggered. If the state value is 1, the rear view and side view are triggered to perform secondary detection and tracking of the corresponding functions, and the abnormal behaviors and states are continuously tracked and captured. If it is a false detection, the main view detection result is updated and corrected. Step 10: The counting module counts the target IDs set in step 6 by category in the R1 area set in step 2 to obtain the traffic flow information; Step 11: Information summary module, which records all detection and behavior warning information, stores such information in the database on the one hand, and transmits the data to the information center through the communication module on the other hand for real-time warning.

Images