Traffic abnormal event detection method, electronic device, storage medium and program product

Abstract

The present invention provides a method for detecting abnormal traffic events, an electronic device, a storage medium and a program product, the method comprising acquiring traffic video data and radar data; inputting an average frame image of the traffic video data into a vehicle detection model, performing suspicious vehicle detection, and obtaining suspicious vehicles and their position information output by the vehicle detection model; clustering the position information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filtering the position information based on the clustering result to obtain the target position information of the abnormal vehicle, and performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle. The present invention occupies less memory resources, can be processed on a terminal device, and does not need to be processed on a remote device, thereby avoiding the situation where the remote transmission data is misdetected or missed, thereby improving the accuracy of detecting abnormal traffic events.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for detecting abnormal traffic events, electronic equipment, storage medium and program product. Background Art

[0002] The detection of vehicle abnormal events is one of the most demanding and time-sensitive tasks in the field of transportation. It plays an important role in preventing secondary accidents and promptly handling current traffic abnormal events. At present, human observation of whether abnormalities occur in traffic videos is inefficient and labor-intensive. Therefore, an intelligent abnormal event detection system is needed for traffic.

[0003] At present, the abnormal event detection system detects vehicles and makes abnormal judgments through optical flow method, inter-frame difference method and background difference method, but it is only applicable to simple scenes and has poor anti-interference ability. For example, when encountering rainy and snowy weather, the pixel values ​​of the picture change greatly, and the segmented foreground has a lot of noise, so the above methods are very likely to cause detection errors and are difficult to apply in practice. Based on this, the current traffic detection system mostly uses deep learning methods. However, the current deep learning methods are too complex and occupy a lot of memory resources, which makes it impossible to process on the terminal device, so decisions can only be made in the command center.

[0004] However, when the detection section is in a remote area or tunnel, the signal may be limited or degraded, and the detection data cannot be transmitted over the network or cannot be transmitted completely, resulting in false detections and missed detections, which reduces the accuracy of traffic abnormality event detection. Summary of the invention

[0005] The present invention provides a traffic abnormality event detection method, electronic equipment, storage medium and program product, which are used to solve the defect of low accuracy of traffic abnormality event detection in the prior art and realize high-accuracy traffic abnormality event detection.

[0006] The present invention provides a method for detecting abnormal traffic events, comprising:

[0007] Acquire traffic video data and radar data;

[0008] Inputting the average frame image of the traffic video data into a vehicle detection model to perform suspicious vehicle detection, and obtaining suspicious vehicles and their location information output by the vehicle detection model;

[0009] Performing clustering processing on the location information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result;

[0010] If the suspicious vehicle is an abnormal vehicle, the location information is filtered based on the clustering result to obtain the target location information of the abnormal vehicle, and a decision-level fusion is performed based on the radar data and the target location information to obtain the abnormal event category of the abnormal vehicle.

[0011] According to a method for detecting abnormal traffic events provided by the present invention, the step of inputting the average frame image of the traffic video data into a vehicle detection model comprises:

[0012] Based on a preset averaging period, sampling the frame images of the traffic video data to obtain a frame image set;

[0013] Performing average calculation on the frame image set to obtain an average frame image;

[0014] The average frame image is input into a vehicle detection model.

[0015] According to a method for detecting abnormal traffic events provided by the present invention, judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result includes:

[0016] If the number of location information of the cluster center of the clustering result is greater than a preset number, the suspicious vehicle is determined to be an abnormal vehicle;

[0017] If the number of location information of the cluster center of the clustering result is less than or equal to a preset number, the suspicious vehicle is determined to be a normal vehicle.

[0018] According to a method for detecting abnormal traffic events provided by the present invention, the decision-level fusion is performed based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle, including:

[0019] Analyzing and processing the radar data to obtain the speed direction, average speed and acceleration of the abnormal vehicle;

[0020] Based on the speed direction, the average speed, the acceleration and the target position information, a decision-level fusion is performed to obtain the abnormal event category of the abnormal vehicle.

[0021] According to a method for detecting abnormal traffic events provided by the present invention, the decision-level fusion is performed based on the speed direction, the average speed, the acceleration and the target position information to obtain the abnormal event category of the abnormal vehicle, including:

[0022] If the speed direction is opposite to the preset speed direction, it is determined that the abnormal vehicle has a reverse driving event;

[0023] If the average speed is less than a preset average speed, it is determined that the abnormal vehicle is in a traffic congestion event;

[0024] If the acceleration is greater than a preset acceleration, it is determined that the abnormal vehicle has a traffic accident event;

[0025] If the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is located in the emergency lane area, then it is determined that the abnormal vehicle has an emergency lane parking event;

[0026] If the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is not located in the emergency lane area, it is determined that the abnormal vehicle has an illegal parking event.

[0027] A traffic abnormality event detection method provided by the present invention is applied to an edge device, wherein the edge device includes a memory module, a central processing unit, a graphics processing unit, a storage device, and a network port device;

[0028] The network port device is used to obtain the traffic video data and the radar data;

[0029] The central processing unit is used to perform average frame processing to obtain the average frame image;

[0030] The memory module is used to store the average frame image and the radar data;

[0031] The graphics processor is used to read the engine file in the memory, and deserialize the engine file to obtain the vehicle detection model, so as to perform inference on the average frame image through the vehicle detection model;

[0032] The central processor is further used to perform clustering processing on the location information;

[0033] The central processing unit is also used for decision-level fusion.

[0034] According to a method for detecting abnormal traffic events provided by the present invention, the memory module includes a first shared memory space and a second shared memory space, and the first shared memory space and the second shared memory space are read and written alternately.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned abnormal traffic event detection methods are implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned abnormal traffic event detection methods are implemented.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned abnormal traffic event detection methods are implemented.

[0038] The traffic abnormal event detection method, electronic device, storage medium and program product provided by the present invention obtain traffic video data and radar data; input the average frame image of the traffic video data into the vehicle detection model, perform suspicious vehicle detection, obtain the suspicious vehicle and its position information output by the vehicle detection model; perform clustering processing on the position information, obtain the clustering result, and judge whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filter the position information based on the clustering result to obtain the target position information of the abnormal vehicle, perform decision-level fusion based on the radar data and the target position information, and obtain the abnormal event category of the abnormal vehicle. The present invention only needs to input traffic video data and radar data to perform abnormal event detection. Compared with the multi-source input traffic event detection system, it can occupy less memory resources, so that it can be processed on the terminal device, and there is no need to transmit each modal data back to the command center for processing, so as to avoid the situation of false detection and missed detection of remote transmission data, thereby improving the accuracy of traffic abnormal event detection. At the same time, compared with the original frame image, the average frame image can blur the normal vehicle information, thereby avoiding the interference of normal vehicle information, and then highlighting the abnormal vehicle information, and finally further improving the accuracy of traffic abnormal event detection. In addition, the location information is clustered to determine whether the suspicious vehicle is an abnormal vehicle. Compared with directly subjecting the suspicious vehicle to abnormal event detection, the present invention can avoid subjecting normally traveling suspicious vehicles to abnormal event category detection, thereby further improving the accuracy of abnormal traffic event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 One of the flow charts of the abnormal traffic event detection method provided by the present invention;

[0041] Figure 2 The second flowchart of the abnormal traffic event detection method provided by the present invention;

[0042] Figure 3 A schematic diagram of an edge device provided by the present invention;

[0043] Figure 4 A schematic diagram of the system architecture provided by the present invention;

[0044] Figure 5 The figure is a schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Figure 1 One of the flow charts of the abnormal traffic event detection method provided by the present invention is as follows Figure 1 As shown, the abnormal traffic event detection method provided by the present invention includes:

[0047] Step 110, acquiring traffic video data and radar data;

[0048] In this embodiment, the abnormal traffic event detection method can be applied to highway scenarios, especially to expressway scenarios, that is, the abnormal traffic event detection method based on expressways.

[0049] The traffic video data includes vehicle information on the target detection section, that is, traffic video data obtained by shooting the target detection section. The traffic video data can be obtained by a video acquisition device (such as a camera or a video camera).

[0050] In one embodiment, the abnormal traffic event detection method is applied to a abnormal traffic event detection system, and the abnormal traffic event detection system is connected to a camera; the camera is used to obtain the traffic video data in real time and transmit the traffic video data to the abnormal traffic event detection system.

[0051] In another embodiment, the abnormal traffic event detection method is applied to an edge device, and the edge device includes a network port device; the network port device is used to obtain the traffic video data.

[0052] The network port device may be a POE (Power Over Ethernet) network port. The network port device is used to obtain traffic video data collected by the video acquisition device, that is, to obtain traffic video data transmitted by the video acquisition device, that is, the network port device is used to connect with the video acquisition device to transmit road information to the edge device.

[0053] Among them, radar data is used to analyze and calculate the speed, speed direction, acceleration and average speed of the target vehicle. The radar data is collected by radar equipment.

[0054] In one embodiment, the abnormal traffic event detection method is applied to an abnormal traffic event detection system, and the abnormal traffic event detection system communicates with a radar; the radar is used to obtain the radar data in real time and transmit the radar data to the abnormal traffic event detection system.

[0055] In another embodiment, the abnormal traffic event detection method is applied to an edge device, and the edge device includes a network port device; the network port device is used to obtain the radar data.

[0056] The network port device may be a POE network port. The network port device is used to obtain radar data collected by the radar, that is, to obtain radar data transmitted by the radar, that is, the network port device is used to communicate with the radar to transmit radar information to the edge device.

[0057] Furthermore, the traffic video data and the radar data are stored in a shared memory space of a memory module.

[0058] Step 120, inputting the average frame image of the traffic video data into a vehicle detection model to perform suspicious vehicle detection, and obtaining suspicious vehicle and position information output by the vehicle detection model;

[0059] In this embodiment, the average frame image is obtained by performing average frame processing on the traffic video data, and the average frame processing includes sampling processing and average calculation processing.

[0060] Specifically, the step of inputting the average frame image of the traffic video data into the vehicle detection model includes:

[0061] Step 121, sampling the frame images of the traffic video data based on a preset averaging period to obtain a frame image set;

[0062] Step 122, performing average calculation on the frame image set to obtain an average frame image;

[0063] Step 123, inputting the average frame image into a vehicle detection model.

[0064] In this embodiment, the preset averaging period can be set according to actual needs. For example, the preset averaging period is 3 seconds, and 30 seconds is used as a detection period. Then, 30 frames of images can be collected every 3 seconds and averaged to obtain an average frame image, that is, 10 average frame images are generated in each detection period, and then the 10 average frame images of a detection period are input into the vehicle detection model.

[0065] The number of frame images in the frame image set can be set according to actual needs. For example, in each preset averaging period, 30 frame images are sampled to form a frame image set.

[0066] In this embodiment, the traffic video data is processed by averaging frames, which can blur the normal vehicle information, thereby avoiding the interference of the normal vehicle information, and then highlighting the abnormal vehicle information, and finally improving the accuracy of abnormal traffic event detection.

[0067] The vehicle detection model is a machine learning model, specifically, a deep learning model. The vehicle detection model can be trained on another device, and then the trained vehicle detection model is deployed on the current device.

[0068] In one embodiment, the vehicle detection model is obtained based on the training of the YOLOv5 neural network. The YOLOv5 neural network can be retrained to enable it to detect models such as cars, trucks, and buses. That is, the output categories of the network can be modified to 3 categories, so that the vehicle detection model can detect vehicles and models.

[0069] Furthermore, the above step 120 includes:

[0070] The average frame image of the traffic video data is input into the vehicle detection model to detect suspicious vehicles, and the suspicious vehicles and their vehicle types and location information output by the vehicle detection model are obtained. The vehicle types include cars, trucks, buses and other models.

[0071] Among them, the suspicious vehicle may be an abnormal vehicle or a normal vehicle. Therefore, it is necessary to further determine whether the suspicious vehicle is an abnormal vehicle.

[0072] In one embodiment, the abnormal traffic event detection method is applied to an abnormal traffic event detection system, which includes a preprocessing module and a suspicious vehicle detection module; the preprocessing module is used to perform average frame processing on traffic video data to obtain an average frame image, and is also used to receive radar data and write the average frame image and the radar data into a shared memory; the suspicious vehicle detection module is used to detect suspicious vehicles on the average frame image obtained by the preprocessing module.

[0073] The preprocessing module includes the connection between the traffic abnormal event detection system and the camera, and the communication between the traffic abnormal event detection system and the radar. The camera is used to obtain traffic video data in real time and read the video frames of the traffic video data into the memory DRAM. The CPU obtains the average frame image by sampling and calculating the video frames, and writes the average frame image into the shared memory. In addition, while calculating the average frame image, the traffic abnormal event detection system continuously receives radar data transmitted by the radar through the TCP transmission protocol, that is, traffic parameter data, and writes the radar data into the shared memory.

[0074] The suspicious vehicle detection module includes the scheduling of GPU and the reasoning of neural network. Specifically, the traffic abnormal event detection system reads the vehicle detection model from ROM to GPU through the memory control structure. The GPU reads the average frame image from the shared memory and performs neural network reasoning. The reasoning results (suspicious vehicles and their vehicle types and location information) are input into the abnormal vehicle reasoning module for the next step of reasoning confirmation.

[0075] In another embodiment, the abnormal traffic event detection method is applied to an edge device, which includes a central processing unit, a memory and a graphics processor; the central processing unit is used to perform average frame processing to obtain the average frame image; the graphics processor is used to read the engine file in the memory and deserialize the engine file to obtain the vehicle detection model.

[0076] Specifically, the GPU graphics processor reads the engine file of the neural network in the ROM memory, reads the average frame image in the shared memory space into the vehicle detection model, performs accelerated reasoning on the vehicle detection model, and obtains suspicious vehicle detection results (suspicious vehicles and their vehicle types and location information).

[0077] Step 130, clustering the location information to obtain a clustering result, and determining whether the suspicious vehicle is an abnormal vehicle based on the clustering result;

[0078] In this embodiment, the algorithm used in the clustering process may be a K-means algorithm or a K-center point algorithm.

[0079] Specifically, judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result includes:

[0080] Step 131, if the number of location information of the cluster center of the clustering result is greater than a preset number, the suspicious vehicle is determined to be an abnormal vehicle;

[0081] Step 132: If the number of location information of the cluster center of the clustering result is less than or equal to a preset number, the suspicious vehicle is determined to be a normal vehicle.

[0082] In this embodiment, the preset number can be set according to actual needs, for example, 5, 6, 7, etc., and is not specifically limited here.

[0083] In a specific embodiment, assuming that the preset number is 6, when there are more than 6 vehicle location information in the cluster center, the suspicious vehicle is considered to be an abnormal vehicle, so that the radar information fusion module can subsequently determine the category of the abnormal event; otherwise, the suspicious vehicle is considered to be a vehicle that moves relatively slowly in the video screen but is driving normally, and no subsequent processing is performed.

[0084] Step 140: If the suspicious vehicle is an abnormal vehicle, the location information is filtered based on the clustering result to obtain the target location information of the abnormal vehicle, and a decision-level fusion is performed based on the radar data and the target location information to obtain the abnormal event category of the abnormal vehicle.

[0085] Specifically, based on the above clustering process, that is, by performing K-means clustering on the central position parameters of the suspicious vehicle to filter the normal vehicle information, the position information (target position information) of the abnormal vehicle is obtained.

[0086] It should be noted that obtaining the abnormal event category of abnormal vehicles can assist the transportation department in better managing road traffic, especially for highways.

[0087] Among them, abnormal event categories may include wrong-way events, traffic congestion events, traffic accidents, emergency lane parking events, and illegal parking events.

[0088] In one embodiment, the abnormal traffic event detection method is applied to a abnormal traffic event detection system, and the abnormal traffic event detection system includes a radar information fusion module; the radar information fusion module is used to infer the location information of a suspicious vehicle, determine whether the suspicious vehicle is an abnormal vehicle, and perform decision-level fusion of the inference result with the radar data received by the abnormal traffic event detection system to determine the abnormal event category of the abnormal vehicle.

[0089] The radar information fusion module includes the reasoning of the position information of suspicious vehicles, the detection of radar data anomalies, and the fusion of radar data. Specifically, the center position parameters of the suspicious vehicle are subjected to K-means clustering to filter the normal vehicle information, and the position information (target position information) of the abnormal vehicle is obtained. The radar data transmitted from the radar in the shared memory to the traffic anomaly event detection system is read to obtain the traffic parameters detected by the radar, and then the image detection results (target position information of the abnormal vehicle) and the radar data are fused at the decision level to determine the abnormal event category of the abnormal vehicle.

[0090] In another embodiment, the abnormal traffic event detection method is applied to an edge device, and the edge device includes a central processing unit; the central processing unit is also used to cluster the location information of suspicious vehicles; the central processing unit is also used to perform decision-level fusion.

[0091] Furthermore, the abnormal traffic event detection system may also include a system communication coordination module, which is used to link the preprocessing module, the suspicious vehicle detection module, and the radar information fusion module to achieve real-time and uninterrupted detection of abnormal traffic events.

[0092] The system communication coordination module is used to coordinate the mutual calls between the various modules in the traffic anomaly event detection system, and realize the information exchange between modules through the TCP protocol. The preprocessing module can be used as a TCP client, and the port is randomly bound by the system. The suspicious vehicle detection module is used as the communication server, and the port is bound to a non-special port number. After the average frame processing of the preprocessing module is completed, a request is sent to the suspicious vehicle reasoning module, and the suspicious vehicle detection module is called to obtain the detection results (suspicious vehicles and their vehicle types and location information) by reading the average frame image in the specified shared memory and using the GPU for network reasoning.

[0093] Furthermore, after the above step 130, if the suspicious vehicle is a normal vehicle, the next detection cycle is entered, that is, returning to the above step 110.

[0094] It can be understood that, at present, the abnormal event detection system includes a single-source input traffic detection system and a multi-source input traffic detection system. The single-source input traffic detection system usually relies only on video information and obtains the detection result by processing the video information. However, it can detect fewer types of traffic events. Although the multi-source input traffic detection system can detect more types of traffic events, it needs to input more detection data. Since the current analysis algorithm for multiple detection data is too complicated and cannot be processed on the terminal device, it can only make decisions in the command center. However, when the detection section is in a remote area or tunnel, the signal may be limited or degraded, and the detection data cannot be transmitted over the network or cannot be transmitted completely, resulting in false detection and missed detection, resulting in a decrease in the accuracy of traffic abnormal event detection. Based on this, the embodiment of the present invention only needs to input traffic video data and radar data to perform abnormal event detection. Compared with the multi-source input traffic event detection system, it can occupy less memory resources, so that it can be processed on the terminal device without processing on the remote device, thereby avoiding the situation of false detection and missed detection of remotely transmitted data, thereby improving the accuracy of traffic abnormal event detection.

[0095] According to the abnormal traffic event detection method of the embodiment of the present invention, by acquiring traffic video data and radar data; inputting the average frame image of the traffic video data into the vehicle detection model, performing suspicious vehicle detection, and obtaining the suspicious vehicle and its position information output by the vehicle detection model; clustering the position information to obtain the clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filtering the position information based on the clustering result to obtain the target position information of the abnormal vehicle, and performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle. The embodiment of the present invention only needs to input traffic video data and radar data to detect abnormal events. Compared with the multi-source input traffic event detection system, it can occupy less memory resources, so that it can be processed on the terminal device without processing on the remote device, thereby avoiding the situation of false detection and missed detection of remote transmission data, thereby improving the accuracy of traffic abnormal event detection. At the same time, compared with the original frame image, the average frame image can blur the normal vehicle information, thereby avoiding the interference of normal vehicle information, and then highlighting the abnormal vehicle information, and finally further improving the accuracy of traffic abnormal event detection. In addition, the location information is clustered to determine whether the suspicious vehicle is an abnormal vehicle. Compared with directly subjecting the suspicious vehicle to abnormal event detection, the embodiment of the present invention can avoid subjecting normally traveling suspicious vehicles to abnormal event category detection, thereby further improving the accuracy of abnormal traffic event detection.

[0096] Furthermore, based on the above first embodiment, a second embodiment of the abnormal traffic event detection method of the present invention is proposed. Figure 2 The second flow chart of the abnormal traffic event detection method provided by the present invention is as follows: Figure 2 As shown, in this embodiment, in the above step 140, based on the radar data and the target position information, a decision-level fusion is performed to obtain the abnormal event category of the abnormal vehicle, including:

[0097] Step 141, analyzing and processing the radar data to obtain the speed direction, average speed and acceleration of the abnormal vehicle;

[0098] Step 142: Perform decision-level fusion based on the speed direction, the average speed, the acceleration and the target position information to obtain the abnormal event category of the abnormal vehicle.

[0099] In this embodiment, the speed direction is used to indicate the driving direction of the vehicle, so as to determine whether the vehicle is traveling in the wrong direction based on the speed direction.

[0100] The average speed is used to represent the average speed of all vehicles in the current detection cycle, so as to determine whether the current detection section has traffic congestion based on the average speed.

[0101] Acceleration is used to indicate changes in the vehicle's driving speed, so as to determine whether the vehicle has been involved in a traffic accident based on the acceleration. If no traffic accident has occurred, the type of abnormal parking event is determined based on the target position information. The abnormal parking event type may include emergency lane parking events and illegal parking events, etc.

[0102] Specifically, the above step 142 includes:

[0103] Step 1421, if the speed direction is opposite to the preset speed direction, it is determined that the abnormal vehicle has a wrong-way event;

[0104] Among them, the preset speed direction is the speed direction corresponding to the normal driving direction of the current detection section.

[0105] In a specific embodiment, after the radar data is read, when a vehicle speed in the radar data in the current detection cycle is a negative value, it is considered that a vehicle reverse driving event occurs.

[0106] Step 1422, if the average speed is less than a preset average speed, it is determined that the abnormal vehicle has a traffic congestion event;

[0107] The preset average speed can be set according to actual conditions, for example, 20 m / s.

[0108] In a specific embodiment, if the average speed of vehicles in the current detection period is lower than 20 m / s, it is considered that a traffic congestion event exists.

[0109] Step 1423: if the acceleration is greater than a preset acceleration, it is determined that a traffic accident event occurs with the abnormal vehicle;

[0110] The preset acceleration can be set according to the actual situation, for example, 15m / s 2 .

[0111] In a specific embodiment, after the current detection cycle determines that an abnormal vehicle is detected, if there is an acceleration exceeding 15m / s in the radar data 2 , it is believed that there was a traffic accident.

[0112] Step 1424: if the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is located in the emergency lane area, then it is determined that the abnormal vehicle has an emergency lane parking event;

[0113] Step 1425: If the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is not located in the emergency lane area, it is determined that the abnormal vehicle has an illegal parking event.

[0114] In a specific embodiment, when there is no abnormal acceleration value in the radar data, an abnormal parking event is judged based on the target position information of the abnormal vehicle. When the coordinates of the abnormal vehicle are in the emergency lane area, it is considered to be an emergency lane parking event. When the coordinates of the abnormal vehicle are not in the emergency lane area, it is considered to be an illegal parking event.

[0115] In this embodiment, based on the speed direction, average speed, acceleration and target position information of the abnormal vehicle, a variety of abnormal event categories of the abnormal vehicle can be obtained, so that the embodiment of the present invention only needs to input traffic video data and radar data to perform abnormal event detection. Compared with the multi-source input traffic event detection system, it can occupy less memory resources, so that it can be processed on the terminal device without processing on the remote device, thereby avoiding the situation of false detection and missed detection in the remote transmission data, thereby improving the accuracy of traffic abnormal event detection.

[0116] Furthermore, based on the first and / or second embodiments above, a third embodiment of the traffic abnormality event detection method of the present invention is proposed. In this embodiment, the traffic abnormality event detection method is applied to an edge device, and the edge device includes a memory module, a central processing unit, a graphics processing unit, a storage device, and a network port device;

[0117] In this embodiment, the edge device may be an NVIDIA edge device.

[0118] The network port device is used to obtain the traffic video data and the radar data;

[0119] The network port device may be a POE (Power Over Ethernet) network port.

[0120] The network port device is used to obtain the traffic video data collected by the video acquisition device, that is, it is used to obtain the traffic video data transmitted by the video acquisition device. In other words, the network port device is used to connect with the video acquisition device to transmit the road surface information to the edge device.

[0121] The network port device is used to obtain radar data collected by the radar, that is, to obtain radar data transmitted by the radar. In other words, the network port device is used to communicate with the radar to transmit the radar information to the edge device.

[0122] The central processing unit is used to perform average frame processing to obtain the average frame image;

[0123] Specifically, the central processor samples the frame images of the traffic video data based on a preset averaging period to obtain a frame image set; and performs an average calculation on the frame image set to obtain an average frame image.

[0124] The memory module is used to store the average frame image and the radar data;

[0125] Among them, the memory module can be a DRAM, and the memory module can open up two shared memory spaces to avoid conflicts caused by simultaneous access to a piece of memory by read and write operations during real-time detection of edge devices.

[0126] Furthermore, the memory module includes a first shared memory space and a second shared memory space, and the first shared memory space and the second shared memory space are read and written alternately.

[0127] Specifically, the first shared memory space and the second shared memory space are used to alternately store the average frame image and radar data in the current detection cycle. When the first shared memory space is accessed by the suspicious vehicle detection module, the average frame image and radar data are written into the second shared memory space, and when the second shared memory space is accessed by the suspicious vehicle detection module, the average frame image and radar data are written into the first shared memory space.

[0128] The graphics processor is used to read the engine file in the memory and deserialize the engine file to obtain the vehicle detection model;

[0129] Specifically, the engine file of the neural network in the ROM memory 1 is read through the graphics processing unit (GPU), and the average frame image in the shared memory space is read into the vehicle detection model, and the neural network is accelerated to infer and obtain the suspicious vehicle detection result (suspicious vehicle and its vehicle type and location information)

[0130] The central processor is further used to perform clustering processing on the location information;

[0131] Specifically, the location information is clustered through the central processing unit to obtain a clustering result. If the number of location information of the cluster center of the clustering result is greater than a preset number, the suspicious vehicle is determined to be an abnormal vehicle; if the number of location information of the cluster center of the clustering result is less than or equal to a preset number, the suspicious vehicle is determined to be a normal vehicle.

[0132] The central processing unit is also used for decision-level fusion.

[0133] Specifically, the radar data is analyzed and processed by a central processing unit to obtain the speed direction, average speed and acceleration of the abnormal vehicle; based on the speed direction, the average speed, the acceleration and the target position information, a decision-level fusion is performed to obtain the abnormal event category of the abnormal vehicle.

[0134] It should be noted that the structure of the edge device and the abnormal traffic event detection method thereof in this embodiment can refer to the above-mentioned first embodiment and second embodiment, and will not be described one by one here.

[0135] For ease of understanding, refer to Figure 3 , Figure 3 Schematic diagram of the edge device provided by the present invention. In the NVIDIA edge device 1. Two shared memory spaces 3 are opened in the 8GB DRAM memory module 2 to avoid conflicts caused by simultaneous access to a piece of memory during real-time detection. The CPU central processing unit 4 is used to perform operations such as averaging images, receiving radar data, abnormal vehicle reasoning, radar-visual information decision-level fusion, and TCP communication within the system. The GPU graphics processor 5 reads the neural network engine file in the ROM memory 11, reads the average frame image in the shared memory space 3 into the neural network, accelerates the reasoning of the neural network, and obtains suspicious vehicle detection results. The memory controller structure (Memory Controller Fabric) 6 is used to implement memory access by various parts inside the development board. The POE network port 7 is used to connect the camera 10 to transmit road information to the edge device. The POE network port 7 is also used to communicate with the radar 9 to obtain traffic parameter information detected by the radar, and send the results obtained for the current detection cycle (abnormal event category of abnormal vehicles) to the monitoring center 8.

[0136] In this embodiment, the abnormal traffic event detection method can be deployed to the edge computing device, so that abnormal event detection can be performed in real time directly in remote sections of the road and tunnels with poor signals. Compared with processing on remote devices, it has stronger practicality and avoids false detection caused by data transmission jams due to poor signals, thereby improving the accuracy of abnormal traffic event detection. There is no need to transmit video recordings to the monitoring center, saving a lot of data transmission costs.

[0137] The abnormal traffic event detection system provided by the present invention is described below. The abnormal traffic event detection system described below and the abnormal traffic event detection method described above can be referred to each other.

[0138] For ease of understanding, refer to Figure 4 , Figure 4 The system architecture diagram provided by the present invention is as follows: The traffic abnormality event detection system includes a preprocessing module, a suspicious vehicle detection module, and a radar information fusion module.

[0139] The main functions of the preprocessing module are: receiving video data, calculating the average frame image, receiving radar data packets for analysis, and putting the average frame image and radar analysis results into shared memory. On roads (such as highways), normal vehicles will remain in a driving state, and abnormal vehicles will be in a stopped state in abnormal events such as traffic accidents, emergency lane parking, and illegal parking. After the average frame processing, the normal vehicle information is blurred and the abnormal vehicle information is highlighted. In addition, the preprocessing module receives radar data and analyzes it according to the communication protocol. The system opens up two shared memory spaces in the memory, which are named shared memory 1 and shared memory 2 for convenience of description, to alternately store the average frame image and radar data in the current detection cycle. When shared memory 1 is accessed by the suspicious vehicle detection module, the average frame image and radar data are written to shared memory 2, and when shared memory 2 is accessed by the suspicious vehicle detection module, the average frame image and radar data are written to shared memory 1.

[0140] The main function of the suspicious vehicle detection module is to perform vehicle detection on the average frame image obtained by the preprocessing module, and all detected vehicles are considered suspicious vehicles. The suspicious vehicle detection module requires training of the vehicle detection network. In this embodiment, the YOLOv5 network is retrained, and the vehicles are divided into three categories: cars, trucks, and buses. The output category of the network is modified to 3 categories, so that the network can detect vehicles and vehicle models. The trained network is called a vehicle detection model in this embodiment and is deployed to the terminal device. The vehicle detection model detects 10 average frame images of a detection cycle, and saves the detected vehicle type and location information as a json format file for reading by the abnormal vehicle reasoning module.

[0141] The main function of the radar information fusion module is to process the vehicle position information obtained by the suspicious vehicle detection module, determine whether the suspicious vehicle is an abnormal vehicle, and fuse it with the radar data. First, read the suspicious vehicle detection results, and then perform K-means clustering on the results. When there are more than 6 vehicle position information in the cluster center, the suspicious vehicle is considered an abnormal vehicle, and the radar information fusion module determines the category of the abnormal event; otherwise, the suspicious vehicle is considered to be a relatively slow but normal vehicle, and no subsequent processing is performed. Secondly, read the radar data from the shared memory, and combine the results of the abnormal vehicle reasoning module to determine abnormal events such as traffic accidents, illegal parking, and emergency lane parking for abnormal vehicles, and use radar data to make judgments on events such as reversing and traffic congestion.

[0142] Specifically, to achieve real-time monitoring of the traffic anomaly event detection system. The camera and radar continuously monitor the road conditions, and input the video frame sequence and radar detection information into the traffic anomaly event detection system based on radar-vision fusion. When a traffic accident occurs within the current detection cycle, the average frame image generated by the system preprocessing module will highlight the abnormal vehicle information involved in the accident, and the normal vehicles that have not been involved in the accident will be blurred by the average, leaving only weak residual vehicle information that does not affect the detection in the image. The vehicle has a sudden change in acceleration due to the traffic accident, so the radar information received by the preprocessing module will show an acceleration greater than 15m / s 2 acceleration value. After the current detection cycle ends, the average frame image and radar information are saved in shared memory 1, and a message is sent to the server through the TCP protocol, indicating that shared memory 1 can be read by the server. At this time, the entire system enters the next detection cycle. In the next detection cycle, the preprocessing module continues to process the average frame and receive radar information, and writes the result to shared memory 2. After the server receives the client request, it reads the average frame image and radar information from shared memory 1. After the average frame image is read by the suspicious vehicle detection module, it is inferred to obtain the type and location information of the abnormal vehicle, and the location information is input into the abnormal vehicle reasoning module. The cluster center is calculated by the elbow rule of K-means clustering. If the number of vehicle location information in the cluster center exceeds 6, the suspicious vehicle is considered to be an accident vehicle, and then the radar information fusion module reads the accident vehicle information and radar information. There is an accident vehicle during the detection cycle, and its acceleration is greater than 15m / s 2 , it is determined that an abnormal traffic accident occurred in the previous cycle, the type of accident vehicle is recorded, and reported to the command center.

[0143] The abnormal traffic event detection device provided by the present invention is described below. The abnormal traffic event detection device described below and the abnormal traffic event detection method described above can be referred to each other.

[0144] In this embodiment, the abnormal traffic event detection device includes:

[0145] An acquisition module, used to acquire traffic video data and radar data;

[0146] A detection module, used for inputting the average frame image of the traffic video data into a vehicle detection model, performing suspicious vehicle detection, and obtaining suspicious vehicle and position information output by the vehicle detection model;

[0147] A clustering module, used for clustering the location information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result;

[0148] A decision-making module is used to filter the location information based on the clustering result to obtain the target location information of the abnormal vehicle if the suspicious vehicle is an abnormal vehicle, and perform decision-level fusion based on the radar data and the target location information to obtain the abnormal event category of the abnormal vehicle.

[0149] Figure 5 is a schematic diagram of the physical structure of an electronic device, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the abnormal traffic event detection method, which includes: obtaining traffic video data and radar data; inputting the average frame image of the traffic video data into the vehicle detection model, performing suspicious vehicle detection, and obtaining the suspicious vehicle and its position information output by the vehicle detection model; clustering the position information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filtering the position information based on the clustering result to obtain the target position information of the abnormal vehicle, and performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle.

[0150] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0151] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the traffic abnormal event detection method provided by the above-mentioned methods, and the method includes: obtaining traffic video data and radar data; inputting the average frame image of the traffic video data into a vehicle detection model, performing suspicious vehicle detection, and obtaining the suspicious vehicle and its position information output by the vehicle detection model; clustering the position information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filtering the position information based on the clustering result to obtain the target position information of the abnormal vehicle, and performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle.

[0152] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the traffic abnormal event detection method provided by the above-mentioned methods, the method comprising: acquiring traffic video data and radar data; inputting the average frame image of the traffic video data into a vehicle detection model, performing suspicious vehicle detection, and obtaining the suspicious vehicle and its position information output by the vehicle detection model; clustering the position information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; if the suspicious vehicle is an abnormal vehicle, filtering the position information based on the clustering result to obtain the target position information of the abnormal vehicle, and performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle.

[0153] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal traffic events, characterized in that: include: Acquire traffic video data and radar data; Inputting the average frame image of the traffic video data into a vehicle detection model to perform suspicious vehicle detection, and obtaining suspicious vehicles and their location information output by the vehicle detection model; Performing clustering processing on the location information to obtain a clustering result, and judging whether the suspicious vehicle is an abnormal vehicle based on the clustering result; If the suspicious vehicle is an abnormal vehicle, the location information is filtered based on the clustering result to obtain the target location information of the abnormal vehicle, and a decision-level fusion is performed based on the radar data and the target location information to obtain the abnormal event category of the abnormal vehicle; The determining whether the suspicious vehicle is an abnormal vehicle based on the clustering result includes: If the number of location information of the cluster center of the clustering result is greater than a preset number, the suspicious vehicle is determined to be an abnormal vehicle; If the number of location information of the cluster center of the clustering result is less than or equal to a preset number, the suspicious vehicle is determined to be a normal vehicle.

2. The abnormal traffic event detection method according to claim 1, characterized in that: The step of inputting the average frame image of the traffic video data into a vehicle detection model comprises: Based on a preset averaging period, sampling the frame images of the traffic video data to obtain a frame image set; Performing average calculation on the frame image set to obtain an average frame image; The average frame image is input into a vehicle detection model.

3. The abnormal traffic event detection method according to claim 1, characterized in that: The performing decision-level fusion based on the radar data and the target position information to obtain the abnormal event category of the abnormal vehicle includes: Analyzing and processing the radar data to obtain the speed direction, average speed and acceleration of the abnormal vehicle; Based on the speed direction, the average speed, the acceleration and the target position information, a decision-level fusion is performed to obtain the abnormal event category of the abnormal vehicle.

4. The abnormal traffic event detection method according to claim 3, characterized in that: The performing decision-level fusion based on the speed direction, the average speed, the acceleration and the target position information to obtain the abnormal event category of the abnormal vehicle includes: If the speed direction is opposite to the preset speed direction, it is determined that the abnormal vehicle has a reverse driving event; If the average speed is less than a preset average speed, it is determined that the abnormal vehicle is in a traffic congestion event; If the acceleration is greater than a preset acceleration, it is determined that the abnormal vehicle has a traffic accident event; If the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is located in the emergency lane area, then it is determined that the abnormal vehicle has an emergency lane parking event; If the acceleration is less than or equal to the preset acceleration, and it is determined based on the target position information that the abnormal vehicle is not located in the emergency lane area, it is determined that the abnormal vehicle has an illegal parking event.

5. The abnormal traffic event detection method according to any one of claims 1 to 4, characterized in that: Applied to edge devices, the edge devices include memory modules, central processing units, graphics processing units, storage devices and network port devices; The network port device is used to obtain the traffic video data and the radar data; The central processing unit is used to perform average frame processing to obtain the average frame image; The memory module is used to store the average frame image and the radar data; The graphics processor is used to read the engine file in the memory, and deserialize the engine file to obtain the vehicle detection model, so as to perform inference on the average frame image through the vehicle detection model; The central processor is further used to perform clustering processing on the location information; The central processing unit is also used for decision-level fusion.

6. The abnormal traffic event detection method according to claim 5, characterized in that: The memory module includes a first shared memory space and a second shared memory space, and the first shared memory space and the second shared memory space are read and written alternately.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the abnormal traffic event detection method according to any one of claims 1 to 6 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal traffic event detection method according to any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the abnormal traffic event detection method according to any one of claims 1 to 6 are implemented.

Images