A video-based intelligent detection system and method for multiple data source events on expressways

Abstract

The application relates to the expressway field and aims to provide an expressway multi-data-source event intelligent detection system and method based on video, so as to reduce the number of traffic event reports and the workload of monitoring personnel. The technical scheme is an expressway multi-data-source event intelligent detection system based on video, characterized in that the system comprises: a collection module used for collecting traffic event videos and extracting traffic event data from the videos; a fusion module used for fusing and comparing the traffic event data; and a feedback module used for auditing and feeding back the traffic event data that cannot be fused.

Description

Technical Field

[0001] This invention relates to the field of highways, specifically to an intelligent event detection system and method for highways using video as the primary data source. Background Technology

[0002] Intelligent event detection is a crucial component of smart highways and a key support for road perception and control. Currently, there are various anomaly detection methods on the market based on data sources such as video, radar, and floating cars. While these methods demonstrate good detection results in the laboratory, their performance in engineering applications is often unsatisfactory, mainly due to the following six problems:

[0003] 1) The engineering environment is complex and changeable. Fog, rain reflection, signs on the roadside or median strip, and engineering facilities under road construction can all cause false alarms. Moreover, since the factors that cause false alarms cannot be eliminated in time, repeated false alarms are very likely to occur. In practice, the same false alarm often occurs dozens of times in a short period of time, which increases the false alarm rate and greatly increases the workload of monitoring personnel.

[0004] 2) The daily operation and management tasks of road operation companies (such as road warning vehicles, road construction areas, etc.) are easily mistaken for abnormal events and alarms, which increases the false alarm rate and the workload of monitoring personnel.

[0005] 3) Duplicate alarms may occur between different detection devices from the same data source, between different data sources, and between intelligent event detection and other manual reporting channels such as traffic police or telephone. If a simple one-size-fits-all fusion method is adopted, it may cause omissions or duplicate reports of traffic events. It is necessary to comprehensively consider various factors such as traffic event type, data source, and event location to set appropriate spatiotemporal fusion rules.

[0006] 4) Currently, none of the automated detection methods for abnormal events can achieve 100% accuracy. Therefore, manual review is required before the event can be officially released. However, other methods besides video cannot provide image information for monitoring personnel to review, which prevents a closed loop from being formed for actual management operations.

[0007] 5) In practice, the number of reported abnormal events is enormous, but the number of events that affect road operations and require monitoring personnel intervention is very small. In engineering applications, it has been found that more than 90% of reported events are abnormal stoppage events. The vast majority of these stoppage events are temporary stops on the hard shoulder or in the temporary parking lane of the harbor, usually for drivers to stop temporarily to drink water, make phone calls, etc., and they will drive away on their own within a few minutes without any action. However, these stops cannot be completely blocked, because some vehicles do indeed pull over due to malfunctions or tire blowouts, requiring timely assistance.

[0008] 6) In congestion situations, a large number of abnormal stoppage events will be reported in a short period of time, which seriously affects the normal work of monitoring personnel. The congestion areas identified by the event intelligent detection method are often scattered and blocky due to the limitations of equipment deployment location. In addition, there are often errors in lane-level congestion detection. Simply using the detected congestion areas to shield them can easily miss congestion areas, resulting in poor shielding effect or shielding areas that are too large, causing missed reports. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings in the above-mentioned background technology and provide a video-based intelligent event detection system and method for multiple data sources on highways, so as to reduce the number of traffic incident reports and reduce the workload of monitoring personnel.

[0010] The technical solution of this invention is:

[0011] A video-based intelligent event detection system for highways using multiple data sources, characterized in that the system comprises:

[0012] The acquisition module is used to acquire traffic incident videos and extract traffic incident data from them;

[0013] The fusion module is used to fuse and compare traffic incident data;

[0014] The feedback module is used to review and provide feedback on traffic incident data that cannot be integrated.

[0015] The acquisition module includes a video data source, a PTZ data source, other data sources, and a video processing unit; the video data source includes a bullet camera; the PTZ data source includes a dome camera or a PTZ camera; other data sources include millimeter-wave radar, floating vehicles, manual inspection, integrated radar-visual equipment, QR code alarms, and telephone alarms.

[0016] A video-based intelligent event detection method for highways using multiple data sources, for use in the aforementioned system, includes the following steps:

[0017] 1) The acquisition module collects traffic incident videos, processes them to obtain traffic incident data, and reports it to the fusion module;

[0018] 2) The fusion module compares and merges traffic incident data with historical incident data through the fusion mechanism. If fusion is successful, the detection ends; otherwise, the traffic incident data is saved as historical incident data and reported to the feedback module, proceeding to step 3).

[0019] 3) The feedback module generates alarms for traffic incident data, which are then reviewed by the monitor to obtain feedback incident data:

[0020] When the review result is "confirmed", the feedback event data will be released externally;

[0021] When the review result is "No processing required", the feedback event data will not be processed.

[0022] When the review is deemed a "false alarm" or "suspended", the feedback event data will be sent back to the fusion module.

[0023] Step 1) includes: a video data source collects traffic event videos and the video processing unit processes them to obtain traffic event data; or, other data sources collect traffic events and send them to the video processing unit, the video processing unit controls the PTZ data source at the corresponding location to collect traffic event videos, and the video processing unit then processes the traffic event videos to obtain traffic event data.

[0024] The fusion module is equipped with an automatic update mechanism, a comparison pool, and a rule pool;

[0025] The automatic update mechanism includes setting different retention times for historical event data based on different event types, and deleting historical event data that has reached the retention time to ensure the stability of the comparison pool size;

[0026] The comparison pool is used to store historical event data; the rule pool is used to store spatiotemporal fusion rules.

[0027] The fusion mechanism in step 2) includes:

[0028] ① Based on the event type of the traffic incident data, find the matching event type in the spatiotemporal fusion rules, and denote it as set A;

[0029] ②Based on set A, find matching historical event data in the comparison pool, denoted as set B;

[0030] ③ Compare the traffic incident data with the historical incident data in set B one by one to determine whether they meet the threshold of the corresponding spatiotemporal fusion rule. If historical incident data that meets the threshold of the spatiotemporal fusion rule is found, it means that they can be fused; otherwise, they cannot be fused.

[0031] In the feedback module, the monitor enters feedback event data, which is then fed back to the comparison pool and saved as historical event data. The event types of the entered feedback event data include construction rescue, congestion, and accidents.

[0032] In step 3),

[0033] When the report is deemed a "false alarm", the event type of the feedback event data will be changed to "false alarm" and the data will be saved in the comparison pool as historical event data.

[0034] When the review is "suspended", the event type of the feedback event data will be changed to "suspended" and the data will be fed back to the comparison pool and saved as historical event data.

[0035] The traffic incident data includes the following event types: stopped, departing, construction, pedestrian, speeding, congestion, littering, illegal vehicles, slow speed, driving against traffic, and lane changing;

[0036] The event types of the feedback event data include construction rescue, congestion, accident, false alarm, suspended, confirmed, and no action required;

[0037] The event types in the historical event data include: stopped, departing, construction, pedestrian, speeding, congestion, littering, illegal vehicles, low speed, driving against traffic, lane changing, construction rescue, congestion, accident, false alarm, and suspended.

[0038] The beneficial effects of this invention are:

[0039] This invention enables intelligent detection of highway incidents primarily based on video. It acquires and extracts data from traffic incident videos through the linkage of multiple data sources, and sets targeted spatiotemporal fusion rules for different traffic incident types and locations. The data is then fused using a comparison pool. The fused results are submitted to monitoring personnel for review and feedback. The monitoring personnel's judgment can be released to external meetings or fed back to the comparison pool. This invention utilizes traffic incident information acquired from multiple data sources, processes it, and extracts effective information. It minimizes the workload of monitoring personnel without reducing the false alarm rate, making incident fusion more accurate and efficient, reducing false alarm rates, and solving the problem of spatiotemporal scaling differences in traffic incidents from different data sources. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the event intelligent detection system of the present invention.

[0041] Figure 2 This is a schematic diagram of the fusion mechanism of the present invention.

[0042] Figure 3 This is a diagram showing the distribution of parking times.

[0043] Figure 4 This is a diagram showing the time distribution of trucks.

[0044] Figure 5 This is a diagram showing the time distribution of passenger buses.

[0045] Figure 6 This is a diagram showing the distribution of parking time for construction vehicles.

[0046] Figure 7 This is a diagram illustrating the event false negative rate.

[0047] Figure 8 This is a diagram illustrating the false alarm rate of events.

[0048] Figure 9This is a schematic diagram illustrating the use of the congestion propagation mechanism in this invention.

[0049] Figure 10 This is a schematic diagram of the present invention without using a congestion propagation mechanism.

[0050] Figure 11 This is the field table for traffic event data in this invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] I. Intelligent Event Detection System

[0053] like Figure 1 As shown, a video-based intelligent event detection system for highways with multiple data sources includes a data acquisition module, a data fusion module, and a feedback module.

[0054] The acquisition module is used to acquire traffic incident videos and extract traffic incident data from them. The acquisition module includes a video data source, a PTZ (pan-tilt-zoom) data source, other data sources, and a video processing unit. The video data source includes bullet cameras. The PTZ data source includes dome cameras or PTZ cameras. The other data sources include millimeter-wave radar, floating vehicles, manual patrols, integrated radar-visual systems, QR code alarms, and telephone alarms.

[0055] The fusion module is used to compare and fuse traffic event data. The fusion module includes a comparison pool and a rule pool. The comparison pool stores historical event data. The rule pool stores spatiotemporal fusion rules.

[0056] The feedback module is used to review and provide feedback on traffic incident data that cannot be integrated.

[0057] II. Intelligent Event Detection Methods

[0058] A video-based intelligent event detection method for highways using multiple data sources includes the following steps:

[0059] 1) The acquisition module collects traffic incident videos, processes them to obtain traffic incident data, and then reports it to the fusion module; specifically, this includes:

[0060] The video data source collects traffic incident videos, which are then processed by the video processing unit to obtain traffic incident data, and then reported to the fusion module; or a PTZ linkage mechanism is used: other data sources collect traffic incidents and send them to the video processing unit, which controls the PTZ data source at the corresponding location to collect traffic incident videos, and then processes the traffic incident videos to obtain traffic incident data, which is then reported to the fusion module.

[0061] The video processing unit extracts traffic event data from the video using event detection algorithms (existing technology). Traffic events that the video processing unit can detect include stopped vehicles, vehicles leaving the road, construction, pedestrians, speeding, congestion, littering, illegal vehicles, slow speeds, driving against traffic, and lane changes.

[0062] The location information of both the PTZ camera data source and other data sources is stored in the video processing unit. Several preset positions are set within the control range of each dome camera or PTZ camera to ensure full coverage of the control range. The coverage area corresponding to each preset position is recorded in the video processing unit. When other data sources report traffic events, the preset position that covers the event can be found, and the dome camera or PTZ camera can be remotely controlled to switch to the corresponding preset position for event detection.

[0063] The PTZ linkage mechanism can incorporate other data sources with lower accuracy, such as floating car or situational anomaly event identification, into the system. Events can only be reported after verification by the PTZ camera or dome camera, ensuring the accuracy of the events. Due to their inherent characteristics, other data sources often produce errors in the station number and event of the same event. This is especially true for anomaly event detection based on floating cars. Due to the low accuracy of GPS and the difficulty in synchronizing the clocks of all on-board terminals of a large number of floating cars, there will be deviations in the spatiotemporal attributes of the event. Through the PTZ linkage mechanism, the event is ultimately reported by the PTZ camera or dome camera, which can ensure the consistency of station number and time in the system. This is beneficial for monitoring personnel management and is also the basis for the system's construction and rescue shielding mechanism and the monitoring personnel feedback mechanism. PTZ linkage provides the monitoring personnel with verifiable image information, forming a business closed loop.

[0064] The fields of the traffic incident data are as follows: Figure 11 As shown. The event types in the traffic event data include stopped vehicles, vehicles leaving, construction, pedestrians, speeding, congestion, littering, illegal vehicles, slow speeds, driving against traffic, and lane changes.

[0065] 2) The fusion module uses a fusion mechanism to compare traffic event data with historical event data in the comparison pool. If they can be fused (the difference between the event time and event location of both is less than the threshold), the detection ends. Otherwise, the traffic event data is saved as historical event data in the comparison pool and reported to the feedback module, proceeding to step 3).

[0066] The integration mechanism includes:

[0067] ① Based on the event type of the traffic incident data, find the matching event type in the spatiotemporal fusion rules, and denote it as set A;

[0068] ②Based on set A, find matching historical event data in the comparison pool, denoted as set B;

[0069] ③ Compare the traffic incident data with the historical incident data in set B one by one to determine whether they meet the threshold of the corresponding spatiotemporal fusion rule. If historical incident data that meets the threshold of the spatiotemporal fusion rule is found, it means that they can be fused; otherwise, they cannot be fused.

[0070] The fields of the spatiotemporal fusion rule include: event type of traffic event data, event type of historical event data, time threshold, and spatial threshold. The spatiotemporal fusion rule needs to be manually configured in advance.

[0071] The fusion module is equipped with an automatic update mechanism: different retention times are first set manually for historical event data of different event types, and the corresponding historical event data is deleted when the retention time is reached to ensure the stability of the comparison pool size.

[0072] The historical event data includes the following event types: stopped, departing, construction, pedestrian, speeding, congestion, littering, illegal vehicles, low speed, driving against traffic, lane changing, construction and rescue, congestion, accident, false alarm, and pending. The first 11 categories are obtained from saved traffic event data, while the last 5 are obtained from feedback.

[0073] 3) The feedback module alarms the traffic incident data and the monitor reviews and provides feedback (monitor feedback mechanism) to obtain feedback event data; the event types of the feedback event data include construction rescue, congestion, accident, false alarm, suspended, confirmed, and no action required.

[0074] Traffic incident data is displayed in the feedback module, including basic information such as incident type, alarm time, alarm description, vehicle type, location, direction, alarm source, and incident images. Monitors review each data entry, viewing screenshots of the traffic incident and video playback of a short period before and after the incident. They can also control nearby dome or pan-tilt cameras to view the incident location.

[0075] The monitoring staff verifies the reported events by using screenshots, video playback, and on-site video footage.

[0076] When the monitor approves the request as "confirmed", the feedback event data will be released to the public.

[0077] When the monitor reviews the data as "no action required," it means that there is no impact on road operation and management, and the feedback event data will not be processed.

[0078] When the monitor verifies that a report is a "false alarm", the event type of the feedback event data is changed to "false alarm" and fed back to the comparison pool to be saved as historical event data. It is also fed back to the acquisition module, which retains the image data for subsequent training and optimization of the event detection algorithm of the video processing unit. In addition, the acquisition module will also filter the same type of traffic event data of the same device in the same location within a short period of time to avoid repeated false alarms.

[0079] When the monitor approves the event as "suspended", the event type of the feedback event data will be changed to "suspended" and fed back to the comparison pool to be saved as historical event data. The monitor will continue to monitor events at nearby locations, and the video processing unit will automatically block all events at nearby locations.

[0080] Meanwhile, the monitor can enter feedback event data, with the event type being construction rescue, congestion, or accident. The feedback module will send these three types of feedback event data to the comparison pool and save them as historical event data.

[0081] Other mechanisms

[0082] The feedback module also includes a page fusion mechanism: traffic event data with similar time and location are put into an event list. The monitor only needs to verify the list once to complete the batch review and feedback, instead of operating on each reported traffic event data separately, which reduces the workload of the monitor.

[0083] The data acquisition module also includes a departure matching mechanism and a construction and rescue shielding mechanism.

[0084] The departure matching mechanism includes: when the video processing unit detects a "stopped" vehicle in the traffic incident video, it calculates the stopping time. If the vehicle remains stationary after the set time, the traffic incident data is then reported. Since most traffic incidents on highways involve vehicles stopping, and the vast majority of these are temporary stops on hard shoulders or harbor parking areas that do not require intervention, often due to drivers stopping briefly for reasons such as drinking water or answering phone calls, and then leaving on their own after a few minutes, filtering out low-risk temporary stops significantly reduces the frequency of incident alarms, thereby reducing the workload of monitoring personnel.

[0085] The construction and rescue shielding mechanism includes manual import or automatic detection: 1) Manual import: The monitor imports the construction plan events into the comparison pool in advance, or the monitor records the construction and rescue events into the comparison pool in real time; 2) Automatic detection: When the video processing unit detects two cones and a construction vehicle or two cones and a construction worker simultaneously in the traffic event video, it indicates that the construction and rescue event has started and reports the start information. The video processing unit searches for cones with similar positions in the nearby area to calculate the construction and rescue range, and shields all traffic event data within the construction and rescue range. When the last cone within the construction and rescue range disappears, it indicates that the construction and rescue event has ended and reports the end information.

[0086] The data acquisition module and feedback module also include a congestion propagation mechanism:

[0087] 1) Congestion fusion processing of feedback events

[0088] When the location (station number) of traffic event data is within the range of historical event data in the "congestion" category, the data is merged under the following conditions (same road segment, same direction, same lane, closest time, and event station number within 300 meters before and after the congestion). If the event type of the merged traffic event data is "congestion", the location (station number) range of the historical event data needs to be extended. That is, the minimum value of the starting station number of the traffic event data and the starting station number of the historical event data is taken as the starting station number of the new congestion boundary, or the maximum value of the ending station number of the traffic event data and the ending station number of the historical event data is taken as the ending station number of the new congestion boundary.

[0089] The historical event data includes two categories: 1) "Congestion" feedback event data entered by the monitor and saved in the comparison pool; 2) "Congestion" traffic event data is reviewed by the monitor as "false alarm" or "suspended" and then the feedback event data is saved in the comparison pool.

[0090] When traffic incident data is fused and compared, two fields are added: start and end station numbers. Generally, the start and end station numbers are the same for both incidents. Figure 11 The equipment station number), construction and congestion are special. Construction will expand the station number range based on the nearby cones detected, while congestion will expand 300m upstream and downstream based on the detected event station number.

[0091] 2) Congestion fusion processing of events at the data acquisition end

[0092] The retention time of "congestion" traffic event data input by the acquisition module in the comparison pool is less than or equal to 30 minutes;

[0093] Traffic event data that is not “congestion”, “parking” or “departure” is merged and compared with historical event data of “congestion”. If the traffic event data and historical event data are on the same road segment, in the same direction, in adjacent lanes (including the same lane), are close in time, and the traffic event mileage is within 600 meters before and after the historical event mileage, they are considered to be the same event.

[0094] First, extend the range of historical event data by station number (similar to the extension process for congestion fusion processing of feedback events), and refresh the retention time of historical event data in the comparison pool. Then, determine whether the interval between traffic event data and historical event data exceeds 30 seconds. If it exceeds 30 seconds, modify the position (station number) of the traffic event data to the station number of the historical event data (for the next step of page fusion) and report it to the feedback module for warning. If it does not exceed 30 seconds, do not issue a warning (is fusion necessary?).

[0095] Due to the limited sensing area, existing intelligent event detection methods typically lack a global perspective on congestion areas, focusing on point identification and thus being incomplete. Manually reported congestion areas are often imprecise, merely rough estimates from monitors. The congestion propagation mechanism integrates the congestion areas detected by the video processing unit with those reported by monitors at the feedback end, combining propagation rules to determine the congestion area, thereby filtering out numerous abnormal stoppage alarms within the congested area.

[0096] This invention includes data acquisition events and feedback events.

[0097] The events collected refer to the events (traffic event data) reported by the collection module, which are divided into 11 categories: stopped, leaving, construction, pedestrian, speeding, congestion, littering, illegal vehicles, low speed, driving against traffic, and lane changing.

[0098] Feedback events refer to events (feedback event data) returned by the feedback module, categorized into seven types: construction rescue, congestion, accident, confirmed, no action required, false alarm, and suspended. Construction rescue, congestion, and accident events are generated by the monitor's input, while confirmed, no action required, false alarm, and suspended events are automatically generated based on the monitor's handling of the reported events. Confirmed and no action required events have no impact on the fusion process; only false alarms and suspended events participate in the fusion.

[0099] Traffic incident data and feedback incident data stored in a comparison pool are called historical incident data.

[0100] The comparison pool is used to compare incoming events (traffic event data) with historical events (historical event data) to determine whether they can be reported. The comparison pool includes data acquisition events that were not ultimately merged and reported to the event feedback module, as well as all feedback events in the feedback module except for confirmed and no-processing events. The event types in the comparison pool include 11 categories (traffic event data) plus 5 categories (construction and rescue, congestion, accidents, confirmed, and no-processing feedback events), for a total of 16 categories.

[0101] The comparison pool has a cleanup mechanism with different retention times for different event types, ranging from tens of minutes to several days (depending on the needs). Events that have exceeded the retention time will be automatically removed from the comparison pool to ensure that the comparison pool does not expand indefinitely.

[0102] In the fusion module, since the events collected need to be compared with the comparison pool (11 * 16 = 176 categories), 176 sets of spatiotemporal fusion rules need to be set in the rule pool. The spatiotemporal fusion rules include the two event types to be compared, spatial thresholds, and temporal thresholds. Only data whose event types match, and whose event location is less than the spatial threshold and event time is less than the temporal threshold, can be fused.

[0103] III. Test

[0104] This invention was tested for 6 days on a 5km section of a highway in Zhejiang Province, and the results are as follows:

[0105] 1. Of the 1,263 traffic incidents that occurred, this invention only reported 987, achieving a fusion rate of 21.85% without compromising detection accuracy.

[0106] 2. Ten construction and rescue operations occurred, resulting in 1263 detected traffic incidents. The construction and rescue incident filtering mechanism filtered out 182 incidents, reducing the number of incidents by 14.41%. Furthermore, due to the complex conditions in construction and rescue areas, false alarms were prone to occur. Of the 22 false alarms detected during the test period, 4 occurred in construction and rescue areas. The construction and rescue incident filtering mechanism reduced the false alarm rate by 18.18%.

[0107] 3. Of the 987 reported incidents, 544 involved vehicle stoppages and 422 involved vehicles leaving the lane, accounting for 97.87% of the total incidents. The lane distribution of the stoppage incidents is as follows:

[0108] Table 1 Lane Distribution of Stoppage Events

[0109] Lane outside the lane One lane Two lanes Three lanes Hard shoulder or harbor total Number of grounding incidents 13 12 8 19 492 544

[0110] Note: "Outside the lane" indicates that the event occurred outside the lane, typically outside the central median strip or the hard shoulder of the road. The detection range was slightly expanded during algorithm configuration to prevent missed detections due to camera shift.

[0111] Analysis of the table above shows that stopping on the hard shoulder accounted for 90.44% of the total number of stops. Of the 492 hard shoulder stopping incidents, 422 could be linked to leaving the road. Analysis of these 422 incidents reveals the following distribution of stopping time. Figure 3 As shown.

[0112] Of the 422 incidents of vehicle outages, 197 involved trucks, 175 involved buses, and 50 involved construction vehicles. The distribution of outage times for different vehicle types is as follows: Figure 4 , Figure 5 , Figure 6 As shown.

[0113] Overall, the distribution of stopping times for trucks and buses is not significantly different, with 79.70% and 80.57% of stops lasting less than 6 minutes, respectively. For construction vehicles, due to frequent stop-and-go traffic during inspections, 78% of stops were less than 4 minutes. Based on this, it is estimated that if stopping on the hard shoulder for less than 6 minutes were not reported, it could reduce both stopping and leaving incidents by 340 each, accounting for 68.90% of the total reported incidents.

[0114] 4. For example Figure 9 and Figure 10 As shown, assuming the congestion fusion threshold is 300 meters, events D1, D2, and D3 in the figure are collected sequentially with a time difference greater than 30 seconds (if the time difference is less than 30 seconds, the data will not be sent).

[0115] When using the propagation mechanism ( Figure 9 Event D3 is 340 meters away from event D1. However, because D1 and D2 are merged, the station boundary of D1 is expanded. That is, the distance between event D3 and event D1 is equal to the distance between event D3 and event D2, which is 290 meters. This is less than the threshold of 300 meters. Finally, the event stations D1, D2, and D3 (k22+500) are sent and merged on the page using the same station number.

[0116] The specific process is as follows: Events D1, D2, and D3 are collected sequentially, and the time difference is greater than 30 seconds. D1 sends the event station number k22+500 and stores it in the comparison pool as (k22+200, k22+800). The difference between the termination station number of D2 and D1 is 55, which is less than the 300 threshold. Therefore, D2 sends the same station number as D1 and merges it on the page. The comparison pool is also expanded to include D1 station number (k22+200, k22+855). However, D2 is no longer stored in the comparison pool. The difference between D3 and the termination station number of D1, k22+855, is 290, which is less than the 300 threshold. Therefore, D3 sends the same station number as D1 and expands the comparison pool to include D1 (k22+200, k23+145).

[0117] When the propagation mechanism is not used ( Figure 10The merging of D1 and D2 does not expand the station boundary of D1. Therefore, the distance between event D3 and event D1 is still 340 meters, which exceeds the threshold of 300 meters. They are sent with their respective event station numbers. The final sent event station numbers are D1, D2 (k22+500), and D3 (k23+145).

[0118] The specific process is as follows: Events D1, D2, and D3 are collected sequentially, and the time difference is greater than 30 seconds. D1 sends the event station number k22+500 and stores it in the comparison pool as (k22+200, k22+800). The difference between the termination station number of D2 and D1 is 55, which is less than the 300 threshold. D1 and D2 are merged, and D2 is not sent. The comparison pool only contains D1. The difference between the termination station number of D3 and D1 is 340, which is greater than the 300 threshold. Then D3 sends the event station number k23+145 and stores it in the comparison pool as (k22+845, k22+445).

[0119] This invention was also tested for one year on a 48km section of a highway in Zhejiang Province, and the results are as follows:

[0120] An average of 8,645 traffic incidents were detected daily, with 3,403 reported, resulting in an incident fusion rate of 60.64%. Through the departure matching mechanism, only 553 incidents per day required actual operation by the monitor, accounting for 16.25% of the reported incidents.

[0121] And the false alarm rate and the missed alarm rate are as follows Figure 7 and Figure 8 As shown, the higher false alarm rates on July 4th and 5th, and July 11th and 12th were due to network instability affecting the video stream transmission.

[0122] The accompanying drawings illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

Claims

1. A video-based intelligent detection system for multi-data source events on highways, characterized in that, The system includes: The acquisition module is used to acquire traffic incident videos and extract traffic incident data from them; The fusion module is used to fuse and compare traffic incident data; The feedback module is used to review and provide feedback on traffic incident data that cannot be integrated. The acquisition module includes a video data source, a PTZ data source, other data sources, and a video processing unit; the video data source includes a bullet camera; the PTZ data source includes a dome camera or a PTZ camera; other data sources include millimeter-wave radar, floating vehicles, manual inspection, integrated radar-visual equipment, QR code alarms, and telephone alarms; The detection method of the system includes the following steps: Step 1) The acquisition module collects traffic incident videos, processes them to obtain traffic incident data, and reports it to the fusion module; Step 2) The fusion module compares and merges traffic incident data with historical incident data through a fusion mechanism. If fusion is successful, the detection ends; otherwise, the traffic incident data is saved as historical incident data and reported to the feedback module, proceeding to Step 3). Step 3) The feedback module generates an alarm for the traffic incident data, which is then reviewed by the monitor to obtain the feedback incident data: When the review is "confirmed", the feedback event data will be released externally; When the review result is "No processing required", the feedback event data will not be processed. When the review is deemed a "false alarm" or "suspended", the feedback event data will be sent to the fusion module.

2. The intelligent detection method of multiple data source events of a video-based highway according to claim 1, characterized in that: Step 1) includes: a video data source collects traffic event videos and the video processing unit processes them to obtain traffic event data; or, other data sources collect traffic events and send them to the video processing unit, the video processing unit controls the PTZ data source at the corresponding location to collect traffic event videos, and the video processing unit then processes the traffic event videos to obtain traffic event data.

3. The intelligent detection method of multiple data source events of a video-based highway according to claim 2, characterized in that: The fusion module is equipped with an automatic update mechanism, a comparison pool, and a rule pool; The automatic update mechanism includes setting different retention times for historical event data based on different event types, and deleting historical event data that has reached the retention time to ensure the stability of the comparison pool size; The comparison pool is used to store historical event data; the rule pool is used to store spatiotemporal fusion rules.

4. The intelligent event detection method for highways based primarily on video and multiple data sources according to claim 3, characterized in that: The fusion mechanism described in step 2) includes: ① Based on the event type of the traffic incident data, find the matching event type in the spatiotemporal fusion rules, and denote it as set A; ②Based on set A, find matching historical event data in the comparison pool, denoted as set B; ③ Compare the traffic incident data with the historical incident data in set B one by one to determine whether they meet the threshold of the corresponding spatiotemporal fusion rule. If historical incident data that meets the threshold of the spatiotemporal fusion rule is found, it means that they can be fused; otherwise, they cannot be fused.

5. The intelligent detection method of multiple data source events of a video-based highway according to claim 4, characterized in that: In the feedback module, the monitor enters feedback event data, which is then fed back to the comparison pool and saved as historical event data. The event types of the entered feedback event data include construction rescue, congestion, and accidents.

6. The intelligent detection method of multiple data source events of a video-based highway according to claim 5, characterized in that: In step 3): When the report is deemed a "false alarm", the event type of the feedback event data will be changed to "false alarm" and the data will be saved in the comparison pool as historical event data. When the review is "suspended", the event type of the feedback event data will be changed to "suspended" and the data will be sent to the comparison pool to be saved as historical event data.

7. The intelligent detection method of multiple data source events of a video-based highway according to claim 6, characterized in that: The traffic incident data includes the following event types: stopped, departing, construction, pedestrian, speeding, congestion, littering, illegal vehicles, slow speed, driving against traffic, and lane changing; The event types of the feedback event data include construction rescue, congestion, accident, false alarm, suspended, confirmed, and no action required; The event types in the historical event data include: stopped, departing, construction, pedestrian, speeding, congestion, littering, illegal vehicles, low speed, driving against traffic, lane changing, construction rescue, congestion, accident, false alarm, and suspended.

Images