Highway abnormal event detection and alarm method based on deep learning

Abstract

The invention relates to the field of traffic monitoring and management, and discloses an expressway abnormal event detection and alarm method based on deep learning, and the method comprises the following steps: collecting multi-modal data along an expressway; analyzing the video data through a deep learning model, and detecting a spatial position and a motion track of the vehicle; fusing radar sensing data, and constructing a space-time trajectory of the vehicle according to the time sequence; identifying an abnormal event based on deviation analysis of the vehicle trajectory; triggering an alarm signal according to a detection result, and performing graded alarm according to the type, the position and the severity of the abnormal event; and storing the related data of the abnormal event, and generating an event report. The video data and the radar sensing data are integrated, multi-mode information is fully utilized, the limitation of a single data source in complex scenes (such as rain and fog weather and light change) is solved, and the adaptability of the system in multiple scenes is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic monitoring and management, and in particular to a method for detecting and alarming abnormal events on highways based on deep learning. Background Art

[0002] Expressways are important infrastructure for modern transportation. Their high speed and closed management characteristics not only improve transportation efficiency, but also put forward higher requirements for traffic safety and management. However, the occurrence of abnormal events on expressways often has a serious impact on traffic safety and traffic efficiency. Abnormal events include but are not limited to driving in the wrong direction, parking, speeding, driving at a low speed, occupying emergency lanes, pedestrians, and littering. These abnormal events may lead to traffic congestion, secondary accidents, and serious casualties and property losses. Therefore, how to detect and alarm abnormal events in a timely manner has become one of the important research contents of expressway traffic management.

[0003] In the prior art, the detection and alarm methods for abnormal events on highways mainly include the following categories: detection methods based on fixed rules, detection methods based on traditional computer vision, and detection methods based on single-modal sensor data. However, these methods have many limitations in practical applications: Traditional detection systems usually rely on pre-set rules (such as speed thresholds, parking time limits, etc.) to determine abnormal events. However, this method cannot handle dynamic and complex traffic environments, such as the failure of rules caused by occlusion or congestion between vehicles in multi-lane scenarios. In addition, such methods have low recognition capabilities for abnormal events and are prone to false positives and false negatives.

[0004] Traditional computer vision technology detects abnormal events by extracting features and matching patterns from video images. However, changes in light (such as day-night alternation and shadow effects), weather interference (such as rainy and foggy weather), and changes in vehicle density in highway scenes often lead to inaccurate feature extraction, resulting in a significant decrease in detection performance. In addition, such methods have limited ability to capture dynamic trajectories, making it difficult to accurately analyze the motion state of vehicles. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a method for detecting and alarming abnormal events on highways based on deep learning, which solves the problem of how to use multimodal data fusion and deep learning technology in complex highway scenarios to achieve accurate detection of abnormal events, real-time alarm and efficient data management, so as to improve highway traffic safety and management efficiency.

[0006] To achieve the above objectives, the present invention is implemented by the following technical scheme: a method for detecting and alarming abnormal events on highways based on deep learning, comprising the following steps: Collect multimodal data along the highway; Analyze video data through deep learning models to detect the spatial position and movement trajectory of the vehicle; Fuse radar perception data and construct the spatiotemporal trajectory of the vehicle based on the time series; Identify abnormal events based on deviation analysis of vehicle trajectories; Trigger alarm signals based on detection results, and perform graded alarms based on the type, location, and severity of abnormal events; Store relevant data of abnormal events and generate event reports.

[0007] Preferably, the analyzing of the video data by a deep learning model to detect the spatial position and motion trajectory of the vehicle comprises the following steps: Perform frame preprocessing on the collected video data, use the deep learning model to detect vehicles on each frame of the preprocessed video data, and generate the bounding box of each vehicle and the coordinates of its center point in the image; The coordinates of the center point of each vehicle in multiple frames of video are correlated and analyzed according to the time series to form the continuous motion trajectory of the vehicle; The generated motion trajectory is supplemented with speed information combined with the vehicle radar perception data, and the motion trajectory is expanded into a spatiotemporal trajectory including position, time and speed; The vehicle's spatiotemporal trajectory is compared with the centerline trajectory of the road lane to analyze the vehicle's movement direction and trajectory deviation degree, providing input for abnormal event detection.

[0008] Preferably, fusing radar perception data and constructing the spatiotemporal trajectory of the vehicle according to the time series comprises the following steps: Extract the speed information, distance information and location coordinates of each vehicle from the radar perception data; Match the center point coordinates of each vehicle in the video data with the spatial position coordinates of the radar perception data to establish the association between the vehicle in the video data and the radar perception data; According to the time series, the coordinates of the center points of the vehicles in the video data are spliced ​​to form a preliminary spatial trajectory; Based on the preliminary spatial trajectory, combined with the vehicle speed information in the radar perception data, the speed is embedded into the trajectory data as a time series feature to construct a complete spatiotemporal trajectory; The time intervals in the radar perception data are smoothed by an interpolation algorithm to ensure the continuity of the space-time trajectory in the time dimension, and finally a three-dimensional space-time trajectory including position, speed and time is generated.

[0009] Preferably, the deviation analysis based on the vehicle trajectory and identifying abnormal events comprises the following steps: Compare the space-time trajectory of the vehicle with the centerline trajectory of the road lane, and calculate the offset distance of the vehicle trajectory from the centerline of the lane; Analyze the vehicle's moving direction and determine whether there is any abnormality in the moving direction by comparing the angle between the vehicle's moving direction and the normal driving direction of the lane; Combine the vehicle's spatiotemporal trajectory and radar perception data to calculate the deviation between the vehicle's actual speed and the road speed limit; By constructing a trajectory deviation energy function, the vehicle speed deviation and trajectory deviation degree are jointly analyzed to identify whether the vehicle is driving in the wrong direction, speeding, driving at a low speed, or has abnormal trajectory behavior; According to the deviation analysis results, vehicles whose trajectory energy function values ​​exceed the threshold are marked and classified as wrong-way vehicles, abnormally parked vehicles, or other types of abnormal events.

[0010] Preferably, triggering an alarm signal according to the detection result and performing graded alarm according to the type, location and severity of the abnormal event includes the following steps: Classify the events according to the type of abnormal events detected, determine the location of the abnormal events, and calibrate the specific lane and distance location of the abnormal events by analyzing the vehicle's spatiotemporal trajectory and matching it with the road geographic information; Calculate the severity of the incident by combining the type of abnormal incident, the location of the incident, and the parameters of vehicle speed and trajectory deviation amplitude; Set alarm priorities according to severity, trigger emergency alarm signals for high-priority events, and trigger regular alarm signals for medium- and low-priority events; When an alarm signal is triggered, an alarm is sounded in a variety of ways and the alarm signal is suppressed to prevent the same abnormal event from repeatedly triggering an alarm in a short period of time.

[0011] Preferably, storing relevant data of the abnormal event and generating an event report comprises the following steps: After an abnormal event is detected, video data from 10 seconds before and 20 seconds after the event is extracted and stored together with the trajectory data of the abnormal event, vehicle speed information, and classification results of the abnormal event; The stored abnormal event data is classified and marked through the data annotation module, and the stored abnormal event data is generated into a structured record form; Generate an event report based on the abnormal event data. The event report includes the type, location, time, severity, and related tracks and video screenshots of the abnormal event. It provides multi-dimensional query and export functions by time, event type and camera number, supports exporting generated event reports in the form of electronic documents, regularly cleans and archives stored data, and performs long-term storage management of high-priority event data.

[0012] Preferably, the highway abnormal event detection and alarm system based on deep learning includes: Data acquisition module, used to collect video data and vehicle radar perception data along the highway; A data processing module is used to pre-process the collected data and synchronize the multi-modal data in time; Deep learning analysis module, used for vehicle detection and trajectory extraction on preprocessed data; Abnormal event detection module, used to identify abnormal events based on deviation analysis of vehicle trajectories; An alarm module, which is used to trigger alarms based on the type, location and severity of abnormal events; The data storage and reporting module is used to store relevant data of abnormal events and generate event reports.

[0013] Preferably, the abnormal event detection module performs trajectory fusion of multi-camera data by a distributed optimization method, and the distributed optimization method calculates the consistency of vehicle trajectories collected by multiple cameras by an alternating direction multiplier method.

[0014] Preferably, the alarm module supports multiple alarm modes, including: real-time pop-up alarm; sound and light alarm; mobile client alarm.

[0015] Preferably, the deep learning model is a deep learning model based on a spatiotemporal convolutional neural network, which is used to extract time series features and spatial features in video data and output the vehicle's trajectory position, movement direction and speed.

[0016] The present invention provides a method for detecting and alarming abnormal events on highways based on deep learning. It has the following beneficial effects: 1. The present invention integrates video data and radar perception data, making full use of multimodal information, and solves the limitations of a single data source in complex scenarios (such as rainy and foggy weather, light changes). Video data provides the spatial location characteristics of the vehicle, and radar data supplements dynamic information such as speed, distance and direction. Through time synchronization and data fusion, high-quality spatiotemporal trajectories are generated, providing a reliable foundation for subsequent deep learning analysis and abnormal event detection, and significantly improving the adaptability of the system in multiple scenarios.

[0017] 2. The present invention introduces trajectory deviation analysis, comprehensively considers multi-dimensional factors such as vehicle position offset, speed anomaly and direction deviation, defines trajectory energy function, and realizes quantitative judgment of abnormal events. Compared with traditional rule-based or single feature-based methods, the present invention can more accurately identify abnormal events in complex traffic environments, such as reverse driving, abnormal parking, speeding, low-speed driving, etc., and greatly reduces the false alarm rate and missed alarm rate.

[0018] 3. The present invention designs a deep learning model based on spatiotemporal convolutional neural network (ST-CNN) and combines it with the attention mechanism to extract spatial and temporal features from video frames and accurately identify vehicle trajectories and motion states. At the same time, the attention mechanism enables the model to focus on key areas and ignore interference information in the background. It is particularly suitable for detection in densely packed and complex scenes, significantly improving the detection accuracy and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Example: Please refer to the attached Figure 1 , an embodiment of the present invention provides a method for detecting and alarming abnormal events on a highway based on deep learning, comprising the following steps: S1, collect multimodal data along the highway; In the present invention, data collection is the basic link to realize the detection and alarm of abnormal events on highways. Through the real-time collection and fusion of multimodal data, the present invention can realize the comprehensive perception of abnormal events on highways and provide accurate input data for subsequent deep learning analysis, trajectory construction and abnormal event identification. It should be noted that the present invention is particularly designed for the complexity of the highway environment (such as multiple lanes, bad weather, day and night alternation, etc.), and a system architecture compatible with multimodal data collection is designed to ensure data quality and time synchronization.

[0022] Specifically, the data collection of the present invention includes video data and radar perception data. The video data is mainly used to capture the spatial characteristics of the vehicle, while the radar data provides the speed, position and direction information of the vehicle. As a possible implementation method, the present invention uses a variety of data processing methods to synchronize and fuse the collected raw data in order to adapt to the subsequent deep learning model.

[0023] In this embodiment, data collection mainly includes the following technical contents: As an option, multiple high-definition cameras and radar devices are deployed along specific sections of the highway (such as toll booths, ramps, tunnel entrances and exits, etc.) to form a multimodal data collection network. The layout of cameras and radar devices needs to be designed according to the specific needs of the number, width and monitoring range of lanes to ensure effective coverage of the spatial and speed information of all vehicles.

[0024] Specifically, the camera uses an industrial-grade camera with high resolution (1920×1080) and high frame rate (30 frames per second). The video data it collects can clearly capture the movement status, position and shape characteristics of the vehicle in each lane; the radar equipment uses a millimeter-wave radar that can measure the vehicle's speed, distance and direction of movement in real time. The sampling frequency is 10Hz, and the detection range covers the width and length of each lane.

[0025] In one possible implementation, both the camera and the radar device are equipped with a timestamp synchronization module to ensure the consistency of the video data and the radar data in the time dimension. The application of timestamp synchronization technology avoids the data misalignment problem caused by the different sampling frequencies of the acquisition devices, thereby providing accurate timing input for subsequent deep learning analysis.

[0026] It should be noted that video data collection and radar data collection are carried out simultaneously. As an exemplary solution, the camera captures the two-dimensional spatial position of the vehicle in real time, while the radar device provides the speed and distance information of the corresponding vehicle. In this case, the spatial information and motion characteristics of the vehicle can be described in multiple dimensions through the joint data of video and radar. For example, in a three-lane scene on a certain road section, the data collected by the camera can identify the position coordinates (x, y) of each vehicle, and the data perceived by the radar can further provide the vehicle's speed v and the distance d from the collection device.

[0027] In order to ensure the quality of the collected data, the present invention performs preprocessing operations on the original video data. Specifically, the preprocessing includes denoising, image enhancement and frame correction. For example, in some embodiments, in order to cope with the possible illumination changes (such as strong light, shadows and night lights) in the highway environment, median filtering and histogram equalization techniques are used to optimize the video data. The radar data is Kalman filtered to remove sudden measurement noise, and the radar signal is smoothed by a fitting technique based on the least squares method to enhance the accuracy of the data.

[0028] It is understood that in some embodiments, in order to achieve data management of multiple cameras and multiple radar devices, the present invention designs a distributed data acquisition network. Through the collaborative work of data acquisition nodes, the raw data of all acquisition devices are transmitted to the central data processing server at the back end in real time. A high-speed optical fiber network is used during the transmission process, and a data compression algorithm is combined to reduce transmission delays.

[0029] As an extended technical content, the present invention introduces a data redundancy mechanism to prevent data loss due to failure of a camera or radar device. For example, in some embodiments, by setting overlapping camera and radar monitoring areas, compensation acquisition of adjacent areas is achieved. Even if the acquisition device in a certain area fails, the adjacent device can cover the area to ensure the integrity of the collected data.

[0030] It should be emphasized that the present invention has designed a compatibility strategy specifically for adverse weather conditions (such as fog, heavy rain, etc.). Specifically, in some embodiments, the camera and radar equipment are equipped with a protective housing and an automatic cleaning device to reduce the impact of weather on the acquisition quality. At the same time, in order to solve the problem that video data may not be clear in adverse weather, the present invention uses radar data as an auxiliary input, and relies on the speed and distance information of the radar to extract key features when the visual features of the vehicle cannot be identified.

[0031] Through the above-mentioned technical means of data collection, the present invention can provide high-quality, synchronized multimodal data, providing a solid foundation for subsequent deep learning analysis, trajectory construction and abnormal event detection. It should be noted that although the specific technical content described above is the preferred implementation method of the present invention, those skilled in the art can make appropriate adjustments to the deployment location, collection range and data processing strategy of the data acquisition equipment according to actual needs without affecting the core idea of ​​the technical solution of the present invention.

[0032] S2. Analyze video data through deep learning models to detect the spatial position and movement trajectory of the vehicle; In the present invention, analyzing video data through a deep learning model is a key step in realizing the spatial position recognition and motion trajectory extraction of vehicles. The present invention designs a spatiotemporal feature extraction method based on deep learning, combining the vehicle features and time series information in the video data to accurately detect the spatial position and trajectory changes of the vehicle. It should be noted that the present invention not only locates the vehicle position in the static frame, but also can use the time series modeling method to obtain the motion trajectory of the vehicle in continuous frames, thereby providing high-quality input data for subsequent trajectory deviation analysis and abnormal event recognition.

[0033] Specifically, the present invention uses a deep learning model with a spatiotemporal convolutional neural network as the core, and realizes accurate extraction of vehicle positions and trajectories by gradually processing and analyzing video frames. The present invention optimizes the model for complex scenes that may occur in highway environments (such as occlusion, light changes, and small vehicle spacing) to improve the accuracy and robustness of detection.

[0034] In this embodiment, the specific steps of analyzing video data through a deep learning model are as follows: As an option, the present invention first preprocesses the collected video data to ensure the quality and consistency of the input data. The preprocessing of video data includes noise removal, frame correction and scale adjustment. Specifically, for video image blur caused by environmental noise (such as dust or raindrops from high-speed driving), the present invention uses median filtering technology to remove noise; for frame offsets caused by changes in camera installation angles or vehicle speeds, an image correction algorithm is used to align video frames; in addition, in order to adapt to the input requirements of the deep learning model, the resolution of the video frame is uniformly adjusted to 224×224 pixels.

[0035] In one possible implementation, the preprocessed video data is input into a deep learning model for feature extraction. The present invention adopts the ST-CNN model, which can simultaneously capture the spatial features and temporal features of video frames. Specifically, ST-CNN consists of two main parts: a spatial convolution module and a temporal convolution module.

[0036] It should be noted that the spatial convolution module extracts the spatial features of the vehicle in the video frame, such as the location, shape and color features of the vehicle, through a multi-layer convolutional neural network. The process of the convolution operation can be described as: Among them, I i,j Represents the grayscale value of the pixel in the input video frame, K m,n is the convolution kernel, f i,j is the eigenvalue after the convolution operation.

[0037] Specifically, in the present invention, each frame of video is divided into several specific areas, and the bounding box (BoundingBox) of each area is calculated by a convolutional neural network, and the center point coordinates (x, y) of the vehicle are generated. In this way, the position of the vehicle in each frame can be accurately calibrated.

[0038] The temporal convolution module uses a three-dimensional convolutional neural network (3D-CNN) to model the time series of consecutive frames. As an exemplary method, the temporal convolution module captures the dynamic change information of the vehicle in multiple frames by extracting features in the time dimension. The process of temporal convolution can be expressed as: Among them, T, H, and W represent the number of time frames, spatial height, and spatial width, respectively. t,x,y represents the input data in time frame, K i,j,k Represents the temporal convolution kernel.

[0039] Through the processing of the time convolution module, the present invention can continuously model the vehicle trajectory in the time dimension and extract the movement characteristics of the vehicle at different times, including speed, acceleration and direction change information.

[0040] It is understandable that in order to improve the detection accuracy of the model, the present invention introduces an attention mechanism into the ST-CNN model. Specifically, the attention mechanism can perform weighted processing on the key areas in the video frame, thereby focusing on the significant feature areas of the vehicle and ignoring the interference information in the background. The calculation process of the attention mechanism is: Among them, Q and K represent the query value and key value of the feature vector respectively, d k is the dimension of the vector, α i represents the attention weight.

[0041] In a specific implementation of the present invention, the output of the ST-CNN model includes the bounding box, center point coordinates, and classification information of each vehicle in each frame. For example, the system can identify that the center point coordinates of a vehicle in the 10th frame are (120, 200), the bounding box is (100, 180, 140, 220), and the classification information is "sedan".

[0042] It should be noted that the detected vehicle position information will be further input into the trajectory tracking algorithm to generate a continuous motion trajectory of the vehicle. In a preferred embodiment, the present invention uses a Kalman filter algorithm to smooth the vehicle trajectory, thereby eliminating the trajectory interruption problem caused by occlusion or camera resolution limitation. For example, if the center point coordinates of a vehicle in the 5th frame are (110, 190) and the center point coordinates of the 7th frame are (130, 210), the Kalman filter algorithm can reasonably estimate the center point position of the vehicle in the 6th frame.

[0043] As an extended technical content, the present invention also supports adaptation to scenes with severe occlusion or poor lighting conditions. In this case, by adjusting the convolution kernel size of the model and the weight distribution of the attention mechanism, the detection capability of the vehicle position can be enhanced, thereby improving the robustness of trajectory extraction.

[0044] Through the above steps, the present invention can efficiently extract the spatial position and motion trajectory of the vehicle from the video data, providing accurate and reliable input data for subsequent abnormal event detection. It should be emphasized that although the specific implementation described above is the preferred technical solution of the present invention, those skilled in the art can make appropriate adjustments to the model architecture, parameter configuration and preprocessing steps according to actual needs without affecting the core idea of ​​the technical solution of the present invention.

[0045] S3, integrates radar perception data and constructs the spatiotemporal trajectory of the vehicle based on the time series; The present invention achieves accurate modeling of the vehicle's motion state by fusing video data and radar perception data to construct the vehicle's spatiotemporal trajectory according to the time series. Video data is used to provide the vehicle's two-dimensional spatial position, while radar perception data provides the vehicle's speed, direction of movement, and distance information. The combination of the two generates a complete vehicle spatiotemporal trajectory. It should be noted that the present invention solves the data inconsistency problem caused by the different sampling frequencies of video data and radar data through timestamp synchronization technology and data fusion algorithm, and ensures the continuity and accuracy of the spatiotemporal trajectory through trajectory smoothing.

[0046] In this embodiment, the specific steps of fusing radar perception data and constructing the spatiotemporal trajectory of the vehicle are as follows: As an option, the present invention first performs time synchronization processing on the video data and radar data to ensure the consistency of the time dimension between different data sources. Specifically, the timestamp of the video data is generated by the internal clock of the camera, while the timestamp of the radar data is provided by the radar acquisition device. In order to achieve synchronization, the unified network time protocol (NTP) is used to calibrate the camera and radar equipment, and the time error of the acquired data is controlled within 1 millisecond.

[0047] In one possible implementation, the coordinates of the center point of the vehicle extracted from the video data are matched with the physical features in the radar data to achieve cross-modal data association. Specifically, for each vehicle, the two-dimensional coordinates (x, y) in the video data are mapped to the spatial position of the radar data, and the matching is completed by calculating the relative distance d and angle θ between the vehicle and the radar device. The matching relationship can be expressed as: Wherein, (x0, y0) is the position of the radar device, and d represents the distance between the vehicle and the radar device.

[0048] It should be noted that the present invention uses an optimal matching algorithm to optimize the many-to-many mapping between video data and radar data to solve the data matching conflict problem that may be caused when multiple vehicles appear at the same time. For example, in a three-lane scene, if the radar data detects the position and speed of three vehicles at the same time, the present invention compares the spatial consistency between the video data and the radar data to perform a one-to-one match for each vehicle.

[0049] Specifically, the time-space trajectory of the vehicle is generated by combining video data and radar data through time series analysis. The present invention defines the time-space trajectory of each vehicle as: R(t)=[x(t),y(t),v(t)] Among them, (x(t), y(t)) is the two-dimensional spatial position of the vehicle at time t, and v(t) is the speed of the vehicle at time t.

[0050] For example, in a three-lane highway scenario, the trajectory data of a vehicle may include the following: Timestamp: t = 1, 2, 3, ..., n Spatial position: (x, y) = (120, 200), (125, 210), (130, 220), ... Speed: v = 25m / s, 26m / s, 27m / s, ... It is understandable that since the actual motion trajectory of the vehicle may be blocked or the sampling frequency of the device is limited, resulting in partial data missing, the present invention uses a trajectory interpolation algorithm to complete the missing data. In a preferred embodiment, the B-spline interpolation method is used to smooth the trajectory to ensure the continuity of the trajectory data in the time dimension. The mathematical expression of the interpolation algorithm is: Among them, B i (t) represents the B-spline basis function, P i Represents a control point.

[0051] It should be noted that the present invention fully considers the dynamic characteristics of vehicle speed changes when smoothing the trajectory. Specifically, if a vehicle has a sudden speed change in the time series (such as sudden deceleration or acceleration), the present invention adjusts the smoothness of the interpolation algorithm through dynamic weights to avoid the trajectory being too smooth and losing motion details. For example, in a scene where a vehicle suddenly decelerates from a constant speed to a stop, the interpolation algorithm retains the time point of the speed change by reducing the smoothness parameter.

[0052] In some embodiments, in order to improve the applicability of radar data in adverse weather conditions (such as rainy and foggy weather), the present invention further integrates the acceleration information of the vehicle and enhances the robustness of the vehicle's motion trajectory through multimodal data. Specifically, the acceleration information is calculated using the following formula: Where v(t) and v(t+1) are the velocities at time t and t+1 respectively, and Δt is the time interval.

[0053] It is understood that the present invention is not only applicable to the trajectory generation of a single vehicle, but also can realize parallel trajectory construction in multi-lane and multi-vehicle scenarios. For example, in a four-lane scenario, the system can simultaneously generate the spatiotemporal trajectories of multiple vehicles on each lane and avoid trajectory intersection or duplication through optimization algorithms.

[0054] Through the above steps, the present invention realizes the efficient fusion of video data and radar data, constructs the complete spatiotemporal trajectory of the vehicle, and provides accurate and reliable input data for the subsequent abnormal event identification. It should be emphasized that although the specific implementation method described above is the preferred solution of the present invention, those skilled in the art can make appropriate adjustments to the data processing method, interpolation algorithm and trajectory modeling method according to actual application requirements without affecting the core idea of ​​the technical solution of the present invention.

[0055] S4, Identify abnormal events based on deviation analysis of vehicle trajectories; The present invention identifies abnormal events on highways by performing deviation analysis on the spatiotemporal trajectory of vehicles. The deviation of vehicle trajectory includes not only the spatial offset between the position and the centerline of the lane, but also the direction of movement, speed, and deviation from normal traffic flow. The present invention designs a set of deviation analysis methods based on trajectory energy functions, which quantify and combine the multidimensional factors of trajectory deviation (such as spatial position, speed and direction) to ensure that abnormal behaviors such as reverse driving, abnormal parking, speeding, low-speed driving, and occupying emergency lanes can be accurately identified. It should be noted that through this deviation analysis method, the present invention can adapt to complex traffic environments, such as multi-lane scenarios, vehicle-dense areas, and dynamic traffic flow changes.

[0056] In this embodiment, the specific steps of identifying abnormal events based on the deviation analysis of vehicle trajectories are as follows: As an option, the deviation analysis of vehicle trajectory uses normal traffic flow as a reference standard. Normal traffic flow can be established by statistical methods or historical data. In one possible implementation, a normal motion trajectory model is constructed based on the lane centerline, which includes the spatial centerline position of the lane, the normal driving speed range, and the driving direction. Lane centerline trajectory L c is defined as: L c ={(x i ,y i )|i=1,2,…,N} Among them, (x i ,y i ) represents the position of the point on the lane centerline, and N is the number of discrete points on the centerline.

[0057] Specifically, for each vehicle's spatiotemporal trajectory T(t) = [x(t), y(t), v(t)], first calculate the offset distance of the vehicle position relative to the lane centerline. The offset distance can be calculated by the Euclidean distance from the vehicle position to the lane centerline: Among them, d 偏移 Indicates the shortest distance from the vehicle's current position to the center line of the lane.

[0058] In a preferred embodiment, in order to further analyze whether the moving direction of the vehicle is consistent with the normal traffic flow direction, the present invention adopts an angle calculation method. Assume that the velocity vector of the vehicle is v(t) = [v x (t),v y (t)], the average direction vector of the lane is v c =[v cx ,v cy ], the angle θ between the two can be calculated by the following formula: If cosθ<0, the vehicle may be driving in the wrong direction.

[0059] It is understandable that a single offset distance or direction angle may not be sufficient to fully characterize the trajectory deviation, so the present invention combines multiple deviation factors through a trajectory energy function. The trajectory energy function is defined as: in, represents the velocity vector of the vehicle, V c (t) represents the average velocity vector of the normal lane, T(t) represents the actual position trajectory of the vehicle, and λ is the weight coefficient used to balance the speed deviation and position deviation.

[0060] It should be noted that by setting the threshold δ of the trajectory energy function, the present invention can flexibly adjust the recognition sensitivity of abnormal events. When E(T)>δ, it is determined that the vehicle has abnormal behavior. Specifically: If the direction deviation of the vehicle is significant (cosθ<0), and the speed and position deviations are large, it is identified as a wrong-way vehicle; If the vehicle speed is close to zero and the position deviation exceeds the normal range, it is identified as abnormal parking; If the vehicle speed far exceeds the normal speed range, it is identified as speeding.

[0061] In a possible implementation, in order to adapt to complex traffic scenarios with multiple lanes and dense vehicles, the present invention further introduces a dynamic traffic flow analysis module. By calculating the statistical distribution of vehicle density and speed over a period of time, the reference standard of trajectory deviation is dynamically adjusted. For example, in a congested scenario with a large vehicle density, the threshold limit on position deviation is relaxed to reduce the false alarm rate.

[0062] For example, in a three-lane highway scenario, the trajectory deviation analysis results of a certain vehicle show that its energy function value is E(T) = 12.5, which exceeds the set threshold δ = 10. Further analysis found that the direction deviation of the vehicle is cosθ = -0.3, and the offset distance d 偏移=2.5. Based on the above information, the system determines that the vehicle is driving against traffic and marks it.

[0063] As an extended technical content, the present invention also supports trajectory deviation analysis in complex environments (such as rainy and foggy weather or at night). Under these conditions, the vehicle speed information is supplemented by combining radar perception data to improve the robustness of the analysis. For example, in low visibility conditions, the speed data provided by the radar Can be used to replace the limited velocity vector in video analysis.

[0064] Through the above steps, the present invention can accurately perform deviation analysis on vehicle trajectories and efficiently identify abnormal events based on the analysis results. It should be emphasized that although the specific implementation method described above is a preferred solution of the present invention, those skilled in the art can make appropriate adjustments to the definition of the energy function, the selection of weight parameters, and the deviation analysis method according to actual application requirements without affecting the core idea of ​​the technical solution of the present invention.

[0065] S5. Trigger an alarm signal based on the detection results, and perform graded alarms based on the type, location, and severity of the abnormal event; The present invention designs a hierarchical alarm mechanism by analyzing the vehicle trajectory deviation detection results, combining the type, location and severity of the abnormal event, so as to promptly issue warnings to the traffic management system or relevant personnel. The alarm mechanism can not only provide real-time alarms for events, but also classify and prioritize abnormal events according to their degree of hazard. The alarm mechanism of the present invention adopts multi-mode output, supports sound and light warnings, pop-up notifications from the control center, and push notifications from mobile clients, and can adapt to different highway management scenarios. It should be noted that the present invention designs an alarm suppression function for the same abnormal event to avoid redundant information interference caused by repeated triggering of alarm signals.

[0066] In this embodiment, the specific steps of triggering an alarm signal and performing a graded alarm according to the detection results are as follows: As an option, the present invention first classifies the detected abnormal events. The classification of abnormal events is based on the trajectory deviation analysis results, combined with the vehicle's motion state (such as speed, direction, position) and the actual road conditions (such as lane type, traffic flow density). Specifically, abnormal events include vehicles traveling in the wrong direction, abnormal parking, speeding, slow driving, occupying emergency lanes, pedestrians, and vehicles throwing objects.

[0067] In one possible implementation, each abnormal event is assigned a corresponding priority. The priority is based on the severity of the event, which is quantified by the following formula: S 事件 =α1·d 偏移 +α2·v异常 +α3·θ 偏差 in: d 偏移 is the distance the vehicle deviates from the center line of the lane; v 异常 is the deviation of the vehicle speed from the normal speed range; θ 偏差 is the angle between the vehicle's travel direction and the normal direction; α1, α2, and α3 are weight coefficients, which are set according to the actual road conditions.

[0068] It should be noted that when the severity S 事件 When the preset threshold is exceeded, the system will trigger a high priority alarm signal. For example, for a wrong-way vehicle, due to the severity of its direction deviation and position offset, it is usually given the highest priority to ensure that relevant personnel can handle it in the first time.

[0069] Specifically, the present invention supports multiple modes of alarm output. As an exemplary implementation, when an alarm signal is triggered, the system sends alarm information to the following output modules: Sound and light warning module: triggers sound and light warnings on monitoring equipment near the location where the abnormal event occurs to alert surrounding vehicles and drivers.

[0070] Control center notification module: sends real-time pop-up notifications to the control center of the traffic management system, which include the type, location, time and vehicle information of the abnormal event.

[0071] Mobile client push module: pushes alarm information to the mobile terminal of traffic management personnel through the network, supporting remote monitoring and emergency response.

[0072] In a preferred embodiment, the alarm signal also includes a link to the video recording and trajectory data, so that the management personnel can quickly obtain the details of the event. For example, for an alarm of a wrong-way vehicle, the system not only provides its current location coordinates, but also attaches the vehicle's movement trajectory in the past 10 seconds and related video screenshots.

[0073] It should be noted that in order to avoid repeated alarms for the same abnormal event, the present invention has designed an alarm suppression function. Specifically, the system generates a unique event identifier (ID) when an abnormal event is first detected, and suppresses repeated alarm signals of the same ID within a short period of time. For example, when a vehicle is detected and an alarm is triggered for driving against the flow, the same alarm signal will not be triggered again within 30 seconds.

[0074] It is understandable that the alarm classification mechanism of the present invention can also dynamically adapt to different road environments and traffic flow conditions. In the case of high traffic flow density, the weight parameters α1, α2, and α3 in the severity formula can be adjusted to prioritize events that have a greater impact on traffic flow. For example, in congested sections, abnormal parking may be given a higher priority; while in unblocked sections, vehicles traveling in the wrong direction may be given a higher priority.

[0075] For example, in a three-lane highway scenario, a vehicle is detected driving in the opposite direction in the main lane, and the system calculates its severity S 事件 =15.8, exceeding the high-priority alarm threshold δ = 10. Therefore, the system immediately triggers the highest level of alarm signal, notifies nearby vehicles through the sound and light warning module, pushes real-time pop-up information to the traffic management platform through the control center notification module, and sends event details to the mobile terminal of the road section manager.

[0076] As an extended technical content, the alarm mechanism of the present invention also supports the storage and retrospective analysis of alarm data. When the alarm is triggered, the system automatically records the trajectory and video data 10 seconds before and 20 seconds after the event, and attaches the storage path of the data to the alarm signal for subsequent review and processing.

[0077] Through the above steps, the present invention realizes real-time alarm and graded processing of abnormal events, which not only improves the accuracy and timeliness of the alarm, but also reduces the possibility of information redundancy through multi-mode output and suppression function. It should be emphasized that although the specific implementation method described above is the preferred solution of the present invention, those skilled in the art can make appropriate adjustments to the severity calculation formula, the output mode of the alarm module, and the time window of the suppression function according to actual needs without affecting the core idea of ​​the technical solution of the present invention.

[0078] S6. Store relevant data of the abnormal event and generate an event report.

[0079] The present invention provides support for subsequent event backtracking, statistical analysis and management decision-making by storing relevant data of abnormal events and generating event reports. The stored data includes multi-dimensional information such as the category, time, location, trajectory information, video clips, etc. of the abnormal event, and is archived in a structured form. The generation of event reports can integrate a variety of data to present detailed information of abnormal events in an intuitive and clear manner. It should be noted that the present invention specifically considers the effectiveness of data storage and long-term management strategies during the design process, while supporting rapid retrieval and multi-dimensional query of event data.

[0080] In this embodiment, the specific steps of storing the relevant data of the abnormal event and generating the event report are as follows: As an option, the present invention will immediately store the spatiotemporal trajectory data and video data related to the event after detecting an abnormal event. Specifically, the trajectory data includes the time series position, speed information, deviation analysis results, and trajectory deviation energy function value of the abnormal vehicle. Exemplarily, the trajectory storage content of a vehicle can be expressed as: Timestamp: t = 0, 1, 2, ..., n Spatial position: (x, y) = (120, 300), (125, 305), (130, 310), ... Speed ​​information: v=30m / s, 29m / s,… Offset distance: d 偏移 =2.5m,2.7m,… Trajectory energy value: E(T)=10.5,12.0,… Specifically, the storage range of video data includes the images 10 seconds before and 20 seconds after the event. The video files will be stored in the central server in an efficient encoding format (such as H.265) to optimize storage space when the data volume is large.

[0081] In one possible implementation, event data is stored in a database in a structured format. The data entry for each abnormal event includes the following fields: Event Number: A unique identifier used to distinguish between different events.

[0082] Event category: such as wrong-way driving, parking, speeding, etc.

[0083] Occurrence time: The specific time point when the event was detected.

[0084] Geolocation: The precise geographic coordinates or lane number where the incident occurred.

[0085] Vehicle trajectory: A time series of position, velocity, and direction.

[0086] Severity: Severity value calculated based on trajectory deviation analysis.

[0087] Video storage path: The file location where video data is stored.

[0088] It should be noted that in order to ensure the integrity and accuracy of event data, the present invention introduces a data verification mechanism in the storage process. By double-verifying the time series consistency of trajectory data and the integrity of video data, it is ensured that the stored content can fully reflect the characteristics of abnormal events.

[0089] As a preferred method, the present invention can automatically generate a detailed event report based on the stored event data. Specifically, the event report will be generated in the form of an electronic document, including: Title and Summary: A brief description of the name, time, and location of the event.

[0090] Event details: event category, trajectory analysis results, video screenshots, and severity rating.

[0091] Trajectory chart: Displays the deviation of the vehicle trajectory through a visual chart, including the two-dimensional spatial path diagram of the trajectory and the speed change curve.

[0092] Video Link: Additional video storage path for quick playback.

[0093] For example, in a three-lane highway scenario, a vehicle is detected and an alarm is triggered due to wrong-way behavior. The generated content of the event report may include: Title: Wrong-way Vehicle Detection Incident Report Time: December 21, 2024 10:32:45 Location: Main lane of Section A of the expressway (pile number K123+500) Category: Retrograde Severity: High (S=15.8) Attachments: trajectory chart, video screenshots and full video link.

[0094] It is understandable that the event report of the present invention can not only provide detailed information of a single event, but also support statistics and export of multiple events by dimensions such as time, event type and camera number. As an extended function, the present invention provides a data query module, and users can quickly retrieve relevant event data by setting query conditions (such as time range, event type) and generate batch statistical reports.

[0095] In some embodiments, in order to optimize the use of storage space, the present invention designs a data archiving and cleaning mechanism. The data of low-priority events will be automatically archived after being stored for a period of time, while the data of high-priority events will be marked as long-term storage to ensure the availability of key data. The archived data will be compressed and stored, while retaining index information for subsequent retrieval.

[0096] As an extended technical content, the present invention also supports a multi-node backup strategy for event data. By synchronizing key data to multiple storage nodes, data loss caused by single-node failure can be avoided. For example, in some cases, the data of vehicles traveling in the wrong direction will be backed up to the traffic management center and the cloud storage platform at the same time.

[0097] Through the above steps, the present invention realizes efficient storage and visual report generation of abnormal event related data, and provides a complete and reliable decision support basis for the traffic management system. It should be emphasized that although the specific implementation method described above is the preferred solution of the present invention, those skilled in the art can make appropriate adjustments to the data storage format, event report content and data management strategy according to actual needs without affecting the core idea of ​​the technical solution of the present invention.

[0098] Please refer to the attached Figure 2 The present invention also provides a highway abnormal event detection and alarm system based on deep learning, including: Data acquisition module, used to collect video data and vehicle radar perception data along the highway; A data processing module is used to pre-process the collected data and synchronize the multi-modal data in time; Deep learning analysis module, used for vehicle detection and trajectory extraction on preprocessed data; Abnormal event detection module, used to identify abnormal events based on deviation analysis of vehicle trajectories; An alarm module, which is used to trigger alarms based on the type, location and severity of abnormal events; The data storage and reporting module is used to store relevant data of abnormal events and generate event reports.

[0099] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0100] Through the collaborative work of the above modules, the present invention realizes real-time detection, alarm and report generation of abnormal events on expressways, and the system has high efficiency, robustness and scalability.

[0101] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting and alarming abnormal events on highways based on deep learning, characterized in that: The following steps are involved: Collect multimodal data along the highway; Analyze video data through deep learning models to detect the spatial position and movement trajectory of the vehicle; Fuse radar perception data and construct the spatiotemporal trajectory of the vehicle based on the time series; Identify abnormal events based on deviation analysis of vehicle trajectories; Trigger alarm signals based on detection results, and perform graded alarms based on the type, location, and severity of abnormal events; Store relevant data of abnormal events and generate event reports.

2. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1 is characterized in that: The method of analyzing the video data by a deep learning model to detect the spatial position and motion trajectory of the vehicle includes the following steps: Perform frame preprocessing on the collected video data, use the deep learning model to detect vehicles on each frame of the preprocessed video data, and generate the bounding box of each vehicle and the coordinates of its center point in the image; The coordinates of the center point of each vehicle in multiple frames of video are correlated and analyzed according to the time series to form the continuous motion trajectory of the vehicle; The generated motion trajectory is supplemented with speed information combined with the vehicle radar perception data, and the motion trajectory is expanded into a spatiotemporal trajectory including position, time and speed; The vehicle's spatiotemporal trajectory is compared with the centerline trajectory of the road lane to analyze the vehicle's movement direction and trajectory deviation degree, providing input for abnormal event detection.

3. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1 is characterized in that: The fusing of radar perception data and constructing the spatiotemporal trajectory of the vehicle according to the time series includes the following steps: Extract the speed information, distance information and location coordinates of each vehicle from the radar perception data; Match the center point coordinates of each vehicle in the video data with the spatial position coordinates of the radar perception data to establish the association between the vehicle in the video data and the radar perception data; According to the time series, the coordinates of the center points of the vehicles in the video data are spliced ​​to form a preliminary spatial trajectory; Based on the preliminary spatial trajectory, combined with the vehicle speed information in the radar perception data, the speed is embedded into the trajectory data as a time series feature to construct a complete spatiotemporal trajectory; The time intervals in the radar perception data are smoothed by an interpolation algorithm to ensure the continuity of the space-time trajectory in the time dimension, and finally a three-dimensional space-time trajectory including position, speed and time is generated.

4. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1 is characterized in that: The deviation analysis based on vehicle trajectory and identification of abnormal events includes the following steps: Compare the space-time trajectory of the vehicle with the centerline trajectory of the road lane, and calculate the offset distance of the vehicle trajectory from the centerline of the lane; Analyze the vehicle's moving direction and determine whether there is any abnormality in the moving direction by comparing the angle between the vehicle's moving direction and the normal driving direction of the lane; Combine the vehicle's spatiotemporal trajectory and radar perception data to calculate the deviation between the vehicle's actual speed and the road speed limit; By constructing a trajectory deviation energy function, the vehicle speed deviation and trajectory deviation degree are jointly analyzed to identify whether the vehicle is driving in the wrong direction, speeding, driving at a low speed, or has abnormal trajectory behavior; According to the deviation analysis results, vehicles whose trajectory energy function values ​​exceed the threshold are marked and classified as wrong-way vehicles, abnormally parked vehicles, or other types of abnormal events.

5. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1 is characterized in that: The triggering of an alarm signal according to the detection result and the graded alarming according to the type, location and severity of the abnormal event include the following steps: Classify the events according to the type of abnormal events detected, determine the location of the abnormal events, and calibrate the specific lane and distance location of the abnormal events by analyzing the vehicle's spatiotemporal trajectory and matching it with the road geographic information; Calculate the severity of the incident by combining the type of abnormal incident, the location of the incident, and the parameters of vehicle speed and trajectory deviation amplitude; Set alarm priorities according to severity, trigger emergency alarm signals for high-priority events, and trigger regular alarm signals for medium- and low-priority events; When an alarm signal is triggered, an alarm is sounded in a variety of ways and the alarm signal is suppressed to prevent the same abnormal event from repeatedly triggering an alarm in a short period of time.

6. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1 is characterized in that: The storing of the relevant data of the abnormal event and generating the event report comprises the following steps: After an abnormal event is detected, video data from 10 seconds before and 20 seconds after the event is extracted and stored together with the trajectory data of the abnormal event, vehicle speed information, and classification results of the abnormal event; The stored abnormal event data is classified and marked through the data annotation module, and the stored abnormal event data is generated into a structured record form; Generate an event report based on the abnormal event data. The event report includes the type, location, time, severity, and related tracks and video screenshots of the abnormal event. It provides multi-dimensional query and export functions by time, event type and camera number, supports exporting generated event reports in the form of electronic documents, regularly cleans and archives stored data, and performs long-term storage management of high-priority event data.

7. A highway abnormal event detection and alarm system based on deep learning, according to any one of claims 1 to 6, characterized in that: include: Data acquisition module, used to collect video data and vehicle radar perception data along the highway; A data processing module is used to pre-process the collected data and synchronize the multi-modal data in time; Deep learning analysis module, used for vehicle detection and trajectory extraction on preprocessed data; Abnormal event detection module, used to identify abnormal events based on deviation analysis of vehicle trajectories; An alarm module, which is used to trigger alarms based on the type, location and severity of abnormal events; The data storage and reporting module is used to store relevant data of abnormal events and generate event reports.

8. The highway abnormal event detection and alarm system based on deep learning according to claim 7 is characterized in that: The abnormal event detection module performs trajectory fusion of multi-camera data through a distributed optimization method, and the distributed optimization method calculates the consistency of vehicle trajectories collected by multiple cameras through an alternating direction multiplier method.

9. The highway abnormal event detection and alarm system based on deep learning according to claim 7 is characterized in that: The alarm module supports multiple alarm modes, including: real-time pop-up alarm; sound and light alarm; mobile client alarm.

10. The method for detecting and alarming abnormal events on highways based on deep learning according to claim 1, characterized in that: The deep learning model is a deep learning model based on a spatiotemporal convolutional neural network, which is used to extract time series features and spatial features in video data and output the vehicle's trajectory position, movement direction and speed.

Images