Highway abnormal event intelligent identification method based on multi-mode perception

Abstract

The invention discloses an intelligent highway abnormal event identification method based on multi-mode perception, and relates to the field of congestion identification, and the method comprises the steps: collecting the dynamic data of a highway in real time through a sensor network; feature extraction is carried out based on historical and real-time traffic flow data of each road section of the expressway, the road congestion condition is judged through feature fusion, and a congestion index is generated; based on the congestion event, triggering traceability analysis, and generating a feature vector of the source area; carrying out model training through a deep reinforcement learning model, obtaining a trained congestion reason identification and classification model, and obtaining a congestion reason label; and generating early warning information based on an identification result, and updating a navigation route through the linkage navigation platform. The method has the advantages that abnormal events on the expressway are accurately recognized by fusing video monitoring, radar, weather and toll station data, real-time traffic early warning is provided through the deep learning and prediction model, and road safety and traffic efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of congestion identification, and in particular to an intelligent identification method for abnormal events on highways based on multimodal perception. Background Technology

[0002] The intelligent identification method for abnormal events on highways based on multimodal perception is an advanced system that combines multiple sensing technologies such as vision, sound, and radar. By comprehensively utilizing these technologies, the system can monitor and analyze various abnormal events occurring on highways in real time. This information, after real-time processing and intelligent analysis, can quickly and accurately identify abnormal situations and promptly issue alerts to relevant management departments, thereby effectively preventing accidents and improving highway traffic safety.

[0003] Current intelligent methods for identifying highway anomalies on the market use a single data source or a limited number of data modalities, such as relying solely on video surveillance or radar data, leading to limitations in information acquisition and insufficient accuracy. Secondly, some methods lack effective data fusion and dynamic update mechanisms during feature extraction and analysis, making it difficult to achieve high accuracy and real-time performance in traffic condition assessment and prediction. Furthermore, some systems lack deep learning models to support congestion source tracing and cause identification, resulting in insufficient ability to cope with complex traffic scenarios and difficulty in accurately identifying and quickly responding to various traffic anomalies. Summary of the Invention

[0004] To improve existing methods, this paper proposes an intelligent identification method for abnormal events on highways based on multimodal perception. This method integrates data from video surveillance, radar, meteorology, and toll stations using multimodal perception technology to accurately identify abnormal events on highways. It also utilizes deep learning and prediction models to provide real-time traffic warnings, optimize traffic flow and route planning, and improve road safety and traffic efficiency.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for intelligent identification of highway anomaly events based on multimodal perception includes: Real-time dynamic data of highways are collected through sensor networks, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity and toll station traffic status data. Feature extraction is performed on historical and real-time traffic flow data of various sections of the highway, and the road congestion situation is determined by feature fusion to generate a congestion index. When the congestion index exceeds a preset threshold, it is determined to be a congestion event and triggers source tracing analysis. By tracing back the video data of the congested area, video surveillance data of the source area of ​​the congestion is obtained, and feature extraction is performed to generate the feature vector of the source area. Based on the acquired feature vectors, a deep reinforcement learning model is used to train the model and obtain a congestion cause identification and classification model. The acquired feature vectors are then input into the model to obtain congestion cause labels, which include: traffic accidents, weather, and toll stations. Based on the identification results, an early warning message containing the location, cause, and expected duration is generated, and the navigation route is updated through linkage with the navigation platform.

[0006] Preferably, the real-time acquisition of dynamic highway data via sensor networks, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity, and toll station traffic status data, specifically includes: Real-time data on road condition changes is captured by cameras along the highway. Real-time monitoring of vehicle speed, position, and trajectory using microwave radar; Monitoring road visibility and rainfall intensity using meteorological sensors; Data on traffic flow, speed, and transaction information are obtained from toll stations and electronic toll collection systems. Based on the collected data, time synchronization and alignment are performed by connecting to an NTP time server. The data will be stored in segments according to road segment IDs, and data packets will be generated based on fixed time windows, containing all collected data within that time period.

[0007] Preferably, the step of extracting features based on historical and real-time traffic flow data of each section of the highway, and determining road congestion through feature fusion to generate a congestion index specifically includes: Historical and real-time traffic flow data for each section of the highway are preprocessed, and key features are extracted, including lane occupancy rate, average headway, speed variation coefficient, visibility impact factor, rainfall intensity index, and traffic efficiency. Based on a fixed time window for generating data packets, features are dynamically extracted and outliers are corrected. The corrected feature data is normalized, and weights are calculated through a two-layer attention mechanism, including environmental weights and traffic weights. The congestion index is calculated based on the assigned dynamic weights, and the congestion level is classified based on the magnitude of the congestion index.

[0008] Preferably, the step of determining a congestion event based on a congestion index exceeding a preset threshold and triggering source tracing analysis, specifically involves obtaining video surveillance data of the congestion source area by tracing back video data of the congested area, extracting features, and generating a feature vector of the source area, including: Based on the classification of congestion index levels, including smooth traffic, slow traffic, light congestion, and heavy congestion, congestion events are defined as those with a congestion index of light or heavy congestion. Based on the determination of a congestion event, a source tracing analysis is triggered, and a backtracking time window is calculated. The spatial range is divided by expanding along the traffic flow direction with the core congestion point as the center. Based on video data within the retrospective time window, the congestion propagation is inverted by reversing the vehicle motion vector frame by frame, starting from the current congestion frame, to determine the congestion source area. Feature extraction is performed based on the identified congestion source areas to obtain visual, environmental, and temporal features within the areas, and feature vectors are generated.

[0009] Preferably, the model is trained using a deep reinforcement learning model based on historical event data to obtain a trained congestion cause identification and classification model. The obtained feature vectors are then input into the model to obtain congestion cause labels, which include: traffic accidents, weather, and toll stations. Specifically, these include: Based on the characteristic data of congestion source areas in historical event data, normalization processing is performed and used as model input; The sub-network based on the deep reinforcement learning model includes a spatial feature extractor, a temporal feature extractor, and an environmental feature processor, which perform feature extraction processing respectively; Model training is performed based on the extracted feature data. In the initialization phase, 10% of the samples are supervised learning through a pre-trained feature extractor to obtain a congestion cause identification and classification model trained by reinforcement learning. The feature vectors of the congestion source areas acquired in real time are input into the model, and congestion cause labels are added to each congestion source area. The congestion cause labels include traffic accidents, weather, and toll stations.

[0010] Preferably, the step of generating warning information including location, cause, and estimated duration based on the identification results, and updating the navigation route through the linkage navigation platform, specifically includes: Based on the congestion cause identification and classification results and the characteristic data of the congestion source area, the location data and cause type data of the congestion source area are obtained; Based on historical congestion data, different prediction logics are adopted according to the cause type. The model is trained through an LSTM neural network model to build a congestion duration prediction model. The location data and cause type data of the congestion source area are input into the model to obtain the expected duration of the congestion source area. Early warning information is generated based on the location data of the source area of ​​congestion, the type of cause data, and the expected duration, and the early warning level is classified. Based on the early warning information, the navigation platform updates the navigation route in real time.

[0011] Compared with the prior art, the advantages of the present invention are: By integrating multimodal data sources such as video surveillance, microwave radar, meteorological sensors, and toll station data, comprehensive dynamic information of highways is captured, ensuring high accuracy and diversity of data. Secondly, feature extraction and fusion are performed using historical and real-time traffic flow data, combined with a two-layer attention mechanism to generate a congestion index, accurately determining congestion levels and improving the reliability of traffic status analysis. Source tracing analysis quickly locates congestion sources through video data retrospection and vehicle motion vector analysis, generating feature vectors to provide a reliable basis for subsequent cause identification. A deep reinforcement learning model, combining spatial, temporal, and environmental features, efficiently trains a congestion cause classification model, accurately identifying causes such as traffic accidents, weather, or toll stations, with accurate and highly interpretable classification results. Furthermore, a congestion duration prediction model based on an LSTM neural network generates early warning information, including location, cause, and estimated duration, providing real-time updates for navigation platforms, optimizing route planning, and reducing travel delays. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method proposed in this invention; Figure 2 This is a schematic diagram of the data collection proposed in this invention; Figure 3 This is a schematic diagram illustrating the generation of the congestion index proposed in this invention; Figure 4 This is a schematic diagram illustrating the generation of source region feature vectors proposed in this invention; Figure 5 This is a schematic diagram illustrating the acquisition of congestion cause labels proposed in this invention; Figure 6 This is a schematic diagram illustrating the generation of early warning information proposed in this invention. Detailed Implementation

[0013] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0014] See Figure 1 As shown, the intelligent identification method for highway anomaly events based on multimodal perception includes: Step 1: Collect real-time dynamic data of the highway through sensor networks, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity and toll station traffic status data; Step 2: Based on historical and real-time traffic flow data of each section of the highway, feature extraction is performed, and the road congestion situation is determined by feature fusion to generate a congestion index; Step 3: When the congestion index is higher than the preset threshold, it is determined to be a congestion event and the source analysis is triggered. By tracing back the video data of the congested area, the video surveillance data of the source area of ​​the congestion is obtained, and feature extraction is performed to generate the feature vector of the source area. Step 4: Based on the acquired feature vectors, train the model using a deep reinforcement learning model to obtain the trained congestion cause identification and classification model. Input the acquired feature vectors into the model to obtain congestion cause labels, which include: traffic accidents, weather, and toll stations. Step 5: Based on the identification results, generate warning information including location, cause, and expected duration, and update the navigation route through the linkage navigation platform.

[0015] See Figure 2 As shown, real-time dynamic data of the highway is collected through a sensor network, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity, and toll station traffic status data. Specifically, this includes: Real-time data on road condition changes is captured by cameras along the highway. Real-time monitoring of vehicle speed, position, and trajectory using microwave radar; Monitoring road visibility and rainfall intensity using meteorological sensors; Data on traffic flow, speed, and transaction information are obtained from toll stations and electronic toll collection systems. Based on the collected data, time synchronization and alignment are performed by connecting to an NTP time server. The data will be stored in segments according to road segment IDs, and data packets will be generated based on fixed time windows, containing all collected data within that time period.

[0016] Specifically, cameras capture road conditions in real time and acquire image or video stream data. After processing the image data, traffic flow and road condition information can be extracted. By measuring the speed of a vehicle using microwave radar, the radar can provide the vehicle's position data in space. By analyzing the differences between multiple position data points, the vehicle's trajectory can be deduced. Visibility on roads is monitored using meteorological sensors, and rainfall is also monitored using meteorological sensors. The data system at toll stations acquires transaction data such as the number of vehicles passing through, vehicle speed, and records the toll fees and duration of vehicle passage. The timestamps of the acquired data are synchronized via an NTP time server. Each piece of sensor data should be timestamped to ensure time alignment between data sources. Data packets are generated based on a fixed time window. All collected data are organized into a single data packet according to the time interval. The generated data packets are stored in the database according to the road segment ID, timestamp, and data content, and backups are made to prevent loss.

[0017] See Figure 3 As shown, features are extracted based on historical and real-time traffic flow data of various highway sections. Through feature fusion, road congestion is determined, and a congestion index is generated. Specifically, this includes: Historical and real-time traffic flow data for each section of the highway are preprocessed, and key features are extracted, including lane occupancy rate, average headway, speed variation coefficient, visibility impact factor, rainfall intensity index, and traffic efficiency. Based on a fixed time window for generating data packets, features are dynamically extracted and outliers are corrected. The corrected feature data is normalized, and weights are calculated through a two-layer attention mechanism, including environmental weights and traffic weights. The congestion index is calculated based on the assigned dynamic weights, and the congestion level is classified based on the magnitude of the congestion index.

[0018] Specifically, based on the preprocessed data, some common traffic features are extracted: lane occupancy rate represents the proportion of vehicles occupying each lane at a certain moment, reflecting the degree of road congestion; headway refers to the time interval between two vehicles, reflecting the tension of traffic flow; speed variation coefficient reflects the degree of fluctuation in vehicle speed, which can quantify the stability of traffic flow; and traffic efficiency measures the number of vehicles passing through the highway per unit time, reflecting the road's capacity. Based on the generated time window data packets, features are dynamically extracted to capture changes in traffic flow in different time periods. The dynamically extracted features may be affected by instantaneous anomalies. Outliers are corrected by using the moving average method or by using historical data to correct feature values ​​that do not meet expectations. The weights are calculated using a two-layer attention mechanism. The environmental weight reflects the impact of factors such as weather, visibility, and rainfall on traffic flow, while the traffic weight reflects the impact of factors such as vehicle density and speed on congestion. Combining the environmental and traffic weights, a congestion index is calculated by weighted averaging. Based on the value of the congestion index, the road congestion is divided into different levels, including smooth traffic, slow traffic, light congestion, and heavy congestion.

[0019] See Figure 4 As shown, a congestion event is identified when the congestion index exceeds a preset threshold, triggering source tracing analysis. This involves backtracking video data from the congested area to obtain video surveillance data from the source area, extracting features, and generating a feature vector for the source area. Specifically, this includes: Based on the classification of congestion index levels, including smooth traffic, slow traffic, light congestion, and heavy congestion, congestion events are defined as those with a congestion index of light or heavy congestion. Based on the determination of a congestion event, a source tracing analysis is triggered, and a backtracking time window is calculated. The spatial range is divided by expanding along the traffic flow direction with the core congestion point as the center. Based on video data within the retrospective time window, the congestion propagation is inverted by reversing the vehicle motion vector frame by frame, starting from the current congestion frame, to determine the congestion source area. Feature extraction is performed based on the identified congestion source areas to obtain visual, environmental, and temporal features within the areas, and feature vectors are generated.

[0020] Specifically, mild and severe congestion are marked as congestion events. When a congestion event is identified, source analysis is required to trace the source of the congestion and divide the spatial range along the direction of traffic flow. Based on the density and speed information of traffic flow, a spatial range is defined, and the data within this range is used for source analysis. By reviewing video data within a time window, the motion state of vehicles in each frame is analyzed based on their movement trajectories. The motion vector of each video frame can be extracted, reflecting the speed and direction of changes in vehicle position. For each frame, the vehicle's velocity vector and position information are calculated. Then, based on the motion patterns of historical frames, the congestion propagation path is inferred backwards. Based on the frame-by-frame reverse analysis results, areas where vehicle motion vectors significantly slow down or stop are identified; these are typically the source areas of congestion. These areas usually gradually expand, forming congestion zones. After identifying the source area of ​​congestion, features are extracted from that area. Visual features are mainly extracted from video data, including traffic density, vehicle speed distribution, and road occupancy. Environmental features include weather, visibility, and rainfall factors, including visibility and rainfall intensity. Temporal features mainly refer to the time period in which congestion events occur. Visual features, environmental features, and temporal features are combined into a unified feature vector.

[0021] See Figure 5 As shown, based on historical event data, a deep reinforcement learning model is used to train the model, obtaining a completed congestion cause identification and classification model. The acquired feature vectors are input into the model to obtain congestion cause labels, which include: traffic accidents, weather, and toll stations. Specifically, these include: Based on the characteristic data of congestion source areas in historical event data, normalization processing is performed and used as model input; The sub-network based on the deep reinforcement learning model includes a spatial feature extractor, a temporal feature extractor, and an environmental feature processor, which perform feature extraction processing respectively; Model training is performed based on the extracted feature data. In the initialization phase, 10% of the samples are supervised learning through a pre-trained feature extractor to obtain a congestion cause identification and classification model trained by reinforcement learning. The feature vectors of the congestion source areas acquired in real time are input into the model, and congestion cause labels are added to each congestion source area. The congestion cause labels include traffic accidents, weather, and toll stations.

[0022] Specifically, the model consists of multiple sub-networks, each responsible for different types of feature extraction and processing. The spatial feature extractor is mainly responsible for extracting features related to spatial location, traffic density, road occupancy, etc. The temporal feature extractor is used to capture time series features, mainly focusing on features that change over time, such as traffic flow and vehicle speed. The environmental feature processor is responsible for processing environmental features, mainly factors such as weather, visibility, and rainfall intensity. During the training process of the model, 10% of the samples in the historical event data are first supervised learning through the pre-training stage in order to initialize the network weights; After pre-training is completed, the model enters the reinforcement learning training phase. The goal of reinforcement learning is to optimize the decision-making strategy through a trial-and-error process. Specifically, it uses environmental feedback to guide the model to update its weights. Once trained, the model receives feature data from real-time collected congestion source areas as input. Based on the input feature data, the model outputs a congestion cause label.

[0023] See Figure 6 As shown, based on the identification results, a warning message is generated that includes the location, cause, and estimated duration. The navigation route is then updated via a linked navigation platform. Specifically, this includes: Based on the congestion cause identification and classification results and the characteristic data of the congestion source area, the location data and cause type data of the congestion source area are obtained; Based on historical congestion data, different prediction logics are adopted according to the cause type. The model is trained through an LSTM neural network model to build a congestion duration prediction model. The location data and cause type data of the congestion source area are input into the model to obtain the expected duration of the congestion source area. Early warning information is generated based on the location data of the source area of ​​congestion, the type of cause data, and the expected duration, and the early warning level is classified. Based on the early warning information, the navigation platform updates the navigation route in real time.

[0024] Specifically, independent LSTM units are designed for different causes of congestion. The training set is divided according to the cause type, and multiple LSTM sub-models are trained. Different causes of congestion have different characteristics, so specific prediction logic needs to be designed for each cause type. For traffic accidents: the duration of congestion caused by traffic accidents is usually related to the severity of the accident and the efficiency of handling. For weather: the congestion caused by weather is closely related to the duration and intensity of meteorological conditions, and needs to be combined with meteorological forecast data. For toll stations: the congestion caused by toll stations is related to traffic flow and toll station processing capacity, and usually has a periodic pattern. Based on the location data of the congestion source area, the cause type data, and the predicted duration, early warning information is generated, and the early warning level is divided according to the severity of the congestion. The early warning information includes location, cause type, duration, and congestion index. Based on the generated warning information, the navigation platform adjusts the navigation route in real time to avoid congested areas. When new warning information is generated or existing warning information is updated, the navigation platform recalculates the route.

[0025] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0026] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0027] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent identification of abnormal events on highways based on multimodal perception, characterized in that, include: Real-time dynamic data of highways are collected through sensor networks, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity and toll station traffic status data. Feature extraction is performed on historical and real-time traffic flow data of various sections of the highway, and the road congestion situation is determined by feature fusion to generate a congestion index. When the congestion index exceeds a preset threshold, it is determined to be a congestion event and triggers source tracing analysis. By tracing back the video data of the congested area, video surveillance data of the source area of ​​the congestion is obtained, and feature extraction is performed to generate the feature vector of the source area. Based on the acquired feature vectors, a deep reinforcement learning model is used to train the model and obtain a congestion cause identification and classification model. The acquired feature vectors are then input into the model to obtain congestion cause labels, which include: traffic accidents, weather, and toll stations. Based on the identification results, an early warning message containing the location, cause, and expected duration is generated, and the navigation route is updated through linkage with the navigation platform.

2. The intelligent identification method for highway abnormal events based on multimodal perception according to claim 1, characterized in that, The real-time acquisition of dynamic highway data through sensor networks, including video surveillance data, vehicle speed and trajectory data detected by microwave radar, road visibility, rainfall intensity, and toll station traffic status data, specifically includes: Real-time data on road condition changes is captured by cameras along the highway. Real-time monitoring of vehicle speed, position, and trajectory using microwave radar; Monitoring road visibility and rainfall intensity using meteorological sensors; Data on traffic flow, speed, and transaction information are obtained from toll stations and electronic toll collection systems. Based on the collected data, time synchronization and alignment are performed by connecting to an NTP time server. The data will be stored in segments according to road segment IDs, and data packets will be generated based on fixed time windows, containing all collected data within that time period.

3. The intelligent identification method for highway abnormal events based on multimodal perception according to claim 1, characterized in that, The process of extracting features from historical and real-time traffic flow data of various highway sections, and then using feature fusion to determine road congestion and generate a congestion index specifically includes: Historical and real-time traffic flow data for each section of the highway are preprocessed, and key features are extracted, including lane occupancy rate, average headway, speed variation coefficient, visibility impact factor, rainfall intensity index, and traffic efficiency. Based on a fixed time window for generating data packets, features are dynamically extracted and outliers are corrected. The corrected feature data is normalized, and weights are calculated through a two-layer attention mechanism, including environmental weights and traffic weights. The congestion index is calculated based on the assigned dynamic weights, and the congestion level is classified based on the magnitude of the congestion index.

4. The intelligent identification method for highway abnormal events based on multimodal perception according to claim 1, characterized in that, The process of determining a congestion event based on a congestion index exceeding a preset threshold and triggering source tracing analysis involves retrieving video data from the congested area to obtain video surveillance data from the source area, extracting features, and generating a feature vector for the source area. Specifically, this includes: Based on the classification of congestion index levels, including smooth traffic, slow traffic, light congestion, and heavy congestion, congestion events are defined as those with a congestion index of light or heavy congestion. Based on the determination of a congestion event, a source tracing analysis is triggered, and a backtracking time window is calculated. The spatial range is divided by expanding along the traffic flow direction with the core congestion point as the center. Based on video data within the retrospective time window, the congestion propagation is inverted by reversing the vehicle motion vector frame by frame, starting from the current congestion frame, to determine the congestion source area. Feature extraction is performed based on the identified congestion source areas to obtain visual, environmental, and temporal features within the areas, and feature vectors are generated.

5. The intelligent identification method for highway abnormal events based on multimodal perception according to claim 1, characterized in that, The method involves training a deep reinforcement learning model based on historical event data to obtain a congestion cause identification and classification model. The acquired feature vectors are then input into the model to obtain congestion cause labels. These labels include: traffic accidents, weather, and toll stations. Specifically, they include: Based on the characteristic data of congestion source areas in historical event data, normalization processing is performed and used as model input; The sub-network based on the deep reinforcement learning model includes a spatial feature extractor, a temporal feature extractor, and an environmental feature processor, which perform feature extraction processing respectively; Model training is performed based on the extracted feature data. In the initialization phase, 10% of the samples are supervised learning through a pre-trained feature extractor to obtain a congestion cause identification and classification model trained by reinforcement learning. The feature vectors of the congestion source areas acquired in real time are input into the model, and congestion cause labels are added to each congestion source area. The congestion cause labels include traffic accidents, weather, and toll stations.

6. The intelligent identification method for highway abnormal events based on multimodal perception according to claim 1, characterized in that, The process of generating early warning information containing location, cause, and estimated duration based on the identification results, and updating the navigation route through the linkage navigation platform, specifically includes: Based on the congestion cause identification and classification results and the characteristic data of the congestion source area, the location data and cause type data of the congestion source area are obtained; Based on historical congestion data, different prediction logics are adopted according to the cause type. The model is trained through an LSTM neural network model to build a congestion duration prediction model. The location data and cause type data of the congestion source area are input into the model to obtain the expected duration of the congestion source area. Early warning information is generated based on the location data of the source area of ​​congestion, the type of cause data, and the expected duration, and the early warning level is classified. Based on the early warning information, the navigation platform updates the navigation route in real time.

Images