Highway anomaly event detection method and system fusing multi-modal data calibration

Abstract

The application provides a highway abnormal event detection method and system fusing multi-modal data calibration, and belongs to the technical field of traffic detection, and comprises the following steps: collecting highway traffic data, determining time series data of a target highway section in a specified time period; performing dynamic programming, Kalman filtering and feature fusion on the time series data to determine a plurality of aligned timestamp prediction state vectors and fusion feature vectors; constructing a traffic heterogeneous graph of each time window and determining a node feature vector of each node of each traffic heterogeneous graph; determining an abnormal label of each node of the traffic heterogeneous graph of each time window to realize multi-classification detection of abnormal events of the target highway section. The method can solve the problems of data time sequence misalignment and noise interference, accurately capture the spatiotemporal dual correlation, realize accurate multi-class abnormality determination, form a closed loop through multi-link cooperation, improve data quality and feature representation capability, and provide more accurate and efficient decision support for traffic control.

Description

Technical Field

[0001] This invention relates to the field of traffic detection technology, and in particular to a method and system for detecting abnormal events on highways that integrates multimodal data calibration. Background Technology

[0002] Early highway anomaly detection relied on manual patrols and single-sensor monitoring, which was inefficient and had limited coverage. With technological advancements, automated detection methods based on single-modal data emerged, such as using cameras to identify congestion or radar to monitor abnormal vehicle speeds. However, these methods suffer from limitations in data dimensionality and weak anti-interference capabilities. While multimodal data fusion was subsequently introduced, it lacked effective temporal alignment and noise calibration mechanisms, and often employed traditional machine learning models that failed to uncover the complex spatial relationships between road segments, lanes, and vehicles. This resulted in low anomaly identification accuracy, insufficient multi-category differentiation capabilities, and difficulty adapting to the dynamic and complex traffic environment of highways.

[0003] Therefore, the present invention provides a method and system for detecting abnormal events on highways by integrating multimodal data calibration. Summary of the Invention

[0004] This invention provides a method and system for detecting abnormal events on highways by integrating multimodal data calibration. It collects highway traffic data within a specified time period for a target highway segment, determines the time-series data for that period, performs dynamic programming, Kalman filtering, and feature fusion to determine multiple aligned timestamps of predicted state vectors and fused feature vectors, constructs a traffic heterogeneity map for each time window within a continuous time window, and determines the node feature vector of each node in the traffic heterogeneity map for each time window. It also determines the anomaly label for each node in the traffic heterogeneity map for each time window, achieving multi-class detection of abnormal events on the target highway segment. This method can solve the problems of data temporal misalignment and noise interference, accurately capture spatiotemporal dual correlations, strengthen differentiated feature aggregation, achieve accurate multi-class anomaly judgment and multi-stage collaborative closed-loop formation, improve data quality and feature representation capabilities, effectively reduce false negative and false positive rates, adapt to complex traffic scenarios, and provide more accurate and efficient decision support for traffic management.

[0005] On the one hand, the present invention provides a method for detecting highway anomalies by integrating multimodal data calibration, comprising: Step 1: Collect highway traffic data within a specified time period using sensor arrays based on the target highway segment; and determine the time series data for the target highway segment within the specified time period based on the highway traffic data. Step 2: Perform dynamic programming, Kalman filtering, and feature fusion on the time series data of the target highway segment within a specified time period to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Step 3: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window. Step 4: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, determine the anomaly label of each node in the traffic heterogeneous graph for each time window, and realize multi-class detection of abnormal events in the target highway segment.

[0006] The highway anomaly detection method based on fused multimodal data calibration provided by the present invention collects highway traffic data within a specified time period based on a sensor array of a target highway segment, including: The sensor array includes radar, cameras, and road surface sensors; Based on the radar and radar sampling frequency of the target highway segment, radar data is collected at each collection timestamp within a specified time period of the target highway segment. The radar data includes radar echo intensity and radar velocity. Based on the cameras and camera sampling frequency of the target highway segment, the number of vehicles at each sampling timestamp within a specified time period of the target highway segment is collected. Based on the road surface sensors and the sampling frequency of the road surface sensors of the target highway section, the current of the road surface sensors at each sampling timestamp within a specified time period of the target highway section is collected. Based on radar data collected at all timestamps within a specified time period for the target highway segment, the number of vehicles, and the current of road surface sensors, the highway traffic data for the specified time period for the target highway segment is determined.

[0007] The highway anomaly detection method based on fused multimodal data calibration provided by the present invention determines time series data of a target highway segment within a specified time period based on highway traffic data of the target highway segment, including: The radar echo intensity and radar velocity in the radar data collected at all timestamps within the highway traffic data of the target highway segment within a specified time period are respectively determined as the first and second collection values. Based on the first collection value of all timestamps, the first time series within the specified time period of the target highway segment is determined. At the same time, based on the second collection value of all timestamps, the second time series within the specified time period of the target highway segment is determined. The number of vehicles at each collection timestamp in the highway traffic data of the target highway segment within a specified time period is determined as the third collection value. Based on the third collection values ​​of all collection timestamps, the third time series within the specified time period of the target highway segment is determined. The road surface sensor current of all collected timestamps in the highway traffic data of the target highway segment within a specified time period is determined as the fourth collected value. Based on the fourth collected value of all collected timestamps, the fourth time series within the specified time period of the target highway segment is determined. Based on the first, second, third, and fourth time series within a specified time period of the target highway segment, determine the time series data within the specified time period of the target highway segment.

[0008] The highway anomaly event detection method based on fused multimodal data calibration provided by the present invention performs dynamic programming, Kalman filtering, and feature fusion on time series data within a specified time period of a target highway segment to determine multiple aligned timestamps of predicted state vectors and fused feature vectors, including: DTW dynamic programming is performed on every two time series in the time series data within a specified time period of the target highway segment to determine the aligned time data. The aligned time data includes the first aligned sequence, the second aligned sequence, the third aligned sequence, and the fourth aligned sequence. Each aligned sequence includes the collected values ​​of multiple aligned timestamps. Based on all the alignment sequence acquisition values ​​for each alignment timestamp in the alignment time data, determine the alignment observation vector for each alignment timestamp; Kalman filtering is applied to the alignment observation vector for each alignment timestamp to determine the predicted state vector for each alignment timestamp after spatial calibration. MLP feature fusion is performed on the predicted state vector of each aligned timestamp after spatial calibration to determine the fused feature vector of each aligned timestamp.

[0009] The highway anomaly event detection method based on fused multimodal data calibration provided by the present invention constructs a traffic heterogeneity map for each time window within a continuous time window based on the predicted state vectors and fused feature vectors of all aligned timestamps, and determines the node feature vector of each node in the traffic heterogeneity map of each time window, including: A time sliding window mechanism is adopted to divide all fused feature vectors with aligned timestamps into continuous time windows; A traffic heterogeneity graph is constructed using the fused feature vector with the latest aligned timestamp for each time window in a continuous time window. Multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation of each node in the traffic heterogeneous graph for each time window, as well as the node feature vector of each node.

[0010] The highway anomaly event detection method based on fused multimodal data calibration provided by the present invention constructs a traffic heterogeneity graph from the fused feature vector with the latest aligned timestamp for each time window in a continuous time window, including: Obtain the lane data of the target highway segment, which includes multiple lanes, lane number of each lane, relationship label between each pair of lanes, and spatial label between each lane and the target highway segment; Based on the predicted state vectors of all aligned timestamps for each time window of the target highway segment and the lane numbers of all lanes in the segment lane data, the vehicle lane data for each time window of the target highway segment is determined. The vehicle lane data includes multiple vehicles, the vehicle number of each vehicle, the interaction label between each pair of vehicles, and the driving lane number of each vehicle. For each time window, determine the lane number of each lane in the road segment lane data, the target highway segment, and the vehicle number of each vehicle in the vehicle lane data, as nodes for each time window, and determine the set of node types for each time window for the lane number, the target highway segment, and the vehicle number. For each time window, determine the relationship label between every two lanes in the road segment lane data, the spatial label between each lane and the target highway segment, and the interaction label between every two vehicles in the vehicle lane data. These are the edges for each time window. Also, determine the set of edge types for each time window for all labels in the relationship label, spatial label, and interaction label. Based on the nodes, node type set, edges, and edge type set of each time window, construct a traffic heterogeneity graph for each time window of the target highway segment.

[0011] According to the highway anomaly event detection method based on fused multimodal data calibration provided by the present invention, multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph of each time window to determine the updated feature vector after feature propagation and aggregation of each node in the traffic heterogeneous graph of each time window, as well as the node feature vector of each node, including: Based on the traffic heterogeneity graph of each time window of the target highway segment, determine the set of neighboring nodes of each node in the traffic heterogeneity graph of each time window; The initialization vector of each node of the traffic heterogeneous graph of the target highway segment is determined based on the fused feature vector of all aligned timestamps of each time window. Based on the node type set and edge type set of each time window, features are collected from all neighbors in the neighbor node set of each node in the traffic heterogeneous graph of each time window. Based on the HGNN model, iterative feature propagation and aggregation are performed on each label in the edge type set of each time window to determine the updated feature vector after feature propagation and aggregation of each layer of each node in the traffic heterogeneous graph of each time window. The HGNN model includes a multi-head attention mechanism for the propagation and aggregation of features at each layer within each time window. Based on the HGNN model, for each attention head of each layer of features propagation and aggregation in each time window, and for each node of each layer of features propagation and aggregation in the traffic heterogeneous graph, the attention score and attention weight of each node of each attention head of the traffic heterogeneous graph in each time window and each node in the set of neighboring nodes of the node are calculated. Based on the attention weights of each node of each attention head in the traffic heterogeneous graph for each time window and all nodes in the set of neighboring nodes of the node, the aggregated feature vector of each node in the traffic heterogeneous graph for each time window is calculated. Based on the aggregated feature vector of each node in the traffic heterogeneous graph for each time window, the node feature vector of each node in the traffic heterogeneous graph for each time window is determined.

[0012] The highway anomaly event detection method based on fused multimodal data calibration provided by the present invention determines the anomaly label of each node in the traffic heterogeneity map for each time window based on the node feature vector of each node in the traffic heterogeneity map for each time window, thereby achieving multi-classification detection of anomaly events on the target highway segment, including: The node feature vector of each node in the traffic heterogeneous graph of each time window is input into the fully connected layer to determine the node label and abnormal event data of each node in the traffic heterogeneous graph of each time window. The node label includes normal nodes and abnormal nodes. If the node label is normal node, the abnormal event data is empty. If the node label is abnormal, the abnormal event data includes the abnormal event category and the abnormal probability. If any node in the traffic heterogeneity graph of the time window is labeled as abnormal, the window label of the time window is determined to be abnormal. In the traffic heterogeneous graph of each time window with an abnormal label, the abnormal event category with the highest abnormal probability in the abnormal event data of each node with an abnormal label is selected as the abnormal label of the node in the traffic heterogeneous graph of the time window. Based on the traffic heterogeneous graph with all window labels as abnormal, the abnormal labels of all nodes are abnormal, realizing the detection of abnormal events on highways.

[0013] The highway anomaly event detection method based on fused multimodal data calibration provided by the present invention uses the node labels of a traffic heterogeneous graph where all time windows are anomalous as the anomaly labels for all anomalous nodes, thereby realizing highway anomaly event detection, including: Based on all nodes in the traffic heterogeneous graph of each time window with an abnormal window label, whose node labels are abnormal and whose node type is lane number or target highway segment, the traffic heterogeneous graph of each time window with an abnormal window label is divided to determine the window heterogeneous graph of each time window with an abnormal window label. Feature extraction is performed on the window heterogeneity graph of each time window with an abnormal label to determine the window anomaly distribution vector; Based on the time window where all window labels are abnormal, the abnormal labels of all nodes with abnormal node labels and node type is vehicle number in the traffic heterogeneous graph are determined; Based on the vehicle type label vector and the traffic heterogeneity graph of each window label being abnormal, determine the vehicle label probability vector of each window label being abnormal time window. The window anomaly distribution vector and vehicle label probability vector of two adjacent time windows with anomaly labels are sequentially input into the anomaly meta-evolution model to determine the anomaly meta-evolution vector of the next time window in the two adjacent time windows and the feature label of each feature in the anomaly meta-evolution vector, wherein the feature label includes distribution features and vehicle features. The abnormal evolution vectors of the latter time window of all two adjacent time windows are input into the time series prediction model to determine the predictive meta-evolution vector and the prediction label of each feature in the predictive meta-evolution vector. The prediction label includes distribution features and vehicle features. Based on the features of all predicted labels in the predictor meta-evolution vector that are distributional features and the traffic heterogeneity map of the last time window with anomaly window label, predictor anomaly data is identified. Based on the predicted abnormal data and all predicted features with vehicle characteristics in the predicted meta-evolution vector, the predicted type label mapping data is determined.

[0014] On the other hand, the present invention provides a highway anomaly event detection system that integrates multimodal data calibration, comprising: Data Acquisition Module: Collects highway traffic data within a specified time period based on sensor arrays of the target highway segment, and determines time series data of the target highway segment within the specified time period based on the highway traffic data of the target highway segment; Determining Module: Performs dynamic programming, Kalman filtering, and feature fusion on time series data within a specified time period for the target highway segment to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Construction module: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window; Detection module: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, the module determines the anomaly label of each node in the traffic heterogeneous graph for each time window, thereby achieving multi-class detection of abnormal events in the target highway segment.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: By collecting highway traffic data within a specified time period for a target highway segment, the system determines the time-series data for that period. After dynamic programming, Kalman filtering, and feature fusion, it determines multiple aligned timestamp-aligned predicted state vectors and fused feature vectors. This constructs a traffic heterogeneity map for each time window within a continuous time window, and determines the node feature vector and anomaly label for each node in the traffic heterogeneity map for each time window. This enables multi-class detection of abnormal events on the target highway segment. It solves the problems of data temporal misalignment and noise interference, accurately captures spatiotemporal dual correlations, strengthens differentiated feature aggregation, achieves accurate multi-class anomaly judgment, and forms a closed loop through multi-stage collaboration. This improves data quality and feature representation capabilities, effectively reduces false negative and false positive rates, adapts to complex traffic scenarios, and provides more accurate and efficient decision support for traffic management. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the highway anomaly event detection method based on multimodal data calibration provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the highway abnormal event detection system that integrates multimodal data calibration provided in an embodiment of the present invention; Figure 3 This is a change diagram of the traffic heterogeneity map of the target highway segment in the highway anomaly event detection method that integrates multimodal data calibration provided in this embodiment of the invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. Example 1:

[0019] This invention provides a method for detecting abnormal events on highways by fusing multimodal data calibration, such as... Figure 1 As shown, it includes: Step 1: Collect highway traffic data within a specified time period using sensor arrays based on the target highway segment; and determine the time series data for the target highway segment within the specified time period based on the highway traffic data. Step 2: Perform dynamic programming, Kalman filtering, and feature fusion on the time series data of the target highway segment within a specified time period to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Step 3: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window. Step 4: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, determine the anomaly label of each node in the traffic heterogeneous graph for each time window, and realize multi-class detection of abnormal events in the target highway segment.

[0020] In this embodiment, the sensor array includes radar, cameras, and road surface sensors, which collect multi-dimensional data such as radar echo intensity, radar speed, number of vehicles, and road surface sensor current within a specified time period. These collected raw highway traffic data are organized and associated according to the chronological order of their collection timestamps, binding the sensor data corresponding to each timestamp one-to-one to form a dataset arranged in chronological order. This dataset represents the time-series data of the target highway segment within the specified time period.

[0021] In this embodiment, due to differences in sampling frequencies among different sensors, time-series data suffers from temporal misalignment. Dynamic programming, specifically dynamic time warping, is used to stretch and compress the time-series curves, achieving time alignment of multi-source data and obtaining a unified aligned timestamp. An aligned observation vector is constructed for the multi-dimensional data of each aligned timestamp. This vector is then filtered using Kalman filtering to remove noise interference and correct spatial biases, outputting an accurate predicted state vector. Finally, an MLP model is used to nonlinearly integrate the predicted state vectors, uncovering potential cross-dimensional correlations and eliminating information redundancy, ultimately yielding a fused feature vector for each aligned timestamp.

[0022] In this embodiment, a time sliding window mechanism is employed to divide the fused feature vector aligned with timestamps into time windows of consecutive time periods. Based on the predicted state vector and fused feature vector of each window, a traffic heterogeneous graph is constructed by combining road segment and lane data. Nodes cover road segments, lanes, and vehicles, and edges correspond to the relationships between various elements. After initializing the nodes with the fused feature vector, the HGNN model iteratively propagates and aggregates the association features of different types of edges through multi-layer graph convolution and multi-head attention operations, quantifies the importance of neighboring nodes, and gradually updates the node features, ultimately obtaining the node feature vector for each node.

[0023] In this embodiment, the node feature vector of each node is input into a fully connected layer and mapped to a multi-abnormal event classification space. The layer outputs the anomalous probability and category corresponding to each type of anomalous event, and selects the category with the highest probability as the node's anomalous label. By integrating the anomalous labels of all nodes in the heterogeneous graph for each time window, the distribution and correlation patterns of different types of node labels are analyzed. This process uncovers anomalous features reflected by multi-node linkages, accurately determining whether an anomaly has occurred and its specific type, thus achieving highway anomalous event detection and comprehensively covering common highway anomalous scenarios.

[0024] The beneficial effects of the above technical solution are as follows: By collecting highway traffic data within a specified time period for the target highway segment, determining the time series data within the specified time period for the target highway segment, and performing dynamic programming, Kalman filtering, and feature fusion to determine multiple aligned timestamp prediction state vectors and fused feature vectors, a traffic heterogeneity map for each time window within a continuous time window is constructed, and the node feature vector of each node in the traffic heterogeneity map of each time window is determined. The anomaly label of each node in the traffic heterogeneity map of each time window is also determined, enabling multi-class detection of abnormal events on the target highway segment. This solution can solve the problems of data temporal misalignment and noise interference, accurately capture spatiotemporal dual correlations, strengthen differentiated feature aggregation, achieve accurate judgment of multi-class anomalies and multi-stage collaborative closed-loop formation, improve data quality and feature representation capabilities, effectively reduce the false negative and false positive rates, adapt to complex traffic scenarios, and provide more accurate and efficient decision support for traffic management. Example 2:

[0025] This invention provides a method for detecting abnormal events on highways based on multimodal data calibration. The method involves collecting highway traffic data within a specified time period using a sensor array for a target highway segment, including: The sensor array includes radar, cameras, and road surface sensors; Based on the radar and radar sampling frequency of the target highway segment, radar data is collected at each collection timestamp within a specified time period of the target highway segment. The radar data includes radar echo intensity and radar velocity. Based on the cameras and camera sampling frequency of the target highway segment, the number of vehicles at each sampling timestamp within a specified time period of the target highway segment is collected. Based on the road surface sensors and the sampling frequency of the road surface sensors of the target highway section, the current of the road surface sensors at each sampling timestamp within a specified time period of the target highway section is collected. Based on radar data collected at all timestamps within a specified time period for the target highway segment, the number of vehicles, and the current of road surface sensors, the highway traffic data for the specified time period for the target highway segment is determined.

[0026] In this embodiment, the sensor group is a collection of traffic data acquisition devices deployed on the target highway section. The radar is used to capture information related to vehicle movement, the camera is used to count the number of vehicles, and the road surface sensor is used to sense the physical signals when vehicles pass by. The three work together to achieve the acquisition of multi-dimensional traffic data.

[0027] In this embodiment, the radar sampling frequency is 10Hz, and the radar velocity measurement error is... The camera sampling frequency is 1Hz, and the camera error is... The road surface sensor sampling frequency is 1Hz, and the road surface sensor error is... .

[0028] In this embodiment, the radar collects data at each collection timestamp within a specified time period according to its own set sampling frequency. The sampling frequency determines the time interval for radar data collection. Each collection timestamp records the corresponding radar data. Among them, the radar echo intensity can reflect the reflection characteristics of the target object and can help identify targets such as vehicles. The radar speed directly reflects the vehicle's driving speed. These data together constitute the traffic status record in the radar dimension.

[0029] In this embodiment, the camera takes pictures and analyzes the target highway segment at each collection timestamp within a specified time period according to its own sampling frequency. The number of vehicles at that time is counted by image recognition technology. The sampling frequency of the camera determines the collection density of vehicle quantity data. This data can intuitively reflect the traffic flow density of the highway at different times.

[0030] In this embodiment, the road surface sensor is a device installed on the highway road surface. Based on its own sampling frequency, it records its own current changes at each collection timestamp within a specified time period. When a vehicle runs over the road surface sensor, it causes a change in the physical state of the sensor, which in turn causes the current to fluctuate. Therefore, the current data of the road surface sensor can indirectly reflect the vehicle's passage situation and help determine the vehicle's driving trajectory and passage time.

[0031] In this embodiment, radar data collected at all timestamps within a specified time period, vehicle counts from cameras, and current data recorded by road surface sensors are integrated and linked together according to the order of timestamps to form a complete dataset covering the time period. This dataset is the highway traffic data of the target highway segment during that time period, and it contains multi-dimensional traffic status information.

[0032] The beneficial effects of the above technical solution are as follows: by collecting highway traffic data within a specified time period based on the sensor group of the target highway section, the richness of the data can be improved, providing data support for determining the time series data of the target highway section within a specified time period. Example 3:

[0033] This invention provides a method for detecting abnormal events on highways by fusing multimodal data calibration. Based on highway traffic data of a target highway segment, it determines time-series data of the target highway segment within a specified time period, including: The radar echo intensity and radar velocity in the radar data collected at all timestamps within the highway traffic data of the target highway segment within a specified time period are respectively determined as the first and second collection values. Based on the first collection value of all timestamps, the first time series within the specified time period of the target highway segment is determined. At the same time, based on the second collection value of all timestamps, the second time series within the specified time period of the target highway segment is determined. The number of vehicles at each collection timestamp in the highway traffic data of the target highway segment within a specified time period is determined as the third collection value. Based on the third collection values ​​of all collection timestamps, the third time series within the specified time period of the target highway segment is determined. The road surface sensor current of all collected timestamps in the highway traffic data of the target highway segment within a specified time period is determined as the fourth collected value. Based on the fourth collected value of all collected timestamps, the fourth time series within the specified time period of the target highway segment is determined. Based on the first, second, third, and fourth time series within a specified time period of the target highway segment, determine the time series data within the specified time period of the target highway segment.

[0034] In this embodiment, radar data corresponding to each collection timestamp is extracted one by one from the integrated highway traffic data within a specified time period of the target highway segment. The radar echo intensity in the radar data at each collection timestamp is explicitly defined as the first acquisition value, and the radar speed in the radar data at the same collection timestamp is explicitly defined as the second acquisition value. Based on the chronological order of the collection timestamps, the first acquisition values ​​corresponding to all collection timestamps are arranged chronologically to form the first time series within the specified time period of the target highway segment. This series fully records the dynamic change of radar echo intensity over time. Simultaneously, using the same time sorting rules, the second acquisition values ​​corresponding to all collection timestamps are arranged sequentially to form the second time series, accurately presenting the temporal fluctuations of vehicle radar speed throughout the specified time period.

[0035] In this embodiment, a first time series within a specified time period of the target highway segment is determined based on the first collected values ​​of all collected timestamps, and a second time series within the specified time period of the target highway segment is determined based on the second collected values ​​of all collected timestamps, as expressed as: ; ; in, Indicates the first time series. Indicates the second time series. These represent the first collected values ​​at the 1st, 1st, and 1st timestamps of the highway traffic data within a specified time period for the target highway segment. These represent the second collection values ​​of the first, i-th, and T-th collection timestamps in the highway traffic data of the target highway segment within a specified time period, respectively, where T represents the number of collection timestamps within the specified time period.

[0036] In this embodiment, vehicle quantity data obtained from camera statistics is extracted from highway traffic data of the target highway segment over a specified time period, corresponding to each collection timestamp, and this vehicle quantity is defined as the third collection value. Strictly following the chronological order of the collection timestamps, the third collection values ​​corresponding to all collection timestamps are arranged one by one to construct a third time series for the target highway segment within that specified time period. This series can clearly reflect the changing patterns of traffic density within the target highway segment at different times.

[0037] In this embodiment, the third time series within a specified time period of the target highway segment is determined based on the third collected value of all collected timestamps, and is expressed as follows: ; in, Indicates the third time series. These represent the third collection values ​​of the first, i-th, and T-th collection timestamps in the highway traffic data within a specified time period for the target highway segment.

[0038] In this embodiment, road surface sensor current data corresponding to each collection timestamp is extracted from highway traffic data of the target highway segment over a specified time period and defined as the fourth collection value. Following the order of the collection timestamps from earliest to latest, all the fourth collection values ​​corresponding to the collection timestamps are organized in an orderly manner to form the fourth time series for the target highway segment within that specified time period. This series completely records the changes in road surface sensor current over time.

[0039] In this embodiment, the fourth time series within a specified time period of the target highway segment is determined based on the fourth acquisition value of all acquisition timestamps, and is expressed as: ; in, This represents the fourth time series. These represent the fourth collection values ​​of the first, i-th, and T-th collection timestamps in the highway traffic data within a specified time period for the target highway segment.

[0040] In this embodiment, the first time series, the second time series, the third time series and the fourth time series obtained within a specified time period of the target highway segment are integrated to determine the time series data of the target highway segment within that specified time period.

[0041] The beneficial effects of the above technical solution are as follows: Based on the highway traffic data of the target highway segment, the time series data of the target highway segment within a specified time period can be determined, which can realize the time series integration of multi-source data and improve the accuracy and efficiency of subsequent data processing. Example 4:

[0042] This invention provides a method for detecting abnormal events on highways by fusing multimodal data calibration. The method performs dynamic programming, Kalman filtering, and feature fusion on time-series data within a specified time period of a target highway segment to determine multiple aligned timestamps of predicted state vectors and fused feature vectors, including: DTW dynamic programming is performed on every two time series in the time series data within a specified time period of the target highway segment to determine the aligned time data. The aligned time data includes the first aligned sequence, the second aligned sequence, the third aligned sequence, and the fourth aligned sequence. Each aligned sequence includes the collected values ​​of multiple aligned timestamps. Based on all the alignment sequence acquisition values ​​for each alignment timestamp in the alignment time data, determine the alignment observation vector for each alignment timestamp; Kalman filtering is applied to the alignment observation vector for each alignment timestamp to determine the predicted state vector for each alignment timestamp after spatial calibration. MLP feature fusion is performed on the predicted state vector of each aligned timestamp after spatial calibration to determine the fused feature vector of each aligned timestamp.

[0043] In this embodiment, due to differences in the sampling frequencies of different sensors, the acquisition timestamps of the first, second, third, and fourth time series cannot completely correspond, resulting in a timing misalignment problem. By performing DTW dynamic programming on each pair of these four time series, and leveraging the temporal stretching and compression capabilities of the dynamic time warping algorithm, the optimal alignment path between each pair of sequences is found, enabling precise matching of the originally misaligned acquisition timestamps and forming a unified aligned timestamp. Based on this aligned timestamp, the four original time series are transformed into corresponding aligned sequences: the first aligned sequence corresponds to the aligned data of the original first time series, the second aligned sequence corresponds to the aligned data of the original second time series, the third aligned sequence corresponds to the aligned data of the original third time series, and the fourth aligned sequence corresponds to the aligned data of the original fourth time series. Each aligned sequence contains acquisition values ​​corresponding to multiple aligned timestamps, achieving temporal dimension unification of multi-source time series data.

[0044] In this embodiment, DTW dynamic programming is performed on every two time series within a specified time period of the target highway segment to determine the aligned time series for each time series in the time series data. The DTW dynamic programming recursive formula is expressed as: ; in, This represents the minimum cumulative distance of the optimal alignment path between the first i-th collection timestamp of time series a and the first j-th collection timestamp of time series b within a specified time period of the target highway segment. This represents the a-th acquisition value of the i-th acquisition timestamp in the a-th time series of time series data. and the b-th collected value at the j-th collected timestamp in the b-th time series. Matching error, , This represents the minimum cumulative distance of the optimal alignment path between the first i-1 collection timestamps of time series a and the first j collection timestamps of time series b within a specified time period of the target highway segment. This represents the minimum cumulative distance of the optimal alignment path between the first i-th collection timestamp of time series a and the first j-1 collection timestamps of time series b within a specified time period of the target highway segment. This represents the minimum cumulative distance of the optimal alignment path between the first i-1 collection timestamps of time series a and the first j-1 collection timestamps of time series b within a specified time period of the target highway segment.

[0045] In this embodiment, It can be calculated The Euclidean distance is calculated as follows: These can be radar echo intensity, radar speed, number of vehicles, and road sensor current, respectively.

[0046] In this embodiment, in the alignment time data, each alignment timestamp corresponds to a single acquired value from four alignment sequences. These acquired values ​​originate from four dimensions: radar echo intensity, radar speed, vehicle quantity, and road sensor current. For each alignment timestamp, the acquired values ​​from the first, second, third, and fourth alignment sequences at that timestamp are integrated to construct a multi-dimensional vector. This vector is the alignment observation vector for that alignment timestamp. This vector centrally carries the acquired information from multiple sensors at the same alignment timestamp.

[0047] In this embodiment, although the aligned observation vector achieves time alignment, it may still contain noise data generated during sensor acquisition due to environmental interference and equipment errors. Furthermore, spatial deviations may exist between data from different dimensions. By performing a Kalman filter operation on the aligned observation vector for each aligned timestamp, the optimal estimation capability of the filtering algorithm is utilized to remove noise interference from the vector and correct the spatial deviations of the data, resulting in a spatially calibrated predicted state vector. This vector can more accurately reflect the true state of the traffic system at the corresponding aligned timestamp, eliminating interference factors in the original data.

[0048] In this embodiment, the predicted state vector obtained by Kalman filtering contains effective data in multiple dimensions, but the data in each dimension are independent of each other, resulting in information redundancy, and potential cross-dimensional correlations are not explored. The MLP model is used to perform feature fusion on the predicted state vector aligned to each timestamp. Leveraging the nonlinear transformation capability of the multilayer perceptron, the multi-dimensional data is integrated and optimized, uncovering potential correlations between different dimensions, eliminating redundant information, and strengthening core features.

[0049] The beneficial effects of the above technical solution are: dynamic programming, Kalman filtering and feature fusion are performed on time series data within a specified time period of the target highway segment to determine multiple aligned timestamp prediction state vectors and fused feature vectors, which can provide data fidelity, realize the qualitative upgrade from independent data to fused features, and improve data quality and feature representation capabilities. Example 5:

[0050] This invention provides a method for detecting abnormal events on highways based on fused multimodal data calibration. Based on the predicted state vectors and fused feature vectors of all aligned timestamps, a traffic heterogeneity graph for each time window within a continuous time window is constructed, and the node feature vector of each node in the traffic heterogeneity graph for each time window is determined, including: A time sliding window mechanism is adopted to divide all fused feature vectors with aligned timestamps into continuous time windows; A traffic heterogeneity graph is constructed using the fused feature vector with the latest aligned timestamp for each time window in a continuous time window. Multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation of each node in the traffic heterogeneous graph for each time window, as well as the node feature vector of each node.

[0051] In this embodiment, the fused feature vectors of all aligned timestamps are time-series data arranged in chronological order, covering multi-source fused information at each aligned moment within a specified time period. By employing a time sliding window mechanism, with a fixed window duration and sliding step size, and using the aligned timestamps as a reference, continuous fused feature vectors are sequentially extracted from the start of the time-series data according to the step size, forming multiple non-overlapping or partially overlapping continuous time windows. Each time window contains a segment of fused feature vectors corresponding to consecutive aligned timestamps, achieving segmented processing of long-time-series fused feature data and transforming the overall time-series data into multiple time-segmented datasets.

[0052] In this embodiment, each time window contains multiple consecutive aligned timestamps and corresponding fused feature vectors. The fused feature vector with the latest aligned timestamp reflects the final traffic state within that time window and is the core representation of the traffic state during that period. Based on this fused feature vector with the latest aligned timestamp, a traffic heterogeneity map for the corresponding time window is constructed.

[0053] In this embodiment, for each time window, a heterogeneous traffic graph is constructed. Multi-layer graph convolution operations are used to mine spatial relationships between nodes. Each layer of graph convolution propagates and aggregates the features of each node with those of its neighbors, integrating node features with the relationship information of its surrounding neighbors, thus achieving initial feature enhancement. Simultaneously, multi-head attention operations are introduced. During feature propagation and aggregation at each layer, multiple independent attention branches accurately capture the differences in the strength of relationships between different nodes, making feature aggregation more targeted. After the collaborative operation of each layer of graph convolution and multi-head attention, an updated feature vector for each node at that layer is obtained, recording the state after feature propagation and aggregation at that layer. After multi-layer iterative processing, a node feature vector for each node is finally obtained. This vector contains both the node's own attribute information and the relationship information of multiple layers of neighboring nodes, forming a comprehensive feature representation of the node.

[0054] The beneficial effects of the above technical solution are as follows: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, a traffic heterogeneous map of each time window within a continuous time window is constructed, and the node feature vector of each node in the traffic heterogeneous map of each time window is determined. This can improve the accuracy and richness of feature representation and provide high-quality feature support with both temporal segmentation characteristics and spatial correlation depth for subsequent traffic anomaly detection. Example 6:

[0055] This invention provides a method for detecting abnormal events on highways based on fused multimodal data calibration. The method constructs a traffic heterogeneity graph from the fused feature vector with the latest aligned timestamp for each time window in a continuous time window, including: Obtain the lane data of the target highway segment, which includes multiple lanes, lane number of each lane, relationship label between each pair of lanes, and spatial label between each lane and the target highway segment; Based on the predicted state vectors of all aligned timestamps for each time window of the target highway segment and the lane numbers of all lanes in the segment lane data, the vehicle lane data for each time window of the target highway segment is determined. The vehicle lane data includes multiple vehicles, the vehicle number of each vehicle, the interaction label between each pair of vehicles, and the driving lane number of each vehicle. For each time window, determine the lane number of each lane in the road segment lane data, the target highway segment, and the vehicle number of each vehicle in the vehicle lane data, as nodes for each time window, and determine the set of node types for each time window for the lane number, the target highway segment, and the vehicle number. For each time window, determine the relationship label between every two lanes in the road segment lane data, the spatial label between each lane and the target highway segment, and the interaction label between every two vehicles in the vehicle lane data. These are the edges for each time window. Also, determine the set of edge types for each time window for all labels in the relationship label, spatial label, and interaction label. Based on the nodes, node type set, edges, and edge type set of each time window, construct a traffic heterogeneity graph for each time window of the target highway segment.

[0056] In this embodiment, basic lane data corresponding to the target highway segment is actively collected. This type of data is the core basis for depicting the spatial layout and relationship between the road segment and lanes. The data includes all lanes existing within the road segment, with each lane assigned a unique lane number to distinguish different lanes and avoid confusion. Simultaneously, a relationship label is defined between every two lanes, which describes the actual association attributes between the lanes. Furthermore, a spatial label is determined between each lane and the target highway segment, which defines the spatial affiliation and positional relationship of the lane within the road segment.

[0057] In this embodiment, the relationship labels include adjacent: adjacent to each other on the same cross section, front-wheel drive: connected end to end in the same driving direction, parallel: parallel for a long distance but not necessarily adjacent, opposite: opposite lanes, etc.

[0058] In this embodiment, spatial labels include membership: the road segment is the parent and the lane is the child; geometric inclusion: the lane centerline falls completely within the road segment polygon; and boundary alignment: the lane boundary is parallel to or coincides with the road segment edge line.

[0059] In this embodiment, based on the predicted state vectors of all aligned timestamps within each time window, and combined with all lane numbers in the road segment lane data, the driving lane number of each vehicle within that time window is determined by matching the vehicle position information in the predicted state vectors with the lane spatial range corresponding to the lane number. Based on the vehicle's position, speed, and other state information, the interaction behavior between every two vehicles is analyzed, and corresponding interaction tags are generated. This information collectively constitutes the vehicle lane data for each time window, fully presenting the relationship between vehicles and lanes, and between vehicles themselves, within that time period.

[0060] In this embodiment, interactive tags include following: two vehicles are in the same lane, the longitudinal distance between the front of the following vehicle and the rear of the preceding vehicle is less than 120m, and the relative speed is less than 2m / s; side by side: two vehicles are in adjacent lanes, the lateral distance is less than the width of one lane, and the longitudinal overlap length is greater than half the length of the vehicle body; approaching: the longitudinal distance is rapidly shortened, and the speed difference is less than -3m / s, etc.

[0061] In this embodiment, lane numbers for all lanes are extracted from the road segment lane data of each time window, and each lane number is treated as an independent node; the entire target highway segment is treated as a node; and vehicle numbers for each vehicle are extracted from the vehicle lane data, and each vehicle number is treated as an independent node. These three types of elements together constitute all the nodes of the traffic heterogeneity graph for each time window. Simultaneously, the lane nodes corresponding to lane numbers, the road segment nodes corresponding to the target highway segment, and the vehicle nodes corresponding to vehicle numbers are classified into three node types, forming the node type set for that time window.

[0062] In this embodiment, the relationship label between every two lanes in the road segment lane data is used as an edge connecting the corresponding two lane nodes to represent the association attribute between lanes; the spatial label between each lane and the target highway segment is used as an edge connecting the corresponding lane node and the road segment node to represent their spatial affiliation; and the interaction label between every two vehicles in the vehicle lane data is used as an edge connecting the corresponding two vehicle nodes to represent the interaction behavior between vehicles. The connection relationships corresponding to the above three types of labels constitute all the edges of the traffic heterogeneous graph for each time window. Collecting all specific label content from the relationship labels, spatial labels, and interaction labels forms the edge type set for that time window, clarifying the attribute categories of different edges in the heterogeneous graph, and achieving accurate differentiation of the associations of nodes with different properties.

[0063] In this embodiment, nodes determined for each time window are used as core elements. Nodes are categorized and organized according to their type set to ensure that different types of nodes belong to their respective categories. Based on the edge type set, edges of each type are precisely connected to their corresponding nodes. For example, edges representing lane adjacency are connected to the corresponding two lane nodes, edges representing vehicle following relationships are connected to the corresponding two vehicle nodes, and edges representing lane affiliation are connected to the corresponding lane node and road segment node. In this way, nodes, node types, edges, and edge types are organically integrated to construct a traffic heterogeneous graph corresponding to each time window. This heterogeneous graph can accurately depict the existence state of highway road segments, lanes, and vehicles within that time period and their complex interrelationships, forming a structured graph representation of the traffic state during that time period.

[0064] In this embodiment, a traffic heterogeneity graph is constructed from the fused feature vector with the latest aligned timestamp for each time window in a continuous time window, as shown below: ; in, This represents the traffic heterogeneity graph for the t-th time window. Let represent the set of time nodes and the set of time edges for the t-th time window, respectively. The beneficial effects of the above technical solution are as follows: constructing a traffic heterogeneous graph by merging the latest fused feature vector of the aligned timestamp of each time window in a continuous time window can improve the adaptability and representation depth of the heterogeneous graph to traffic scenarios, provide a more realistic graph structure foundation for subsequent multi-layer graph convolution and multi-head attention operations, and effectively enhance the ability to mine spatiotemporal correlation features. Example 7:

[0065] This invention provides a method for detecting abnormal events on highways by fusing multimodal data calibration. It performs multi-layer graph convolution and multi-head attention operations on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation for each node in the traffic heterogeneous graph for each time window, as well as the node feature vector for each node. This includes: Based on the traffic heterogeneity graph of each time window of the target highway segment, determine the set of neighboring nodes of each node in the traffic heterogeneity graph of each time window; The initialization vector of each node of the traffic heterogeneous graph of the target highway segment is determined based on the fused feature vector of all aligned timestamps of each time window. Based on the node type set and edge type set of each time window, features are collected from all neighbors in the neighbor node set of each node in the traffic heterogeneous graph of each time window. Based on the HGNN model, iterative feature propagation and aggregation are performed on each label in the edge type set of each time window to determine the updated feature vector after feature propagation and aggregation of each layer of each node in the traffic heterogeneous graph of each time window. The HGNN model includes a multi-head attention mechanism for the propagation and aggregation of features at each layer within each time window. Based on the HGNN model, for each attention head of each layer of features propagation and aggregation in each time window, and for each node of each layer of features propagation and aggregation in the traffic heterogeneous graph, the attention score and attention weight of each node of each attention head of the traffic heterogeneous graph in each time window and each node in the set of neighboring nodes of the node are calculated. Based on the attention weights of each node of each attention head in the traffic heterogeneous graph for each time window and all nodes in the set of neighboring nodes of the node, the aggregated feature vector of each node in the traffic heterogeneous graph for each time window is calculated. Based on the aggregated feature vector of each node in the traffic heterogeneous graph for each time window, the node feature vector of each node in the traffic heterogeneous graph for each time window is determined.

[0066] In this embodiment, each time window corresponds to a traffic heterogeneous graph depicting the traffic state during that time period. The graph contains three types of nodes: road segments, lanes, and vehicles, as well as various types of associated edges. For each node in the heterogeneous graph of that time window, based on the connection relationships of the edges in the graph, all other nodes directly connected to that node are selected. These directly connected nodes together constitute the neighbor node set of that node. For example, the neighbor node set of a vehicle node includes the node of its lane and the adjacent vehicle nodes before and after it; the neighbor node set of a lane node includes the adjacent lane nodes and the road segment nodes to which it belongs. Determining the neighbor node set clarifies the local association range of each node in the heterogeneous graph.

[0067] In this embodiment, the fused feature vectors of all aligned timestamps within each time window are high-quality, multi-dimensional data that have undergone time alignment, Kalman filter calibration, and MLP feature fusion. For different types of nodes in the heterogeneous graph of each time window, information matching the node attributes is extracted from the fused feature vectors, and the nodes are assigned initial feature representations, i.e., initialization vectors. For example, the initialization vector of a vehicle node extracts information such as the vehicle's speed and position from the fused feature vector; the initialization vector of a lane node extracts information such as the lane's traffic flow and traffic status from the fused feature vector; and the initialization vector of a road segment node extracts information such as the overall traffic flow and congestion level of the road segment from the fused feature vector. The initialization vectors provide the nodes with basic feature representations that have the advantages of multi-source data fusion.

[0068] In this embodiment, based on the node type set and edge type set of each time window, features are collected from all neighbors in the neighbor node set of each node in the traffic heterogeneous graph of each time window. Then, based on the HGNN model, iterative feature propagation and aggregation are performed on each label in the edge type set of each time window to determine the updated feature vector of each node in the traffic heterogeneous graph of each time window after feature propagation and aggregation at each layer. The update formula and updated feature vector of the (l+1)th layer are expressed as follows: ; ; Among them, t This represents the set of edge types for the t-th time window. Let represent the set of neighboring nodes of the i-th node in the traffic heterogeneity graph at time window t. Let represent the set of neighboring nodes of the j-th node in the traffic heterogeneity graph at time t. Represents the first edge in the set of edge types. One tag, This represents the first edge in the set of edge types for the t-th time window. The learnable weight matrix for each label, This represents the activation function. This indicates that the i-th node of the traffic heterogeneity graph in the t-th time window has passed through... Update the feature vector after layer feature propagation and aggregation. This indicates that the j-th node of the traffic heterogeneity graph in the t-th time window has passed through... Update the feature vector after layer feature propagation and aggregation. Let represent the updated feature vector of the i-th node in the traffic heterogeneity graph at time window t, after L layers of feature propagation and aggregation. Let represent the number of nodes in the set of all neighbor nodes of the j-th node in the traffic heterogeneous graph at time window t. Let L represent the node feature vector of the i-th node in the traffic heterogeneous graph at the t-th time window, and L represent the number of layers for feature propagation and aggregation.

[0069] In this embodiment, when l=0, This represents the initialization vector of the i-th node in the traffic heterogeneity graph at the t-th time window.

[0070] In this embodiment, the first Each tag can be a relationship tag, a spatial tag, or an interaction tag.

[0071] In this embodiment, for example, the 60-second aligned timestamp is slid through a time sliding window mechanism with a window length of 10 seconds and a step size of 1 second, generating a total of 6 consecutive time windows. This process propagates and aggregates the node features in the traffic heterogeneity map. The change graph of the traffic heterogeneity map of the target highway segment within 60 seconds is shown below. Figure 3 As shown.

[0072] In this embodiment, the convolution operation of each layer of the traffic heterogeneity graph for each time window is represented as follows: ; in, This represents the adjacency matrix of the t-th time window. The degree matrix represents the adjacency matrix of the t-th time window. This represents the learning weight matrix for the t-th time window. This represents the t-th time window. The node feature matrix of the layer, This represents the t-th time window. The node feature matrix of the layer.

[0073] In this embodiment, when l+1= hour, Let represent the spatiotemporal feature matrix of the t-th time window.

[0074] In this embodiment, the node feature matrix is ​​determined by the updated feature vector of all nodes in the traffic heterogeneous graph of the corresponding time window after feature propagation and aggregation at the corresponding layer. Each row of the node feature matrix is ​​the updated feature vector of a node.

[0075] In this embodiment, for example, the 60-second aligned timestamp is slid through a time sliding window mechanism with a window length of 10 seconds and a step size of 1 second, generating a total of 6 consecutive time windows.

[0076] In this embodiment, a multi-head attention mechanism is incorporated into the HGNN model's feature propagation and aggregation process for each time window. This mechanism does not aggregate features in a single dimension, but rather captures the association strength between nodes and different types of neighbors and edge types through multiple independent attention branches. Each attention head has an independent parameter system, allowing it to selectively focus on a specific type of node association or a specific dimension of feature interaction. For example, some attention heads focus on the association between a vehicle and its lane, some on the interaction between a vehicle and adjacent vehicles, and some on the spatial association between a lane and its road segment. This mechanism allows feature propagation and aggregation to move beyond an indiscriminate average fusion mode, achieving precise differentiation of the importance of different neighbors.

[0077] In this embodiment, based on the HGNN model, for each attention head that propagates and aggregates features at each layer for each time window, the attention score and attention weight of each node of each attention head in the traffic heterogeneous graph of each time window and each node in the set of neighboring nodes of the node are calculated, and expressed as: ; ; in, Let represent the attention scores of the i-th node and the j-th node of the k-th attention head in the traffic heterogeneity graph during the t-th time window. Let m represent the set of neighboring nodes of the m-th node in the traffic heterogeneity graph at time t. These represent the traffic heterogeneity graph at time window t, where the i-th and j-th nodes pass through respectively. Update the feature vector after layer feature propagation and aggregation. Let represent the learnable weight matrix of the k-th attention head in the traffic heterogeneity graph of the t-th time window. Let represent the parameter vector of the k-th attention head in the traffic heterogeneity graph of the t-th time window. Let represent the attention scores of the i-th node and the m-th node of the k-th attention head in the traffic heterogeneity graph during the t-th time window. This represents the attention weights of the i-th and j-th nodes of the k-th attention head in the traffic heterogeneous graph for the t-th time window.

[0078] In this embodiment, based on the attention weights of each node of each attention head in the traffic heterogeneous graph of each time window and all nodes in the neighbor node set of that node, the aggregated feature vector of each node in the traffic heterogeneous graph of each time window is calculated; based on the aggregated feature vector of each node in the traffic heterogeneous graph of each time window, the node feature vector of each node in the traffic heterogeneous graph of each time window is determined, denoted as: ; ; ; in, Let represent the aggregated feature subvector of the i-th node of the k-th attention head in the traffic heterogeneity graph of the t-th time window. Let represent the aggregated feature vector of the i-th node of all attention heads in the traffic heterogeneity graph at time window t. Let K represent the node feature vector of the i-th node in the traffic heterogeneity graph at the t-th time window, and K represent the number of attention heads.

[0079] The beneficial effects of the above technical solution are as follows: multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation of each node in the traffic heterogeneous graph for each time window, as well as the node feature vector of each node. This achieves refined and categorized feature aggregation, so that the node feature vector of the node has both its own attributes and accurate correlation information, improving the richness and accuracy of feature representation, adapting to the complex node interaction scenarios of highways, and providing high-quality feature support for anomaly detection. Example 8:

[0080] This invention provides a method for detecting abnormal events on highways based on multimodal data calibration. Based on the node feature vector of each node in the traffic heterogeneity graph for each time window, the method determines the anomaly label for each node in the traffic heterogeneity graph for each time window, achieving multi-class classification detection of abnormal events on a target highway segment. The method includes: The node feature vector of each node in the traffic heterogeneous graph of each time window is input into the fully connected layer to determine the node label and abnormal event data of each node in the traffic heterogeneous graph of each time window. The node label includes normal nodes and abnormal nodes. If the node label is normal node, the abnormal event data is empty. If the node label is abnormal, the abnormal event data includes the abnormal event category and the abnormal probability. If any node in the traffic heterogeneity graph of the time window is labeled as abnormal, the window label of the time window is determined to be abnormal. In the traffic heterogeneous graph of each time window with an abnormal label, the abnormal event category with the highest abnormal probability in the abnormal event data of each node with an abnormal label is selected as the abnormal label of the node in the traffic heterogeneous graph of the time window. Based on the traffic heterogeneous graph with all window labels as abnormal, the abnormal labels of all nodes are abnormal, realizing the detection of abnormal events on highways.

[0081] In this embodiment, the node feature vector of each node in the traffic heterogeneous graph of each time window is a core feature representation optimized by the multi-layer feature propagation and aggregation multi-head attention mechanism of the HGNN model. This vector comprehensively carries the node's own attributes, the node's association information with its neighbors, and the importance differences of different association dimensions. The node feature vector of each node is input one by one into a fully connected layer. The role of the fully connected layer is to map the high-dimensional node feature vector to a preset abnormal event classification space, which covers various common highway abnormal event types such as traffic accidents, vehicle malfunctions, and traffic congestion. The fully connected layer outputs the node label of each node and the quantified value of the matching degree of various abnormal events corresponding to the node label being abnormal, i.e., the abnormal probability, through non-linear transformation and dimensionality compression of the node feature vector. It also clarifies the specific abnormal event category corresponding to each abnormal probability. These abnormal event categories and their corresponding abnormal probabilities together constitute the abnormal event data of each node.

[0082] In this embodiment, the abnormal event data of each node in the traffic heterogeneous graph of the time window with the window label "abnormal" contains multiple abnormal event categories and the corresponding abnormal probability for each category. These probability values ​​intuitively reflect the matching degree between the node's feature vector and the features of various abnormal events. For each node in the heterogeneous graph of each time window with the window label "abnormal," the abnormal probability values ​​of all abnormal event categories in its abnormal event data are compared, and the abnormal event category with the highest probability is selected and directly set as the abnormal label for that node. For example, if the abnormal event data of a vehicle node shows that the probability of vehicle malfunction is higher than that of traffic accidents and traffic congestion, then the abnormal label for that vehicle node is vehicle malfunction. Similarly, if the abnormal event data of a lane node shows that the probability of traffic congestion is the highest, then the abnormal label for that lane node is traffic congestion. In this way, each node is given a clear abnormal attribute identifier, enabling accurate determination of the abnormal state of a single node.

[0083] The beneficial effects of the above technical solution are as follows: Based on the node feature vector of each node in the traffic heterogeneous graph of each time window, the anomaly label of each node in the traffic heterogeneous graph of each time window is determined, realizing multi-class detection of anomaly events in the target highway segment. It can fully explore the representational value of the correlation between nodes for anomaly events, improve the accuracy and comprehensiveness of anomaly event identification, adapt to the complex scenario of multiple anomalies intertwined on highways, and provide efficient and accurate decision support for traffic management. Example 9:

[0084] This invention provides a method for detecting highway anomaly events by fusing multimodal data calibration. It uses the node labels of a traffic heterogeneous graph where all time windows are anomalous as the anomaly labels for all anomalous nodes, thus achieving highway anomaly event detection. The method includes: Based on all nodes in the traffic heterogeneous graph of each time window with an abnormal window label, whose node labels are abnormal and whose node type is lane number or target highway segment, the traffic heterogeneous graph of each time window with an abnormal window label is divided to determine the window heterogeneous graph of each time window with an abnormal window label. Feature extraction is performed on the window heterogeneity graph of each time window with an abnormal label to determine the window anomaly distribution vector; Based on the time window where all window labels are abnormal, the abnormal labels of all nodes with abnormal node labels and node type is vehicle number in the traffic heterogeneous graph are determined; Based on the vehicle type label vector and the traffic heterogeneity graph of each window label being abnormal, determine the vehicle label probability vector of each window label being abnormal time window. The window anomaly distribution vector and vehicle label probability vector of two adjacent time windows with anomaly labels are sequentially input into the anomaly meta-evolution model to determine the anomaly meta-evolution vector of the next time window in the two adjacent time windows and the feature label of each feature in the anomaly meta-evolution vector, wherein the feature label includes distribution features and vehicle features. The abnormal evolution vectors of the latter time window of all two adjacent time windows are input into the time series prediction model to determine the predictive meta-evolution vector and the prediction label of each feature in the predictive meta-evolution vector. The prediction label includes distribution features and vehicle features. Based on the features of all predicted labels in the predictor meta-evolution vector that are distributional features and the traffic heterogeneity map of the last time window with anomaly window label, predictor anomaly data is identified. Based on the predicted abnormal data and all predicted features with vehicle characteristics in the predicted meta-evolution vector, the predicted type label mapping data is determined.

[0085] In this embodiment, all time windows labeled as abnormal are selected. For the traffic heterogeneous graph corresponding to each such time window, all nodes with abnormal node labels and node types belonging to lane number or target highway segment are extracted. Using these selected abnormal nodes as the core, the associated edges of these nodes in the original traffic heterogeneous graph and other nodes connected to these edges are retained. The original traffic heterogeneous graph is then trimmed and divided to form a window heterogeneous graph that focuses only on lane and road segment abnormality-related nodes and their relationships. This window heterogeneous graph eliminates redundant nodes and edges unrelated to lane and road segment abnormalities, enabling precise focus on the core abnormal area.

[0086] In this embodiment, for each time window labeled as abnormal, feature extraction is performed on the window heterogeneous graph corresponding to the abnormal node type, which is either lane number or target highway segment. The abnormal distribution probability is calculated, i.e., the proportion of lane nodes of each abnormal category to the total number of lane nodes in the window heterogeneous graph, and the proportion of abnormal road segment nodes. Distribution range features are extracted, including the continuous distribution length of abnormal lanes, the range of lane numbers involved, and the specific station interval of the abnormal road segment. These extracted features are integrated into a structured vector, namely the window abnormal distribution vector, where the abnormal category can be traffic accidents, vehicle malfunctions, traffic congestion, etc.

[0087] In this embodiment, the abnormal labels of all nodes with vehicle ID as the node type and abnormal label in the traffic heterogeneous graph of all time windows with abnormal window labels are collected. These abnormal labels are then classified and statistically analyzed to clarify the existence of various types of vehicle abnormal labels. Information such as the presence and frequency of each type of abnormal label is converted into a vector form, forming a vehicle label vector. The vehicle abnormal labels include vehicle malfunction, vehicle congestion, etc.

[0088] In this embodiment, using the vehicle type label vector as a reference, for each time window where the label is abnormal, the probability distribution of various abnormal labels among nodes with vehicle number as the node type within that window is statistically analyzed. Specifically, the proportion of the number of nodes with each type of vehicle abnormal label to the total number of abnormal vehicle nodes within that window is calculated, and these proportions are arranged according to the feature order in the vehicle type label vector to form a vehicle label probability vector. This vector accurately depicts the probability distribution characteristics of vehicle abnormality types within a single abnormal time window.

[0089] In this embodiment, two adjacent time windows labeled as "abnormal" are selected chronologically. The window anomaly distribution vector and vehicle label probability vector of the preceding time window are extracted, and the two types of vectors corresponding to the following time window are also extracted. These four sets of vectors are input into the anomaly meta-evolution model. The model analyzes the changes and correlation strengths of the road segment anomaly distribution (such as the proportion and range of abnormal lanes) and vehicle anomaly probability (such as the proportion of vehicle congestion) between the two time windows to uncover the evolutionary characteristics of anomalies from the former to the latter. The model outputs the anomaly evolution meta-vector of the following time window, which encapsulates the core features of anomaly evolution. Simultaneously, each feature in the vector is assigned a feature label: if the feature originates from the window anomaly distribution vector (such as changes in the distribution range of abnormal lanes), it is labeled as a distribution feature; if the feature originates from the vehicle label probability vector (such as changes in vehicle congestion probability), it is labeled as a vehicle feature, thus achieving the classification and identification of evolutionary features.

[0090] In this embodiment, the anomaly meta-evolution model adopts a dual-input feature fusion architecture. Its core is to extract the evolutionary essence by comparing the feature differences and correlations between two adjacent anomaly time windows. First, it receives the window anomaly distribution vector and vehicle label probability vector from the previous time window, and the corresponding two types of vectors from the next time window. A multilayer perceptron performs a nonlinear transformation on the four types of vectors, mapping the distribution-related features and vehicle-related features to a high-dimensional feature space. A feature discrimination mechanism is introduced, assigning an initial label to each high-dimensional feature based on its source. Then, a gating unit filters key evolutionary information, focusing on retaining core evolutionary features such as anomaly range changes and anomaly probability fluctuations, while eliminating redundant noise. Finally, the filtered evolutionary features are condensed into low-dimensional anomaly meta-vectors, and the feature label for each feature is determined based on the initial label, achieving structured and classified representation of evolutionary information.

[0091] In this embodiment, the abnormal evolution vectors corresponding to the next time window in all two adjacent abnormal time windows are collected and arranged in chronological order to form a complete temporal evolution vector sequence. This sequence is input into a temporal prediction model. The model learns the temporal dependencies of historical evolution vectors to capture long-term trends in abnormal evolution, such as changes in the expansion rate of distribution features and the type conversion patterns of vehicle features. Based on the learned temporal patterns, the model outputs a predicted meta-evolution vector, which represents the abnormal evolution trend in the next time window. Simultaneously, following the feature label classification rules, each feature in the predicted meta-evolution vector is assigned a predicted label: features originating from historical distribution feature evolution are labeled as distribution features, and features originating from historical vehicle feature evolution are labeled as vehicle features, ensuring the traceability of the predicted feature categories.

[0092] In this embodiment, the time-series prediction model adopts a time-series sequence modeling architecture, the core of which is to learn the temporal patterns of historical evolution sequences and infer future trends. First, it collects the anomalous evolutionary vectors corresponding to all adjacent anomalous windows, forming a complete temporal evolutionary vector sequence in chronological order. This sequence is then input into the model's temporal feature extraction layer. If an LSTM structure is used, the key trends in historical evolution, such as long-term congestion expansion rates, are selectively remembered and short-term noise interference is forgotten through the synergistic effect of the input gate, forget gate, and output gate. If a Transformer encoder is used, the association weights of evolutionary vectors at different time steps are calculated through a self-attention mechanism to accurately capture the temporal patterns of evolution, such as the periodicity of a certain type of anomalous evolution. Through deep learning of historical temporal features, the model outputs a predictive meta-evolutionary vector, which represents the anomalous evolution trend of future time windows. Simultaneously, following the feature labeling rules of the anomalous meta-evolution model, a predictive label is assigned to each feature in the predictive meta-evolutionary vector based on the inheritance relationship of historical feature labels.

[0093] In this embodiment, the traffic heterogeneity map of the last time window with the window label indicating anomaly is used as the basis. This map contains the latest distribution and relationships of abnormal lanes and road segments. Features with distribution characteristics for all predicted labels are extracted from the predictor meta-evolution vector. These features reflect the evolution trend of anomalies at the lane and road segment levels, such as the expansion direction of abnormal lane ranges and the extension trend of abnormal road segments. Combining the current distribution status of abnormal nodes in the traffic heterogeneity map of the last time window, the spatial distribution of anomalies in the next time window is deduced. Information such as the predicted number of abnormal lanes, the corresponding lane number range, and the station interval of the target highway segment of the anomaly are determined and integrated to form predicted anomaly data, accurately defining the spatial coverage of future anomalies.

[0094] In this embodiment, the predicted anomaly data clarifies the spatial range of future anomalies, including specific intervals of abnormal lanes and road segments. Features with vehicle characteristics as predicted labels are extracted from the predicted meta-evolution vector. These features reflect the evolutionary trends of vehicle anomaly types, such as changes in vehicle failure probabilities and the distribution trends of vehicle congestion types. Combining the spatial range defined by the predicted anomaly data, the predicted distribution proportion of various vehicle anomaly types within this range is analyzed, such as the predicted proportion of vehicle failures and vehicle congestion within a predicted abnormal lane. The anomaly spatial range is then bound to the corresponding vehicle anomaly type and its predicted proportion to form predicted type label mapping data, clearly presenting the distribution patterns of vehicle anomaly types within different anomaly spatial ranges.

[0095] The beneficial effects of the above technical solution are as follows: Based on the node labels of the traffic heterogeneous graph where all time windows are abnormal, the abnormal labels of all abnormal nodes are used to realize the detection of abnormal events on highways. This can improve the foresight and comprehensiveness of abnormal detection and further provide accurate data support for the early deployment of response measures for traffic control. Example 10:

[0096] This invention provides a highway anomaly event detection system that integrates multimodal data calibration, such as... Figure 2 As shown, it includes: Data Acquisition Module: Collects highway traffic data within a specified time period based on sensor arrays of the target highway segment, and determines time series data of the target highway segment within the specified time period based on the highway traffic data of the target highway segment; Determining Module: Performs dynamic programming, Kalman filtering, and feature fusion on time series data within a specified time period for the target highway segment to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Construction module: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window; Detection module: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, the module determines the anomaly label of each node in the traffic heterogeneous graph for each time window, thereby achieving multi-class detection of abnormal events in the target highway segment.

[0097] The beneficial effects of the above technical solution are as follows: By collecting highway traffic data within a specified time period for the target highway segment, determining the time series data within the specified time period for the target highway segment, and performing dynamic programming, Kalman filtering, and feature fusion to determine multiple aligned timestamp prediction state vectors and fused feature vectors, a traffic heterogeneity map for each time window within a continuous time window is constructed, and the node feature vector of each node in the traffic heterogeneity map of each time window is determined. The anomaly label of each node in the traffic heterogeneity map of each time window is also determined, enabling multi-class detection of abnormal events on the target highway segment. This solution can solve the problems of data temporal misalignment and noise interference, accurately capture spatiotemporal dual correlations, strengthen differentiated feature aggregation, achieve accurate judgment of multi-class anomalies and multi-stage collaborative closed-loop formation, improve data quality and feature representation capabilities, effectively reduce the false negative and false positive rates, adapt to complex traffic scenarios, and provide more accurate and efficient decision support for traffic management.

[0098] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal events on highways by integrating multimodal data calibration, characterized in that, include: Step 1: Collect highway traffic data within a specified time period using sensor arrays based on the target highway segment; and determine the time series data for the target highway segment within the specified time period based on the highway traffic data. Step 2: Perform dynamic programming, Kalman filtering, and feature fusion on the time series data of the target highway segment within a specified time period to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Step 3: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window. Step 4: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, determine the anomaly label of each node in the traffic heterogeneous graph for each time window, and realize multi-class detection of abnormal events in the target highway segment.

2. The method for detecting highway anomalies based on multimodal data calibration according to claim 1, characterized in that, A sensor array based on a target highway segment collects highway traffic data over a specified time period, including: The sensor array includes radar, cameras, and road surface sensors; Based on the radar and radar sampling frequency of the target highway segment, radar data is collected at each collection timestamp within a specified time period of the target highway segment. The radar data includes radar echo intensity and radar velocity. Based on the cameras and camera sampling frequency of the target highway segment, the number of vehicles at each sampling timestamp within a specified time period of the target highway segment is collected. Based on the road surface sensors and the sampling frequency of the road surface sensors of the target highway section, the current of the road surface sensors at each sampling timestamp within a specified time period of the target highway section is collected. Based on radar data collected at all timestamps within a specified time period for the target highway segment, the number of vehicles, and the current of road surface sensors, the highway traffic data for the specified time period for the target highway segment is determined.

3. The method for detecting highway anomalies based on multimodal data calibration according to claim 2, characterized in that, Based on highway traffic data for the target highway segment, time-series data for a specified time period for the target highway segment is determined, including: The radar echo intensity and radar velocity in the radar data collected at all timestamps within the highway traffic data of the target highway segment within a specified time period are respectively determined as the first and second collection values. Based on the first collection value of all timestamps, the first time series within the specified time period of the target highway segment is determined. At the same time, based on the second collection value of all timestamps, the second time series within the specified time period of the target highway segment is determined. The number of vehicles at each collection timestamp in the highway traffic data of the target highway segment within a specified time period is determined as the third collection value. Based on the third collection values ​​of all collection timestamps, the third time series within the specified time period of the target highway segment is determined. The road surface sensor current of all collected timestamps in the highway traffic data of the target highway segment within a specified time period is determined as the fourth collected value. Based on the fourth collected value of all collected timestamps, the fourth time series within the specified time period of the target highway segment is determined. Based on the first, second, third, and fourth time series within a specified time period of the target highway segment, the time series data within the specified time period of the target highway segment are determined.

4. The highway anomaly event detection method based on multimodal data calibration according to claim 3, characterized in that, Dynamic programming, Kalman filtering, and feature fusion are performed on time series data of the target highway segment within a specified time period to determine multiple aligned timestamp prediction state vectors and fused feature vectors, including: DTW dynamic programming is performed on every two time series in the time series data within a specified time period of the target highway segment to determine the aligned time data. The aligned time data includes the first aligned sequence, the second aligned sequence, the third aligned sequence, and the fourth aligned sequence. Each aligned sequence includes the collected values ​​of multiple aligned timestamps. Based on all the alignment sequence acquisition values ​​for each alignment timestamp in the alignment time data, determine the alignment observation vector for each alignment timestamp; Kalman filtering is applied to the alignment observation vector for each alignment timestamp to determine the predicted state vector for each alignment timestamp after spatial calibration. MLP feature fusion is performed on the predicted state vector of each aligned timestamp after spatial calibration to determine the fused feature vector of each aligned timestamp.

5. The highway anomaly event detection method based on multimodal data calibration according to claim 4, characterized in that, Based on the predicted state vectors and fused feature vectors of all aligned timestamps, a traffic heterogeneity map for each time window within a continuous time window is constructed, and the node feature vector of each node in the traffic heterogeneity map of each time window is determined, including: A time sliding window mechanism is adopted to divide all fused feature vectors with aligned timestamps into continuous time windows; A traffic heterogeneity graph is constructed using the fused feature vector with the latest aligned timestamp for each time window in a continuous time window. Multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation of each node in the traffic heterogeneous graph for each time window, as well as the node feature vector of each node.

6. The method for detecting highway anomalies based on multimodal data calibration according to claim 5, characterized in that, For each time window in a continuous time window, the latest aligned timestamp of the fused feature vector is used to construct a traffic heterogeneity graph, including: Obtain the lane data of the target highway segment, which includes multiple lanes, lane number of each lane, relationship label between each pair of lanes, and spatial label between each lane and the target highway segment; Based on the predicted state vectors of all aligned timestamps for each time window of the target highway segment and the lane numbers of all lanes in the segment lane data, the vehicle lane data for each time window of the target highway segment is determined. The vehicle lane data includes multiple vehicles, the vehicle number of each vehicle, the interaction label between each pair of vehicles, and the driving lane number of each vehicle. For each time window, determine the lane number of each lane in the road segment lane data, the target highway segment, and the vehicle number of each vehicle in the vehicle lane data, as nodes for each time window, and determine the set of node types for each time window for the lane number, the target highway segment, and the vehicle number. For each time window, determine the relationship label between every two lanes in the road segment lane data, the spatial label between each lane and the target highway segment, and the interaction label between every two vehicles in the vehicle lane data. These are the edges for each time window. Also, determine the set of edge types for each time window for all labels in the relationship label, spatial label, and interaction label. Based on the nodes, node type set, edges, and edge type set of each time window, construct a traffic heterogeneity graph for each time window of the target highway segment.

7. The method for detecting highway anomalies based on multimodal data calibration according to claim 5, characterized in that, Multi-layer graph convolution and multi-head attention operations are performed on the traffic heterogeneous graph for each time window to determine the updated feature vector after feature propagation and aggregation for each node in the traffic heterogeneous graph for each time window, as well as the node feature vector for each node, including: Based on the traffic heterogeneity graph of each time window of the target highway segment, determine the set of neighboring nodes of each node in the traffic heterogeneity graph of each time window; The initialization vector of each node of the traffic heterogeneous graph of the target highway segment is determined based on the fused feature vector of all aligned timestamps of each time window. Based on the node type set and edge type set of each time window, features are collected from all neighbors in the neighbor node set of each node in the traffic heterogeneous graph of each time window. Based on the HGNN model, iterative feature propagation and aggregation are performed on each label in the edge type set of each time window to determine the updated feature vector after feature propagation and aggregation of each layer of each node in the traffic heterogeneous graph of each time window. The HGNN model includes a multi-head attention mechanism for the propagation and aggregation of features at each layer within each time window. Based on the HGNN model, for each attention head of each layer of features propagation and aggregation in each time window, and for each node of each layer of features propagation and aggregation in the traffic heterogeneous graph, the attention score and attention weight of each node of each attention head of the traffic heterogeneous graph in each time window and each node in the set of neighboring nodes of the node are calculated. Based on the attention weights of each node of each attention head in the traffic heterogeneous graph for each time window and all nodes in the set of neighboring nodes of the node, the aggregated feature vector of each node in the traffic heterogeneous graph for each time window is calculated. Based on the aggregated feature vector of each node in the traffic heterogeneous graph for each time window, the node feature vector of each node in the traffic heterogeneous graph for each time window is determined.

8. The method for detecting highway anomalies based on multimodal data calibration according to claim 1, characterized in that, Based on the node feature vector of each node in the traffic heterogeneity graph for each time window, the anomaly label of each node in the traffic heterogeneity graph for each time window is determined, realizing multi-class detection of abnormal events in the target highway segment, including: The node feature vector of each node in the traffic heterogeneous graph of each time window is input into the fully connected layer to determine the node label and abnormal event data of each node in the traffic heterogeneous graph of each time window. The node label includes normal nodes and abnormal nodes. If the node label is normal node, the abnormal event data is empty. If the node label is abnormal, the abnormal event data includes the abnormal event category and the abnormal probability. If any node in the traffic heterogeneity graph of the time window is labeled as abnormal, the window label of the time window is determined to be abnormal. In the traffic heterogeneous graph of each time window with an abnormal label, the abnormal event category with the highest abnormal probability in the abnormal event data of each node with an abnormal label is selected as the abnormal label of the node in the traffic heterogeneous graph of the time window. Based on the traffic heterogeneous graph with all window labels as abnormal, the abnormal labels of all nodes are abnormal, realizing the detection of abnormal events on highways.

9. The method for detecting highway anomalies based on multimodal data calibration according to claim 8, characterized in that, Based on the node labels of a traffic heterogeneous graph where all time windows are anomalous, the system uses the anomalous labels of all nodes to achieve highway anomaly event detection, including: Based on all nodes in the traffic heterogeneous graph of each time window with an abnormal window label, whose node labels are abnormal and whose node type is lane number or target highway segment, the traffic heterogeneous graph of each time window with an abnormal window label is divided to determine the window heterogeneous graph of each time window with an abnormal window label. Feature extraction is performed on the window heterogeneity graph of each time window with an abnormal label to determine the window anomaly distribution vector; Based on the time window where all window labels are abnormal, the abnormal labels of all nodes with abnormal node labels and node type is vehicle number in the traffic heterogeneous graph are determined; Based on the vehicle type label vector and the traffic heterogeneity graph of each window label being abnormal, determine the vehicle label probability vector of each window label being abnormal time window. The window anomaly distribution vector and vehicle label probability vector of two adjacent time windows with anomaly labels are sequentially input into the anomaly meta-evolution model to determine the anomaly meta-evolution vector of the next time window in the two adjacent time windows and the feature label of each feature in the anomaly meta-evolution vector, wherein the feature label includes distribution features and vehicle features. The abnormal evolution vectors of the latter time window of all two adjacent time windows are input into the time series prediction model to determine the predictive meta-evolution vector and the prediction label of each feature in the predictive meta-evolution vector. The prediction label includes distribution features and vehicle features. Based on the features of all predicted labels in the predictor meta-evolution vector that are distributional features and the traffic heterogeneity map of the last time window with anomaly window label, predictor anomaly data is identified. Based on the predicted abnormal data and all predicted features with vehicle characteristics in the predicted meta-evolution vector, the predicted type label mapping data is determined.

10. A highway anomaly event detection system integrating multimodal data calibration, characterized in that, A highway anomaly detection method for performing the fusion multimodal data calibration as described in any one of claims 1 to 9, comprising: Data Acquisition Module: Collects highway traffic data within a specified time period based on sensor arrays of the target highway segment, and determines time series data of the target highway segment within the specified time period based on the highway traffic data of the target highway segment; Determining Module: Performs dynamic programming, Kalman filtering, and feature fusion on time series data within a specified time period for the target highway segment to determine the predicted state vector and fused feature vector with multiple aligned timestamps; Construction module: Based on the predicted state vectors and fused feature vectors of all aligned timestamps, construct the traffic heterogeneity map for each time window within the continuous time window, and determine the node feature vector of each node in the traffic heterogeneity map of each time window; Detection module: Based on the node feature vector of each node in the traffic heterogeneous graph for each time window, the module determines the anomaly label of each node in the traffic heterogeneous graph for each time window, thereby achieving multi-class detection of abnormal events in the target highway segment.

Images