A traffic event intelligent identification and early warning method and system based on multi-source perception data fusion

Abstract

The present application belongs to the field of intelligent transportation and digital twin technology, in particular to a traffic event intelligent identification and early warning method and system based on multi-source perception data fusion, comprising the following steps: step S1, multi-source perception data acquisition and space-time alignment: through the laid three-dimensional monitoring network, real-time acquisition of multi-source heterogeneous perception data including roadside high-definition video image, radar-video integrated machine point cloud data and meteorological environment monitoring data, after cleaning each data, space-time alignment and registration are carried out according to unified time stamp and space coordinate system, forming standardized space-time continuous perception data flow. The traffic event intelligent identification and early warning method and system based on multi-source perception data fusion, by constructing a 'data level-feature level-decision level' hierarchical fusion mechanism, and introducing a multi-head self-attention feature fusion module composed of multiple Transformer encoder layers at the feature level, significantly improves the accuracy and robustness of traffic event identification in complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and digital twin technology, and in particular to a method and system for intelligent identification and early warning of traffic incidents based on multi-source sensing data fusion. Background Technology

[0002] Improving traffic safety across the board is a crucial guarantee for building a strong transportation nation. Urban roads experience frequent changes in vehicle trajectories and complex driving scenarios. Once an emergency occurs, it can cause regional traffic congestion, blockages, and even traffic accidents, resulting in serious traffic safety problems.

[0003] Currently, urban road traffic monitoring methods mainly rely on single video images or fixed-point loop data. These methods have significant technical limitations: in complex scenarios such as insufficient lighting at night, inclement weather, or target obstruction, the sensing capability of a single sensor decreases drastically, resulting in a limited information dimension. This makes it difficult to meet the accuracy and real-time requirements of modern urban traffic proactive management in identifying dangerous driving behaviors such as speeding, driving in the wrong direction, rapid acceleration, and rapid deceleration, as well as events such as traffic congestion and accidents. Furthermore, existing systems often fuse multi-source heterogeneous data through simple voting on decision-making results, lacking deep collaboration and adaptive weighting mechanisms at the feature level, leading to insufficient multi-source data fusion and inaccurate event diagnosis.

[0004] Therefore, there is an urgent need for a technical solution that can deeply integrate multi-source heterogeneous sensing data to achieve high-precision, real-time identification and early warning of complex traffic events. Summary of the Invention

[0005] Based on existing technical problems, this invention proposes a method and system for intelligent identification and early warning of traffic incidents based on multi-source sensing data fusion.

[0006] This invention proposes a method for intelligent identification and early warning of traffic incidents based on multi-source sensing data fusion, comprising the following steps:

[0007] Step S1: Acquisition and Spatiotemporal Alignment of Multi-Source Sensing Data: Through the deployed three-dimensional monitoring network, multi-source heterogeneous sensing data, including roadside high-definition video images, point cloud data from integrated radar and vision systems, and meteorological and environmental monitoring data, are acquired in real time. After cleaning each data, spatiotemporal alignment and registration are performed based on a unified timestamp and spatial coordinate system to form a standardized spatiotemporal continuous sensing data stream.

[0008] Step S2, Data-level fusion: Pixel-level enhancement is performed on the spatiotemporally aligned video images, and point cloud data from multiple frames is stitched together at the point cloud level to generate the fused enhanced image data volume and dense point cloud data volume.

[0009] Step S3, Feature-level Fusion: A parallel deep learning feature extraction network is used to extract visual feature maps from the enhanced image data volume, geometric and motion feature maps from the dense point cloud data volume, and environmental feature vectors from meteorological and environmental monitoring data. The visual feature maps, geometric and motion feature maps are concatenated and then input together with the environmental feature vectors into a multi-head self-attention feature fusion module. The module dynamically learns the association weights of different modal features at different spatial locations and time nodes to generate a fusion feature map rich in multimodal context information.

[0010] Step S4, Decision-level Fusion and Event Recognition: Based on the fused feature map, four sub-task models are executed in parallel. Then, the preliminary recognition results output by each sub-task model are input into a trainable weight network for dynamic weighted fusion. After comprehensive logical judgment, the final confidence level of various traffic events is obtained. ;

[0011] Step S5, Event Analysis and Tiered Early Warning: For traffic events with confidence levels exceeding the threshold, analyze and assess their severity, scope of impact, and development trend by combining real-time traffic flow data, and generate differentiated early warning information for release according to preset tiered early warning rules.

[0012] Preferably, in step S4, the preliminary identification results output by each sub-task model are dynamically weighted and fused, and the final confidence level... The calculation formula is: ;

[0013] In the formula, The classification probability vector output by the sub-model for identifying illegal driving behavior; The classification probability vector output by the traffic operation event detection sub-model; The classification probability vector output by the dangerous driving behavior recognition sub-model; The classification probability vector output by the vehicle interaction behavior recognition sub-model;

[0014] , , , These are the fusion weight coefficients dynamically generated by the trainable weight network, corresponding to the four sub-models mentioned above.

[0015] Preferably, the four sub-task models are a traffic violation behavior recognition sub-model, a traffic operation event detection sub-model, a dangerous driving behavior recognition sub-model, and a vehicle interaction behavior recognition sub-model.

[0016] The illegal driving behavior recognition sub-model uses an improved LRCN network to identify illegal driving behaviors of vehicles;

[0017] The traffic operation event detection sub-model uses an LSTM-NN network to detect and classify traffic operation events; the dangerous driving behavior recognition sub-model uses a combination of CNN and LSTM to identify dangerous driving behaviors.

[0018] The vehicle interaction behavior recognition sub-model uses a spatiotemporal graph convolutional network to recognize vehicle interaction behavior;

[0019] The fusion weight coefficients satisfy the following normalization condition: .

[0020] Preferably, the trainable weight network is dynamically generated. to The process is as follows:

[0021] The environmental feature vector generated in step S3 with by , , , Cascaded probability vectors formed by sequential concatenation The vectors are concatenated to form a joint input vector. The calculation formula is: ;

[0022] The joint input vector The input is fed into a three-layer fully connected weighted network, and after being normalized by the Softmax function, the fused weight coefficients are output. , , , .

[0023] Preferably, the improvements in the improved LRCN network in step S4 are as follows:

[0024] After the backbone network in the CNN feature extraction stage, a channel attention (SE) module is introduced to weight each channel of the feature vector, strengthening the feature channels related to violations. In the LSTM temporal modeling stage, a bidirectional LSTM is used instead of a unidirectional LSTM to capture the bidirectional temporal dependencies.

[0025] Preferably, the spatiotemporal graph convolutional network in step S4 specifically performs the following recognition process:

[0026] Using the target vehicle and its neighboring vehicles as nodes, and the relative positions and speeds between vehicles as edges, a dynamic spatiotemporal graph is constructed. The node features are the fused feature vectors of each vehicle, and the edge features are the distance and speed difference codes between vehicles. After multi-layer spatiotemporal graph convolution, the target vehicle node features are classified, and the interaction behavior classification probability vector is output. .

[0027] Preferably, a traffic incident intelligent identification and early warning system based on multi-source sensing data fusion includes:

[0028] The multi-source data access module is used to access and parse multi-source heterogeneous data from roadside video, radar-visual integrated machine, and meteorological monitoring equipment;

[0029] The data preprocessing and spatiotemporal synchronization module is used to clean and reduce noise from multi-source data, and to perform spatiotemporal alignment based on a unified timestamp and spatial coordinate system.

[0030] The hierarchical fusion and intelligent recognition engine has built-in data-level fusion units, feature-level fusion units, and decision-level fusion units;

[0031] The data-level fusion unit generates enhanced image data volumes and dense point cloud data volumes; the feature-level fusion unit integrates a deep learning feature extraction network and a multi-head self-attention feature fusion module; the decision-level fusion unit deploys multiple parallel traffic event detection and recognition sub-models, as well as a trainable weight network, for dynamically weighted fusion of the preliminary results of each sub-model.

[0032] The event analysis and early warning module is used to assess the impact of identified traffic events and generate and issue graded early warning information.

[0033] Preferably, the multiple parallel sub-models deployed in the decision-level fusion unit specifically include:

[0034] The improved LRCN network, with its CNN part followed by a channel attention SE module and its temporal part using a bidirectional LSTM, is used to identify illegal driving behaviors such as speeding, driving in the wrong direction, driving at low speed, and illegal parking.

[0035] LSTM-NN networks are used to detect and classify operational events such as traffic congestion, traffic jams, and traffic accidents.

[0036] A model combining CNN and LSTM is used to identify dangerous driving behaviors such as rapid acceleration, rapid deceleration, and sudden steering.

[0037] Spatiotemporal graph convolutional networks are used to identify vehicle interaction behaviors such as following and lane changing.

[0038] Preferably, the deep learning feature extraction network in the feature-level fusion unit is specifically configured as follows:

[0039] The SlowFast network is used to extract spatiotemporal visual features from video images;

[0040] The PointNet++ network is used to extract the geometric and motion features of the point cloud;

[0041] A fully connected network is used to process meteorological data to generate environmental feature vectors.

[0042] Preferably, the event analysis and early warning module has a preset three-level early warning system: red, orange, and yellow.

[0043] A red alert is generated when a traffic accident or multi-vehicle collision hazard is identified.

[0044] An orange alert is generated when a dangerous single-vehicle driving incident or a severe traffic congestion event is identified.

[0045] A yellow alert is generated when a general violation is identified.

[0046] The beneficial effects of this invention are as follows:

[0047] 1. By constructing a hierarchical fusion mechanism of "data level-feature level-decision level", and introducing a multi-head self-attention feature fusion module composed of multiple Transformer encoder layers at the feature level, the accuracy and robustness of traffic event recognition in complex scenarios are significantly improved.

[0048] 2. By setting up a trainable weighted network at the decision level, and replacing the traditional fixed rules or average voting mechanism with adaptive dynamic weighted fusion, intelligent integrated decision-making with multiple models for dynamic environments is achieved.

[0049] 3. By making targeted structural improvements to the key sub-task model and introducing a spatiotemporal graph convolutional network to model multi-vehicle interaction behavior, the identification dimensions and early warning capabilities of traffic active safety management have been further expanded. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of a traffic incident intelligent identification and early warning method based on multi-source sensing data fusion proposed in this invention.

[0051] Figure 2 This is a system module structure diagram of a traffic incident intelligent identification and early warning system based on multi-source sensing data fusion proposed in this invention;

[0052] Figure 3 This is a schematic diagram illustrating the working principle of the hierarchical fusion and intelligent recognition engine of a traffic incident intelligent recognition and early warning system based on multi-source sensing data fusion proposed in this invention.

[0053] Figure 4 This is a schematic diagram of the working principle of the data-level fusion unit in the hierarchical fusion and intelligent recognition engine of a traffic incident intelligent identification and early warning system based on multi-source sensing data fusion proposed in this invention.

[0054] Figure 5This is a schematic diagram of the working principle of the feature-level fusion unit in the hierarchical fusion and intelligent recognition engine of a traffic incident intelligent identification and early warning system based on multi-source sensing data fusion proposed in this invention.

[0055] Figure 6 This is a schematic diagram of the working principle of the decision-level fusion unit in the hierarchical fusion and intelligent recognition engine of a traffic incident intelligent identification and early warning system based on multi-source perception data fusion proposed in this invention. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0057] Example 1

[0058] Reference Figure 1 A method for intelligent identification and early warning of traffic incidents based on multi-source sensing data fusion includes the following steps:

[0059] Step S1: Acquisition and Spatiotemporal Alignment of Multi-Source Sensing Data: Through the deployed three-dimensional monitoring network, multi-source heterogeneous sensing data, including roadside high-definition video images, point cloud data from integrated radar and vision systems, and meteorological and environmental monitoring data, are acquired in real time. After cleaning each data, spatiotemporal alignment and registration are performed based on a unified timestamp and spatial coordinate system to form a standardized spatiotemporal continuous sensing data stream.

[0060] Step S2, Data-level fusion: Pixel-level enhancement is performed on the spatiotemporally aligned video images, and point cloud data from multiple frames is stitched together at the point cloud level to generate the fused enhanced image data volume and dense point cloud data volume.

[0061] Step S3, Feature-level Fusion: A parallel deep learning feature extraction network is used to extract visual feature maps from the enhanced image data volume, geometric and motion feature maps from the dense point cloud data volume, and environmental feature vectors from meteorological and environmental monitoring data. The visual feature maps, geometric and motion feature maps are concatenated and then input together with the environmental feature vectors into a multi-head self-attention feature fusion module. The module dynamically learns the association weights of different modal features at different spatial locations and time nodes to generate a fusion feature map rich in multimodal context information.

[0062] Step S4, Decision-level Fusion and Event Recognition: Based on the fused feature map, four sub-task models are executed in parallel. Then, the preliminary recognition results output by each sub-task model are input into a trainable weight network for dynamic weighted fusion. After comprehensive logical judgment, the final confidence level of various traffic events is obtained. ;

[0063] The four sub-task models are: illegal driving behavior recognition sub-model, traffic operation event detection sub-model, dangerous driving behavior recognition sub-model, and vehicle interaction behavior recognition sub-model.

[0064] The illegal driving behavior recognition sub-model uses an improved LRCN network to identify illegal driving behaviors of vehicles;

[0065] The traffic operation event detection sub-model uses an LSTM-NN network to detect and classify traffic operation events; the dangerous driving behavior recognition sub-model uses a combination of CNN and LSTM to identify dangerous driving behaviors.

[0066] The vehicle interaction behavior recognition sub-model uses a spatiotemporal graph convolutional network to recognize vehicle interaction behavior;

[0067] In step S4, the preliminary recognition results output by each sub-task model are dynamically weighted and fused, and the final confidence level... The calculation formula is: ;

[0068] In the formula, The classification probability vector output by the sub-model for identifying illegal driving behavior; The classification probability vector output by the traffic operation event detection sub-model; The classification probability vector output by the dangerous driving behavior recognition sub-model; The classification probability vector output by the vehicle interaction behavior recognition sub-model;

[0069] , , , These are the fusion weight coefficients dynamically generated by the trainable weight network, corresponding to the four sub-models mentioned above.

[0070] The fusion weight coefficients satisfy the following normalization condition: .

[0071] The trainable weight network is dynamically generated. to The process is as follows:

[0072] The environmental feature vector generated in step S3 with by , , , Cascaded probability vectors formed by sequential concatenation The vectors are concatenated to form a joint input vector. The calculation formula is: ;

[0073] The joint input vector The input is fed into a three-layer fully connected weighted network, and after being normalized by the Softmax function, the fused weight coefficients are output. , , , .

[0074] The improvements to the LRCN network in step S4 are as follows:

[0075] After the backbone network in the CNN feature extraction stage, a channel attention (SE) module is introduced to weight each channel of the feature vector, strengthening the feature channels related to violations. In the LSTM temporal modeling stage, a bidirectional LSTM is used instead of a unidirectional LSTM to capture the bidirectional temporal dependencies.

[0076] The spatiotemporal graph convolutional network in step S4 specifically performs the following recognition process:

[0077] Using the target vehicle and its neighboring vehicles as nodes, and the relative positions and speeds between vehicles as edges, a dynamic spatiotemporal graph is constructed. The node features are the fused feature vectors of each vehicle, and the edge features are the distance and speed difference codes between vehicles. After multi-layer spatiotemporal graph convolution, the target vehicle node features are classified, and the interaction behavior classification probability vector is output. .

[0078] Step S5, Event Analysis and Tiered Early Warning: For traffic events with confidence levels exceeding the threshold, analyze and assess their severity, scope of impact, and development trend by combining real-time traffic flow data, and generate differentiated early warning information for release according to preset tiered early warning rules.

[0079] By constructing a hierarchical fusion mechanism of "data-level-feature-level-decision-level" and introducing a multi-head self-attention feature fusion module composed of multiple Transformer encoder layers at the feature level, the accuracy and robustness of traffic event recognition in complex scenarios are significantly improved. This invention utilizes a multi-head self-attention mechanism at the feature level to dynamically learn the global correlation weights between visual features, geometric motion features, and environmental features. This enables the fused feature map to intelligently strengthen key modalities and suppress noisy modalities based on the current scene. For example, at night or in rainy or foggy weather, the model automatically strengthens the geometric motion features of radar point clouds while suppressing visual noise in video images. This completely solves the problems of insufficient single-modal perception capabilities and false alarms caused by simple fusion methods, achieving all-weather, high-precision traffic event perception.

[0080] Example 2

[0081] Reference Figures 2-6A traffic incident intelligent identification and early warning system based on multi-source sensing data fusion, according to Embodiment 1, includes:

[0082] The multi-source data access module is used to access and parse multi-source heterogeneous data from roadside video, radar-visual integrated machine, and meteorological monitoring equipment;

[0083] The data preprocessing and spatiotemporal synchronization module is used to clean and reduce noise from multi-source data, and to perform spatiotemporal alignment based on a unified timestamp and spatial coordinate system.

[0084] The hierarchical fusion and intelligent recognition engine has built-in data-level fusion units, feature-level fusion units, and decision-level fusion units;

[0085] The data-level fusion unit generates enhanced image data volumes and dense point cloud data volumes; the feature-level fusion unit integrates a deep learning feature extraction network and a multi-head self-attention feature fusion module; the decision-level fusion unit deploys multiple parallel traffic event detection and recognition sub-models, as well as a trainable weight network, for dynamically weighted fusion of the preliminary results of each sub-model.

[0086] The multiple parallel sub-models deployed in the decision-level fusion unit specifically include:

[0087] The improved LRCN network, with its CNN part followed by a channel attention SE module and its temporal part using a bidirectional LSTM, is used to identify illegal driving behaviors such as speeding, driving in the wrong direction, driving at low speed, and illegal parking.

[0088] LSTM-NN networks are used to detect and classify operational events such as traffic congestion, traffic jams, and traffic accidents.

[0089] A model combining CNN and LSTM is used to identify dangerous driving behaviors such as rapid acceleration, rapid deceleration, and sudden steering.

[0090] Spatiotemporal graph convolutional networks are used to identify vehicle interaction behaviors such as following and lane changing.

[0091] The deep learning feature extraction network in the feature-level fusion unit is specifically configured as follows:

[0092] The SlowFast network is used to extract spatiotemporal visual features from video images;

[0093] The PointNet++ network is used to extract the geometric and motion features of the point cloud;

[0094] A fully connected network is used to process meteorological data to generate environmental feature vectors.

[0095] The event analysis and early warning module is used to assess the impact of identified traffic events and generate and issue graded early warning information.

[0096] The event analysis and early warning module has a preset three-level early warning system: red, orange, and yellow.

[0097] A red alert is generated when a traffic accident or multi-vehicle collision hazard is identified.

[0098] An orange alert is generated when a dangerous single-vehicle driving incident or a severe traffic congestion event is identified.

[0099] A yellow alert is generated when a general violation is identified.

[0100] By making targeted structural improvements to the key sub-task model and introducing a spatiotemporal graph convolutional network to model multi-vehicle interaction behavior, this invention further expands the identification dimensions and early warning capabilities of proactive traffic safety management. In the violation identification sub-model, a channel attention (SE) module is introduced after the CNN backbone network to strengthen violation-related feature channels, and the unidirectional LSTM is improved to a bidirectional LSTM, fully capturing the bidirectional temporal dependencies of violation behaviors, thus improving the completeness of identification of behaviors such as illegal parking and driving against traffic. Simultaneously, a spatiotemporal graph convolutional network is introduced to construct a dynamic spatiotemporal graph with vehicles as nodes and relative motion relationships as edges, for the first time incorporating vehicle interaction behaviors such as following and lane changing into the fusion judgment system, achieving a leap from single-vehicle behavior recognition to road network-level interaction risk recognition.

[0101] By setting up a trainable weighted network at the decision level, and replacing the traditional fixed rules or average voting mechanism with adaptive dynamic weighted fusion, intelligent integrated decision-making with multiple models for dynamic environments is achieved. The weighted network in this invention takes environmental feature vectors as conditional input, and after passing through fully connected layers and Softmax normalization, dynamically outputs the fusion weight coefficients of each sub-model. Through supervised training with historical data, this network can learn to adaptively assign trust values ​​to each sub-model under different illumination, weather, and road conditions (e.g., placing more trust in the visual violation recognition model during the day, and more trust in the radar-based dangerous driving and trajectory interaction models at night). This overcomes the shortcomings of fixed fusion strategies that cannot adapt to environmental changes, significantly improving the comprehensive recognition confidence of multiple types of traffic events.

[0102] Example 3

[0103] Reference Figures 1-6 In the demonstration section, the following sensing devices were deployed according to the "air-space-ground" collaborative deployment plan:

[0104] Roadside fixed monitoring layer: A set of integrated radar-visual units and high-definition intelligent cameras are deployed every 200 meters to achieve seamless coverage of the entire road section. Among them, the integrated radar-visual unit operates at a frequency of 77GHz and can output structured point cloud data containing target position, speed, and acceleration, with a detection range of 250 meters; the high-definition camera has a resolution of 1920×1080 and a frame rate of 25fps.

[0105] Meteorological environment perception layer: Miniature weather stations are deployed at key nodes along the road to collect environmental parameters in real time in four dimensions: temperature, humidity, visibility, and road surface slipperiness, at a frequency of 1Hz.

[0106] Clock synchronization and spatial reference: All sensing devices achieve microsecond-level clock synchronization through the NTP protocol and GPS / BeiDou dual-mode timing module; the spatial coordinate system adopts the CGCS2000 national geodetic coordinate system and is mapped to the road network topology of the high-precision map.

[0107] Step S1: Acquisition and spatiotemporal alignment of multi-source sensing data. The multi-source data access module receives video, point cloud and meteorological data. The data preprocessing module performs Gaussian filtering and point cloud SOR filtering. Microsecond-level synchronization is achieved through NTP and GPS time synchronization. The point cloud is projected onto the image coordinate system using calibration extrinsic parameters to complete spatiotemporal alignment.

[0108] The system's multi-source data access module receives three heterogeneous data streams in real time from roadside high-definition cameras, radar-visual integrated units, and micro weather stations via gigabit fiber optic networks and 5G mobile communication networks. The data preprocessing and spatiotemporal synchronization modules perform the following operations respectively:

[0109] Data cleaning: Gaussian filtering is applied to the video stream for noise reduction, with a Gaussian kernel size of 3×3 and a standard deviation σ=1.0; point cloud data is filtered based on statistical outlier removal, with the number of neighboring points k=50 and the standard deviation multiple threshold set to 2.0, to remove abnormal reflection points and noise points.

[0110] Spatiotemporal alignment: Extract the UTC timestamps of all data, and align the radar point cloud data and meteorological data using linear interpolation, based on the video frame timestamps (40ms interval per frame). Spatially, transform the radar point cloud coordinates to the image pixel coordinate system using a calibrated extrinsic parameter matrix (rotation matrix R and translation vector T), achieving pixel-by-pixel spatial registration between the point cloud and the image, ultimately forming a standard sensing data stream that is temporally synchronized and spatially aligned.

[0111] Step S2: Data-level fusion. The data-level fusion unit uses the Zero-DCE++ network to enhance the image at night and AOD-Net to remove fog in foggy weather. Point cloud frames are accumulated in a 0.5-second window, and dense point clouds are generated using the ICP algorithm.

[0112] The data-level fusion unit of the layered fusion and intelligent recognition engine performs the following operations:

[0113] Image enhancement: For nighttime or low-light scenes, the Zero-DCE++ low-light enhancement network is used to adaptively enhance the brightness and restore the color of video images at the pixel level. The network input is a single frame of low-light image, and the output is the enhanced normal brightness image. For foggy scenes, the AOD-Net defogging network is used for sharpening. The network input is a foggy image, and the output is a clear image after defogging.

[0114] Point cloud densification: The sparse point cloud output from the Rayvision all-in-one camera (typically containing 200 to 500 effective points per frame) is accumulated over a 0.5-second time window, involving approximately 12 frames of point cloud data. The ICP algorithm is used to register and stitch the continuous frame point clouds. The maximum number of iterations for the ICP algorithm is set to 100, and the convergence threshold is a root mean square error of less than 1 × 10⁻⁶. -6 After densification processing, a dense point cloud data volume with 2,000 to 5,000 effective points per frame is generated, which more completely restores the vehicle's outline and motion trajectory.

[0115] Step S3: Feature-level fusion, feature-level fusion unit extracts features:

[0116] Visual features, SlowFast network outputs visual feature maps ;

[0117] Geometric features, PointNet++ outputs vehicle-level geometric motion feature maps ;

[0118] Environmental characteristics and meteorological data are output through a three-layer fully connected coding network. .

[0119] Will and After splicing and dimensional reduction, with The token sequence is fed into a 4-layer Transformer encoder to generate a fused feature map. .

[0120] The feature-level fusion unit employs a parallel heterogeneous network for multimodal feature extraction, and then achieves deep fusion through a multi-head self-attention mechanism. Specific implementation details are as follows:

[0121] Multimodal feature extraction includes:

[0122] Visual Feature Extraction: A SlowFast network is used to extract spatiotemporal visual feature maps from the enhanced video image data. The SlowFast network is configured as follows: the slow path input frame rate is 8fps, using a 3DResNet-50 backbone network; the fast path input frame rate is 25fps, using a lightweight 3D convolutional backbone network; the two paths are fused laterally at Stage 3, Stage 4, and Stage 5 of the network.

[0123] Final output visual feature map The dimensions are [8×56×56×512], where T=8 represents the temporal features extracted from 8 consecutive frames, H=56 and W=56 are the height and width of the feature map, respectively, and C... v =512 represents the number of feature channels.

[0124] Geometric and Motion Feature Extraction: PointNet++ network is used to extract the geometric shape and precise motion features of vehicles from dense point cloud data. PointNet++ employs a multi-scale grouping strategy, using multi-scale sphere queries with radii of 0.5m, 1.0m, and 2.0m for hierarchical point set abstraction, with 16, 32, and 64 sampling points at each scale, respectively. The network outputs pointwise geometric features of dimension [N×256], where N is the number of points per frame for the target vehicle. The pointwise features are aggregated into a single-target global feature vector of dimension [1×256] using a symmetric function (MaxPooling), and then concatenated with the global features of 8 consecutive frames for that target to form a vehicle-level spatiotemporal geometric feature map of dimension [8×256]. .

[0125] Environmental feature extraction: Temperature (T) collected by the weather station emp (Unit: °C) and humidity (H) um (Unit: %), Visibility (V) is (Unit: m) Road surface wet / slippery condition (S) lip Four numerical parameters (dimensionless, ranging from 0 to 1) are input into a three-layer fully connected encoding network after Min-Max normalization. The network structure is as follows: input layer (4 neurons) → first hidden layer (16 neurons, ReLU activation function) → output layer (32 neurons), generating an environmental feature vector of dimension [1×32]. .

[0126] Multi-head self-attention fusion, which integrates visual feature maps (Dimensions [8×56×56×512]) and geometric feature map (Dimension [8×256]) Concatenate along the channel dimension to obtain a concatenated feature map of dimension [8×56×56×768], then reduce the dimension to [8×56×56×256] using a 1×1 convolution. Flatten the spatial dimension of this feature map to [8×3136×256], and copy and expand it to the same dimension as the environment feature vector. The sequences are spliced ​​together to form a serialized feature token sequence of length L = 8 × 3136 + 1.

[0127] The token sequence is input to a multi-head self-attention feature fusion module consisting of four stacked Transformer encoder layers. Each encoder layer is configured as follows:

[0128]

[0129] After layer-by-layer abstraction and interaction through four encoder layers, the model outputs a fused feature map rich in multimodal contextual information. .

[0130] Step S4: Decision-level fusion and event recognition.

[0131] The decision-level fusion unit executes four sub-tasks in parallel:

[0132] The improved LRCN network consists of a ResNet-50 followed by an SE attention module, and then a bidirectional LSTM is used to model the temporal sequence, outputting... .

[0133] LSTM-NN network, output And the scope of impact.

[0134] CNN-LSTM model, MobileNetV2+LSTM, output .

[0135] Spatiotemporal graph convolutional network is used to construct a dynamic spatiotemporal graph of vehicles and output the result. .

[0136] Concatenate the above four probability vectors as follows ,and According to the formula After concatenation, the weights are input into a three-layer fully connected weighted network, and Softmax normalization is used to obtain dynamic weights. to The normalization condition is met. The final confidence level is calculated using the formula... calculate.

[0137] Decision-level fusion units based on fusion feature maps Four sub-task recognition models are deployed in parallel and adaptively fused through a trainable weight network.

[0138] The sub-task model definition includes a sub-model for recognizing illegal driving behavior, a sub-model for detecting traffic operation events, a sub-model for recognizing dangerous driving behavior, and a sub-model for recognizing vehicle interaction behavior.

[0139] The sub-model for identifying illegal driving behavior, as described in this embodiment, improves upon the standard LRCN in two ways: Channel Attention (SE) module: After the ResNet-50 backbone network in the CNN feature extraction stage, a channel attention (SE) module is introduced. This module first performs global average pooling on the 2048-dimensional feature vector to obtain a 1×1×2048 channel descriptor. Then, after passing through two fully connected layers (compression ratio r=16, i.e., 2048→128→2048) and sigmoid activation, the weight coefficients for each channel are obtained. These weights are then multiplied channel-by-channel with the original feature vector to strengthen the feature channels related to illegal behaviors (such as abnormal vehicle position or abnormal driving direction).

[0140] Bidirectional LSTM temporal modeling: The feature vector sequence with channel attention is input into a bidirectional LSTM. The bidirectional LSTM consists of a forward LSTM and a backward LSTM, each with a hidden layer dimension of 128. The forward and backward hidden states are concatenated into a 256-dimensional vector. The network outputs classification probability vectors for each violation category (no violation, speeding, wrong-way driving, slow driving, illegal parking). It has 5 dimensions, and the sum of its components is 1.

[0141] The traffic event detection sub-model employs a three-layer stacked LSTM structure, with each hidden layer having a dimension of 128, followed by a fully connected classification layer and a parallel regression layer. The input is the fused features of consecutive frames, and the classification layer outputs a traffic event state classification probability vector. The dimension is 4, corresponding to normal traffic, traffic congestion, traffic jams and traffic accidents respectively, and the sum of each component is 1; the regression layer outputs the predicted influence radius (unit: meters) and the expected duration (unit: minutes).

[0142] The dangerous driving behavior recognition sub-model first uses the lightweight MobileNetV2 to compress the spatial features of the keyframe fusion features, with the width multiplier of MobileNetV2 set to 1.0 and the output feature dimension to 1280. Then, a single-layer LSTM is used to perform temporal modeling on the compressed features of consecutive frames, with the LSTM hidden layer dimension to be 128. Finally, a fully connected classification layer is used to output the dangerous driving behavior classification probability vector. The dimension is 4, corresponding to normal driving, rapid acceleration / deceleration, sudden turning, and other dangerous driving, with the sum of each component being 1.

[0143] The vehicle interaction behavior recognition sub-model uses the target vehicle and its eight neighboring vehicles as nodes (a total of nine nodes), and the relative positions and speeds between vehicles as edges to construct a dynamic spatiotemporal graph. The graph node features are the fused feature vectors of each vehicle (256 dimensions), and the edge features are the distance (in meters) and speed difference (in m / s) between vehicles (16 dimensions). After three layers of spatiotemporal graph convolution (spatial graph kernel size = 3×3, temporal convolution kernel size = 9), a fully connected classification is performed on the target vehicle node features, outputting an interaction behavior classification probability vector. The dimension is 4, corresponding to free driving, following, lane changing, and overtaking, and the sum of each component is 1.

[0144] Dynamically weighted fusion is used to initially output four probability vectors from the four sub-task models, which are 5-dimensional, 4-dimensional, 4-dimensional, and 4-dimensional respectively. These vectors are then concatenated sequentially to form a 17-dimensional cascaded probability vector. .

[0145] The trainable weighted network is designed as a three-layer fully connected network, with the following specific structure:

[0146]

[0147] The inference phase workflow: The environmental feature vector generated in step S3... Cascaded probability vectors of 32-dimensional and 17-dimensional dimensions The vectors are concatenated to form a 49-dimensional joint input vector. The calculation formula is as follows: In the formula, ⊕ represents the vector concatenation operation.

[0148] Joint input vector The weights are fed into a weighted network, normalized by the Softmax function, and then output as four dynamic weight coefficients. , , , .

[0149] The mathematical expression for the Softmax function is as follows: , i=1,2,3,4; where z i This represents the original output value of the i-th node in the output layer of the fully connected network. Due to the properties of the Softmax function, the output weight coefficients automatically satisfy the normalization condition:

[0150] .

[0151] Final overall confidence level It is calculated using the following weighted summation formula:

[0152]

[0153] Pick The category with the highest probability is taken as the final identification result, and its probability value is the comprehensive confidence level.

[0154] Training method for the weighted network: A dataset containing 10,000 training samples is constructed, with each sample containing feature data under different environmental conditions (illuminance, weather, road conditions). Each environmental condition type has at least 500 samples covering it. The labels for each training sample are generated by three traffic management experts who independently label the optimal fusion weights for each sub-model according to the corresponding environment. The average of the three experts' labels is used as the label. The loss function is mean squared error loss, the optimizer is Adam, and the initial learning rate is set to 1×10⁻⁶. -4 The training batch size is 32, the training rounds are 100, and an early stopping strategy is adopted, which stops training when the validation set loss no longer decreases for 10 consecutive rounds.

[0155] Step S5: Event Analysis and Tiered Early Warning. The event analysis and early warning module assesses the impact of traffic events with a confidence level exceeding a threshold (set at 85%). If identified as a "traffic accident," the module combines the predicted impact range output from the LSTM-NN network regression head to delineate the affected lanes and estimated dissipation time on a high-precision map. If identified as "dangerous driving," the module locks the target vehicle ID, predicts its trajectory for the next 3 seconds, and assesses the risk of collision with surrounding vehicles.

[0156] The tiered early warning rules are preset to three levels: red, orange, and yellow.

[0157] Red Alert: Triggered when a traffic accident or multi-vehicle collision hazard is identified with a confidence level exceeding 90%, the alert information is directly pushed to the traffic police command center's large screen and the terminals of relevant law enforcement personnel.

[0158] Orange alert: Triggered when dangerous driving behavior of a single vehicle is identified (such as sudden acceleration, sudden deceleration, sudden turning) or severe traffic congestion (congestion index exceeding 0.7) with a confidence level exceeding 85%, the alert information is released to the guidance screens of surrounding road sections.

[0159] Yellow alert: Triggered when a general violation (such as illegal parking or slow driving) is identified and the confidence level exceeds 85%, a warning message is generated and video evidence is saved.

[0160] All warning information is highlighted on the digital twin management platform, automatically popping up the associated video surveillance footage and vehicle identity information, and storing an evidence chain containing 15-second video clips before and after the event.

[0161] Example 4

[0162] The system was deployed on key roads surrounding the "Lantern World" scenic area in the eastern new city of Zigong to verify its practical application under peak tourist flow scenarios during holidays. During holidays such as the Spring Festival and Lantern Festival, the "Lantern World" scenic area sees over 100,000 visitors per day, leading to a surge in traffic on surrounding roads. Many vehicles cruise slowly in search of parking spaces, easily causing severe traffic congestion and accidents. A demonstration area of ​​approximately 3.6 kilometers was selected on key roads surrounding the scenic area, where 18 integrated radar-visual systems, 18 high-definition intelligent cameras, and 3 mini weather stations were deployed.

[0163] The test period was from 18:30 to 21:30 on the Lantern Festival (February 24, 2024), with a total of approximately 12,000 vehicles monitored. The weather that day was cloudy turning overcast, with temperatures ranging from 9°C to 14°C, visibility of approximately 2.5 kilometers, and dry road surfaces.

[0164] During the 3-hour test period, the system's identification and early warning results for various traffic incidents are as follows:

[0165] illegal parking 326 cases 316 cases 96.9% 96.8% yellow driving at low speed 215 cases 204 cases 94.9% 94.2% yellow sudden deceleration / sudden turn 53 cases 50 94.3% 93.4% orange color Traffic congestion events 12 times 12 times 100% 97.1% orange color Potential collision risk 7 cases 7 cases 100% 91.3% red Total / Average 613 cases / time 589 or more per time 96.1% 94.3% —

[0166] In a typical case at 20:15, the system identified a potential three-vehicle chain collision risk: a sedan, searching for a parking space near the entrance of a scenic area, suddenly decelerated from 35 km / h to 8 km / h within 1.8 seconds, prompting two other vehicles behind it to brake suddenly and change lanes to avoid it. The system captured the interaction behavior of the three vehicles using a spatiotemporal graph convolutional network. Under low-light conditions at night (approximately 8.5 lux), the trainable weighted network automatically adjusted the weight allocation to... , , , After weighted fusion, the output of "multi-vehicle collision danger" has a confidence level of 94.7%, triggering a red alert. It is pushed to the traffic police command center 15.2 seconds in advance, winning valuable time for on-site handling and effectively preventing actual collision accidents from occurring.

[0167] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A traffic event intelligent identification and early warning method based on multi-source perception data fusion, characterized in that, Includes the following steps: Step S1: Acquisition and Spatiotemporal Alignment of Multi-Source Sensing Data: Through the deployed three-dimensional monitoring network, multi-source heterogeneous sensing data, including roadside high-definition video images, point cloud data from integrated radar and vision systems, and meteorological and environmental monitoring data, are acquired in real time. After cleaning each data, spatiotemporal alignment and registration are performed based on a unified timestamp and spatial coordinate system to form a standardized spatiotemporal continuous sensing data stream. Step S2, Data-level fusion: Pixel-level enhancement is performed on the spatiotemporally aligned video images, and point cloud data from multiple frames is stitched together at the point cloud level to generate the fused enhanced image data volume and dense point cloud data volume. Step S3, Feature-level Fusion: A parallel deep learning feature extraction network is used to extract visual feature maps from the enhanced image data volume, geometric and motion feature maps from the dense point cloud data volume, and environmental feature vectors from meteorological and environmental monitoring data. The visual feature maps, geometric and motion feature maps are concatenated and then input together with the environmental feature vectors into a multi-head self-attention feature fusion module. The module dynamically learns the association weights of different modal features at different spatial locations and time nodes to generate a fusion feature map rich in multimodal context information. Step S4, Decision-level Fusion and Event Recognition: Based on the fused feature map, four sub-task models are executed in parallel. Then, the preliminary recognition results output by each sub-task model are input into a trainable weight network for dynamic weighted fusion. After comprehensive logical judgment, the final confidence level of various traffic events is obtained. ; Step S5, Event Analysis and Tiered Early Warning: For traffic events with confidence levels exceeding the threshold, analyze and assess their severity, scope of impact, and development trend by combining real-time traffic flow data, and generate differentiated early warning information for release according to preset tiered early warning rules. 2.The traffic event intelligent identification and early warning method based on multi-source perception data fusion according to claim 1, characterized in that: The four sub-task models are: illegal driving behavior recognition sub-model, traffic operation event detection sub-model, dangerous driving behavior recognition sub-model, and vehicle interaction behavior recognition sub-model. The illegal driving behavior recognition sub-model uses an improved LRCN network to identify illegal driving behaviors of vehicles; The traffic operation event detection sub-model uses an LSTM-NN network to detect and classify traffic operation events; The dangerous driving behavior recognition sub-model uses a combination of CNN and LSTM to identify dangerous driving behaviors; The vehicle interaction behavior recognition sub-model uses a spatiotemporal graph convolutional network to recognize vehicle interaction behavior; The step S4 dynamically weights and fuses the preliminary recognition results output by each sub-task model, and the final confidence The calculation formula is: ; In the formula, The classification probability vector output by the sub-model for identifying illegal driving behavior; The classification probability vector output by the traffic operation event detection sub-model; The classification probability vector output by the dangerous driving behavior recognition sub-model; The classification probability vector output by the vehicle interaction behavior recognition sub-model; , , , are respectively dynamically generated fusion weight coefficients corresponding to the four sub-task models by the trainable weight network. 3.The traffic incident intelligent identification and early warning method based on multi-source perception data fusion according to claim 2, characterized in that: The fusion weight coefficients satisfy the following normalization condition: .

4. The traffic event intelligent identification and early warning method based on multi-source perception data fusion according to claim 3, characterized in that: The trainable weight network dynamically generates to the process, specifically: The environmental feature vector generated in step S3 with by , , , Cascaded probability vectors formed by sequential concatenation The vectors are concatenated to form a joint input vector. The calculation formula is: ; The joint input vector The input is fed into a three-layer fully connected weighted network, and after being normalized by the Softmax function, the fused weight coefficients are output. , , , .

5. The intelligent identification and early warning method for traffic incidents based on multi-source sensing data fusion according to claim 4, characterized in that: The improvements to the LRCN network in step S4 are as follows: After the backbone network in the CNN feature extraction stage, a channel attention (SE) module is introduced to weight each channel of the feature vector, strengthening the feature channels related to violations. In the LSTM temporal modeling stage, a bidirectional LSTM is used instead of a unidirectional LSTM to capture the bidirectional temporal dependencies.

6. The traffic event intelligent identification and early warning method based on multi-source perception data fusion according to claim 5, characterized in that: The spatiotemporal graph convolutional network in step S4 specifically performs the following recognition process: Using the target vehicle and its neighboring vehicles as nodes, and the relative positions and speeds between vehicles as edges, a dynamic spatiotemporal graph is constructed. The node features are the fused feature vectors of each vehicle, and the edge features are the distance and speed difference codes between vehicles. After multi-layer spatiotemporal graph convolution, the target vehicle node features are classified, and the interaction behavior classification probability vector is output. . 7.The intelligent identification and early warning system based on multi-source perception data fusion traffic event intelligent identification and early warning method according to claim 6, characterized in that, include: The multi-source data access module is used to access and parse multi-source heterogeneous data from roadside video, radar-visual integrated machine, and meteorological monitoring equipment; The data preprocessing and spatiotemporal synchronization module is used to clean and reduce noise from multi-source data, and to perform spatiotemporal alignment based on a unified timestamp and spatial coordinate system. The hierarchical fusion and intelligent recognition engine has built-in data-level fusion units, feature-level fusion units, and decision-level fusion units; The data-level fusion unit generates enhanced image data volumes and dense point cloud data volumes; the feature-level fusion unit integrates a deep learning feature extraction network and a multi-head self-attention feature fusion module; the decision-level fusion unit deploys multiple parallel traffic event detection and recognition sub-models, as well as a trainable weight network, for dynamically weighted fusion of the preliminary results of each sub-model. The event analysis and early warning module is used to assess the impact of identified traffic events and generate and issue graded early warning information. 8.The traffic incident intelligent identification and warning system based on multi-source perception data fusion of claim 7, characterized in that: The multiple parallel sub-models deployed in the decision-level fusion unit specifically include: The improved LRCN network, with its CNN part followed by a channel attention SE module and its temporal part using a bidirectional LSTM, is used to identify illegal driving behaviors such as speeding, driving in the wrong direction, driving at low speed, and illegal parking. LSTM-NN networks are used to detect and classify traffic congestion, traffic jams, and traffic accident events. A model combining CNN and LSTM is used to identify dangerous driving behaviors such as rapid acceleration, rapid deceleration, and sudden steering. Spatiotemporal graph convolutional networks are used to identify vehicle interaction behaviors such as following and lane changing. 9.The traffic incident intelligent identification and warning system based on multi-source perception data fusion of claim 7, characterized in that: The deep learning feature extraction network in the feature-level fusion unit is specifically configured as follows: The SlowFast network is used to extract spatiotemporal visual features from video images; The PointNet++ network is used to extract the geometric and motion features of the point cloud; A fully connected network is used to process meteorological data to generate environmental feature vectors. 10.The traffic incident intelligent identification and warning system based on multi-source perception data fusion of claim 7, wherein: The event analysis and early warning module has a preset three-level early warning system: red, orange, and yellow. A red alert is generated when a traffic accident or multi-vehicle collision hazard is identified. An orange alert is generated when a dangerous single-vehicle driving incident or a severe traffic congestion event is identified. A yellow alert is generated when a general violation is identified.

Images