Traffic event detection method based on data fusion and related device

Abstract

The application provides a traffic event detection method based on data fusion and related equipment, the method comprises the following steps: preprocessing and feature extraction are performed on collected multi-modal traffic data to obtain a standardized feature vector; a heterogeneous graph representation is constructed according to the standardized feature vector, graph convolution operation and attention weighting are performed on the heterogeneous graph representation to obtain a multi-modal fusion feature; and inference calculation is performed on the multi-modal fusion feature by using a preset traffic event detection model to obtain a traffic event classification result and an abnormal probability. By modeling the space-time dependency relationship between multi-modal data and using the attention mechanism to realize adaptive feature fusion, the accuracy and robustness of traffic anomaly event detection are significantly improved, and the method is suitable for real-time anomaly monitoring scenes in a highway traffic management system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a traffic incident detection method and related equipment based on data fusion. Background Technology

[0002] With the acceleration of urbanization, highway traffic management faces increasingly severe challenges. Timely detection of abnormal events such as traffic accidents and congestion is crucial for ensuring road safety and improving traffic efficiency. Traditional traffic anomaly detection methods mainly rely on a single data source, such as surveillance camera or sensor data, and identify anomalies through statistical models or rule matching. However, traffic events themselves have complex spatiotemporal characteristics and multimodal manifestations, and a single data source often cannot comprehensively capture the complete information of abnormal events.

[0003] In existing technologies, although some solutions attempt to fuse data from multiple sensors, they mostly use simple feature stitching or weighted averaging for data fusion. This fusion method ignores the inherent correlation and spatiotemporal dependence between different modal data, resulting in the failure to fully utilize the complementary information of heterogeneous data. In complex traffic scenarios, the detection accuracy is not high, making it difficult to meet the needs of real-time and reliable traffic anomaly detection. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problems of low detection accuracy caused by insufficient multimodal data fusion and failure to effectively capture the spatiotemporal dependencies between heterogeneous data in existing traffic anomaly detection methods. This invention provides a traffic incident detection method based on data fusion, wherein the adaptive control method includes: The collected multimodal traffic data is preprocessed and features are extracted to obtain standardized feature vectors; A heterogeneous graph representation is constructed based on the standardized feature vectors. Graph convolution and attention weighting are then performed on the heterogeneous graph representation to obtain multimodal fusion features. Based on the multimodal fusion features, inference calculations are performed using a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

[0005] The present invention also provides a traffic incident detection device based on data fusion, the traffic incident detection device based on data fusion comprising: The data preprocessing module is used to preprocess and extract features from the collected multimodal traffic data to obtain standardized feature vectors; The heterogeneous graph fusion module is used to construct a heterogeneous graph representation based on the standardized feature vector, and to perform graph convolution and attention weighting on the heterogeneous graph representation to obtain multimodal fusion features; The event reasoning module is used to perform reasoning calculations based on the multimodal fusion features and a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

[0006] The present invention also provides a traffic incident detection device based on data fusion, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; the at least one processor invokes the instructions in the memory to cause the traffic incident detection device based on data fusion to perform the steps of the traffic incident detection method based on data fusion described above.

[0007] The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described traffic incident detection method based on data fusion.

[0008] The aforementioned traffic incident detection method and related equipment based on data fusion preprocess and extract features from collected multimodal traffic data to obtain standardized feature vectors. A heterogeneous graph representation is constructed based on these standardized feature vectors, and graph convolution and attention weighting are performed on this representation to obtain multimodal fusion features. Based on these multimodal fusion features, inference calculations are performed using a pre-defined traffic incident detection model to obtain traffic incident classification results and anomaly probabilities. This invention significantly improves the accuracy and robustness of traffic anomaly detection by modeling the spatiotemporal dependencies between multimodal data and utilizing an attention mechanism to achieve adaptive feature fusion. It is suitable for real-time anomaly monitoring scenarios in highway traffic management systems.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the first embodiment of the traffic incident detection method based on data fusion in this invention. Figure 2 This is a schematic diagram of a second embodiment of the traffic incident detection method based on data fusion in this invention. Figure 3 This is a schematic diagram of one embodiment of the traffic incident detection device based on data fusion in this invention. Figure 4 This is a schematic diagram of one embodiment of the traffic incident detection device based on data fusion in this invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0014] To facilitate understanding of this embodiment, a traffic incident detection method based on data fusion disclosed in this invention will first be described in detail. For example... Figure 1 As shown, this method includes the following steps: 101. Preprocess and extract features from the collected multimodal traffic data to obtain standardized feature vectors; In this embodiment, the preprocessing and feature extraction of the collected multimodal traffic data to obtain a standardized feature vector includes: denoising the collected image data, point cloud data, sound data, and temperature data respectively to obtain denoised modal data; performing spatiotemporal alignment on the denoised modal data by matching data collected by different sensors in the same time window and spatial location to obtain spatiotemporally aligned multimodal data; normalizing the spatiotemporally aligned multimodal data respectively to obtain normalized features for each modality; and concatenating the normalized features of each modality to obtain a standardized feature vector.

[0015] Specifically, for image data, traffic scenes can be captured in real time by surveillance cameras along highways. These images are subject to noise due to changes in lighting and weather conditions during acquisition. Gaussian filtering or median filtering can be used to denoise the images, and convolution operations are performed on each pixel and its neighborhood to remove random noise points. Point cloud data is collected by roadside LiDAR sensors. Filtering can be performed based on the distance attributes and reflection intensity of the point cloud, removing outliers with abnormal distances or reflection intensities to obtain high-quality denoised point cloud data. Sound data is collected by sound sensors on both sides of the road, capturing engine noise, braking sounds, and abnormal collision sounds from vehicles. However, background noise such as wind and rain can also be mixed in. Bandpass filtering can be used to retain sound signals within a specific frequency range, filtering out low-frequency and high-frequency noise components. Then, Fourier transform is performed on the filtered sound signal to convert the time-domain signal to the frequency domain, extracting the spectral characteristics of the sound. Temperature data is collected by temperature sensors buried in the road surface, and fluctuates due to factors such as climate and sunlight. Data can be smoothed using moving averages or exponential smoothing to remove short-term random fluctuations and retain the overall trend of temperature changes.

[0016] After obtaining the denoised modal data, differences in temporal and spatial references exist between the data collected by different sensors. For example, the sampling frequencies of cameras and LiDAR differ, leading to variations in temporal resolution. Therefore, spatiotemporal alignment of this data is necessary. This can be achieved by setting a unified time window and matching the modal data collected within that window. For spatial alignment, the spatial coordinate systems of the collected data differ due to variations in sensor installation location and orientation. A coordinate transformation matrix can be calculated based on the sensor's installation location and orientation to transform point cloud data, image data, etc., into a unified road coordinate system, resulting in spatiotemporally aligned multimodal data.

[0017] For spatiotemporally aligned data, due to significant differences in the dimensions and numerical ranges of each modality, normalization is necessary. A min-max normalization method can be used to standardize the numerical ranges of each modality to the same interval. For each pixel value in the image data, normalization is calculated based on its relationship to the minimum and maximum values ​​in the image. Point cloud data, sound spectrum data, and temperature data are processed using a similar normalization method to obtain normalized features for each modality. These normalized features are then concatenated in a predetermined order to form a unified feature vector. This feature vector contains multimodal information from images, point clouds, sound, and temperature; it is the standardized feature vector.

[0018] 102. Construct a heterogeneous graph representation based on the standardized feature vector, and perform graph convolution and attention weighting on the heterogeneous graph representation to obtain multimodal fusion features; In this embodiment, when constructing the heterogeneous graph representation, the image features, point cloud features, sound features, and temperature features in the standardized feature vector can be mapped to different types of nodes in the heterogeneous graph. For example, the nodes corresponding to the image features are labeled as first-type nodes, the nodes corresponding to the point cloud features are labeled as second-type nodes, the nodes corresponding to the sound features are labeled as third-type nodes, and the nodes corresponding to the temperature features are labeled as fourth-type nodes, thereby obtaining a heterogeneous node set containing multiple node types.

[0019] After obtaining the heterogeneous node set, edge connections between nodes need to be constructed based on the spatiotemporal correlations between modalities. For any two nodes in the heterogeneous node set, the temporal and spatial distances of their corresponding data can be calculated. The temporal distance can be calculated by the difference in the timestamps of the data acquisition between the two nodes, and the spatial distance can be calculated based on the coordinates of the sensor's installation location. Then, the existence of a correlation between the two nodes is determined based on the calculated temporal and spatial distances. When both the temporal and spatial distances are less than a preset threshold, a correlation is considered to exist between the two nodes, and the corresponding positions are marked as correlated in the node correlation matrix. Based on the node correlation matrix, edge connections are constructed for the correlated node pairs, and edge weights are calculated based on the correlation strength. The correlation strength can be determined based on the reciprocals of the temporal and spatial distances; the closer the distance, the stronger the correlation, and the larger the corresponding edge weight, resulting in a heterogeneous graph representation containing the node set, edge set, and edge weights.

[0020] After obtaining the heterogeneous graph representation, graph convolution can be performed on each node in the heterogeneous graph. Based on the edge set in the heterogeneous graph representation, the set of neighboring nodes for each node can be determined. The edge weights in the neighboring node set are normalized so that the sum of the weights of all neighboring edges for each node equals one. Then, the features of each node in the neighboring node set are weighted and summed according to the normalized edge weights. The result of the weighted sum is concatenated with the current node's own features to obtain aggregated features. Linear transformation and nonlinear activation operations are performed on the aggregated features. The linear transformation is implemented using a weight matrix and a bias term, while the nonlinear activation uses the ReLU activation function to obtain the node hidden state representation of the current layer. The node hidden state representation of the current layer is used as the input to the next layer, and the above aggregation and transformation operations are repeated. After passing through a preset number of network layers, the node representation after graph convolution is obtained.

[0021] After obtaining the node representations after graph convolution, it is necessary to calculate the attention score for each modality node. For each modality node, an attention network can be used to process its node representation to obtain the attention score for that node. The attention network can contain fully connected layers and activation functions, mapping the node representation to the attention score through learned parameters. The attention scores of all modality nodes are then normalized using a softmax function, ensuring that the sum of the attention weights for all modality nodes equals one. Based on the normalized attention weights, the representations of each modality node are weighted and fused. This involves multiplying each modality node representation by its corresponding attention weight and then summing the results to obtain a multimodal fusion feature. This fusion feature integrates multimodal information such as image, point cloud, sound, and temperature, and is adaptively weighted according to the importance of each modality.

[0022] 103. Based on the multimodal fusion features, inference calculations are performed using a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

[0023] In this embodiment, the step of obtaining traffic event classification results and anomaly probabilities by performing inference calculations based on the multimodal fusion features and a preset traffic event detection model includes: inputting the multimodal fusion features into the input layer of the preset traffic event detection model, performing forward propagation calculations through a multi-layer neural network to obtain feature representations of each traffic event category; performing a softmax operation on the feature representations of each traffic event category to obtain the probability distribution of each event category; selecting the category with the highest probability value as the traffic event classification result based on the probability distribution, and outputting the corresponding probability value as the anomaly probability.

[0024] Specifically, the traffic incident detection model can employ a multi-layer fully connected neural network structure. The multimodal fusion features obtained above are input into the input layer of the detection model. These fusion features contain multimodal information such as image, point cloud, sound, and temperature after heterogeneous graph fusion. Upon receiving the fusion features, the input layer passes them to the first hidden layer. In the hidden layer, the fusion features are multiplied by the layer's weight matrix, a bias term is added, and then a nonlinear transformation is performed using an activation function. The activation function can be the ReLU function, which introduces nonlinearity, allowing the network to learn complex mapping relationships. After processing by the first hidden layer, the output is passed to the next hidden layer, repeating the matrix operations, bias addition, and activation function processing. Through layer-by-layer processing across multiple hidden layers, the network can progressively extract and transform features, mapping the fusion features to a feature space more suitable for the classification task.

[0025] After forward propagation through multiple hidden layers, the output of the last hidden layer is passed to the output layer. The number of neurons in the output layer corresponds to the number of traffic event categories. For example, if the traffic event categories include normal traffic, traffic congestion, traffic accidents, and weather-related anomalies, then the output layer contains four neurons. The output layer performs a linear transformation on the output of the last hidden layer to obtain the feature representations for each traffic event category. These feature representations are unnormalized raw scores.

[0026] Then, a softmax operation is performed on the feature representations of each traffic event category. The softmax function transforms the original scores into a probability distribution, ensuring that the sum of the probabilities of all categories equals one. Through the softmax operation, the probability values ​​of the current input data belonging to each traffic event category can be obtained, resulting in the probability distribution of each event category. This probability distribution reflects the model's confidence that the current traffic scenario belongs to different event categories.

[0027] After obtaining the probability distribution of each event category, the traffic event classification result can be determined based on the probability distribution. The probability values ​​of each category in the probability distribution are traversed, and the category with the highest probability value is identified as the traffic event classification result. For example, if the probability value of the traffic congestion category is the highest, then the current traffic scene is classified as a traffic congestion event. Simultaneously, this highest probability value is output as the anomaly probability, which represents the model's confidence level in this classification result. If the anomaly probability is high, it means that the model believes that this type of traffic anomaly event does exist in the current scene, and corresponding alarm or notification mechanisms can be triggered. If the anomaly probability is low, although the model provides a classification result, the confidence level is not high, and further judgment may be needed by combining other information. Through this reasoning and calculation process, the traffic event detection model can accurately identify and classify various anomaly events in highway traffic scenes based on multimodal fusion features, providing timely anomaly event information to traffic management departments and ensuring road traffic safety.

[0028] Furthermore, after performing a softmax operation on the feature representations of each traffic event category to obtain the probability distribution of each event category, the method further includes: calculating the Euclidean distance between the multimodal fusion feature and the feature center of each traffic event category; normalizing the Euclidean distance to obtain the feature deviation; calculating a confidence adjustment coefficient based on the feature deviation using a preset confidence mapping function; decreasing the confidence adjustment coefficient when the feature deviation exceeds a preset threshold and increasing the confidence adjustment coefficient when the feature deviation is below the preset threshold; scaling the probability values ​​in the probability distribution of each event category based on the confidence adjustment coefficient; and re-normalizing the scaled probability values ​​to obtain the corrected probability distribution.

[0029] Specifically, after obtaining the probability distribution of each event category, a correction process can be performed on the probability distribution to further improve the reliability of the classification results. For the currently input multimodal fusion feature, the Euclidean distance between the fusion feature and the feature centers of each traffic event category can be calculated. The feature centers of each traffic event category can be obtained during the model training phase by clustering or statistically analyzing the features of the training samples. Each category corresponds to a feature center, which represents the typical position of that category in the feature space. When calculating the Euclidean distance between the multimodal fusion feature and the feature center of a certain category, the fusion feature and the category center can be regarded as two points in the feature space. The straight-line distance between these two points is calculated. The smaller the distance, the closer the fusion feature is to the typical features of that category.

[0030] After calculating the Euclidean distances between the multimodal fusion features and the feature centers of each traffic event category, the minimum and maximum values ​​of these distances can be identified. Then, each Euclidean distance is normalized to map the distance values ​​to a uniform range. Normalization can be achieved by subtracting the minimum distance value from each distance value and then dividing by the difference between the maximum and minimum distance values, yielding the feature deviation. The feature deviation reflects the degree to which the current fused feature deviates from the nearest category center. A smaller deviation indicates that the fused feature is relatively close to the feature center of a certain category, and the classification result is relatively reliable; a larger deviation indicates that the fused feature is in a relatively ambiguous position in the feature space, and the reliability of the classification result may be lower.

[0031] Based on the obtained feature deviation, a confidence adjustment coefficient can be calculated using a preset confidence mapping function. This confidence mapping function can be customized to meet specific needs. For example, it can be set so that when the feature deviation exceeds a certain preset threshold, the confidence adjustment coefficient is set to a value less than one, thus reducing the confidence in the probability distribution. Conversely, when the feature deviation is below the preset threshold, the confidence adjustment coefficient is set to a value greater than one, increasing the confidence in the probability distribution. More complex mapping functions can also be used to make the confidence adjustment coefficient change smoothly with variations in feature deviation.

[0032] After obtaining the confidence adjustment coefficient, the probability values ​​in the probability distribution of each event category can be scaled using this coefficient. For each category probability value in the probability distribution, it is multiplied by the confidence adjustment coefficient to obtain the scaled probability value. If the confidence adjustment coefficient is less than one, the probability values ​​of each category will be reduced, and the probability distribution will become flatter, indicating that the model is less certain about the classification of each category. If the confidence adjustment coefficient is greater than one, the probability values ​​of each category will be amplified, and the differences in the probability distribution will be more obvious, highlighting the model's bias towards a certain category.

[0033] Since the sum of the probability values ​​for each category may no longer equal one after scaling, the scaled probability values ​​need to be renormalized. This can be done by calculating the sum of all scaled probability values ​​and then dividing each scaled probability value by this sum to obtain the corrected probability distribution. The corrected probability distribution takes into account the positional information of the fused features in the feature space, more accurately reflecting the model's confidence in traffic event classification and improving the reliability and accuracy of traffic event detection.

[0034] In this embodiment, standardized feature vectors are obtained by preprocessing and extracting features from the collected multimodal traffic data. A heterogeneous graph representation is constructed based on these standardized feature vectors, and graph convolution and attention weighting are performed on this representation to obtain multimodal fusion features. Based on these multimodal fusion features, inference calculations are performed using a pre-defined traffic event detection model to obtain traffic event classification results and anomaly probabilities. This invention significantly improves the accuracy and robustness of traffic anomaly detection by modeling the spatiotemporal dependencies between multimodal data and utilizing an attention mechanism to achieve adaptive feature fusion. It is suitable for real-time anomaly monitoring scenarios in highway traffic management systems.

[0035] Please see Figure 2 Another embodiment of the traffic incident detection method based on data fusion in this application includes: 201. Preprocess and extract features from the collected multimodal traffic data to obtain standardized feature vectors; In this embodiment, step 201 is similar to step 101, and will not be described again here.

[0036] 202. Map each modal feature in the standardized feature vector to different types of nodes in the heterogeneous graph, and construct edge connections between nodes based on the spatiotemporal correlation between modalities to obtain the heterogeneous graph representation; In this embodiment, the step of mapping each modal feature in the standardized feature vector to different types of nodes in the heterogeneous graph, and constructing edge connections between nodes based on the spatiotemporal correlation between modalities to obtain a heterogeneous graph representation includes: mapping each modal feature in the standardized feature vector to different types of nodes in the heterogeneous graph to obtain a heterogeneous node set; calculating the temporal and spatial distances between nodes in the heterogeneous node set, determining whether there is a correlation between nodes based on the temporal and spatial distances, and obtaining a node correlation matrix; constructing edge connections for node pairs with correlation based on the node correlation matrix, calculating edge weights based on the correlation strength, and obtaining a heterogeneous graph representation containing a node set, an edge set, and edge weights.

[0037] Specifically, when mapping standardized feature vectors to heterogeneous graph nodes, it is necessary to distinguish them according to the type of each modality feature. For image features in the standardized feature vectors, they can be mapped to image type nodes. Each image type node contains the corresponding image feature information, the timestamp of image acquisition, and the camera location coordinates. Point cloud features are mapped to point cloud type nodes, which store the point cloud feature data and the time and spatial location information of the LiDAR acquisition data. Sound features are mapped to sound type nodes, containing the sound's spectral characteristics and the sound sensor's acquisition time and location. Temperature features are mapped to temperature type nodes, which record the temperature numerical characteristics and the temperature sensor's acquisition time and installation location. Through this mapping process, a heterogeneous node set containing image nodes, point cloud nodes, sound nodes, and temperature nodes is obtained. Different types of nodes represent different modalities of traffic data in the graph structure.

[0038] After obtaining the heterogeneous node set, it is necessary to calculate the temporal and spatial distances between the nodes to determine their relationships. For any two nodes in the heterogeneous node set, the temporal distance can be calculated based on the timestamp information stored in the nodes. For example, if an image node was acquired at a certain moment and a sound node was acquired at a slightly later moment, the difference between the two timestamps is the temporal distance between the two nodes. The temporal distance reflects the proximity of the acquisition times of the two modal data; the smaller the temporal distance, the closer the two data points are in time, potentially describing traffic conditions during the same time period. The spatial distance is calculated based on the sensor location coordinates stored in the nodes. For example, if a camera is located at one location on a highway and a temperature sensor is located at another location, the spatial distance between them can be calculated based on their coordinates. The spatial distance reflects the geographical proximity of the two sensors; data collected by sensors that are closer together often correspond to traffic conditions on the same or adjacent road segments.

[0039] Based on the calculated temporal and spatial distances, it can be determined whether a relationship exists between nodes. Temporal and spatial distance thresholds can be set; when the temporal distance between two nodes is less than both thresholds, a relationship is considered to exist between them. In the node relationship matrix, the positions of corresponding node pairs are marked as indicating a relationship. The rows and columns of the node relationship matrix correspond to the nodes in the heterogeneous node set, and the elements in the matrix indicate whether a corresponding node pair has a relationship.

[0040] Based on the node association matrix, edges can be constructed to connect node pairs that are associated. For node pairs marked as associated in the matrix, an edge is added to the heterogeneous graph to connect the two nodes. Then, edge weights need to be calculated based on the association strength, which can be determined based on temporal and spatial distances. The weighted sum of the reciprocals of the temporal and spatial distances can be used as a measure of association strength; the smaller the temporal and spatial distances, the larger the corresponding reciprocals, and the stronger the association. Each edge is assigned a corresponding edge weight based on the association strength; a larger edge weight indicates a stronger association between the two nodes. Through the above process, a heterogeneous graph representation containing a set of nodes, a set of edges, and edge weights is obtained. This heterogeneous graph structure can effectively express the spatiotemporal associations between multimodal traffic data.

[0041] 203. For each node in the heterogeneous graph representation, aggregate the feature information of its neighboring nodes, calculate the aggregated node features through weighted summation, and obtain the node representation after graph convolution; In this embodiment, the step of aggregating the feature information of its neighboring nodes for each node in the heterogeneous graph representation and obtaining the aggregated node features through weighted summation to obtain the node representation after graph convolution includes: determining the set of neighboring nodes for each node based on the set of edges in the heterogeneous graph representation; normalizing the edge weights in the set of neighboring nodes to obtain normalized edge weights; performing weighted summation on the node features in the set of neighboring nodes for each node based on the normalized edge weights; concatenating the weighted summation result with the current node's own features to obtain aggregated features; performing linear transformation and nonlinear activation operations on the aggregated features to obtain the node hidden state representation of the current layer; using the node hidden state representation of the current layer as the input of the next layer, repeating the above aggregation and transformation operations, and obtaining the node representation after graph convolution after a preset number of layers.

[0042] Specifically, 203. For each node in the heterogeneous graph representation, aggregate the feature information of its neighboring nodes, calculate the aggregated node features through weighted summation, and obtain the node representation after graph convolution; In this embodiment, the step of aggregating the feature information of its neighboring nodes for each node in the heterogeneous graph representation and obtaining the aggregated node features through weighted summation to obtain the node representation after graph convolution includes: determining the set of neighboring nodes for each node based on the set of edges in the heterogeneous graph representation; normalizing the edge weights in the set of neighboring nodes to obtain normalized edge weights; performing weighted summation on the node features in the set of neighboring nodes for each node based on the normalized edge weights; concatenating the weighted summation result with the current node's own features to obtain aggregated features; performing linear transformation and nonlinear activation operations on the aggregated features to obtain the node hidden state representation of the current layer; using the node hidden state representation of the current layer as the input of the next layer, repeating the above aggregation and transformation operations, and obtaining the node representation after graph convolution after a preset number of layers.

[0043] Specifically, when performing graph convolution operations on nodes in a heterogeneous graph, it is necessary to first determine the neighboring nodes of each node based on the edge set in the heterogeneous graph representation. For a given node in the heterogeneous graph, all edges in the edge set are traversed to find other nodes directly connected to that node. These directly connected nodes constitute the set of neighboring nodes of that node. For example, for an image node, it may be connected to surrounding point cloud nodes, sound nodes, and other image nodes by edges; these nodes connected by edges are the neighboring nodes of that image node. The number of neighboring nodes may vary for different nodes, depending on the node's connections to other nodes in the heterogeneous graph.

[0044] After determining the set of neighboring nodes for each node, the edge weights in the neighboring node set need to be normalized. For all the neighboring edge weights of a given node, the sum of these edge weights can be calculated, and then the weight of each edge can be divided by this sum to obtain the normalized edge weights. Normalization ensures that the sum of all the neighboring edge weights of a node equals one. This allows the neighborhood information of different nodes to be fused at the same scale during the subsequent weighted summation process, avoiding feature scale differences caused by varying numbers of neighboring nodes.

[0045] Then, a weighted summation is performed on the features of each node in the set of neighboring nodes of each node. For each neighboring node of a given node, the feature of that neighboring node is multiplied by the corresponding normalized edge weight, and then the weighted features of all neighboring nodes are summed to obtain a weighted aggregation result of the neighborhood features. This weighted aggregation result integrates the feature information of the neighboring nodes and distinguishes the contribution of different neighboring nodes according to the edge weight; neighboring nodes with larger weights have a greater impact on the aggregation result. The weighted summation result is then concatenated with the features of the current node itself. This concatenation can be achieved by connecting the two feature vectors in a dimensional direction to obtain the aggregated features. The aggregated features contain both the information passed from the neighboring nodes and retain the original feature information of the current node itself.

[0046] The aggregated features are subjected to linear transformation and nonlinear activation operations. The linear transformation, implemented using a weight matrix and a bias term, maps the aggregated features to a new feature space. Specifically, the aggregated features are multiplied by the weight matrix, and then a bias term is added. A nonlinear activation function, such as ReLU, is then applied to the result of the linear transformation. ReLU sets values ​​less than zero to zero and leaves values ​​greater than zero unchanged, introducing nonlinearity. After the linear transformation and nonlinear activation operations, the hidden state representation of the nodes in the current layer is obtained. This hidden state representation is an update result of the node features after one graph convolution operation.

[0047] To further extract deeper features, the hidden state representation of the nodes in the current layer can be used as the input to the next layer, and the aforementioned neighborhood aggregation, feature concatenation, linear transformation, and nonlinear activation operations can be repeatedly performed. Through multi-layer graph convolution operations, nodes can gradually fuse information from a wider range of neighborhoods, capturing more complex topological structures and feature relationships in heterogeneous graphs.

[0048] 204. Based on the node representations after the graph convolution, calculate the attention score of each modality node, normalize the attention score to obtain the attention weight, and perform weighted fusion of the node representations of each modality based on the attention weight to obtain multimodal fusion features; In this embodiment, after obtaining the node representations after graph convolution, it is necessary to evaluate the importance of nodes of different modalities to achieve adaptive feature fusion. For image nodes, point cloud nodes, sound nodes, and temperature nodes, since their contribution to abnormal event detection varies in different traffic scenarios, an attention mechanism is needed to dynamically adjust the weights of each modality.

[0049] Specifically, attention scores can be calculated by processing the representations of nodes in each modality using an attention network. An attention network can contain one or more fully connected layers. For each modality's node representation, it is input into the attention network, processed through linear transformations and activation functions in the fully connected layers, to obtain the attention score for that modality's node. For example, for the representation of an image node, a numerical value is obtained through the attention network mapping, reflecting the importance of the image modality in the current traffic scene. The same method can be used to obtain the attention scores for point cloud nodes, sound nodes, and temperature nodes. The attention scores for different modalities may vary significantly, reflecting the different roles of each modality in a specific scenario.

[0050] After obtaining the attention scores for each modality node, these scores need to be normalized so that the sum of the attention weights for all modalities equals one. The softmax function can be used for normalization. The softmax function maps any real value to a probability value between zero and one, ensuring that the sum of all values ​​is one. Applying the softmax function to the attention scores of each modality yields the normalized attention weights. For example, if the attention score of an image node is high, after softmax normalization, the attention weight corresponding to the image modality will also be relatively large, indicating that the image information is more important in the current scene.

[0051] Based on the obtained attention weights, weighted fusion of node representations from each modality can be performed. For each modality's node representation, it is multiplied by its corresponding attention weight to obtain the weighted node representation for that modality. Then, the weighted node representations of the image, point cloud, sound, and temperature modalities are summed to obtain a multimodal fusion feature. This fusion feature integrates information from various modalities and distinguishes the contribution of each modality based on the attention weights. In traffic congestion scenarios, image and point cloud modalities may receive higher weights because vehicle density information is mainly represented by visual and spatial data. In accident scenarios, sound modalities may receive higher weights because sound features such as collision sounds and braking sounds are crucial for accident detection.

[0052] 205. Based on the multimodal fusion features, inference calculations are performed using a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

[0053] In this embodiment, step 205 is similar to step 103, and will not be described again here.

[0054] In this embodiment, standardized feature vectors are obtained by preprocessing and extracting features from the collected multimodal traffic data. A heterogeneous graph representation is constructed based on these standardized feature vectors, and graph convolution and attention weighting are performed on this representation to obtain multimodal fusion features. Based on these multimodal fusion features, inference calculations are performed using a pre-defined traffic event detection model to obtain traffic event classification results and anomaly probabilities. This invention significantly improves the accuracy and robustness of traffic anomaly detection by modeling the spatiotemporal dependencies between multimodal data and utilizing an attention mechanism to achieve adaptive feature fusion. It is suitable for real-time anomaly monitoring scenarios in highway traffic management systems.

[0055] The traffic incident detection method based on data fusion in the embodiments of the present invention has been described above. The traffic incident detection device based on data fusion in the embodiments of the present invention will be described below. Please refer to the following for details regarding the traffic incident detection device based on data fusion. Figure 3 One embodiment of the traffic incident detection device based on data fusion in this invention includes: Data preprocessing module 301 is used to preprocess and extract features from the collected multimodal traffic data to obtain standardized feature vectors; The heterogeneous graph fusion module 302 is used to construct a heterogeneous graph representation based on the standardized feature vector, and to perform graph convolution operation and attention weighting on the heterogeneous graph representation to obtain multimodal fusion features; The event reasoning module 303 is used to perform reasoning calculations based on the multimodal fusion features and a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

[0056] In this embodiment of the invention, the traffic incident detection device based on data fusion operates the aforementioned traffic incident detection method based on data fusion. The device preprocesses and extracts features from the collected multimodal traffic data to obtain standardized feature vectors. Based on these standardized feature vectors, a heterogeneous graph representation is constructed. Graph convolution and attention weighting are then applied to this heterogeneous graph representation to obtain multimodal fusion features. Based on these multimodal fusion features, inference calculations are performed using a pre-defined traffic incident detection model to obtain traffic incident classification results and anomaly probabilities. This invention significantly improves the accuracy and robustness of traffic anomaly detection by modeling the spatiotemporal dependencies between multimodal data and utilizing an attention mechanism to achieve adaptive feature fusion. It is suitable for real-time anomaly monitoring scenarios in highway traffic management systems.

[0057] above Figure 3The traffic incident detection device based on data fusion in this embodiment of the invention will be described in detail from the perspective of unitized functional entities. The traffic incident detection device based on data fusion in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0058] Figure 4 This is a schematic diagram of a traffic incident detection device based on data fusion provided in an embodiment of the present invention. The traffic incident detection device 400 based on data fusion can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more units (not shown in the diagram), each unit may include a series of instruction operations on the traffic incident detection device 400 based on data fusion. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the traffic incident detection device 400 based on data fusion to implement the steps of the aforementioned traffic incident detection method based on data fusion.

[0059] The traffic incident detection device 400 based on data fusion may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The illustrated structure of the traffic incident detection device based on data fusion does not constitute a limitation on the traffic incident detection device based on data fusion provided by the present invention. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0060] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the traffic event detection method based on data fusion.

[0061] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0062] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A traffic incident detection method based on data fusion, characterized in that, The traffic incident detection method based on data fusion includes: The collected multimodal traffic data is preprocessed and features are extracted to obtain standardized feature vectors; A heterogeneous graph representation is constructed based on the standardized feature vectors. Graph convolution and attention weighting are then performed on the heterogeneous graph representation to obtain multimodal fusion features. Based on the multimodal fusion features, inference calculations are performed using a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

2. The traffic incident detection method based on data fusion according to claim 1, characterized in that, The preprocessing and feature extraction of the collected multimodal traffic data to obtain standardized feature vectors includes: The collected image data, point cloud data, sound data, and temperature data are denoised separately to obtain the denoised modal data. Spatiotemporal alignment is performed on the denoised modal data, and data collected by different sensors in the same time window and spatial location are matched to obtain spatiotemporally aligned multimodal data; The spatiotemporally aligned multimodal data are normalized to obtain normalized features for each modality, and the normalized features of each modality are concatenated to obtain a standardized feature vector.

3. The traffic incident detection method based on data fusion according to claim 1, characterized in that, The step of constructing a heterogeneous graph representation based on the standardized feature vector, and performing graph convolution and attention weighting on the heterogeneous graph representation to obtain multimodal fusion features includes: Each modal feature in the standardized feature vector is mapped to a different type of node in the heterogeneous graph. Edge connections between nodes are constructed based on the spatiotemporal correlation between modalities to obtain the heterogeneous graph representation. For each node in the heterogeneous graph representation, the feature information of its neighboring nodes is aggregated, and the aggregated node features are obtained by weighted summation to obtain the node representation after graph convolution. Based on the node representations after graph convolution, the attention score of each modality node is calculated, the attention score is normalized to obtain the attention weight, and the representations of each modality node are weighted and fused according to the attention weight to obtain multimodal fusion features.

4. The traffic incident detection method based on data fusion according to claim 3, characterized in that, The process of mapping each modal feature in the standardized feature vector to different types of nodes in the heterogeneous graph, and constructing edge connections between nodes based on the spatiotemporal correlation between modalities to obtain the heterogeneous graph representation includes: Each modal feature in the standardized feature vector is mapped to a different type of node in the heterogeneous graph to obtain a heterogeneous node set. For the nodes in the heterogeneous node set, calculate the temporal distance and spatial distance between the nodes, determine whether there is an association relationship between the nodes based on the temporal distance and spatial distance, and obtain the node association relationship matrix; Based on the node association matrix, edge connections are constructed for node pairs with association relationships, and edge weights are calculated based on the association strength to obtain a heterogeneous graph representation containing a set of nodes, a set of edges, and edge weights.

5. The traffic incident detection method based on data fusion according to claim 3, characterized in that, For each node in the heterogeneous graph representation, the feature information of its neighboring nodes is aggregated, and the aggregated node features are calculated by weighted summation to obtain the node representation after graph convolution, including: Based on the set of edges in the heterogeneous graph representation, determine the set of neighboring nodes for each node, and normalize the edge weights in the set of neighboring nodes to obtain normalized edge weights. For each node's features in its neighborhood node set, a weighted sum is calculated based on the normalized edge weights. The weighted sum is then concatenated with the current node's own features to obtain aggregated features. Perform linear transformation and nonlinear activation operations on the aggregated features to obtain the hidden state representation of the nodes in the current layer; The hidden state representation of the nodes in the current layer is used as the input to the next layer. The above aggregation and transformation operations are repeated, and after a preset number of layers, the node representation after graph convolution is obtained.

6. The traffic incident detection method based on data fusion according to claim 1, characterized in that, The step of performing inference calculations based on the multimodal fusion features using a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities includes: The multimodal fusion features are input into the input layer of a preset traffic event detection model, and after forward propagation calculation through a multi-layer neural network, feature representations of each traffic event category are obtained. A softmax operation is performed on the feature representations of each traffic event category to obtain the probability distribution of each event category; Based on the probability distribution, the category with the highest probability value is selected as the traffic event classification result, and the corresponding probability value is output as the anomaly probability.

7. The traffic incident detection method based on data fusion according to claim 6, characterized in that, After performing a softmax operation on the feature representations of each traffic event category to obtain the probability distribution of each event category, the method further includes: Calculate the Euclidean distance between the multimodal fusion feature and the feature center of each traffic event category, and normalize the Euclidean distance to obtain the feature deviation. Based on the feature deviation, a confidence adjustment coefficient is calculated using a preset confidence mapping function. When the feature deviation exceeds a preset threshold, the confidence adjustment coefficient is reduced; when the feature deviation is below the preset threshold, the confidence adjustment coefficient is increased. Based on the confidence adjustment coefficient, the probability values ​​in the probability distribution of each event category are scaled and adjusted, and the scaled probability values ​​are renormalized to obtain the corrected probability distribution.

8. A traffic incident detection device based on data fusion, characterized in that, The traffic incident detection device based on data fusion includes: The data preprocessing module is used to preprocess and extract features from the collected multimodal traffic data to obtain standardized feature vectors; The heterogeneous graph fusion module is used to construct a heterogeneous graph representation based on the standardized feature vector, and to perform graph convolution and attention weighting on the heterogeneous graph representation to obtain multimodal fusion features; The event reasoning module is used to perform reasoning calculations based on the multimodal fusion features and a preset traffic event detection model to obtain traffic event classification results and anomaly probabilities.

9. A traffic incident detection device based on data fusion, characterized in that, The traffic incident detection device based on data fusion includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the data fusion-based traffic incident detection device to perform the steps of the data fusion-based traffic incident detection method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the traffic incident detection method based on data fusion as described in any one of claims 1-7.

Images