Traffic anomaly processing method and device, electronic equipment and storage medium

Abstract

The application discloses a traffic anomaly processing method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring multi-modal data associated with traffic information in a target area; processing the multi-modal data based on a pre-trained traffic anomaly detection model to obtain traffic anomaly evaluation attributes; wherein the traffic anomaly detection model at least comprises a feature extraction module corresponding to each mode, a multi-head attention mechanism module and a Bayesian inference sub-model; when the traffic anomaly evaluation attributes meet a preset traffic anomaly condition, determining to-be-used data related to traffic anomaly processing according to the multi-modal data; processing the to-be-used data based on a pre-trained deep reinforcement learning model to determine a traffic relief strategy, and performing traffic relief processing on the target area based on the traffic relief strategy. The application improves the accuracy and efficiency of traffic anomaly detection, improves the efficiency of abnormal traffic relief, and ensures the timeliness and comprehensiveness of traffic relief.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for handling traffic anomalies. Background Technology

[0002] In urban traffic management, in order to improve road traffic efficiency and safety, abnormal traffic conditions are usually detected and managed.

[0003] Currently, existing methods for handling traffic anomalies mainly rely on manual monitoring and traffic anomaly detection based on single-modal data, followed by manual traffic management based on the anomaly detection results. Manual monitoring suffers from high labor costs, untimely and inaccurate anomaly detection, while single-modal data-based anomaly detection suffers from low detection accuracy and a high false alarm rate. Furthermore, manually determining traffic management strategies is insufficient to handle complex real-world traffic scenarios, resulting in delayed and incomplete traffic management. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for handling traffic anomalies, which improves the accuracy and efficiency of traffic anomaly detection, while also improving the efficiency of managing abnormal traffic and ensuring the timeliness and comprehensiveness of traffic management.

[0005] According to one aspect of the present invention, a traffic anomaly handling method is provided, the method comprising: Acquire multimodal data associated with traffic information within the target area; wherein, the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; The multimodal data is processed based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein, the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area; When the traffic anomaly assessment attributes meet the preset traffic anomaly conditions, the data to be used related to traffic anomaly handling is determined based on the multimodal data; The data to be used is processed based on a pre-trained deep reinforcement learning model to determine a traffic management strategy, and traffic management is carried out on the target area based on the traffic management strategy.

[0006] According to another aspect of the present invention, a traffic anomaly handling device is provided, the device comprising: A multimodal data acquisition module is used to acquire multimodal data associated with traffic information within a target area; wherein, the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; An anomaly assessment attribute determination module is used to process the multimodal data based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein, the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area; The data to be used determination module is used to determine the data to be used related to traffic anomaly handling based on the multimodal data when the traffic anomaly assessment attribute meets the preset traffic anomaly conditions. The traffic management module is used to process the data to be used based on a pre-trained deep reinforcement learning model, determine the traffic management strategy, and perform traffic management processing on the target area based on the traffic management strategy.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the traffic anomaly handling method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the traffic anomaly handling method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the traffic anomaly handling method as described in any embodiment of the present invention.

[0010] The technical solution of this invention ensures the comprehensiveness of the acquired data by obtaining multimodal data associated with traffic information within a target area. The multimodal data is processed using a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes, thereby improving the accuracy of traffic anomaly detection. When the traffic anomaly assessment attributes meet preset traffic anomaly conditions, the data to be used, associated with traffic anomaly handling, is determined based on the multimodal data. That is, when a traffic anomaly is determined to exist in the target area, the data to be used for handling the traffic anomaly is acquired, providing data support for subsequent determination of traffic management strategies. The data to be used is processed using a pre-trained deep reinforcement learning model to obtain traffic management strategies, achieving millisecond-level real-time determination of management strategies, improving the traffic efficiency of the target area, and reducing traffic congestion duration. This invention solves the problems of untimely and inaccurate traffic anomaly detection in the prior art by analyzing multimodal data, thereby improving the accuracy and efficiency of traffic anomaly detection. At the same time, it solves the problems of lagging and incompleteness of existing traffic management strategies. By generating real-time dynamic traffic management strategies through deep enhancement models, it improves the efficiency of traffic management in case of abnormal traffic and ensures the timeliness and comprehensiveness of traffic management.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a traffic anomaly handling method provided in an embodiment of the present invention; Figure 2 This is a flowchart of a traffic anomaly handling method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a traffic anomaly handling device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the traffic anomaly handling method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] Example 1 Figure 1 This is a flowchart of a traffic anomaly handling method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where traffic anomalies are detected in a target area, and a dynamic traffic diversion strategy corresponding to the abnormal traffic is determined to alleviate the abnormal traffic in the target area. This method can be executed by a traffic anomaly handling device, which can be implemented in hardware and / or software. The traffic anomaly handling device can be configured in electronic devices such as mobile phones, computers, or servers. Figure 1 As shown, the method includes: S110. Obtain multimodal data related to traffic information within the target area.

[0018] The target area can be the region where traffic anomaly detection is currently required. Traffic anomalies can include traffic accidents, traffic congestion, etc. Multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information for at least one vehicle. Road image information can be acquired in real time by roadside cameras integrated into a Road Side Unit (RSU). Road image information can include at least two images to be processed. The images to be processed include at least one traffic participation element corresponding to the road in the target area. Traffic participation elements can be understood as components that constitute the traffic system and affect traffic operation and safety. Optionally, traffic participation elements can include: road users (e.g., pedestrians, drivers of motor vehicles or non-motorized vehicles), infrastructure (e.g., roads, traffic signals, traffic facilities), and vehicles (e.g., motor vehicles, non-motorized vehicles).

[0019] Radar point cloud data can be three-dimensional point cloud data of traffic participation elements. Radar point cloud data can be acquired by millimeter-wave radar devices integrated in roadside units (RSUs).

[0020] At least one vehicle can be located within the target area. Vehicle driving information may include at least: vehicle speed, acceleration, direction of travel, angle of travel, vehicle location (latitude and longitude), braking status, steering angle, and throttle status. Vehicle driving information can be collected through in-vehicle IoT devices.

[0021] Specifically, the roadside unit integrates multimodal sensors to collect multimodal data related to traffic information within the target area. This includes acquiring road image information within the target area through roadside cameras integrated into the roadside unit, and acquiring radar point cloud data of at least one traffic participation element within the target area through millimeter-wave radar integrated into the roadside unit. Vehicle driving information is also collected through onboard IoT devices in vehicles within the target area to obtain driving information for at least one vehicle.

[0022] For example, the roadside camera device of the roadside unit can be a 4K camera with 30 frames per second, used to acquire visual images (corresponding to the images to be processed mentioned above) to obtain road image information. The millimeter-wave radar device of the roadside unit can be a 77GHz millimeter-wave radar with a range accuracy of 0.1 meters and a speed accuracy of 0.5 km / h, used to acquire vehicle speed / distance point clouds (corresponding to the radar point cloud data mentioned above). The acquired road image information and radar point cloud data are transmitted in real time to the Roadside Intelligent Processing Unit (RIPU). Correspondingly, the in-vehicle IoT device sends the acquired vehicle driving information to the Roadside Intelligent Processing Unit through the Vehicle-to-Everything (V2X) communication module, so that the Roadside Intelligent Processing Unit can perform traffic anomaly detection, traffic management decision determination, and power consumption optimization based on the acquired multimodal data. It should be noted that the V2X communication module follows the 5.9GHz DSRC / C-V2X protocol with a latency of less than 10 milliseconds, used to achieve real-time coordination between vehicles, roadside units, and traffic lights within the target area. In addition, the above also includes a solar power unit consisting of a 50W photovoltaic panel and a 48V lithium battery, which supports field deployment, has a battery life of more than 30 days, and is suitable for field deployment.

[0023] Taking a high-accident area (e.g., intersections, ramps) on a main urban road or highway as an example, the method of this invention is applied to a Roadside Intelligent Processing Unit (RIPU). The RIPU integrates a 4K camera and a 77GHz millimeter-wave radar to collect road image information and radar point cloud data in real time at a frequency of 30 Hz. Simultaneously, an onboard IoT device transmits vehicle driving information such as speed, acceleration, and vehicle location via a V2X communication module. This allows for the detection of traffic accidents within the target area using high-resolution 3840x2160 images (the image to be processed in the road image information), radar point cloud data, and vehicle telemetry data (vehicle driving information). The detection process can be completed within 100 milliseconds.

[0024] It should be noted that one roadside unit can be deployed every 500 meters, equipped with a 4K camera and 77GHz radar, and powered by solar energy. One roadside intelligent processing unit can be deployed every 2 kilometers, connecting 8 roadside units. The V2X communication module is based on the C-V2X protocol, covering an area with a radius of 1 kilometer. The V2X communication module supports dynamic adjustments of traffic lights and vehicles through millisecond-level communication, reducing congestion recovery time by 73%.

[0025] S120. Based on the pre-trained traffic anomaly detection model, multimodal data is processed to obtain traffic anomaly assessment attributes.

[0026] The traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model. The feature extraction module corresponding to each modality is used to extract features for that modality. This module includes: a feature extraction module corresponding to the road image information, a feature extraction module corresponding to the radar point cloud data, and a feature extraction module corresponding to the vehicle driving information. The multi-head attention mechanism is used to capture the correlation between features from different modalities, enabling feature fusion across all modalities. The Bayesian inference sub-model is used to determine traffic anomaly assessment attributes based on the features corresponding to all modalities and the fused features.

[0027] Traffic anomaly assessment attributes are used to reflect the existence of abnormal traffic events within a target area. Abnormal traffic events can be traffic congestion events or traffic accidents. Optionally, the probability of traffic accidents occurring within the target area can be determined using traffic anomaly assessment attributes.

[0028] Specifically, based on the feature extraction module corresponding to each modality in the traffic anomaly detection model, feature extraction processing is performed on the corresponding modality data to obtain the features to be processed for each modality. The multi-head attention mechanism module of the traffic anomaly detection model then fuses the features to be processed for all modalities to obtain fused features. Finally, a Bayesian inference sub-model processes the features to be processed for all modalities and the fused features to obtain the traffic anomaly assessment attributes.

[0029] In this embodiment of the invention, the method for determining traffic anomaly assessment attributes may be as follows: processing modal data based on the feature extraction module corresponding to each modality to obtain the features to be processed corresponding to each modality; performing feature fusion processing on the features to be processed corresponding to each modality based on the multi-head attention mechanism module to obtain fused features; and processing the fused features and the features to be processed corresponding to each modality based on the Bayesian inference sub-model to obtain traffic anomaly assessment attributes.

[0030] The features to be processed can be the modal features obtained by the feature extraction module of the corresponding modality after extracting features from the corresponding modality data. The fused features can be the features obtained by fusing the features to be processed corresponding to all modalities.

[0031] Specifically, a feature extraction module corresponding to road image information processes the road image information to obtain features to be processed. A feature extraction module corresponding to radar point cloud data processes the radar point cloud data to obtain features to be processed. A feature extraction module corresponding to vehicle driving information processes the vehicle driving information to obtain features to be processed. A multi-head attention mechanism module performs feature fusion processing on the features to be processed corresponding to road image information, radar point cloud data, and vehicle driving information to obtain fused features. A Bayesian inference sub-model infers the fused features, the features to be processed corresponding to road image information, radar point cloud data, and vehicle driving information to obtain traffic anomaly assessment attributes.

[0032] Optionally, the features to be processed corresponding to each modality include: image features to be processed, point cloud features to be processed, and vehicle features to be processed. The method for feature extraction from road image information can be as follows: for at least two images to be processed in the road image information, perform resolution adjustment and normalization processing on each of the at least two images to be processed to obtain at least two input images; for each of the at least two input images, extract features from each input image to obtain image features to be processed.

[0033] The image to be processed can be a road image acquired by a roadside camera device, including at least one traffic participation element. The input image can be an image obtained by sequentially adjusting the resolution and normalizing the image to be processed. Optionally, the feature extraction module corresponding to the road image information can be a feature extraction module based on a YOLOv5 convolutional neural network (CNN).

[0034] Specifically, for at least two images to be processed included in the road image information, each image is sequentially subjected to resolution adjustment processing to obtain an image with adjusted resolution. The at least two images with adjusted resolution are then normalized to obtain an input image corresponding to each image with adjusted resolution, thus obtaining at least two input images. Feature extraction is performed on each input image to obtain the image features corresponding to each input image.

[0035] For example, following the above example, the acquired images to be processed are sequentially adjusted to a resolution of 224x224, and all adjusted images are normalized to obtain the input image. Feature extraction is then performed on the input image to obtain the features of the image to be processed.

[0036] Correspondingly, the method for feature extraction of radar point cloud data can be as follows: downsampling the radar point cloud data and performing hierarchical feature extraction on the downsampling radar point cloud data to obtain at least one local point cloud feature; processing the at least one local point cloud feature based on at least two first fully connected layers and a first activation function to obtain the point cloud feature to be processed.

[0037] The local point cloud features can be features corresponding to a portion of the point cloud data in the downsampled radar point cloud data. Optionally, the feature extraction module corresponding to the radar point cloud data can be the feature extraction module of the PointNet++ network. At least two first fully connected layers and a first activation function can be fully connected layers and activation functions in the feature extraction module corresponding to the radar point cloud data. Accordingly, when the feature extraction module corresponding to the radar point cloud data can be the feature extraction module of the PointNet++ network, the first fully connected layer can be two fully connected layers of the PointNet++ network, and the first activation function can be the ReLU activation function.

[0038] Specifically, to improve data processing efficiency, radar point cloud data can be downsampled to obtain downsampled radar point cloud data. Farthest Point Sampling (FPS) is then performed on the downsampled radar point cloud data to identify at least one keypoint. For each keypoint, the distance between each keypoint and other points in the downsampled radar point cloud data is calculated. Based on the distance information between each keypoint and other points, multiple neighboring points corresponding to each keypoint are determined (e.g., other points in a spherical region with a radius of 0.2 meters centered on the keypoint can be considered as neighboring points). These neighboring points are then aggregated to obtain a local point cloud. Feature extraction is performed on the local point cloud to obtain local point cloud features.

[0039] At least one local point cloud feature is compressed using at least one first fully connected layer in the feature extraction module corresponding to the radar point cloud data to obtain a first point cloud feature. The first point cloud feature is then processed using a first activation function to obtain a second point cloud feature. The second point cloud feature is then processed using at least one first fully connected layer in the feature extraction module corresponding to the radar point cloud data to obtain the final point cloud feature.

[0040] Optionally, each point in the radar point cloud data may include three-dimensional point cloud coordinates. And speed. When the feature extraction module corresponding to the radar point cloud data can be a PointNet++ network feature extraction module, the radar point cloud data is downsampled to 1024 points, and the spatial and dynamic features of the downsampled radar point cloud data are extracted using the PointNet++ network feature extraction module to obtain 128-dimensional point cloud features to be processed. The specific processing can be shown in the following function: ; in, This represents the downsampled radar point cloud data. Referring to the example above, for radar point cloud data generated by a 77GHz millimeter-wave radar, each point contains: (x, y, z) coordinates (meters) and velocity (km / h). The radar generates approximately 10,000 points per frame, which are downsampled to 1024 points (voxel grid, 0.1-meter resolution) to reduce computational load. This indicates the hierarchical processing of the PointNet++ network. For example, it performs farthest point sampling on downsampled radar point cloud data to select 1024 key points, and then determines 32 neighboring points around each key point (spherical query, radius 0.2 meters), aggregating them into local point cloud features. This represents the features of a local point cloud. This represents the weights of the first fully connected layer in the PointNet++ network, used to compress point cloud features from 1024×4 dimensions to 256 dimensions. This represents the bias vector of the first fully connected layer in the PointNet++ network, used to match the output dimension. This represents the first activation function. This represents the weights of the second fully connected layer in the PointNet++ network, used to reduce the point cloud features from 256 dimensions to 128 dimensions. This represents the bias vector of the second first fully connected layer in the PointNet++ network, used to match the output dimension. This represents the 128-dimensional point cloud features to be processed, used to characterize the spatial and dynamic features of the point cloud (such as vehicle spacing and speed anomalies).

[0041] Correspondingly, the method for extracting features from vehicle driving information can be as follows: normalize the driving data of multiple driving dimensions in the vehicle driving information to obtain normalized driving data; process the normalized driving data based on at least two second fully connected layers and at least two second activation functions to obtain the vehicle features to be processed.

[0042] The multiple driving dimensions can include at least: driving speed, driving acceleration, driving direction, driving angular velocity, driving latitude and longitude, braking status, driving steering angle, and throttle status. Correspondingly, the driving data can be data under the corresponding driving dimensions. Normalized driving data can be data obtained by normalizing the driving data from multiple driving dimensions to a preset data range. The preset data range can be a data range from 0 to 1.

[0043] Optionally, the feature extraction module corresponding to the vehicle driving information can be implemented using a multilayer perceptron (MLP). The second fully connected layer and the second activation function can be the fully connected layer and activation function in the feature extraction module corresponding to the vehicle driving information. Accordingly, when the feature extraction module corresponding to the vehicle driving information can be implemented using a multilayer perceptron, the second fully connected layer can be a fully connected layer of the MLP, and the second activation function can be the ReLU activation function.

[0044] Specifically, the driving data across multiple dimensions in the vehicle driving information is normalized to obtain normalized driving data. The normalized driving data across multiple dimensions is then processed using at least one second fully connected layer of the feature extraction module to obtain a first vehicle feature. The first vehicle feature is then processed using at least one second activation function of the feature extraction module to obtain a second vehicle feature. The second vehicle feature is then processed using at least one second fully connected layer of the feature extraction module to obtain a third vehicle feature. Finally, the third vehicle feature is processed using at least one second activation function of the feature extraction module to obtain the final vehicle feature.

[0045] Optionally, if the feature extraction module corresponding to the vehicle driving information can be implemented by a multilayer perceptron, the driving data collected by the in-vehicle IoT device in eight driving dimensions, namely driving speed (km / h), driving acceleration (m / s²), etc., can be processed. 2 The driving direction (degrees), driving angular velocity (degrees / second), driving latitude and longitude, braking status (0 / 1), driving steering angle (degrees), and throttle status (0 / 1) are normalized to [0,1] to obtain normalized driving data. The normalized driving data is then processed through the second fully connected layer and the second activation function of the multilayer perceptron to obtain 128-dimensional vehicle features to capture abnormal vehicle behaviors (such as sudden braking and abnormal steering). The specific processing can be shown in the following function: ; in, This represents normalized driving data. This represents the weights of the first and second fully connected layers in the MLP, used to increase the feature dimension from 8 to 64. Correspondingly, This represents the bias vector of the first second fully connected layer corresponding to the MLP. This represents the second activation function corresponding to the MLP. This represents the weights of the second fully connected layer corresponding to the MLP, used to increase the feature dimension from 64 to 128. This represents the bias vector of the second fully connected layer corresponding to the MLP. It represents 128-dimensional vehicle features to be processed, used to characterize vehicle behavior patterns. Each dimension represents a vehicle behavior feature (such as emergency braking + steering), capturing individual vehicle behavior and compensating for the macroscopic limitations of vision and radar.

[0046] Optionally, after obtaining the image features, point cloud features, and vehicle features to be processed, the multi-head attention mechanism module can integrate the image features (256 dimensions), point cloud features (128 dimensions), and vehicle features (128 dimensions) to obtain a 512-dimensional fused feature using the following function. in, Indicates the features of the image to be processed. This represents the point cloud features to be processed. This indicates the characteristics of the vehicle to be processed. (Through...) It can capture the above three modal correlations (such as visual smoke + radar stationary + vehicle emergency braking).

[0047] Correspondingly, the Bayesian inference sub-model can determine the traffic anomaly assessment attributes using the following function: ; in, Indicates the features of the image to be processed. This represents the point cloud features to be processed. Indicates the characteristics of the vehicle to be processed. This indicates a traffic anomaly assessment attribute, used to quantify the likelihood of traffic anomalies (traffic accidents) occurring. This represents the prior probability (usually determined based on the historical anomaly probability of the target area, and can be set to 0.1). This represents the normalization factor, which is usually estimated based on historical samples.

[0048] After determining the features of the image to be processed and the features of the point cloud to be processed, a collision risk attribute can be determined by combining traffic anomaly assessment attributes, and then used as a parameter in the deep reinforcement learning model. Accordingly, the collision risk attribute can be determined as follows: For at least two input images, convolution processing is performed on the features of the image to be processed corresponding to the input images to determine the bounding box information of multiple objects to be verified in the input images; overlap analysis is performed on the bounding box information corresponding to each pair of objects to be verified to determine the intersection-union ratio (IURR) information between each pair of objects to be verified; motion anomaly detection is performed based on the object velocity information corresponding to the input images and the IURR information between each pair of objects to be verified to obtain the collision assessment attribute; anomaly assessment is performed on the features of the point cloud to be processed to determine the point cloud anomaly assessment attribute; and the collision risk attribute is determined based on the point cloud anomaly assessment attribute, the collision assessment attribute, and the traffic anomaly assessment attribute.

[0049] The objects to be verified can be traffic-related elements such as vehicles and obstacles identified in the input image. Bounding box information characterizes the region of the identified object in the input image. Bounding box intersection-over-union (OCU) information characterizes the degree of overlap between the bounding boxes of any two objects. Object velocity information represents the speed of the object in the image at the time the image corresponding to the input image was acquired. It should be noted that, typically, object velocity information refers to vehicle velocity. Collision assessment attributes characterize the probability of a collision between the objects in the input image and the objects being verified. Point cloud anomaly assessment attributes characterize the probability of point cloud anomalies within the target area. Collision risk attributes characterize the probability of a secondary collision within the target area.

[0050] Specifically, for at least two input images, convolution processing is performed on the features of the corresponding input images to identify multiple objects to be verified in the input images and determine the bounding box information corresponding to each object. Overlap analysis is then performed on the bounding box information of every two objects to be verified to determine the intersection-union ratio (IUGR) of the bounding boxes between each pair of objects.

[0051] For multiple objects to be verified, object velocity change information is determined based on the object velocity information of the current object in the current input image and the object velocity information of the corresponding object in the previous input image. The intersection-over-union (IoU) ratio of the bounding boxes between the current object and other objects in the current input image is multiplied by the object velocity change information to determine the collision sub-evaluation attributes between the current object and other objects. All collision sub-evaluation attributes are then fused to obtain the collision evaluation attributes.

[0052] Accordingly, anomaly assessment is performed on the point cloud features to be processed, resulting in point cloud anomaly assessment attributes corresponding to the features. The point cloud anomaly assessment attributes, collision assessment attributes, and traffic anomaly assessment attributes are then weighted and fused to obtain collision risk attributes.

[0053] Optionally, in conjunction with the above, after obtaining the image features corresponding to each input image, the image features can be processed using a YOLOv5 convolutional neural network (CNN) to identify traffic participation elements such as vehicles and obstacles in the input image, and determine the bounding boxes and confidence scores corresponding to these traffic participation elements (corresponding to the bounding box information mentioned above). By analyzing the degree of overlap and motion anomalies (e.g., sudden stopping) of traffic participation elements such as vehicles and obstacles, a collision severity score (corresponding to the collision assessment attributes mentioned above) is determined. The specific processing procedure can be determined using the following function: ; in, This represents the intersection-over-union (IoU) ratio of the bounding boxes between object i and object j to be verified. It belongs to the data range [0,1] and is used to measure the degree of overlap between the objects to be verified. This represents the velocity change information of object i in two consecutive frames of the input image. When the object to be verified is a vehicle, This represents the speed change (in km / h) of vehicle i between two consecutive input frames (time difference 33.3 ms), reflecting whether there was sudden braking or abnormal acceleration. For example, if the vehicle decelerates from 60 km / h to 30 km / h... The speed is 30 km / h. This indicates that for all objects to be verified... of and Summation of products, that is, summing up the evaluation attributes of all collision sub-categories to synthesize all potential collision possibilities in the target area. This represents the collision severity score, which is the collision assessment attribute mentioned above.

[0054] Accordingly, after obtaining the point cloud features to be processed based on the PointNet++ network, the point cloud features to be processed can be processed by the following function to detect stationary vehicles or abnormal trajectories (such as speed suddenly dropping to 0) and obtain the point cloud anomaly probability, that is, the point cloud anomaly evaluation attribute mentioned above.

[0055] ; in, This represents the point cloud features to be processed. Indicates weight, Represents the bias vector, through and Determine the point cloud anomaly probability corresponding to the 128-dimensional point cloud features to be processed. . This represents the probability of point cloud anomalies, used to characterize the likelihood of point cloud anomalies. It corresponds to the point cloud anomaly assessment attribute mentioned above and belongs to the data range of [0,1].

[0056] Based on the above, collision risk attributes can be determined using the following function: ; in, The collision risk attribute (i.e., the change in the number of secondary collisions occurring on abnormal road sections in the target area, [0,1]) is used to reflect traffic safety. This represents the collision severity score, which is the collision assessment attribute mentioned above. This represents the probability of point cloud anomalies, corresponding to the point cloud anomaly assessment attributes mentioned above. This indicates the attributes of traffic anomaly assessment. , , These weights can be determined experimentally. Optional, , , .

[0057] S130. When the traffic anomaly assessment attributes meet the preset traffic anomaly conditions, determine the data to be used related to traffic anomaly handling based on multimodal data.

[0058] The preset traffic anomaly conditions can be pre-set conditions that the traffic anomaly assessment attributes must meet. Optionally, if the traffic anomaly assessment attribute exceeds a preset traffic anomaly attribute threshold, the traffic anomaly assessment attribute is determined to meet the preset traffic anomaly conditions. The preset traffic anomaly attribute threshold can be a pre-set standard value for the traffic anomaly assessment attribute. For example, the preset traffic anomaly attribute threshold could be 0.7.

[0059] The data to be used in relation to traffic anomaly handling can be traffic data determined based on multimodal data, used to generate traffic management strategies. Optionally, the data to be used may include: location information of traffic anomalies in the target area, traffic congestion length, traffic flow information, location information of rescue vehicles, traffic light status information, etc.

[0060] Specifically, when the traffic anomaly assessment attribute is greater than the preset traffic anomaly attribute threshold, it is determined that the traffic anomaly assessment attribute meets the preset traffic anomaly conditions. Multimodal data is analyzed to determine the data to be used related to traffic anomaly handling, so as to determine the traffic diversion strategy for the target area through the data to be used.

[0061] S140. Based on the pre-trained deep reinforcement learning model, process the data to be used, determine the traffic diversion strategy, and perform traffic diversion processing on the target area based on the traffic diversion strategy.

[0062] Deep Reinforcement Learning (DRL) models are used to analyze the data to determine the optimal traffic management strategy for the current scenario in the target area. The traffic management strategy can be a method for managing traffic within the target area. Optionally, the traffic management strategy can determine action instructions for at least one control system associated with traffic management within the target area, so that the control system executes the action instructions to achieve traffic management. For example, the control system can be a system that controls the state of traffic lights, managing traffic flow within the target area by controlling the on / off state of the traffic lights.

[0063] Specifically, the pre-trained deep reinforcement learning model processes the data to be used, determines the optimal traffic management strategy for the target area under the current scenario, and performs traffic management processing on the target area based on the traffic management strategy.

[0064] It should be noted that the deep reinforcement learning model can be pre-trained in the cloud with 10 million simulations to achieve an inference latency of less than 100 milliseconds and a model compression to 50MB (INT8 quantization).

[0065] The technical solution of this embodiment ensures the comprehensiveness of the acquired data by obtaining multimodal data associated with traffic information within the target area. The multimodal data is processed using a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes, thereby improving the accuracy of traffic anomaly detection. When the traffic anomaly assessment attributes meet preset traffic anomaly conditions, the data to be used, associated with traffic anomaly handling, is determined based on the multimodal data. That is, when a traffic anomaly is determined to exist in the target area, the data to be used for handling the traffic anomaly is acquired, providing data support for subsequent determination of traffic management strategies. The data to be used is processed using a pre-trained deep reinforcement learning model to obtain traffic management strategies, achieving millisecond-level real-time determination of management strategies, improving the traffic efficiency of the target area, and reducing traffic congestion duration. This invention solves the problems of untimely and inaccurate traffic anomaly detection in the prior art by analyzing multimodal data, thereby improving the accuracy and efficiency of traffic anomaly detection. At the same time, it solves the problems of lagging and incompleteness of existing traffic management strategies. By generating real-time dynamic traffic management strategies through deep enhancement models, it improves the efficiency of traffic management in case of abnormal traffic and ensures the timeliness and comprehensiveness of traffic management.

[0066] Example 2 Figure 2This is a flowchart of a traffic anomaly handling method provided in Embodiment 2 of the present invention. This embodiment is a preferred embodiment of the above embodiments. For specific implementation details, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the method includes: S210. Obtain multimodal data related to traffic information within the target area.

[0067] The multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle.

[0068] S220. Based on the pre-trained traffic anomaly detection model, multimodal data is processed to obtain traffic anomaly assessment attributes.

[0069] The traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model. The traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area.

[0070] S230. When the traffic anomaly assessment attributes meet the preset traffic anomaly conditions, determine the data to be used related to traffic anomaly handling based on multimodal data.

[0071] After obtaining the collision risk attributes, a deep reinforcement learning model can be used to process the data to be processed and the collision risk attributes to obtain traffic management strategies.

[0072] S240. At least three third fully connected layers of the policy network based on the deep reinforcement learning model process the data to be used in sequence to obtain state features.

[0073] The deep reinforcement learning model includes a policy network and a value network. The policy network (Actor) is used to analyze the data to be used in relation to traffic anomaly handling. The policy network can include at least three fully connected layers, namely the third fully connected layer mentioned above. The state features can be the state feature representation determined by the policy network in processing the data to be used.

[0074] Specifically, after determining the data to be used related to traffic anomaly handling, the state vector corresponding to the data to be used can be determined. The state vector is then normalized and input into a deep reinforcement learning model. The normalized state vector is then processed by at least three third fully connected layers of the policy network of the deep reinforcement learning model to obtain state features.

[0075] Optionally, in conjunction with the above, when the traffic anomaly assessment attribute exceeds a preset traffic anomaly attribute threshold, the Roadside Intelligent Processing Unit (RIPU) can determine that a traffic anomaly or traffic accident exists in the target area. Based on multimodal data, the data to be used in the target area associated with the traffic anomaly is determined. This data is processed to obtain a 37-dimensional state vector, including information such as the location of the traffic anomaly (e.g., traffic accident coordinates), traffic congestion length, traffic flow (vehicles / minute), rescue vehicle location, traffic light status, RIPU computational load, and multimodal sensor battery level. After normalizing the 37-dimensional state vector, it is input into the Actor-Critic architecture of the Deep Reinforcement Learning (DRL) model, so that the three fully connected layers (256→512→256 neurons) contained in the policy network (Actor) (corresponding to at least three third fully connected layers mentioned above) generate state features using the ReLU activation function. Specific processing details can be found in the following function: ; in, It represents 256-dimensional state features, used to characterize higher-order features of traffic conditions (such as congestion severity and rescue priority). The 37-dimensional state vector includes accident coordinates (x, y, 2D, unit: meters), congestion length (1D, unit: meters), traffic flow (1D, unit: vehicles / minute), rescue vehicle location (x, y, 2D), traffic light status (green / red / yellow, 3D, one-hot encoded), RIPU computational load (1D, 0-1), sensor battery level (1D, 0-1), and other traffic parameters (such as vehicle speed distribution and road segment type, approximately 26 dimensions). This represents the weights of the first third fully connected layer of the policy network. This represents the bias of the first third fully connected layer in the policy network, which is used to adjust the feature dimension from 37 to 256. This represents the weights of the second and third fully connected layers of the policy network. This represents the bias of the second and third fully connected layers in the policy network. These layers are used to adjust the feature dimension from 256 to 512. This represents the weights of the third fully connected layer in the policy network. This indicates the bias of the third fully connected layer of the policy network, which is used to adjust the feature dimension from 512 to 256.

[0076] S250. The third activation function based on the policy network is used to process the state features to obtain the action probability distribution.

[0077] The third activation function can be understood as the activation function in the policy network. Optionally, the third activation function can be a softmax function, used to convert state features into a probability distribution. In this embodiment of the invention, the third activation function is used to convert state features into an action probability distribution. The action probability distribution can be understood as the probability of performing a corresponding traffic management action. Corresponding traffic management actions may include: traffic light phase adjustment, lane closure, diversion guidance, and emergency lane opening, etc.

[0078] Specifically, the state features are processed using the third activation function based on the policy network to obtain the action probability distribution: ; in, This represents a 12-dimensional probability distribution of actions (such as adjusting traffic light phases, lane closures, traffic diversion guidance, and emergency lane opening). This represents the output layer weights of the policy network. This indicates the output layer bias of the policy network. Represents 256-dimensional state features. As an activation function, it can be used... The function selects the traffic management action with the highest probability.

[0079] S260. Based on the value network, conduct value assessment of the data to be used and determine the state value assessment attributes that match the state characteristics.

[0080] The Critic network is used to evaluate the value of the current traffic state corresponding to the data to be used, and to measure long-term returns (such as traffic management effectiveness) to enhance the stability of the policy network's action choices. The state value assessment attribute can be used to characterize the expected returns of the current traffic state corresponding to the data to be used.

[0081] Specifically, the value network can use the following function to evaluate the value of the data to be used, so as to obtain state value evaluation attributes that match the state characteristics: ; in, This represents a 37-bit state vector corresponding to the data to be used. , , The weights of the value network (Critic), where, The feature dimensions were adjusted from 37 to 256. The feature dimensions were adjusted from 256 to 512. The feature dimension was adjusted from 512 dimensions to 1 dimension. , , The bias of the value network (Critic) is used to match the output dimension of the corresponding layer. This represents the state value assessment attribute, used to characterize the expected return of the current state (1-dimensional, floating-point number, range [-infinity, infinity]).

[0082] S270. Based on multimodal data, energy consumption information of the data acquisition device corresponding to the multimodal data, and collision risk attributes, determine the data to be calculated.

[0083] The data acquisition device corresponding to multimodal data can be understood as a multimodal sensor that collects multimodal data. For example, the data acquisition device may include the roadside camera device for collecting road image information, the radar device for collecting radar point cloud data, and the communication module for transmitting vehicle driving information. Energy consumption information can be understood as the sensor energy consumption of the data acquisition device. The data to be calculated may be data determined based on multimodal data, energy consumption information, and collision risk attributes.

[0084] Specifically, based on road image information and radar point cloud data from the multimodal data, the length of traffic congestion in the target area is determined. Then, based on the radar point cloud data and vehicle driving information from the multimodal data, the average vehicle speed in the target area is determined. Based on the traffic congestion length and average vehicle speed, the duration of traffic congestion in the target area is determined. Correspondingly, road traffic efficiency is determined based on the multimodal data. Road traffic efficiency, traffic congestion duration, energy consumption information, and collision risk attributes are identified as data to be calculated.

[0085] For example, in conjunction with the above, the Roadside Intelligent Processing Unit (RIPU) determines the traffic congestion duration in real time using multimodal sensors (corresponding to the data acquisition devices mentioned above). Specifically, the process can be as follows: Based on road image information and radar point cloud data, determine the length of continuous road segments with stationary or low-speed vehicles (less than 5 km / h) before and after the traffic anomaly in the target area; that is, obtain the traffic congestion length. Then, determine the average vehicle speed within the target area using radar point cloud data and vehicle driving information. Substituting the traffic congestion length and the average vehicle speed into the following function yields the traffic congestion duration: ; in, Indicates the length of traffic congestion, This indicates the average speed of the vehicle (km / h). Indicates the duration of traffic congestion (in minutes).

[0086] It should also be noted that, to adapt to the subsequent preset reward function, the traffic congestion duration can be normalized based on the maximum historical congestion duration determined from historical data, so as to obtain a normalized traffic congestion duration in the range of [0,1]. ; in, Indicates the duration of traffic congestion. Indicates the maximum historical congestion duration, for example, It can be 10 minutes. This indicates the normalized duration of traffic congestion.

[0087] Accordingly, based on multimodal data, the number of vehicles passing through each minute on road sections with traffic anomalies within the target area is determined. And the road capacity (e.g., 1200 vehicles / hour). Divide the number of vehicles passing through per minute by the road capacity to obtain the road traffic efficiency. .

[0088] The determined power consumption of the multimodal sensor is approximately 4 watts. Based on historical data, the maximum historical power consumption is determined to be approximately 10 watts. Therefore, the normalized power consumption information... It can be 0.4.

[0089] The determined normalized traffic congestion duration, road traffic efficiency, and normalized energy consumption information, combined with the collision risk attributes identified above, are used as the data to be calculated.

[0090] S280. Evaluate the data to be calculated based on the preset reward function and determine the reward evaluation attributes.

[0091] The preset reward function can be a pre-determined function. The reward evaluation attribute can be used to weigh congestion time, traffic efficiency, energy consumption, and collision risk to evaluate the effectiveness of the action.

[0092] Specifically, the following preset reward function can be used to evaluate road traffic efficiency, normalized traffic congestion duration, normalized energy consumption information, and collision risk attributes: ; in, Indicates normalized traffic congestion duration, Indicates road traffic efficiency. This represents the normalized energy consumption information. This indicates the collision risk attribute. This indicates the reward evaluation attribute.

[0093] S290. Adjust the action probability distribution based on the reward evaluation attribute and the state value evaluation attribute to obtain the traffic diversion strategy, and carry out traffic diversion processing on the target area based on the traffic diversion strategy.

[0094] Specifically, based on the reward evaluation attributes and state value evaluation attributes, a proximal optimization strategy is used to update the policy network and value network to adjust the action probability distribution, thus obtaining the traffic management strategy. This can be implemented using the following function: ; ; in, The immediate reward (which is a reward evaluation attribute that adjusts according to changes in the action probability distribution) is determined by processing the data to be calculated based on the reward evaluation function, which changes with the action probability distribution. This represents the discount factor, which can be set to 0.99. This indicates the current state's value assessment attribute. It can represent the next state value assessment attribute (Critic network output), which changes with the action probability distribution. The Critic network is used to measure the quality of actions. Greater than 0: The traffic management action is better than expected, and the Actor network should increase the probability of this action. , representing the ratio of the probability distributions of the new and old actions, is output by the Actor network. This indicates the trimming parameter (0.2, range 0.8-1.2).

[0095] For example, in conjunction with the above, after determining that there is a traffic anomaly in the target area, the Roadside Intelligent Processing Unit (RIPU) can generate the optimal traffic management strategy (such as adjusting the traffic light phase or closing lanes) within 92 milliseconds at the 2-kilometer road segment node in the target area based on the accident information and traffic status (corresponding to the data to be used mentioned above).

[0096] Optionally, traffic management of the target area based on the traffic management strategy includes: determining at least one action command corresponding to the traffic management strategy; for the at least one action command, sending the action command to the control system associated with the action command, so that the control system executes the traffic management action based on the action command.

[0097] At least one action command may include one or more traffic management commands such as traffic light duration adjustment commands and vehicle route replanning commands. The control system associated with the action command may be a system for controlling corresponding equipment to perform traffic management actions based on the action command. For example, a control system for controlling traffic light duration, and a control system for performing vehicle route replanning.

[0098] Specifically, at least one action command corresponding to the traffic management strategy is determined, and the action command is sent to a control system adapted to the action command, so that the control system controls the corresponding equipment to perform traffic management actions that match the action command.

[0099] Optionally, after the RIPU generates a traffic management strategy, the traffic light controller, vehicle navigation system, and emergency command center execute traffic management actions within a 1-kilometer radius of the accident scene in the target area. Action commands are broadcast within 10 milliseconds via the V2X communication module to ensure rapid congestion relief and priority passage for rescue vehicles. Specifically: The RIPU sends action commands (traffic light phase duration, lane status, and GPS coordinates of the diversion path), see the following command set functions: ; Among them, instruction set This includes: the phase duration associated with the traffic light duration adjustment command. Lane status (0 / 1) associated with vehicle route replanning instructions. and path coordinates Delay < 10 milliseconds.

[0100] Accordingly, the traffic light controller (the control system associated with the traffic light duration adjustment command) receives the command (e.g., extending the north-south green light by 30 seconds), updates the phase cycle (minimum 30 seconds), and prioritizes emergency vehicles. For specific duration adjustments, please refer to the following function: ; in, This indicates the adjusted phase duration of the traffic lights. This indicates the default period (usually set to 30 seconds). This indicates the DRL adjustment amount.

[0101] The in-vehicle navigation system plans the optimal vehicle route coordinate sequence based on the coordinates of traffic accidents and the length of traffic congestion within the target area to determine the optimal vehicle path. The V2X communication module works in conjunction to reduce the risk of secondary collisions.

[0102] ; in, Represents the minimum distance to the new vehicle path. Indicates the duration of traffic congestion, This is the weighting factor, which is usually set to 0.5. Indicates the new vehicle route.

[0103] RIPU uploads traffic anomaly information (traffic accident coordinates, severity) to the emergency command center, integrating traffic accidents and regional traffic flow at a global level to optimize traffic diversion and cross-regional traffic management.

[0104] ; in, Indicates the coordinates of traffic accidents within the target area. Indicates traffic flow (vehicles / minute) in the target area. This indicates the severity of the traffic accident.

[0105] In conjunction with the above, the embodiments of the present invention further include: for multiple computing nodes to be used, acquiring in real time the node operation status data corresponding to the computing nodes to be used; wherein, the multiple computing nodes to be used include at least: local computing nodes, edge computing nodes, and cloud computing nodes; evaluating and processing the node operation status data based on a target energy consumption assessment model to determine the energy consumption assessment attributes corresponding to the computing nodes to be used; and determining a target computing node based on the energy consumption assessment attributes of each computing node to be used, so as to perform operations corresponding to traffic anomaly handling based on the target computing node.

[0106] The multiple computing nodes to be used can be computing nodes used for traffic anomaly handling methods. These multiple computing nodes include at least: local computing nodes, edge computing nodes, and cloud computing nodes. Node operating status data can be used to characterize the node operating status of the computing nodes to be used. Optionally, the node status data can be an 8-dimensional state vector. When the node operating status data includes data such as the battery power (SOC, data range [0,1]) of the multimodal sensor, CPU utilization, network latency information (milliseconds), data complexity information processed (MB / s), and signal strength, the node operating status data can be represented as: ; in, Represents node state data, an 8-dimensional state vector, normalized to [0,1]. Indicates battery level, Indicates CPU utilization, Indicates network latency information, This indicates information about data complexity.

[0107] The target energy consumption assessment model can be used to assess the energy consumption of node operating status data. The energy consumption assessment attribute represents the energy consumption of the computing node to be used. The target computing node can be determined from multiple computing nodes to be used, and is the node used to execute the corresponding data processing step in the traffic anomaly handling method.

[0108] Specifically, for multiple computing nodes to be used, including local computing nodes, edge computing nodes, and cloud computing nodes, the node status and operation data of each node to be used are acquired in real time. The node operation status data of each node to be used is then evaluated using a target energy consumption assessment model to obtain energy consumption assessment attributes. Based on the energy consumption assessment attributes of each node to be used, a target computing node is determined from the multiple nodes to be used, and data processing operations corresponding to traffic anomaly handling are performed based on the target computing node. Optionally, to optimize node energy consumption, the RIPU's task processor can dynamically determine the computing node (local roadside unit, RIPU, or cloud) to perform data processing operations corresponding to traffic anomaly handling based on the real-time acquired node status data, covering a 2-kilometer RSU network, and perform these operations simultaneously to ensure efficient resource utilization.

[0109] Based on the deep Q-network in the target energy consumption assessment model, the expected returns for selecting local computing nodes, edge computing nodes (RIPU offloading, transferring data processing tasks corresponding to traffic anomaly handling from local devices to RIPU processing), or cloud computing nodes (cloud offloading, transferring data processing tasks corresponding to traffic anomaly handling from local devices to remote cloud servers for execution) are determined. Based on these expected returns, the target computing nodes are then selected. Specifically: ; This represents the 3D (local processing, RIPU unloading, cloud unloading) and the expected return of the three actions. Represents the node's operational status data, an 8-dimensional state vector. , , For the weights of a deep Q-network, The feature dimensions were adjusted from 8 to 64. Adjust the feature dimension from 64 to 64. The feature dimension was adjusted from 64 dimensions to 3 dimensions. , , This is the bias of the deep Q-network, used to match the output dimension of the corresponding layer.

[0110] Determine the action with the highest Q value based on the expected return. : ; It should be noted that when the battery level is below 30%, the local compute node is forced to process data to protect the battery. ; in, This indicates the action of selecting a compute node to be used. "local" indicates that the local compute node will process the data, and "SOC" indicates the battery level.

[0111] Energy consumption assessment for different computing nodes to be used: ; in, Indicates energy consumption assessment attributes, This indicates the amount of data being processed; "local" indicates that the processing is done on the local computing node. This indicates edge computing node processing. This indicates processing by cloud computing nodes.

[0112] Optionally, the deep Q-network can be optimized by prioritizing experience replay to update the target compute node: ; in, This represents the loss of a deep Q-network. This indicates an immediate reward (negative energy consumption). This represents the discount factor (0.99). This represents the maximum Q value of the next state. This represents the maximum Q-value in the current state. Determining the target computing node using a deep Q-network can reduce power consumption by 67%.

[0113] The technical solution of this embodiment ensures the comprehensiveness of the acquired data by obtaining multimodal data associated with traffic information within the target area. The multimodal data is processed using a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes, thereby improving the accuracy of traffic anomaly detection. When the traffic anomaly assessment attributes meet preset traffic anomaly conditions, the data to be used, associated with traffic anomaly handling, is determined based on the multimodal data. That is, when a traffic anomaly is determined to exist in the target area, the data to be used for handling the traffic anomaly is acquired, providing data support for subsequent determination of traffic management strategies. A policy network based on a deep reinforcement learning model processes the data to be used to obtain an action probability distribution, and a value network evaluates the value of the data to be used to obtain state value assessment attributes. Based on the multimodal data, the energy consumption information of the data acquisition device corresponding to the multimodal data, and collision risk attributes, the data to be calculated is determined. The data to be calculated is evaluated according to a preset reward function to determine the reward assessment attributes. The action probability distribution is adjusted based on the reward assessment attributes and state value assessment attributes to obtain a traffic management strategy, and traffic management is performed on the target area based on the traffic management strategy. This invention achieves real-time traffic management decisions within less than 100 milliseconds through policy and value networks, improving traffic efficiency in target areas and reducing traffic congestion duration. It addresses the problems of untimely and inaccurate traffic anomaly detection in existing technologies by analyzing multimodal data, thus improving the accuracy and efficiency of anomaly detection. Simultaneously, it solves the problems of lagging and incomplete existing traffic management strategies by generating real-time dynamic traffic management strategies through deep reinforcement models, improving the efficiency of anomaly traffic management and ensuring the timeliness and comprehensiveness of traffic management.

[0114] Example 3 Figure 3 This is a schematic diagram of a traffic anomaly handling device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a multimodal data acquisition module 310, an anomaly assessment attribute determination module 320, a data to be used determination module 330, and a traffic management module 340.

[0115] A multimodal data acquisition module 310 is used to acquire multimodal data related to traffic information within a target area; wherein the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; an anomaly assessment attribute determination module 320 is used to process the multimodal data based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events within the target area; a data to be used determination module 330 is used to determine data to be used related to traffic anomaly handling based on the multimodal data when the traffic anomaly assessment attributes meet preset traffic anomaly conditions; a traffic diversion module 340 is used to process the data to be used based on a pre-trained deep reinforcement learning model to determine a traffic diversion strategy, and perform traffic diversion processing on the target area based on the traffic diversion strategy.

[0116] The technical solution of this embodiment ensures the comprehensiveness of the acquired data by obtaining multimodal data associated with traffic information within the target area. The multimodal data is processed using a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes, thereby improving the accuracy of traffic anomaly detection. When the traffic anomaly assessment attributes meet preset traffic anomaly conditions, the data to be used, associated with traffic anomaly handling, is determined based on the multimodal data. That is, when a traffic anomaly is determined to exist in the target area, the data to be used for handling the traffic anomaly is acquired, providing data support for subsequent determination of traffic management strategies. The data to be used is processed using a pre-trained deep reinforcement learning model to obtain traffic management strategies, achieving millisecond-level real-time determination of management strategies, improving the traffic efficiency of the target area, and reducing traffic congestion duration. This invention solves the problems of untimely and inaccurate traffic anomaly detection in the prior art by analyzing multimodal data, thereby improving the accuracy and efficiency of traffic anomaly detection. At the same time, it solves the problems of lagging and incompleteness of existing traffic management strategies. By generating real-time dynamic traffic management strategies through deep enhancement models, it improves the efficiency of traffic management in case of abnormal traffic and ensures the timeliness and comprehensiveness of traffic management.

[0117] Based on the above embodiments, optionally, the anomaly assessment attribute determination module includes: a feature extraction unit, used to process modal data based on the feature extraction module corresponding to each modality to obtain the features to be processed corresponding to each modality; a feature fusion unit, used to perform feature fusion processing on the features to be processed corresponding to each modality based on the multi-head attention mechanism module to obtain fused features; and an assessment attribute determination unit, used to process the fused features and the features to be processed corresponding to each modality based on a Bayesian inference sub-model to obtain traffic anomaly assessment attributes.

[0118] Optionally, the features to be processed corresponding to each modality include: image features to be processed, point cloud features to be processed, and vehicle features to be processed. The feature extraction unit includes: an image feature extraction subunit, used to perform resolution adjustment and normalization processing on at least two images to be processed in the road image information to obtain at least two input images; and to extract features from each of the at least two input images to obtain image features to be processed; a point cloud feature extraction subunit, used to perform downsampling processing on the radar point cloud data and perform hierarchical feature extraction on the downsampling radar point cloud data to obtain at least one local point cloud feature; and to process at least one of the local point cloud features according to at least two first fully connected layers and a first activation function to obtain point cloud features to be processed; and a vehicle feature extraction subunit, used to normalize driving data of multiple driving dimensions in the vehicle driving information to obtain normalized driving data; and to process the normalized driving data according to at least two second fully connected layers and at least two second activation functions to obtain vehicle features to be processed.

[0119] Optionally, the device further includes: a collision assessment attribute determination module, configured to: perform convolution processing on the image features corresponding to the at least two input images to determine the bounding box information of multiple objects to be verified in the input images; perform overlap analysis on the bounding box information corresponding to each pair of objects to be verified to determine the cross-union ratio (CUI) information between each pair of objects to be verified; perform motion anomaly detection based on the object velocity information corresponding to the input images and the CUI information between each pair of objects to be verified to obtain a collision assessment attribute; perform anomaly assessment on the point cloud features to be processed to determine the point cloud anomaly assessment attribute; and determine a collision risk attribute based on the point cloud anomaly assessment attribute, the collision assessment attribute, and the traffic anomaly assessment attribute.

[0120] Optionally, the deep reinforcement learning model includes a policy network and a value network. The traffic management module includes: a traffic management strategy determination unit, used to process the data to be used sequentially based on at least three third fully connected layers of the policy network to obtain state features; process the state features based on the third activation function of the policy network to obtain an action probability distribution; evaluate the value of the data to be used based on the value network to determine a state value evaluation attribute matching the state features; determine the data to be calculated based on the multimodal data, the energy consumption information of the data acquisition device corresponding to the multimodal data, and the collision risk attribute; evaluate the data to be calculated according to a preset reward function to determine a reward evaluation attribute; and adjust the action probability distribution based on the reward evaluation attribute and the state value evaluation attribute to obtain a traffic management strategy.

[0121] Optionally, the traffic management module includes: a traffic management unit, configured to determine at least one action command corresponding to the traffic management strategy; and for at least one action command, to send the action command to a control system associated with the action command, so that the control system performs traffic management actions based on the action command.

[0122] Optionally, the device further includes: a node energy consumption assessment module, used to acquire in real time node operation status data corresponding to multiple computing nodes to be used; wherein the multiple computing nodes to be used include at least: local computing nodes, edge computing nodes, and cloud computing nodes; the node operation status data is evaluated and processed based on a target energy consumption assessment model to determine the energy consumption assessment attribute corresponding to the computing nodes to be used; and a target computing node is determined based on the energy consumption assessment attribute of each computing node to be used, so as to perform operations corresponding to traffic anomaly handling based on the target computing node.

[0123] The traffic anomaly handling device provided in the embodiments of the present invention can execute the traffic anomaly handling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0124] Example 4 Figure 4This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0125] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0126] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0127] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as traffic anomaly handling methods.

[0128] In some embodiments, the traffic malfunction handling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the traffic malfunction handling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the traffic malfunction handling method by any other suitable means (e.g., by means of firmware).

[0129] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] Computer programs for implementing the traffic anomaly handling method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0132] Example 5 Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a traffic malfunction handling method, the method comprising: Acquire multimodal data associated with traffic information within the target area; wherein, the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; The multimodal data is processed based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein, the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area; When the traffic anomaly assessment attributes meet the preset traffic anomaly conditions, the data to be used related to traffic anomaly handling is determined based on the multimodal data; The data to be used is processed based on a pre-trained deep reinforcement learning model to determine a traffic management strategy, and traffic management is carried out on the target area based on the traffic management strategy.

[0133] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0136] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0137] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for handling traffic anomalies, characterized in that, include: Acquire multimodal data associated with traffic information within the target area; wherein, the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; The multimodal data is processed based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein, the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area; When the traffic anomaly assessment attributes meet the preset traffic anomaly conditions, the data to be used related to traffic anomaly handling is determined based on the multimodal data; The data to be used is processed based on a pre-trained deep reinforcement learning model to determine a traffic management strategy, and traffic management is carried out on the target area based on the traffic management strategy.

2. The method according to claim 1, characterized in that, The multimodal data processing based on the pre-trained traffic anomaly detection model yields traffic anomaly assessment attributes, including: The modal data is processed based on the feature extraction module corresponding to each modality to obtain the features to be processed for each modality; Based on the multi-head attention mechanism module, feature fusion processing is performed on the features to be processed corresponding to each modality data to obtain fused features; Based on the Bayesian inference sub-model, the fused features and the features to be processed corresponding to each modal data are processed to obtain traffic anomaly assessment attributes.

3. The method according to claim 2, characterized in that, The features to be processed corresponding to each modality include: image features to be processed, point cloud features to be processed, and vehicle features to be processed. The road image information is processed based on the feature extraction module corresponding to the road image information to obtain the image features to be processed, including: For at least two images to be processed in the road image information, resolution adjustment and normalization are performed on the at least two images to be processed respectively to obtain at least two input images; For the at least two images to be input, feature extraction is performed on each image to obtain the features of the image to be processed; Accordingly, the radar point cloud data is processed by the feature extraction module corresponding to the radar point cloud data to obtain the point cloud features to be processed, including: The radar point cloud data is downsampled, and the downsampled radar point cloud data is subjected to hierarchical feature extraction to obtain at least one local point cloud feature. At least one of the local point cloud features is processed based on at least two first fully connected layers and a first activation function to obtain the point cloud feature to be processed; Accordingly, the feature extraction module corresponding to the vehicle driving information processes the vehicle driving information to obtain the vehicle features to be processed, including: The driving data of multiple driving dimensions in the vehicle driving information are normalized to obtain normalized driving data; The normalized driving data is processed based on at least two second fully connected layers and at least two second activation functions to obtain the vehicle features to be processed.

4. The method according to claim 3, characterized in that, The method further includes: For the at least two images to be input, convolution processing is performed on the image features corresponding to the images to be input to determine the bounding box information of multiple objects to be verified in the images to be input. Perform overlap analysis on the bounding box information corresponding to each pair of objects to be verified, and determine the intersection-union ratio information of the bounding boxes between each pair of objects to be verified. Motion anomaly detection is performed based on the object velocity information corresponding to the input image and the bounding box intersection-union ratio information between every two objects to be verified to obtain collision evaluation attributes. Anomaly assessment is performed on the features of the point cloud to be processed to determine the point cloud anomaly assessment attributes; Based on the point cloud anomaly assessment attributes, the collision assessment attributes, and the traffic anomaly assessment attributes, collision risk attributes are determined.

5. The method according to claim 4, characterized in that, The deep reinforcement learning model includes a policy network and a value network. Based on the pre-trained deep reinforcement learning model, the data to be used is processed to determine traffic management strategies, including: The data to be used is processed sequentially by at least three third fully connected layers of the policy network to obtain state features; The state features are processed based on the third activation function of the policy network to obtain the action probability distribution; Based on the value network, the data to be used is valued to determine the state value assessment attributes that match the state characteristics. Based on the multimodal data, the energy consumption information of the data acquisition device corresponding to the multimodal data, and the collision risk attribute, the data to be calculated is determined; The data to be calculated is evaluated based on a preset reward function to determine the reward evaluation attributes; The action probability distribution is adjusted based on the reward evaluation attribute and the state value evaluation attribute to obtain a traffic management strategy.

6. The method according to claim 1, characterized in that, The traffic management process for the target area based on the traffic management strategy includes: Determine at least one action command corresponding to the traffic management strategy; For at least one of the action commands, the action command is sent to the control system associated with the action command, so that the control system performs traffic management actions based on the action command.

7. The method according to claim 1, characterized in that, The method further includes: For multiple computing nodes to be used, the node running status data corresponding to the computing nodes to be used is acquired in real time; wherein, the multiple computing nodes to be used include at least: local computing nodes, edge computing nodes, and cloud computing nodes; The node operation status data is evaluated and processed based on the target energy consumption assessment model to determine the energy consumption assessment attributes corresponding to the computing node to be used. Based on the energy consumption assessment attributes of each computing node to be used, a target computing node is determined, and operations corresponding to traffic anomaly handling are performed based on the target computing node.

8. A traffic anomaly handling device, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal data associated with traffic information within a target area; wherein, the multimodal data includes road image information, radar point cloud data corresponding to at least one traffic participation element, and vehicle driving information of at least one vehicle; An anomaly assessment attribute determination module is used to process the multimodal data based on a pre-trained traffic anomaly detection model to obtain traffic anomaly assessment attributes; wherein, the traffic anomaly detection model includes at least: a feature extraction module corresponding to each modality, a multi-head attention mechanism module, and a Bayesian inference sub-model, and the traffic anomaly assessment attributes are used to reflect whether there are traffic anomaly events in the target area; The data to be used determination module is used to determine the data to be used related to traffic anomaly handling based on the multimodal data when the traffic anomaly assessment attribute meets the preset traffic anomaly conditions. The traffic management module is used to process the data to be used based on a pre-trained deep reinforcement learning model, determine the traffic management strategy, and perform traffic management processing on the target area based on the traffic management strategy.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the traffic abnormality handling method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the traffic malfunction handling method according to any one of claims 1-7.

Images