Road abnormal event processing method, device and storage medium
By deploying multimodal models on vehicles for road anomaly event identification and automatic reporting, the problem of blind spots in road anomaly event monitoring has been solved, achieving full coverage and timely response, and improving the efficiency and safety of traffic management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SZ ZHUOYU TECH CO LTD
- Filing Date
- 2026-03-03
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, there are regulatory blind spots in the monitoring of abnormal road events outside specific areas, which leads to delays in the discovery and reporting of events, making it impossible to handle them in a timely manner and affecting traffic safety and management efficiency.
By deploying multimodal models on vehicles, road anomaly events can be identified by acquiring multimodal data, and the identification results can be automatically reported to the target receiving end according to the preset reporting strategy, so as to achieve full coverage and timely response.
It eliminates blind spots in traditional monitoring, improves the timeliness of identifying and handling abnormal road events, ensures that events can be dealt with in a timely manner, and enhances the responsiveness of traffic management.
Smart Images
Figure CN122157478A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent transportation technology, and in particular to a method, device and storage medium for handling abnormal road events. Background Technology
[0002] With rapid socio-economic development and a continuous increase in the number of motor vehicles, road traffic pressure is increasing daily, and various traffic violations have become a major cause of traffic congestion and accidents. Therefore, effective monitoring of traffic violations and timely identification and reporting of abnormal road events such as traffic congestion and accidents are crucial for traffic management departments to ensure road safety and regulate driving behavior.
[0003] In existing technologies, the identification of abnormal road events primarily relies on specialized equipment (such as fixed high-definition cameras and radar speed detectors) deployed at traffic intersections and key road sections to monitor roads in specific areas in real time, automatically capturing and recording vehicle violations within those areas. However, abnormal events occurring on roads outside these designated areas are typically discovered and reported to traffic management departments by individuals. The time from discovery by pedestrians or traffic management personnel to reporting usually takes a considerable amount of time, leading to untimely handling of abnormal events occurring on roads outside these designated areas. Summary of the Invention
[0004] This application provides a method, device, and storage medium for handling abnormal road events, which can improve the ability to proactively detect abnormal events in non-fixed-point monitoring areas while ensuring the timeliness of event response, thereby improving the efficiency of handling abnormal road traffic events.
[0005] Firstly, this application provides a method for handling abnormal road events, including:
[0006] Acquire multimodal data of the vehicle's surrounding environment while the vehicle is running;
[0007] The multimodal data is processed by a multimodal model to identify road anomalies. When a road anomaly is identified, an anomaly identification result is generated, which includes the anomaly type, hazard level, and key evidence.
[0008] According to the preset reporting strategy, the abnormal event identification results are reported to the corresponding target receiving end.
[0009] In one possible implementation, the multimodal model includes a feature fusion layer and an anomaly detection layer; the process of identifying road anomaly events using the multimodal model includes:
[0010] The multimodal data is temporally aligned and feature fused through the feature fusion layer to obtain an environmental perception feature vector;
[0011] The anomaly recognition layer performs road anomaly event recognition processing on the environmental perception feature vector, and outputs at least one candidate anomaly event type and the confidence level corresponding to each candidate anomaly event type.
[0012] If the confidence level corresponding to any candidate abnormal event type is greater than or equal to the confidence level threshold, then it is determined that the road abnormal event exists;
[0013] If the confidence level corresponding to each candidate abnormal event type is less than the confidence level threshold, then it is determined that there is no abnormal road event.
[0014] In one possible implementation, generating the abnormal event identification result includes:
[0015] For each candidate abnormal event type with a confidence level greater than or equal to the confidence level threshold, an abnormal event corresponding to the candidate abnormal event type is determined, and the candidate abnormal event type is determined as the abnormal event type of the abnormal event.
[0016] The hazard level is determined based on the degree of interference the abnormal event causes to the passage of surrounding traffic participants;
[0017] Based on the time of occurrence of the abnormal event, data within a preset time window is extracted from the multimodal data as the key evidence;
[0018] Based on the abnormal event type, the hazard level, and the key evidence, a corresponding abnormal event identification result is generated.
[0019] In one possible implementation, determining the hazard level based on the degree of interference the abnormal event causes to surrounding traffic participants includes:
[0020] If the abnormal event does not interfere with the passage of surrounding traffic participants, then the hazard level is determined to be a relatively minor level.
[0021] If the abnormal event causes traffic disruption to the surrounding traffic participants but does not pose a safety risk, then the hazard level is determined to be of the general level.
[0022] If the abnormal event poses a collision risk, causes lane congestion, or disrupts traffic to surrounding road users, the hazard level is determined to be severe.
[0023] In one possible implementation, reporting the abnormal event identification result to the corresponding target receiving end according to a preset reporting strategy includes:
[0024] The corresponding target receiving end is determined by querying the pre-stored reporting strategy table based on the abnormal event type.
[0025] The abnormal event identification results are reported to the target receiving end via the vehicle-mounted wireless communication link.
[0026] In one possible implementation, the road abnormal events include at least one of the following categories: road violation events, traffic abnormal events, urban management abnormal events, and parking abnormal events;
[0027] The multimodal data includes at least two of the following: image data or video data from vehicle-mounted cameras, point cloud data from millimeter-wave radar or lidar, vehicle attitude data from inertial measurement units, map navigation data, and road scene data. The road scene data is obtained by processing sensor perception data using a combination of driver assistance algorithms.
[0028] In one possible implementation, the method further includes:
[0029] Receive model deployment requests sent by the cloud server;
[0030] In response to the model deployment request, a response message is sent to the cloud server, the response message indicating whether the deployment of the multimodal model is agreed upon;
[0031] When the response message indicates agreement to deploy the multimodal model, the system receives the multimodal model sent by the cloud server and deploys the multimodal model.
[0032] Secondly, this application provides a device for handling abnormal road events, comprising:
[0033] The acquisition module is used to acquire multimodal data of the vehicle's surrounding environment when the vehicle is started.
[0034] The first processing module is used to perform road anomaly event identification processing on the multimodal data through a multimodal model, and when it is determined that there is a road anomaly event, it generates anomaly event identification results, which include anomaly event type, hazard level and key evidence.
[0035] The second processing module is used to report the abnormal event identification results to the corresponding target receiving end according to a preset reporting strategy.
[0036] Thirdly, this application provides an electronic device, including: a processor and a memory;
[0037] The memory stores computer-executed instructions;
[0038] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.
[0039] Fourthly, this application provides a vehicle, including: a vehicle body, and electronic equipment as described in the third aspect.
[0040] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.
[0041] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.
[0042] This application provides a method, device, and storage medium for handling abnormal road events. The method includes: acquiring multimodal data of the vehicle's surrounding environment when the vehicle starts; performing road anomaly event identification processing on the multimodal data using a multimodal model; generating anomaly event identification results when an abnormal road event is determined to exist; and then reporting the anomaly event identification results to the corresponding target receiving end according to a preset reporting strategy. The anomaly event identification results include the anomaly event type, hazard level, and key evidence. In the above process, the synergistic effect of the multimodal model deployed on the vehicle and the preset reporting strategy can eliminate traditional monitoring blind spots while reducing manual intervention, thereby improving the timeliness and accuracy of road anomaly event identification and reporting. Attached Figure Description
[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0044] Figure 1 A flowchart illustrating an embodiment of the method for handling abnormal road events provided in this application;
[0045] Figure 2 A flowchart illustrating Embodiment 2 of the method for handling abnormal road events provided in this application;
[0046] Figure 3 A flowchart illustrating Embodiment 3 of the method for handling abnormal road events provided in this application;
[0047] Figure 4 A flowchart illustrating Embodiment 4 of the method for handling abnormal road events provided in this application;
[0048] Figure 5A flowchart illustrating an example of a method for handling abnormal road events provided in this application embodiment;
[0049] Figure 6 A schematic diagram of the structure of the road abnormality event processing device provided in the embodiments of this application;
[0050] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0053] With rapid socio-economic development, the number of motor vehicles has continued to rise, leading to a significant increase in road traffic flow and increasingly prominent road traffic pressure. Frequent traffic violations not only disrupt normal road traffic order but also become a major cause of traffic congestion and accidents, seriously threatening road safety and the public's right to travel. Therefore, efficient and comprehensive monitoring of various traffic violations, timely and accurate identification of road congestion, traffic accidents, and other abnormal road events, and rapid reporting of these events have become crucial for traffic management departments to strengthen road control, ensure road safety, regulate driving behavior, and improve emergency response efficiency. This is of great significance for promoting intelligent transportation construction and achieving refined road traffic management.
[0054] In related technologies, the identification and monitoring of abnormal road events mainly rely on fixed monitoring solutions. This involves deploying dedicated monitoring equipment (such as fixed high-definition cameras and radar speed detectors) in key road sections and areas such as traffic intersections, highway entrances and exits, and urban main roads to monitor road traffic conditions in real time. This automatically captures and records traffic violations such as running red lights, speeding, and illegal parking, providing support for traffic management departments to identify and handle violations.
[0055] However, due to limitations imposed by geographical location, construction costs, and wiring conditions, the deployment of fixed monitoring equipment can only cover a few key locations and areas, failing to extend to urban secondary roads, rural roads, and non-key sections of highways. This results in numerous blind spots in the monitoring of abnormal road events. Currently, there are no effective automatic identification and reporting methods for these blind spots—namely, abnormal events such as traffic violations, traffic congestion, and traffic accidents occurring on roads outside specific areas. These events typically rely on accidental discovery by passersby, drivers, or traffic management personnel, who then manually report them to the traffic management department.
[0056] More significantly, the entire process from the discovery of a road anomaly to its reporting often involves multiple steps, including event confirmation, information processing, finding reporting channels, and submitting relevant information. This process is time-consuming and can easily lead to delays in reporting anomalies. The efficiency of handling road anomalies is directly related to the timeliness of reporting. Delayed reporting prevents traffic management departments from promptly grasping the situation at the scene and quickly implementing emergency response and traffic control measures. This can not only exacerbate traffic congestion but also lead to secondary accidents due to untimely handling of traffic accidents and other emergencies, further increasing casualties and property damage, and seriously affecting road safety and management efficiency.
[0057] To address the aforementioned issues, the inventors propose a method for handling road anomaly events, comprising: after vehicle startup, acquiring multimodal data (such as images, point clouds, etc.) of the vehicle's surrounding environment; performing road anomaly event identification processing on the multimodal data using a multimodal model to accurately determine the existence of road anomalies; and generating anomaly event identification results containing the anomaly event type, hazard level, and key evidence if anomalies are found; subsequently, reporting the anomaly event identification results to the corresponding target receiving end according to a preset reporting strategy to ensure timely handling of road anomalies. In this process, the organic integration of the vehicle-deployed multimodal model and the strategy-driven reporting mechanism fully leverages the vehicle's full-coverage capability as a mobile monitoring node, eliminating blind spots in traditional fixed monitoring. Simultaneously, through automated identification and accurate reporting, the efficiency of handling road anomalies can be significantly improved, ensuring timely and reliable event processing.
[0058] The application scenarios of the embodiments of this application will be described below first.
[0059] The methods for handling abnormal road events provided can be widely applied in various industries such as intelligent connected vehicles, traffic management, parking management, auto insurance, and urban management. These include, but are not limited to: automatic identification and reporting of abnormal events such as traffic accidents, road obstacles, and traffic violations by intelligent connected vehicles on urban roads, highways, and rural roads; real-time perception and rapid response of abnormal events across the entire road network by traffic management departments; intelligent supervision of illegal parking, occupation of parking spaces, and obstruction of passageways by parking lot managers; accident liability determination and claims assessment by insurance institutions based on structured event evidence; and dynamic inspection and collaborative handling of urban appearance issues such as street vending and illegal operations by urban management departments.
[0060] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0061] Figure 1 This is a flowchart illustrating an embodiment of the method for handling abnormal road events provided in this application. Please refer to [link / reference]. Figure 1 The method includes:
[0062] S101. Acquire multimodal data of the vehicle's surrounding environment when the vehicle is started.
[0063] The execution subject of the method embodiments of this application can be an electronic device or a road anomaly event processing device installed in an electronic device. The road anomaly event processing device can be implemented by software or by a combination of software and hardware. The road anomaly event processing device can be a processor in an electronic device. For ease of understanding, the technical solution of this application will be described below using an electronic device as an example.
[0064] In this step, multimodal data for abnormal event identification can be acquired after the vehicle is powered on and put into operation.
[0065] In one alternative implementation, the above-mentioned "acquisition" includes, but is not limited to: acquiring raw signals in real time from onboard sensors, calling intermediate processing results from the driver assistance system, and reading external context information from the onboard information system.
[0066] Specifically, multimodal data includes at least two of the following: image or video data from vehicle-mounted cameras, point cloud data from millimeter-wave radar or lidar, vehicle attitude data output by an inertial measurement unit (IMU), map navigation data, and road scene data. Among these, road scene data is a structured environmental understanding result obtained by processing sensor perception data (such as image data and point cloud data) using combined driver assistance algorithms; examples include object detection boxes, lane line fitting parameters, or the movement trajectories of traffic participants.
[0067] In one alternative implementation, the vehicle-mounted camera may include at least one of a forward-looking, side-looking, or surround-view camera, and the map navigation data (such as lane attributes and speed limit information) may come from a high-precision map or navigation system.
[0068] Furthermore, the combination of multimodal data can be flexibly adapted according to differences in vehicle configuration. For example, in vehicles equipped only with basic Advanced Driver Assistance Systems (ADAS), video data from onboard cameras, vehicle attitude data, and map navigation data can be used; in advanced intelligent connected vehicles, point cloud data from LiDAR and road scene data output by combined ADAS through combined driver assistance algorithms can be fused.
[0069] For example, when vehicle A is started, multimodal data of the environment surrounding vehicle A can be acquired. This multimodal data may include video data collected by the vehicle A's forward-facing camera, point cloud data generated by the onboard LiDAR, and map navigation data.
[0070] S102. Use a multimodal model to process multimodal data for road anomaly event identification, and generate anomaly event identification results when it is determined that a road anomaly event exists.
[0071] In this step, multimodal data can be input into a pre-deployed multimodal model, which performs fusion analysis and anomaly detection, and generates anomaly event identification results when anomaly events are determined to exist.
[0072] Optionally, abnormal road events include at least one of the following categories: road violation events, traffic abnormal events, urban management abnormal events, and parking abnormal events; where "road" refers to all kinds of public or semi-public passage and parking spaces involved in vehicle operation, including but not limited to urban public roads, roadside parking lanes, public parking lots, and transportation hub areas.
[0073] In one specific implementation, the multimodal model includes a feature fusion layer and an anomaly recognition layer. The feature fusion layer first performs temporal alignment and feature fusion on the multimodal data to generate an environment-aware feature vector. Then, the anomaly recognition layer outputs multiple candidate anomaly event types and their corresponding confidence levels based on the environment-aware feature vector.
[0074] If the confidence level of any candidate abnormal event type is greater than or equal to a preset threshold, an abnormal road event is determined to exist, and the process proceeds to the result generation stage; otherwise, it is considered a normal traffic scenario and no further processing is triggered. The generated abnormal event identification results include three parts: abnormal event type, hazard level, and key evidence, which is data extracted from multimodal data within a preset time window around the time of the event.
[0075] For example, after vehicle A starts, the electronic device acquires video data collected by its forward-facing camera, point cloud data generated by LiDAR, and map navigation data, and inputs the above multimodal data into a multimodal model. After feature fusion and recognition processing, the model detects that vehicle B has been driving continuously in the bus lane of an urban main road for more than 10 seconds, and there is no emergency avoidance situation in the vicinity. The model outputs a confidence level of 0.92 for the "occupying the bus lane" event, which is higher than the preset threshold of 0.85. Based on this, it is determined that there is an abnormal road event, and the corresponding abnormal event identification result is generated, including: the abnormal event type is "occupying the bus lane", the hazard level is "general", and the key evidence is a video clip centered on the time of the event, 5 seconds before and after, and the corresponding point cloud data.
[0076] S103. According to the preset reporting strategy, the abnormal event identification results are reported to the corresponding target receiving end.
[0077] In this step, the generated anomaly identification result, combined with a preset reporting strategy, can be reported to the corresponding target receiver. Specifically, the pre-stored reporting strategy table can be queried based on the anomaly type in the anomaly identification result to determine the corresponding target receiver; then, the anomaly identification result is reported to the target receiver via the vehicle-mounted wireless communication link.
[0078] The vehicle-mounted wireless communication link refers to the wireless data transmission channel supported by the communication module installed in the vehicle, used to send vehicle-side information to an external platform or receiver. This communication link can be implemented based on various wireless communication technologies, including fourth-generation mobile communication technology (4G), 5G, vehicle-to-everything (V2X), wireless Fidelity (Wi-Fi), and satellite communication. In practical applications, one or more of these technologies can be selected and combined according to network conditions and business requirements.
[0079] Optionally, the reporting strategy table can predefine the mapping relationship between abnormal event types and target receivers. For example, traffic violation events (such as running red lights, crossing lines, and occupying bus lanes) correspond to the traffic management department's management platform; traffic abnormal events (such as traffic accidents, road obstacles, and road collapses) correspond to the road administration or emergency command center; urban management abnormal events (such as street vending and illegal operation) correspond to the urban comprehensive management platform; and parking abnormal events (such as occupying private parking spaces, fuel vehicles occupying charging spaces, and blocking passages) correspond to the management of the corresponding parking lot.
[0080] In one specific implementation, the information of the target receiving end can be obtained by the electronic device in different ways. For example, for fixed jurisdictional units such as traffic management departments, their communication interface addresses can be pre-configured in the memory of the electronic device; while for dynamic scenarios such as shopping malls or public parking lots, the electronic device can automatically connect to the network to query the contact information of the corresponding parking lot management based on the current location of the vehicle, such as obtaining its message receiving channel through the application programming interface (API) provided by a third-party service platform, and pushing alarm information and key evidence to the management through a secure encrypted channel.
[0081] For example, electronic devices can query the reporting strategy table based on the abnormal event type "occupying a bus lane" in the abnormal event identification results, determine the target receiving end as the traffic management department's management platform, and report the abnormal event identification results, including the abnormal event type "occupying a bus lane", the hazard level "general", and video clips and corresponding point cloud data 5 seconds before and after the time of the event, to the management platform via the 5G network.
[0082] In this embodiment, multimodal data of the vehicle's surrounding environment can be acquired when the vehicle is started. A multimodal model is then used to process this data for road anomaly event identification. Upon determining the presence of an anomaly, an anomaly event identification result is generated and reported to the corresponding target receiver according to a preset reporting strategy. The anomaly event identification result includes the anomaly event type, hazard level, and key evidence. By utilizing the vehicle as a mobile sensing node and combining intelligent identification and strategy-based reporting using a multimodal model, road anomalies can be detected and reported promptly, improving the efficiency of anomaly event handling and the level of road management.
[0083] Figure 2 This is a flowchart illustrating Embodiment Two of the method for handling abnormal road events provided in this application. Please refer to... Figure 2Based on the above embodiments, the multimodal model includes a feature fusion layer and an anomaly recognition layer. Step S102, which involves using the multimodal model to perform road anomaly event recognition processing on the multimodal data, may include the following steps:
[0084] S201. The multimodal data is time-aligned and feature-fused through the feature fusion layer to obtain the environmental perception feature vector.
[0085] In this step, the acquired multimodal data can be input into the feature fusion layer of the multimodal model. The feature fusion layer first synchronizes the multimodal data in time to ensure that the multimodal data are on the same spatiotemporal reference. Then, through the cross-modal feature extraction and fusion mechanism, the time-aligned multimodal data can be mapped to a unified high-dimensional semantic space to generate an environmental perception feature vector that can comprehensively represent the current traffic scene.
[0086] Optionally, time alignment can be achieved using hardware timestamp synchronization or software interpolation alignment. For example, for high-frequency vehicle attitude data (e.g., 100Hz) and low-frequency image data (e.g., 30Hz) or LiDAR point cloud data (e.g., 10Hz), based on the timestamps of each sensor, linear interpolation, spline interpolation, or nearest neighbor matching methods can be used to align the low-frequency data to the keyframes of the high-frequency data in the time dimension, or to aggregate the high-frequency data to the sampling time of the low-frequency frames, thereby constructing a multimodal data frame sequence with a unified time reference.
[0087] Optionally, in terms of feature fusion, the subsequent multimodal data can be feature-encoded separately and then fused using methods such as concatenation, weighted summation, or attention mechanisms to generate a unified feature representation that integrates multi-source information. For example, a cross-modal attention mechanism can be introduced to dynamically adjust the contribution weights of different modalities, such as image data, point cloud data, and vehicle posture data, in the current scene, thereby improving the ability of the environmental perception feature vector to express complex traffic conditions.
[0088] S202. The anomaly recognition layer performs road anomaly event recognition processing on the environmental perception feature vector, and outputs at least one candidate anomaly event type and the confidence level corresponding to each candidate anomaly event type.
[0089] In this step, the environmental perception feature vector output by the feature fusion layer can be input into the anomaly recognition layer of the multimodal model. The anomaly recognition layer performs classification reasoning on the environmental perception feature vector and outputs one or more candidate anomaly event types and their corresponding confidence scores.
[0090] Optionally, the anomaly detection layer can adopt a multi-label classification architecture to support scenarios where multiple anomaly events coexist. The confidence score of each candidate anomaly event type represents the multimodal model's belief that the event occurred in the current traffic scenario, and the value is typically a real number between 0 and 1. For example, when a vehicle is detected driving in a bus lane without emergency avoidance behavior, the model can output a confidence score of 0.92 for the "occupying the bus lane" event and a confidence score of 0.08 for the "driving normally" event.
[0091] Optionally, the anomaly detection layer can employ deep learning structures such as Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), or Transformer to perform classification and reasoning on environmental perception feature vectors.
[0092] For example, when vehicle A is driving to an urban intersection, it detects that vehicle B ahead has crossed the stop line while the red light is on and continues to pass through. At the same time, it senses that there is a significant pothole on the road surface ahead in its lane. The anomaly recognition layer can output two candidate anomaly event types: "running a red light" and "road obstacle". The confidence level for "running a red light" is 0.94, and the confidence level for "road obstacle" is 0.88.
[0093] S203. If the confidence level corresponding to any candidate abnormal event type is greater than or equal to the confidence level threshold, then it is determined that there is a road abnormal event.
[0094] In this step, the confidence level of each candidate abnormal event type output by the anomaly identification layer can be compared with the preset confidence level threshold. If the confidence level of at least one candidate abnormal event type is greater than or equal to the confidence level threshold, it is determined that there is a road abnormal event in the current traffic scenario.
[0095] Optionally, the confidence threshold can be set to a fixed value or dynamically adjusted based on the type of abnormal event, the complexity of the road scene, or the vehicle's driving status. For example, for event types with high hazard levels (such as running a red light or road collapse), a lower threshold can be used to improve detection sensitivity and avoid missed detections; while for event types that are susceptible to environmental interference (such as temporary parking or false detections due to obstruction), a higher threshold can be used to suppress false alarms, thereby achieving a reasonable balance between detection rate and accuracy.
[0096] For example, if the confidence threshold is set to a fixed value of 0.8, the confidence scores of "running a red light" (0.94) and "road obstacle" (0.88) output by the anomaly detection layer are both greater than the confidence threshold of 0.8, so it can be directly determined that there is an abnormal road event.
[0097] S204. If the confidence level corresponding to each candidate abnormal event type is less than the confidence level threshold, then it is determined that there is no abnormal road event.
[0098] In this step, the confidence scores of each candidate abnormal event type output by the anomaly identification layer can be compared with a preset confidence threshold. If the confidence scores of all candidate abnormal event types are less than the confidence threshold, it is determined that there are no abnormal road events in the current traffic scenario.
[0099] For example, if the confidence threshold is a fixed value of 0.8, and the confidence of all candidate abnormal event types output by the anomaly recognition layer is less than 0.8, then it can be directly determined that there are no abnormal road events.
[0100] In this embodiment, the multimodal data can be temporally aligned and fused using a feature fusion layer in the multimodal model to obtain an environmental perception feature vector. This vector is then processed by an anomaly detection layer to identify road anomaly events, outputting at least one candidate anomaly event type and a confidence level for each type. By comparing this confidence level with a confidence threshold, the existence of a road anomaly event can be accurately determined. In this process, the feature fusion layer effectively integrates multimodal data to construct a high-dimensional, unified environmental perception feature vector. The anomaly detection layer then performs deep semantic reasoning based on this feature vector, enabling the classification, identification, and confidence quantification of various road anomaly events, thereby improving the model's ability to understand complex traffic behaviors.
[0101] Furthermore, the road anomaly event processing method provided in this application introduces a confidence threshold mechanism to quantify the identification results, thereby avoiding false triggers caused by noise or interference. In addition, this mechanism supports fixed or dynamic threshold configuration, which can effectively suppress low-confidence false alarms while ensuring that high-risk events are not missed, significantly improving the accuracy of anomaly event determination.
[0102] Figure 3 This is a flowchart illustrating Embodiment 3 of the method for handling abnormal road events provided in this application. Please refer to... Figure 3 Based on the above embodiment two, when it is determined that there is a road anomaly event, the generation of anomaly event identification results in step S102 may include the following steps:
[0103] S301. For each candidate abnormal event type with a confidence level greater than or equal to the confidence level threshold, determine the abnormal event corresponding to the candidate abnormal event type, and determine the candidate abnormal event type as the abnormal event type of the abnormal event.
[0104] In this step, for each candidate abnormal event type with a confidence level greater than or equal to the confidence level threshold, the abnormal event corresponding to that candidate abnormal event type can be determined, and that candidate abnormal event type can be used as the abnormal event type of the determined abnormal event.
[0105] It should be understood that "event type" usually refers to the possible event category label output by the multimodal model based on the environmental perception feature vector; while "abnormal event" refers to a specific event instance that has been confirmed as actually occurring after being filtered by a confidence threshold.
[0106] For example, when the anomaly detection layer outputs a confidence level of 0.94 for "running a red light" and a confidence level of 0.88 for "road obstacle", and the confidence threshold is 0.8, since both are greater than the confidence threshold of 0.8, two abnormal events can be identified, with the abnormal event types being "running a red light" and "road obstacle" respectively.
[0107] S302. Determine the hazard level based on the degree of interference the abnormal event causes to surrounding traffic participants.
[0108] In this step, for each identified abnormal event, the corresponding hazard level can be determined based on the actual or potential degree of interference with the safety and efficiency of surrounding traffic participants.
[0109] Traffic participants include, but are not limited to, road users such as vehicles, pedestrians, and non-motorized vehicles; the hazard level can be quantified using preset grading standards (such as minor, moderate, and severe) to characterize the severity of the abnormal event, and specific grading rules can be customized according to actual application needs.
[0110] In one specific implementation, if the abnormal event does not interfere with the passage of surrounding traffic participants, the hazard level is determined to be a minor level; if the abnormal event interferes with the passage of surrounding traffic participants but does not pose a safety risk, the hazard level is determined to be a moderate level; if the abnormal event poses a collision risk, causes lane congestion, or causes traffic interruption to surrounding traffic participants, the hazard level is determined to be a severe level.
[0111] For example, if vehicle B briefly picks up or drops off passengers in a non-restricted area without obstructing the view of other vehicles or hindering pedestrians, the hazard level can be determined to be relatively minor.
[0112] For example, if vehicle C is traveling at low speed in a bus lane, causing a bus behind it to slow down, but without triggering a sudden braking or lane-changing conflict, then the hazard level can be determined as moderate.
[0113] For example, if vehicle D breaks down and stops across the driving lane on a main urban road, causing multiple vehicles to swerve and resulting in traffic congestion in a localized area, the hazard level can be determined as severe.
[0114] For example, an "running a red light" incident, because it causes vehicles or pedestrians traveling normally in the opposite direction to have to swerve to avoid it, posing a high risk of collision, can be classified as a severe incident; an "obstacle on the road" incident, because the obstacle is small and located on the side of the road, and does not affect the normal passage of vehicles, can be classified as a minor incident.
[0115] S303. Based on the time of occurrence of the abnormal event, extract data within a preset time window from the multimodal data as key evidence.
[0116] In this step, based on the time of occurrence of the abnormal event, data within a time window centered on that time and extending before and after it for a preset duration can be extracted from the multimodal data as key evidence of the abnormal event.
[0117] The occurrence time of the abnormal event can refer to the point in time when the multimodal model determines that the abnormal event is true, typically corresponding to the frame or timestamp where the confidence level first exceeds the confidence threshold. This moment can be accurately recorded by the system clock or sensor timestamps, serving as a benchmark for event localization and evidence collection.
[0118] Optionally, the preset duration can be flexibly configured according to the type of abnormal event and evidence collection needs. For example, it can be set to 5 seconds before and 3 seconds after the event to cover the entire process of the event's cause, occurrence, and initial impact. For continuous events (such as illegal parking), the backward window can be extended to reflect the event's ongoing status.
[0119] For key evidence, multimodal data or its compressed form within a preset time window can be included. Examples include video streams and LiDAR point cloud sequences, to ensure that the event process is traceable and verifiable.
[0120] In one alternative implementation, the electronic device can automatically trigger the interception of key evidence after determining that an abnormal event has occurred, and synchronously intercept multi-source data for the corresponding time period from the local cache or multimodal data stream according to a preset time window range.
[0121] For example, regarding the "running a red light" incident, if it occurs at 10:15:23, multimodal data from the time window of 10:15:18 to 10:15:26 can be extracted from video data, LiDAR point cloud data, and map navigation data as key evidence. Among them, video data is used to record the behavior of vehicle B running the red light, point cloud data is used to reconstruct the three-dimensional spatial state of the intersection, and map navigation data is used to confirm the location of the intersection and the attributes of the traffic lights.
[0122] For example, regarding the "road obstacle" anomaly event, if it occurs at 14:30:45, multimodal data from the time window of 14:30:40 to 14:30:48 can be extracted from video data, LiDAR point cloud data, and map navigation data as key evidence. Among them, video data is used to observe the appearance of the obstacle and the reaction of surrounding vehicles, point cloud data is used to accurately measure the geometric dimensions and position of the obstacle, and map navigation data is used to determine whether the location belongs to the driving lane or the emergency lane, thus assisting in assessing the impact of the event.
[0123] S304. Based on the type of abnormal event, the severity level, and key evidence, generate the corresponding abnormal event identification results.
[0124] In this step, the identified abnormal event types, corresponding hazard levels, and key evidence can be structurally integrated to generate complete abnormal event identification results.
[0125] For example, for the "running a red light" anomaly event, the generated identification results may include: the anomaly event type is "running a red light", the hazard level is "severe", the occurrence time is 10:15:23, and the key evidence includes the forward-looking video, LiDAR point cloud, and intersection map data from 10:15:18 to 10:15:26; for the "road obstacle" anomaly event, the generated identification results may include: the anomaly event type is "road obstacle", the hazard level is "relatively minor", the occurrence time is 14:30:45, and the key evidence includes the multimodal data packet from 14:30:40 to 14:30:48, used to fully describe the event context.
[0126] Furthermore, based on a preset reporting strategy, the results of abnormal event identification can be reported to the corresponding target receiving end. For example, for a "running a red light" abnormal event, the electronic device can report the abnormal event identification results, including the event type, hazard level, and key evidence, to the traffic management department's management platform via the 5G network, according to the preset reporting strategy; for a "road obstacle" abnormal event, the corresponding abnormal event identification results can be pushed to the road administration department or the city emergency command center via the 5G network, so as to arrange for clearing or issue traffic warnings in a timely manner.
[0127] In this embodiment, after determining the existence of a road anomaly, its specific type and hazard level can be further identified, and relevant multimodal data can be automatically extracted as key evidence to generate a structurally complete anomaly event identification result. By organically integrating event type, hazard level, and key evidence, comprehensive, accurate, and traceable event information can be provided to the target receiver, significantly improving the response efficiency and decision-making accuracy of anomaly event handling.
[0128] Figure 4 This is a flowchart illustrating Embodiment 4 of the method for handling abnormal road events provided in this application. Please refer to... Figure 4 Based on any of the above embodiments, before step S102 is executed, the method for handling the road anomaly event further includes:
[0129] S401: Receive the model deployment request sent by the cloud server.
[0130] In this step, the electronic device can receive a model deployment request from the cloud server. This model deployment request can be a negotiating message, used to inquire whether the vehicle is willing to install or update a specified multimodal model.
[0131] Optionally, the model deployment request can include basic information about the multimodal model to be deployed, such as model name, version number, applicable scenarios, and model size. The cloud server can dynamically push better multimodal models to eligible vehicles based on real-time traffic conditions, model iteration progress, or regional management needs. This mechanism enables on-demand distribution and scenario-based adaptation of model capabilities, while retaining deployment decision-making power on the vehicle side, ensuring that the update process balances driving safety and user control.
[0132] In one specific implementation, the multimodal model is centrally trained on a cloud server. The training dataset comes from a large amount of multimodal data transmitted from vehicles. After anonymization, labeling, and temporal alignment, this data can form training samples with real labels, which are used to supervise the model in learning abnormal behavior patterns in complex traffic scenarios.
[0133] Through end-to-end training, the multimodal model output by the cloud server can receive multi-source heterogeneous multimodal data as input and output structured recognition results, specifically including one or more candidate abnormal event types and their corresponding confidence levels. This model possesses cross-modal fusion and semantic reasoning capabilities, and can effectively identify various road abnormal events such as "running a red light," "road obstacles," and "illegal lane changes."
[0134] S402. In response to the model deployment request, a response message is sent to the cloud server. The response message indicates whether or not the deployment of the multimodal model is agreed upon.
[0135] In this step, after receiving the model deployment request, the electronic device can determine whether to accept the deployment based on the current vehicle operating status and return a response message to the cloud server.
[0136] The vehicle's operating status can include: whether the vehicle is in motion, whether there is sufficient remaining storage space, whether the current network connection is available, and whether the battery power is sufficient.
[0137] Specifically, the response message is used to inform the cloud server whether it agrees to deploy the multimodal model, and may include an explicit instruction of "agree" or "reject"; if rejected, a brief reason may also be attached, such as insufficient storage space, so that the cloud server can process it later.
[0138] S403. When the response message indicates agreement to deploy the multimodal model, receive the multimodal model sent by the cloud server and deploy the multimodal model.
[0139] In this step, if the response message indicates agreement to deploy the multimodal model, the electronic device can download the multimodal model data package from the cloud server and deploy it after the download is complete.
[0140] Optionally, the download process can be based on the currently available wireless communication link and support optimization mechanisms such as resume download or compressed transmission to reduce bandwidth consumption and improve download efficiency.
[0141] Furthermore, after downloading, the integrity and legitimacy of the multimodal model's data package can be verified. This may include comparing the hash value of the model file with the verification information carried in the model deployment request, and verifying whether the digital signature was issued by a trusted cloud server. Only after successful verification is the model allowed to be loaded onto the vehicle's local storage to prevent damage or tampering of the model from affecting the accuracy of anomaly identification and vehicle safety.
[0142] In this embodiment, the system can receive a model deployment request from a cloud server, respond to the request by sending a response message to the cloud server indicating whether it agrees to deploy the multimodal model, and receive the multimodal model from the cloud server and complete the deployment when the response message indicates agreement. This process enables autonomous control of model deployment by the vehicle, taking into account user wishes, preventing unauthorized model deployment from interfering with the operation of the vehicle equipment, and ensuring stable operation of the vehicle equipment.
[0143] Figure 5 This is a flowchart illustrating an example of a method for handling abnormal road events provided in this application. Please refer to [link / reference]. Figure 5 The method may include:
[0144] S501, Strategy Issuance.
[0145] Optionally, relevant departments (such as traffic management departments, urban management departments, road administration departments, or parking lot operators) may send a multimodal model deployment request to the vehicle terminal through the cloud server according to actual management needs. After receiving the request, the vehicle terminal may decide whether to accept the deployment based on the current operating status and user authorization. If it agrees, it will download the multimodal model from the cloud server and complete the local deployment.
[0146] S502, Road Abnormal Behavior Recognition.
[0147] Optionally, after the vehicle-side completes the deployment of the multimodal model, it can analyze the current traffic scenario based on the acquired multimodal data of the vehicle's surrounding environment to determine whether there are any abnormal road events.
[0148] The multimodal data includes at least two of the following: image or video data from vehicle-mounted cameras, point cloud data from millimeter-wave radar or lidar, vehicle attitude data from inertial measurement units, map navigation data, and road scene data. The road scene data is obtained by processing sensor-perceived data using a combination of driver assistance algorithms.
[0149] In one alternative implementation, road abnormal behavior recognition is not limited to vehicle-side execution; it can also be accomplished in conjunction with other data acquisition devices. For example, dashcams, action cameras, or roadside sensing devices can transmit the collected multimodal data to a cloud server with model inference capabilities. The multimodal model deployed on the cloud server processes the multimodal data to complete the identification and subsequent reporting of abnormal events.
[0150] Furthermore, the road abnormal behavior recognition method provided in this application can reuse the existing sensor and algorithm outputs (such as road scene data) of the combined driver assistance system to improve recognition efficiency and robustness, but does not depend on the combined driver assistance system. For vehicles that are not equipped with combined driver assistance hardware, as long as they have basic multimodal perception capabilities (such as cameras and IMUs), they can deploy multimodal models and execute this method to realize the recognition and reporting of road abnormal events.
[0151] S503, Parking Management Telephone Inquiry.
[0152] Optionally, when abnormal behavior is detected in the parking lot and the vehicle is not pre-configured with the corresponding parking lot management's communication address, the contact information of the parking lot management can be automatically queried based on the current geographical location of the vehicle through the application programming interface (API) provided by a third-party service platform or a local database. The contact information may include a message receiving port or a data reporting address for subsequent abnormal event reporting.
[0153] S504. Reporting of abnormal behavior in parking lots.
[0154] Optionally, for confirmed abnormal events occurring in the parking lot, the vehicle can push the structured identification results, including the type of abnormal event, the level of harm, and multimodal key evidence captured centered on the time of the event, to the receiving platform of the queried parking lot management via the vehicle wireless communication link, so as to support the timely verification and management measures taken by the management.
[0155] In one specific implementation, abnormal parking lot behavior may include: illegal parking, occupying private parking spaces, and gasoline vehicles occupying charging spaces.
[0156] S505, reporting of traffic violations.
[0157] Optionally, for abnormal events confirmed to have occurred on public roads, the vehicle can push the corresponding abnormal event identification results to the traffic management department's management platform via the vehicle-mounted wireless communication link to assist in traffic operation management.
[0158] In one specific implementation, traffic violations may include: running red lights, driving over the line, occupying the bus lane, driving against traffic, obscuring license plates, and failing to yield to pedestrians.
[0159] S506, Reporting of Abnormal Traffic Behaviors.
[0160] Optionally, for abnormal events that are confirmed to affect road traffic safety or operational efficiency, the vehicle can push the corresponding abnormal event identification results to the road administration department or the city emergency command center through the vehicle wireless communication link to assist in road maintenance and emergency response.
[0161] In one specific implementation, abnormal traffic behaviors may include: large potholes in the road, road surface collapse, vehicles that have broken down and are parked sideways, and obstacles such as tires on the road.
[0162] S507. Reporting of Abnormal Urban Management Behaviors.
[0163] Optionally, for abnormal events confirmed to occur in public areas and affect urban appearance or public safety, the vehicle can push the corresponding abnormal event identification results to the receiving platform of the urban management department through the vehicle wireless communication link to assist in urban operation and governance.
[0164] In one specific implementation, abnormal urban management behaviors may include: illegal parking, illegal street vending, illegal operation, and illegal dumping.
[0165] Optionally, if the confidence level of all candidate event types in the abnormal event identification results is lower than the preset threshold, the identification result can be discarded and the subsequent reporting process can be avoided to prevent invalid information from interfering with the management platform.
[0166] The method for handling abnormal road events provided in this application embodiment can be referred to the technical solution shown in the above method embodiment for specific execution process. The implementation principle and beneficial effects are similar, and will not be repeated here.
[0167] Figure 6 This is a schematic diagram of the road anomaly event processing device provided in an embodiment of this application. Please refer to... Figure 6 The road abnormality event handling device 10 includes:
[0168] The acquisition module 11 is used to acquire multimodal data of the vehicle's surrounding environment when the vehicle is started.
[0169] The first processing module 12 is used to perform road anomaly event identification processing on multimodal data through a multimodal model, and when it is determined that there is a road anomaly event, it generates anomaly event identification results, including anomaly event type, hazard level and key evidence.
[0170] The second processing module 13 is used to report the abnormal event identification results to the corresponding target receiving end according to the preset reporting strategy.
[0171] The road abnormality event processing device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0172] In one possible implementation, the multimodal model includes a feature fusion layer and an anomaly detection layer; the first processing module 12 is specifically used for:
[0173] The feature fusion layer performs temporal alignment and feature fusion on multimodal data to obtain environmental perception feature vectors;
[0174] The anomaly detection layer performs road anomaly event identification processing on the environmental perception feature vector, outputting at least one candidate anomaly event type and the confidence level corresponding to each candidate anomaly event type.
[0175] If the confidence level of any candidate abnormal event type is greater than or equal to the confidence level threshold, then it is determined that there is a road abnormal event.
[0176] If the confidence level for each candidate abnormal event type is less than the confidence level threshold, then it is determined that there are no abnormal road events.
[0177] In one possible implementation, the first processing module 12 is specifically used for:
[0178] For each candidate abnormal event type with a confidence level greater than or equal to the confidence level threshold, determine the abnormal event corresponding to the candidate abnormal event type, and determine the candidate abnormal event type as the abnormal event type of the abnormal event.
[0179] The severity level is determined based on the degree of disruption that the abnormal event causes to the passage of surrounding traffic participants;
[0180] Based on the time of occurrence of the abnormal event, data within a preset time window is extracted from the multimodal data as key evidence;
[0181] Based on the type of abnormal event, the severity level, and key evidence, corresponding abnormal event identification results are generated.
[0182] In one possible implementation, the first processing module 12 is specifically used for:
[0183] If the abnormal event does not interfere with the passage of surrounding traffic participants, the hazard level is determined to be relatively minor.
[0184] If an abnormal event causes traffic disruption to surrounding road users but does not pose a safety risk, the hazard level is determined to be moderate.
[0185] If an abnormal event poses a collision risk to surrounding road users, causes lane congestion, or disrupts traffic, the hazard level is determined to be severe.
[0186] In one possible implementation, the second processing module 13 is specifically used for:
[0187] The pre-stored reporting strategy table is queried according to the type of abnormal event to determine the corresponding target receiving end;
[0188] The abnormal event identification results are reported to the target receiving end through the vehicle-mounted wireless communication link.
[0189] In one possible implementation, road abnormal events include at least one of the following categories: road violation events, traffic abnormal events, urban management abnormal events, and parking abnormal events;
[0190] Multimodal data includes at least two of the following: image or video data from vehicle-mounted cameras, point cloud data from millimeter-wave radar or lidar, vehicle attitude data from inertial measurement units, map navigation data, and road scene data. The road scene data is obtained by processing sensor perception data through a combination of driver assistance algorithms.
[0191] In one possible implementation, the first processing module 12 is further configured to:
[0192] Receive model deployment requests sent by the cloud server;
[0193] In response to the model deployment request, a response message is sent to the cloud server. The response message indicates whether the deployment of the multimodal model is agreed upon.
[0194] When the response message indicates agreement to deploy the multimodal model, the system receives the multimodal model sent by the cloud server and deploys the multimodal model.
[0195] The road abnormality event processing device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0196] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 7 The electronic device 20 provided in this embodiment can be a controller in a vehicle. For example, it could be a driver domain controller, a cockpit domain controller, or a vehicle controller. The electronic device 20 may include at least one processor 21 and a memory 22. Optionally, the device 20 may also include a communication component 23. The processor 21, memory 22, and communication component 23 are connected via a bus 24.
[0197] In the specific implementation process, at least one processor 21 executes computer execution instructions stored in memory 22, causing at least one processor 21 to perform the above-described method.
[0198] The specific implementation process of processor 21 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0199] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0200] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0201] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0202] This application also provides a vehicle, including: a vehicle body, and as follows: Figure 7 The electronic device shown is used to implement the road abnormality event handling method in the above embodiments. Its implementation principle and beneficial effects are similar, and will not be described again here.
[0203] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0204] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0205] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0206] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0207] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0209] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0210] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0211] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0212] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for handling abnormal road events, characterized in that, include: Acquire multimodal data of the vehicle's surrounding environment while the vehicle is running; The multimodal data is processed by a multimodal model to identify road anomalies. When a road anomaly is identified, an anomaly identification result is generated, which includes the anomaly type, hazard level, and key evidence. According to the preset reporting strategy, the abnormal event identification results are reported to the corresponding target receiving end.
2. The method according to claim 1, characterized in that, The multimodal model includes a feature fusion layer and an anomaly detection layer; The process of identifying road anomaly events using a multimodal model on the multimodal data includes: The multimodal data is temporally aligned and feature fused through the feature fusion layer to obtain an environmental perception feature vector; The anomaly recognition layer performs road anomaly event recognition processing on the environmental perception feature vector, and outputs at least one candidate anomaly event type and the confidence level corresponding to each candidate anomaly event type. If the confidence level corresponding to any candidate abnormal event type is greater than or equal to the confidence level threshold, then it is determined that the road abnormal event exists. If the confidence level corresponding to each candidate abnormal event type is less than the confidence level threshold, then it is determined that there is no abnormal road event.
3. The method according to claim 2, characterized in that, The generated abnormal event identification results include: For each candidate abnormal event type with a confidence level greater than or equal to the confidence level threshold, an abnormal event corresponding to the candidate abnormal event type is determined, and the candidate abnormal event type is determined as the abnormal event type of the abnormal event. The hazard level is determined based on the degree of interference the abnormal event causes to the passage of surrounding traffic participants; Based on the time of occurrence of the abnormal event, data within a preset time window is extracted from the multimodal data as the key evidence; Based on the abnormal event type, the hazard level, and the key evidence, a corresponding abnormal event identification result is generated.
4. The method according to claim 3, characterized in that, The determination of the hazard level based on the degree of interference of the abnormal event with the passage of surrounding traffic participants includes: If the abnormal event does not interfere with the passage of surrounding traffic participants, then the hazard level is determined to be a relatively minor level. If the abnormal event causes traffic disruption to the surrounding traffic participants but does not pose a safety risk, then the hazard level is determined to be of the general level. If the abnormal event poses a collision risk, causes lane congestion, or disrupts traffic to surrounding road users, the hazard level is determined to be severe.
5. The method according to any one of claims 1-4, characterized in that, The step of reporting the abnormal event identification result to the corresponding target receiving end according to the preset reporting strategy includes: The corresponding target receiving end is determined by querying the pre-stored reporting strategy table based on the abnormal event type. The abnormal event identification results are reported to the target receiving end via the vehicle-mounted wireless communication link.
6. The method according to any one of claims 1-4, characterized in that, The abnormal road events include at least one of the following categories: road violation events, traffic abnormal events, urban management abnormal events, and parking abnormal events; The multimodal data includes at least two of the following: image data or video data from vehicle-mounted cameras, point cloud data from millimeter-wave radar or lidar, vehicle attitude data from inertial measurement units, map navigation data, and road scene data. The road scene data is obtained by processing sensor perception data using a combination of driver assistance algorithms.
7. The method according to any one of claims 1-4, characterized in that, The method further includes: Receive model deployment requests sent by the cloud server; In response to the model deployment request, a response message is sent to the cloud server, the response message indicating whether the deployment of the multimodal model is agreed upon; When the response message indicates agreement to deploy the multimodal model, the system receives the multimodal model sent by the cloud server and deploys the multimodal model.
8. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.
9. A vehicle, characterized in that, include: The vehicle body, and the electronic device as described in claim 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.