A method, device, and storage medium for identifying high-risk scenarios in intelligent vehicle operation based on deep learning.

Abstract

This invention relates to the field of risk identification technology, specifically to a method, device, and storage medium for identifying high-risk scenarios in intelligent vehicle operation based on deep learning. The method includes collecting multimodal data of intelligent vehicle operation scenarios; determining the spatiotemporal evolution of risk features based on a fused feature sequence; updating a global risk feature map based on the spatiotemporal evolution; determining risk markers for the operation scenario based on the updated global risk feature map; and distinguishing between static and dynamic risk regions within the risk markers based on the unupdated global risk feature map and the fused feature sequence, thereby obtaining high-risk candidate regions corresponding to the operation scenario. This invention, by collecting multimodal data in real time and combining it with deep learning for data fusion and encoding, can accurately identify high-risk scenarios in intelligent vehicle operation, dynamically assess and predict potential risks, thus providing more accurate decision support for intelligent vehicles and preventing possible accidents.

Description

Technical Field

[0001] This application relates to the field of risk identification technology, and in particular to a method, device and storage medium for identifying high-risk scenarios in the operation of intelligent vehicles based on deep learning. Background Technology

[0002] Intelligent vehicles, also known as autonomous vehicles, are cars that combine artificial intelligence, sensors, communication technologies, and advanced control systems. They can achieve autonomous driving, environmental perception, decision-making, and action execution to varying degrees, reducing reliance on drivers and even eliminating the need for driver intervention in fully autonomous driving scenarios.

[0003] Currently, with the rapid development of autonomous driving technology, intelligent vehicles are gradually becoming an important part of the future transportation system. However, in complex and dynamic traffic environments, how to ensure the safety of intelligent vehicles has become an urgent problem to be solved. Traditional vehicle safety systems often rely on preset rules and sensor data, which are difficult to fully cope with complex risk scenarios. In actual driving, intelligent vehicles face constantly changing environments and complex traffic scenarios. The environment around the vehicle and the motion state of the target will change over time. Traditional static decision-making systems cannot cope with the complexity of this spatiotemporal evolution. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device and storage medium for identifying high-risk scenarios in intelligent vehicle operation based on deep learning, so as to solve the problem that existing vehicle safety systems often rely on preset rules and sensor data, which are difficult to comprehensively cope with complex risk scenarios.

[0005] To achieve the above objectives, this application provides the following technical solution: A deep learning-based method for identifying high-risk operating scenarios of intelligent vehicles includes: Multimodal data of intelligent vehicle operation scenarios are collected, including visual images, LiDAR point clouds, V2X communication data, and high-precision map information; multimodal feature fusion and encoding are performed on the multimodal data to obtain a fused feature sequence; The spatiotemporal evolution of risk features is determined based on the fused feature sequence, and the global risk feature map is updated based on the spatiotemporal evolution, wherein the global risk feature map is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. Based on the updated global risk feature map, the risk markers of the operation scenario are determined. Based on the global risk feature map before the update and the fused feature sequence, the static risk regions and dynamic risk regions in the risk markers are distinguished to obtain the high-risk candidate regions corresponding to the operation scenario. Based on the high-risk candidate areas corresponding to the operating scenario, the reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario are determined. Based on the location distribution of the reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene, the risk attention index of each target in the current scene is determined; based on the risk attention index of each target in the current scene, the risk level of each target in the current operating scene is determined.

[0006] Preferably, the multimodal data is subjected to multimodal feature fusion and encoding to obtain a fused feature sequence, including: The multimodal data is categorized into visual sub-data, radar sub-data, V2X sub-data, and map sub-data based on data type. Feature extraction is performed on each sub-data set to obtain the modal feature vector for each sub-data set; The correlation degree between the modal feature vector of each sub-data and the modal feature vectors of the other sub-data is determined to obtain the cross-modal feature mapping network of the multimodal data; Obtain the attention weight of each feature channel in the cross-modal feature mapping network, and determine the fusion weight of each modal feature based on the attention weight of each feature channel; Multiple modal features corresponding to each data segment in the multimodal data are obtained, and the fusion features of each data segment in the multimodal data are determined based on the fusion weights and attention weights of the modal features. The fusion feature sequence of the multimodal data is determined based on the fusion features of each data segment in the multimodal data.

[0007] Preferably, determining the spatiotemporal evolution of risk features based on the fused feature sequence includes: Determine the effective sensing range of the multimodal data, and construct a spatiotemporal grid cell set based on this range; The correlation between each feature in the fused feature sequence in the time and space dimensions is obtained, and all units including risk correlation in the spatiotemporal grid unit set are obtained. The intersection of the feature evolution trajectory and the feature node in the fused feature sequence is obtained. The unit is used to characterize a local feature set initialized based on the fused feature sequence, which represents a spatiotemporal range. The unit records the feature weight, spatiotemporal coordinates and correlation strength. Determine the risk evolution distance from each unit to the intersection point of the feature node, and based on the risk evolution distance of the unit, determine the risk space labeling of the multimodal data as low-risk area, medium-risk area and high-risk area, so as to obtain the spatiotemporal evolution of risk features.

[0008] Preferably, updating the global risk feature map based on the spatiotemporal evolution includes: Locate units in the spatiotemporal evolution within the global risk feature map; For the identified unit, the global risk value of the unit in the global risk feature map is updated based on the risk evolution distance of the unit in the spatiotemporal evolution. For units that are not found, the units are added to the global risk feature map, and the risk evolution distance of the units in the spatiotemporal evolution is used as their global risk value in the global risk feature map.

[0009] Preferably, based on the global risk feature map before the update and the fused feature sequence, static risk regions and dynamic risk regions in the risk markers are distinguished to obtain high-risk candidate regions corresponding to the operating scenario, including: The feature changes in the fused feature sequence are compared with the feature baseline of the global risk feature map before the update to obtain dynamic risk elements; The dynamic risk elements are associated with the risk areas in the risk markers to obtain static risk areas and dynamic risk areas; High-risk candidate regions are selected from static and dynamic risk regions based on a preset high-risk threshold.

[0010] Preferably, based on the high-risk candidate areas corresponding to the operating scenario, the reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario are determined, including: For any road segment, the high-risk candidate area of ​​the road segment is used as a reference point, and the road network unit in which the reference point is located in the topological road network is used as a reference road network unit. Among them, environmental feature points and historical risk data of all road sections are obtained in advance, and the feature points are processed by the road network topology construction algorithm to obtain the topology road network corresponding to the operation scenario. The key nodes in the topology road network are environmental feature points. The topology road network is obtained by the road network topology construction algorithm based on high-precision map. The road network topology construction algorithm is used to fuse road centerline, intersection and lane line features to generate road network units. The area formed by the reference road network unit in the topological road network and the road network unit directly connected to the reference road network unit is designated as the key monitoring area of ​​the road segment. Environmental feature points and historical accident sites in the key monitoring areas will be used as reference risk points for key monitoring of the road sections.

[0011] Preferably, based on the location distribution of the reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene, the risk attention index of each target in the current scene is determined, including: For any key monitoring area, adjacent road network units belonging to the same risk type in the key monitoring area are merged to obtain a local risk area; Determine the correlation degree of each reference risk point in each risk local area to obtain the first risk characteristic value of the corresponding reference risk point; Determine the shortest distance between each reference risk point and the current location of the intelligent vehicle to obtain the second risk characteristic value of the corresponding reference risk point; The local risk complexity of each reference risk point is determined based on the first and second risk characteristic values ​​of each reference risk point. Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained; The risk environment complexity of the current road segment is obtained by weighting the local risk complexity of all reference risk points. The risk environment complexity and the risk exposure level are weighted to obtain the risk correlation level of the current scenario; Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained; For any target in the current scene, obtain the target's motion speed, acceleration, and turning angular velocity. Based on the target's motion parameters and the degree of risk correlation, obtain the target's risk attention index.

[0012] Preferably, the risk exposure level of the intelligent vehicle in the current scene is obtained based on the positional relationship between environmental feature points and target motion feature points in the current scene, including: Sort the spatial distances between each environmental feature point and each target motion feature point in the current scene to obtain a distance sequence; The distance sequence is divided into multiple risk intervals; the matching degree between the corresponding environmental feature points and the reference risk points in each risk interval is determined to obtain the risk correlation degree of each interval. The intervals with risk correlation greater than a preset correlation threshold are divided into high correlation intervals, and the intervals with risk correlation less than or equal to the preset correlation threshold are divided into low correlation intervals. The target motion feature points in the current scene and the corresponding environmental feature points in the highly correlated intervals are grouped into the same cluster as a high-risk cluster; the corresponding environmental feature points in the low-correlation intervals are grouped into the same cluster as a low-risk cluster. The difference in spatial distribution variance between the high-risk cluster and the low-risk cluster is normalized to obtain the risk exposure level of the intelligent vehicle in the current scenario.

[0013] To achieve the above objectives, this application also provides the following technical solutions: A deep learning-based intelligent vehicle operation high-risk scenario recognition device, applicable to the aforementioned deep learning-based intelligent vehicle operation high-risk scenario recognition method, includes: The feature fusion module is used to collect multimodal data of intelligent vehicle operation scenarios, wherein the multimodal data includes visual images, LiDAR point clouds, V2X communication data and high-precision map information; the multimodal data is fused and encoded to obtain a fused feature sequence; The graph modeling module is used to determine the spatiotemporal evolution of risk features based on the fused feature sequence, and to update the global risk feature graph based on the spatiotemporal evolution, wherein the global risk feature graph is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. The region segmentation module is used to determine the risk markers of the operating scenario based on the updated global risk feature map, and to distinguish the static risk regions and dynamic risk regions in the risk markers based on the global risk feature map before the update and the fused feature sequence, so as to obtain the high-risk candidate regions corresponding to the operating scenario. The risk monitoring module is used to determine the reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario based on the high-risk candidate areas corresponding to the operating scenario. The risk identification module is used to determine the risk attention index of each target in the current scene based on the location distribution of the reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene; and to determine the risk level of each target in the current operating scene based on the risk attention index of each target in the current scene.

[0014] To achieve the above objectives, this application also provides the following technical solutions: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep learning-based intelligent vehicle operation high-risk scenario identification method.

[0015] This application, through real-time acquisition of multimodal data and fusion and encoding of the data using deep learning, can accurately identify high-risk scenarios in the operation of intelligent vehicles. It can dynamically assess and predict potential risks, thus providing more accurate decision support for intelligent vehicles and preventing possible accidents. Furthermore, through risk labeling, it can effectively distinguish between static and dynamic risk areas, providing more detailed risk analysis for intelligent vehicle operation. It can identify which risk factors are caused by fixed environmental conditions (such as traffic signs and road conditions) and which are caused by surrounding dynamic elements (such as pedestrians and other vehicles). This distinction allows the vehicle to take appropriate countermeasures based on specific situations. Moreover, through spatiotemporal evolution analysis, the global risk feature map is updated, and the system can adjust according to real-time scene changes, the vehicle's surrounding environment, and the motion state of targets, enabling intelligent vehicles to respond more promptly and accurately to different operating environments. Finally, by integrating multiple data sources and combining deep learning technology, it can enhance the recognition and prediction capabilities of the autonomous driving system in complex and dynamic environments, not only improving system reliability but also enabling the vehicle to respond more quickly to sudden risk scenarios, ensuring driving safety. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of an embodiment of a deep learning-based method for identifying high-risk scenarios in intelligent vehicle operation, as described in this application. Figure 2 This is an architecture diagram of one embodiment of a deep learning-based intelligent vehicle operation high-risk scene recognition device according to this application.

[0017] Attached reference numerals: 1. Feature fusion module; 2. Map modeling module; 3. Region division module; 4. Risk monitoring module; 5. Risk identification module. Detailed Implementation

[0018] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the present invention proposes a method for identifying high-risk scenarios in intelligent vehicle operation based on deep learning, comprising: S1. Collect multimodal data of intelligent vehicle operation scenarios, including visual images, LiDAR point clouds, V2X communication data, and high-precision map information; perform multimodal feature fusion and encoding on the multimodal data to obtain a fused feature sequence; S2. Determine the spatiotemporal evolution of risk features based on the fused feature sequence, and update the global risk feature map based on the spatiotemporal evolution. The global risk feature map is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. S3. Based on the updated global risk feature map, determine the risk markers of the operation scenario. Based on the global risk feature map before the update and the fused feature sequence, distinguish between static risk areas and dynamic risk areas in the risk markers to obtain high-risk candidate areas corresponding to the operation scenario. S4. Based on the high-risk candidate areas corresponding to the operating scenario, determine the reference risk points that intelligent vehicles need to focus on monitoring in the topological road network corresponding to the operating scenario; S5. Based on the location distribution of reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene, determine the risk attention index of each target in the current scene; and determine the risk level of each target in the current operating scene based on the risk attention index of each target in the current scene.

[0020] In this invention, visual images are from vehicle-mounted cameras, used to identify road conditions, pedestrians, other vehicles, etc.; LiDAR (Light Detection and Ranging) obtains 3D point cloud data through laser ranging to help the car perceive its surroundings; V2X communication data is communication data between vehicles and other vehicles or infrastructure, transmitting traffic signals, road condition information, etc.; high-precision map information can provide detailed road network information, including lanes, intersections, traffic signs, etc. For example, on a highway, a car detects a vehicle ahead through a camera, LiDAR confirms the vehicle's precise location and distance, the V2X system receives a signal that the vehicle ahead is moving slowly, and the high-precision map shows that there is a curve ahead; through the fusion of these data, the system generates a feature sequence containing all environmental features; based on the fused feature sequence, the system needs to analyze the risks that may occur at different times and spatial points, called spatiotemporal evolution; by analyzing this evolution, the system can update a global risk feature map, reflecting the risk status of different areas at different times; as time progresses... As time progresses, changes in vehicle and pedestrian flow data update the risk feature map of the area. The updated global risk feature map can be used to mark static risk areas (such as roadblocks and road depressions) and dynamic risk areas (such as other vehicles and pedestrians) in the current operating scenario. By comparing the maps before and after the update, the system can identify high-risk candidate areas that require special attention. For example, in urban roads, there is a long-standing pothole at the edge of the lane (static risk area), while a car suddenly slows down ahead (dynamic risk area). The system marks these as high-risk areas by comparing them with the previous map. Based on the high-risk areas, the system further analyzes the reference risk points in the topological road network (such as roads and intersections) that need to be monitored. According to the location of the reference risk points, the location and movement state of the target objects (such as other vehicles and pedestrians), the system calculates the risk attention index, which is the degree of impact of a target on the risk in the current scenario. Finally, the system classifies the risk level of each target based on the risk attention index.

[0021] In an optional embodiment, multimodal feature fusion and encoding are performed on the multimodal data to obtain a fused feature sequence, including: Multimodal data is categorized into visual sub-data, radar sub-data, V2X sub-data, and map sub-data based on data type. Feature extraction is performed on each sub-data set to obtain the modal feature vector for each sub-data set; Determine the correlation degree between the modal feature vector of each sub-data and the modal feature vectors of the remaining sub-data to obtain the cross-modal feature mapping network for multimodal data; Obtain the attention weight of each feature channel in the cross-modal feature mapping network, and determine the fusion weight of each modal feature based on the attention weight of each feature channel; Multiple modal features corresponding to each data segment in the multimodal data are obtained, and the fusion features of each data segment in the multimodal data are determined based on the fusion weights and attention weights of the modal features. The fusion feature sequence of multimodal data is determined based on the fusion features of each data segment in the multimodal data.

[0022] It's important to note that each type of sub-data may not be directly usable for analysis in its original form. Therefore, feature extraction is necessary for each sub-data, extracting features that represent its key information to obtain the modal feature vector for each sub-data. Different modal feature vectors correspond to different data types (e.g., image features, radar point cloud features, etc.). To understand the correlation between different modal data, the correlation degree between the modal feature vector of each sub-data and the feature vectors of other sub-data needs to be calculated. By calculating these correlation degrees, a cross-modal feature mapping network can be established, which helps the system understand the relationships between different modal data. For example, suppose visual sub-data (pedestrians in an image) and radar sub-data (objects in a radar point cloud) have a certain spatial correlation. Through the cross-modal feature mapping network, the system can understand the correspondence between the pedestrian positions in the visual image and the objects scanned by the radar, thereby better identifying pedestrians in the current scene. In multimodal data, different modal data contribute differently to risk prediction and decision-making. By calculating attention weights, the system can determine which feature channels (e.g., visual features, radar features) are more important in the current task. Based on these attention weights, for each... Each modal feature is assigned a fusion weight, representing the importance of that modality in the final fusion. For example, in complex traffic scenarios, visual sub-data may be more important (e.g., recognizing traffic signs and pedestrians), while in adverse weather conditions, radar data may be more critical (e.g., accurately identifying road obstacles). Based on an attention mechanism, the system assigns different fusion weights to visual and radar features respectively. Once each modal feature has its corresponding fusion weight and attention weight, the system can perform weighted fusion of data from different modalities to obtain the fusion feature of each data segment. The fusion feature is the weighted sum of the modal features, representing... This data segment provides comprehensive information across various dimensions. For example, in a scenario on an urban road, the system fuses image information (such as the positions of pedestrians and other vehicles) and radar information (such as the precise distance and speed of objects) around the vehicle based on the weights of visual and radar sub-data. The resulting fused features will help intelligent vehicles understand the scenario more comprehensively. By fusing features from multiple data segments (data blocks in time or space), a complete fused feature sequence can be formed. This feature sequence contains comprehensive information from multiple data segments across multiple modalities, which can be used for subsequent scenario understanding and risk prediction.

[0023] In an optional embodiment, determining the spatiotemporal evolution of risk features based on the fused feature sequence includes: Determine the effective sensing range of multimodal data, and construct a spatiotemporal grid cell set based on this range; The correlation between each feature in the fused feature sequence in the time and space dimensions is obtained, and all units including risk correlation in the spatiotemporal grid unit set are obtained. The intersection of the feature evolution trajectory and the feature node in the fused feature sequence is obtained. The unit is used to represent a local feature set in a spatiotemporal range based on the fused feature sequence initialization. The unit records the feature weight, spatiotemporal coordinates and correlation strength. The risk evolution distance from each unit to the intersection point of the feature node is determined. Based on the risk evolution distance of the unit, the risk space corresponding to the multimodal data is labeled as low-risk area, medium-risk area and high-risk area, so as to obtain the spatiotemporal evolution of risk features.

[0024] It should be noted that in the analysis of multimodal data, the first step is to define an "effective sensing range," which is the range within which the sensor can effectively sense and extract features. This range may be determined by the working range of devices such as cameras, radar, and V2X data. Based on this sensing range, a set of spatiotemporal grid cells is constructed. Each spatiotemporal grid cell represents a local feature set and records the feature weights, spatiotemporal coordinates, and correlation strength with other cells. Each grid cell has spatiotemporal coordinates (representing its spatial location and temporal index) and the risk characteristics of that area. The fused feature sequence is a feature set obtained through feature extraction and fusion of multimodal data. In both temporal and spatial dimensions, there may be certain correlations between fused features. By analyzing these correlations, risk information in spatiotemporal grid cells can be matched and correlated with feature sequences to further identify risk areas. For example, at a certain moment, a smart car's camera and radar simultaneously detect a car braking suddenly and a pedestrian crossing the road ahead. These sensor information come from different sensor modalities (vision and radar), are synchronous in time, and are located in the same spatial area. This information is combined through a fused feature sequence, and by analyzing the temporal and spatial correlations of each feature, it can provide insights for subsequent risk prediction. Clues; In the analysis of multimodal data, features evolve with changes in time and space; feature evolution trajectories describe the dynamic changes of these features over time and space; by obtaining the intersection points of feature nodes, we can find the intersection points of these evolution trajectories with the spatiotemporal grid cell set, thereby determining which areas have potential risks; for example: suppose a car is traveling at high speed, and the system detects a vehicle braking suddenly ahead; this sudden braking action is a feature evolution trajectory; by analyzing the vehicle's braking information and the fused feature sequence of the intelligent vehicle, we can determine the intersection point of this evolution trajectory with the grid cell where the intelligent vehicle is currently located; for example, the features of sudden braking may An intersection with a certain spatiotemporal cell ahead indicates that the intelligent vehicle is about to enter a potentially high-risk area. Once the intersection of the feature evolution trajectory and the spatiotemporal grid cell is determined, the "risk evolution distance" from each grid cell to the feature node intersection can be calculated. This distance represents the spatiotemporal relationship between a feature event (such as sudden braking, obstacle appearance, etc.) and a certain grid cell, reflecting the risk level of that grid cell. Based on the risk evolution distance, the grid cells in the entire spatiotemporal range can be divided into different risk zones (low, medium, and high risk). By calculating the risk evolution distance of these cells and comparing it with a predefined threshold, each grid cell can be risk-marked.For example, assuming a traffic accident occurs 200 meters ahead, the intelligent vehicle will use this information to assess the risk of different areas: areas farther from the accident site (e.g., beyond 300 meters) will be marked as low-risk areas, areas closer to the accident site (e.g., within 100 meters) will be marked as high-risk areas, and areas between 100 and 200 meters may be marked as medium-risk areas. Finally, the risk characteristic evolution of all spatiotemporal grid units in both time and space dimensions will be integrated into a complete spatiotemporal evolution map. This can be used to dynamically adjust the intelligent vehicle's driving strategy to ensure driving safety.

[0025] In an optional embodiment, updating the global risk feature map based on spatiotemporal evolution includes: Locate units in the spatiotemporal evolution within the global risk feature map; For the identified units, the global risk value of the units in the global risk feature map is updated based on the risk evolution distance of the units in the spatiotemporal evolution. For units that are not found, the units are added to the global risk feature map, and the risk evolution distance of the unit in the spatiotemporal evolution is used as its global risk value in the global risk feature map.

[0026] It should be noted that during the spatiotemporal evolution process, it may be necessary to search for units at a specific moment and spatial location, which are the focus of the current risk analysis. For example, intelligent transportation systems may identify certain specific spatiotemporal areas based on real-time sensing data (such as traffic accidents, weather changes, etc.) and query the risk data of these areas in the global risk feature map. For example, suppose an intelligent transportation system in a city is monitoring the real-time risk situation of different road segments; at a certain moment, the system detects a traffic accident on the road ahead, and this area is located in the city center; the system will search whether there is already relevant risk data for this area in the global risk map; if there is... Once a corresponding spatiotemporal unit is found in the spectrum, the system updates its global risk value based on the unit's "risk evolution distance." Risk evolution distance is a measure describing the degree of change and propagation of a feature or event in time and space, typically reflecting the impact of a risk event on the region. This update can more accurately reflect the current risk status of the unit. If no corresponding spatiotemporal unit is found in the global risk feature map, it means that the unit may be a newly emerging region or a region that has not been previously monitored. In this case, the system adds the unit to the global risk map and assigns it an initial global risk value based on its risk evolution distance.

[0027] In an optional embodiment, static and dynamic risk regions in the risk markers are distinguished based on the global risk feature map before the update and the fused feature sequence to obtain high-risk candidate regions corresponding to the operating scenario, including: By comparing the feature changes in the fused feature sequence with the feature baseline of the global risk feature map before the update, dynamic risk elements are obtained. By associating dynamic risk elements with risk areas in risk markers, static and dynamic risk areas are obtained. High-risk candidate regions are selected from static and dynamic risk regions based on a preset high-risk threshold.

[0028] It should be noted that comparing the feature changes in the fused feature sequence with the feature baseline in the global risk feature map before the update aims to identify feature changes. The feature baseline in the global risk feature map represents the risk status of the area under normal conditions. Dynamic risk elements refer to risk fluctuations caused by feature changes, usually reflecting the increase or decrease in risk due to certain factors (such as sudden events, disasters, etc.). By comparing the fused feature sequence (such as traffic flow changes) with the baseline (risk value under normal flow) in the global risk feature map, the system can detect changes in the traffic risk characteristics of the area and extract "dynamic risk elements," such as an increased probability of traffic accidents. In the system, risk markers represent known risk areas, such as areas where natural disasters or traffic accidents have occurred in the past. Dynamic risk elements are associated with these static risk areas, thus dividing static and dynamic risk areas. Static risk areas refer to those areas where the risk value is relatively stable or whose risk changes can be predicted based on historical data. Dynamic risk areas are those areas where the risk changes significantly due to current events or feature changes. It is dynamic; after identifying static and dynamic risk areas, the system filters based on preset high-risk thresholds (e.g., if the risk value of an area exceeds a certain set standard, the area is considered a high-risk area). The purpose of this step is to identify potential high-risk areas from static and dynamic areas, that is, areas with high risk levels that require special attention. The system detects a sudden increase in traffic flow in a certain area of ​​the city at a certain moment, and the real-time monitored traffic accident rate also increases. By comparing the baseline data of this area before (such as traffic flow and accident rate under normal conditions), the system identifies a significant change and extracts the "dynamic risk element," that is, the probability of accidents occurring in this area has increased. The system compares the associated risk areas with the already marked risk areas. The city center area was previously marked as a static risk area due to a high number of traffic accidents, but with the change in traffic flow, this area is further marked as a dynamic risk area. Based on preset high-risk thresholds (e.g., a risk value exceeding 8 is considered high-risk), the system filters the city center area and certain old building areas as high-risk candidate areas, and then prompts management personnel to intervene (such as increasing traffic control, strengthening building safety inspections, etc.).

[0029] In an optional embodiment, reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario are determined based on high-risk candidate areas corresponding to the operating scenario, including: For any road segment, the high-risk candidate area of ​​the road segment is used as the benchmark point, and the road network unit in which the benchmark point is located in the topological road network is used as the reference road network unit. Among them, environmental feature points and historical risk data of all road sections are obtained in advance, and the feature points are processed by the road network topology construction algorithm to obtain the topology road network corresponding to the operation scenario. The key nodes in the topology road network are environmental feature points. The topology road network is obtained by the road network topology construction algorithm based on high-precision map. The road network topology construction algorithm is used to fuse road centerline, intersection and lane line features to generate road network units. The area formed by the reference road network unit in the topological road network and the road network unit directly connected to the reference road network unit is regarded as the key monitoring area of ​​the road segment; Environmental feature points and historical accident sites in key monitoring areas will be used as reference risk points for key monitoring of road sections.

[0030] It should be noted that for any road segment, the system first identifies high-risk candidate areas; these areas are considered as reference points. Then, through the road network topology, the system locates the road segment unit where the reference point is located and treats it as a reference road network unit. For example, suppose a city's intelligent transportation system detects a high traffic accident rate on a main road; the system considers this road segment a "high-risk candidate area." To perform more accurate analysis, the system collects environmental feature points and historical risk data (such as traffic accident frequency, road conditions, etc.) for all road segments in advance. Then, it uses a "road network topology construction algorithm" to process these feature points, thereby obtaining a topology road network that conforms to the current scenario. The road network topology construction algorithm combines information such as road centerlines, intersections, and lane lines to generate a complete road network topology. The key nodes in the topology are these environmental feature points, which help describe the road structure and risk distribution. For example, when constructing the road network topology, the system may identify the intersections of each road based on a high-precision map. Key nodes such as lane dividers will be included in the road network topology as "environmental feature points." The system defines reference road network units and other road network units directly connected to them as key monitoring areas for road segments. This is to monitor other road segments that may be affected by the reference point (high-risk area) and ensure that potential risks can be detected in a timely manner. For example, if the reference road network unit is an intersection with frequent traffic accidents, the system will select several directly connected roads around the intersection as "key monitoring areas." For example, main roads and side roads next to the intersection will also be monitored to detect potential risks caused by changes in traffic flow or road conditions in a timely manner. In the key monitoring areas, the system will focus on environmental feature points and historical accident sites. Environmental feature points include factors that may affect traffic safety, such as traffic lights, intersections, and school zones. Historical accident sites refer to areas where multiple accidents have occurred in the past. The system will monitor these points as "reference risk points" to detect and warn of potential traffic risks as early as possible.

[0031] In an optional embodiment, based on the location distribution of reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene, a risk concern index for each target in the current scene is determined, including: For any key monitoring area, adjacent road network units belonging to the same risk type within the key monitoring area are merged to obtain a local risk area; Determine the correlation degree of each reference risk point in each risk local area to obtain the first risk characteristic value of the corresponding reference risk point; Determine the shortest distance between each reference risk point and the current location of the intelligent vehicle to obtain the second risk characteristic value of the corresponding reference risk point; The local risk complexity of each reference risk point is determined based on the first and second risk characteristic values ​​of each reference risk point. Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained; The risk environment complexity of the current road segment is obtained by weighting the local risk complexity of all reference risk points. By weighting the complexity of the risk environment with the degree of risk exposure, the risk correlation degree of the current scenario is obtained; Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained; For any target in the current scenario, obtain the target's velocity, acceleration, and turning angular velocity. Based on the target's motion parameters and the degree of risk correlation, obtain the target's risk concern index.

[0032] It should be noted that the system analyzes key monitoring areas, merging adjacent road network units belonging to the same risk type into a larger area called a risk local area. The purpose is to focus the analysis on areas with similar risk characteristics, thereby more effectively assessing potential risks. For reference risk points (such as traffic lights, intersections, etc.) within each risk local area, the system calculates a first risk characteristic value based on its correlation with other factors within the area. This characteristic value reflects the risk level of the reference risk point in the current area. For example, if a traffic light at an intersection frequently causes traffic accidents due to malfunctions, then the first risk characteristic value of that traffic light will be higher. The system will calculate the correlation between each reference risk point and the intelligent... The shortest distance between the current location of the intelligent vehicle and the reference risk point is calculated. This step aims to consider the distance between the intelligent vehicle and the reference risk point; the closer the distance, the higher the potential risk. This distance will be used to calculate the second risk characteristic value. The local risk complexity of each reference risk point consists of two factors: the first risk characteristic value and the second risk characteristic value. The system combines these two values ​​to comprehensively evaluate the complexity of the reference risk point, thereby reflecting the overall risk level of the risk point in the current environment. The risk exposure degree of the intelligent vehicle in a certain scenario refers to the risk level of the vehicle in the current environment. This degree depends not only on the risk points in the surrounding environment but also on factors such as the vehicle's position and speed. Based on environmental feature points (such as traffic lights, intersections, etc.) By calculating the relative position of the vehicle and the target motion feature points (such as the vehicle's current position), the system can determine the risk exposure level of the intelligent vehicle. For example, if the vehicle is approaching an intersection with a high incidence of traffic accidents and the traffic lights at that intersection are malfunctioning, then the vehicle's risk exposure level is high. The system weights the local risk complexity of all reference risk points in the current road segment to obtain the risk environment complexity of that road segment; this value reflects the overall risk level of the road segment. The system also weights the risk environment complexity of the road segment with the risk exposure level of the intelligent vehicle to obtain the risk correlation degree; this value comprehensively reflects the overall risk level faced by the intelligent vehicle in the current scenario. For example, if the vehicle is in a high-risk road segment... If a road segment is traveling at a high speed and the road environment is highly complex, then the risk correlation of that road segment will be very high. The system calculates the risk concern index of the target (intelligent vehicle or other traveling target) based on the motion parameters (such as speed, acceleration, steering angular velocity, etc.) of the target in the current scene, combined with the aforementioned risk correlation. This index reflects the overall risk level of the intelligent vehicle in the current scene and determines whether corresponding risk warnings or intervention measures need to be taken. For example, if an intelligent vehicle is traveling at a high speed and is approaching a high-risk area (such as an accident-prone intersection), its risk concern index will be very high, and the system may issue a warning to remind the driver to slow down or take other safety measures.

[0033] In an optional embodiment, the risk exposure level of the intelligent vehicle in the current scene is obtained based on the positional relationship between environmental feature points and target motion feature points in the current scene, including: Sort the spatial distances between each environmental feature point and each target motion feature point in the current scene to obtain a distance sequence; The distance sequence is divided into multiple risk intervals; the matching degree between the corresponding environmental feature points and the reference risk points in each risk interval is determined to obtain the risk correlation degree of each interval. The intervals with a risk correlation greater than a preset correlation threshold are classified as high correlation intervals, and the intervals with a risk correlation less than or equal to the preset correlation threshold are classified as low correlation intervals. The target motion feature points in the current scene and the corresponding environmental feature points in the highly correlated intervals are grouped into the same cluster as a high-risk cluster; the corresponding environmental feature points in the low-correlation intervals are grouped into the same cluster as a low-risk cluster. The difference in spatial distribution variance between high-risk and low-risk clusters is normalized to obtain the risk exposure level of intelligent vehicles in the current scenario.

[0034] It should be noted that the system calculates the spatial distance between each environmental feature point (such as traffic signs, intersections, etc.) and each target motion feature point (such as the position of the intelligent vehicle) in the current scene; then, these distances are sorted in ascending order to obtain a distance sequence. The sorted sequence helps the system understand the spatial relationship between the target and each environmental feature point; the smaller the distance, the closer the relationship between the intelligent vehicle and that environmental feature point. The distance sequence is divided into multiple risk intervals, meaning that the system divides the spatial distance between the feature point and the target into several intervals based on the distance size. Each interval represents a spatial relationship within a certain range, and the system uses these intervals to analyze the risk situation at different distances. For each risk interval, the system analyzes the correlation between the environmental feature points within that interval and reference risk points (such as historical traffic accident data, traffic light status, etc.). The goal of this process is to evaluate the impact of each environmental feature point on the risk within that interval, thereby obtaining the risk correlation degree of that interval. Based on a preset correlation degree threshold, the system divides the interval into high-correlation intervals and low-correlation intervals. The high-risk interval indicates a strong risk relationship between environmental feature points and the target, while the low-risk interval indicates a weaker risk relationship. Based on the previously defined high-risk and low-risk intervals, the system clusters target motion feature points (such as the location of the intelligent vehicle) with environmental feature points in the high-risk interval into a high-risk cluster, and environmental feature points in the low-risk interval into a low-risk cluster. The system calculates the difference in spatial distribution variance between the high-risk and low-risk clusters. Spatial distribution variance reflects the distribution of points within each cluster; a larger variance indicates a more dispersed distribution of points within the cluster, indicating greater risk variation in that area. By calculating the difference in variance and normalizing it, the system ultimately determines the risk exposure level of the intelligent vehicle. This level reflects the risk level faced by the intelligent vehicle in the current scenario. For example, if environmental feature points in high-risk areas are concentrated, while those in low-risk areas are dispersed, the variance difference between the two will be larger, resulting in a higher risk exposure level for the intelligent vehicle.

[0035] like Figure 2 As shown, the present invention proposes a deep learning-based intelligent vehicle operation high-risk scene recognition device, which is applicable to the aforementioned deep learning-based intelligent vehicle operation high-risk scene recognition method, comprising: Feature fusion module 1 is used to collect multimodal data of intelligent vehicle operation scenarios, including visual images, LiDAR point clouds, V2X communication data and high-precision map information; multimodal feature fusion and encoding are performed on the multimodal data to obtain a fused feature sequence; The graph modeling module 2 is used to determine the spatiotemporal evolution of risk features based on the fused feature sequence and update the global risk feature graph based on the spatiotemporal evolution. The global risk feature graph is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. The region division module 3 is used to determine the risk markers of the operation scenario based on the updated global risk feature map, and to distinguish between static risk areas and dynamic risk areas in the risk markers based on the global risk feature map before the update and the fused feature sequence, so as to obtain the high-risk candidate areas corresponding to the operation scenario. Risk monitoring module 4 is used to determine the reference risk points that need to be monitored by the intelligent vehicle in the topological road network corresponding to the operating scenario based on the high-risk candidate areas corresponding to the operating scenario. Risk identification module 5 is used to determine the risk attention index of each target in the current scene based on the location distribution of reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene; and to determine the risk level of each target in the current operating scene based on the risk attention index of each target in the current scene.

[0036] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a deep learning-based method for identifying high-risk scenarios in intelligent vehicle operation.

[0037] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases.

[0038] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning, characterized in that, include: Multimodal data of intelligent vehicle operation scenarios are collected, including visual images, LiDAR point clouds, V2X communication data, and high-precision map information; multimodal feature fusion and encoding are performed on the multimodal data to obtain a fused feature sequence; The spatiotemporal evolution of risk features is determined based on the fused feature sequence, and the global risk feature map is updated based on the spatiotemporal evolution, wherein the global risk feature map is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. The risk markers for the operational scenario are determined based on the updated global risk feature map; The feature changes in the fused feature sequence are compared with the feature baseline of the global risk feature map before the update to obtain dynamic risk elements; The dynamic risk elements are associated with the risk areas in the risk markers to obtain static risk areas and dynamic risk areas; High-risk candidate regions are selected from static and dynamic risk regions based on a preset high-risk threshold. Based on the high-risk candidate areas corresponding to the operating scenario, the reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario are determined. For any key monitoring area, adjacent road network units belonging to the same risk type in the key monitoring area are merged to obtain a local risk area; Determine the correlation degree of each reference risk point in each risk local area to obtain the first risk characteristic value of the corresponding reference risk point; Determine the shortest distance between each reference risk point and the current location of the intelligent vehicle to obtain the second risk characteristic value of the corresponding reference risk point; The local risk complexity of each reference risk point is determined based on the first and second risk characteristic values ​​of each reference risk point. Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained; The risk environment complexity of the current road segment is obtained by weighting the local risk complexity of all reference risk points. The risk environment complexity and the risk exposure level are weighted to obtain the risk correlation level of the current scenario; For any target in the current scene, obtain the target's motion speed, acceleration, and turning angular velocity; and obtain the target's risk attention index based on the target's motion parameters and the risk correlation degree. The risk level of each target in the current operating scenario is determined based on the risk attention index of each target in the current scenario.

2. The method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning according to claim 1, characterized in that, The multimodal data is subjected to multimodal feature fusion and encoding to obtain a fused feature sequence, including: The multimodal data is categorized into visual sub-data, radar sub-data, V2X sub-data, and map sub-data based on data type. Feature extraction is performed on each sub-data set to obtain the modal feature vector for each sub-data set; The correlation degree between the modal feature vector of each sub-data and the modal feature vectors of the other sub-data is determined to obtain the cross-modal feature mapping network of the multimodal data; Obtain the attention weight of each feature channel in the cross-modal feature mapping network, and determine the fusion weight of each modal feature based on the attention weight of each feature channel; Multiple modal features corresponding to each data segment in the multimodal data are obtained, and the fusion features of each data segment in the multimodal data are determined based on the fusion weights and attention weights of the modal features. The fusion feature sequence of the multimodal data is determined based on the fusion features of each data segment in the multimodal data.

3. The method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning according to claim 2, characterized in that, Determining the spatiotemporal evolution of risk characteristics based on the fused feature sequence includes: Determine the effective sensing range of the multimodal data, and construct a spatiotemporal grid cell set based on this range; The correlation between each feature in the fused feature sequence in the time and space dimensions is obtained, and all units including risk correlation in the spatiotemporal grid unit set are obtained. The intersection of the feature evolution trajectory and the feature node in the fused feature sequence is obtained. The unit is used to characterize a local feature set initialized based on the fused feature sequence, which represents a spatiotemporal range. The unit records the feature weight, spatiotemporal coordinates and correlation strength. Determine the risk evolution distance from each unit to the intersection point of the feature node, and based on the risk evolution distance of the unit, determine the risk space labeling of the multimodal data as low-risk area, medium-risk area and high-risk area, so as to obtain the spatiotemporal evolution of risk features.

4. The method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning according to claim 3, characterized in that, Updating the global risk feature map based on the spatiotemporal evolution includes: Locate units in the spatiotemporal evolution within the global risk feature map; For the identified unit, the global risk value of the unit in the global risk feature map is updated based on the risk evolution distance of the unit in the spatiotemporal evolution. For units that are not found, the units are added to the global risk feature map, and the risk evolution distance of the units in the spatiotemporal evolution is used as their global risk value in the global risk feature map.

5. The method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning according to claim 4, characterized in that, Based on the high-risk candidate regions corresponding to the aforementioned operating scenarios, the reference risk points that intelligent vehicles need to focus on monitoring in the topological road network corresponding to the operating scenarios are determined, including: For any road segment, the high-risk candidate area of ​​the road segment is used as a reference point, and the road network unit in which the reference point is located in the topological road network is used as a reference road network unit. Among them, environmental feature points and historical risk data of all road sections are obtained in advance, and the feature points are processed by the road network topology construction algorithm to obtain the topology road network corresponding to the operation scenario. The key nodes in the topology road network are environmental feature points. The topology road network is obtained by the road network topology construction algorithm based on high-precision map. The road network topology construction algorithm is used to fuse road centerline, intersection and lane line features to generate road network units. The area formed by the reference road network unit in the topological road network and the road network unit directly connected to the reference road network unit is designated as the key monitoring area of ​​the road segment. Environmental feature points and historical accident sites in the key monitoring areas will be used as reference risk points for key monitoring of the road sections.

6. The method for identifying high-risk operating scenarios of intelligent vehicles based on deep learning according to claim 5, characterized in that, Based on the positional relationship between environmental feature points and target motion feature points in the current scene, the risk exposure level of the intelligent vehicle in the current scene is obtained, including: Sort the spatial distances between each environmental feature point and each target motion feature point in the current scene to obtain a distance sequence; The distance sequence is divided into multiple risk intervals; the matching degree between the corresponding environmental feature points and the reference risk points in each risk interval is determined to obtain the risk correlation degree of each interval. The intervals with risk correlation greater than a preset correlation threshold are divided into high correlation intervals, and the intervals with risk correlation less than or equal to the preset correlation threshold are divided into low correlation intervals. The target motion feature points in the current scene and the corresponding environmental feature points in the highly correlated intervals are grouped into the same cluster as a high-risk cluster; the corresponding environmental feature points in the low-correlation intervals are grouped into the same cluster as a low-risk cluster. The difference in spatial distribution variance between the high-risk cluster and the low-risk cluster is normalized to obtain the risk exposure level of the intelligent vehicle in the current scenario.

7. A deep learning-based intelligent vehicle operation high-risk scene recognition device, applicable to the deep learning-based intelligent vehicle operation high-risk scene recognition method according to any one of claims 1 to 6, characterized in that, include: The feature fusion module is used to collect multimodal data of intelligent vehicle operation scenarios, wherein the multimodal data includes visual images, LiDAR point clouds, V2X communication data and high-precision map information; the multimodal data is fused and encoded to obtain a fused feature sequence; The graph modeling module is used to determine the spatiotemporal evolution of risk features based on the fused feature sequence, and to update the global risk feature graph based on the spatiotemporal evolution, wherein the global risk feature graph is used to characterize the initialization of risk features based on the fused feature sequence at the initial time. The region segmentation module is used to determine the risk markers of the operating scenario based on the updated global risk feature map, and to distinguish the static risk regions and dynamic risk regions in the risk markers based on the global risk feature map before the update and the fused feature sequence, so as to obtain the high-risk candidate regions corresponding to the operating scenario. The risk monitoring module is used to determine the reference risk points that the intelligent vehicle needs to focus on monitoring in the topological road network corresponding to the operating scenario based on the high-risk candidate areas corresponding to the operating scenario. The risk identification module is used to determine the risk attention index of each target in the current scene based on the location distribution of the reference risk points in the topological road network, the positional relationship between environmental feature points and target motion feature points in the current scene, and the motion state of each target in the current scene; and to determine the risk level of each target in the current operating scene based on the risk attention index of each target in the current scene.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images