Knowledge and data hybrid driven occlusion potential hazard scene cognition method

Abstract

The application discloses a kind of knowledge and data hybrid drive's field of vision obstruction potential dangerous scene cognition method, steps include: step 1, perceive scene image information, position time information;Step 2, detect scene target;Step 3, generate scene graph;Step 4, carry out knowledge graph multi-hop reasoning, obtain potential implicit information in scene;Step 5, utilize Bert model and carry out feature coding, obtain as data-driven input vector matrix;Step 6, based on ST-GCN realizes scene cognition, vector matrix that completes feature coding is input into ST-GCN model, obtains the final result of scene cognition, namely, this.The method of the application, compared with simply based on road scene image information carries out scene cognition with better migration and generalization, consider the influence of coupling relationship between potential dangerous scene macro-micro space-time constituting factor to scene cognition, can more complete and accurate cognition current potential dangerous scene.

Description

Technical Field

[0001] This invention belongs to the field of vehicle traffic hazard scene cognition technology, and relates to a knowledge and data hybrid-driven method for recognizing potential hazard scenes of visual occlusion. Background Technology

[0002] In the field of autonomous driving, the underlying architecture and most technical issues have been resolved, but 5% of the "long tail" problems remain unresolved, which is a significant factor currently hindering the development of autonomous driving. Currently, research on recognizing potential hazards in autonomous driving scenarios with obstructed views attempts to address this issue using vehicle-to-everything (V2X) or vehicle-to-infrastructure (V2I) systems. While these systems relying on connected technologies help perceive the status of traffic participants in areas with obstructed views, they still face numerous challenges beyond addressing issues like network reliability and security. For example, V2X technology relies on mobile phones or other smart devices carried by vehicles / pedestrians to establish V2X communication to avoid potential hazards, but this method is susceptible to communication interruptions due to missing or damaged smart devices. V2I technology heavily relies on information acquired by roadside sensors, but due to the highly random distribution of potential hazards, roadside sensors cannot achieve comprehensive and permanent coverage of all potential hazards, leading to a lack of perception of important road information. Furthermore, roadside intelligent sensing struggles to cover all urban roads and non-road areas.

[0003] To address the safety hazards posed by potential dangerous scenarios, a human-like cognitive model for potential dangerous scenarios involving visual field occlusion can be developed from a single-vehicle intelligence perspective. This model combines the vehicle itself with artificial intelligence technology, employing a hybrid knowledge and data-driven approach. Recognizing potential dangerous scenarios involving visual field occlusion for autonomous vehicles is a multi-dimensional issue, involving the impact of interactions between the autonomous vehicle and surrounding traffic participants, the influence of the surrounding environment, and the time period in which the scenario occurs. Therefore, scenario recognition requires a multi-dimensional consideration of the interactive relationships between other participants in the traffic scenario. Summary of the Invention

[0004] The purpose of this invention is to provide a knowledge- and data-driven method for recognizing potential dangerous scenarios with occlusion, which solves the problem that existing technologies have difficulty in comprehensively considering the relative motion between autonomous vehicles and virtual traffic participants, as well as the influence of the contextual semantics of the surrounding macroscopic traffic environment on scene cognition, resulting in insufficient cognition of potential dangerous scenarios with occlusion and inadequate real-time accuracy.

[0005] The technical solution adopted in this invention is a knowledge- and data-driven method for recognizing potential hazards in visual occlusion scenarios, implemented according to the following steps:

[0006] Step 1: Perceive scene image information and location and time information;

[0007] Step 2: Detect scene targets.

[0008] The YOLO v8 algorithm was used to detect explicit feature information of traffic participants in the scene, including the participants and their attribute information.

[0009] Step 3: Generate scene diagram.

[0010] The PSG model is used to output all relationships in the image and to locate objects using accurate segmentation blocks. A scene graph is generated based on these features and relationships and stored as SPO triples.

[0011] Step 4: Perform multi-hop reasoning based on the knowledge graph.

[0012] A knowledge graph multi-hop reasoning method is used to perform human-like knowledge reasoning on the entity and relationship information represented in the scene graph to obtain potential implicit information in the scene;

[0013] Step 5: Perform feature encoding using the BERT model.

[0014] The explicit and implicit information of the scene is integrated and feature encoding based on the BERT model is performed to obtain a vector matrix as data-driven input;

[0015] Step 6: Implement scene recognition based on ST-GCN.

[0016] The vector matrix with completed feature encoding is input into the ST-GCN model to obtain the final result of scene cognition.

[0017] The beneficial effects of this invention include the following aspects:

[0018] 1) The method of this invention integrates knowledge graph and deep learning technologies to construct a data and knowledge-driven cognitive method for the vision occlusion scenario of autonomous driving system. It mines the semantic information of road scene and has better transferability and generalization compared with scene cognition based solely on road scene image information.

[0019] 2) In the process of scene cognition, the method of the present invention considers the influence of the coupling relationship between the macro and micro spatiotemporal constituent factors of potential dangerous scenes on scene cognition.

[0020] 3) By using spatiotemporal graph convolution technology based on ST-GCN, we can more completely and accurately understand the current potential dangerous scenarios. ST-GCN takes into account the coupling relationship between various interactors in the scenario, which solves the problem that conventional methods for recognizing potential traffic hazards only consider the one-way interaction between autonomous vehicles and potential traffic participants. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the process of recognizing potentially dangerous scenarios with obstructed field of vision using the method of the present invention.

[0022] Figure 2 This is a schematic diagram of an embodiment of the field-of-view occlusion method used in the present invention.

[0023] Figure 3a A frame of the simplified embodiment scene diagram, Figure 3b The scene graph generated after extracting the list of scene graph objects and their attributes;

[0024] Figure 4 This is a schematic diagram of scene map information fusion used in the method of the present invention;

[0025] Figure 5 This is a schematic diagram of knowledge graph multi-hop reasoning used in the method of this invention;

[0026] Figure 6 This is a schematic diagram illustrating the cognitive process of potential visual obstruction hazards in the method of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0028] Reference Figure 1 The concept and principle of the method of this invention are as follows:

[0029] This invention provides a method for recognizing potential traffic hazards caused by obstructed vision in autonomous vehicles. This human-like scenario recognition method employs a hybrid knowledge- and data-driven approach to enhance the comprehensiveness and reliability of recognizing potential hazards under obstructed vision. Firstly, the knowledge-driven approach utilizes knowledge graph reasoning technology. Knowledge graphs can represent entities and relationships within potential traffic hazard scenarios using a network-like knowledge structure. Knowledge graph reasoning technology can think and reason like a human, thereby obtaining important implicit information within the scenario, forming a crucial foundation for human-like scenario recognition. Secondly, the data-driven approach, based on knowledge graph reasoning, adds the characteristic information inherent in the scene itself, using this information as input, to perform data-driven scenario recognition under obstructed vision.

[0030] Considering the macro and micro spatiotemporal scene information of the current autonomous vehicle, the macro spatiotemporal factors are the current time and location (city or suburb), the micro spatiotemporal information are the speed, acceleration and other factors of the autonomous vehicle and the scene interactors, and the traffic location of the autonomous vehicle (residential area, commercial area, school, etc.), and the event information under normal traffic conditions are also taken into account.

[0031] Using knowledge graphs such as Neo4j, a real-time knowledge graph for autonomous vehicles is constructed, considering macro and micro spatiotemporal information in driving scenarios, as well as autonomous driving event information, to build a complete knowledge graph of potential dangers and risks caused by field of view occlusion, such as... Figure 4 As shown. Figure 4 The map consists of a time layer map composed of prior knowledge, a space layer map, and an image layer map of the autonomous vehicle's real-time scene. By fusing the map composed of the two prior knowledge maps and the image layer map of the autonomous vehicle's real-time scene, a complete scene map map of potential risks of field of view occlusion is obtained.

[0032] The system generates scene graphs for the current autonomous driving scenario. Based on the constructed knowledge graph of potential traffic hazards, it uses the feature attributes and relationships in the autonomous driving scene graph as input for knowledge graph reasoning to obtain explicit and implicit scene features. Data-driven technology is used to encode these features and then input them into the cognitive model to achieve complete cognition of potential hazards caused by visual occlusion.

[0033] The method of the present invention is based on the following framework for setting up the steps and processes: scene image information perception, location and time information perception, scene target detection (algorithm), panoptic scene graph generation (PSG) based on panoramic segmentation, knowledge graph multi-hop reasoning (method), feature encoding based on the BERT model, and cognitive model based on ST-GCN.

[0034] In summary, the knowledge- and data-driven method for recognizing potential hazards in visual field occlusion, as described in this invention, is implemented according to the following steps:

[0035] Step 1: Perceive scene image information and location / time information.

[0036] The system uses an in-vehicle camera to perceive real-time scene photos at certain time intervals, and an in-vehicle GPS to perceive the GPS information of the scene. At the same time, it uses the Baidu API to process the obtained real-time scene photos and GPS information to obtain the required location and time information.

[0037] Step 2: Detect scene targets.

[0038] The YOLO v8 algorithm was used to detect explicit feature information of traffic participants in the scene, including the participants and their attribute information.

[0039] Step 3: Generate scene diagram.

[0040] The PSG model comprehensively outputs all relationships in the image (including relationships between objects, between objects and the background, and between backgrounds), and uses accurate segmentation blocks to locate objects. Based on these feature information and relationships, a scene graph (also known as a directed graph) is generated, and this information is stored in the form of SPO triples (entity, relation, entity).

[0041] Step 4: Perform multi-hop reasoning based on the knowledge graph.

[0042] A knowledge graph multi-hop reasoning method is used to perform human-like knowledge reasoning on the entity and relationship information represented in the scene graph to obtain potential implicit information in the scene;

[0043] Step 5: Perform feature encoding using the BERT model.

[0044] The explicit and implicit information of the scene is integrated and feature encoding based on the BERT model is performed to obtain a vector matrix as data-driven input;

[0045] Step 6: Implement scene recognition based on ST-GCN.

[0046] The vector matrix with completed feature encoding is input into the ST-GCN model to obtain the final result of scene cognition.

[0047] Example:

[0048] Reference Figure 6 According to the above-described process of the method of the present invention, it is specifically implemented according to the following steps:

[0049] Step 1: Input the time-series scene photos perceived by the vehicle's onboard camera and the vehicle's GPS sensor information. The scene photos are a series of image frames extracted at certain intervals from real-time video generated by the camera sensors on the autonomous vehicle, such as... Figure 1 As shown in section a); the Baidu API is used to process the scene photos and GPS information in a unified manner to obtain the required location and time information, such as... Figure 1 As shown in part b).

[0050] Step 2, using YOLO v8 to... Figure 2 The scene image shown is used for object detection, and feature information is extracted from the scene image to obtain a feature set Objects = {O1, O2, ...} and an attribute set Attributes = {O1: [x, y, type]}. For example, in Figure 2 Within the target set, information such as car 1, car 2, bus, pedestrian 1, pedestrian 2, and brake lights on the bus were detected.

[0051] Step 3, as follows Figure 1As shown in section c), a scene graph reflecting the relationships between feature elements in a scene photograph is constructed in the SPG-based full-scene graph generation model. For example, in Figure 2 The relationship information shown includes that car 1 is to the left of car 2, there is a bus directly in front, and pedestrians are crossing the road on the sidewalk. For each input information (including image information, GPS information, etc.), the output is represented by G. T ={O n A n The triplet information is used to represent the corresponding scene relationships, and this is used to construct a scene graph. For example... Figure 3a The image shown is a frame of a scene photograph in this embodiment. Figure 3b The scene graph is generated after extracting the list of scene graph objects and their relationships, and contains edge connection nodes of various types.

[0052] Step 4, multi-hop reasoning based on knowledge graph, such as Figure 1 Part d) of the algorithm takes the triplet information from the scene graph as input and performs knowledge reasoning on the potential danger risk scene graph of field of view occlusion to obtain explicit and implicit information for each frame of the scene. For example, ... Figure 5 ,Depend on Figure 1 In part a) of the diagram, at the spatial layer, the GPS sensor on the autonomous vehicle 1 obtains the macro-location (city, Xi'an) and the micro-location (the entrance of Xiwang Primary School) via the Baidu Maps API. From the relationship between location and time, it derives attribute information of the target features, such as speed and acceleration. From traffic lights, it infers micro-locations such as traffic intersections, and from the intersections and the school, it infers that this is a high-traffic area. At the temporal layer, the time information from the vehicle's GPS determines that it is noon on Sunday, thus inferring that this time period is a peak hour. At the scene feature layer, the bus is an obstructed area, and the brake lights are on, indicating the possible presence of a pedestrian suddenly appearing from behind a window.

[0053] Step 5: Perform BERT-based feature encoding on the features of the local scene, such as converting occluded areas, high-traffic areas, peak time periods, and traffic participants into feature vectors to construct a feature matrix. Figure 1 Part e) in the text.

[0054] Step 6, Scene cognition based on ST-GCN, such as Figure 1 Part f) takes the scene feature matrix as input to classify the potential hazard types in the current scene, such as... Figure 1 Part g) in the text. Figure 5In this embodiment, features such as high-traffic areas, peak hours, traffic participants, and "ghost pedestrians" (people suddenly appearing from behind obstacles) are encoded and input into a pre-trained ST-GCN network to classify and recognize the current scene type. For example, when the classification result is labeled as "high-risk," the current scene can be determined to be of a high-risk type.

Claims

1. A knowledge and data hybrid driven field of view occlusion potential hazard scene awareness method, characterized in that, The specific implementation is as follows: Step 1, sensing scene image information, position time information, Specifically: applying a vehicle-mounted camera to sense real-time scene photos within a certain time interval, applying a vehicle-mounted GPS to sense the GPS information of the scene, and simultaneously applying Baidu API to process the obtained real-time scene photos and GPS information to obtain the required position and time information; Step 2, detecting scene targets, Detecting the explicit feature information of traffic participants in the scene through the YOLO v8 algorithm, including participant and attribute information; Step 3, generating a scene graph, Based on the PSG model, all relationships in the image are output, and accurate segmentation blocks are used to locate objects; all relationships include object-to-object relationships, object-to-background relationships, and background-to-background relationships; Based on these feature information and relationships, a scene graph is generated, and these information is stored in the form of SPO triples; SPO triples include entities, relationships, and entities; Step 4, multi-hop reasoning of knowledge graph, Using the multi-hop reasoning method of the knowledge graph, the entity and relationship information represented in the scene graph is subjected to human-like knowledge reasoning to obtain potential implicit information in the scene; Step 5, feature encoding using the Bert model, Integrate explicit and implicit information of the scene and perform feature encoding based on the Bert model to obtain a vector matrix as a data-driven input; Convert the information of occluded areas, high pedestrian flow areas, peak time periods, and traffic participants into feature vectors to construct a feature matrix; Step 6, scene cognition based on ST-GCN, With the scene feature matrix as input, the potential danger type classification of the current scene is realized, and the vector matrix after feature encoding is input into the ST-GCN model to obtain the final result of scene cognition.

Images