Automatic driving scene evolution knowledge graph construction method and risk prediction evaluation method

Abstract

The invention discloses an automatic driving scene evolution knowledge graph construction method and a risk prediction evaluation method, and belongs to the technical field of automatic driving. The automatic driving scene evolution knowledge graph construction method comprises the steps of generating a traffic element entity relation graph of each frame according to driving data of an unmanned vehicle; generating a scene node corresponding to each frame based on the traffic element entity relation graph; after scene nodes are screened based on the traffic element entity relation graph, the edge probability of the scene nodes is calculated according to the example traffic element entity relation graph, so that a scene evolution knowledge graph is generated; wherein the example traffic element entity relation graph is a traffic element entity relation graph of the screened scene nodes. According to the invention, the probability that the current scene will be converted into each scene due to the behavior can be predicted and the risk assessment under the target scene can be predicted through the behavior decision guidance opinions provided by the scene evolution knowledge graph.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a method for constructing an autonomous driving scenario evolution knowledge graph and a risk prediction and assessment method. Background Art

[0002] As a powerful knowledge representation and reasoning tool, knowledge graphs have theoretical advantages such as completeness and accuracy, which can help provide correct guidance for practical tasks and thus help agents complete tasks.

[0003] Autonomous driving systems improve road safety and travel efficiency by reducing human errors, optimizing traffic flow and energy consumption, improving travel convenience, and alleviating driving fatigue. Autonomous driving also supports shared travel, reduces transportation costs, promotes the development of smart cities, and makes overall transportation more efficient and sustainable. A good autonomous driving knowledge graph can effectively help autonomous driving systems make autonomous driving decisions more safely, reliably, and quickly, and improve the explainability of decisions.

[0004] In practice, there are some limitations in applying traditional knowledge graphs to the field of autonomous driving, which are mainly manifested in three aspects: First, they usually only record the entity relationships in specific scenarios, and the semantics of these nodes are often vague and cannot accurately describe the entity relationships in the scene. For example, the vehicle should give way to pedestrians, but there is a lack of more detailed and limited description of the relative relationship between the vehicle and pedestrians, such as the relative position relationship between the vehicle and pedestrians, the vehicle's own state, and the state of the pedestrian. Secondly, traditional knowledge graphs cannot capture the scene evolution relationship that occurs with the behavior of the subject, lack of correlation between scenes, and cannot estimate risks based on the actions of the current scene subject, thus failing to provide the best behavioral decision-making advice. Summary of the invention

[0005] In response to the above problems, the present invention provides a method for constructing an autonomous driving scenario evolution knowledge graph and a risk prediction and assessment method. Through the behavioral decision-making guidance provided by the scenario evolution knowledge graph, it is possible to predict the probability that the current scenario will be converted to various scenarios due to behavior and predict the risk assessment under the target scenario.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for constructing an autonomous driving scenario evolution knowledge graph is provided, comprising:

[0007] Generate an entity relationship diagram of traffic elements in each frame based on the driving data of the unmanned vehicle;

[0008] Generate a scene node corresponding to each frame based on the traffic element entity relationship diagram;

[0009] After the scene nodes are screened based on the traffic element entity relationship diagram, the out-edge probability of the scene nodes is calculated according to the example traffic element entity relationship diagram to generate a scene evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after screening.

[0010] Optionally, the driving data includes: multi-view time-series image data, laser radar time-series point cloud data and sensor data;

[0011] The generating of the traffic element entity relationship diagram of each frame according to the driving data of the unmanned vehicle includes:

[0012] Frame alignment based on multi-view temporal image data and lidar temporal point cloud data i Extract global semantic information and obtain global semantic information Among them, scene i Indicates frame i The scene identifier, id ij Indicates frame i The unique identifier of the jth traffic element entity in c ij Indicates frame i The traffic element category occupied by the jth traffic element entity in s ij Indicates frame i The pixel area occupied by the jth traffic element entity in ij Indicates frame i Identified as a traffic element entity id ij Point set, |frame i | indicates frame i The number of traffic element entities under the unmanned vehicle pixel area s ij and point set p ij are all empty sets;

[0013] Frame using sensor data i Extract the entity parameters of traffic elements and combine them with global semantic information Get global semantic information Among them, loc ij Indicates the entity id of the traffic element ij The location information, v ij Indicates the entity id of the traffic element ij The speed information, a ij Indicates the entity id of the traffic element ij Acceleration information, color ij Indicates the entity id of the traffic element ijColor information, text ij Indicates the entity id of the traffic element ij Text information, b ij Indicates the entity id of the traffic element ij Behavioral information;

[0014] Based on the global semantic information Get any two traffic element entity IDs ij The relationship between traffic element entities ik , and combined with global semantic information Get global semantic information FM i = <scene i ,G i >; Among them, the traffic element entity graph G i = <E i ,N i >, a directed edge set E used to represent entity relationships i ={e ik |k=1,2,…,m i},e ik frame i The kth traffic element entity relationship, m i is the number of traffic element relationships in the current frame, and each traffic element entity relationship e ik is a triplet of (id, r, id), which is a key-value pair set N used to represent node information. i ={id ij :(c ij ,s ij ,p ij ,loc ij ,v ij ,a ij ,color ij ,text ij ,b ij )|j=1,2,…,|frame i |}, each element is a key-value pair of id:(c,s,p,loc,v,a,color,t,b).

[0015] Optionally, the frame based on the multi-view time-series image data and the laser radar time-series point cloud data i Extract global semantic information and obtain global semantic information include:

[0016] Use the pre-trained image panoptic semantic segmentation model imagePerform global semantic segmentation on multi-view images frame by frame, track traffic elements in multi-view images, and use different labels to identify pixel areas in the image sequence to obtain the image global semantic segmentation result SSRes image ;

[0017] Use the pre-trained point cloud global semantic segmentation model Model pc Perform global semantic segmentation on the LiDAR point cloud data frame by frame, track the traffic elements appearing in the point cloud, and use different labels to identify the point cloud subsets in the sequence point cloud data to obtain the point cloud global semantic segmentation result SSRes pc ;

[0018] The global semantic segmentation result of the image is SSRes image And point cloud global semantic segmentation result SSRes pc Make a match;

[0019] Generate global semantic information based on the matching results and the semantic information of the unmanned vehicle

[0020] Optionally, generating a scene node corresponding to each frame based on the traffic element entity relationship diagram includes:

[0021] Obtain node non-association semantic attribute classification rules and node association semantic attribute classification rules; wherein the non-association semantic attribute category classification rules are rules generated based on non-association semantic attributes that will not affect other entities or relationships between entities, and the association semantic attribute category classification rules are rules generated based on association semantic attributes that will affect other entities or relationships between entities;

[0022] Traverse each frame and create the current frame source semantic node set Set s and the target semantic node set Set t ;

[0023] Get the traffic element entity relationship graph G of the current frame i = <E i ,N i >

[0024] Traverse the traffic element entity relationship graph G i Traffic element entity n s , and generate each traffic element entity n s Non-associative semantic rule result Rule nr (n s ) and the traffic element entity n s With neighboring node n t The result of the associated semantic rule Rule nr (n s,n t );

[0025] According to the traffic element entity n s Category c s and non-associative semantic rule results Rule nr (n s ), create source node category label s , and label the source node category s Add to the source semantic node set Set s middle;

[0026] According to the neighboring node n t Category c t and associated semantic rule result Rule nr (n s ,n t ), create the target node category label t , and label the target node category t Add to the target semantic node set Set t middle;

[0027] Traverse traffic element entity node n s and neighboring node n t The relationship between them is used to establish the source node category label s and target node category label t directed edges between ;

[0028] Based on the source semantic node set Set s , target semantic node set Set t , source node category label s and target node category label t The directed edges between them and the entity relationship graph G of traffic elements i , generate the scene node S corresponding to the current frame i = <Set s ,Set t ,G i >.

[0029] Optionally, the filtering of scene nodes based on the traffic element entity relationship diagram includes:

[0030] Randomly add a scene node to the scene node collection;

[0031] Traverse all frames and obtain the traffic element entity relationship graph G of the current frame i ;

[0032] Traverse the scene node set to obtain the traffic element entity relationship graph G′ corresponding to a scene node i ;

[0033] Determine the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Whether they are the same, and if they are the same, the result of the associated semantic rule of the neighboring nodes of the corresponding traffic element entity node Rule nr (n s ,n t ) are the same;

[0034] In the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Different, or traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i The result of the association semantic rule of the neighboring nodes of the same but corresponding traffic element entity node nr (n s ,n t ) In different cases, the scene node corresponding to the current frame is added to the scene node collection.

[0035] Optionally, calculating the out-edge probability of the scene node according to the example traffic element entity relationship graph includes:

[0036] Traverse each frame in the driving data;

[0037] Determine the current frame i and the next frame i+1 Are the scene nodes the same?

[0038] If the current frame i and the next frame i+1 If the scene node is the same as the next frame i+1 =frame (i+1)+1 , and re-execute the judgment of the current frame i and the next frame i+1 Are the scene nodes the same?

[0039] If the current frame i and the next frame i+1 If the scene node is the same as the current frame i The corresponding scene node and the next frame i+1 Add an edge to the corresponding scene node; the attributes of the edge include: the action type and the number of times the action type occurs in the scene evolution knowledge graph, the number corresponding to the current frame framei The outgoing edge evolution probability of the corresponding scene node.

[0040] According to a second aspect of an embodiment of the present disclosure, a risk prediction and assessment method is provided, the method comprising:

[0041] Construct an entity relationship diagram of traffic elements of a scene S;

[0042] Compare the similarity between the traffic element entity relationship graph of the scene S and the example traffic element entity relationship graph corresponding to the scene evolution knowledge graph constructed based on any of the above-mentioned autonomous driving scene evolution knowledge graph construction methods, and obtain the scene node of the scene S in the evolution knowledge graph;

[0043] The risk probability of executing each action under the scenario node is calculated on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

[0044] Optionally, calculating the risk probability of executing each action under the scenario node on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S includes:

[0045] According to the execution action A r The time consumed T is used to obtain the number of frames required. Δt is the time interval between frames;

[0046] Based on the scenario evolution knowledge graph, obtain The scene S′ after frames, and the probability of reaching each scene S′ is calculated using the dynamic programming algorithm;

[0047] Calculate the scenario risk of scenario S′ according to the example traffic element entity relationship diagram and risk indicators of scenario S′, wherein the risk indicators include: TTC, comfort, safety, closest vehicle distance, traffic speed and traffic density;

[0048] Based on the probability of reaching each scene S′ and the scene risk of the scene S′, a risk assessment result under the scene S is obtained.

[0049] According to a third aspect of an embodiment of the present disclosure, a system for constructing a knowledge graph for autonomous driving scenario evolution is provided, which is characterized by comprising:

[0050] A first entity relationship diagram generation module, used to generate an entity relationship diagram of traffic elements of each frame according to the driving data of the unmanned vehicle;

[0051] A scene node attribute generation module, used to generate scene nodes based on the traffic element entity relationship diagram;

[0052] The scenario evolution knowledge graph generation module is used to filter the scene nodes based on the traffic element entity relationship diagram, and then calculate the out-edge probability of the scene nodes according to the example traffic element entity relationship diagram to generate a scenario evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after filtering.

[0053] According to a fourth aspect of an embodiment of the present disclosure, a risk prediction and assessment system is provided, comprising:

[0054] A second entity relationship diagram generation module is used to construct a traffic element entity relationship diagram of a scene S;

[0055] A similarity calculation module, used to compare the similarity between the traffic element entity relationship graph of the scene S and the example traffic element entity relationship graph corresponding to the scene evolution knowledge graph constructed based on any of the above-mentioned autonomous driving scene evolution knowledge graph construction methods, to obtain the scene node of the scene S in the evolution knowledge graph;

[0056] The risk assessment result generation module is used to calculate the risk probability of executing each action under the scenario node on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

[0057] According to a fifth aspect of an embodiment of the present disclosure, a computer device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for constructing an autonomous driving scenario evolution knowledge graph provided in the first aspect of the present disclosure or the risk prediction and assessment method provided in the second aspect of the present disclosure is implemented.

[0058] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for constructing a knowledge graph for evolving autonomous driving scenarios provided in the first aspect of the present disclosure or the risk prediction and assessment method provided in the second aspect of the present disclosure is implemented.

[0059] Compared with the prior art, the present invention has at least the following beneficial effects.

[0060] (1) Based on the actual image data and point cloud data, global semantic information is extracted frame by frame, and attribute semantic classification rules are formulated to refine the information in the scene and obtain relatively unified entity semantic information. In this way, a single scene knowledge graph is established and the example traffic element entity relationship graph is retained.

[0061] (2) Global semantic information is extracted frame by frame based on the actual image data and point cloud data, and a single scene knowledge graph is constructed based on the global semantic information. The single scene knowledge graph of the current frame is used as the source node, the next frame is used as the target node, and the action of the unmanned vehicle in the current frame is used as a sample. The probability of action scene evolution from the source node to the target node is counted, and the scene evolution knowledge graph is constructed based on this, which solves the problem of lack of correlation between scenes in traditional knowledge graphs.

[0062] (3) Based on the constructed scenario evolution knowledge graph, a method is provided to calculate the risks inherent in each action decision in the current scenario. According to the behavioral decision in the current scenario, the scenario evolution knowledge graph is used to obtain the risk estimation of the behavioral decision based on the scenario matching algorithm, dynamic programming algorithm and scenario risk index calculation method, which solves the problem that the traditional knowledge graph cannot evaluate the action risk and cannot provide the optimal behavioral decision opinion. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of a method for constructing an autonomous driving scenario evolution knowledge graph according to an exemplary embodiment.

[0064] Figure 2 The present invention is a flowchart of a risk prediction and evaluation method according to an exemplary embodiment.

[0065] Figure 3 is a diagram showing an example of TTC risk estimation according to an exemplary embodiment.

[0066] Figure 4 It is a block diagram of a system for constructing an autonomous driving scenario evolution knowledge graph according to an exemplary embodiment.

[0067] Figure 5 is a block diagram of a risk prediction and evaluation system according to an exemplary embodiment.

[0068] Figure 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0070] The present invention provides a method for constructing a knowledge graph of autonomous driving scene evolution. Figure 1 As shown, the process includes the following steps (step 1 to step 3).

[0071] Step 1: Generate an entity relationship diagram of traffic elements for each frame based on the driving data of the unmanned vehicle.

[0072] Step 1.1: Extract global semantic information based on multi-view temporal image data and lidar temporal point cloud data.

[0073] First, describe the data used. The data used should cover all traffic scenarios, and each scenario should contain rich traffic element information. The data for constructing the autonomous driving knowledge graph is a collection of time-series image data and time-series point cloud data. Each element in the collection is a dictionary, and the collection is called a data segment. The key of the data segment dictionary is the unique identifier frame number, and the value is a tuple constructed from multiple images and point cloud data from different perspectives in the current frame. For the sake of clarity in the following description, Pair = {framei: (imagei, pci) | i = 1, 2, ..., L} is used to describe a data segment, where i is the serial index of the data frame, L is the total number of frames of the Pair, and framei is the unique identifier frame number of the data segment. It should be noted that imagei and pci are both collections, and the elements are images and point cloud sets from multiple perspectives in the current frame, respectively.

[0074] Step 1.1.1: Use the pre-trained global semantic segmentation model to extract semantic information of traffic elements in each frame image data and point cloud data.

[0075] The purpose of semantic segmentation of image data and point cloud data is to identify the entity-related information and scene information of traffic elements in the data. The entity-related information includes the pixel area occupied by the entity in the image, the point set occupied in the point cloud data, and the category information of the entity. The entity-related information will provide information for the subsequent construction of the nodes of the scene knowledge graph; the scene information includes the category information of the current scene, and the scene information will provide an identifier for the subsequent construction of the scene knowledge graph.

[0076] The identifiers used to mark pixel areas and point cloud data are traffic-related traffic element classes and scene classes. Traffic element classes include but are not limited to pedestrians, cars, trucks, vans, buses, police cars, fire trucks, bicycles, electric vehicles, motorcycles, traffic lights, traffic signs, roadblocks and other traffic elements; scene classes include but are not limited to intersections, straight roads, curves, traffic roundabouts, highways, overpasses, crosswalks and other scenes.

[0077] For image data, the global semantic segmentation result includes pixel regions and scene classes with identifiers; for point cloud data, the global semantic segmentation result includes point sets with identifiers. Different pixel regions or point clouds belonging to the same category are distinguished by different serial number suffixes.

[0078] Use the pre-trained image panoptic semantic segmentation model imagePerform global semantic segmentation on multi-view images frame by frame, track traffic elements in the images, and use different labels to identify pixel areas in the image sequence. image =Model image (Pair) = {frame i :{id ij :(s ij ,c ij )|j=1,2,...,|frame i |}|i=1,2,...,L} is used to identify the result of global semantic segmentation of the image. ij is the unique identifier of the jth traffic element entity in the i-th frame, s ij and c ij represent the pixel area and traffic element category occupied by the jth traffic element entity in the i-th frame, respectively. i | indicates frame i The number of traffic element entities under the model is , and L is the total number of frames of the Pair. pc Perform global semantic segmentation on the LiDAR point cloud data frame by frame, track the traffic elements appearing in the point cloud, use different tags to identify the point cloud subsets in the sequence point cloud data, and use SSRes pc =Model pc (Pair) = {{frame i :{id ij :(p ij ,c ij )|j=1,2,...,|frame i |}|i=1,2,...,L}}. Among them, p ij and c ij Respectively represent the id in the i-th frame ij Point sets and traffic feature categories.

[0079] Step 1.1.2: Match the semantic segmentation results of image data and point cloud data.

[0080] Use a matching method to match the image semantic segmentation result with the point cloud semantic segmentation result. For example, the point cloud data can be projected onto the corresponding viewing plane, and the proportion of the point set with a certain ID (unique entity identifier, corresponding to an entity in the scene) falling in each pixel area in the viewing plane is calculated. The pixel area identifier with the largest proportion is matched with the point set ID. The category c in the image semantic segmentation result ij And category c in the point cloud semantic segmentation result ijIt can be used as an auxiliary verification. The scene class result in the image semantic segmentation result is used as the scene identification scene of the current frame. i .

[0081] After matching the semantic segmentation results of the image data with the semantic segmentation results of the point cloud data, the global semantic information in each frame is obtained, such as FM i = <scene i ,{id ij :(c ij ,s ij ,p ij )|j=1,2,...,|frame i |}>, where s ij and p ij Respectively represent the pixel area occupied by the jth traffic element entity in the i-th frame and the pixel area marked as id ij The point set.

[0082] Step 1.1.3: Supplement the semantic information of the unmanned vehicle.

[0083] Based on the relevant information of the unmanned vehicle in the corresponding frame, create an information key-value pair belonging to the unmanned vehicle and add it to FM i The key-value pair corresponding to the driverless car is s ij and p ij are all empty sets.

[0084] Step 1.2: Traffic element parameter extraction based on global semantic information and sensor data.

[0085] This step aims to extract relevant parameters of various traffic elements in each frame in order to construct a more complete knowledge graph of scene evolution. Traffic element parameters are descriptions of physical elements in the scene, adding additional attributes to physical elements to provide more complete information. In this invention, global semantic information and corresponding sensor data are used to extract traffic element parameters. The global semantic information uses the global semantic segmentation result obtained in step 1.1, and the sensor data uses the recorded data of various sensors aligned with the image point cloud data. The extracted traffic element parameters include but are not limited to parameters such as position, speed, acceleration, color, text, and behavior. The following explains how to calculate the relevant parameters mentioned. The information of a certain traffic element in the current frame is FM i = <scene i ,{id ij :(c ij ,s ij ,p ij )|j=1,2,...,|frame i |}>.

[0086] Step 1.2.1: Extract the location information of each traffic element based on global semantic information.

[0087] Traversing the traffic element information of each frame, the traffic element id can be calculated using the following formula ij Location information:

[0088]

[0089] Among them, item is the point belonging to the jth entity in the i-th frame in the point cloud semantic segmentation result, p ij Indicates that the i-th frame of point cloud data is identified as id ij The point set, |p ij | indicates that the i-th frame of point cloud data is identified as id ij The number of point cloud data in the collection.

[0090] Step 1.2.2: Extract speed information of each traffic element based on global semantic information.

[0091] Traversing the traffic element information of each frame, the traffic element id can be calculated using the following formula ij Speed ​​information:

[0092]

[0093] Among them, Δt is the time interval between two frames.

[0094] Step 1.2.3: Extract acceleration information of each traffic element based on global semantic information.

[0095] Traversing the traffic element information of each frame, the traffic element id can be calculated using the following formula ij Acceleration information:

[0096]

[0097] Step 1.2.4: Extracting color information of each traffic element based on global semantic information

[0098] Traversing the traffic element information of each frame, the traffic element id can be calculated using the following formula ij Color information:

[0099]

[0100] Among them, s ij represents the pixel area occupied by the jth traffic element entity in the i-th frame, |s ij | represents the number of pixels occupied by the jth traffic element entity in the i-th frame, and pixel represents pixels.

[0101] Step 1.2.5: Extract text information of each traffic element based on global semantic information.

[0102] Traverse the traffic element information of each frame, according to s ij Extract the image from the original image data to get the image fragment image seg . Use the pre-trained text recognition model Model text For image fragment image seg Recognize the text in text ij =Model text (image seg )

[0103] Step 1.2.6: Extract the behaviors of various traffic elements based on global semantic information.

[0104] First, formulate the behavior set of various traffic elements and the judgment conditions of each behavior. For example, the behavior states of small vehicles include but are not limited to straight driving, turning, stationary, reversing, etc., where the judgment condition for straight driving is "the minimum degree of the swing angle of the initial frame position point of the vehicle in n frames that can cover the subsequent position point does not exceed the threshold ". According to the relevant parameters of the vehicle, determine whether the straight-ahead condition is met. If it is met, set b ij for this behavior, otherwise continue to traverse other behavior states, where b ij Indicates the behavior information of the jth traffic element in the i-th frame. Traverse the traffic element information of each frame to obtain the behavior information of all traffic elements. It should be noted that any two behavior state judgment conditions of a certain type of traffic element cannot be met at the same time.

[0105] Step 1.2.7: Autonomous vehicle parameter extraction based on sensor data

[0106] Based on the relevant sensor data loaded on the unmanned vehicle, refer to the relevant parameter calculation method from step 1.2.1 to step 1.2.6 to obtain parameters including but not limited to the position, speed, acceleration, color, behavior, etc. of the unmanned vehicle, and add them to the corresponding key-value pairs of the unmanned vehicle.

[0107] Through the above steps, the FM obtained in step 1.1 i Supplemented to obtain more complete traffic element semantic information FM′ i = <scene i ,{id ij :(c ij ,s ij ,p ij ,loc ij ,v ij ,a ij ,color ij,text ij ,b ij )|j=1,2,...,|frame i |}>. For clarity, the relevant information of the jth entity in the i-th frame after parameter extraction is shown in Table 1.

[0108] Table 1 Entity parameter description table

[0109]

[0110]

[0111] Step 1.3: Extract traffic element relationships based on global semantic information and traffic element parameter information.

[0112] Traffic element relationships are used to describe the directed edges and their attributes between traffic element entities in a scene. For example, vehicle 1 follows vehicle 2. Here, following means that vehicle 1 points to vehicle 2. Traffic element relationships are extracted through a relationship judgment model. The connection between the attributes of traffic element entities in each frame or whether certain requirements are met is input to analyze and judge whether there is a relationship between traffic elements.

[0113] Select two traffic element entities whose relationship needs to be determined, extract and summarize the global semantic information extracted in step 1.2, and obtain the semantic information between the two traffic element entities, such as "the distance between vehicle one and vehicle two is always kept within five to ten meters", "the traffic light is red, and vehicle one is stationary", etc., and input the summarized relevant information into the pre-trained relationship judgment model to obtain the relationship between the two traffic element entities.

[0114] According to the results of relationship extraction, the FM obtained in step 1.2 i Supplement to obtain more complete semantic information of traffic elements:

[0115] FM i = <scene i ,G i >

[0116] Among them, the traffic element entity graph G i = <E i ,N i >, a directed edge set E used to represent entity relationships i ={e ik |k=1,2,…,m i},e ik frame i The kth traffic element entity relationship, m i is the number of traffic element relationships in the current frame, and each traffic element entity relationship eik is a triplet of (id, r, id), which is a key-value pair set N used to represent node information. i ={id ij :(c ij ,s ij ,p ij ,loc ij ,v ij ,a ij ,color ij ,text ij ,b ij )|j=1,2,…,|frame i |}, each element is a key-value pair of id:(c,s,p,loc,v,a,color,t,b).

[0117] Step 2: Generate scene nodes corresponding to each frame based on the traffic element entity relationship diagram.

[0118] Before proceeding to the next steps, some concepts need to be introduced. The semantic information contained in the global semantic information obtained through the above steps is mainly divided into two categories, non-associative semantics and associative semantics. Non-associative semantics only depends on the relevant attributes of the traffic element entity itself, such as "the speed of vehicle one is thirty kilometers per hour"; associative semantics depends on the attributes of a pair of traffic element entities that have a relationship, such as "the distance between vehicle one and vehicle two is ten meters, and vehicle two is in front of vehicle one".

[0119] Step 2.1: Classify the attributes of traffic element entities.

[0120] When constructing knowledge graph nodes, we hope that the entities have attribute categories determined by attributes, such as "vehicles with faster speed, the front vehicle that is slowing down", etc. Among them, faster speed is a non-associative semantic category of the speed attribute, deceleration is a non-associative semantic category of the behavior attribute, and the front vehicle is an associated semantic category of the location attribute. However, the traffic element entities in the global semantic information obtained through the above steps do not have the above-mentioned attribute categories. Therefore, it is necessary to classify according to the relevant attributes of the entities.

[0121] Step 2.1.1: Formulate classification rules for non-associative semantic attribute categories of traffic element entities.

[0122] First, the attribute categories of the non-associative semantics of each traffic element entity are classified. The attribute classification standards under each entity category are established. For example, for the category of small cars of traffic elements, the attribute classification standards related to speed are established for different traffic scenes. "Speed ​​is less than or equal to 20 kilometers per hour" can be recorded as "slow speed" category, "speed is greater than 20 kilometers per hour and less than or equal to 40 kilometers per hour" can be recorded as "normal speed" category, and "speed is greater than 40 millimeters per hour" can be recorded as "fast speed" category. It should be noted that for a certain category of traffic elements, some attributes will not have an impact, such as the color of small vehicles, the behavior of traffic lights, etc. Such attributes are called irrelevant attributes under the category of traffic elements. For irrelevant attributes of a certain traffic element category, they can be optionally discarded to eliminate redundant information. In addition, the classification of non-associative semantic attribute categories only needs to be performed on attributes with discrete attribute values. Discrete attributes can directly use attribute values ​​as attribute categories.

[0123] Step 2.1.2: Formulate classification rules for the semantic attribute categories associated between traffic element entities.

[0124] Before classifying the relevant semantic attribute categories between traffic element entities, some concepts need to be introduced. For a certain entity n, taking entity n as the source entity, the set of entities that can be reached through a single relationship in the traffic element entity graph of the current frame is called the neighborhood of entity n, denoted as δ(n).

[0125] It should be noted that a reference entity must be selected when classifying the associated semantic attribute category of an entity. In the present invention, the associated semantic attribute category classification takes a certain entity as a reference entity and performs classification on the neighborhood entities of the entity. Taking the location attribute as an example, there are two entities, Vehicle One and Vehicle Two, in the entity relationship diagram of the traffic element of the current frame, and there is one relationship (Vehicle One, Overtaking, Vehicle Two). When Vehicle Two is located in front of Vehicle One, the associated semantic category of Vehicle Two to Vehicle One is the "front vehicle" category. It should be noted that the front vehicle category is only meaningful when Vehicle One is used as a reference entity. When separated from the reference entity of Vehicle One, the "front vehicle" category loses its meaning.

[0126] Step 2.2: Generate scene nodes.

[0127] After the above steps, the current results include the traffic element entity relationship diagram, node non-association semantic attribute classification rules, and node association semantic attribute classification rules. First, obtain the traffic element entity relationship diagram G of the current frame i = <E i ,N i >, entity non-associative semantic attribute classification rule Rule nr , entity association semantic attribute classification rule Ruler It should be noted that Rule nr and Rule r Contains classification rules for all relevant attributes.

[0128] Step 2.2.1: Create the source semantic entity set Set of the current frame s and the target semantic entity set Set t .

[0129] Step 2.2.2: Get the entity relationship graph G of traffic elements in the frame i = <E i ,N i >.

[0130] Step 2.2.3: Traverse the entity information N in the current frame i , according to the currently traversed entity n s Category c s The result of the semantic rule Rule that is not associated with this entity nr (n s ), create source entity category label s , and added to the source semantic entity set Set s middle.

[0131] Step 2.2.4: Traverse the neighboring entities n of the current entity t , according to the neighborhood entity n t Category c t and associated semantic rule result Rule nr (n s ,n t ), create the target entity category label t , and added to the target semantic entity set Set t middle.

[0132] Step 2.2.5: Traverse the current entity n s and neighborhood entity n t The relationship between them is used to establish the source entity category label s and target entity category label t The type of directed edges is the same as the currently traversed relationship.

[0133] Step 2.2.6: The algorithm returns the source semantic entity set Set s , target semantic entity set Set t , and the traffic element entity relationship diagram G as the scene node S of the current frame i = <Set s ,Set t ,G i>.

[0134] Step 3: After filtering the scene nodes based on the traffic element entity relationship diagram, calculate the out-edge probability of the scene nodes according to the example traffic element entity relationship diagram to generate a scene evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after filtering.

[0135] Step 3.1: Create a scene node collection.

[0136] Based on the above steps, we have obtained the global semantic information and knowledge graph for each scenario. Next, we will introduce how to construct the knowledge graph for each scenario in the scenario evolution.

[0137] Step 3.1.1: Initially, a scene node constructed at a random frame is added to the scene node set.

[0138] Step 3.1.2: Traverse all frames and obtain the scene node S of the current frame i = <Set s ,Set t ,G i >.

[0139] Step 3.1.3: Traverse the scene node set and obtain the reference scene node S i ′= <Set s ′,Set t ′,G′ i >The traffic element entity relationship diagram G′ i .

[0140] Step 3.1.4: According to relevant algorithms, such as the nauty algorithm, determine the traffic element entity relationship diagram G i Entity relationship diagram with traffic elements G′ i Whether the entity relationship graphs of the traffic elements are the same and whether the association semantics of the neighborhood nodes of the corresponding nodes are the same, if the above conditions are met, the current frame is discarded; if the scene node set is traversed and no scene node that meets the conditions is found, the entity relationship graph of the scene traffic element is added to the entity relationship graph set, and the corresponding scene node is added to the scene node set.

[0141] It should be noted that each element in the scene node set corresponds to a traffic element entity relationship diagram, and the current traffic element entity relationship diagram is used as a scene example traffic element entity relationship diagram. When checking the scene node set, if no scene is found that is the same as the current scene, it means that the current scene is to be added to the scene node set, which means that it has not appeared before.

[0142] Step 3.2: Scenario evolution probability statistics.

[0143] Based on the above steps, the example traffic element entity relationship diagram of all scenes is obtained. In order to better describe the evolution process of driving experience scenes, the actions and evolution scenes that lead to the scenes shown in the data are counted. Traverse each frame in the data to determine whether the scene nodes of the current frame and the next frame are the same. If they are the same, traverse the next frame; if they are not the same, find the example traffic element entity relationship diagram corresponding to the current frame and the next frame in the scene node set, and add an edge between the scene node corresponding to the current frame and the scene node corresponding to the next frame. The edge has two attributes, namely the action type and the number of times; when adding an edge, an edge with the same action type is found, and the number of edges of the same type is increased by one.

[0144] After traversing all frames, traverse the scene node set and calculate the evolution probability corresponding to each outgoing edge according to the outgoing edge action type of the current scene node. For example, there is an edge with the action of "left lane change" between scene node 1, scene node 2 and scene node 3, and the number of times is 2 and 8 respectively. Then after calculation, the probability of scene node 1 evolving to scene node 2 through the action of "left lane change" is 0.2, and the probability of evolving to scene node 3 is 0.8.

[0145] Based on the above-mentioned autonomous driving scenario evolution knowledge graph construction method, the present invention also proposes a risk prediction and assessment method, such as Figure 2 As shown, the steps include steps A to C.

[0146] Step A: Construct an entity relationship diagram of traffic elements of a scene S.

[0147] For a scene S to be detected, the present invention can first generate a traffic element entity relationship diagram of the scene S according to the description in step 1 of the method for constructing an autonomous driving scene evolution knowledge graph.

[0148] Step B: Compare the similarity between the entity relationship graph of traffic elements of the scene to be tested and the example entity relationship graph of traffic elements corresponding to the scene evolution knowledge graph to obtain the scene node of the scene S in the evolution knowledge graph.

[0149] This embodiment uses the graph edit distance as the measure of graph similarity and selects the example traffic element entity relationship graph with the greatest similarity to obtain the scene node of scene S in the scene evolution knowledge graph. For example, given G1=<N1,E1> ,G2=<N2,E2> , respectively set the node replacement, deletion, and addition costs, and edge replacement, deletion, and addition costs. Remember that the number of nodes in G2 is greater than the number of nodes in G1. Take G1 with fewer nodes as a reference, take the corresponding number of nodes from G2 and arrange and combine them to form a node map. First, calculate the edit cost of the G2 mapping set and G1, and then calculate the cost of deleting the free nodes of G2. The sum of the two is the total edit cost. If the node has attributes, the node distance needs to be set according to the relevant attributes. For example, if the node has only one speed attribute, the absolute value of the speed difference can be used as the node distance, and when replacing, deleting, and adding nodes, the node distance is used as the node cost coefficient to calculate the node edit cost.

[0150] Step C: Calculate the risk probability of executing each action under the scenario node on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

[0151] When in the current scene, the risk probability of performing each action in the current scene can be calculated based on the scene evolution knowledge graph. It takes a certain amount of time to complete a driving action, which can be divided into several time frames. Assuming that the current action to be completed is divided into n time frames, the evolution probability corresponding to the last time frame can be calculated frame by frame based on the dynamic programming algorithm. According to the example traffic element entity relationship diagram of all possible evolved scenes in the last time frame, the scene risk is comprehensively evaluated based on relevant indicators such as TTC, comfort, safety, closest vehicle distance, traffic speed, traffic density, etc. The evaluation result is used as the evolution risk corresponding to the action in the current scene.

[0152] Take TTC as an example. Figure 3 As shown in the figure, in the current scenario, the action that the unmanned vehicle will take is "right lane change", and the action "right lane change" takes time T, and the total time required is time frame.

[0153] Based on the dynamic programming algorithm, remember fra i,J is the jth scene node in the Ith frame, fra i indicates fra i,j The set of all components can be represented by fra i-1 Evolved through behavior b. In the scene evolution knowledge graph, use represents the evolution from the initial frame to the frame after i time frames i,j The evolution probability of the initial frame only contains fra 0,1 ,and It can be calculated by the following transfer equation

[0154]

[0155] in, For from fra i-1,k Evolved to fra through behavior b i,j The evolution probability of .

[0156] Through the above process, we can calculate The probability of reaching n scenes after a time frame is

[0157] Based on the information of the example traffic element entity relationship diagram of n scenarios, the TTC under these n scenarios is calculated according to the following TTC formula: n}:

[0158]

[0159] Among them, d is the distance between the unmanned vehicle and the front vehicle, v rel is the relative speed difference, ∈ is a very small quantity used to prevent the denominator from being zero.

[0160] Based on the above results, the TTC risk value of the "right lane change" action can be calculated as:

[0161]

[0162] In summary, due to the limitations of traditional knowledge graphs, it is particularly important to build a scene evolution knowledge graph in the field of autonomous driving. The scene evolution knowledge graph uses a single scene as a node, retains global semantic information through a graph structure, and the nodes are connected by directed edges consisting of the behavior of the subject and the evolution probability. This new type of knowledge graph takes the behavior of the subject as the core, which can not only provide guidance for its behavioral decision-making, but also has the advantages of predicting the probability that the current scene will be converted to various scenes due to the behavior and predicting the risk assessment under the target scene.

[0163] The present invention has formulated attribute semantic classification rules, and established an example traffic element entity relationship diagram based on global semantic information. Specifically, the present invention extracts global semantic information frame by frame based on actual image data and point cloud data, and formulates attribute semantic classification rules to refine the information in the scene to obtain relatively unified entity semantic information. In this way, a traffic element entity relationship diagram is established, and an example traffic element entity relationship diagram is retained. Relatively unified entity semantic information can semantically summarize the traffic element entity relationship diagram and avoid the problem of retaining too much redundant information.

[0164] The present invention is based on a data-driven approach to calculate the probability of scene evolution caused by the behavior of an unmanned vehicle and to establish a scene evolution knowledge graph. Specifically, the present invention extracts global semantic information frame by frame based on actual image data and point cloud data, and constructs a single scene knowledge graph based on global semantic information. The current frame single scene knowledge graph is used as the source node, the next frame as the target node, and the action of the unmanned vehicle in the current frame is used as a sample to statistically analyze the probability of action scene evolution from the source node to the target node, and to construct a scene evolution knowledge graph based on this, solving the problem of lack of correlation between scenes in traditional knowledge graphs.

[0165] The present invention is based on the scene evolution knowledge graph, predicts the evolution scene according to the unmanned vehicle behavior, and estimates the risk in each evolution scene. Specifically, based on the constructed scene evolution knowledge graph, the present invention provides a method for calculating the risks contained in each action decision in the current scene. According to the behavior decision in the current scene, the scene evolution knowledge graph is used to obtain the risk estimate of the behavior decision based on the scene matching algorithm, dynamic programming algorithm and scene risk index calculation method, which solves the problem that the traditional knowledge graph cannot evaluate the action risk and cannot provide the optimal behavior decision opinion.

[0166] Exemplary Systems

[0167] Figure 4 A system for constructing a knowledge graph of autonomous driving scene evolution is shown according to an exemplary embodiment. Figure 4 The system includes: a first entity relationship diagram generation module 410, a scene node attribute generation module 420 and a scene evolution knowledge graph generation module 430.

[0168] A first entity relationship diagram generation module 410, for generating an entity relationship diagram of traffic elements of each frame according to the driving data of the unmanned vehicle;

[0169] A scene node attribute generation module 420, for generating a scene node corresponding to each frame based on the traffic element entity relationship diagram;

[0170] The scene evolution knowledge graph generation module 430 is used to filter the scene nodes based on the traffic element entity relationship diagram, and then calculate the out-edge probability of the scene nodes according to the example traffic element entity relationship diagram to generate a scene evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after filtering.

[0171] Optionally, the driving data includes: multi-view time-series image data, laser radar time-series point cloud data and sensor data;

[0172] The first entity relationship diagram generation module 410 is also used to: frame based on the multi-view time-series image data and the laser radar time-series point cloud data i Extract global semantic information and obtain global semantic information Among them, scene i Indicates frame i The scene identifier, id ij Indicates frame i The unique identifier of the jth traffic element entity in c ij Indicates frame i The traffic element category occupied by the jth traffic element entity in s ij Indicates frame i The pixel area occupied by the jth traffic element entity in ij Indicates frame i Identified as a traffic element entity id ij Point set, |frame i | indicates frame i The number of traffic element entities under the unmanned vehicle pixel area s ij and point set p ij All are empty sets; use sensor data to frame frame i Extract the entity parameters of traffic elements and combine them with global semantic information Get global semantic information Among them, loc ij Indicates the entity id of the traffic element ij The location information, v ij Indicates the entity id of the traffic element ij The speed information, a ij Indicates the entity id of the traffic element ij Acceleration information, color ij Indicates the entity id of the traffic element ij Color information, text ij Indicates the entity id of the traffic element ij Text information, b ij Indicates the entity id of the traffic element ij Behavioral information; based on the global semantic information Get any two traffic element entity IDs ij The relationship between traffic element entities ik , and combined with global semantic information Get global semantic information FM i = <scene i ,Gi >; Among them, the traffic element entity graph G i = <E i ,N i >, a directed edge set E used to represent entity relationships i ={e ik |k=1,2,…,m i},e ik frame i The kth traffic element entity relationship, m i is the number of traffic element relationships in the current frame, and each traffic element entity relationship e ik is a triplet of (id, r, id), which is a key-value pair set N used to represent node information. i ={id ij :(c ij ,s ij ,p ij ,loc ij ,v ij ,a ij ,color ij ,text ij ,b ij )|j=1,2,…,|frame i |}, each element is a key-value pair of id:(c,s,p,loc,v,a,color,t,b).

[0173] Optionally, the first entity relationship graph generation module 410 is further used to: use a pre-trained image panoramic semantic segmentation model Model image Perform global semantic segmentation on multi-view images frame by frame, track traffic elements in multi-view images, and use different labels to identify pixel areas in the image sequence to obtain the image global semantic segmentation result SSRes image ; Use the pre-trained point cloud global semantic segmentation model Model pc Perform global semantic segmentation on the LiDAR point cloud data frame by frame, track the traffic elements appearing in the point cloud, and use different labels to identify the point cloud subsets in the sequence point cloud data to obtain the point cloud global semantic segmentation result SSRes pc ; The global semantic segmentation result of the image SSRes image And point cloud global semantic segmentation result SSRes pc Perform matching; generate global semantic information based on the matching results and the semantic information of the unmanned vehicle

[0174] Optionally, the scene node attribute generation module 420 is also used to: obtain node non-associated semantic attribute classification rules and node associated semantic attribute classification rules; wherein the non-associated semantic attribute category classification rules are rules generated based on non-associated semantic attributes that will not affect other entities or relationships between entities, and the associated semantic attribute category classification rules are rules generated based on associated semantic attributes that will affect other entities or relationships between entities; obtain node non-associated semantic attribute classification rules and node associated semantic attribute classification rules; wherein the non-associated semantic attribute category classification rules are rules generated based on non-associated semantic attributes that will not affect other entities or relationships between entities, and the associated semantic attribute category classification rules are rules generated based on associated semantic attributes that will affect other entities or relationships between entities; traverse each frame and create a current frame source semantic node set Set s and the target semantic node set Set t ; Get the traffic element entity relationship graph G of the current frame i = <E i ,N i >; Traverse the traffic element entity relationship graph G i Traffic element entity n s , and generate each traffic element entity n s Non-associative semantic rule result Rule nr (n s ) and the traffic element entity n s With neighboring node n t The result of the associated semantic rule Rule nr (n s ,n t );According to the traffic element entity n s Category c s and non-associative semantic rule results Rule nr (n s ), create source node category label s , and label the source node category s Add to the source semantic node set Set s According to the neighborhood node n t Category c t and associated semantic rule result Rule nr (n s ,n t ), create the target node category label t , and label the target node category t Add to the target semantic node set Set t Middle; traverse traffic element entity node n s and neighboring node nt The relationship between them is used to establish the source node category label s and target node category label t Directed edges between; based on the source semantic node set Set s , target semantic node set Set t , source node category label s and target node category label t The directed edges between them and the entity relationship graph G of traffic elements i , generate the scene node S corresponding to the current frame i = <Set s ,Set t ,G i >.

[0175] Optionally, the scene evolution knowledge graph generation module 430 is further used to: randomly add a scene node to the scene node set; traverse all frames to obtain the traffic element entity relationship graph G of the current frame i ; Traverse the scene node set to obtain the traffic element entity relationship graph G′ corresponding to a scene node i ; Determine the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Whether they are the same, and if they are the same, the result of the associated semantic rule of the neighboring nodes of the corresponding traffic element entity node Rule nr (n s ,n t ) are the same; in the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Different, or traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i The result of the association semantic rule of the neighboring nodes of the same but corresponding traffic element entity node nr (n s ,n t ) In different cases, the scene node corresponding to the current frame is added to the scene node collection.

[0176] Optionally, the scene evolution knowledge graph generation module 430 is further used to: traverse each frame in the driving data; determine the current frame frame i and the next frame i+1 Are the scene nodes the same? If the current frame i and the next frame i+1 If the scene node is the same as the next frame i+1 =frame (i+1)+1, and re-execute the judgment of the current frame i and the next frame i+1 Are the scene nodes the same? If the current frame i and the next frame i+1 If the scene node is the same as the current frame i The corresponding scene node and the next frame i+1 Add an edge to the corresponding scene node; the attributes of the edge include: the action type and the number of times the action type occurs in the scene evolution knowledge graph, the number corresponding to the current frame frame i The outgoing edge evolution probability of the corresponding scene node.

[0177] Figure 5 A risk prediction and evaluation system is shown according to an exemplary embodiment. Figure 5 The system includes: a second entity relationship diagram generating module 510, a similarity calculating module 520 and a risk assessment result generating module 530.

[0178] A second entity relationship diagram generation module 510, for constructing a traffic element entity relationship diagram of a scene S;

[0179] A similarity calculation module 520 is used to compare the similarity between the traffic element entity relationship graph of the scene S and the example traffic element entity relationship graph corresponding to the scene evolution knowledge graph, so as to obtain the scene node of the scene S in the evolution knowledge graph;

[0180] The risk assessment result generation module 530 is used to calculate the risk probability of executing each action under the scenario node on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

[0181] Optionally, the risk assessment result generating module 530 is further configured to: r The time consumed T is used to obtain the number of frames required. Δt is the time interval between frames; based on the scene evolution knowledge graph, obtain The scene S′ after frames is obtained, and the probability of reaching each scene S′ is calculated by using a dynamic programming algorithm; the scene risk of scene S′ is calculated according to the example traffic element entity relationship diagram and risk indicators of scene S′, and the risk indicators include: TTC, comfort, safety, closest vehicle distance, traffic speed and traffic density; based on the probability of reaching each scene S′ and the scene risk of the scene S′, the risk assessment result under scene S is obtained.

[0182] The exemplary system is a system embodiment corresponding to the exemplary method described above. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0183] Exemplary Electronic Devices

[0184] Figure 6 1 is a block diagram of an electronic device 600 according to an exemplary embodiment. The electronic device 600 may be a terminal, a notebook computer, a desktop computer, a server, a computer cluster or other types of electronic devices.

[0185] Reference Figure 6 , the electronic device 600 may include at least one processor 610 and a memory 620. The processor 610 may execute instructions stored in the memory 620. The processor 610 is communicatively connected to the memory 620 via a data bus. In addition to the memory 620, the processor 610 may also be communicatively connected to an input device 630, an output device 640, and a communication device 650 via a data bus.

[0186] The processor 610 may be any conventional processor, such as a central processing unit (CPU), a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0187] The memory 620 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0188] In the disclosed embodiment, executable instructions are stored in the memory 620, and the processor 610 can read the executable instructions from the memory 620 and execute the instructions to implement all or part of the steps of the autonomous driving scenario evolution knowledge graph construction method or the risk prediction and assessment method in the above exemplary embodiments.

[0189] Exemplary computer-readable storage media

[0190] In addition to the above methods and devices, the exemplary embodiments of the present disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to implement all or part of the steps described in the above exemplary embodiments.

[0191] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0192] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of readable storage media include: static random access memory (SRAM) with one or more wires electrically connected, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0193] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for constructing a knowledge graph for autonomous driving scenario evolution, characterized in that: include: Generate an entity relationship diagram of traffic elements in each frame based on the driving data of the unmanned vehicle; Generate a scene node corresponding to each frame based on the traffic element entity relationship diagram; After the scene nodes are screened based on the traffic element entity relationship diagram, the out-edge probability of the scene nodes is calculated according to the example traffic element entity relationship diagram to generate a scene evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after screening.

2. The method for constructing a knowledge graph for autonomous driving scenario evolution according to claim 1, characterized in that: The driving data includes: multi-view time-series image data, laser radar time-series point cloud data and sensor data; The generating of the traffic element entity relationship diagram of each frame according to the driving data of the unmanned vehicle includes: Frame alignment based on multi-view temporal image data and lidar temporal point cloud data i Extract global semantic information and obtain global semantic information Among them, scene i Indicates frame i The scene identifier, id ij Indicates frame i The unique identifier of the jth traffic element entity in c ij Indicates frame i The traffic element category occupied by the jth traffic element entity in s ij Indicates frame i The pixel area occupied by the jth traffic element entity in ij Indicates frame i Identified as a traffic element entity id ij Point set, |frame i | indicates frame i The number of traffic element entities under the unmanned vehicle pixel area s ij and point set p ij are all empty sets; Frame using sensor data i Extract the entity parameters of traffic elements and combine them with global semantic information Get global semantic information Among them, loc ij Indicates the entity id of the traffic element ij The location information, v ij Indicates the entity id of the traffic element ij The speed information, a ij Indicates the entity id of the traffic element ij Acceleration information, color ij Indicates the entity id of the traffic element ij Color information, text ij Indicates the entity id of the traffic element ij Text information, b ij Indicates the entity id of the traffic element ij Behavioral information; Based on the global semantic information Get any two traffic element entity IDs ij The relationship between traffic element entities ik , and combined with global semantic information Get global semantic information FM i = <scene i ,G i >; Among them, the traffic element entity graph G i = <E i ,N i >, a directed edge set E used to represent entity relationships i ={e ik |k=1,2,…,m i },e ik frame i The kth traffic element entity relationship, m i is the number of traffic element relationships in the current frame, and each traffic element entity relationship e ik is a triplet of (id, r, id), which is a key-value pair set N used to represent node information. i ={id ij :(c ij ,s ij ,p ij ,loc ij ,v ij ,a ij ,color ij ,text ij ,b ij )|j=1,2,…,|frame i |}, each element is a key-value pair of id:(c,s,p,loc,v,a,color,t,b).

3. The method for constructing a knowledge graph for autonomous driving scenario evolution according to claim 2, characterized in that: The method is based on the frame alignment of multi-view temporal image data and laser radar temporal point cloud data i Extract global semantic information and obtain global semantic information include: Use the pre-trained image panoptic semantic segmentation model image Perform global semantic segmentation on multi-view images frame by frame, track traffic elements in multi-view images, and use different labels to identify pixel areas in the image sequence to obtain the image global semantic segmentation result SSRes image ; Use the pre-trained point cloud global semantic segmentation model Model pc Perform global semantic segmentation on the LiDAR point cloud data frame by frame, track the traffic elements appearing in the point cloud, and use different labels to identify the point cloud subsets in the sequence point cloud data to obtain the point cloud global semantic segmentation result SSRes pc ; The global semantic segmentation result of the image is SSRes image And point cloud global semantic segmentation result SSRes pc Make a match; Generate global semantic information based on the matching results and the semantic information of the unmanned vehicle 4. The method for constructing a knowledge graph for autonomous driving scenario evolution according to claim 2, characterized in that: Generating scene nodes corresponding to each frame based on the traffic element entity relationship diagram includes: Obtain node non-association semantic attribute classification rules and node association semantic attribute classification rules; wherein the non-association semantic attribute category classification rules are rules generated based on non-association semantic attributes that will not affect other entities or relationships between entities, and the association semantic attribute category classification rules are rules generated based on association semantic attributes that will affect other entities or relationships between entities; Traverse each frame and create the current frame source semantic node set Set s and the target semantic node set Set t ; Get the traffic element entity relationship graph G of the current frame i = <E i ,N i > Traversing the traffic element entity relationship graph G i Traffic element entity n s , and generate each traffic element entity n s Non-associative semantic rule result Rule nr (n s ) and the traffic element entity n s With neighboring node n t The result of the associated semantic rule Rule nr (n s ,n t ); According to the traffic element entity n s Category c s and non-associative semantic rule results Rule nr (n s ), create source node category label s , and label the source node category s Add to the source semantic node set Set s middle; According to the neighboring node n t Category c t and associated semantic rule result Rule nr (n s ,n t ), create the target node category label t , and label the target node category t Add to the target semantic node set Set t middle; Traverse traffic element entity node n s and neighboring node n t The relationship between them is used to establish the source node category label s and target node category label t directed edges between ; Based on the source semantic node set Set s , target semantic node set Set t , source node category label s and target node category label t The directed edges between them and the entity relationship graph G of traffic elements i , generate the scene node S corresponding to the current frame i = <Set s ,Set t ,G i >.

5. The method for constructing a knowledge graph for autonomous driving scenario evolution according to claim 4, characterized in that: The screening of scene nodes based on the traffic element entity relationship diagram includes: Randomly add a scene node to the scene node collection; Traverse all frames and obtain the traffic element entity relationship graph G of the current frame i ; Traverse the scene node set to obtain the traffic element entity relationship graph G′ corresponding to a scene node i ; Determine the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Whether they are the same, and if they are the same, the result of the associated semantic rule of the neighboring nodes of the corresponding traffic element entity node Rule nr (n s ,n t ) are the same; In the traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i Different, or traffic element entity relationship diagram G i With the reference traffic element entity relationship diagram G′ i The result of the association semantic rule of the neighboring nodes of the same but corresponding traffic element entity node nr (n s ,n t ) In different cases, the scene node corresponding to the current frame is added to the scene node collection.

6. The method for constructing a knowledge graph for autonomous driving scenario evolution according to claim 1, characterized in that: The outbound edge probability of the scene node is calculated based on the example traffic element entity relationship graph, including: Traverse each frame in the driving data; Determine the current frame i and the next frame i+1 Are the scene nodes the same? If the current frame i and the next frame i+1 If the scene node is the same as the next frame i+1 =frame (i+1)+1 , and re-execute the judgment of the current frame i and the next frame i+1 Are the scene nodes the same? If the current frame i and the next frame i+1 If the scene node is the same as the current frame i The corresponding scene node and the next frame i+1 Add an edge to the corresponding scene node; the attributes of the edge include: the action type and the number of times the action type occurs in the scene evolution knowledge graph, the number corresponding to the current frame frame i The outgoing edge evolution probability of the corresponding scene node.

7. A risk prediction and assessment method, characterized in that: The method comprises: Construct an entity relationship diagram of traffic elements of a scene S; Compare the similarity between the entity relationship graph of traffic elements of the scene S and the example entity relationship graph of traffic elements corresponding to the scene evolution knowledge graph constructed based on the method for constructing the autonomous driving scene evolution knowledge graph according to any one of claims 1 to 6, and obtain the scene node of the scene S in the evolution knowledge graph; The risk probability of executing each action under the scenario node is calculated on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

8. The risk prediction and assessment method according to claim 7, characterized in that: The risk probability of executing each action under the scene node is calculated on the scene evolution knowledge graph to obtain the risk assessment result under the scene S, including: According to the execution action A r The time consumed T is used to obtain the number of frames required. Δt is the time interval between frames; Based on the scenario evolution knowledge graph, obtain The scene S′ after frames, and use the dynamic programming algorithm to calculate the probability of reaching each scene S′; Calculate the scenario risk of scenario S′ according to the example traffic element entity relationship diagram and risk indicators of scenario S′, wherein the risk indicators include: TTC, comfort, safety, closest vehicle distance, traffic speed and traffic density; Based on the probability of reaching each scene S′ and the scene risk of the scene S′, a risk assessment result under the scene S is obtained.

9. A system for constructing a knowledge graph for autonomous driving scenario evolution, characterized in that: include: A first entity relationship diagram generation module, used to generate an entity relationship diagram of traffic elements of each frame according to the driving data of the unmanned vehicle; A scene node attribute generation module, used to generate scene nodes corresponding to each frame based on the traffic element entity relationship diagram; The scenario evolution knowledge graph generation module is used to filter the scene nodes based on the traffic element entity relationship diagram, and then calculate the out-edge probability of the scene nodes according to the example traffic element entity relationship diagram to generate a scenario evolution knowledge graph; wherein the example traffic element entity relationship diagram is the traffic element entity relationship diagram of the scene nodes after filtering.

10. A risk prediction and assessment system, characterized in that: include: A second entity relationship diagram generation module is used to construct a traffic element entity relationship diagram of a scene S; A similarity calculation module, used to compare the similarity between the traffic element entity relationship graph of the scene S and the example traffic element entity relationship graph corresponding to the scene evolution knowledge graph constructed based on the method for constructing the autonomous driving scene evolution knowledge graph according to any one of claims 1 to 6, to obtain the scene node of the scene S in the evolution knowledge graph; The risk assessment result generation module is used to calculate the risk probability of executing each action under the scenario node on the scenario evolution knowledge graph to obtain the risk assessment result under the scenario S.

Images