Automatic driving safety decision-making method and system fusing scene evolution knowledge graph and large language model

Abstract

The invention relates to an automatic driving safety decision-making method and system fusing a scene evolution knowledge graph and a large language model, and belongs to the field of automatic driving. The method comprises the following steps: acquiring driving data obtained in the driving process of the unmanned vehicle; generating a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the security decision model is constructed based on a large language model; selecting an example traffic element entity relation graph in the scene evolution knowledge graph based on the driving data, and obtaining probable evolution scene sets and probabilities corresponding to each evolution scene in the scene evolution knowledge graph in combination with the number of time frames required for executing a behavior decision result; judging a risk assessment result of the behavior decision result based on the evolution scene set and the probability corresponding to each evolution scene; and under the condition that the risk assessment result meets the requirement, enabling the unmanned vehicle to execute the decision result. According to the invention, more accurate driving behavior suggestions are provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to an autonomous driving safety decision-making method and system that integrates a scene evolution knowledge graph and a large language model. Background Art

[0002] Knowledge Graph is a semantic network that uses a graph model to describe entities and the relationships between entities. Its basic components are the "entity-relationship-entity" triples, as well as entity attribute information of entities and their related attribute values. Each node in the knowledge graph represents an entity in the real world, and each edge is an association relationship between entities or between entities and attributes.

[0003] The autonomous driving system has many advantages, such as improving traffic safety, reducing traffic congestion, improving traffic efficiency, and saving labor costs. A good autonomous driving knowledge graph can help the autonomous driving system take better safety measures, make quick decisions, and improve the explainability of decisions.

[0004] In recent years, large models of autonomous driving based on large language models have received widespread attention. Using large language models as a base to achieve autonomous driving of unmanned vehicles can not only enable the autonomous driving system to better understand human instructions, but also make behavioral decisions with higher interpretability. However, large language models often have the characteristics of black boxes, poor interpretability, fuzzy ability boundaries, lack of professionalism, and are prone to "hallucination" problems. As a method of knowledge representation, the scene evolution knowledge graph has the advantages of professionalism, completeness, and correctness, and it plays an important role in integrating, understanding, and applying knowledge, providing a foundation for various intelligent applications. Although the knowledge graph has many advantages, it also has disadvantages such as poor flexibility and lack of intelligence. Summary of the invention

[0005] The present invention provides an autonomous driving safety decision-making method and system that integrates a scene evolution knowledge graph and a large language model, which can provide more accurate driving behavior recommendations.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for autonomous driving safety decision-making integrating a scene evolution knowledge graph and a large language model is provided, the method comprising:

[0007] Step S1: Acquire driving data obtained during the driving process of the unmanned vehicle;

[0008] Step S2: generating a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model;

[0009] Step S3: Based on the driving data, an example traffic element entity relationship diagram is selected in the scene evolution knowledge graph, and combined with the number of time frames T required to execute the behavior decision result, a possible evolution scene set G and the probability p corresponding to each evolution scene are obtained in the scene evolution knowledge graph. i ;

[0010] Step S4: Based on the probability p corresponding to each evolution scenario set G i The risk assessment results for determining the outcome of the behavioral decision;

[0011] Step S5: When the risk assessment result meets the requirements, the unmanned vehicle executes the behavior decision result.

[0012] Optionally, generating a prompt template of a safety decision model according to the driving data to obtain a behavior decision result includes:

[0013] Extracting data information from the driving data using a pre-trained perception model; wherein the driving data includes: image data, laser radar point cloud data and sensor data, and the data information includes: entity information and environmental information;

[0014] Filling single-entity semantic chain information based on the entity information and the environment information to generate multiple single-entity semantic chains corresponding to each entity;

[0015] Generate semantic information of each entity according to multiple single-entity semantic chains corresponding to the entity;

[0016] Filter the semantic information of the entity;

[0017] The filtered semantic information is filled into an input template of a security decision model so that the security decision model outputs a behavior decision result.

[0018] Optionally, screening the single entity semantic chain includes:

[0019] Arrange the entities in the scene in non-descending order according to their straight-line distance from the unmanned vehicle, and retain the single-entity semantic chains corresponding to the top k entities in the sorting;

[0020] and,

[0021] Define the dangerous behavior of the entity and the risk factor of the dangerous behavior, and after arranging the risk factors in non-descending order, retain the single entity semantic chains corresponding to the entities in the first k positions of the sort.

[0022] Optionally, an example traffic element entity relationship diagram is selected from the scene evolution knowledge graph based on the driving data, including:

[0023] Based on the driving data, obtaining a traffic element entity relationship diagram V of the current scene;

[0024] Traversing the entity nodes in the entity relationship graph V of the traffic elements of the current scene, and classifying the entity nodes by non-associative semantic attributes according to the non-associative semantic attribute category classification rules, and classifying the domain nodes of the entity nodes by associated semantic attributes according to the associated semantic attribute category classification rules, thereby generating the entity relationship graph V′ of the traffic elements of the current scene; wherein the non-associative semantic attribute category classification rules are rules generated based on non-associative semantic attributes that will not affect other entities or relationships between entities, and the associated semantic attribute category classification rules are rules generated based on associated semantic attributes that will affect other entities or relationships between entities;

[0025] An example traffic element entity relationship diagram is selected based on the similarity between the traffic element entity relationship diagram V′ of the current scene and each example traffic element entity relationship diagram in the scene evolution knowledge graph.

[0026] Optionally, the acquiring, based on the driving data, a traffic element entity relationship graph V of the current scene includes:

[0027] Preprocessing the driving data using a pre-trained panoramic semantic segmentation model and a pre-trained parameter extraction model to obtain entity data and attribute data of each entity in the current scene; wherein the entity data includes: a unique identifier, a category, a pixel area, and point cloud data;

[0028] Based on the attribute data, a pre-trained relationship extraction model is used to obtain relationship elements between entities, and the relationship elements are combined with the entity data and the attribute data to obtain global semantic information in the current scene;

[0029] According to the global semantic information in the current scene, a traffic element entity relationship graph V of the current scene is obtained.

[0030] Optionally, selecting an example traffic element entity relationship diagram by comparing the traffic element entity relationship diagram V′ of the current scene with each example traffic element entity relationship diagram in the scene evolution knowledge graph includes:

[0031] For the traffic element entity relationship graph V′ of the current scene and an example traffic element entity relationship graph, the node editing cost between nodes with the same attribute is calculated by combining the classification results of non-associated semantic attributes, the classification results of associated semantic attributes and the difference between corresponding attributes;

[0032] Map the nodes in the example traffic element entity relationship graph to the traffic element entity relationship graph V′ of the current scene, and calculate the total editing cost by combining the node editing cost and the set node replacement cost, node deletion cost, node addition cost, edge replacement cost, edge deletion cost, and edge addition cost;

[0033] According to the total editing cost, an example traffic element entity relationship diagram is selected in the scene evolution knowledge graph.

[0034] Optionally, based on the probability p corresponding to the evolution scene set G and each evolution scene i The risk assessment results for determining the behavior decision results include:

[0035] Based on each evolution scenario g i The values ​​of each risk indicator under this condition are used to generate the evolution scenario g i wherein the risk indicators include: time to collision, safe response time, distance to collision, safety and comfort;

[0036] According to the probability p corresponding to the evolution scenario i With this evolution scenario g i vector, generating the behavior decision result B r The risk estimation vector risk(B r );

[0037] Based on the risk indicators, a judgment matrix A is generated by using the analytic hierarchy process and the 1-9 scaling method;

[0038] Get Matrix Among them, the vector is a one-dimensional column vector with the jth bit set to 1 and the rest bits set to 0 and length num, {c1,c2,...,c m} is the subscript set of risk indicators, m is the number of risk indicators, and num is the number of risk indicators;

[0039] The judgment matrix A is transformed by using the matrix B to obtain the judgment matrix

[0040] Calculate the matrix by the product method The transformation matrix A tran After that, according to the transformation matrix A tran Get the weight vector

[0041] According to the weight vector and the risk estimation vector risk(B r ) to obtain a risk assessment result of the behavioral decision result.

[0042] Optionally, when the risk assessment result meets the requirements, after the unmanned vehicle executes the behavior decision result, the method further includes:

[0043] Recording the environment experienced by the unmanned vehicle frame by frame to obtain a traffic element entity relationship diagram of each frame during the execution of the behavior by the unmanned vehicle;

[0044] The scene nodes are matched by calculating the similarity between the entity relationship graph of traffic elements in each frame and the example entity relationship graph of traffic elements in the scene evolution knowledge graph;

[0045] When the unmanned vehicle performs a behavior action based on the matched scene node, all scene nodes experienced during the execution process are added to the behavior recorder set Among them, B num is the number of actions recorded and executed in the current scenario. Indicates that the current scene node passes the behavior action b i The number of scene nodes S that have arrived;

[0046] Set the behavior threshold γ;

[0047] When the action i Satisfy the conditions Delete all the behavior actions b of the current scene node i The outgoing edges are based on the recorder collection Scene BC All actions in b i Relevant information, recalculate the behavior evolution probability of the corresponding scene node, create the corresponding outgoing edge, and at the same time convert all the numbers Set to 0.

[0048] Optionally, the method further comprises:

[0049] When the risk assessment result does not meet the requirements, a risk estimation counter and a maximum risk estimation round τ are set, and a prompt template of the security decision model is reset; wherein the prompt template of the reset security decision model includes a risk assessment result element;

[0050] In each round of risk estimation, a prompt template of the safety decision model is generated by combining the risk prediction and behavior suggestion corresponding to the risk assessment result, so as to obtain a new behavior decision result, and then re-execute from step S3 to perform risk assessment on the new behavior decision result;

[0051] If the current risk estimation round count is greater than the risk estimation maximum round τ, manual intervention is requested.

[0052] According to a second aspect of an embodiment of the present disclosure, there is provided an autonomous driving safety decision-making system integrating a scene evolution knowledge graph and a large language model, the system comprising:

[0053] A data acquisition module is used to obtain driving data obtained during the driving process of the unmanned vehicle;

[0054] A behavior decision module, used to generate a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model;

[0055] A decision evaluation module is used to select an example traffic element entity relationship diagram in the scene evolution knowledge graph based on the driving data, and obtain a possible evolution scene set G and a probability p corresponding to each evolution scene in the scene evolution knowledge graph in combination with the number of time frames T required to execute the behavior decision result. i ; Based on the probability p corresponding to the evolution scenario set G and each evolution scenario i The risk assessment results for determining the outcome of the behavioral decision;

[0056] The decision execution module is used to enable the unmanned vehicle to execute the behavior decision result when the risk assessment result meets the requirements.

[0057] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the autonomous driving safety decision-making method that integrates the scene evolution knowledge graph and the large language model provided by the first aspect of the present disclosure is implemented.

[0058] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the autonomous driving safety decision-making method that integrates the scene evolution knowledge graph and the large language model provided by the first aspect of the present disclosure is implemented.

[0059] Compared with the prior art, the present invention has at least the following beneficial effects.

[0060] (1) A method is proposed to use precise entity linking technology to ensure accurate matching between environmental entities and entities in the knowledge graph, and to use relationship extraction technology to carefully identify the behavioral rules between entities in order to extract accurate driving behavior recommendations from the knowledge graph.

[0061] (2) For the first time, a method for autonomous driving safety decision-making that integrates knowledge graphs and large language models is proposed, which complements the advantages and disadvantages of scene evolution knowledge graphs and large language models. The scene evolution knowledge graph is based on past driving experience data, and statistical driving scenes are based on the probabilistic graph model of driving behavior. It can also use dynamic programming algorithms to infer scene risks after multiple time steps. The information contained in it can provide professional and complete external knowledge, and the large language model can flexibly use external prompts to produce safer and better behavioral decisions. Therefore, the autonomous driving safety decision-making technology that integrates scene evolution knowledge graphs and large language models is of great significance. A good scene evolution knowledge graph can make up for the lack of scene understanding ability of the large language model and help the large language model make more reasonable and safe behavioral decisions.

[0062] (3) Based on the data collected during the driving process of the autonomous vehicle, including but not limited to image data, lidar data, and sensor data, by adding behavior recorders to scene nodes, the evolution of scene nodes caused by behaviors is counted, and the evolution relationship of scene nodes in the scene evolution knowledge graph is recalculated based on the statistical data, and the relevant information in the scene evolution knowledge graph is updated.

[0063] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flowchart of an autonomous driving safety decision-making method that integrates a scene evolution knowledge graph and a large language model according to an exemplary embodiment.

[0065] Figure 2 It is a block diagram of an autonomous driving safety decision-making system that integrates a scene evolution knowledge graph and a large language model according to an exemplary embodiment.

[0066] Figure 3 is a block diagram of an electronic device 300 according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0068] The present invention provides an autonomous driving safety decision-making method that integrates scene evolution knowledge graph and large language model. It can predict the probability of evolving scenes and make behavioral decision risk judgments based on real-time driving data and during the delay of large language model inference by using the established scene evolution knowledge graph in the field of autonomous driving. Based on the behavioral decision risk estimation results, it is judged whether the decision result of the large model is safe and whether it is necessary to intervene in the scene evolution knowledge graph to help the large model make safer and more complete behavioral decisions. Figure 1 As shown, the autonomous driving safety decision-making technology integrating the scene evolution knowledge graph and the large language model in the present invention includes the following steps S1 to S5.

[0069] Step S1: Acquire driving data obtained during the driving process of the unmanned vehicle.

[0070] In one embodiment of the present invention, the driving data includes image data, lidar point cloud data, sensor data and other information.

[0071] Step S2: Generate a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model.

[0072] Step 2.1: Use the pre-trained perception model to extract data information from the driving data.

[0073] Based on the image data, lidar point cloud data, sensor data and other information obtained during the real-time driving of the unmanned vehicle, the pre-trained perception model Model per Extract information, including but not limited to real-time entity information Message ent , such as speed, position, acceleration, behavior, and environmental information Message envir , such as the area, scene type, traffic regulations and other information.

[0074] Step 2.2: Fill in the single-entity semantic chain information based on the entity information and environment information, and generate multiple single-entity semantic chains corresponding to each entity.

[0075] Single-entity semantic information is used to describe the semantic description of a specific entity in the current scene, which is expressed through multiple single-entity semantic chain structures. A single-entity semantic chain is derived from a sentence based on semantic description. A sentence can be composed of multiple entities and relationships alternating, such as "pedestrians are on the side of the road and riding bicycles", where "pedestrians", "side of the road" and "bicycles" are all entities, and "at" and "riding" are relationships. The semantic chain composed of constantly alternating entities and relationships is the semantic expression that describes the first entity.

[0076] Based on the extracted entity information and environment information, all the entities identified in the entity information are traversed, and single entity semantic chain information is filled for the identified single entity to obtain multiple single entity semantic chains of the currently traversed entity. For example, {pedestrians are riding bicycles on the side of the road, pedestrians are talking on the phone}.

[0077] Step 2.3: Generate semantic information of each entity based on multiple single-entity semantic chains corresponding to the entity.

[0078] Traverse all entities in the current scene, and traverse all single entity semantic chains belonging to the entity, build the semantic information of the entity, and obtain the semantic information Message of each entity in the current scene scene ={Ent i |i=1,2,...,n}, n is the number of entities in the current scene, Ent i ={sentemce ij |j=1,2,...,|Ent i |},|Ent i | is the number of sentences used to describe the semantics of the entity, sentence ij is the jth semantic sentence describing the entity.

[0079] Step 2.4: Filter the semantic information of the entity.

[0080] For large model generation tasks, the input sequence of the large model often has a length limit, and the entity information in the current scene may be too much, resulting in that all of it cannot be used as context information for the large model input. Therefore, the extracted entity information needs to be screened. The present invention provides two screening methods based on distance and dangerous behavior.

[0081] The distance-based screening method uses the distance between the entity in the scene and the unmanned vehicle as the evaluation criterion. The entities in the scene are sorted in non-descending order according to the straight-line distance from the unmanned vehicle. The front entity information is preferentially selected as the context of the large model input. It is considered that the entity information in the first k positions is relatively critical, and the remaining entity information is relatively uncritical.

[0082] Based on the screening method of dangerous behaviors, we first need to define the dangerous behaviors of each category of entities, such as "a motor vehicle is running a red light at an intersection", "a vehicle is driving at high speed at a zebra crossing", etc., and give each dangerous behavior a corresponding risk factor. The setting of the risk factor can be based on existing traffic regulations or customized by the user. The risk factors of entities that are performing dangerous behaviors in the scene are arranged in non-ascending order, and the front entity information is preferentially selected as the context of the large model input. It is considered that the first k entity information is relatively critical, and the remaining entity information is relatively uncritical.

[0083] The information filtered above is recorded as Message scene = <M c ,M nc >, where M c is a set of entity information arranged in non-descending order according to importance, M nc It is a relatively uncritical collection of entity information.

[0084] Step 2.5: Fill the filtered semantic information into the input template of the security decision model so that the security decision model outputs a behavioral decision result.

[0085] Use the pre-trained large language model for downstream tasks. It has been found in practice that giving the large model appropriate input context can better help the large model complete downstream tasks. Prompt engineering is carried out based on the pre-trained task form of the large model, aiming to find an input template that can better help the large model complete downstream tasks. In one embodiment, the present invention provides an input template template that can be used to guide the large model to make autonomous driving decisions:

[0086] template=[P 1:i ][M c ][P j:k ][x][P l:t ][mask][P v:max ]

[0087] Among them, [M c ] is the relatively important entity semantic information obtained through the screening in step 2.2, [x] is the original problem description, [mask] is the model input information, P 1:i , P j:k , P l:t , P v:max All are Prompt contents, and max is the longest input sequence length of the large model.

[0088] Based on the above input template, given an instance:

[0089] template=The more important entity information in the scene is: [M c ], requirement: [x], what action to take: [mask][]

[0090] Among them, [mask] is the location where the large model will be output, and [] means there is no prompt content here.

[0091] In actual use, the above template can be used to construct a large model input. In addition, other methods can be used to construct a template for large model input, such as using P-tuning to form prompt content through soft prompts, so that the large model can train itself to obtain the most suitable template for the current task.

[0092] Finally, according to the constructed input template template, the obtained entity information set M c Fill in the input template template as the input of the large model. In one embodiment, the input template template can be expressed as:

[0093] template = [The following important entity information exists in the current scene:][Pedestrian 1 is walking on the zebra crossing, pedestrian 1 is directly in front of the driverless car, vehicle 1 is slowing down in the lane, vehicle 1 is on the right side of the driverless car][; Based on the above information, the following questions need to be answered:][What is the safest driving behavior at present?][The answer to the above question is:

[0094] ][mask][]

[0095] Among them, [mask] is the location where the large model will be output, and [] means there is no prompt content here.

[0096] Input the filled template into the big model, and after the big model makes a decision, output the decision information. According to the structure of the template, extract the [mask] part of the message from the template. mask The resulting message mask It is a semantic representation of vehicle behavior and may not be directly executed by the unmanned vehicle, such as Message mask = "The safest driving behavior at present is to gradually reduce the speed." Among them, "the safest driving behavior at present is" is irrelevant information, and "gradually reduce the speed" is a semantic expression of the behavior. "Gradually reduce the speed" does not indicate at what acceleration the speed is reduced, so subsequent processing is required to obtain clearer behavioral decision information.

[0097] Message mask The information is processed to obtain the output B of the large model r For example, Message mask For "The safest driving behavior at present is to gradually reduce the speed.", extract the semantically expressed decision information: "gradually reduce the speed", and according to the rules, transform the semantically expressed decision information into an input B that is acceptable to the vehicle. r , for example: "{acceleration: -2.3, angle: 0}".

[0098] Step S3: Based on the driving data, an example traffic element entity relationship diagram is selected in the scene evolution knowledge graph, and combined with the number of time frames T required to execute the behavior decision result, a possible evolution scene set G and the probability p corresponding to each evolution scene are obtained in the scene evolution knowledge graph. i .

[0099] When using a large model for downstream task reasoning, text generation is performed serially during the reasoning phase, which takes a certain amount of time and results in a more obvious reasoning delay. In order to make up for the disadvantage of reasoning delay in large models and make rational use of this time, the scene evolution knowledge graph can be used to predict the scene evolution and estimate the risks of various behaviors that the unmanned vehicle can take in the current scene. Before performing scene evolution prediction and risk estimation, it is necessary to match scene nodes based on the entity information and environmental information of the current scene to find out which scene node in the scene evolution graph the current scene belongs to.

[0100] Step 3.1: Based on the driving data, obtain the traffic element entity relationship diagram V of the current scene.

[0101] Based on the image data, lidar point cloud data and sensor data obtained during real-time driving, a pre-trained panoramic semantic segmentation model is used for preprocessing to obtain the unique identifier, category, pixel area and point cloud data of each entity in the current scene; based on the image data, lidar point cloud data, sensor data and semantic segmentation result data obtained during real-time driving, a pre-trained parameter extraction model is used to obtain the attribute data of each entity, including but not limited to attribute parameters such as position, speed, acceleration, color, text, behavior, etc.; based on the results of the parameter extraction model, a pre-trained relationship extraction model is used to obtain the relationship elements between each entity, and combined with the results of the parameter extraction model to obtain the global semantic information in the current scene, and obtain the entity relationship diagram of the traffic elements in the current scene.

[0102] Step 3.2: Traverse the entity nodes in the traffic element entity relationship graph V of the current scene, and classify the entity nodes by non-associative semantic attributes according to the non-associative semantic attribute category classification rules, and classify the domain nodes of the entity nodes by associated semantic attributes according to the associated semantic attribute category classification rules, and then generate the traffic element entity relationship graph V′ of the current scene.

[0103] Based on the traffic element entity relationship diagram and non-association semantic attribute category classification rules obtained in step 3.1, traverse all entity nodes in the traffic element entity relationship diagram, and classify the entity nodes in the traffic element entity relationship diagram in the current scene by non-association semantic attributes; among them, the non-association semantic attribute category classification rules are a set of identification and classification rules for non-association semantic attributes such as "speed" and "color" that will not affect other entities or the relationship between entities, and are used to classify such entity attributes into categories such as "faster speed" and "red".

[0104] Based on the traffic element entity relationship diagram and the classification rules of associated semantic attributes obtained in step 3.1, all entity nodes in the traffic element entity relationship diagram are traversed, and the domain nodes of the entity nodes in the traffic element entity relationship diagram in the current scene are classified by associated semantic attributes, and the results are recorded in the current node. Among them, the classification rules of associated semantic attributes are a set of identification and classification rules for non-associated semantic attributes such as "location" that will affect other entities or relationships between entities, and are used to classify such entity attributes into categories such as "front car" and "rear car".

[0105] Step 3.3: Select an example traffic element entity relationship diagram based on the similarity between the traffic element entity relationship diagram V′ of the current scene and each example traffic element entity relationship diagram in the scene evolution knowledge graph.

[0106] According to the graph edit distance as the measure of graph similarity, the similarity between the current scene traffic element entity relationship graph and the elements in the scene node set is calculated in turn, and the example traffic element entity relationship graph with the greatest similarity is selected as the corresponding node of the current scene. For example, given two example traffic element entity relationship graphs G1 =<N1,E1> ,G2=<N2,E2> , respectively set the node replacement, deletion, and addition costs, and edge replacement, deletion, and addition costs. Remember that the number of nodes in G2 is greater than the number of nodes in G1. Take G1 with fewer nodes as a reference, take the corresponding number of nodes from G2 and arrange and combine them to form a node map. First, calculate the edit cost of the G2 mapping set and G1, and then calculate the cost of deleting the free nodes of G2. The sum of the two is the total edit cost. If the node has attributes, it is necessary to set the node distance according to the relevant attributes. For example, if the node has only one speed attribute, the absolute value of the speed difference can be used as the node distance, and when replacing, deleting, and adding nodes, the node distance is used as the node cost coefficient to calculate the node edit cost.

[0107] Step S4: Based on the probability p corresponding to each evolution scenario set G i The risk assessment result of determining the outcome of the behavioral decision.

[0108] Behavioral decision results based on the output of the large model Br , using the scene evolution knowledge graph to calculate behavior B r The number of time frames required is the number of frames T. Based on the scene evolution knowledge graph, a dynamic programming algorithm is used to calculate the probability P of evolving to all possible evolution scenes after T time frames = {p i |i=1,2,...,s num} and the scene set G = {g i |i=1,2,...,s num}, where s num For Behavior B r The number of possible scenarios after completing the behavior, g i For Behavior B r The example traffic element entity relationship diagram recorded in the scene evolution knowledge graph for the i-th scene that may evolve after completing the behavior; traverse the scene node set S and calculate g i The risk indicators include but are not limited to TTC (time-to-collision), STN (safe time-to-noraml), DTC (distanece-to-collision), safety, comfort and other indicators, and the risk indicator vector set V = {v i |i=1,2,...,s num}, where v i g i The vector composed of the following risk indicators; after obtaining the above results, the behavior B can be calculated according to the following formula r The risk estimate vector is:

[0109]

[0110] Next, a comprehensive evaluation method is used to comprehensively evaluate the behavior risk. The present invention provides a feasible evaluation scheme, which uses the hierarchical analysis method and the 1-9 scaling method to give the judgment matrix A of all risk evaluation indicators. The index subscript set participating in the evaluation task is {c1, c2, ..., c m}, where m is the number of indicators involved in the risk assessment task. is a one-dimensional column vector with the i-th bit being 1 and the rest being 0 and length num, denoted by the matrix The judgment matrix of the evaluation task can be obtained according to the following formula:

[0111]

[0112] The matrix can be calculated according to the product method The transformation matrix Atran and the weight vector

[0113] Calculate the comprehensive risk assessment result of the current scenario according to the following formula: eva :

[0114]

[0115] Step S5: When the risk assessment result meets the requirements, the unmanned vehicle executes the behavior decision result.

[0116] The present invention sets a risk threshold δ. If the comprehensive risk assessment result risk eva If it is less than or equal to the risk threshold δ, the behavior decision result of the large model is accepted.

[0117] In a preferred embodiment, after accepting the behavioral decision results of the large model, the present invention also needs to record the environment experienced by the unmanned vehicle, extract the entity relationship graph of traffic elements in each frame during the execution of the unmanned vehicle, and calculate the entity relationship graph of traffic elements in each frame and the scene node similarity graph in the scene evolution knowledge graph, and perform scene node matching.

[0118] When the unmanned vehicle performs a certain action, a behavior recorder collection is added to all the scene nodes experienced by the unmanned vehicle during the execution process. Among them, B num is the number of actions recorded and executed in the current scenario. Indicates the current scene node, through behavior b i The number of scene nodes S that have arrived. Set the behavior threshold γ. When a behavior b recorded by a scene node i The following conditions are met:

[0119]

[0120] Then delete all the nodes in the current scene that are related to behavior b i Related outgoing edges, based on Scene BC All behaviors i The relevant information recalculates the behavior evolution probability of the corresponding scene node and creates the corresponding outgoing edge. Set to 0.

[0121] In a preferred embodiment, when the risk assessment result does not meet the requirements, step S6 is further included: iterative optimization of large model decisions.

[0122] Based on the risk assessment in step 4, if the behavior decision generated by the large model cannot pass the risk assessment, the large model input needs to be adjusted based on the risk assessment results. Set the risk estimation counter and the maximum risk estimation round τ. The initial value of the risk estimation counter is 0, and the risk estimation counter increases by 1 for each iteration. If the counter value is greater than τ, output a risk warning and request manual intervention; otherwise, continue with the following steps.

[0123] First, the large model input template needs to be adjusted. The present invention provides a feasible template for integrating risk assessment results. risk template:

[0124] template risk =[P 1:i ][M c ][P j:k ][risk][P e:q ][x][P l:t ][mask][P v:max ]

[0125] Among them, [risk] is the content related to the risk assessment results, P e:q The same is true for the prompt content.

[0126] Similarly, other methods can be used to construct templates for large model inputs. risk , such as P-tuning, which uses soft prompts to form prompt content, so that the large model can train itself to obtain the most suitable template for the current task risk .

[0127] Fill the risk prediction and behavior suggestions of the knowledge graph into the template risk , the filled templante risk As a new input for the large model, re-input the large model and execute step S4.

[0128] In summary, the present invention proposes a method that uses precise entity linking technology to ensure accurate matching of environmental entities with entities in the knowledge graph, and uses relationship extraction technology to carefully identify the behavioral rules between entities to extract accurate driving behavior suggestions from the knowledge graph. It achieves fast matching of scene nodes in the scene evolution knowledge graph, ensures that risk estimation is completed within the large model inference delay time, and gives accurate behavioral decision-making opinions.

[0129] This paper proposes for the first time a safe decision-making method for autonomous driving that integrates knowledge graphs and large language models. Through the collaborative mechanism of knowledge graphs and large language models, it effectively integrates the structured professional knowledge decision logic of knowledge graphs and the deep data analysis and reasoning capabilities of large language models, which can significantly reduce the possibility of autonomous driving systems making high-risk driving decisions and enhance the safety of autonomous driving.

[0130] The present invention is based on the data collected during the driving process of the autonomous vehicle, including but not limited to image data, lidar data and sensor data. By adding behavior recorders to scene nodes, the evolution of scene nodes through behavior is counted, and the evolution relationship of scene nodes in the scene evolution knowledge graph is recalculated based on the statistical data, and the relevant information in the scene evolution knowledge graph is updated at the same time. The dynamic update and optimization of the scene evolution knowledge graph is realized, and the knowledge graph is ensured to continuously adapt to environmental changes.

[0131] Exemplary Systems

[0132] Figure 2 An autonomous driving safety decision-making method integrating scene evolution knowledge graph and large language model is shown according to an exemplary embodiment. Figure 2 The device includes: a data acquisition module 210, a behavior decision module 220, a decision evaluation module 230 and a decision execution module 240.

[0133] The data acquisition module 210 is used to acquire the driving data obtained during the driving process of the unmanned vehicle;

[0134] A behavior decision module 220, for generating a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model;

[0135] The decision evaluation module 230 is used to select an example traffic element entity relationship diagram in the scene evolution knowledge graph based on the driving data, and obtain the possible evolution scene set G and the probability p corresponding to each evolution scene in the scene evolution knowledge graph in combination with the number of time frames T required to execute the behavior decision result. i ; Based on the probability p corresponding to the evolution scenario set G and each evolution scenario i The risk assessment results for determining the outcome of the behavioral decision;

[0136] The decision execution module 240 is used to enable the unmanned vehicle to execute the behavior decision result when the risk assessment result meets the requirements.

[0137] Optionally, the behavior decision module 220 is used to: use a pre-trained perception model to extract data information from the driving data; wherein the driving data includes: image data, lidar point cloud data and sensor data, and the data information includes: entity information and environmental information; based on the entity information and the environmental information, fill in the single entity semantic chain information to generate multiple single entity semantic chains corresponding to each entity; based on the multiple single entity semantic chains corresponding to each entity, generate the semantic information of the entity; filter the semantic information of the entity; fill the filtered semantic information into the input template of the safety decision model, so that the safety decision model outputs the behavior decision result.

[0138] Optionally, the behavior decision module 220 is used to: sort the entities in the scene in non-descending order according to their straight-line distance from the unmanned vehicle, and retain the single-entity semantic chains corresponding to the entities in the first k positions of the sort; and define the dangerous behavior of the entity and the risk coefficient of the dangerous behavior, and after sorting the risk coefficient in non-descending order, retain the single-entity semantic chains corresponding to the entities in the first k positions of the sort.

[0139] Optionally, the decision evaluation module 230 is used to obtain a traffic element entity relationship graph V of the current scene based on the driving data; traverse the entity nodes in the traffic element entity relationship graph V of the current scene, and classify the entity nodes by non-associated semantic attributes according to the non-associated semantic attribute category classification rules, and classify the domain nodes of the entity nodes by associated semantic attributes according to the associated semantic attribute category classification rules, and then generate a traffic element entity relationship graph V′ of the current scene; wherein the non-associated semantic attribute category classification rules are rules generated based on non-associated semantic attributes that will not affect other entities or relationships between entities, and the associated semantic attribute category classification rules are rules generated based on associated semantic attributes that will affect other entities or relationships between entities; select an example traffic element entity relationship graph through the similarity between the traffic element entity relationship graph V′ of the current scene and each example traffic element entity relationship graph in the scene evolution knowledge graph.

[0140] Optionally, the decision evaluation module 230 is used to: pre-process the driving data using a pre-trained panoramic semantic segmentation model and a pre-trained parameter extraction model, respectively, to obtain entity data and attribute data of each entity in the current scene; wherein the entity data includes: a unique identifier, a category, a pixel area, and point cloud data; based on the attribute data, use a pre-trained relationship extraction model to obtain the relationship elements between the entities, and combine them with the entity data and the attribute data to obtain the global semantic information in the current scene; based on the global semantic information in the current scene, obtain the traffic element entity relationship diagram V of the current scene.

[0141] Optionally, the decision evaluation module 230 is used to: for the traffic element entity relationship diagram V′ of the current scene and an example traffic element entity relationship diagram, calculate the node editing cost between nodes with the same attributes by combining the non-associated semantic attribute classification results, the associated semantic attribute classification results and the difference between corresponding attributes; map the nodes in the example traffic element entity relationship diagram to the traffic element entity relationship diagram V′ of the current scene, and calculate the total editing cost by combining the node editing cost and the set node replacement cost, node deletion cost, node addition cost, edge replacement cost, edge deletion cost and edge addition cost; and select an example traffic element entity relationship diagram in the scene evolution knowledge graph based on the total editing cost.

[0142] Optionally, the decision evaluation module 230 is used to: based on each evolution scenario g i The values ​​of each risk indicator under this condition are used to generate the evolution scenario g i vector; wherein the risk indicators include: time to collision, safe recovery time, distance to collision, safety and comfort; according to the probability p corresponding to the evolution scenario i With this evolution scenario g i vector, generating the behavior decision result B r The risk estimation vector risk(B r ); Based on the risk indicators, the judgment matrix A is generated by using the analytic hierarchy process and the 1-9 scaling method; the matrix is ​​obtained Among them, the vector is a one-dimensional column vector with the jth bit set to 1 and the rest bits set to 0 and length num, {c1,c2,...,c m} is the subscript set of risk indicators, m is the number of risk indicators, and num is the number of risk indicators; the judgment matrix A is transformed using the matrix B to obtain the judgment matrix Calculate the matrix by the product method The transformation matrix A tran After that, according to the transformation matrix A tran Get the weight vector According to the weight vector and the risk estimation vector risk(B r ) to obtain a risk assessment result of the behavioral decision result.

[0143] Optionally, the decision execution module 240 is used to record the environment experienced by the unmanned vehicle frame by frame to obtain the traffic element entity relationship diagram of each frame during the execution of the behavior by the unmanned vehicle; by calculating the similarity between the traffic element entity relationship diagram of each frame and the example traffic element entity relationship diagram in the scene evolution knowledge graph, the scene node matching is performed; when the unmanned vehicle performs a behavior action based on the matched scene node, all the scene nodes experienced during the execution process are added to the behavior recorder set Among them, B num is the number of actions recorded and executed in the current scenario. Indicates that the current scene node passes the behavior action b i The number of scene nodes S that have arrived; setting the behavior threshold γ; when the behavior action b i Satisfy the conditions Delete all the behavior actions b of the current scene node i The outgoing edges are based on the recorder collection Scene BC All actions in b i Relevant information, recalculate the behavior evolution probability of the corresponding scene node, create the corresponding outgoing edge, and at the same time convert all the numbers Set to 0.

[0144] Optionally, the decision execution module 240 is used to set the risk estimation counter and the maximum risk estimation round τ, and reset the prompt template of the security decision model when the risk assessment result does not meet the requirements; wherein the prompt template of the reset security decision model contains the risk assessment result element; in each risk estimation round, the prompt template of the security decision model is generated in combination with the risk prediction and behavior suggestion corresponding to the risk assessment result, and after obtaining a new behavior decision result, re-execution is started to perform risk assessment on the new behavior decision result; if the current risk estimation round count is greater than the maximum risk estimation round τ, manual intervention is requested.

[0145] The exemplary device is an embodiment of the device corresponding to the above exemplary method. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0146] Exemplary Electronic Devices

[0147] Figure 3 A block diagram of an electronic device 300 is shown according to an exemplary embodiment. The electronic device 300 may be a terminal, a notebook computer, a desktop computer, a server, a computer cluster or other types of electronic devices.

[0148] Reference Figure 3, the electronic device 300 may include at least one processor 310 and a memory 320. The processor 310 may execute instructions stored in the memory 320. The processor 310 is communicatively connected to the memory 320 via a data bus. In addition to the memory 320, the processor 310 may also be communicatively connected to an input device 330, an output device 340, and a communication device 350 via a data bus.

[0149] The processor 310 may be any conventional processor, such as a central processing unit (CPU), a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0150] The memory 320 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0151] In the disclosed embodiment, executable instructions are stored in the memory 320, and the processor 310 can read the executable instructions from the memory 320 and execute the instructions to implement all or part of the steps of the autonomous driving safety decision-making method that integrates the scene evolution knowledge graph and the large language model in the above exemplary embodiment.

[0152] Exemplary computer-readable storage media

[0153] In addition to the above methods and devices, the exemplary embodiments of the present disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to implement all or part of the steps described in the above exemplary embodiments.

[0154] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0155] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of readable storage media include: static random access memory (SRAM) with one or more wires electrically connected, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0156] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for autonomous driving safety decision-making that integrates scene evolution knowledge graph and large language model, characterized in that: The method comprises: Step S1: Acquire driving data obtained during the driving process of the unmanned vehicle; Step S2: generating a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model; Step S3: Based on the driving data, an example traffic element entity relationship diagram is selected in the scene evolution knowledge graph, and combined with the number of time frames T required to execute the behavior decision result, a possible evolution scene set G and the probability p corresponding to each evolution scene are obtained in the scene evolution knowledge graph. i ; Step S4: Based on the probability p corresponding to each evolution scenario set G i The risk assessment results for determining the outcome of the behavioral decision; Step S5: When the risk assessment result meets the requirements, the unmanned vehicle executes the behavior decision result.

2. The method according to claim 1, characterized in that Generate a prompt template of a safety decision model according to the driving data to obtain a behavior decision result, including: Extracting data information from the driving data using a pre-trained perception model; wherein the driving data includes: image data, laser radar point cloud data and sensor data, and the data information includes: entity information and environmental information; Filling single-entity semantic chain information based on the entity information and the environment information to generate multiple single-entity semantic chains corresponding to each entity; Generate semantic information of each entity according to multiple single-entity semantic chains corresponding to the entity; Filter the semantic information of the entity; The filtered semantic information is filled into an input template of a security decision model so that the security decision model outputs a behavior decision result.

3. The method according to claim 2, characterized in that Screening the single entity semantic chain includes: Arrange the entities in the scene in non-descending order according to their straight-line distance from the unmanned vehicle, and retain the single-entity semantic chains corresponding to the top k entities in the sorting; and, Define the dangerous behavior of the entity and the risk factor of the dangerous behavior, and after arranging the risk factors in non-descending order, retain the single entity semantic chains corresponding to the entities in the first k positions of the sort.

4. The method according to claim 1, characterized in that An example traffic element entity relationship diagram is selected from the scene evolution knowledge graph based on the driving data, including: Based on the driving data, obtaining a traffic element entity relationship diagram V of the current scene; The entity nodes in the traffic element entity relationship graph V of the current scene are traversed, and the non-associated semantic attribute classification of the entity nodes is performed according to the non-associated semantic attribute classification rule, and the associated semantic attribute classification of the domain nodes of the entity nodes is performed according to the associated semantic attribute classification rule, and the traffic element entity relationship graph V of the current scene is generated. ′ ; Wherein, the non-associative semantic attribute category classification rule is a rule generated based on non-associative semantic attributes that will not affect other entities or relationships between entities, and the associative semantic attribute category classification rule is a rule generated based on associative semantic attributes that will affect other entities or relationships between entities; Through the traffic element entity relationship diagram V of the current scene ′ An example traffic element entity relationship diagram is selected based on its similarity with each example traffic element entity relationship diagram in the scenario evolution knowledge graph.

5. The method according to claim 4, characterized in that The step of obtaining a traffic element entity relationship diagram V of a current scene based on the driving data includes: Preprocessing the driving data using a pre-trained panoramic semantic segmentation model and a pre-trained parameter extraction model to obtain entity data and attribute data of each entity in the current scene; wherein the entity data includes: a unique identifier, a category, a pixel area, and point cloud data; Based on the attribute data, a pre-trained relationship extraction model is used to obtain relationship elements between entities, and the relationship elements are combined with the entity data and the attribute data to obtain global semantic information in the current scene; According to the global semantic information in the current scene, a traffic element entity relationship graph V of the current scene is obtained.

6. The method according to claim 4, characterized in that Through the traffic element entity relationship diagram V of the current scene ′ Based on the similarity with each example traffic element entity relationship diagram in the scene evolution knowledge graph, an example traffic element entity relationship diagram is selected, including: For the traffic element entity relationship diagram V of the current scene ′ With an example traffic element entity relationship graph, the node editing cost between nodes with the same attribute is calculated by combining the classification results of non-associated semantic attributes, the classification results of associated semantic attributes and the difference between corresponding attributes; Map the nodes in the example traffic element entity relationship graph to the traffic element entity relationship graph V of the current scene ′ , and calculate the total editing cost by combining the node editing cost and the set node replacement cost, node deletion cost, node addition cost, edge replacement cost, edge deletion cost, and edge addition cost; According to the total editing cost, an example traffic element entity relationship diagram is selected in the scene evolution knowledge graph.

7. The method according to claim 1, characterized in that Based on the probability p corresponding to the evolution scene set G and each evolution scene i The risk assessment results for determining the behavior decision results include: Based on each evolution scenario g i The values ​​of each risk indicator under this condition are used to generate the evolution scenario g i wherein the risk indicators include: time to collision, safe response time, distance to collision, safety and comfort; According to the probability p corresponding to the evolution scenario i With this evolution scenario g i vector, generating the behavior decision result B r The risk estimation vector risk(B r ); Based on the risk indicators, a judgment matrix A is generated by using the analytic hierarchy process and the 1-9 scaling method; Get Matrix Among them, the vector is a one-dimensional column vector with the jth bit set to 1 and the rest bits set to 0 and length num, {c1,c2,...,c m } is the subscript set of risk indicators, m is the number of risk indicators, and num is the number of risk indicators; The judgment matrix A is transformed by using the matrix B to obtain the judgment matrix Calculate the matrix by the product method The transformation matrix A tran After that, according to the transformation matrix A tran Get the weight vector According to the weight vector and the risk estimation vector risk(B r ) to obtain a risk assessment result of the behavioral decision result.

8. The method according to any one of claims 1 to 7, characterized in that: When the risk assessment result meets the requirements, after the unmanned vehicle executes the behavior decision result, the method further includes: Recording the environment experienced by the unmanned vehicle frame by frame to obtain a traffic element entity relationship diagram of each frame during the execution of the behavior by the unmanned vehicle; The scene nodes are matched by calculating the similarity between the entity relationship graph of traffic elements in each frame and the example entity relationship graph of traffic elements in the scene evolution knowledge graph; When the unmanned vehicle performs a behavior action based on the matched scene node, all scene nodes experienced during the execution process are added to the behavior recorder set Among them, B num is the number of actions recorded and executed in the current scenario. Indicates that the current scene node passes the behavior action b i The number of scene nodes S that have arrived; Set the behavior threshold γ; When the action i Satisfy the conditions Delete all the behavior actions b of the current scene node i The outgoing edges are based on the recorder collection Scene BC All actions in b i Relevant information, recalculate the behavior evolution probability of the corresponding scene node, create the corresponding outgoing edge, and at the same time convert all the numbers Set to 0.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When the risk assessment result does not meet the requirements, a risk estimation counter and a maximum risk estimation round τ are set, and a prompt template of the security decision model is reset; wherein the prompt template of the reset security decision model includes a risk assessment result element; In each round of risk estimation, a prompt template of the safety decision model is generated by combining the risk prediction and behavior suggestion corresponding to the risk assessment result, so as to obtain a new behavior decision result, and then re-execute from step S3 to perform risk assessment on the new behavior decision result; If the current risk estimation round count is greater than the risk estimation maximum round τ, manual intervention is requested.

10. An autonomous driving safety decision-making system integrating scene evolution knowledge graph and large language model, characterized in that: The system comprises: A data acquisition module is used to obtain driving data obtained during the driving process of the unmanned vehicle; A behavior decision module, used to generate a prompt template of a safety decision model according to the driving data to obtain a behavior decision result; wherein the safety decision model is constructed based on a large language model; A decision evaluation module is used to select an example traffic element entity relationship diagram in the scene evolution knowledge graph based on the driving data, and obtain a possible evolution scene set G and a probability p corresponding to each evolution scene in the scene evolution knowledge graph in combination with the number of time frames T required to execute the behavior decision result. i ; Based on the probability p corresponding to the evolution scenario set G and each evolution scenario i The risk assessment results for determining the outcome of the behavioral decision; The decision execution module is used to enable the unmanned vehicle to execute the behavior decision result when the risk assessment result meets the requirements.

Images