Safety reinforcement learning method for autonomous driving system based on LLM and KG constraints

Abstract

The present invention discloses a safety reinforcement learning method for an autonomous driving system based on LLM and KG constraints, including: collecting environmental information at the initial time t; generating a question sentence through a constraint reward question-answering machine constructed by a large language model and a knowledge graph Q t , and then generate real-time constraint text C t ; will constrain the text C t With status information S t Combined with the data passed to the agent, take action A t ; Get the new state at time t+1 S t+1 、Prompt questions Q t+1 and constraint text C t+1 , and use the constraint reward question answerer to calculate the action at the previous time t A t Evaluate and give rewards R ´; will constrain the text C t+1 With status information S t+1 And comprehensive rewards R + R ´Then input to the agent, and the agent takes action at time t+1 A t+1 ; Perform iterative safety learning and convergence strategy. The method of the present invention solves the problem that traditional safety reinforcement learning based on text constraints is difficult to automatically generate corresponding constraints according to the environment by using LLM and KG to generate safety constraints in the process of autonomous driving reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving decision-making methods, and relates to a safety reinforcement learning method for an autonomous driving system based on a large language model and knowledge graph constraints. Background Art

[0002] In recent years, reinforcement learning has been successful in solving complex sequential decision-making tasks and has made significant progress in the field of autonomous driving. However, traditional reinforcement learning algorithms may take overly aggressive actions in pursuit of global maximum rewards, which may lead to major accidents in the field of autonomous driving. To address this problem, many researchers have tried to introduce safety reinforcement learning in the field of autonomous driving to improve learning efficiency without sacrificing safety. Safety reinforcement learning solves decision-making problems in complex and sensitive environments through safety constraints during reinforcement learning, thereby ensuring the safety and reliability of the agent's strategy.

[0003] One of the most common methods currently used for constrained safety reinforcement learning is the Lagrangian method, but it is difficult to implement when facing systems in complex, large-scale or dynamic environments. Compared with the data experience obtained by the agent through random exploration, human expert experience has stronger security. Therefore, some scholars have proposed language text constraints. By collecting expert libraries of trajectory segments and structured language constraints, complex behaviors are mapped to constraints using language to guide reinforcement learning agents. Although this method is more flexible and efficient, the constraints set by relying on expert experience require a lot of manual intervention, and different scenarios often mean different text constraints. Rapidly changing scenarios mean that text constraints also need to change rapidly to match the scenario constraint requirements. Therefore, it is difficult to meet the constraint requirements of autonomous driving for constantly changing scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a safety reinforcement learning method for an autonomous driving system based on a large language model (LLM) and a knowledge graph (KG) constraint. By using LLM to generate safety constraints in the autonomous driving reinforcement learning process, the problem that traditional safety reinforcement learning based on text constraints is difficult to automatically generate corresponding constraints according to the environment is solved.

[0005] The technical solution adopted by the present invention is a safety reinforcement learning method for an autonomous driving system based on LLM and KG constraints, which is specifically implemented in the following steps:

[0006] Step 1: Collect the status information at the initial time t and scene image information, and generating a scene graph using the scene image information;

[0007] Step 2: Generate questions using a constrained reward question-answering machine built using a large language model and knowledge graph Q t , and then generate real-time constraint text ;

[0008] Step 3: Constrain the text With status information Combined with the data passed to the agent, take action ;

[0009] Step 4: Get the new state at time t+1 、Prompt questions and constraint text , and use the constraint reward from step 2 to reward the question answerer for its action at the previous time t Evaluate and give rewards ;

[0010] Step 5: Generate the With status information And comprehensive rewards The joint input is given to the agent, and the agent takes action at time t+1 ;

[0011] Step 6: Repeat steps 4-5 to perform iterative security learning and converge the strategy.

[0012] The present invention is also characterized in that:

[0013] Step 2 is as follows:

[0014] Taking the scene graph structure generated in step 1 as input, the prompt engineering is combined with the offline knowledge graph and the evidence path is obtained by using the large language model to find the paths to the key information nodes of the autonomous vehicle in the traffic scene, forming an online knowledge graph, and the triple information in the online knowledge graph is input as a prompt word to the large language model. Through the prompt engineering, the large language model gives real-time constraint text .

[0015] Step 3 is as follows:

[0016] By defining the action space, the large language model is used to constrain the encoder to generate the constraint text generated in step 2 Encode into a numerical matrix that can be directly calculated and combine the numerical matrix with the current state information Combined with the data transmitted to the agent, the agent decides the action according to different pre-set reinforcement learning strategies .

[0017] In step 3, based on the action space of the autonomous driving vehicle, the action is simplified by replacing it with numbers [0, 1, 2…], and the constraint function of formula (1) is used. Constraining Agent Behavior:

[0018] (1)

[0019] In formula (1), represents the action at the current time t, represents the state information at the current time t, Indicates the state information of the next time t+1, is a set of constraints formed by numerical matrices, representing safety constraints in safety reinforcement learning; when f=0, it indicates the current action It is a constrained action, and the execution action is False; when f=1, the execution action is True.

[0020] Step 4 is as follows:

[0021] Collect the environmental information at time t+1 and the actions taken by the agent at the previous time t Interact with the environment to generate new states , and repeat step 2 to obtain the prompt question at time t+1 and constraint text , and use the constraint reward question-answering machine in step 2 to evaluate the action taken by the agent at the previous time t , according to the action The result is used to generate its judgment on the action, which is formed into a numerical score by the large language model, that is, the reward .

[0022] Combined rewards in step 5 The reward generated in step 4 Rewards from the environment Comprehensive formation; among them, the reward of strengthening learning its own environment Includes: collision reward, distance reward, time step reward, speed reward and successful braking reward.

[0023] The beneficial effects of the present invention are:

[0024] (1) The method of the present invention introduces a large language model to replace manual work and uses the text semantic technology of the large language model. The large language model can provide certain constraints in the current scene based on the interrogative sentences generated by the current environment and the current specific situation, and can provide the agent with the relationship between the agent and the environment in a timely and accurate manner, thereby coping with the ever-changing scenes in the field of autonomous driving. It solves the problem that the traditional method that relies on manual generation of safety reinforcement learning constraints cannot meet the real-time safety and reliability requirements of ever-changing scenes, and can generate text constraints in real time according to the scene;

[0025] (2) The method of the present invention introduces a knowledge graph based on a large language model, that is, a constraint reward question-answering machine that combines LLM and KG is used to generate safety constraints, which overcomes the shortcomings of LLM itself in terms of factual accuracy and interpretability, improves the accuracy of the text generated by LLM, ensures that it is consistent with real-world knowledge, thereby reducing language and text errors and ensuring the safety and reliability of decision results;

[0026] (3) The method of the present invention constructs a method based on language constraints to guide safety reinforcement learning by using a large language model and knowledge graph. Autonomous driving intelligent vehicles not only rely on the constraints and reward conditions of traditional reinforcement learning, but also combine the safety constraints and rewards generated in real time by the constraint reward question answerer based on scene information to optimize decision-making. This method allows the intelligent agent to quickly adapt and make safer and more reliable decisions when facing complex and dynamically changing environments;

[0027] (4) When the constraint reward question-answering machine in the method of the present invention generates safety text constraints, each observation is accompanied by text information and corresponding reward information, which can effectively provide the interpretability of the training process. In addition, since the intervention of safety constraints restricts the action space, prohibited unsafe behaviors are eliminated before the agent selects actions, narrowing the action selection space, thereby avoiding useless and dangerous behavior trial and error, and improving learning efficiency while ensuring the safety of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 is a process block diagram of the method of the present invention;

[0030] Figure 3 It is a schematic diagram of the process of fusing a large language model with a knowledge graph to generate a constrained text in the method of the present invention;

[0031] Figure 4 is a schematic diagram of the characteristics of the simulation scene state of Example 1 of the present invention;

[0032] Figure 5is the scene graph structure of the example scene in embodiment 1 of the present invention;

[0033] Figure 6 It is an online knowledge graph of traffic participants in the example scenario of Example 1 of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The method of the present invention establishes a human-guided safety reinforcement learning training model, which is a method that constructs a constraint reward question-answering machine through a large language model and a knowledge graph, and autonomously generates safety reinforcement learning text constraints, thereby improving the decision safety, explainability and training efficiency of the autonomous driving system. The method first collects environmental information, uses a large language model with the assistance of a knowledge graph to generate environmental interrogative sentences at the current moment , and then generate text constraints , and then encode the generated constraints, and then the agent combines the state information Taking Action with Security Constraints , Evaluation of Constrained Reward Question Answerers Based on Large Language Models and Knowledge Graphs , give rewards ; During the training process, the state information , Text constraints , and the rewards of reinforcement learning itself and The final reward is superimposed and fed back to the intelligent agent, and this method is used for safe reinforcement learning.

[0036] The present invention is based on the LLM and KG constraints of the autonomous driving system safety reinforcement learning method, such as Figure 1 and Figure 2 As shown, the specific implementation steps are as follows:

[0037] Step 1: Collect environmental information at the initial time t:

[0038] Collect scene status information such as speed, location, and heading of traffic participants in the scene ,At the same time, the on-board camera, sensors, etc. are used to obtain the image information of the current environment, and the scene graph generation algorithm is used to generate the corresponding scene graph.

[0039] Step 2: Combine the triples contained in the scene graph information with the previously constructed offline knowledge graph to construct an online knowledge graph, and use the online knowledge graph to prompt the large language model to ask questions and generate question sentences. Q t , and then generate real-time constraint text through the constraint reward question answerer built by the large language model and knowledge graph , specify certain behaviors that the intelligent vehicle is prohibited from performing.

[0040] The above constraint reward question answerer is formed by integrating a large language model with a knowledge graph, such as Figure 3 As shown in the figure, the basic reasoning process is as follows: take the scene graph structure generated in step 1 as input, use the big language model to obtain the evidence path through the prompt engineering combined with the offline knowledge graph, find the paths to the key information nodes of the autonomous vehicle in the traffic scene, form an online knowledge graph, and input the triple information in the online knowledge graph as the prompt word to the big language model. Through the prompt engineering, the big language model gives corresponding constraints for different scenes. , thereby realizing the function of the constrained reward question answerer.

[0041] The prompt question As input to the large language model interface, such as ChatGPT, combined with the knowledge graph to obtain the constraint text of this environment at the current time t , by combining knowledge graphs, the accuracy of text generation by large language models and the interpretability of the reasoning process are improved.

[0042] Step 3: Define the action space and use the large language model to constrain the encoder to generate the constraint text in step 2. Encode into a numerical matrix that can be directly calculated and combine the numerical matrix with the current state information Combined with the data, it is sent to the agent, which decides the action based on different pre-set reinforcement learning strategies. Specifically:

[0043] Based on the action space of the autonomous vehicle, the action is simplified by replacing the action with numbers [0, 1, 2…] and the constraint text is converted into Encoded into a numerical matrix that the agent can directly calculate and stored in the constraint set , represents the safety constraint in safety reinforcement learning, through the constraint function of formula (1) Constraining agent behavior.

[0044] (1)

[0045] when =0, indicating the current action It is a constrained action, and the execution action is False; when = 1, the action is True. Therefore, for each given state ,The attributes of the constrained actions are local, and this method can effectively avoid dangerous actions during autonomous driving and play an effective safety protection role.

[0046] Next, combine the numerical matrix with the current state information Combined with the transmission to the intelligent agent, the TD3 reinforcement learning algorithm is selected, and the strategy network logs the value matrix and state information Fusion to create action .

[0047] Step 4: Collect the environmental information at time t+1 and calculate the actions taken by the agent at the previous time t Interact with the environment to generate new states , and repeat step 2 to obtain the prompt question at time t+1 and constraint text , and use the constraint reward question-answering machine in step 2 to evaluate the action taken by the agent at the previous time t , according to the action The result is used to generate its judgment on the action, which is formed into a numerical score by the large language model, that is, the reward .

[0048] Step 5: Generate the With status information And comprehensive rewards The joint input is given to the agent, and the agent takes action at time t+1 .

[0049] Among them, the comprehensive reward is the reward generated in step 4 Rewards from the environment Comprehensively form the total reward feedback of the agent at time t+1 .

[0050] The reward function in the method of the present invention consists of two parts. The first part is the traditional reinforcement learning reward , including: collision reward , Distance Reward , time step reward , Speed ​​Reward and rewards for successful braking , the total reward function is shown in formula (2):

[0051] (2)

[0052] The second part is based on the LLM of the constrained reward question answerer as a behavior evaluator, which evaluates each action of the agent. Conduct an assessment and obtain the assessment reward ,For example: = (1, the behavior result is good, -1, the behavior result is bad, 0, other). The reward will be evaluated Compared with traditional reinforcement learning rewards Combined, the reward of the agent at time t is formed: , so the security reinforcement learning in the method of the present invention is composed of the tuple < , , , , , , , >Define, where is the state set, It's an action set. is the discount factor, Given the current state and actions The next state The conditional probability ( | , ), It is a collection of text constraint specifications. The intelligent vehicle selects actions based on safety constraints, comprehensive rewards, and observation status.

[0053] Step 6: Iterative security learning and strategy convergence:

[0054] Use the TD3 algorithm as the reinforcement learning algorithm for safety reinforcement training, repeat the above steps 4-5 for safety reinforcement learning, constantly interact with the environment, collect information, make decisions, obtain feedback, and update its strategy; after multiple iterations, the strategy of the intelligent agent gradually stabilizes and converges to an optimal strategy or a near-optimal strategy.

[0055] Embodiment 1:

[0056] Reference Figure 1 According to the above process of the present invention, the following steps are specifically implemented:

[0057] Taking the traffic scene on the right side of the urban road as an example, the characteristics of its state scene graph are as follows: Figure 4 As shown, the vehicle's speed, position, heading and other scene status information are collected ; The image information of the current environment is obtained through the vehicle-mounted camera, sensor, etc., and the traffic scene graph is extracted from the image using the Roadscene2vec algorithm to obtain the scene graph structure. The scene graph includes the position relationship between the autonomous vehicle and the blocking vehicle and the traffic relationship with other traffic participants. The scene graph structure obtained in this embodiment is as follows Figure 5 shown.

[0058] The triple entities contained in the scene graph are introduced into the large language model, such as ChatGPT. The relevant evidence paths are searched in the offline knowledge graph through the prompt engineering, and the paths related to the autonomous vehicle in the traffic scene are obtained, as shown in Table 1. Taking a certain time node as an example, a set of head entities (time and traffic participants) are input to the LLM, and the evidence paths are obtained through the prompt engineering. Finally, all the reasoning paths are fused to form an online knowledge graph, and the online knowledge graph under the occlusion scene on the right is obtained (such as Figure 6 As shown in Table 2), the triple information in the online knowledge graph is used as the prompt word, and the constraint reward question answerer of the online knowledge graph is integrated with the large language model to give the prohibited actions and action evaluations in the current environment through prompt engineering questions, as shown in Table 2.

[0059] Table 1 Evidence path acquisition

[0060]

[0061] Table 2 Example of KG+LLM Constrained Reward Question Answerer

[0062]

[0063] The state and action space of safety reinforcement learning are defined as shown in Table 3 and Table 4, where Table 3 is the state space. Defining the state space based on the example scenario is conducive to the agent making appropriate action choices based on the environment. Table 4 is the action space. In autonomous driving, the action space includes steering, acceleration, braking, etc. The agent needs to choose the appropriate action according to the current state to drive safely and efficiently. In the present invention, the state space is prohibited from being the same as the action space, and based on the action space of the autonomous driving vehicle, the action is simplified by replacing the action with numbers [0, 1, 2...], which also enables the large language model to better understand the action, thereby making a more accurate evaluation; by using the constraint encoder to encode the constraint text "no acceleration" into a numerical matrix that the agent can directly calculate, and store it in the constraint set , then the constraint function of formula (1) is If the value is 0, the "acceleration" action will not be performed, thus constraining the behavior of the intelligent vehicle.

[0064] Table 3 State space settings

[0065]

[0066] Table 4 Action space settings

[0067]

[0068] Define the reward function, based on the scenario and constraint reward question answerer. The reward function in the method of the present invention consists of two parts. The first part is the traditional reinforcement learning reward , including: collision reward, distance reward, time step reward, speed reward and successful braking reward. Its reward function is shown in formula (2), where the formulas for each part of the reward are as follows:

[0069] (1) Collision Reward:

[0070] (3)

[0071] In the above formula:

[0072] ——Collision penalty coefficient;

[0073] ——Collision indication function, returns 1 if a collision occurs, otherwise returns 0;

[0074] (2) Distance Reward:

[0075] (4)

[0076] In the above formula:

[0077] ——The distance between the autonomous vehicle and the vehicle whose field of vision is blocked;

[0078] ——The hidden state characteristics of the scene;

[0079] ——Reward coefficients for implicit state characteristics in different scenarios;

[0080] (3) Time step reward:

[0081] (5)

[0082] In the above formula:

[0083] ——Reward coefficients for implicit state characteristics in different scenarios;

[0084] (4) Speed ​​Reward:

[0085] (6)

[0086] In the above formula:

[0087] ——autonomous vehicle speed;

[0088] ——Target speed under different scenarios’ hidden state characteristics;

[0089] ——Reward coefficients for implicit state characteristics in different scenarios;

[0090] (5) Braking reward:

[0091] (7)

[0092] In the above formula:

[0093] - The position of the vehicle that blocks the view;

[0094] ——The length of the vehicle that blocks the view;

[0095] ——Speed ​​of the vehicle that blocks the view;

[0096] ——reward coefficient;

[0097] The second part is to evaluate actions based on the constraint reward question answerer , for each action of the agent For evaluation, the large language model will generate its evaluation reward for the action based on the result of the action , the specific formula is shown in formula (8):

[0098] (8)

[0099] In the above formula:

[0100] — Rewards that constrain the answerer to produce good evaluations of behavior;

[0101] — Constrain the question answerer to reward behaviors that produce bad evaluations.

[0102] The large language model evaluation reward Compared with traditional reinforcement learning Comprehensively, the total reward feedback of the agent is formed. Formally, the reward of the agent at time t is the evaluation reward of LLM and traditional rewards Combination of: .

[0103] Based on the above principles, the tuple < , , , , , , , >Define and update the TD3 algorithm, and set the algorithm parameters in Table 5. These parameters jointly affect the performance and convergence speed of the TD3 algorithm in training. Use new action functions and reward functions, repeat the above steps for safety reinforcement learning, and the intelligent vehicle selects actions based on safety constraints, comprehensive rewards, and observation states. Continuously interact with the environment, collect environmental information, make decisions, obtain feedback based on rewards, and update its policy network; after multiple iterations, when the reward value obtained is the maximum, the agent's strategy converges to an optimal strategy or a near-optimal strategy.

[0104] Table 5 TD3 algorithm parameter settings

[0105]

[0106] Embodiment 2:

[0107] The safety reinforcement learning method for autonomous driving system based on LLM and KG constraints is implemented in the following steps:

[0108] Step 1: Collect the status information at the initial time t and scene image information, and generating a scene graph using the collected scene image information;

[0109] Step 2: Generate questions using a constrained reward question-answering machine built using a large language model and knowledge graph Q t , and then generate real-time constraint text ;

[0110] Step 3: Constrain the text With status information Combined with the data passed to the agent, take action ;

[0111] Step 4: Get the new state at time t+1 、Prompt questions and constraint text , and use the constraint reward from step 2 to reward the question answerer for its action at the previous time t Evaluate and give rewards ;

[0112] Step 5: Generate the With status information And comprehensive rewards The joint input is given to the agent, and the agent takes action at time t+1 ;

[0113] Step 6: Repeat steps 4-5 to perform iterative security learning and converge the strategy.

[0114] Embodiment 3:

[0115] Based on Example 2, step 2 is specifically as follows:

[0116] Taking the scene graph structure generated in step 1 as input, the prompt engineering is combined with the offline knowledge graph and the evidence path is obtained by using the large language model to find the paths to the key information nodes of the autonomous vehicle in the traffic scene, forming an online knowledge graph, and the triple information in the online knowledge graph is input as a prompt word to the large language model. Through the prompt engineering, the large language model gives real-time constraint text .

Claims

1. A safety reinforcement learning method for autonomous driving system based on LLM and KG constraints, characterized by: Follow the steps below to implement it: Step 1: Collect the status information at the initial time t and scene image information, and generating a scene graph using the scene image information; Step 2: Generate questions using a constrained reward question-answering machine built using a large language model and knowledge graph Q t , and then generate real-time constraint text ; Step 3: Constrain the text With status information Combined with the data passed to the agent, take action ; Step 4: Get the new state at time t+1 、Prompt questions and constraint text , and use the constraint reward from step 2 to reward the answerer for the action at the previous time t Evaluate and give rewards ; Step 5: Create the With status information And comprehensive rewards The joint input is given to the agent, and the agent takes action at time t+1 ; Step 6: Repeat steps 4-5 to perform iterative security learning and converge the strategy; Step 2 is as follows: Taking the scene graph structure generated in step 1 as input, the prompt engineering is combined with the offline knowledge graph and the evidence path is obtained by using the large language model to find the paths to the key information nodes of the autonomous vehicle in the traffic scene, forming an online knowledge graph, and the triple information in the online knowledge graph is input as a prompt word to the large language model. Through the prompt engineering, the large language model gives real-time constraint text ; Step 3 is as follows: By defining the action space, the large language model is used to constrain the encoder to generate the constraint text generated in step 2 Encode into a numerical matrix that can be directly calculated and combine the numerical matrix with the current state information Combined with the data transmitted to the agent, the agent decides the action according to different pre-set reinforcement learning strategies ; In step 3, based on the action space of the autonomous driving vehicle, the action is simplified by replacing it with numbers [0, 1, 2…], and the constraint function of formula (1) is used. Constraining Agent Behavior: （1） In formula (1), represents the action at the current time t, represents the state information at the current time t, Indicates the state information of the next time t+1, is a set of constraints formed by numerical matrices, representing safety constraints in safety reinforcement learning; When f=0, it indicates the current action It is a constrained action, and the execution action is False; when f=1, the execution action is True; Step 4 is as follows: Collect the environmental information at time t+1 and the actions taken by the agent at the previous time t Interact with the environment to generate new states , and repeat step 2 to obtain the prompt question at time t+1 and constraint text , and use the constraint reward question-answering machine in step 2 to evaluate the action taken by the agent at the previous time t , according to the action The result is used to generate its judgment on the action, which is formed into a numerical score by the large language model, that is, the reward ; Combined rewards in step 5 The reward generated in step 4 Rewards from the environment Comprehensive formation; among them, the reward of strengthening learning its own environment Includes: collision reward, distance reward, time step reward, speed reward and successful braking reward.

Images