An unmanned driving test scene automatic generation method based on reinforcement learning

Abstract

The present application belongs to the technical field of unmanned driving test, and particularly relates to an unmanned driving test scene automatic generation method based on reinforcement learning. In the present application, dynamic elements in the scene are regarded as agents, and the model is continuously trained in the simulation process through reinforcement learning, so that the agents learn to interact with the ego vehicle in the driving process, thereby enabling the unmanned vehicle to face more complex scenes. Because there are multiple pedestrians and multiple vehicles in the scene, a centralized training and decentralized execution architecture and an actor-critic idea are used to make the agents cooperate with each other. In the training stage, each dynamic element needs to inform the central processor module of its observation value and the action made at present, and through the information, the central processor module determines the action to be made by each agent in the next step; in the execution stage, the dynamic element does not need to provide its observation value and action to the central controller any more, but makes a decision by itself, thereby improving the test efficiency.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving testing technology, and in particular to a method for automatically generating autonomous driving test scenarios based on reinforcement learning. Background Technology

[0002] Autonomous driving is a current international hot topic and a crucial support for economic transformation and development, attracting numerous automotive companies to invest in this field. During the development of autonomous driving systems, verifying their safety, particularly their ability to handle unexpected situations, presents a significant challenge.

[0003] To ensure the safety of autonomous driving, much research focuses on generating numerous complex and diverse driving scenarios for safety testing, aiming to identify potential flaws in autonomous driving systems early on. However, given the infinite number of scenarios autonomous driving can encounter, the cost of generating these scenarios in the real world is prohibitive. Therefore, current autonomous driving testing generally employs virtual simulation technology. Some current patents, such as the "Random Automatic Generation Method and System for Autonomous Driving Simulation Test Scenarios" (Patent No.: CN202211301843.6), "A Method and System for Generating Autonomous Driving Simulation Test Scenarios" (Patent No.: CN201910797521.7), and "A Method for Generating Simulation Test Scenarios for Autonomous Driving Systems" (Patent No.: CN202210644626.0), primarily process vehicle data from natural driving, using random sampling or heuristic methods to generate new test scenarios. However, these methods generate a large number of repetitive and useless test scenarios. There are also some invention patents, such as a method for generating key autonomous driving scenarios based on reinforcement learning (patent number: CN202110082493.8) and a method for testing autonomous vehicle collisions based on reinforcement learning (patent number: CN202110799952.4), which mainly generate test scenarios by adjusting environmental parameters, but the autonomous vehicle lacks interaction with the environment during operation.

[0004] Based on the above overview of existing technologies, it can be seen that the technology for generating test scenarios for autonomous driving suffers from low testing efficiency. Therefore, improving the efficiency of autonomous driving test scenario generation is both urgent and necessary. Summary of the Invention

[0005] To overcome the shortcomings of existing autonomous driving test scenario generation methods in terms of diversity and efficiency, this invention provides an automatic autonomous driving scenario generation method based on reinforcement learning. Reinforcement learning is an important branch of artificial intelligence learning, used to control intelligent agents capable of autonomous action in a given environment, continuously improving their behavior through interaction with the environment.

[0006] In this invention, the self-driving vehicle refers to a vehicle controlled by an autonomous driving system in a test scenario, whose trajectory and behavior patterns are controlled by the decision-making module within the autonomous driving system. Dynamic scene elements include pedestrians and other vehicles, which can interfere with the self-driving vehicle's operation.

[0007] This invention treats dynamic elements in a scene as intelligent agents. Through reinforcement learning, the model is continuously trained during simulation, enabling these agents to learn how to interact with their own vehicle during driving. This allows autonomous vehicles to handle more complex scenarios, especially those prone to making incorrect decisions. Because the scene contains multiple pedestrians and vehicles, a centralized training and decentralized execution architecture is adopted to facilitate cooperation among the agents. During the training phase, each dynamic element needs to report its observations and current actions to the central processing unit (CPU). This CPU uses this information to determine the next action for each agent. During the execution phase, the dynamic elements no longer need to provide their observations and actions to the CPU; instead, they make decisions independently, thus improving testing efficiency.

[0008] The technical solution of the present invention is as follows:

[0009] A method for automatically generating autonomous driving test scenarios based on reinforcement learning includes the following steps:

[0010] Step 100: Start the simulation simulator and initialize the world environment in the simulator, including terrain, weather, and visibility.

[0011] Step 200: Randomly generate the location of your own vehicle on the simulation simulator, and randomly generate other vehicles and pedestrians around your own vehicle. The positions and orientations of these vehicles and pedestrians are random. If vehicles overlap, they are regenerated until they no longer conflict with each other.

[0012] Step 300, iterative training process, with the number of iterations i∈[1,N].

[0013] When multiple agents interact with the environment simultaneously, the entire system becomes a multi-agent system. In this case, each agent still follows the goal of reinforcement learning, which is to maximize the obtainable reward. However, changes in the global state of the environment are now related to the joint actions of all agents. Therefore, the impact of joint actions needs to be considered during agent policy learning. This invention uses a centralized training and decentralized execution framework to effectively coordinate the actions of multiple agents. Its main idea is to utilize a central processing unit module to learn the relationship between the global and local states, thereby maximizing the overall reward for all agents. To obtain the maximum reward, the agent needs to select each action according to a certain policy. This policy is continuously updated during training, and depending on the update method, it can be divided into value-based and policy-based methods. This invention adopts the idea of ​​an actor-critic framework, which can select appropriate actions from continuous actions like a policy-based method, and update network parameters based on each action taken by the agent, like a value-based method. In this framework, the actor refers to the agent, which uses a policy-based approach to determine actions based on the environment state. The critic, on the other hand, uses a value-based approach to score the action, and the agent updates its policy based on this score. The Centralized Training with Decentralized Execution (CTDE) policy framework uses a central processing unit module to train the Critic network for all agents, and each agent trains its own Actor network. The environment receives the state as input and the Actor network generates actions. These actions are executed by the environment, simultaneously calculating the reward and the next state. The Critic network uses the current state and the next state to predict the reward and calculates the Temporal Difference (TD) error. The TD error measures the difference between the expected reward and the actual reward for taking an action in a given state. The formula for calculating the TD error δ is as follows:

[0014] δ=r+γV(s′)-V(s)

[0015] Where V(s) is the estimate of the current state, V(s′) is the estimate of the next state, r represents the current reward, and γ represents the discount factor. The Actor and Critic networks then update their network parameters based on the TD error value.

[0016] The parameter update formula for the Critic network is as follows:

[0017]

[0018] Where, θ c α represents the parameters of the Critic network. c It is the learning rate of the Critic network. Let s represent the gradient. t This indicates the state after each action is performed.

[0019] The parameter update formula for the Actor network is as follows:

[0020]

[0021] Where, θ a π(a) represents the parameters of the Actor network. t |s t α is the action probability distribution output by the Actor network. a It is the learning rate of the Actor network.

[0022] The specific training process is as follows:

[0023] Step 301: Run the simulation simulator, give the vehicle a random target location, and treat each pedestrian and other vehicle as an intelligent agent;

[0024] Step 302: Concatenate all actions performed by the agents in the current frame and store them in array a, a = [a_{n+1} - 1}. 1 ,a 2 ,…,a m Where m represents the total number of agents in the environment, a i This represents the action of the i-th agent in the current frame;

[0025] Step 303: Concatenate the observations of all agents in the current frame and store them in array o, o = [o 1 ,o 2 ,…,o m ]; where m represents the total number of agents in the environment, o i This represents the observation value of the i-th agent in the current frame;

[0026] Step 304: Transmit the observations and actions of all agents to the central controller module. The central controller module knows the actions performed by all agents in the current frame, the observation information, and the rewards obtained. Each agent receives a reward based on its state and actions, and each agent's goal is to maximize its total expected reward, calculated using the following formula:

[0027]

[0028] Among them, R i It represents the total expected return for each agent, γ is the discount factor, t is time, T is the time range, and r is the total expected return for each agent. i t It is the reward for each intelligent agent at each step.

[0029] Step 305: The Critic structure in the central processing unit receives the environmental state and actions transmitted by each agent, which includes the value network q(o, a; w) of all agents. i ), w i The values ​​represent the network's weights, 'o' represents the observation, and 'a' represents the action. The value network can score each action of the agent; the value network utilizes the action-value function Q. π (s,a)=E[R|S t =s,a t =a] to generate policy π, and learn the action-value function Q corresponding to the optimal policy by minimizing the loss. * .

[0030] Step 306: The agent trains its own decision network π(a) based on the value network data transmitted back from the central controller module. i |o i ;θ i ), where θ i This represents the weight parameters of the decision network; the decision network can determine its next action based on the scores from the central controller module. The decision network adjusts the policy parameters θ directly by moving along... Use direction to maximize your goals.

[0031] Step 307: At the end of one cycle, the simulation is scored based on the performance of the autonomous vehicle, other vehicles, and pedestrians. If other vehicles and pedestrians collide with each other, the reward is -1; if other vehicles collide with the autonomous vehicle, the reward is +10000. The program also obtains corresponding scores as rewards by detecting the driving quality of the vehicle during the operation. The driving quality includes the number of times the autonomous vehicle brakes suddenly, accelerates suddenly, and turns sharply during the operation.

[0032] Repeat steps 301 to 307 until the specified number of iterations is reached;

[0033] In step 400, the central controller module will not function during execution; the agent will make decisions and determine the actions at each step.

[0034] The beneficial effects of the present invention are as follows: The method of the present invention can automate the testing of autonomous driving by generating effective and complex driving test scenarios, and can efficiently detect the wrong decisions made by autonomous vehicles when facing complex road conditions, so that developers can quickly fix the defects. Attached Figure Description

[0035] Figure 1 This is a flowchart of the autonomous driving scene generation method based on reinforcement learning of the present invention;

[0036] Figure 2 This describes how a simulation simulator works;

[0037] Figures 3(a) to 3(d) These are the four road categories in the simulator;

[0038] Figure 4 This is a flowchart of the CTDE framework;

[0039] Figure 5 This is a flowchart of the Actor-Critic network's workflow;

[0040] Figure 6 This is a flowchart of the multi-agent reinforcement learning process;

[0041] Figure 7 It is a vehicle collision category. Detailed Implementation

[0042] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0043] Figure 1 This is a flowchart illustrating the workflow of the reinforcement learning-based autonomous driving scene generation method provided by this invention. First, the simulator is started. The simulator is used to mimic the physical laws of real life and can be used for low-cost testing of autonomous driving. This invention uses Carla simulation software, an open-source autonomous driving simulator that supports the generation and control of various standard sensors, environments, and dynamic and static scene elements. Figure 2 The diagram illustrates how Carla works. As shown, the simulator consists of a server and a client. The server is primarily responsible for simulating the real world, while the client controls the operation of the simulated world. Users can send commands to the server by writing Python scripts, and the server will execute these commands. The simulator provides rich and complex road environments, such as... Figures 3(a) to 3(d) What is shown are four common types of roads: straight roads, crossroads, three-way intersections, and corners.

[0044] The training process includes the following steps:

[0045] Step 301: Run the simulation simulator, give the vehicle a random target location, and treat each pedestrian and other vehicle as an intelligent agent;

[0046] Step 302: Concatenate all actions performed by the agents in the current frame and store them in array a, a = [a_{n+1} - 1}. 1 ,a 2 ,…,a m The agent is given five actions: moving forward, moving backward, turning left, turning right, and braking.

[0047] Step 303: Concatenate the observations of all agents in the current frame and store them in array o, o = [o 1 ,o 2 ,…,o m The information that each intelligent agent can observe comes from the vehicle's onboard sensors, including LiDAR, Inertial Measurement Unit (IMU), and Obstacle Detector, which can be used to detect the vehicle's speed, steering angle, and information about surrounding obstacles.

[0048] In steps 304 to 307, the CTDE framework was used, and its workflow is as follows: Figure 4 As shown in the diagram, multiple Critic network structures are trained within the central controller module. These Critic networks employ a value network structure, accepting observations and actions from each agent, scoring each action, and returning the score to the agent. They utilize the actor-critic framework, with the structure shown below. Figure 5 As shown, it can select appropriate actions from a series of actions, similar to policy-based methods, or update network parameters based on each action performed by the agent, similar to value-based methods. The agent updates its Actor network structure according to the obtained scores. These Actor networks adopt a policy network structure and can update their network parameters based on the scores given by the Critic network. To enable cooperation among multiple agents, the reinforcement learning algorithm framework used in this invention is as follows: Figure 6As shown. First, the number of agents is defined as N, which is 8 in this example. Each agent contains an Actor network structure, which includes a target policy network and an Actor network. The central processing unit contains N Critic network structures, each containing a target value network and a Critic network. Simultaneously, an experience pool D is initialized. The experience pool is mainly used to address the sample correlation and sample imbalance problems in deep reinforcement learning.

[0049] Step 304: Transmit the observations and actions of all agents to the central controller module. The central controller module knows the actions performed by all agents in the current frame, the observation information, and the rewards obtained.

[0050] In step 305, the Critic structure in the central processing unit receives the environment state and actions transmitted by each agent. The Critic network takes the current environment state and each agent's actions as input and outputs a value Q to evaluate the agent's actions. For each time step t, each agent i bases its actions on the current policy μ. i Select Action And receive rewards for interacting with the environment. and the next state s t+1 , will experience (s t ,a t ,r t ,s t+1 The experience is stored in the experience pool D. For each agent i, a batch of experience is randomly sampled from the experience pool D, and its target value is calculated using the target value network. The calculation formula is as follows:

[0051]

[0052] Where γ is the discount factor. and These represent the target value network and the target policy network, respectively. These are the parameters of the target network. Using the target network can enhance the stability and effectiveness of learning.

[0053] Considering the strategy is θ = {θ1,…,θ} n A parameterized representation of a game with N agents is given by setting the set of all agent policies as μ = {μ1, ..., μ}. n}. Based on the experience gained through learning, the loss function L(θ) is calculated. i To update the critic network parameters, the loss function is calculated using the following formula:

[0054]

[0055] Where y represents the total reward obtained, E represents the expected value, s is the current state, a is the currently chosen action, r is the current reward, and s′ is the next state. It is a centralized action-value function that takes the actions a1, ..., a2 of all agents in the environment. n The system takes the state information s as its input and then outputs the Q value of agent i.

[0056] Then calculate the gradient of the value function. The parameters of the value network are then updated using these gradient values. This process is repeated until the network converges.

[0057] Step 306: The agent trains its own decision network π(a) based on the value network data transmitted back from the central controller module. i |o i ;θ i ), where θ i This represents the weight parameters of the decision network; the decision network can determine its next action based on the scores from the central controller module. The decision network adjusts the policy parameters θ directly by moving along... Use direction to maximize your goals.

[0058] Step 307: At the end of one cycle, the simulation is scored based on the performance of the driver's own vehicle, other vehicles, and pedestrians. The simulator client's application programming interface (API) can be used to check if a collision occurred, and between which two vehicles. It can also obtain the number of times the driver's own vehicle braked suddenly, accelerated sharply, and turned sharply, as well as its distances relative to other vehicles. Based on this data, the driving quality score can be calculated. Figure 7 As shown, collisions between the vehicle and other vehicles are mainly divided into five categories: rear-ending another vehicle, being rear-ended by another vehicle, being collided with by a turning vehicle while traveling straight, colliding with a straight-traveling vehicle while turning, and colliding with a turning vehicle while turning. The driving quality score mainly includes the number of times the vehicle experienced sudden braking, sudden acceleration, and sharp turns. For sudden braking and sudden acceleration, the detector k... a In other words, it can be obtained by calculating the following formula:

[0059]

[0060] Where A x This is the vehicle's longitudinal acceleration ratio, and g is the gravitational constant. If k a If the value exceeds a threshold, it is considered a sudden acceleration or braking. A sharp turn occurs when the autonomous vehicle attempts to steer at a relatively high speed. Because sharp turns are related to the lateral forces applied to the vehicle, the sharp turn detector k... t The calculation formula is as follows:

[0061]

[0062] Where V y and SWA represent lateral velocity and steering wheel angle, respectively. If k t If the score exceeds the threshold, it is considered that a sharp turn has occurred. It can be argued that a higher driving quality score creates a more dangerous driving environment for one's own vehicle.

[0063] This invention automates the above process, enabling the automatic generation of autonomous driving test scenarios. Using reinforcement learning allows for the generation of more complex and diverse driving scenarios, thus improving the efficiency of autonomous driving test scenario generation.

Claims

1. A method for automatically generating autonomous driving test scenarios based on reinforcement learning, characterized in that, Includes the following steps: Step 100: Start the simulation simulator and initialize the world environment in the simulator, including terrain, weather, and visibility; Step 200: Randomly generate the location of your own vehicle on the simulation simulator, and randomly generate other vehicles and pedestrians around your own vehicle; the positions and orientations of these vehicles and pedestrians are random, and if vehicles overlap, they are regenerated until the vehicles no longer conflict with each other. Step 300, iterative training process, with the number of iterations i∈[1,N]; The specific training process is as follows: Step 301: Run the simulation simulator, give the vehicle a random target location, and treat each pedestrian and other vehicle as an intelligent agent; Step 302: Concatenate all actions performed by the agents in the current frame and store them in array a, a = [a_{n+1} - 1}. 1 ,a 2 ,…,a m ]; where m represents the total number of agents in the environment, a i This represents the action of the i-th agent in the current frame; Step 303: Concatenate the observations of all agents in the current frame and store them in array o, o = [o 1 ,o 2 ,…,o m ]; where m represents the total number of agents in the environment, o i This represents the observation value of the i-th agent in the current frame; Step 304: Transmit the observations and actions of all agents to the central controller module. The central controller module knows the actions performed by all agents in the current frame, the observation information, and the rewards obtained. Each agent receives a reward based on its state and actions, and each agent's goal is to maximize its total expected reward, calculated using the following formula: Among them, R i γ is the total expected return for each agent, t is the time period, and T is the time range. It is the reward for each intelligent agent at every step; Step 305: The Critic structure in the central processing unit receives the environmental state and actions transmitted by each agent, which includes the value network q(o, a; w) of all agents. i ), w i The values ​​represent the network's weights, 'o' represents the observation, and 'a' represents the action. The value network scores each action of the agent; the value network utilizes the action-value function Q. π (s,a)=E[R|S t =s,a t =a] to generate policy π, and learn the action-value function Q corresponding to the optimal policy by minimizing the loss. * ; Step 306: The agent trains its own decision network π(a) based on the value network data transmitted back from the central controller module. i |o i ;θ i ), where θ i The weight parameters represent the decision network's parameters; the decision network determines its next action based on the scores from the central controller module; the decision network adjusts the policy parameters θ directly by moving along... Direction to maximize the goal; Step 307: At the end of one cycle, the simulation is scored based on the performance of the autonomous vehicle, other vehicles, and pedestrians. If other vehicles and pedestrians collide with each other, the reward is -1; if other vehicles collide with the autonomous vehicle, the reward is +10000. The program also obtains corresponding scores as rewards by detecting the driving quality of the vehicle during the operation. The driving quality includes the number of times the autonomous vehicle brakes suddenly, accelerates suddenly, and turns sharply during the operation. Repeat steps 301 to 307 until the specified number of iterations is reached; In steps 304-307, the deneutralized execution policy framework CTDE and the actor-critic framework Actor-Critic are employed. Multiple Critic network structures are trained in the central controller module. Each Critic network structure uses a value network structure, accepts observations and actions from each agent, scores each action, and returns the score to the agent. The agent updates its Actor network structure based on the obtained score. The Actor network uses a policy network structure and updates its network parameters according to the score given by the Critic network. Each agent contains one Actor network structure, which includes a target policy network and an Actor network. The central processor contains N Critic network structures, each containing a target value network and a Critic network. Simultaneously, an experience pool D is initialized to address the sample correlation and imbalance problems in deep reinforcement learning. In step 400, the central controller module will not function during execution; the agent will make decisions and determine the actions at each step.

2. The method for automatically generating autonomous driving test scenarios based on reinforcement learning according to claim 1, characterized in that, In step 300, the centralized training and decentralized execution framework CTDE is used to coordinate the actions of multiple agents. Specifically, a central processing unit (CPU) module is used to learn the relationship between global and local states to maximize the overall reward for all agents. To obtain the maximum reward, the agent needs to select each action according to a certain policy. This policy is continuously updated during training and can be divided into two methods based on the update method: value-based and policy-based. The Actor-Critic framework is adopted, where the policy-based method selects appropriate actions from consecutive actions, or the value-based method updates the network parameters based on each action taken by the agent. Here, "Actor" refers to the agent, and the network parameters are updated based on each action taken by the agent. The policy-based approach determines actions based on the environment state. A Critic uses a value-based method to score these actions, and the agent updates its policy accordingly. The deneutralized execution policy framework uses a central processing unit (CPU) module to train the Critic network for all agents. Each agent trains its own Actor network. The environment receives the state as input, and the Actor network generates actions that are executed by the environment, simultaneously calculating the reward and the next state. The Critic network uses the current and next states to predict the reward and calculates the Time Difference (TD) error. The TD error measures the difference between the expected and actual reward for taking an action in a given state. The TD error δ is calculated using the following formula: δ=r+γV(s′)-V(s) Where V(s) is the estimate of the current state, V(s′) is the estimate of the next state, r represents the current reward, and γ represents the discount factor; then the Actor network and Critic network will update the network parameters according to the value of the TD error; The parameter update formula for the Critic network is as follows: Where, θ c α represents the parameters of the Critic network. c It is the learning rate of the Critic network. Let s represent the gradient. t This indicates the state after each action is performed; The parameter update formula for the Actor network is as follows: Where, θ a π(a) represents the parameters of the Actor network. t |s t α is the action probability distribution output by the Actor network. a It is the learning rate of the Actor network.

Images