Safety-critical and aggressive test scene generation method for autonomous vehicle

Abstract

The invention discloses a safety-critical and aggressive test scene generation method for an automatic driving vehicle. The method comprises the following steps: firstly, acquiring vehicle natural distribution and a vehicle driving mode from a natural driving data set to initialize a scene; and then, selecting a background vehicle group taking the self-CAV as a center, and constructing a safety perception adjacency matrix based on TTC correlation between vehicles to represent a safety key state in the vehicle group. Secondly, extracting input graph features by using a graph reinforcement learning model, introducing a reward function system based on a safety key index, enabling an action output by the model to better influence a CAV safety key state, and then generating a safety key and aggressive scene based on the action; and finally, performing a large number of simulation experiments on different CAV control models in different traffic scenes to verify the test effect. According to the method, the performance of different CAV decision models in safety-critical and aggressive scenes can be effectively evaluated in different traffic scenes.

Description

Technical Field

[0001] The present invention relates to the intersection of autonomous driving testing and deep reinforcement learning, and is a method for generating autonomous driving vehicle test scenarios based on deep reinforcement learning. Background Art

[0002] In recent years, as autonomous driving technology has become more and more mature, autonomous vehicles (CAVs) have attracted more and more attention. However, a key factor affecting the development and deployment of autonomous vehicles is the safety testing of autonomous vehicle algorithms. If the vehicle cannot be tested in an extreme and comprehensive manner, the safety of the vehicle cannot be guaranteed. Safety testing of autonomous vehicles in real traffic environments requires not only evaluating their performance under normal driving conditions, but also examining their ability to cope with extreme or complex situations. Therefore, generating test scenarios that can simulate complex traffic conditions in the real world, especially scenarios that can challenge existing autonomous driving systems, is crucial to promoting safety research on autonomous driving technology. However, such scenarios are rare in the real world, and it is difficult to conduct tests based on data-driven methods. Therefore, it is necessary to simulate and generate vehicle test scenarios.

[0003] Most of the current work has fully considered the reality and challenges of generating scenarios, and has restored the natural safe driving state of the driver and the real traffic conditions in the scenario as much as possible. However, in addition to considering the above two factors, it is also necessary to consider unnatural dangerous driving states and dangerous traffic conditions caused by factors such as driver behavior, weather conditions, and vehicle damage. For example, if the driver suffers from road rage, the maneuvers of background vehicles (BVs) will continue to maliciously test and interfere with the CAV, including sudden acceleration, sudden braking, dangerous lane changes, etc. These behaviors are safety-critical aggressive behaviors for CAVs; because of reasons such as driver drowsiness or distraction, slippery road surface, collision or damage of vehicles, the maneuvers of BVs are also safety-critical and aggressive. In the above cases, the behavior of BVs is unnatural and dangerous driving behavior, and may affect the BVs around a BV, thereby indirectly interfering with the behavior of CAVs. For example, two BVs collided at a close distance in front of the CAV and stopped in front of the CAV. This scenario is a safety-critical scenario for CAVs, and CAVs need to make corresponding strategies (lane change or slow down) to avoid the stopped BVs in front while ensuring that they do not collide with the surrounding BVs. Therefore, it is also crucial to evaluate the performance of CAVs in the above scenario. The behavior of BVs in the above scenario is called aggressive and safety-critical maneuvers, and the scenario in which CAVs are in is called a safety-critical and aggressive scenario. Summary of the invention

[0004] In order to overcome the problem that safety-critical and aggressive scenarios are rare in reality and that autonomous vehicles cannot be subjected to extreme and comprehensive testing, the present invention provides a method for generating safety-critical and aggressive test scenarios for autonomous vehicles. The present invention interacts with CAV by controlling BV to perform safety-critical and aggressive maneuvers, thereby generating safety-critical and aggressive test scenarios.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for generating a safety-critical and aggressive test scenario for an autonomous vehicle comprises the following steps:

[0007] Step 1: First, obtain the random natural position distribution and natural driving state distribution of the background vehicle BV from the natural driving dataset, i.e., following or free following, and initialize them into the simulation environment; then, with the autonomous driving vehicle CAV as the center, select the vehicles in front and behind its lane and the vehicles in front and behind the left and right lanes within its observation range as the adversarial vehicle group that directly interacts with the CAV; model the vehicle group as a graph, and construct a safety perception adjacency matrix based on the collision time TTC correlation between vehicles to characterize the safety-critical states within the vehicle group;

[0008] Step 2: A graph reinforcement learning framework is proposed to train the model. The spatial relationship and safety perception relationship between vehicle groups are extracted through a graph convolutional network. The extracted features are then input into the reinforcement algorithm to obtain actions. A reward function system based on safety-critical indicators is introduced to adjust the actions, so that the actions output by the model can better affect the safety-critical state of CAV.

[0009] Step 3: Perform simulation based on the actions output by the model. However, before letting the selected adversarial vehicle execute the maneuver output by the model, use the TTC indicator to determine whether the current adversarial vehicle and CAV are in a safety-critical state. If so, execute the output maneuver, otherwise execute the natural driving maneuver. Through the interaction between the adversarial vehicle and CAV, we finally get the safety-critical and aggressive test scenario SCAS for testing CAV.

[0010] Further, the process of step 1 is as follows:

[0011] Step 1.1, Environment Initialization

[0012] A natural driving test scenario is represented as a series of parameters and possible variables that are predetermined by the operational design domain. The variables are represented as follows:

[0013]

[0014] where x i,jrepresents the variable of the i-th BV at the j-th time step, N represents the number of BVs of interest, T represents the total number of time steps, and X represents the feasible space of the variable;

[0015] By using the spatiotemporal independence relationship between variables to simplify the problem, assuming the Markov property, the changes in variables in the natural driving test scenario can be expressed as the natural driving maneuvers made by all BVs in the scenario for the current CAV vehicle state and its own state, as shown below:

[0016]

[0017] The states and natural driving maneuvers at time steps t=0,…,T are expressed as

[0018]

[0019] in Indicates the status of the CAV, represents the state of the i-th BV, represents the natural driving maneuver of the i-th BV. Assuming that all BVs choose maneuvers simultaneously and independently, then:

[0020]

[0021] Therefore, the goal of initialization is to simulate the natural speed and distance of the vehicle as the starting point of the simulation, that is, to initialize s0. To achieve this goal, the position of the vehicle in the initial area and the speed in the empirical speed distribution are randomly sampled from the data set, and the position and speed of the first vehicle in each lane are determined accordingly; then, the joint distribution of the distance from the front to the rear of the vehicle and the relative speed in the empirical distribution in the data set is queried to determine the initial position and speed of the downstream vehicle for each lane in turn. After the initial position and speed of each vehicle are determined, it is placed in the corresponding position of the simulation environment SUMO. After all lanes are placed according to the natural distribution of vehicles, the environment initialization is completed;

[0022] Step 1.2: Graph Construction

[0023] Taking CAV as the center, select n BVs in its observation range as adversarial vehicles AV to interact directly with CAV, 1≤n≤6≤N, and then define CAV and AVs together as the safety perception map in represents the set of vehicle nodes at time t, represents a CAV vehicle, represents the safety perception relationship between vehicles. The TTC value between vehicles is used to define the above relationship. Therefore, the safety perception relationship between vehicle i and vehicle j is defined as follows:

[0024]

[0025] where δ ttc represents the safety critical threshold value of TTC, represents the TTC value between the i-th vehicle and the j-th vehicle, so the safety perception adjacency matrix is:

[0026]

[0027] We also define the spatial perception map Represents the spatial perception adjacency matrix, which is used to represent the spatial existence relationship between vehicles;

[0028] For two graphs, the state space is represented as in in Indicates the distance between the current vehicle and the CAV, Indicates speed, and represents the lateral and longitudinal position of the vehicle, Indicates the vehicle's heading angle. Indicates the lane the vehicle is in. represents the vehicle acceleration, represents the action of the vehicle in the previous time step.

[0030] Furthermore, the process of step 2 is as follows:

[0031] Step 2.1: A graph reinforcement learning network is constructed to extract input features and output adversarial maneuvers. The network is mainly constructed by the graph convolutional network GCN and the reinforcement learning algorithm PPO. Before using GCN to extract features from the input, the state matrix S is first converted to t Input into the encoder, the state features are mapped to the sample label space, which is expressed as follows:

[0032] S FC =φ FC (S t ) (7)

[0033] Where S FC represents the node feature matrix after FC processing; φ FC represents the FC layer;

[0034] Then use the GCN module to perceive the adjacency matrix from the space and the security-aware adjacency matrix The spatial perception relationship and safety perception relationship between vehicle groups are extracted and expressed as follows:

[0035]

[0036] Among them, SP t and SA t They represent the spatial features and security perception features obtained after GCN processing respectively; and Represents the graph convolution operation on two matrices; D s and D ttc are based on and The calculated degree matrix; W s and W ttc represents the weight matrix; b s and b ttc represents bias; σ is the activation function;

[0037] Then the spatial perception feature SP t , Safety perception features SA t and state characteristics S FC Aggregation, input to the reinforcement learning module. The Actor network first receives the features and outputs a safety-critical action a t , which is expressed as follows:

[0038] F t =φ Cat (S FC , S.P. t , S.A. t ) (10)

[0039]

[0040] in, represents the Actor network, φ Cat represents the feature aggregation operation, F t represents the aggregated feature matrix, σ is the Softmax activation function, represents the RL-filter matrix, used to represent the presence of a vehicle;

[0041] After simulating the action, calculate the reward value of the regular indicator and safety key indicator bonus value Get the reward value R of the action t , then the Critic network evaluates action a t The value of V t , which is expressed as follows:

[0042]

[0043] in, represents the Critic network;

[0044] Finally, the Actor and Critic networks are updated separately by collecting data such as state, action, reward, value, etc. at multiple time steps;

[0045] Step 2.2: Calculate the value of the regular indicator reward function

[0046] Designed a regular indicator reward system R U , which is used to adjust the maneuver of the adversarial vehicle (AV) so that it constantly interacts with the CAV, as shown below:

[0047]

[0048] In order to encourage AVs to collide with CAVs, the collision reward is set to a fixed value, and appropriate penalties are imposed on collisions between other vehicles. Because it is necessary to observe the reaction of CAVs when encountering AVs or collisions between AVs and BVs, that is, when background vehicles collide with BVCs, BVCs are not filtered out during the simulation process, which will cause a chain reaction when BVC occurs, that is, BVs near the BVC will also make evasive or emergency braking actions due to the collision, but these actions may cause collisions between BVs, and eventually multiple vehicles will rear-end in a row; in addition, since the time step required for each collision is different, this will cause the overall reward to fluctuate greatly, making it difficult for the model to learn, so W c Used to limit the final collision reward, It increases nonlinearly and gradually over time, δ step is the time threshold;

[0049] In order to encourage AV to change lanes to the lane where CAV is located to interfere with the normal driving of CAV, a lane change reward R is set. L , the speed reward is The acceleration reward is The purpose of setting these two rewards is to guide AVs to maintain a speed close to that of CAV through continuous acceleration and deceleration, which corresponds to the situation in reality where vehicles accelerate and brake suddenly, thus interfering with the normal driving of CAV;

[0050] Time Reward R time The reward is a fixed value. The purpose of setting this reward is to make the overall test time as long as possible so as to test the vulnerability of CAV in dealing with aggressive behaviors. At the same time, this indicator can balance the fluctuation of the reward value when CAV leaves safely and when CAV collides with AV, so as to make the model learn stably and avoid excessive fluctuation of the strategy. The time weight W time =W c ;

[0051] Step 2.3: Calculate the value of the safety key indicator reward function

[0052] When designing the safety-critical reward system, the calculation frequency of the safety-critical indicators was taken into consideration, that is, whether the indicator will only be calculated in specific scenarios. Finally, the three indicators of TTC, DRAC, and PET were selected to cover most of the scenarios that may be encountered in the test. The TTC indicator can be calculated in any scenario, DRAC has different calculation methods in the following scenario and the lane changing scenario, and the PET indicator is only calculated in the lane changing scenario. Therefore, the calculation frequency of the three indicators should be distributed from high to low; when calculating the reward for the safety-critical indicator, only the indicator value between the i-th AV and CAV needs to be considered, and the indicator value between AVs is not considered. At the same time, those that are not within the safety-critical threshold value range will be filtered, that is, these values ​​will not be rewarded or punished. The calculation method is as follows:

[0053]

[0054] The TTC value indicates the time until a collision occurs based on the current vehicle spacing and speed difference. The shorter the time, the more unavoidable the collision. The calculation is as follows:

[0055]

[0056] in m is the number of TTC values ​​in this range;

[0057] DRAC represents the maximum deceleration required to avoid a collision. The greater the deceleration value, the less likely a collision is to be avoided. The calculation method is as follows:

[0058]

[0059] in is the number of DRAC values ​​in this range. In the lane change scenario, t A The time required for the vehicle A in the forward position to exit the conflict area, dist B represents the distance from vehicle B to the conflict area, δ drac It represents the safety critical threshold value of DRAC, and the conflict area represents the overlapping part of the future predicted trajectories of the two vehicles. For the calculation involving the conflict area, the calculation function is integrated in the SUMO simulator used, so the specific calculation method and detailed definition of the conflict area are not given here;

[0060] PET is used to calculate the time interval between one participant leaving and another participant entering a specified conflict zone. Therefore, the smaller the value of PET, the more inevitable a collision between two vehicles is, and therefore The calculation method is as follows:

[0061] Calculated only in lane change situations (19)

[0062]

[0063] in m is the number of PET values ​​in this range, t A and t B They represent the time when the front vehicle leaves the conflict area and the time when the rear vehicle enters the conflict area, respectively. pet Indicates the safety critical threshold value of PET.

[0064] Furthermore, the process of step 3 is as follows:

[0065] Step 3.1: Use the action controller to select the vehicle that performs the safety-critical action

[0066] The output of the Actor network is defined as 33 discrete maneuvers, expressed as follows:

[0067]

[0068] where a L Indicates a left turn, a R Indicates a right turn, a S Indicates going straight;

[0069] So the action space In order to make the maneuver performed by AV have the greatest impact on CAV, before executing the output maneuver, it is necessary to judge whether the maneuver can be performed at the current time, that is, whether the AV and CAV are in a safety-critical state. The judgment of the safety-critical state is also determined by the TTC indicator, as described in the state construction section; therefore, an action controller is designed to judge the safety-critical state between vehicles. The judgment process is as follows:

[0070]

[0071] in, represents the TTC value between the i-th AV and CAV, Indicates that the vehicle is in a safety-critical state; otherwise, it is not in a safety-critical state. In SCAS, only the AV in the safety-critical state is allowed to perform the adversarial maneuver output by the model, while the AV not in the safety-critical state will perform the natural driving maneuver like the BV;

[0072] Step 3.2: Generate safety-critical and aggressive scenarios

[0073] In summary, to generate SCAS, we only need to use the new AV behavior distribution To replace part of the BV behavior distribution in formula (4) in step 1.1 The new behavior distribution is expressed as follows:

[0074]

[0075] The change of variable x' in the final generated SCAS scenario is expressed as follows:

[0076]

[0077] The technical concept of the present invention is as follows: first, a natural driving dataset is used to simulate a natural driving scenario, and then an adversarial vehicle group is constructed with CAV as the center. The vehicle group is modeled as a graph, and a safety-aware adjacency matrix is ​​constructed based on the TTC correlation between vehicles; features are extracted from the graph through a graph convolutional network, and then a reinforcement learning algorithm and a reward system of introduced safety-critical indicators are used to continuously adjust the model so that its output can affect the safety-critical state of the CAV; finally, the right time and the right vehicle are selected to let it execute the action output by the model to interact with the CAV, and finally a safety-critical and aggressive test scenario is obtained.

[0078] The beneficial effects of the present invention are mainly manifested in that by generating safety-critical and aggressive test scenarios, the performance of the autonomous driving vehicle decision-making and control model in dealing with unnatural dangerous driving states and dangerous traffic conditions can be tested in different complex traffic environments, thereby improving the test of the safety of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a flow chart of a method for generating safety-critical and aggressive test scenarios for autonomous vehicles. DETAILED DESCRIPTION

[0080] The present invention will be further described below in conjunction with the accompanying drawings.

[0081] Reference Figure 1 , a method for generating safety-critical and aggressive test scenarios for autonomous vehicles, comprising the following steps:

[0082] Step 1: First, obtain the random natural position distribution and natural driving state distribution of the background vehicle BV from the natural driving dataset, i.e., following or free following, and initialize them into the simulation environment; then, with the autonomous driving vehicle CAV as the center, select the vehicles in front and behind its lane and the vehicles in front and behind the left and right lanes within its observation range as the adversarial vehicle group that directly interacts with the CAV; model the vehicle group as a graph, and construct a safety perception adjacency matrix based on the collision time TTC correlation between vehicles to characterize the safety-critical states within the vehicle group;

[0083] In step 1, the process of environment initialization and graph construction is as follows:

[0084] Step 1.1, Environment Initialization

[0085] A natural driving test scenario can be represented as a series of parameters predetermined by the operational design domain (such as road type, weather conditions, etc.) and variables that may vary (such as the acceleration of the BV). The variables can be represented as follows:

[0086]

[0087] where x i,j represents the variables (e.g., position and velocity) of the i-th BV at the j-th time step, N represents the number of BVs of interest, T represents the total number of time steps, and X represents the feasible space of variables;

[0088] By using the spatiotemporal independence relationship between variables to simplify the problem, assuming the Markov property, the changes in variables in the natural driving test scenario can be expressed as the natural driving maneuvers made by all BVs in the scenario for the current CAV vehicle state and its own state, as shown below:

[0089]

[0090] The states and natural driving maneuvers at time steps t=0,…,T are expressed as

[0091]

[0092] in Indicates the status of the CAV, represents the state of the i-th BV, represents the natural driving maneuver of the i-th BV. Assuming that all BVs choose maneuvers simultaneously and independently, then:

[0093]

[0094] Therefore, the goal of initialization is to simulate the natural speed and distance of the vehicle as the starting point of the simulation, that is, to initialize s0. To achieve this goal, the position of the vehicle in the initial area and the speed in the empirical speed distribution are randomly sampled from the data set, and the position and speed of the first vehicle in each lane are determined accordingly; then, the joint distribution of the distance from the front to the rear of the vehicle and the relative speed in the empirical distribution in the data set is queried to determine the initial position and speed of the downstream vehicle for each lane in turn. After the initial position and speed of each vehicle are determined, it is placed in the corresponding position of the simulation environment SUMO; after all lanes are placed according to the natural distribution of vehicles, the environment initialization is completed;

[0095] Step 1.2: Graph Construction

[0096] Taking CAV as the center, we select n (1≤n≤6≤N) vehicles in front and behind its lane and in front and behind the left and right lanes within its observation range as adversarial vehicles (AVs) to interact directly with CAV. Then, we define CAV and AVs together as the safety perception map. in represents the set of vehicle nodes at time t, represents a CAV vehicle, represents the safety perception relationship between vehicles, and the TTC value between vehicles is used to define the above relationship. Therefore, the safety perception relationship between vehicle i and vehicle j is defined as follows:

[0097]

[0098] where δ ttc represents the safety critical threshold value of TTC, represents the TTC value between the i-th vehicle and the j-th vehicle. Therefore, the safety perception adjacency matrix is:

[0099]

[0100] We also define the spatial perception map Represents the spatial perception adjacency matrix, which is used to represent the spatial existence relationship between vehicles;

[0101] For two graphs, the state space is represented as in in Indicates the distance between the current vehicle and the CAV, Indicates speed, and represents the lateral and longitudinal position of the vehicle, Indicates the vehicle's heading angle. Indicates the lane the vehicle is in. represents the vehicle acceleration, represents the action of the vehicle in the previous time step;

[0103] Step 2: A graph reinforcement learning framework is proposed to train the model. The spatial relationship and safety perception relationship between vehicle groups are extracted through a graph convolutional network. The extracted features are then input into the reinforcement algorithm to obtain actions. A reward function system based on safety-critical indicators is introduced to adjust the actions, so that the actions output by the model can better affect the safety-critical state of CAV.

[0104] In step 2, the model training process based on graph reinforcement learning is as follows:

[0105] Step 2.1: A graph reinforcement learning network is constructed to extract input features and output adversarial maneuvers. The network is mainly constructed by a graph convolutional network (GCN) and a reinforcement learning algorithm (PPO). Before using GCN to extract features from the input, the state matrix S is first converted to t Input into the encoder (FC Layer), the state features are mapped to the sample tag space, which is expressed as follows:

[0106] S FC =φ FC (S t ) (7)

[0107] Where S FC represents the node feature matrix after FC processing; φ FC Indicates the FC layer.

[0108] Then use the GCN module to perceive the adjacency matrix from the space and the security-aware adjacency matrix The spatial perception relationship and safety perception relationship between vehicle groups are extracted and expressed as follows:

[0109]

[0110] Among them, SP t and SA t They represent the spatial features and security perception features obtained after GCN processing respectively; and Represents the graph convolution operation on two matrices; D s and D ttc are based on and The calculated degree matrix; W s and W ttc represents the weight matrix; b s and b ttc represents bias; σ is the activation function;

[0111] Then the spatial perception feature SP t , Safety perception features SA t and state characteristics S FC Aggregation, input to the reinforcement learning module, the Actor network first receives the features and outputs the safety-critical action a t , which is expressed as follows:

[0112] F t =φ Cat (S FC , S.P. t , S.A. t ) (10)

[0113]

[0114] in, represents the Actor network, φ Cat represents the feature aggregation operation, F t represents the aggregated feature matrix, σ is the Softmax activation function, represents the RL-filter matrix, used to represent the presence of a vehicle;

[0115] After simulating the action, calculate the reward value of the regular indicator and safety key indicator bonus value Get the reward value R of the action t , then the Critic network evaluates action a t The value of V t , which is expressed as follows:

[0116]

[0117] in, represents the Critic network;

[0118] Finally, the Actor and Critic networks are updated separately by collecting data such as state, action, reward, value, etc. at multiple time steps;

[0119] Step 2.2: Calculate the value of the regular indicator reward function

[0120] Designed a regular indicator reward system R U , which is used to adjust the maneuver of the adversarial vehicle (AV) so that it constantly interacts with the CAV, as shown below:

[0121]

[0122] In order to encourage AVs to collide with CAVs, the collision reward is set to a fixed value, and appropriate penalties are imposed on collisions between other vehicles. Because it is necessary to observe the reaction of CAVs when encountering AVs or collisions between AVs and BVs (collectively referred to as background vehicle collisions or BVCs), BVCs are not filtered out during the simulation, which will cause a chain reaction when BVC occurs, that is, BVs near the BVC will also make evasive or emergency braking actions due to the collision, but these actions may cause collisions between BVs, and ultimately multiple vehicles will rear-end collisions; in addition, since the time step required for each collision is different, this will cause the overall reward to fluctuate greatly, making it difficult for the model to learn, so It is used to limit the final collision reward, making it increase nonlinearly and gradually increase over time, δ step is the time threshold set;

[0123] In order to encourage AV to change lanes to the lane where CAV is located to interfere with the normal driving of CAV, a lane change reward R is set. L , the speed reward is The acceleration reward is The purpose of setting these two rewards is to guide AVs to maintain a speed close to that of CAV through continuous acceleration and deceleration, which corresponds to the situation in reality where vehicles accelerate and brake suddenly, thus interfering with the normal driving of CAV;

[0124] Time Reward R time The reward is a fixed value. The purpose of setting this reward is to make the overall test time as long as possible so as to test the vulnerability of CAV in dealing with aggressive behaviors. At the same time, this indicator can balance the fluctuation of the reward value when CAV leaves safely and when CAV collides with AV, so as to make the model learn stably and avoid excessive fluctuation of the strategy. The time weight value W time =W c ;

[0125] Step 2.3: Calculate the value of the safety key indicator reward function

[0126] When designing the safety-critical reward system, the calculation frequency of the safety-critical indicators was taken into consideration, that is, whether the indicator will only be calculated in specific scenarios. The three indicators TTC, DRAC, and PET were selected to cover most of the scenarios that may be encountered in the test. The TTC indicator can be calculated in any scenario, DRAC has different calculation methods in the following scenario and the lane changing scenario, and the PET indicator is only calculated in the lane changing scenario. Therefore, the calculation frequency of the three indicators should be distributed from high to low; when calculating the reward for the safety-critical indicator, only the indicator value between the i-th AV and CAV needs to be considered, and the indicator value between AVs is not considered. At the same time, those that are not within the safety-critical threshold value range will be filtered, that is, these values ​​will not be rewarded or punished. The calculation method is as follows:

[0127]

[0128] The TTC value indicates the time until a collision occurs based on the current vehicle spacing and speed difference. The shorter the time, the more unavoidable the collision. The calculation is as follows:

[0129]

[0130] in m is the number of TTC values ​​in this range;

[0131] DRAC represents the maximum deceleration required to avoid a collision. The greater the deceleration value, the less likely a collision is to be avoided. The calculation method is as follows:

[0132]

[0133] in is the number of DRAC values ​​in this range. In the lane change scenario, t A The time required for the vehicle A in the forward position to exit the conflict area, dist B represents the distance from vehicle B to the conflict area, δ drac It represents the safety critical threshold value of DRAC, and the conflict area represents the overlapping part of the future predicted trajectories of the two vehicles. For the calculation involving the conflict area, the calculation function is integrated in the SUMO simulator used, so the specific calculation method and detailed definition of the conflict area are not given here;

[0134] PET is used to calculate the time interval between one participant leaving and another participant entering a specified conflict zone. Therefore, the smaller the value of PET, the more inevitable a collision between two vehicles is, and therefore The calculation method is as follows:

[0135] Calculated only in lane change situations (19)

[0136]

[0137] in m is the number of PET values ​​in this range, t A and t B They represent the time when the front vehicle leaves the conflict area and the time when the rear vehicle enters the conflict area, respectively. pet Indicates the safety critical threshold value of PET;

[0138] Step 3: Perform simulation based on the actions output by the model. However, before letting the selected adversarial vehicle execute the maneuver output by the model, use the TTC indicator to determine whether the current adversarial vehicle and CAV are in a safety-critical state. If so, execute the output maneuver, otherwise execute the natural driving maneuver. Through the interaction between the adversarial vehicle and CAV, a safety-critical and aggressive test scenario for testing CAV is finally obtained.

[0139] In step 3, the process of simulating the safety-critical action is as follows:

[0140] Step 3.1: Use the action controller to select the vehicle that performs the safety-critical action

[0141] The output of the Actor network is defined as 33 discrete maneuvers, expressed as follows:

[0142]

[0143] where a L Indicates a left turn, a R Indicates a right turn, a s Indicates going straight.

[0144] So the action space In order to make the maneuvers performed by AV have the greatest impact on CAV, before executing the output maneuvers, it is necessary to judge whether the maneuvers can be performed at the current time, that is, whether the AV and CAV are in a safety-critical state. The judgment of the safety-critical state is also determined by the TTC indicator. The reason is as described in the state construction section. Therefore, an action controller is designed to judge the safety-critical state between vehicles. The judgment process is as follows:

[0145]

[0146] in, represents the TTC value between the i-th AV and CAV, Indicates that the vehicle is in a safety-critical state; otherwise, it is not in a safety-critical state. In SCAS, only AVs in a safety-critical state are allowed to perform the adversarial maneuvers output by the model, while AVs that are not in a safety-critical state will perform natural driving maneuvers like BVs;

[0147] Step 3.2: Generate safety-critical and aggressive scenarios

[0148] In summary, to generate SCAS, we only need to use the new AV behavior distribution To replace part of the BV behavior distribution in formula (4) in step 1.1 The new behavior distribution is expressed as follows:

[0149]

[0150] The change of variable x' in the final generated SCAS scenario is expressed as follows:

[0151]

[0152] The simulation experiment process of this embodiment is as follows:

[0153] (1) Setting up the experimental scene

[0154] All experiments were conducted in the open source traffic simulator SUMO, with the random seed set to 21. The TRACI interface was used to generate the test environment. A high-speed driving environment with multiple lanes and different distances and a low-speed urban environment were constructed. All vehicles were controlled at intervals of 100ms, and the number of test rounds for each experiment was 1500.

[0155] (2) Parameter determination

[0156] The model uses the PPO algorithm, and the input and output of the GCN and hidden layers are both 32. During the training process, the Adam optimizer is used, and the learning rate is set to (5*10-5), eps is (10-3), discount factor is (0.99), lambda is (0.96), policy clipping factor is (0.1), batch size is (128), update interval is (200), single batch update rounds are (20), and training rounds are (150). In addition, the collision reward is set Lane Change Reward R L = 10, if the AV changes lanes to the lane where the CAV is, the time reward R time =20. L =0.05,W v =W acc =0.8. Since the maximum time step of a simulation is set to 500, δ step =125. At the same time, relevant data were consulted and the safety critical threshold values ​​of three safety critical indicators were determined as follows: δ ttc =1.22,δ drac =3,δ pet =1.5, and set W ttc =W drac =W pet =3

[0157] (3) CAV test model

[0158] In order to verify the effectiveness of the proposed method, a planner algorithm was selected from the rule-based planner and the learning-based end-to-end planner as the test object, namely CAV-I and CAV-II. CAV-I is constructed by the intelligent traffic model (IDM) and the lane-changing model (MOBIL). The specific parameters a max =1.5, v0=-33.33, δ=4, s0=2.0, T=1.2, a com f f=2, and p=0.0. CAV-II is a DRL model based on the DDQN algorithm. The input of the network is similar to this model, including the speed of up to 7 vehicles, position information XY and lane information, and the output is also the same as this model, that is, 33 discrete action spaces. The reward function includes speed, time and collision. The speed reward maps the speed from [20, 40] (m / s) to [-1, 1]. The time reward is fixed at 10, and the collision reward is divided into two cases: a safe departure reward of 100 and a collision penalty of 20. The weight coefficients of each reward are the same as the corresponding reward coefficients of this model. Two hidden layers with 256 neurons and DDQN are used to train the CAV agent. During the training phase, the Adam optimizer was used, and the learning rate was set to (5*10 -5 ), eps is (10 -4 ), the discount factor is (0.99), the target network update frequency is (1000), the replay memory unit size is (50000), the warm-up step size is (5000), the batch size is (64), and the number of training rounds is (150).

[0159] (4) Experimental results

[0160] The collision rate study tested the IDM model and the DRL model in urban and highway scenarios at different distances and lanes, and compared the collision rates. Table 1 is a table of model simulation collision rate experiments. IDM refers to the test of the IDM model in a highway scenario. DRL refers to the test of the DRL model in a highway scenario. City refers to the test of the IDM model in an urban road scenario. The test results of D2RL are only used for demonstration and not for comparison.

[0161]

[0162] Table 1

[0163] As can be seen from Table 1, in the tests at various distances, the collision rate in the 3-lane scenario is always higher than that in the 2-lane scenario. This is because the model selects certain vehicles as adversarial vehicles and makes them perform specific adversarial actions at specific times. As the number of lanes increases, the number of vehicles that can interact with the CAV will also increase, resulting in an increase in the collision rate. At the same time, the 4-lane scenario was not tested because it is considered that in actual situations, the vehicles that can interact with the CAV are at most the lane where the CAV is located and the vehicles on the left and right lanes of the CAV. Therefore, even if the test is conducted in the 4-lane scenario, the BV in the lane far away from the CAV will not be selected to interact directly with the CAV, and of course it will not have much impact on the final test results.

[0164] At the same time, it can be seen from the data in the table that, with the same number of lanes, an increase in the test distance will also lead to an increase in the collision rate. This is because the longer the test distance and the longer the test time, the more opportunities there are for AV to confront CAV. The more confrontations there are, the more the defects of CAV in dealing with various confrontation maneuvers will be exposed. The lack of interactive events is precisely one of the reasons why it is difficult to conduct tests in real scenarios.

[0165] Then we can find from the table that the collision rate of the DRL-based CAV model is much higher than that of the rule-based CAV model IDM, and even reaches a collision rate of 99.1% in the 4km 3Lane scenario. Because when the DRL model is trained, it is placed in the natural driving environment NDD for training. During the training process, the DRL model is not subject to excessive external interference, so it is impossible to learn the corresponding avoidance maneuvers. When it is placed in the test environment generated by the model, because the AV can continuously make SCAMs and the DRL model has not trained a response strategy, this leads to an extremely high collision rate during the test. However, the relationship between the collision rates of this model in various scenarios is the same as that obtained by the IDM model, that is, as the test distance and the number of lanes increase, the collision rate will increase.

[0166] The last row in the table gives the test results of the IDM model in the scenario generated by the D2RL method. From the test results, it can be seen that in the D2RL scenario, the collision rate will also increase with the increase of the number of lanes and the test distance, but the method of the present invention still has an order of magnitude difference in collision rate. This difference will be demonstrated from the following points to illustrate the feasibility of the method of the present invention:

[0167] First, the method of the present invention is similar to the D2RL method in the core concept, both of which allow the adversarial agent to interact with the CAV at a specific time and perform specific adversarial maneuvers. However, there is a difference in the number of vehicles selected at the selection time: the D2RL method will only select one or two BVs as AVs when they are close to the CAV and the criticality of the action is calculated according to the importance function constructed by them. However, a safety critical indicator is used to measure the safety critical state of up to 6 surrounding vehicles through a safety critical threshold value, and those less than this threshold value are defined as safety critical states. Therefore, the number of AVs selected will be more than that of the D2RL method, which will lead to an increase in the collision rate.

[0168] Secondly, the purpose of the D2RL method is to reduce the time required for testing in the real world. They achieve this goal by learning collision strategies only from real collision events to intensify the occurrence of dangerous scenarios. Therefore, in essence, the D2RL method focuses more on realism and aims to restore as many dangerous scenarios as possible. Therefore, the scenarios generated during the test process include both safe scenarios and dangerous scenarios. Dangerous scenarios in real-life scenarios include those that occur in natural safe driving conditions and those that occur in unnatural dangerous driving conditions. Compared with D2RL, which focuses more on challenges, dangerous scenarios in unnatural dangerous driving conditions are restored during the test, while safe scenarios and dangerous scenarios in natural safe driving conditions are completely ignored.

[0169] In short, the present invention only generates a specific part of the dangerous scenes in the D2RL method and only generates dangerous scenes, so compared with the method of the present invention, the dangerous scenes generated by D2RL appear sparse in all scenes, and the collision rate is naturally lower than that of the present invention.

[0170] Ablation experiment. The model of the present invention is evaluated and analyzed by modules through ablation experiments. The key parts of the model are GCN, reinforcement learning algorithm PPO, and safety-critical indicators. They will be replaced or ablated respectively to verify the effect of the model of the present invention. For different models, only 1-2 modules will be replaced or ablated each time, and each model will be retrained to ensure the fitting of rewards and losses. From the collision rate results of the ablation experiment in Table 2, it can be seen that after replacing or ablating any module, the test effect of the model of the present invention will be reduced to varying degrees. Table 2 is an ablation experiment table,

[0171] 'r / t' means that the module in the model is replaced, and the previous module is replaced by the latter one.

[0172] 'w / o' means that the model has no specific modules. The indicators here refer to safety-critical indicators.

[0173]

[0174] Table 2

[0175] From the collision rate, we can see that the complete model using the PPO algorithm has the highest collision rate among all models.

[0176] Similarly, after removing GCN from the model of the present invention, the collision rate has dropped significantly. Therefore, this shows that it is very effective to use a graph structure to model a vehicle group and extract the spatial perception relationship and safety perception relationship between the vehicle groups through the GCN network. The two extracted features can enhance the aggressiveness of the counter-maneuver, thereby better testing the CAV model.

[0177] For the model that eliminated the critical safety indicators during the training process, there is a significant reduction in the collision rate. This shows that the adversarial maneuvers made by the model trained by the critical safety indicators of the present invention are more likely to affect the safety status of the CAV and cause collisions, which is exactly in line with the purpose of designing the critical safety indicator system of the present invention. After eliminating GCN and critical safety indicators, the collision rate of the model test reached the lowest, and the learned adversarial maneuvers were relatively more inclined to MUM.

[0178] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for generating safety-critical and aggressive test scenarios for autonomous vehicles, characterized in that: The method comprises the following steps: Step 1: First, obtain the random natural position distribution and natural driving state distribution of the background vehicle BV from the natural driving dataset, i.e., following or free following, and initialize them into the simulation environment; then, with the autonomous driving vehicle CAV as the center, select the vehicles in front and behind its lane and the vehicles in front and behind the left and right lanes within its observation range as the adversarial vehicle group that directly interacts with the CAV; model the vehicle group as a graph, and construct a safety perception adjacency matrix based on the collision time TTC correlation between vehicles to characterize the safety-critical states within the vehicle group; Step 2: A graph reinforcement learning framework is proposed to train the model. The spatial relationship and safety perception relationship between vehicle groups are extracted through a graph convolutional network. The extracted features are then input into the reinforcement algorithm to obtain actions. A reward function system based on safety-critical indicators is introduced to adjust the actions, so that the actions output by the model can better affect the safety-critical state of CAV. Step 3: Perform simulation based on the actions output by the model. However, before letting the selected adversarial vehicle execute the maneuver output by the model, use the TTC indicator to determine whether the current adversarial vehicle and CAV are in a safety-critical state. If so, execute the output maneuver; otherwise, execute the natural driving maneuver. Through the interaction between the adversarial vehicle and CAV, we finally obtain the safety-critical and aggressive test scenario SCAS for testing CAV.

2. The method for generating safety-critical and aggressive test scenarios for an autonomous driving vehicle according to claim 1, wherein: The process of step 1 is as follows: Step 1.1, Environment Initialization A natural driving test scenario is represented as a series of parameters and possible variables that are predetermined by the operational design domain. The variables are represented as follows: where x i,j represents the variable of the i-th BV at the j-th time step, N represents the number of BVs of interest, T represents the total number of time steps, and X represents the feasible space of the variable; By using the spatiotemporal independence relationship between variables to simplify the problem, assuming the Markov property, the changes in variables in the natural driving test scenario can be expressed as the natural driving maneuvers made by all BVs in the scenario for the current CAV vehicle state and its own state, as shown below: The states and natural driving maneuvers at time steps t=0,…,T are expressed as in Indicates the status of the CAV, represents the state of the i-th BV, represents the natural driving maneuver of the i-th BV. Assuming that all BVs choose maneuvers simultaneously and independently, then: Therefore, the goal of initialization is to simulate the natural speed and distance of the vehicle as the starting point of the simulation, that is, to initialize s0. To achieve this goal, the position of the vehicle in the initial area and the speed in the empirical speed distribution are randomly sampled from the data set, and the position and speed of the first vehicle in each lane are determined accordingly; then, the joint distribution of the distance from the front to the rear of the vehicle and the relative speed in the empirical distribution in the data set is queried to determine the initial position and speed of the downstream vehicle for each lane in turn. After the initial position and speed of each vehicle are determined, it is placed in the corresponding position of the simulation environment SUMO. After all lanes are placed according to the natural distribution of vehicles, the environment initialization is completed; Step 1.2: Graph construction Taking CAV as the center, select n BVs in its observation range to interact directly with CAV, including the vehicles in front and behind its lane and the vehicles in front and behind its left and right lanes, 1≤n≤6≤N. Then, CAV and AVs are defined as the safety perception map. in represents the set of vehicle nodes at time t, represents a CAV vehicle, represents the safety perception relationship between vehicles. The TTC value between vehicles is used to define the above relationship. Therefore, the safety perception relationship between vehicle i and vehicle j is defined as follows: where δ ttc represents the safety critical threshold value of TTC, represents the TTC value between the i-th vehicle and the j-th vehicle, so the safety perception adjacency matrix is: We also define the spatial perception map Represents the spatial perception adjacency matrix, which is used to represent the spatial existence relationship between vehicles; For two graphs, the state space is represented as in in Indicates the distance between the current vehicle and the CAV, Indicates speed, and represents the lateral and longitudinal position of the vehicle, Indicates the vehicle's heading angle. Indicates the lane the vehicle is in. represents the vehicle acceleration, represents the action of the vehicle in the previous time step.

3. The method for generating safety-critical and aggressive test scenarios for an autonomous driving vehicle according to claim 1 or 2, characterized in that: The process of step 2 is as follows: Step 2.1, a graph reinforcement learning network is constructed to extract input features and output adversarial maneuvers. The network is constructed by the graph convolutional network GCN and the reinforcement learning algorithm PPO. Before using GCN to extract features from the input, the state matrix S is first converted to t Input into the encoder, the state features are mapped to the sample label space, which is expressed as follows: S FC =φ FC (S t ) (7) Where S FC represents the node feature matrix after FC processing; φ FC represents the FC layer; Then use the GCN module to perceive the adjacency matrix from the space and the security-aware adjacency matrix The spatial perception relationship and safety perception relationship between vehicle groups are extracted and expressed as follows: Among them, SP t and SA t They represent the spatial features and security perception features obtained after GCN processing respectively; and Represents the graph convolution operation on two matrices; D s and D ttc are based on and The calculated degree matrix; W s and W ttc represents the weight matrix; b s and b ttc represents bias; σ is the activation function; Then the spatial perception feature SP t , Safety perception features SA t and state characteristics S FC Aggregation, input to the reinforcement learning module, the Actor network first receives the features and outputs the safety-critical action a t , which is expressed as follows: F t =φ Cat (S FC ,SP t ,SA t ) (10) in, represents the Actor network, φ Cat represents the feature aggregation operation, F t represents the aggregated feature matrix, σ is the Softmax activation function, represents the RL-filter matrix, used to represent the presence of a vehicle; After simulating the action, calculate the reward value of the regular indicator and safety key indicator bonus value Get the reward value R of the action t , then the Critic network evaluates action a t The value of V t , which is expressed as follows: in, represents the Critic network; Finally, the Actor and Critic networks are updated separately by collecting state, action, reward, and value data at multiple time steps; Step 2.2: Calculate the value of the regular indicator reward function Designed a regular indicator reward system R U , which is used to adjust the maneuver of the adversarial vehicle AV so that it constantly interacts with the CAV, and is expressed as follows: W c Used to limit the final collision reward, It increases nonlinearly and gradually over time, δ step is the time threshold; the lane change reward R is set L , the speed reward is The acceleration reward is Time Reward R time is a fixed value, and the time weight value W time =W c ; Step 2.3: Calculate the value of the safety key indicator reward function Designed a safety-critical indicator reward function system It is used to adjust the maneuver of the AV to affect the safety-critical state of the CAV and is calculated as follows: The calculation is as follows: in m is the number of TTC values ​​in this range; The calculation method is as follows: in is the number of DRAC values ​​in this range. In the lane change scenario, t A The time required for the vehicle A in the forward position to exit the conflict area, dist B represents the distance from vehicle B to the conflict area, δ drac Indicates the critical safety threshold of DRAC; The calculation method is as follows: Calculated only in lane change situations (19) in m is the number of PET values ​​in this range, t A and t B They represent the time when the front vehicle leaves the conflict area and the time when the rear vehicle enters the conflict area, respectively. pet Indicates the safety critical threshold value of PET.

4. The method for generating safety-critical and aggressive test scenarios for an autonomous driving vehicle according to claim 1 or 2, wherein: The process of step 3 is as follows: Step 3.1: Use the action controller to select the vehicle that performs the safety-critical action The output of the Actor network is defined as 33 discrete maneuvers, expressed as follows: where a L Indicates a left turn, a R Indicates a right turn, a S Indicates going straight; So the action space In order to make the maneuver performed by AV have the greatest impact on CAV, before executing the output maneuver, it is necessary to judge whether the maneuver can be performed at the current time, that is, whether the AV and CAV are in a safety-critical state. The judgment of the safety-critical state is also determined by the TTC indicator. Therefore, an action controller is designed to judge the safety-critical state between vehicles. The judgment process is as follows: in, represents the TTC value between the i-th AV and CAV, Indicates that the vehicle is in a safety-critical state; otherwise, it is not in a safety-critical state. In SCAS, we only allow AVs in a safety-critical state to perform adversarial maneuvers output by the model, while AVs that are not in a safety-critical state will perform natural driving maneuvers like BVs. Step 3.2: Generate safety-critical and aggressive scenarios Using new AV behavior distribution To replace part of the BV behavior distribution in formula (4) in step 1.1 The new behavior distribution is expressed as follows: The changes of variable x' in the generated SCAS scenario are expressed as follows:

Images