Automatic driving dangerous scene generation method and system based on confrontation reward training

Abstract

The invention relates to the field of automatic driving testing, and provides an automatic driving dangerous scene generation method and system based on confrontation reward training, and the method comprises the steps: initializing a driving simulation environment of a multi-lane road; a punishment action is randomly generated based on scene state parameters in the vehicle data, the scene state parameters, the punishment action and corresponding action parameters form a state action pair, and under the condition of no historical action similarity punishment, a state action value is calculated; performing historical action similarity punishment on the punishment action through a punishment function, and calculating a state action value after the historical action similarity punishment is updated; and updating the reinforcement learning model according to the difference value between the state action value in the reinforcement learning model and the expected state action value to guide the generation of an automatic driving dangerous scene. According to the method, the efficiency and the quality of automatic driving system testing are remarkably improved, meanwhile, the scene quality loss is reduced, and the diversity and the practicability of the scene are improved in a short training period.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving testing, and in particular to a method and system for generating dangerous scenarios for autonomous driving based on adversarial reward training. Background Art

[0002] In order to improve the safety of autonomous driving and verify its reliability, comprehensive testing must be carried out before autonomous driving is officially deployed, with the main purpose of discovering any potential failures or dangerous situations. Simulation testing is the main content of autonomous driving testing due to its advantages of high efficiency and low cost. Simulation testing consists of many scenarios, which is defined as the process in which the test vehicle interacts with other traffic participants in a simulated traffic environment to achieve its driving intention within a specified time period. Among them, dangerous scenarios directly demonstrate the capability boundaries of autonomous driving, and therefore play a core role in simulation testing. Due to the rarity of safety-critical events in the real world, it is not enough to rely solely on real-world databases to collect them. Therefore, generating effective dangerous scenarios for autonomous driving testing is an important research direction.

[0003] At present, researchers have applied various principles and methods to scene generation. Among them, the online scene generation method based on deep reinforcement learning has become the main research direction because it can change the scene state in real time according to the feedback of the tested object, which has the advantage of being more challenging. In this field, researchers have conducted in-depth research on scene modeling, reward setting, event probability, multi-vehicle scenes and training convergence.

[0004] Although many achievements have been made in this field, there are still two major problems in terms of actual results:

[0005] (1) The limitation of the adversarial reward mechanism is that the reward is mainly centered on collision, ignoring the risk assessment in the process. This leads to oversimplification of the generated test scenarios, which only focus on collisions and fail to reflect other types of dangers, such as near-collision situations. This deviation is not conducive to truly evaluating the response complexity of the autonomous driving system.

[0006] (2) The application of deep reinforcement learning faces challenges in improving the diversity of test scenarios. Although attempts have been made to address this problem by introducing diversity metrics into the reward function, this approach may destroy the original optimal action strategy, resulting in the generated scenarios being neither practical nor having the expected test value. In addition, the repeated training requirements of this approach make it inefficient in actual operation and difficult to meet the needs of rapid iteration. Summary of the invention

[0007] The purpose of the present invention is to solve the above technical problems and propose a method and system for generating dangerous scenarios for autonomous driving based on adversarial reward training. The method replaces the traditional collision reward with real-time scenario risk assessment to avoid generating unsolvable scenarios, effectively overcome the degradation of scenario quality and the need for repeated training caused by the existing diversity methods, and significantly improve the efficiency and quality of autonomous driving tests. The purpose of the present invention can be achieved by the following technical solutions:

[0008] The present invention provides a method for generating dangerous scenarios for autonomous driving based on adversarial reward training, comprising the following steps:

[0009] Step S1, initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, set the control algorithm and parameters for the target vehicle to perform automatic driving; based on the estimated collision time between the target vehicle and the challenge vehicle, introduce a reward function to encourage the challenge vehicle to perform dangerous test behaviors, and at the same time establish a coordinate system in the simulation environment and collect vehicle data in real time;

[0010] Step S2, based on the scene state parameters in the vehicle data, randomly generate penalty actions, convert the penalty actions into action parameters through action network transformation, form state-action pairs with the scene state parameters, penalty actions and corresponding action parameters, input them into the pre-trained reinforcement learning model, and calculate the state-action value without historical action similarity penalty;

[0011] Step S3, according to the similarity between the current action and the historical action of the challenge vehicle, the penalty action is penalized by the penalty function for the historical action similarity to avoid repeated dangerous test behaviors; the state-action pair is input into the reinforcement learning model to calculate the state-action value after the historical action similarity penalty is updated;

[0012] Step S4, taking the sum of the state action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state action value, and updating the reinforcement learning model according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of dangerous scenarios for autonomous driving.

[0013] Furthermore, establishing a coordinate system in the simulation environment includes:

[0014] Set a specified point on the road centerline as the origin, the road centerline as the longitudinal axis, and set a transverse axis through the origin and perpendicular to the road centerline to establish a coordinate system.

[0015] Furthermore, based on the estimated collision time between the target vehicle and the challenge vehicle, a reward function is introduced to encourage the challenge vehicle to perform dangerous test behaviors, including:

[0016] The estimated collision time is determined based on the scene state parameters of the target vehicle and the challenge vehicle, and the scene risk is quantitatively evaluated. The formula is:

[0017]

[0018] Where TTC is the estimated time to collision, v vel is the relative speed between the target car and the challenge car; a rel is the relative speed between the target car and the challenge car, d re l is the relative distance;

[0019] According to the expected collision time, a reward function is introduced to encourage the challenge vehicle to perform dangerous test behaviors. The formula is:

[0020]

[0021] Among them, r TTC It is a reward value based on the estimated collision time, which is used to optimize the driving strategy of the challenge vehicle.

[0022] Furthermore, based on the scene state parameters in the vehicle data, penalty actions are randomly generated, and the penalty actions are converted into action parameters through action network transformation. The scene state parameters, penalty actions and corresponding action parameters form a state-action pair and are input into the pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated, including:

[0023] Determine and collect the scene state parameters of the target vehicle and the challenge vehicle at each time step; the scene state parameters include the horizontal and vertical positions, horizontal and vertical speeds;

[0024] Define the scope and type of penalty actions, and randomly generate penalty actions for the challenge vehicle;

[0025] The scene state parameters and penalty actions are output through the action network to the corresponding action parameters;

[0026] The scene state parameters, penalty actions and corresponding action parameters are combined into state-action pairs and input into the pre-trained reinforcement learning model. The state-action value is obtained through the value network in the pre-trained reinforcement learning model.

[0027] Furthermore, according to the similarity between the current action and the historical action of the challenging vehicle, the historical action similarity penalty is imposed on the penalty action through the penalty function, including:

[0028] Create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0029] After the challenge car takes random actions under the current scene state parameters, it updates the database and adds action records;

[0030] According to the current scene state parameters, all historical actions are extracted from the database; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain the penalty value R(s,a), the formula is:

[0031]

[0032] Among them, s is the current scene state parameter, a is the current action, and a i History actions, History(s,a) is all the history actions under the current scene state parameters; n is the total number of history actions.

[0033] Furthermore, the reinforcement learning model is updated according to the difference between the state action value and the expected state action value in the reinforcement learning model, and the guidance of generating the dangerous scenario of autonomous driving includes:

[0034] The current scene state parameters and current action are obtained, and combined with the action parameters recommended by the action network to form a state-action pair, which are input into the pre-trained reinforcement learning model and the reinforcement learning model respectively;

[0035] The state-action value obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model;

[0036] Calculate the penalty value of historical action similarity penalty based on the state-action pair;

[0037] The state-action value and penalty value in the pre-trained reinforcement learning model are combined as the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

[0038] Based on the same inventive concept, the present invention also provides an autonomous driving dangerous scene generation system based on adversarial reward training, comprising:

[0039] The scenario generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, and set the control algorithm and parameters for the target vehicle to perform autonomous driving;

[0040] The scenario real-time risk reward evaluation module is used to introduce a reward function to encourage the challenge vehicle to perform dangerous test behaviors based on the estimated collision time between the target vehicle and the challenge vehicle. At the same time, a coordinate system is established in the simulation environment and vehicle data is collected in real time.

[0041] The pre-training module is used to randomly generate penalty actions based on the scene state parameters in the vehicle data, convert the penalty actions into action parameters through action network transformation, and form state-action pairs with the scene state parameters, penalty actions and corresponding action parameters, which are input into the pre-trained reinforcement learning model to calculate the state-action value without historical action similarity penalty.

[0042] The historical action penalty module is used to punish the penalty action based on the similarity between the current action and the historical action of the challenge vehicle through a penalty function to avoid repeated dangerous test behaviors;

[0043] Reinforcement learning training module; used to input state-action pairs into the reinforcement learning model, calculate the state-action value after the historical action similarity penalty is updated; take the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state-action value, and update the reinforcement learning model according to the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of dangerous scenarios for autonomous driving.

[0044] Furthermore, the scenario real-time risk reward evaluation module includes:

[0045] The collision time prediction unit is used to determine the estimated collision time based on the scene state parameters of the target vehicle and the challenge vehicle, and to quantitatively evaluate the scene risk. The formula is:

[0046]

[0047] Where TTC is the estimated time to collision, v vel is the relative speed between the target car and the challenge car; a rel is the relative speed between the target car and the challenge car, d rel is the relative distance;

[0048] The risk assessment unit is used to introduce a reward function to motivate the challenge vehicle to perform dangerous test behaviors based on the expected collision time. The formula is:

[0049]

[0050] Among them, r TTC It is a reward value based on the estimated collision time, which is used to optimize the driving strategy of the challenge vehicle;

[0051] The reward function generation unit is used to dynamically generate reward or penalty signals based on the risk assessment results to encourage the challenge vehicle to take specified actions to deal with dangerous situations.

[0052] Furthermore, it is characterized in that the historical action penalty module includes:

[0053] A database management unit, used for creating a database for each scene state parameter, for storing all historical actions under the scene state parameter;

[0054] Action recording and updating unit: after the challenge car takes random actions under the current scene state parameters, it updates the database and adds action records;

[0055] The historical action retrieval penalty unit extracts all historical actions from the database according to the current scene state parameters; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain the penalty value R(s, a), the formula is:

[0056]

[0057] Among them, s is the current scene state parameter, a is the current action, and a i History actions, History(s, a) is all history actions under the current scene state parameters; n is the total number of history actions.

[0058] Compared with the prior art, the present invention has at least one of the following technical effects:

[0059] The present invention significantly improves the efficiency and quality of autonomous driving system testing, while reducing scene quality loss and increasing scene diversity and practicability within a shorter training cycle, providing strong support for safety verification and functional optimization of autonomous driving technology.

[0060] (1) The present invention effectively overcomes the problems of unsolvable scenarios and non-collision scenarios that may be caused by traditional collision final state rewards through a real-time danger reward mechanism. At the same time, this reward mechanism not only improves the practicality of the scenario, but also enhances the comprehensiveness of the test.

[0061] (2) The present invention adopts a historical action neighborhood penalty strategy to solve the problem of scene quality degradation that may be caused by explicit diversity rewards. By penalizing actions that are too similar, the diversity and richness of the scenes are ensured, thereby more comprehensively testing the responsiveness of the autonomous driving system. Finally, a diversity result is achieved by implementing a single-step difference reward without repeated multiple training. The scenes generated by the present invention greatly reduce the proportion of unsolvable scenes and generate non-collision hazard scenes. Experiments have shown that the present invention improves scene diversity with only two rounds of training, while greatly reducing scene quality loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for describing the embodiment:

[0063] Figure 1 A flowchart of the steps of the method for generating dangerous scenarios for autonomous driving based on adversarial reward training according to the present invention;

[0064] Figure 2 This is a working principle diagram of this embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0066] Example 1

[0067] In the existing technology, the safety verification of autonomous vehicles is carried out through simulation testing, in which the generation of challenging test scenarios is crucial. However, current simulation tests mainly rely on collision scenarios to evaluate the response of the autonomous driving system. Although this method is direct, it cannot fully reflect the various dangerous situations that may be encountered in actual driving. Specifically, on the one hand, the behavior of the attacking vehicle controlled by the collision reward is only to hit the test vehicle as quickly as possible, such as rear-ending or cutting in at a very close distance. This may lead to unsolvable scenarios, which is not conducive to improving the performance of autonomous driving. On the other hand, it cannot generate non-collision dangerous scenarios, which should also be considered and prevented for autonomous driving.

[0068] In addition, although existing deep reinforcement learning methods can dynamically generate test scenarios, they often only focus on collision results and ignore risk changes in the process because of the designed adversarial rewards, resulting in a lack of diversity in the generated scenarios and insufficient coverage of non-collision but equally dangerous situations. Specifically, deep reinforcement learning tends to converge to the optimal solution, resulting in a lack of diversity in the generated scenarios. Existing methods attempt to improve scenario diversity by explicitly adding diversity indicators to the reward function. However, such methods may cause actions to lose their original optimality or even induce extreme actions, resulting in invalid scenarios. In addition, existing methods for improving diversity require retraining based on previously generated scenarios, which is time-consuming and impractical. These limitations greatly reduce the practical value of the generated scenarios.

[0069] Based on these considerations, this application proposes a new method for generating dangerous test scenarios for autonomous driving, which aims to overcome the limitation of existing technologies that only rely on collision results by comprehensively considering real-time risk assessment of collision and non-collision factors. This method adopts a dynamic adversarial reward mechanism, which not only focuses on whether a collision occurs, but also considers the behavior patterns and choices of the vehicle before approaching a collision, so as to more realistically simulate potential dangerous situations. At the same time, by introducing historical action similarity penalties and single-step difference reward strategies, this method effectively improves the diversity and practicality of the scenarios, and avoids the reduction of scenario quality and waste of computing resources due to over-training. The specific implementation methods are as follows:

[0070] like Figure 1 As shown, the present invention provides a method for generating dangerous scenarios for autonomous driving based on adversarial reward training, comprising the following steps:

[0071] Step S1, initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, set the control algorithm and parameters for the target vehicle to perform automatic driving; based on the estimated collision time between the target vehicle and the challenge vehicle, introduce a reward function to encourage the challenge vehicle to perform dangerous test behaviors, and at the same time establish a coordinate system in the simulation environment and collect vehicle data in real time;

[0072] Step S2, based on the scene state parameters in the vehicle data, randomly generate penalty actions, convert the penalty actions into action parameters through action network transformation, form state-action pairs with the scene state parameters, penalty actions and corresponding action parameters, input them into the pre-trained reinforcement learning model, and calculate the state-action value without historical action similarity penalty;

[0073] Step S3, according to the similarity between the current action and the historical action of the challenge vehicle, the penalty action is penalized by the penalty function for the historical action similarity to avoid repeated dangerous test behaviors; the state-action pair is input into the reinforcement learning model to calculate the state-action value after the historical action similarity penalty is updated;

[0074] Step S4, taking the sum of the state action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state action value, and updating the reinforcement learning model according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of dangerous scenarios for autonomous driving.

[0075] Furthermore, establishing a coordinate system in the simulation environment includes:

[0076] Set a specified point on the road centerline as the origin, the road centerline as the longitudinal axis, and set a transverse axis through the origin and perpendicular to the road centerline to establish a coordinate system.

[0077] Furthermore, based on the estimated collision time between the target vehicle and the challenge vehicle, a reward function is introduced to encourage the challenge vehicle to perform dangerous test behaviors, including:

[0078] The estimated collision time is determined based on the scene state parameters of the target vehicle and the challenge vehicle, and the scene risk is quantitatively evaluated. The formula is:

[0079]

[0080] Where TTC is the estimated time to collision, v vel is the relative speed between the target car and the challenge car; a rel is the relative speed between the target car and the challenge car, d re l is the relative distance;

[0081] According to the expected collision time, a reward function is introduced to encourage the challenge vehicle to perform dangerous test behaviors. The formula is:

[0082]

[0083] Among them, r TTC It is a reward value based on the estimated collision time, which is used to optimize the driving strategy of the challenge vehicle.

[0084] Furthermore, based on the scene state parameters in the vehicle data, penalty actions are randomly generated, and the penalty actions are converted into action parameters through action network transformation. The scene state parameters, penalty actions and corresponding action parameters form a state-action pair and are input into the pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated, including:

[0085] Determine and collect the scene state parameters of the target vehicle and the challenge vehicle at each time step; the scene state parameters include the horizontal and vertical positions, horizontal and vertical speeds;

[0086] Define the scope and type of penalty actions, and randomly generate penalty actions for the challenge vehicle;

[0087] The scene state parameters and penalty actions are output through the action network to the corresponding action parameters;

[0088] The scene state parameters, penalty actions and corresponding action parameters are combined into state-action pairs and input into the pre-trained reinforcement learning model. The state-action value is obtained through the value network in the pre-trained reinforcement learning model.

[0089] Furthermore, according to the similarity between the current action and the historical action of the challenging vehicle, the historical action similarity penalty is imposed on the penalty action through the penalty function, including:

[0090] Create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0091] After the challenge car takes random actions under the current scene state parameters, it updates the database and adds action records;

[0092] According to the current scene state parameters, all historical actions are extracted from the database; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain the penalty value R(s,a), the formula is:

[0093]

[0094] Among them, s is the current scene state parameter, a is the current action, and a i History actions, History(s,a) is all the history actions under the current scene state parameters; n is the total number of history actions.

[0095] Furthermore, the reinforcement learning model is updated according to the difference between the state action value and the expected state action value in the reinforcement learning model, and the guidance of generating the dangerous scenario of autonomous driving includes:

[0096] The current scene state parameters and current action are obtained, and combined with the action parameters recommended by the action network to form a state-action pair, which are input into the pre-trained reinforcement learning model and the reinforcement learning model respectively;

[0097] The state-action value obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model;

[0098] Calculate the penalty value of historical action similarity penalty based on the state-action pair;

[0099] The state-action value and penalty value in the pre-trained reinforcement learning model are combined as the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

[0100] Example 2

[0101] Based on the same inventive concept, the present invention also provides an autonomous driving dangerous scene generation system based on adversarial reward training, comprising:

[0102] The scenario generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, and set the control algorithm and parameters for the target vehicle to perform autonomous driving;

[0103] The scenario real-time risk reward evaluation module is used to introduce a reward function to encourage the challenge vehicle to perform dangerous test behaviors based on the estimated collision time between the target vehicle and the challenge vehicle. At the same time, a coordinate system is established in the simulation environment and vehicle data is collected in real time.

[0104] The pre-training module is used to randomly generate penalty actions based on the scene state parameters in the vehicle data, convert the penalty actions into action parameters through action network transformation, and form state-action pairs with the scene state parameters, penalty actions and corresponding action parameters, which are input into the pre-trained reinforcement learning model to calculate the state-action value without historical action similarity penalty.

[0105] The historical action penalty module is used to punish the penalty action based on the similarity between the current action and the historical action of the challenge vehicle through a penalty function to avoid repeated dangerous test behaviors;

[0106] Reinforcement learning training module; used to input state-action pairs into the reinforcement learning model, calculate the state-action value after the historical action similarity penalty is updated; take the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state-action value, and update the reinforcement learning model according to the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of dangerous scenarios for autonomous driving.

[0107] Furthermore, the scenario real-time risk reward evaluation module includes:

[0108] The collision time prediction unit is used to determine the estimated collision time based on the scene state parameters of the target vehicle and the challenge vehicle, and to quantitatively evaluate the scene risk. The formula is:

[0109]

[0110] Where TTC is the estimated time to collision, v vel is the relative speed between the target car and the challenge car; a rel is the relative speed between the target car and the challenge car, d re l is the relative distance;

[0111] The risk assessment unit is used to introduce a reward function to motivate the challenge vehicle to perform dangerous test behaviors based on the expected collision time. The formula is:

[0112]

[0113] Among them, r TTC It is a reward value based on the estimated collision time, which is used to optimize the driving strategy of the challenge vehicle;

[0114] The reward function generation unit is used to dynamically generate reward or penalty signals based on the risk assessment results to encourage the challenge vehicle to take specified actions to deal with dangerous situations.

[0115] Furthermore, it is characterized in that the historical action penalty module includes:

[0116] A database management unit, used for creating a database for each scene state parameter, for storing all historical actions under the scene state parameter;

[0117] Action recording and updating unit: after the challenge car takes random actions under the current scene state parameters, it updates the database and adds action records;

[0118] The historical action retrieval penalty unit extracts all historical actions from the database according to the current scene state parameters; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain the penalty value R(s, a), the formula is:

[0119]

[0120] Among them, s is the current scene state parameter, a is the current action, and a i History actions, History(s, a) is all history actions under the current scene state parameters; n is the total number of history actions.

[0121] Example 3

[0122] In order to more clearly demonstrate the technical details and applications of the present invention, the following content will introduce an exemplary embodiment in detail in combination with the specific system configuration and step implementation process. This embodiment will describe the generation process of the autonomous driving dangerous test scenario, the technical modules involved and their interactions, and how to actually apply the technical solution of the present invention to show how the present invention solves the problems existing in the prior art and improves the effectiveness and efficiency of the autonomous driving system test.

[0123] like Figure 2 As shown, the present invention implements a method for generating dangerous scenarios for autonomous driving through an autonomous driving dangerous scenario generation system based on adversarial reward training, including a scenario generation module, a scenario real-time risk reward evaluation module, a pre-training module, a historical action penalty module, and a reinforcement learning training module. The following is a separate introduction to the structure of each module and the structural connection relationship;

[0124] 1. Scene Generation Module

[0125] This module is used for basic scene settings, such as Figure 2 As described, the test scene in this embodiment includes two vehicles: one is the target vehicle, which is the autonomous driving vehicle being tested, and the other is the attack vehicle controlled by this technical solution, which is the challenge vehicle. The main functions of this module include initializing the driving simulation environment, setting the starting position of the vehicle, setting the parameters of the vehicle movement, and ensuring the diversity and controllability of the scene. Through this setting, the action of the attack vehicle can specifically test the reaction ability and driving strategy of the target vehicle, thereby revealing the performance of the autonomous driving system when facing sudden driving challenges.

[0126] (II) Scenario-based real-time risk-reward evaluation module

[0127] Based on the scene generation module, a coordinate system is established in the simulation environment and vehicle data is collected in real time; the scene state in the vehicle data is expressed as the lateral and longitudinal position and speed of each vehicle [x, y, v x ,v y], and the action of the challenge car is the longitudinal acceleration and the steering wheel angle [acc,angle]. The scenario real-time risk reward evaluation module introduces a reward function to encourage the challenge vehicle to perform dangerous test behaviors based on the estimated collision time between the target car and the challenge car. The specific process is as follows:

[0128] (1) The collision time prediction unit quantitatively describes the scene risk based on the collision time (TTC). The TTC calculation method is:

[0129]

[0130] Where TTC is the estimated time to collision, v vel is the relative speed between the target car and the challenge car; a rel is the relative speed between the target car and the challenge car, d re l is the relative distance.

[0131] (2) Risk Assessment Unit: This unit focuses on how to use time to collision (TTC) to assess the dangerousness of a scenario and the value of the test. TTC, or time to collision, is a measure of the length of time it takes for two vehicles to collide relative to each other. In the testing of autonomous driving systems, TTC is a key indicator used to assess the dangerousness of a scenario.

[0132] When the TTC value is very small, it means that a collision is about to happen, and the autonomous vehicle under test may not have enough time to take effective collision avoidance measures. Although such scenarios are dangerous, they are not very valuable for testing the strategy and reaction capabilities of the autonomous driving system because the time left for the system to react is extremely short.

[0133] On the contrary, when the TTC value is large, the autonomous vehicle has more time to take preventive measures and most operations can be ensured to be safe. Such scenarios are also of limited value in testing the emergency handling capabilities of the system.

[0134] In order to overcome the above problems and optimize the testing process, this technical solution introduces a specific reward function that aims to increase the proportion of scenarios with medium TTC values. Such scenarios are neither situations where emergency collision avoidance is almost impossible nor situations that can be easily handled with sufficient time. They are more suitable for evaluating the decision-making and action efficiency of the autonomous driving system in possible collision situations.

[0135]

[0136] As shown in the above formula, for TTC values ​​close to collision or safe operation range (i.e., very small or maximum TTC), the reward value is low because these scenarios have limited testing value for driving strategies. For scenarios with medium TTC values, the reward value is higher to encourage the generation of such more challenging scenarios that are more critical to testing autonomous driving performance. With such a reward function setting, the system can generate and test scenarios that can truly test the reaction and strategy formulation capabilities of the autonomous driving system more frequently. As shown in the formula:

[0137] When TTC<0, r TTC = exp(TTC)-2; This formula function is used to simulate the rapid response after a collision or the emergency handling of past events; the response value is rapidly increased by the exponential function exp(TTC), because a negative TTC means that a collision is very close or inevitable. This function rises quickly, indicating the severity of the emergency.

[0138] When 0≤TTC<1.6s, r TTC =TTC-0.6, within this range, r TTC The value of increases linearly with the increase of TTC. If TTC is close to 0, r TTC It may be a negative value, which means there is still a high risk of collision; when TTC is close to 1.6 seconds, the value is close to the maximum positive value of 1, which means it is relatively safe. That is, this interval is a response to an impending collision. TTC values ​​in this range mean that the vehicle needs to take immediate measures (such as braking or avoiding) to prevent a collision. Subtracting 0.6 from the formula is to adjust the baseline of the reward so that the response only starts to become positive when TTC approaches 1.6 seconds.

[0139] When 1.6s≤TTC<3.6s, r TTC =-0.2(TTC-2.6) 2 +1.2, this interval provides the maximum reward and is the ideal reaction time for collision prevention. The function reaches its maximum value (1.2) at TTC = 2.6 seconds, encouraging the driving system to maintain or adjust the speed within this time window to maintain a safe distance, that is, the ideal safety response time, which reaches its maximum value when it is close to 2.6 seconds, thereby controlling the attacking car to initiate confrontational actions against the autonomous vehicle according to the above strategy.

[0140] When TTC ≥ 3.6s, r TTC =2(TTC-2.6) -1 -1, in this case, rTTC increases linearly with the increase of TTC, indicating increasing safety, and the reward also increases accordingly, encouraging maintaining or increasing the distance from the vehicle in front.

[0141] Therefore, each formula in this unit aims to guide the autonomous driving system to take the most appropriate action to optimize safety and driving efficiency by rewarding or punishing different TTC situations.

[0142] (III) Pre-training module

[0143] The purpose of the pre-training module is to provide initial training data and action strategies for the reinforcement learning model under the condition of no historical action similarity penalty. The main implementation steps are to set up a multi-lane road in a simulation environment and configure the target vehicle and the challenge vehicle. Simulate and collect state parameters such as the lateral and longitudinal position and speed of each vehicle in different traffic scenarios. Use the Deep Deterministic Policy Gradient (DDPG) algorithm to randomly generate the actions of the attacking vehicle (longitudinal acceleration and steering wheel angle) without considering the influence of historical behavior. Finally, evaluate the effect of the generated action, that is, the immediate reward and long-term value brought by the action under specific conditions. The main feature is to use the advantages of the DDPG algorithm, that is, the processing capability of the continuous action space, to improve the efficiency and accuracy of pre-training. The exploratory nature of the initial training phase is ensured without historical similarity penalty, providing the model with a variety of behavioral strategies.

[0144] (IV) Historical Action Penalty Module

[0145] In the generation of dangerous test scenarios for autonomous driving, it is a challenge to ensure the diversity of the scenarios while maintaining their value for practical applications. Traditional reinforcement learning methods tend to optimize specific performance indicators when designing reward mechanisms, which often results in the generated scenarios being too simple in type and strategy to fully evaluate the responsiveness of the autonomous driving system. In addition, directly using diversity as part of the reward may cause the action selection to deviate from the actual optimal strategy, thereby reducing the effectiveness and practicality of the test.

[0146] To solve the above problems, this module proposes a historical action neighborhood penalty method, which aims to increase the diversity of generated scenes without sacrificing the quality of the scenes. The core idea of ​​this method is to penalize actions that are too similar to historical actions through a penalty function, thereby encouraging the generation of innovative action choices to avoid repeatedly generating overly similar scenes. This method allows the system to explore a wider range of action spaces while maintaining local optimality and generate richer and more diverse test scenes.

[0147] That is to say, mainly considering the optimal characteristics of reinforcement learning, if the rewards in the scene real-time risk-reward evaluation module are directly used for training, the scenes will converge to highly similar situations, and the scene diversity will be seriously lacking, which cannot meet the requirements for testing the performance of autonomous driving systems. Existing research methods attempt to improve scene diversity by explicitly incorporating scene diversity indicators into the reward function. However, this will cause the actions under the new reward function to deviate from expectations, lose their original optimality, and even generate a large number of invalid scenes. Based on this, this scheme proposes a historical action neighborhood penalty method. The idea is that for a new segment, its action should be different from the actions of similar historical states that have been generated, while also ensuring local optimality. We use a penalty function method to impose a relatively large penalty on the historical action neighborhood, and the penalty function is:

[0148]

[0149] (1) Historical action set: For each new state-action pair (s, a), we first retrieve the set of all actions taken under the same state s from the historical database {a i These historical actions represent the vehicle’s past behavior patterns in similar situations.

[0150] (2) Calculate the action difference: for the current action a and each historical action a i , and calculate the difference between them. The difference measure used here is the absolute value difference |aa i |, intuitively reflects the degree of proximity between the two actions in terms of magnitude.

[0151] (3) Accumulated difference value: All the difference values ​​are accumulated to form a total penalty signal. The accumulated value is converted into a penalty value through a negative sign, that is, the more similar the action is to the historical action, the greater the accumulated difference (penalty).

[0152] (4) Applying penalty: The total penalty value R(s,a) is used to adjust the original reward function, which actually encourages the vehicle to explore new and different behavioral strategies by reducing the rewards for current actions that are too similar to historical actions.

[0153] (V) Reinforcement Learning Training Module

[0154] In the context of considering action differences and historical behaviors. This reinforcement learning training module uses the results of pre-training and further optimizes the strategy through a deep reinforcement learning algorithm to achieve the purpose of generating more diverse and challenging test scenarios in a simulated environment. The key is to use the penalty of historical action similarity and the single-step difference reward to enhance the quality and efficiency of the learning process, thereby improving the ability of the autonomous driving system to cope with various potentially dangerous situations. The basic idea is that diversity is not a reward and does not need to pursue the maximum value, but the action difference needs to meet a certain threshold. If this state has selected an action different from the historical trajectory, then the subsequent state will have a different goal from the historical trajectory, improving diversity. Even if it may converge to the historical trajectory, the above process can ensure that it deviates again. In general, the diversity reward only needs to act on the current state and does not affect the subsequent state value. We use the deep reinforcement learning algorithm framework for improvement. The new Q' is defined as:

[0155] Q'(s t ,a t )=R HASP (s t ,a t )+Q(s t ,a t );

[0156] Q'(s t ,a t ) comes from the training results without the historical action neighborhood penalty, as Figure 2 The reward function for training reinforcement learning is the reward function of the scenario real-time risk reward evaluation module. The goal of the value network is to determine Q'(s) under the current state and given penalty action. t ,a t ), and the goal of the action network is to make Q'(s t ,a t ) is maximized. To eliminate the influence of fixed historical trajectories, we use a random penalty action method during training. That is, for any penalty action a, the algorithm can determine the optimal action. When the training is officially completed and run, the historical actions are input. Since the input dimension is only increased by a few actions, the feasibility of training is relatively high.

[0157] The method for diverse dangerous test scenarios of reinforcement learning for autonomous driving considering dynamic adversarial rewards proposed by the present invention overcomes the problem of unsolvable scenarios and non-collision scenarios caused by collision terminal rewards through real-time dangerous rewards, solves the problem of explicit diversity reward scenario loss through historical action neighborhood penalties, and achieves diverse results without repeated multiple training through single-step differential rewards. Moreover, experiments prove that the scenarios generated by the present invention greatly reduce the proportion of unsolvable scenarios and generate non-collision dangerous scenarios. At the same time, the present invention improves scenario diversity with only two rounds of training, while greatly reducing the loss of scenario quality.

[0158] To specifically illustrate the specific usage process in this embodiment, the following makes a specific description in combination with the specific algorithm training and usage process:

[0159] Algorithm Training and Usage Process

[0160] Establish a training environment

[0161] for episodes = 1:Max do

[0162] Reset the environment

[0163] while t < t_max or not collision do

[0164] Obtain the state observation s0

[0165] Obtain the penalty action a pen (Randomly generated during training, and all historical actions in the historical state action library are used to find similar states when generating scenarios)

[0166] Put s0, a pen Input into the action network to obtain the action a[acc, angle]

[0167] Put s0, a pen and a into the historical action domain penalty algorithm to obtain the similarity penalty r

[0168] If it is a training record s0, a pen Record s0, a, r into the replay pool B. If it is scenario generation, record s0, a into the historical state action library

[0169] The environment executes the action a

[0170] end while

[0171] if episodes % training cycle == 0 do

[0172] Randomly take out batch_size groups of data from B

[0173] Put s0, a pen, a is input into the evaluation network, and the state action value q1 considering the historical penalty is obtained. pen , a is input into the evaluation network of the pre-trained reinforcement learning algorithm to obtain the state action value q0 without historical penalty

[0174] Calculate the difference between the expected state action value q0+r and the estimated value q1, update the evaluation network and update the action network

[0175] end if

[0176] end for

[0177] As shown in the above process, a training environment is established, a multi-lane road simulation environment is initialized, and a target vehicle (the autonomous driving vehicle under test) and a challenge vehicle (the attack vehicle) are configured.

[0178] Training loop: For each training episode, the following steps are performed:

[0179] (1) Before starting each new episode, reset the simulation environment to its initial state.

[0180] (2) At each time step t, the following sub-steps are run continuously until the maximum time t_max is reached or a collision occurs:

[0181] (3) Obtain state observation s0, which is the current state from the simulation environment, including information such as the position and speed of all vehicles.

[0182] (4) Get penalty action a pen During training, penalty actions are randomly generated to explore new behavior patterns; during scene generation, all historical actions in similar states in the historical state-action library are used.

[0183] (5) The state observation s0 and the penalty action a pen Input into the action network and calculate the corresponding action parameters a (acceleration and steering angle).

[0184] (6) Input s, apen and a into the historical action domain penalty algorithm to obtain the similarity penalty r.

[0185] (7) Execute the calculated action a and record the data to the appropriate location according to the scene type. The training data enters the playback pool B, and the generated scene data is recorded in the historical state action library.

[0186] (8) Whenever the specified training cycle is reached, a batch of data is randomly selected from the replay pool B, and the sampled data is input into the evaluation network and the pre-trained reinforcement learning algorithm evaluation network to obtain the state-action value q1 considering historical penalties and the state-action value q0 without historical penalties, respectively. The difference between the expected state-action value q0+r and the estimated value q1 is calculated, and this error is used to update the evaluation network and action network to improve the strategy and improve the prediction accuracy.

[0187] Through the above process, it is possible to effectively train and generate diverse and challenging autonomous driving test scenarios, enhancing the autonomous driving system's ability to cope with complex environments.

[0188] Example 4

[0189] In order to show the implementation process and technical details of the present invention in more detail, the following content will be explained in combination with the actual system configuration and operation steps. This embodiment specifically describes how to test and verify the behavior of autonomous driving vehicles by building a multi-lane driving simulation environment, especially the responsiveness when facing potential dangerous scenarios. In addition, this embodiment also details the reinforcement learning algorithms used in this environment, including the configuration, training and implementation process of the algorithms, and how to effectively generate and evaluate dangerous driving scenarios through these algorithms, thereby improving the overall safety and reliability of the autonomous driving system.

[0190] (I) Environment settings:

[0191] A two-lane simulation environment was established, which included a tested autonomous vehicle and an attack vehicle controlled by the system. IDM (Intelligent Driver Model) and MOBIL (Minimizing Overall Braking Induced by Lane Changes) algorithms were used for longitudinal and lateral control respectively, and the basic standards for the operation of autonomous vehicles were established. At the same time, the vehicle's range of action was defined, including maximum acceleration, corner limit and maximum speed, as well as how to deal with situations beyond the boundaries. For example, the maximum acceleration is 5m / s 2 , the turning angle is limited to ±15°, and the maximum speed is limited to 30m / s. The state of the environment is expressed as the lateral and longitudinal position and velocity [x, y, v x ,v y ], the action space of the controlled attack vehicle is the longitudinal acceleration and the steering wheel angle [acc, angle], and the action range is the same as that of the autonomous vehicle. If the attack vehicle exceeds the boundary, the current scene ends and enters the next round, and the boundary is penalized in the reward function.

[0192] (II) Application of reinforcement learning algorithm

[0193] The deep deterministic policy gradient (DDPG) method is used to train the reinforcement learning model without considering the similarity penalty of historical actions. The reinforcement learning model includes an action network, a target action network, a state value network, and a target state value network. Each network consists of two fully connected layers, each with 64 parameters, using RELU and tanh activation functions. The loss function of the state value network is calculated by the following formula:

[0194]

[0195] y t = r(s t ,a t )+γQ'(s t+1 ,A'(s t+1 )|θ Q );

[0196] That is, the loss function L(θ Q ) is used to train the Critic network (value network) in DDPG, whose goal is to minimize the predicted state-action value Q(s t ,a t |θ Q ) and the actual target state-action value y t The error between the two states. And the loss calculation is based on a series of sampled transitions from the environment (state s t , action a t , reward r, and next state s t+ 1).

[0197] Target state-action value y t It is calculated according to the Bellman equation and is used to predict future rewards and train the value network. Specifically, it includes the immediate reward r(s t ,a t ) and the discounted future reward, which is based on the target action A'(s t+1 )|θ Q and the predicted future value calculated by the target value network Q'.

[0198] The loss function L(θ Q ) and the target value y t In the DDPG training process, the loss function guides the update of network weights by comparing the prediction of the value network with the target value calculated by the target network, and finally optimizes the entire strategy. This setting ensures the stability and efficiency of the learning process, and continuously improves the performance of the strategy by continuously adjusting the network parameters.

[0199] The action network is updated by the gradient of the state value network output value:

[0200]

[0201] The formula is the policy network parameter θ A The gradient update is used to maximize the action value function Q; in the formula, J is the objective function associated with the expected return. In a given state s t and action A(s t )|θ Q ), the action value function Q is related to the policy parameter θ A The gradient of θ. That is, find a policy parameter θ A So that in a given state, the action recommended by the strategy can maximize the value of the Q function.

[0202] And the For strategy p β Under the state expectation, the p β is a sampling strategy for efficient sampling in the experience replay buffer. In DDPG, this gradient update enables the policy network (that is, the action selection network) to optimize in the direction of improving long-term returns. The policy network learns to select actions that maximize the predicted future rewards (that is, Q values).

[0203] After each main network update, the target network is soft-updated (τ<<1): θ'=(1-τ)θ'+τθ; generally τ is a very small constant, which is used to stabilize the learning process and avoid training instability caused by too fast changes in the target network parameters.

[0204] The reward function of this round of training is the reward function of the scene real-time risk reward evaluation module, τ is 0.001, the discount coefficient of reinforcement learning is 0.9, and the number of training rounds is 150,000 rounds (based on one iteration of the scene). Then the algorithm proposed by the present invention is trained, with the above-mentioned single-step difference reward action evaluation module as the overall framework, and the action network and value network structure are consistent with the above-mentioned DDPG algorithm. Figure 1 and Figure 2 The training process was carried out for 400,000 rounds. The loss function and update method of the two neural networks are also the same as the above DDPG algorithm.

[0205] After the training, you can Figure 1 and Figure 2 When testing autonomous vehicles, similar historical scenes are judged as the difference in longitudinal distance between two vehicles in two scenes is less than 1m, the lateral distance is less than 1m, and the longitudinal speed is less than 2m / s. To avoid the problem of too many similar scenes and uncertain number of penalty actions, the maximum number of penalty actions recorded in the same similar scene is 5, and this maximum value is used to determine the network input parameter size. If the number of penalty actions is insufficient, the super-actionable action range value is used instead.

[0206] Although the present invention is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for generating dangerous scenarios for autonomous driving based on adversarial reward training, characterized in that: The following steps are involved: Step S1, initializing a driving simulation environment of a multi-lane road, configuring a target vehicle and a challenge vehicle, setting a control algorithm and parameters for the target vehicle to perform automatic driving; based on the estimated collision time between the target vehicle and the challenge vehicle, introducing a reward function to motivate the challenge vehicle to perform a dangerous test behavior, and establishing a coordinate system in the simulation environment and collecting vehicle data in real time; Step S2, based on the scene state parameters in the vehicle data, randomly generate penalty actions, convert the penalty actions into action parameters through action network transformation, form a state-action pair with the scene state parameters, the penalty actions and the corresponding action parameters, input them into the pre-trained reinforcement learning model, and calculate the state-action value without historical action similarity penalty; Step S3, according to the similarity between the current action and the historical action of the challenge vehicle, a historical action similarity penalty is imposed on the penalty action through a penalty function to avoid repeated dangerous test behaviors; Inputting the state-action pair into a reinforcement learning model, and calculating the state-action value after the historical action similarity penalty is updated; Step S4, taking the sum of the state action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state action value, and updating the reinforcement learning model according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of an autonomous driving dangerous scenario.

2. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: Establishing a coordinate system in the simulation environment includes: The coordinate system is established by setting a designated point on the road centerline as the origin, the road centerline as the longitudinal axis, and setting a transverse axis passing through the origin and perpendicular to the road centerline.

3. The method for generating dangerous scenarios for autonomous driving according to claim 2, characterized in that: Based on the estimated collision time between the target vehicle and the challenge vehicle, introducing a reward function to motivate the challenge vehicle to perform a dangerous test behavior includes: The estimated collision time is determined based on the scene state parameters of the target vehicle and the challenge vehicle, and the scene risk is quantitatively evaluated using the formula: Wherein, TTC is the estimated collision time, v vel is the relative speed between the target vehicle and the challenge vehicle; a re l is the relative speed between the target vehicle and the challenge vehicle, d rel is the relative distance; According to the estimated collision time, the reward function is introduced to motivate the challenge vehicle to perform the dangerous test behavior, and the formula is: Among them, r TTC It is a reward value based on the estimated collision time, used to optimize the driving strategy of the challenge vehicle.

4. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: Based on the scene state parameters in the vehicle data, a penalty action is randomly generated, the penalty action is converted into an action parameter through action network transformation, the scene state parameters, the penalty action and the corresponding action parameters form a state-action pair, which is input into the pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated, including: Determine and collect the scene state parameters of the target vehicle and the challenge vehicle at each time step; the scene state parameters include transverse and longitudinal positions, horizontal and vertical speeds; Defining the scope and type of the penalty action, and randomly generating the penalty action of the challenge vehicle; Outputting the corresponding action parameters of the scene state parameters and the penalty action through the action network; The scene state parameters, the penalty action and the corresponding action parameters form the state-action pair, which is input into the pre-trained reinforcement learning model, and the state-action value is obtained through the value network in the pre-trained reinforcement learning model.

5. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: According to the similarity between the current action and the historical action of the challenging vehicle, the historical action similarity penalty is imposed on the penalty action by a penalty function, including: Creating a database for each of the scene state parameters, for storing all historical actions under the scene state parameters; After the challenge vehicle takes the random action under the current scene state parameters, the action record is added to the database; According to the current scene state parameters, all the historical actions are extracted from the database; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain a penalty value R(s,a), and the formula is: Where s is the current scene state parameter, a is the current action, and a i The historical actions, History(s, a) are all the historical actions under the current scene state parameters; n is the total number of historical actions.

6. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: Updating the reinforcement learning model according to the difference between the state action value and the expected state action value in the reinforcement learning model to guide the generation of the dangerous scenario of autonomous driving includes: The current scene state parameters and the current action are obtained, and combined with the action parameters recommended by the action network to form the state-action pair, which are input into the pre-trained reinforcement learning model and the reinforcement learning model respectively; The state-action value obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model; Calculate the penalty value of the historical action similarity penalty according to the state-action pair; The state-action value and the penalty value in the pre-trained reinforcement learning model are combined as the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

7. A system for generating dangerous scenarios for autonomous driving based on adversarial reward training, comprising: A scenario generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, and set the control algorithm and parameters for the target vehicle to perform autonomous driving; A scenario real-time risk reward evaluation module is used to introduce a reward function to motivate the challenge vehicle to perform dangerous test behaviors based on the estimated collision time between the target vehicle and the challenge vehicle, and to establish a coordinate system in the simulation environment and collect vehicle data in real time; A pre-training module, for randomly generating penalty actions based on the scene state parameters in the vehicle data, converting the penalty actions into action parameters through action network transformation, forming a state-action pair with the scene state parameters, the penalty actions and the corresponding action parameters, inputting the state-action pairs into a pre-trained reinforcement learning model, and calculating the state-action value without a historical action similarity penalty; A historical action penalty module is used to perform historical action similarity penalty on the penalty action through a penalty function according to the similarity between the current action and the historical action of the challenge vehicle, so as to avoid repeated dangerous test behaviors; Reinforcement learning training module; Used to input the state-action pair into the reinforcement learning model, calculate the state-action value after the historical action similarity penalty is updated; take the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state-action value, and update the reinforcement learning model according to the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of dangerous scenarios for autonomous driving.

8. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: The scenario real-time risk reward evaluation module includes: A collision time prediction unit is used to determine the expected collision time based on the scene state parameters of the target vehicle and the challenge vehicle, and to quantitatively evaluate the scene risk, and the formula is: Wherein, TTC is the estimated collision time, v vel is the relative speed between the target vehicle and the challenge vehicle; a rel is the relative speed between the target vehicle and the challenge vehicle, d rel is the relative distance; The risk assessment unit is used to introduce the reward function to motivate the challenge vehicle to perform the dangerous test behavior according to the expected collision time, and the formula is: Among them, r TTC A reward value based on the estimated collision time, used to optimize the driving strategy of the challenge vehicle; A reward function generating unit is used to dynamically generate a reward or penalty signal based on the risk assessment result to motivate the challenge vehicle to take a specified action to deal with the dangerous situation.

9. The method for generating dangerous scenarios for autonomous driving according to claim 1, characterized in that: The historical action penalty module includes: A database management unit, used for creating a database for each of the scene state parameters, for storing all historical actions under the scene state parameters; An action recording and updating unit, after the challenge vehicle takes the random action under the current scene state parameters, adds the action record to the database; The historical action retrieval penalty unit extracts all the historical actions from the database according to the current scene state parameters; for each historical action, the difference between the current action and the historical action is calculated, and the difference is accumulated to obtain a penalty value R(s, a), which is: Where s is the current scene state parameter, a is the current action, and a i The historical actions, History(s, a) are all the historical actions under the current scene state parameters; n is the total number of historical actions.

Images