A method and system for generating dangerous scenarios for autonomous driving based on adversarial reward training

Abstract

This invention relates to the field of autonomous driving testing, and provides a method and system for generating hazardous autonomous driving scenarios based on adversarial reward training. The method initializes a multi-lane road driving simulation environment; randomly generates penalty actions based on scene state parameters from vehicle data; pairs the scene state parameters, penalty actions, and corresponding action parameters into state-action pairs; calculates the state-action value under the condition of no historical action similarity penalty; applies a historical action similarity penalty to the penalty actions using a penalty function, and calculates the updated state-action value; and updates the reinforcement learning model based on the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of hazardous autonomous driving scenarios. This invention significantly improves the efficiency and quality of autonomous driving system testing, while reducing scene quality loss, and enhances the diversity and practicality of scenarios within a shorter training cycle.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving testing, and in particular to a method and system for generating dangerous autonomous driving scenarios based on adversarial reward training. Background Technology

[0002] To improve the safety and reliability of autonomous driving, comprehensive testing is essential before its official deployment. The primary goal is to identify any potential malfunctions or hazards. Simulation testing, due to its efficiency and low cost, is a major component of autonomous driving testing. Simulation testing consists of numerous scenarios, defined as the process by which a test vehicle interacts with other traffic participants in a simulated traffic environment within a specified time period to achieve its driving intentions. Hazardous scenarios directly demonstrate the capabilities of autonomous driving and therefore play a central role in simulation testing. Due to the rarity of safety-critical events in the real world, relying solely on real-world databases to collect them is insufficient. Therefore, generating effective hazardous scenarios for autonomous driving testing is an important research direction.

[0003] Currently, researchers have applied various principles and methods to scene generation. Among them, online scene generation methods based on deep reinforcement learning have become a major research direction due to their advantage of being more challenging because they can change the scene state in real time based on the feedback of the test subjects. In this field, researchers have conducted in-depth research on scene modeling, reward settings, event probabilities, multi-vehicle scenes, and training convergence.

[0004] Despite the many achievements made in this field, two major problems remain in terms of practical results:

[0005] (1) The limitations of the adversarial reward mechanism are mainly reflected in the fact that the reward is mainly based on collisions, ignoring the risk assessment in the process. This leads to the generation of overly simplified test scenarios, which only focus on collisions and cannot reflect other types of dangers, such as near-collision situations. This bias is not conducive to the realistic assessment of the complexity of autonomous driving systems.

[0006] (2) The application of deep reinforcement learning faces challenges in improving the diversity of test scenarios. Although attempts have been made to address this issue by introducing diversity metrics into the reward function, this approach may disrupt the original optimal action strategy, resulting in generated scenarios that are neither realistic nor possess the expected testing value. Furthermore, the need for repeated training makes this method inefficient in practice and difficult to meet the demands of rapid iteration. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned technical problems by proposing a method and system for generating hazardous scenarios for autonomous driving based on adversarial reward training. This method replaces traditional collision rewards with real-time scenario risk assessment to avoid generating unsolvable scenarios and effectively overcomes the degradation of scenario quality and the need for repetitive training caused by existing diverse methods. This significantly improves the efficiency and quality of autonomous driving testing. The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides a method for generating hazardous scenarios for autonomous driving based on adversarial reward training, comprising the following steps:

[0009] Step S1: Initialize the driving simulation environment of the multi-lane road and configure the target vehicle and the challenge vehicle. Set the control algorithm and parameters for the target vehicle to perform autonomous driving. Based on the expected collision time of the target vehicle and the challenge vehicle, introduce a reward function to incentivize the challenge vehicle to perform dangerous test behaviors. At the same time, establish a coordinate system in the simulation environment and collect vehicle data in real time.

[0010] Step S2: Based on the scene state parameters in the vehicle data, a penalty action is randomly generated. The penalty action is converted into action parameters through an action network. The scene state parameters, penalty actions, and corresponding action parameters are combined into a state-action pair and input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated.

[0011] Step S3: Based on the similarity between the current action and the historical action of the challenge vehicle, a penalty function is used to penalize the penalty action based on the similarity of the historical action to avoid repeated dangerous testing behaviors; the state-action pair is input into the reinforcement learning model to calculate the state-action value after the historical action similarity penalty is updated.

[0012] Step S4: The sum of the state action value under the condition of no historical action similarity penalty and the penalty value of historical action similarity penalty is used as the expected state action value. The reinforcement learning model is updated according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of autonomous driving dangerous scenarios.

[0013] Furthermore, establishing a coordinate system in the simulation environment includes:

[0014] Establish a coordinate system by setting a specified point on the road centerline as the origin, using the road centerline as the longitudinal axis, and setting a transverse axis that passes through the origin and is perpendicular to the road centerline.

[0015] Furthermore, based on the predicted collision times of the target vehicle and the challenge vehicle, a reward function is introduced to incentivize the challenge vehicle to perform hazardous test behaviors, including:

[0016] The estimated collision time is determined based on the scenario state parameters of the target vehicle and the challenge vehicle, and the scenario risk is quantitatively assessed using the following formula:

[0017]

[0018] Where TTC is the estimated time of collision, v vel The relative speeds of the target car and the challenge car; a rel The relative speeds of the target car and the challenge car, d re l represents the relative distance;

[0019] Based on the expected collision time, a reward function is introduced to incentivize the challenge vehicle to perform dangerous test behaviors. The formula is:

[0020]

[0021] Where, r TTC The reward value, based on the estimated collision time, is used to optimize the driving strategy of the challenge vehicle.

[0022] Furthermore, based on the scene state parameters in the vehicle data, a penalty action is randomly generated. This penalty action is then converted into action parameters using an action network. The scene state parameters, the penalty action, and the corresponding action parameters are combined into a state-action pair, which is then input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated, including:

[0023] Determine and collect the scene state parameters of the target vehicle and the challenge vehicle at each time step; the scene state parameters include lateral and longitudinal positions, horizontal and vertical velocities;

[0024] Define the scope and type of penalty actions, and randomly generate penalty actions for the challenge car;

[0025] The scene state parameters and penalty actions are output as corresponding action parameters through the action network;

[0026] The scene state parameters, penalty actions, and corresponding action parameters are combined to form state-action pairs, which are then input into a pre-trained reinforcement learning model. The state-action value is obtained through the value network in the pre-trained reinforcement learning model.

[0027] Furthermore, based on the similarity between the challenge vehicle's current action and its historical actions, a penalty function is used to apply a historical action similarity penalty to the penalty action, including:

[0028] Create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0029] After the challenge vehicle takes a random action under the current scene state parameters, it updates the database and adds the action record.

[0030] Based on the current scene state parameters, all historical actions are extracted from the database; for each historical action, the difference between the current action and the historical actions is calculated, and the differences are accumulated to obtain the penalty value R(s,a), as shown in the formula:

[0031]

[0032] Where s is the current scene state parameter, a is the current action, and a i History (s, a) represents all historical actions under the current scene state parameters; n is the total number of historical actions.

[0033] Furthermore, the reinforcement learning model is updated based on the difference between the state-action value in the reinforcement learning model and the expected state-action value, guiding the generation of hazardous scenarios for autonomous driving, including:

[0034] The current scene state parameters and current action will be obtained and combined with the action parameters recommended by the action network to form a state-action pair, which will be input into the pre-trained reinforcement learning model and the reinforcement learning model respectively.

[0035] Based on the state-action values ​​obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model;

[0036] The penalty value for similarity penalty based on state-action pairs is calculated.

[0037] The state-action value and penalty value in the pre-trained reinforcement learning model are combined into the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

[0038] Based on the same inventive concept, this invention also provides an autonomous driving hazardous scenario generation system based on adversarial reward training, comprising,

[0039] The scene generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target car and the challenge car, and set the control algorithm and parameters for the target car to achieve autonomous driving.

[0040] The real-time risk reward evaluation module is used to incentivize the challenge vehicle to perform dangerous test behaviors based on the expected collision time of the target vehicle and the challenge vehicle by introducing a reward function. At the same time, a coordinate system is established in the simulation environment and vehicle data is collected in real time.

[0041] The pre-training module is used to randomly generate penalty actions based on scene state parameters in vehicle data. The penalty actions are converted into action parameters through an action network. The scene state parameters, penalty actions, and corresponding action parameters are combined into state-action pairs and input into the pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated.

[0042] The historical action penalty module is used to apply a penalty function to the penalty action based on the similarity between the current action and the historical actions of the challenge vehicle, thereby avoiding repeated dangerous testing behaviors.

[0043] The reinforcement learning training module is used to input state-action pairs into the reinforcement learning model, calculate the state-action value after updating with historical action similarity penalty, and use the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of historical action similarity penalty as the expected state-action value. The reinforcement learning model is updated based on the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of autonomous driving dangerous scenarios.

[0044] Furthermore, the real-time risk reward assessment module for the scenario includes:

[0045] The collision time prediction unit is used to determine the estimated collision time based on the scenario state parameters of the target vehicle and the challenge vehicle, and to quantitatively assess the scenario risk. The formula is as follows:

[0046]

[0047] Where TTC is the estimated time of collision, v vel The relative speeds of the target car and the challenge car; a rel The relative speeds of the target car and the challenge car, d rel It is a relative distance;

[0048] The risk assessment unit is used to incentivize the challenge vehicle to perform hazardous test behaviors based on the expected collision time using a reward function. The formula is as follows:

[0049]

[0050] Where, r TTC The reward value, based on the estimated collision time, is used to optimize the driving strategy of the challenge vehicle.

[0051] The reward function generation unit is used to dynamically generate reward or penalty signals based on the risk assessment results, incentivizing challenge vehicles to take specified actions to deal with dangerous situations.

[0052] Furthermore, the historical action punishment module includes:

[0053] The database management unit is used to create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0054] The motion recording and update unit updates the database and adds motion records after the challenge vehicle takes a random action under the current scene state parameters.

[0055] The historical action retrieval penalty unit extracts all historical actions from the database based on the current scene state parameters. For each historical action, it calculates the difference between the current action and the historical actions, accumulates the differences, and obtains the penalty value R(s, a), as shown in the formula:

[0056]

[0057] Where s is the current scene state parameter, a is the current action, and a i History actions, History(s, a) represents all historical actions under the current scene state parameters; n is the total number of historical actions.

[0058] Compared with the prior art, the present invention has at least one of the following technical advantages:

[0059] This invention significantly improves the efficiency and quality of autonomous driving system testing, while reducing scene quality loss. It also enhances the diversity and practicality of scenarios within a shorter training cycle, providing strong support for the safety verification and functional optimization of autonomous driving technology.

[0060] (1) The present invention effectively overcomes the problems of unsolvable scenarios and non-collision scenarios that may be caused by traditional collision final state rewards through a real-time danger reward mechanism. At the same time, this reward mechanism not only improves the practicality of the scenario, but also enhances the comprehensiveness of the test.

[0061] (2) This invention employs a historical action neighborhood penalty strategy to address the potential degradation in scene quality caused by explicit diversity rewards. By penalizing overly similar actions, it ensures scene diversity and richness, thereby enabling a more comprehensive test of the autonomous driving system's responsiveness. Finally, single-step difference rewards are used to achieve diverse results through multiple training sessions without repetition. Furthermore, the scenes generated by this invention significantly reduce the proportion of unsolvable scenes and generate non-collision hazard scenes. Experiments demonstrate that this invention improves scene diversity with only two rounds of training, while significantly reducing scene quality loss. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below:

[0063] Figure 1 This is a flowchart of the steps of the autonomous driving dangerous scene generation method based on adversarial reward training of the present invention;

[0064] Figure 2 This is a schematic diagram illustrating the working principle of this embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0066] Example 1

[0067] In existing technologies, the safety verification of autonomous vehicles is conducted through simulation testing, in which generating challenging test scenarios is crucial. However, current simulation testing primarily relies on collision scenarios to evaluate the autonomous driving system's response. While this method is direct, it cannot comprehensively reflect the various dangerous situations that may be encountered in real-world driving. Specifically, on the one hand, the aggressive vehicle behavior controlled by collision rewards aims only to collide with the test vehicle as quickly as possible, such as rear-ending or cutting in at extremely close range. This may lead to unmanageable scenarios, which is detrimental to improving autonomous driving performance. On the other hand, it cannot generate non-collision hazard scenarios, which should also be considered and prevented by autonomous driving systems.

[0068] Furthermore, while existing deep reinforcement learning methods can dynamically generate test scenarios, their adversarial reward designs often focus solely on collision outcomes, neglecting the risk variations during the process. This results in a lack of diversity in the generated scenarios and an inability to cover non-collision but equally dangerous situations. Specifically, deep reinforcement learning tends to converge to the optimal solution, leading to a lack of diversity in the generated scenarios. Existing methods attempt to improve scenario diversity by explicitly incorporating diversity metrics into the reward function. However, such methods may cause actions to lose their original optimality or even trigger extreme actions, resulting in ineffective scenarios. Moreover, existing methods for improving diversity require retraining based on previously generated scenarios, which is both time-consuming and impractical. These limitations significantly reduce the practical value of the generated scenarios.

[0069] Based on these considerations, this application proposes a novel method for generating hazardous test scenarios for autonomous driving. This method aims to overcome the limitations of existing technologies that rely solely on collision results by comprehensively considering real-time hazard assessments that take into account both collision and non-collision factors. This method employs a dynamic adversarial reward mechanism, focusing not only on whether a collision occurs but also on the vehicle's behavioral patterns and choices before approaching a collision, to more realistically simulate potential hazardous situations. Simultaneously, by introducing historical action similarity penalties and single-step difference reward strategies, this method effectively improves the diversity and practicality of the scenarios, avoiding the degradation of scenario quality and waste of computational resources caused by overtraining. The specific implementation is as follows:

[0070] like Figure 1 As shown, this invention provides a method for generating dangerous scenarios for autonomous driving based on adversarial reward training, including the following steps:

[0071] Step S1: Initialize the driving simulation environment of the multi-lane road and configure the target vehicle and the challenge vehicle. Set the control algorithm and parameters for the target vehicle to perform autonomous driving. Based on the expected collision time of the target vehicle and the challenge vehicle, introduce a reward function to incentivize the challenge vehicle to perform dangerous test behaviors. At the same time, establish a coordinate system in the simulation environment and collect vehicle data in real time.

[0072] Step S2: Based on the scene state parameters in the vehicle data, a penalty action is randomly generated. The penalty action is converted into action parameters through an action network. The scene state parameters, penalty actions, and corresponding action parameters are combined into a state-action pair and input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated.

[0073] Step S3: Based on the similarity between the current action and the historical action of the challenge vehicle, a penalty function is used to penalize the penalty action based on the similarity of the historical action to avoid repeated dangerous testing behaviors; the state-action pair is input into the reinforcement learning model to calculate the state-action value after the historical action similarity penalty is updated.

[0074] Step S4: The sum of the state action value under the condition of no historical action similarity penalty and the penalty value of historical action similarity penalty is used as the expected state action value. The reinforcement learning model is updated according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of autonomous driving dangerous scenarios.

[0075] Furthermore, establishing a coordinate system in the simulation environment includes:

[0076] Establish a coordinate system by setting a specified point on the road centerline as the origin, using the road centerline as the longitudinal axis, and setting a transverse axis that passes through the origin and is perpendicular to the road centerline.

[0077] Furthermore, based on the predicted collision times of the target vehicle and the challenge vehicle, a reward function is introduced to incentivize the challenge vehicle to perform hazardous test behaviors, including:

[0078] The estimated collision time is determined based on the scenario state parameters of the target vehicle and the challenge vehicle, and the scenario risk is quantitatively assessed using the following formula:

[0079]

[0080] Where TTC is the estimated time of collision, v vel The relative speeds of the target car and the challenge car; a rel The relative speeds of the target car and the challenge car, d re l represents the relative distance;

[0081] Based on the expected collision time, a reward function is introduced to incentivize the challenge vehicle to perform dangerous test behaviors. The formula is:

[0082]

[0083] Where, r TTC The reward value, based on the estimated collision time, is used to optimize the driving strategy of the challenge vehicle.

[0084] Furthermore, based on the scene state parameters in the vehicle data, a penalty action is randomly generated. This penalty action is then converted into action parameters using an action network. The scene state parameters, the penalty action, and the corresponding action parameters are combined into a state-action pair, which is then input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated, including:

[0085] Determine and collect the scene state parameters of the target vehicle and the challenge vehicle at each time step; the scene state parameters include lateral and longitudinal positions, horizontal and vertical velocities;

[0086] Define the scope and type of penalty actions, and randomly generate penalty actions for the challenge car;

[0087] The scene state parameters and penalty actions are output as corresponding action parameters through the action network;

[0088] The scene state parameters, penalty actions, and corresponding action parameters are combined to form state-action pairs, which are then input into a pre-trained reinforcement learning model. The state-action value is obtained through the value network in the pre-trained reinforcement learning model.

[0089] Furthermore, based on the similarity between the challenge vehicle's current action and its historical actions, a penalty function is used to apply a historical action similarity penalty to the penalty action, including:

[0090] Create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0091] After the challenge vehicle takes a random action under the current scene state parameters, it updates the database and adds the action record.

[0092] Based on the current scene state parameters, all historical actions are extracted from the database; for each historical action, the difference between the current action and the historical actions is calculated, and the differences are accumulated to obtain the penalty value R(s,a), as shown in the formula:

[0093]

[0094] Where s is the current scene state parameter, a is the current action, and a i History (s, a) represents all historical actions under the current scene state parameters; n is the total number of historical actions.

[0095] Furthermore, the reinforcement learning model is updated based on the difference between the state-action value in the reinforcement learning model and the expected state-action value, guiding the generation of hazardous scenarios for autonomous driving, including:

[0096] The current scene state parameters and current action will be obtained and combined with the action parameters recommended by the action network to form a state-action pair, which will be input into the pre-trained reinforcement learning model and the reinforcement learning model respectively.

[0097] Based on the state-action values ​​obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model;

[0098] The penalty value for similarity penalty based on state-action pairs is calculated.

[0099] The state-action value and penalty value in the pre-trained reinforcement learning model are combined into the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

[0100] Example 2

[0101] Based on the same inventive concept, this invention also provides an autonomous driving hazardous scenario generation system based on adversarial reward training, comprising,

[0102] The scene generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target car and the challenge car, and set the control algorithm and parameters for the target car to achieve autonomous driving.

[0103] The real-time risk reward evaluation module is used to incentivize the challenge vehicle to perform dangerous test behaviors based on the expected collision time of the target vehicle and the challenge vehicle by introducing a reward function. At the same time, a coordinate system is established in the simulation environment and vehicle data is collected in real time.

[0104] The pre-training module is used to randomly generate penalty actions based on scene state parameters in vehicle data. The penalty actions are converted into action parameters through an action network. The scene state parameters, penalty actions, and corresponding action parameters are combined into state-action pairs and input into the pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the state-action value is calculated.

[0105] The historical action penalty module is used to apply a penalty function to the penalty action based on the similarity between the current action and the historical actions of the challenge vehicle, thereby avoiding repeated dangerous testing behaviors.

[0106] The reinforcement learning training module is used to input state-action pairs into the reinforcement learning model, calculate the state-action value after updating with historical action similarity penalty, and use the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of historical action similarity penalty as the expected state-action value. The reinforcement learning model is updated based on the difference between the state-action value in the reinforcement learning model and the expected state-action value to guide the generation of autonomous driving dangerous scenarios.

[0107] Furthermore, the real-time risk reward assessment module for the scenario includes:

[0108] The collision time prediction unit is used to determine the estimated collision time based on the scenario state parameters of the target vehicle and the challenge vehicle, and to quantitatively assess the scenario risk. The formula is as follows:

[0109]

[0110] Where TTC is the estimated time of collision, v vel The relative speeds of the target car and the challenge car; a rel The relative speeds of the target car and the challenge car, d re l represents the relative distance;

[0111] The risk assessment unit is used to incentivize the challenge vehicle to perform hazardous test behaviors based on the expected collision time using a reward function. The formula is as follows:

[0112]

[0113] Where, r TTC The reward value, based on the estimated collision time, is used to optimize the driving strategy of the challenge vehicle.

[0114] The reward function generation unit is used to dynamically generate reward or penalty signals based on the risk assessment results, incentivizing challenge vehicles to take specified actions to deal with dangerous situations.

[0115] Furthermore, the historical action punishment module includes:

[0116] The database management unit is used to create a database for each scene state parameter to store all historical actions under the scene state parameter;

[0117] The motion recording and update unit updates the database and adds motion records after the challenge vehicle takes a random action under the current scene state parameters.

[0118] The historical action retrieval penalty unit extracts all historical actions from the database based on the current scene state parameters. For each historical action, it calculates the difference between the current action and the historical actions, accumulates the differences, and obtains the penalty value R(s, a), as shown in the formula:

[0119]

[0120] Where s is the current scene state parameter, a is the current action, and a i History actions, History(s, a) represents all historical actions under the current scene state parameters; n is the total number of historical actions.

[0121] Example 3

[0122] To more clearly demonstrate the technical details and applications of this invention, the following content will describe a specific exemplary embodiment in conjunction with a concrete system configuration and implementation process. This embodiment will illustrate how this invention solves the problems existing in the prior art and improves the effectiveness and efficiency of autonomous driving system testing by describing the generation process of autonomous driving hazardous test scenarios, the technical modules involved and their interactions, and how to practically apply the technical solution of this invention.

[0123] like Figure 2 As shown, this invention realizes the method of generating dangerous scenes for autonomous driving through an autonomous driving dangerous scene generation system based on adversarial reward training. It includes a scene generation module, a scene real-time risk reward evaluation module, a pre-training module, a historical action penalty module, and a reinforcement learning training module. The following is a separate introduction to the structure of each module and the structural connection relationship.

[0124] (I) Scene Generation Module

[0125] This module is used for basic scene settings, such as... Figure 2 The test scenario in this embodiment includes two vehicles: one is the autonomous driving vehicle being tested, i.e., the target vehicle, and the other is the attack vehicle controlled by this technical solution, i.e., the challenge vehicle. The main functions of this module include initializing the driving simulation environment, setting the starting position of the vehicles, setting the parameters of vehicle movement, and ensuring the diversity and controllability of the scenario. Through this setup, the actions of the attack vehicle can specifically test the reaction capabilities and driving strategies of the target vehicle, thereby revealing the performance of the autonomous driving system when facing unexpected driving challenges.

[0126] (II) Real-time Risk Reward Assessment Module

[0127] The scene generation module establishes a coordinate system in the simulation environment and collects vehicle data in real time; the scene state in the vehicle data is represented by the lateral and longitudinal positions and velocities of each vehicle [x, y, v]. x ,v yThe target vehicle's actions are longitudinal acceleration and steering wheel angle [acc, angle]. The real-time risk reward evaluation module introduces a reward function to incentivize the challenge vehicle to perform dangerous test behaviors based on the expected collision time between the target vehicle and the challenge vehicle. The specific process is as follows:

[0128] (1) Collision Time Prediction Unit: Based on Collision Time (TTC), this unit quantitatively describes the scenario risk. The TTC calculation method is as follows:

[0129]

[0130] Where TTC is the estimated time of collision, v vel The relative speeds of the target car and the challenge car; a rel The relative speeds of the target car and the challenge car, d re l represents the relative distance.

[0131] (2) Risk Assessment Unit: This unit focuses on how to use Time-to-Collision (TTC) to assess the hazard level and testing value of a scenario. TTC, or Time-to-Collision, is a measurement of the time it is expected that a collision will occur when two vehicles are in relative motion. In the testing of autonomous driving systems, TTC is a key indicator used to assess the hazard of a scenario.

[0132] When the TTC value is very low, it means a collision is imminent, and the tested autonomous vehicle may not have enough time to take effective avoidance measures. Although such scenarios are dangerous, they are not very valuable for evaluating the strategy and reaction capabilities of autonomous driving systems because the time allotted for the system to react is extremely short.

[0133] Conversely, when the TTC value is high, autonomous vehicles have more time to take preventative measures, and the vast majority of operations can be performed safely. Such scenarios are also of limited value for testing the emergency response capabilities of the system.

[0134] To overcome the aforementioned problems and optimize the testing process, this technical solution introduces a specific reward function designed to increase the proportion of scenarios with moderate TTC values. These scenarios are neither situations where emergency collision avoidance is almost impossible, nor situations that can be easily handled with sufficient time; they are more suitable for evaluating the decision-making and action efficiency of autonomous driving systems in potential collision situations.

[0135]

[0136] As shown in the formula above, for TTC values ​​close to the collision or safe operating range (i.e., minimal or maximum TTC), the reward value is lower because these scenarios have limited testing value for driving strategies. For scenarios with moderate TTC values, the reward value is higher to incentivize the generation of such more challenging scenarios that are more critical for testing autonomous driving performance. Through this reward function setting, the system can more frequently generate and test scenarios that truly test the autonomous driving system's reaction and strategy-making capabilities. As shown in the formula:

[0137] When TTC < 0, r TTC =exp(TTC)-2; This formula function is used to simulate a rapid response after a simulated collision or emergency handling of past events; the response value is rapidly increased by the exponential function exp(TTC), because a negative TTC means that a collision was very close or unavoidable. The rapid increase of this function indicates the severity of the emergency.

[0138] When 0 ≤ TTC < 1.6s, r TTC =TTC-0.6, within this range, r TTC The value of r increases linearly with increasing TTC. If TTC approaches 0, r... TTC A negative TTC value indicates a high risk of collision; while a TTC value close to 1.6 seconds, nearing its maximum positive value of 1, signifies relative safety. This range represents the response to an impending collision. A TTC value within this range means the vehicle needs to take immediate action (such as braking or swerving) to prevent a collision. The subtraction of 0.6 in the formula adjusts the baseline of the reward, ensuring the response only becomes aggressive when the TTC approaches 1.6 seconds.

[0139] When 1.6s ≤ TTC < 3.6s, r TTC = -0.2(TTC-2.6) 2 +1.2, this interval provides the maximum reward and is the ideal collision prevention reaction time. The function reaches its maximum value (1.2) at TTC = 2.6 seconds, encouraging the driving system to maintain or adjust speed within this time window to maintain a safe distance, i.e., the ideal safe response time, which reaches its maximum value close to 2.6 seconds, thereby controlling the attacking vehicle to launch an adversarial action against the autonomous vehicle according to the above strategy.

[0140] When TTC ≥ 3.6s, r TTC =2(TTC-2.6) -1 -1. In this case, rTTC increases linearly with the increase of TTC, indicating that it is becoming safer, and the reward increases accordingly, encouraging the maintenance or increase of the distance from the vehicle in front.

[0141] Therefore, each formula in this unit is designed to guide the autonomous driving system to take the most appropriate action by rewarding or penalizing different TTC situations in order to optimize safety and driving efficiency.

[0142] (III) Pre-training Module

[0143] The purpose of the pre-training module is to provide initial training data and action strategies for the reinforcement learning model by leveraging the absence of historical action similarity penalties. Its main implementation steps involve setting up a multi-lane road in a simulation environment and configuring target and challenge vehicles. State parameters such as lateral and longitudinal positions and speeds of each vehicle are simulated and collected in different traffic scenarios. Using the Deep Deterministic Policy Gradient (DDPG) algorithm, actions (longitudinal acceleration and steering wheel angle) of the attacking vehicle are randomly generated, without considering the influence of historical behavior. Finally, the effectiveness of the generated actions is evaluated, i.e., the immediate reward and long-term value brought by the actions in a specific state. The main feature is the advantage of using the DDPG algorithm, namely its ability to handle continuous action spaces, improving the efficiency and accuracy of pre-training. The absence of historical similarity penalties ensures the exploratory nature of the initial training phase, providing the model with diverse behavioral strategies.

[0144] (iv) Historical Action Punishment Module

[0145] In generating hazardous test scenarios for autonomous driving, ensuring scenario diversity while maintaining their practical application value is a challenge. Traditional reinforcement learning methods tend to optimize specific performance metrics when designing reward mechanisms, often resulting in generated scenarios that are too homogeneous in type and strategy, failing to comprehensively evaluate the responsiveness of autonomous driving systems. Furthermore, directly incorporating diversity as part of the reward may cause action selection to deviate from the actual optimal strategy, thereby reducing the effectiveness and practicality of the test.

[0146] To address the aforementioned issues, this module proposes a historical action neighborhood penalty method, which aims to increase the diversity of generated scenarios without sacrificing scenario quality. The core idea of ​​this method is to penalize actions that are too similar to historical actions through a penalty function, thereby encouraging the generation of innovative action choices and avoiding the repetition of overly similar scenarios. This approach allows the system to explore a wider action space and generate richer and more diverse test scenarios while maintaining local optima.

[0147] In other words, considering the optimal characteristics of reinforcement learning, directly using the rewards from the real-time risk reward evaluation module for training would lead to scenarios converging to highly similar situations, resulting in a severe lack of scenario diversity and failing to meet the performance requirements for testing autonomous driving systems. Existing research methods attempt to improve scenario diversity by explicitly incorporating scenario diversity metrics into the reward function. However, this causes actions under the new reward function to deviate from expectations, losing their original optimality and even generating a large number of invalid scenarios. Based on this, this proposal suggests a historical action neighborhood penalty method. The idea is that for a new segment, its action should be different from actions in similar historical states that have already been generated, while also ensuring local optimality. We use a penalty function method to impose a relatively large penalty on the historical action neighborhood, and the penalty function is:

[0148]

[0149] (1) Historical Action Set: For each new state-action pair (s, a), first retrieve the set of all actions {a, b, c} taken in the same state s from the historical database. i These historical actions represent past vehicle behavior patterns in similar situations.

[0150] (2) Calculate the action difference: for the current action a and every historical action a i Calculate the difference between them. The difference measure used here is the absolute difference |aa|. i | It intuitively reflects the degree of similarity between two actions in terms of magnitude.

[0151] (3) Accumulate the difference value: Accumulate all the difference values ​​to form the total penalty signal. This accumulated value is converted into a penalty value through a negative sign, that is, the more similar the action is to the historical action, the greater the accumulated difference (penalty).

[0152] (4) Applying penalties: The total penalty value R(s,a) is used to adjust the original reward function. In fact, by reducing the reward for current actions that are too similar to historical actions, the vehicle is encouraged to explore new and different behavioral strategies.

[0153] (V) Enhanced Learning and Training Module

[0154] In the context of considering action differences and historical behavior, this reinforcement learning training module uses pre-trained results and further optimizes the policy through deep reinforcement learning algorithms to generate more diverse and challenging test scenarios in a simulated environment. The key lies in leveraging penalties for historical action similarity and single-step difference rewards to enhance the quality and efficiency of the learning process, thereby improving the autonomous driving system's ability to cope with various potentially dangerous situations. The basic idea is that diversity is not a reward and does not need to pursue a maximum value; rather, action differences need to meet a certain threshold. If a state has already chosen an action different from the historical trajectory, then subsequent states will target different objectives than the historical trajectory, increasing diversity. Even if it might converge to the historical trajectory, the above process ensures that it deviates again. In general, the diversity reward only applies to the current state and does not affect subsequent state values. We use a deep reinforcement learning algorithm framework for improvement. The new Q' is defined as:

[0155] Q'(s t ,a t ) = R HASP (s t ,a t )+Q(s t ,a t );

[0156] Q'(s t ,a t The results come from training without historical action neighborhood penalties, as... Figure 2 The reward function shown for training reinforcement learning is the reward function of the scene real-time risk reward evaluation module. The goal of the value network is to determine the current state and Q'(s) under a given penalty action. t ,a t The goal of the action network is to make Q'(s) t ,a t To maximize the impact of fixed historical trajectories, we employ a randomized penalty action method during training. That is, for any penalty action 'a', the algorithm can determine the optimal action. When training is complete and running, the historical actions are input. Since the input dimension only increases by a few actions, training feasibility is relatively high.

[0157] The method for diverse dangerous test scenarios of reinforcement learning autonomous driving considering dynamic adversarial rewards proposed by the present invention overcomes the problem of unsolvable scenarios and non-collision scenarios caused by collision terminal rewards through real-time danger rewards, solves the problem of explicit diversity reward scenario loss through historical action neighborhood penalties, and achieves diverse results without repeated multiple training through single-step differential rewards. Moreover, experiments prove that the scenarios generated by the present invention greatly reduce the proportion of unsolvable scenarios and generate non-collision dangerous scenarios. At the same time, the present invention improves scenario diversity with only two rounds of training, while greatly reducing the loss of scenario quality.

[0158] To specifically illustrate the specific usage process in this embodiment, the following is a specific description in combination with the specific algorithm training and usage process:

[0159] Algorithm Training and Usage Process

[0160] Establish a training environment

[0161] for episodes = 1:Max do

[0162] Reset the environment

[0163] while t < t_max or not collision do

[0164] Obtain the state observation s0

[0165] Obtain the penalty action a pen (Randomly generated during training, and all historical actions in the historical state action library are used to find similar states when generating scenarios)

[0166] Put s0, a pen Input into the action network to obtain the action a[acc, angle]

[0167] Put s0, a pen and a into the historical action domain penalty algorithm to obtain the similarity penalty r

[0168] If it is a training record, put s0, a pen and a, r into the replay pool B. If it is scenario generation, record s0, a into the historical state action library

[0169] The environment executes the action a

[0170] end while

[0171] if episodes % training cycle == 0 do

[0172] Randomly take out batch_size groups of data from B

[0173] Put s0, a penInputting 'a' into the evaluation network, we obtain the state-action value q1 considering historical penalties, and then inputting 's0' and 'a' into the evaluation network. pen In the evaluation network of the pre-trained reinforcement learning algorithm, the value q0 of the state-action without historical penalty is obtained.

[0174] Calculate the difference between the expected state action value q0+r and the estimated value q1, and update the evaluation network and action network accordingly.

[0175] end if

[0176] end for

[0177] As shown in the above process, a training environment is established, a multi-lane road simulation environment is initialized, and the target vehicle (the autonomous vehicle under test) and the challenge vehicle (the attack vehicle) are configured.

[0178] Training cycle: For each training cycle (episode), perform the following steps:

[0179] (1) Before starting each new episode, reset the simulation environment to its initial state.

[0180] (2) At each time step t, continue running the following sub-steps until the maximum time t_max is reached or a collision occurs:

[0181] (3) Obtain state observation s0, obtain the current state from the simulation environment, including information such as the position and speed of all vehicles.

[0182] (4) Obtain the penalty action a pen During training, penalty actions are randomly generated to explore new behavioral patterns; during scene generation, all historical actions in similar states from the historical state action library are used.

[0183] (5) Combine the state observation s0 and the penalty action a pen The input is fed into the motion network, and the corresponding motion parameters a (acceleration and steering angle) are calculated.

[0184] (6) Input s, apen and a into the historical action domain penalty algorithm to obtain the similarity penalty r.

[0185] (7) Execute the calculated action a, and record the data in the appropriate location according to the scene type. The training data is put into the playback pool B, and the generated scene data is recorded in the historical state action library.

[0186] (8) Whenever a specified training period is reached, a batch of data is randomly selected from the replay pool B. The sampled data is input into the evaluation network and the pre-trained reinforcement learning algorithm evaluation network. The state-action value q1 considering historical penalties and the state-action value q0 without historical penalties are obtained respectively. The difference between the expected state-action value q0+r and the estimated value q1 is calculated. This error is used to update the evaluation network and the action network to improve the strategy and increase the prediction accuracy.

[0187] Through the above process, diverse and challenging autonomous driving test scenarios can be effectively trained and generated, enhancing the autonomous driving system's ability to cope with complex environments.

[0188] Example 4

[0189] To illustrate the implementation process and technical details of this invention in more detail, the following description will be provided in conjunction with actual system configuration and operation steps. This embodiment specifically describes how to test and verify the behavior of autonomous vehicles, especially their response capabilities in the face of potentially dangerous scenarios, by constructing a multi-lane driving simulation environment. Furthermore, this embodiment details the reinforcement learning algorithms applied in this environment, including the configuration, training, and implementation processes of the algorithms, and how these algorithms can effectively generate and evaluate dangerous driving scenarios, thereby improving the overall safety and reliability of the autonomous driving system.

[0190] (I) Environment Setup:

[0191] A two-lane simulation environment was established, including a tested autonomous vehicle and an attack vehicle controlled by the system. IDM (Intelligent Driver Model) and MOBIL (Minimizing Overall Braking Induced by Lane Changes) algorithms were used for longitudinal and lateral control, respectively, establishing the basic standards for autonomous vehicle operation. The vehicle's range of motion was also defined, including maximum acceleration, turning angle limits, and maximum speed, as well as how to handle situations exceeding these boundaries. For example, the maximum acceleration was set to 5 m / s². 2 The turning angle is limited to ±15°, and the maximum speed is limited to 30 m / s. The environmental state is represented by the lateral and longitudinal positions and velocities of each vehicle [x, y, v]. x ,v y The controlled attack vehicle's action space is defined by its longitudinal acceleration and steering wheel angle [acc, angle], and its range of motion is the same as that of an autonomous vehicle. If the attack vehicle exceeds the boundary, the current scene ends and the next round begins, with a penalty applied in the reward function for exceeding the boundary.

[0192] (II) Application of Reinforcement Learning Algorithms

[0193] The reinforcement learning model is trained using the Deep Deterministic Policy Gradient (DDPG) method, eliminating the need to consider similarity penalties for historical actions. The reinforcement learning model comprises an action network, a target action network, a state-value network, and a target state-value network. Each network consists of two fully connected layers with 64 parameters, employing ReLU and tanh activation functions. The loss function for the state-value network is calculated using the following formula:

[0194]

[0195] y t =r(s t ,a t )+γQ'(s t+1 ,A'(s t+1 )|θ Q );

[0196] That is, the loss function L(θ) Q This is used to train the Critic network (value network) in DDPG, whose objective is to minimize the predicted state-action value Q(s). t ,a t |θ Q ) and the actual target state - action value y t The error between them. And this loss calculation is based on a series of transitions (states) sampled from the environment. t Action a t Reward r, and next state s t+ 1).

[0197] Target state - action value y t It is calculated according to the Bellman equation to predict future rewards and train the value network, specifically including the immediate reward r(s). t ,a t ) and discounted future rewards, the latter based on the target action A'(s) t+1 )|θ Q The predicted future value is calculated using the target value network Q'.

[0198] Loss function L(θ) Q ) and target value y t In the DDPG training process, these two functions complement each other. The loss function guides the update of network weights by comparing the predictions of the value network with the target value calculated by the target network, ultimately optimizing the entire policy. This setup ensures the stability and efficiency of the learning process, continuously improving policy performance by adjusting network parameters.

[0199] The action network updates the value using the gradient of the output value of the state-value network.

[0200]

[0201] The formula is the policy network parameter θ. A The gradient update is used to maximize the action-value function Q; in the formula, J is the objective function associated with the expected reward. Gradient Indicates that in a given state s t and action A(s) t )|θ Q Under the condition that the action value function Q is related to the policy parameter θ, the action value function Q is related to the policy parameter θ A The gradient. That is, finding a policy parameter θ. A This ensures that, given a state, the action recommended by this strategy maximizes the value of the Q function.

[0202] And the above For strategy p β The expectation of the next state, p β This sampling strategy is used for effective sampling within the experience replay buffer. In DDPG, this gradient update enables the policy network (i.e., the action selection network) to optimize in the direction of improving long-term rewards. The policy network learns to select actions that maximize the predicted future reward (i.e., the Q-value).

[0203] After each update of the main network, a soft update is performed on the target network (τ<<1): θ'=(1-τ)θ'+τθ; generally, τ is a very small constant used to stabilize the learning process and avoid training instability caused by the target network parameters changing too quickly.

[0204] The reward function for this round of training is the reward function of the scene real-time risk reward evaluation module, with τ set to 0.001 and the discount factor for reinforcement learning set to 0.9. The training rounds are 150,000 (based on one scene iteration). Subsequently, the algorithm proposed in this invention is trained, using the aforementioned single-step differential reward action evaluation module as the overall framework. The action network and value network structures remain consistent with the aforementioned DDPG algorithm, as referenced. Figure 1 and Figure 2 The process is trained for 400,000 epochs. The loss function and update method for these two neural networks are the same as those of the DDPG algorithm described above.

[0205] After the training is completed, you can proceed as follows Figure 1 and Figure 2 For testing autonomous vehicles, similar historical scenarios are defined as those where the difference in longitudinal distance between the two vehicles is less than 1 meter, the difference in lateral distance is less than 1 meter, and the difference in longitudinal speed is less than 2 m / s. To avoid an excessive number of identical similar scenarios and the resulting uncertainty in the number of penalty actions, the maximum number of penalty actions recorded for the same similar scenario is 5. This maximum value is used to determine the size of the network input parameters. If the number of penalty actions is insufficient, a range of actions exceeding the permissible action range is used instead.

[0206] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for generating hazardous scenarios for autonomous driving based on adversarial reward training, characterized in that, Includes the following steps: Step S1: Initialize the driving simulation environment of the multi-lane road and configure the target vehicle and the challenge vehicle. Set the control algorithm and parameters for the target vehicle to perform autonomous driving. Based on the expected collision time of the target vehicle and the challenge vehicle, introduce a reward function to incentivize the challenge vehicle to perform dangerous test behaviors. At the same time, establish a coordinate system in the simulation environment and collect vehicle data in real time. Specifically, based on the estimated collision time of the target vehicle and the challenge vehicle, a reward function is introduced to incentivize the challenge vehicle to perform dangerous test behaviors, including: The estimated collision time is determined based on the scenario state parameters of the target vehicle and the challenge vehicle, and the scenario risk is quantitatively assessed using the following formula: ； in, The estimated collision time, The relative speed between the target vehicle and the challenge vehicle; The relative speeds of the target vehicle and the challenge vehicle. It is a relative distance; Based on the predicted collision time, the reward function is introduced to incentivize the challenge vehicle to perform the dangerous test behavior, as shown in the formula: ； in, The reward value based on the estimated collision time is used to optimize the driving strategy of the challenge vehicle; Step S2: Based on the scene state parameters in the vehicle data, a penalty action is randomly generated. The penalty action is converted into action parameters through an action network. The scene state parameters, the penalty action, and the corresponding action parameters are combined into a state-action pair and input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the value of the state-action is calculated. Step S3: Based on the similarity between the current action and the historical action of the challenge vehicle, a penalty function is used to penalize the penalty action based on the similarity of the historical action to avoid repeating the dangerous test behavior; the state-action pair is input into the reinforcement learning model to calculate the value of the state-action after the historical action similarity penalty is applied. Step S4: The sum of the state action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty is taken as the expected state action value. The reinforcement learning model is updated according to the difference between the state action value in the reinforcement learning model and the expected state action value to guide the generation of autonomous driving dangerous scenarios.

2. The method for generating hazardous scenarios for autonomous driving according to claim 1, characterized in that, Establishing a coordinate system in the simulation environment includes: The coordinate system is established by setting a designated point on the road centerline as the origin, the road centerline as the longitudinal axis, and a transverse axis passing through the origin and perpendicular to the road centerline.

3. The method for generating hazardous scenarios for autonomous driving according to claim 1, characterized in that, Based on the scene state parameters in the vehicle data, a penalty action is randomly generated. This penalty action is then converted into action parameters using an action network. The scene state parameters, the penalty action, and the corresponding action parameters are combined to form a state-action pair, which is then input into a pre-trained reinforcement learning model. Under the condition of no historical action similarity penalty, the value of the state-action pair is calculated, including: The scene state parameters of the target vehicle and the challenge vehicle at each time step are determined and collected; the scene state parameters include lateral and longitudinal positions, horizontal and vertical velocities; Define the scope and type of the penalty action, and randomly generate the penalty action for the challenge vehicle; The scene state parameters and the penalty action are output through the action network, along with the corresponding action parameters. The scene state parameters, the penalty action, and the corresponding action parameters are combined to form the state-action pair, which is then input into the pre-trained reinforcement learning model. The state-action value is obtained through the value network in the pre-trained reinforcement learning model.

4. The method for generating hazardous scenarios for autonomous driving according to claim 1, characterized in that, Based on the similarity between the current action and the historical actions of the challenge vehicle, the penalty action is penalized for historical action similarity using a penalty function, including: A database is created for each of the scene state parameters to store all historical actions under the scene state parameters; After the challenge vehicle takes the random action under the current scene state parameters, it updates the database and adds the action record. Based on the current scene state parameters, all historical actions are extracted from the database; for each historical action, the difference between the current action and the historical actions is calculated, and the differences are accumulated to obtain a penalty value. The formula is: ； in, The current scene state parameters, For the current action, The historical actions, This refers to all historical actions under the current scene state parameters; n is the total number of historical actions.

5. The method for generating hazardous scenarios for autonomous driving according to claim 1, characterized in that, The reinforcement learning model is updated based on the difference between the state-action value in the reinforcement learning model and the desired state-action value, guiding the generation of dangerous autonomous driving scenarios, including: The current scene state parameters and the current action will be obtained and combined with the action parameters recommended by the action network to form the state-action pair, which will be input into the pre-trained reinforcement learning model and the reinforcement learning model respectively. The state-action value is obtained from the pre-trained reinforcement learning model and the value network in the reinforcement learning model; The penalty value for the similarity penalty of the historical actions is calculated based on the state action pair; The state-action value and the penalty value in the pre-trained reinforcement learning model are combined to form the expected state-action value, and the state-action value output by the reinforcement learning model is adjusted according to the expected state-action value to optimize the reinforcement learning model.

6. An autonomous driving hazardous scenario generation system based on adversarial reward training, comprising, The scene generation module is used to initialize the driving simulation environment of a multi-lane road, configure the target vehicle and the challenge vehicle, and set the control algorithm and parameters for the target vehicle to perform autonomous driving. The scenario real-time risk reward evaluation module is used to introduce a reward function to incentivize the challenge vehicle to perform dangerous test behaviors based on the expected collision time of the target vehicle and the challenge vehicle, while establishing a coordinate system in the simulation environment and collecting vehicle data in real time. The scenario-based real-time risk reward evaluation module includes: The collision time prediction unit is used to determine the estimated collision time based on the scenario state parameters of the target vehicle and the challenge vehicle, and to quantitatively assess the scenario risk, using the following formula: ； in, The estimated collision time, The relative speed between the target vehicle and the challenge vehicle; The relative speeds of the target vehicle and the challenge vehicle. It is a relative distance; The risk assessment unit is used to incentivize the challenge vehicle to perform the dangerous test behavior by introducing the reward function based on the expected collision time, as shown in the formula: ； in, The reward value based on the estimated collision time is used to optimize the driving strategy of the challenge vehicle; The reward function generation unit is used to dynamically generate reward or penalty signals based on the risk assessment results, so as to incentivize the challenge vehicle to take specified actions to deal with dangerous situations; The pre-training module is used to randomly generate penalty actions based on scene state parameters in the vehicle data, convert the penalty actions into action parameters through an action network, and input the scene state parameters, the penalty actions, and the corresponding action parameters into a state-action pair, which is then input into the pre-trained reinforcement learning model to calculate the state-action value under the condition of no historical action similarity penalty. The historical action penalty module is used to apply a penalty function to the penalty action based on the similarity between the current action and the historical actions of the challenge vehicle, thereby avoiding repeated dangerous test behaviors. A reinforcement learning training module is used to input the state-action pair into a reinforcement learning model, calculate the state-action value after updating with the historical action similarity penalty, use the sum of the state-action value under the condition of no historical action similarity penalty and the penalty value of the historical action similarity penalty as the expected state-action value, and update the reinforcement learning model to guide the generation of autonomous driving dangerous scenarios based on the difference between the state-action value in the reinforcement learning model and the expected state-action value.

7. The autonomous driving hazardous scene generation system according to claim 6, characterized in that, The historical action penalty module includes: The database management unit is used to create a database for each of the scene state parameters to store all historical actions under the scene state parameters; The action recording and updating unit updates the database and adds the action record after the challenge vehicle takes the random action under the current scene state parameters; The historical action retrieval and penalty unit extracts all historical actions from the database based on the current scene state parameters; for each historical action, it calculates the difference between the current action and the historical actions, and accumulates the differences to obtain a penalty value. The formula is: ； in, The current scene state parameters, For the current action, The historical actions, This refers to all historical actions under the current scene state parameters; n is the total number of historical actions.

Images