An automatic driving safety key scene generation method based on deep reinforcement learning

Abstract

The application discloses a kind of automatic driving safety key scene generation methods based on deep reinforcement learning.The method is: first, in Carla simulation simulator, the basic scene information of automatic driving is collected;Then based on the obtained basic scene information and the running state information of the measured vehicle, reinforcement learning is carried out, and the initial state of dynamic element is generated;Then the interaction of dynamic element and environment is reinforced learning, and the action sequence of dynamic element is obtained;According to the action sequence of dynamic element, the feedback information of the interaction of dynamic element and environment is obtained;Finally, the network parameters of policy network are adjusted based on the feedback information of interaction until the feedback information of interaction reaches the expectation, and the key safety scene is obtained.The safety key scene generated by the application has the advantages of low cost, high detection efficiency, strong diversity and being applicable to most roads.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving testing, and in particular to a method for generating key safety scenarios for autonomous driving based on deep reinforcement learning. Background Technology

[0002] Autonomous driving is one of the key technologies of modern automobiles. It can not only provide greater traffic convenience and driving comfort, but also help reduce the number of driving-related accidents.

[0003] Testing is essential for verifying the safety and robustness of autonomous driving systems. However, autonomous driving testing faces numerous challenges, such as the time cost of road testing and the safety of testing in extreme scenarios and dangerous conditions. According to research by the RAND Corporation, autonomous driving algorithms need to accumulate at least 17.7 billion kilometers of driving data to reach the level of human drivers. If a fleet of 100 autonomous driving test vehicles were deployed, conducting road tests 24 hours a day at an average speed of 40 kilometers per hour, it would take over 500 years to reach the target mileage—a virtually impossible task. Furthermore, the lack of corresponding traffic regulations and insurance claims mechanisms for autonomous driving restricts the large-scale implementation of road testing. Relying solely on limited open roads and intelligent connected vehicle testing zones is insufficient to meet the robust testing demand for autonomous vehicles. In contrast, virtual simulation testing offers rich scenarios, fast computation speed, high testing efficiency, low resource consumption, good repeatability, and can be embedded in various stages of vehicle development. Using virtual simulation to generate key safety scenarios for autonomous driving allows for rapid testing and verification of autonomous driving systems, revealing system weaknesses.

[0004] Existing methods for generating safety-critical scenarios include three types: data-driven methods, knowledge-based methods, and adversarial methods. Data-driven methods generate scenarios by collecting real-world data, which requires massive amounts of data and is therefore costly. Knowledge-based methods represent scenarios using parameterization and then combine scene elements such as traffic participants and roads according to heuristics or special rules to generate scenarios. This method generates a sufficient number of scenarios, but the detection efficiency is low, and these simple heuristics cannot fully capture the complexity and diversity of real-world traffic scenarios. Adversarial methods use adversarial generative frameworks to generate scenarios. Most existing adversarial generative frameworks are designed for specific basic scenarios such as pedestrian crossings and intersections, controlling dynamic objects, and the generated scenarios are relatively simple. Summary of the Invention

[0005] The purpose of this invention is to provide a method for generating safety-critical scenarios for autonomous driving based on deep reinforcement learning, which has high detection efficiency, diversity, and applicability to most roads.

[0006] The technical solution to achieve the purpose of this invention is: a method for generating key safety scenarios for autonomous driving based on deep reinforcement learning, comprising the following steps:

[0007] Step 1: In the Carla simulation simulator, collect basic scene information for autonomous driving;

[0008] Step 2: Based on the obtained basic scene information and the operating status information of the tested vehicle, perform reinforcement learning to generate the initial state of dynamic elements;

[0009] Step 3: Perform reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of dynamic elements;

[0010] Step 4: Obtain feedback information on the interaction between dynamic elements and the environment based on the action sequence of dynamic elements;

[0011] Step 5: Adjust the network parameters of the policy network based on the interactive feedback information until the interactive feedback information reaches the expected level, thus obtaining the critical security scenario.

[0012] Furthermore, the basic scene information for autonomous driving described in step 1, as described in the Carla simulation simulator, is collected as follows:

[0013] In the Carla simulation simulator, the built-in vehicle drives through the town, and uses the vehicle's camera sensors and depth sensors to collect basic scene information at fixed time intervals, including image data, dynamic feature information and static feature information.

[0014] Furthermore, the basic scene information of the reinforcement learning input state mentioned in step 2 includes a top view of the scene; the test vehicle's operating state information includes the test vehicle's position coordinates, driving speed, and driving direction.

[0015] Furthermore, step 2 involves performing reinforcement learning based on the obtained basic scene information and the operating status information of the tested vehicle to generate the initial state of the dynamic elements, as detailed below:

[0016] Step 2.1: Generate prior map information. Based on the basic scene information and the operating status information of the vehicle under test, determine the areas with higher and lower probabilities in generating safety-critical scenes, and list buildings and flower bed areas as non-generable areas.

[0017] Step 2.2: Extract features from the top-down view of the basic scene using a convolutional neural network;

[0018] Step 2.3: Through the fully connected layer, combined with prior map information and scene top view feature information, generate the initial state information of dynamic elements, including initial position and motion state information.

[0019] Furthermore, step 3 involves performing reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of the dynamic elements, as detailed below:

[0020] Step 3.1: Using a convolutional neural network, extract features from a specific square region centered on the location of the dynamic element to prevent the dynamic element from deviating from the road or colliding with static elements, thereby generating a logically sound safety-critical scenario.

[0021] Step 3.2: Use LSTM to analyze the historical action sequences of dynamic elements to avoid dynamic elements generating past differentiated action sequences and ensure that the action sequences of dynamic elements are consistent with the real world.

[0022] Step 3.3: By combining feature extraction, historical action analysis, and the actions of the tested vehicle with a fully connected layer, the action of the dynamic element at the next time step is generated.

[0023] Step 3.4: The tested autonomous vehicle generates its position for the next time step through environmental perception, planning and decision-making, and control execution.

[0024] Step 3.5: Repeat steps 3.1 to 3.4 until the dynamic element collides with the vehicle under test, or until all time steps are completed, to obtain the action sequence of the dynamic element.

[0025] Furthermore, the feedback information obtained from the interaction between the dynamic elements and the environment based on the action sequence of the dynamic elements in step 4 is as follows:

[0026] Based on the action sequence of the dynamic elements, at each time step, interactive feedback information is generated based on the interaction between the dynamic elements, the tested vehicle, and the environment. This feedback information includes whether a collision occurred between the dynamic elements and the tested autonomous vehicle, and whether the dynamic elements produced non-compliant actions.

[0027] Furthermore, the network parameters of the policy network are adjusted based on the interactive feedback information as described in step 5 until the interactive feedback information reaches the expected level, as follows:

[0028] Each round contains several time steps, and at each time step, an interactive feedback is generated. Based on the interactive feedback, reinforcement learning methods are used to encourage dynamic elements to produce excellent actions and punish dynamic elements to produce poor actions by rewarding or punishing them, until the average feedback in each round gradually stabilizes and reaches the expected standard.

[0029] Furthermore, the reinforcement learning method described herein is as follows:

[0030] The optimization is performed using the policy gradient method of reinforcement learning. The policy in the policy gradient is represented by a network. By inputting different observations into the policy network, the policy network outputs different actions. The policy network determines the output of the action through the parameters in the policy network. The parameters of the policy network are trained to maximize the reward of the dynamic elements.

[0031] Furthermore, the parameters of the training policy network are configured to maximize the reward for dynamic elements, as detailed below:

[0032] Reinforcement learning is used to study how an agent can maximize the reward it can obtain in an uncertain environment. The process of reinforcement learning is actually the interaction between the agent and the environment. The agent obtains the state in the environment and then outputs an action based on its own decision.

[0033] Reinforcement learning is used to generate key safety scenarios for autonomous driving. These scenarios are dynamic, specifically a 60-second scenario from 8:00 to 8:01. Assuming reinforcement learning generates 10 actions per second for different input states, the entire scenario contains 600 time steps. Within each time step, the state is represented by s. t To represent an action, use 'a'. t To represent, use Υ t Let represent the state and action at time t, and let Υ represent the state and action for all time steps, as shown below:

[0034] Υ={s1,a1,s2,a2…,s t ,a t}

[0035] Because the environment is uncertain, within a time step t-1, the environment is performing the same action a. t-1 There is a probability p that generates different states s. t The agent adjusts the state s according to its own policy network. t There will be a probability p θ Producing different actions a t Let the parameters of the policy network be θ, therefore from Υ t-1 Generate Υ t The probability is:

[0036]

[0037] An agent receives different rewards for performing different actions in different states; therefore, at each time step, there is a reward denoted as R(Υ). tLet R represent the sum of all rewards obtained in the entire scenario. In reinforcement learning, the goal of the policy network θ is to obtain the maximum reward R. Because R is a random variable, its value varies depending on the policy θ and the environment. θ The expected value of the reward is expressed by the following formula:

[0038]

[0039] Finally, gradient ascent is used to maximize the reward. The gradient of the reward R in an environment Υ is expressed as:

[0040]

[0041] Furthermore, the key security scenarios described in step 5 are as follows:

[0042] The system visualizes the interaction between the tested vehicle and dynamic elements at each time step in the generated safety-critical scenario, providing an intuitive representation of the generated safety-critical scenario to determine whether it meets the expected standards, and retaining the safety-critical scenario that meets the expected standards.

[0043] Compared with the prior art, the significant advantages of this invention are: (1) the generated safety-critical scenarios are not limited to specific intersections and pedestrian crossings, but can be applied to most roads, making the safety-critical scenarios diverse; (2) by generating different pedestrian action sequences and locations, good collision rates are achieved in different scenarios; (3) basic scenario information is collected in the Carla simulation simulator, saving costs. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the overall process of the method for generating key safety scenarios for autonomous driving based on deep reinforcement learning, as described in this invention.

[0045] Figure 2 This is a schematic diagram illustrating the process of generating dynamic element positions in this invention.

[0046] Figure 3 This is a schematic diagram of the process for generating dynamic element actions in this invention.

[0047] Figure 4 This is a schematic diagram of the process for adjusting the strategy gradient in this invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0049] Combination Figure 1 This invention discloses a method for generating key safety scenarios for autonomous driving based on deep reinforcement learning, comprising the following steps:

[0050] Step 1: In the Carla simulation simulator, collect basic scene information for autonomous driving, as follows:

[0051] During the process of collecting basic autonomous driving scenario data, the basic autonomous driving scenario information is collected by using the built-in vehicle to drive in the town in the Carla 0.8.4 simulation simulator. The surrounding environment data is collected once after a certain time step. The CameraDepth depth camera collects the depth information of the surrounding environment, the CameraRGB conventional camera collects the image of the environment, and the CameraSeg semantic camera classifies different objects in the environment with different colors. The size of each collected image is 125*256. At the same time, the position coordinates of other dynamic elements during the built-in vehicle's driving in the town are recorded.

[0052] Step 2: Based on the obtained basic scene information and the operating status information of the tested vehicle, reinforcement learning is performed to generate the initial state of dynamic elements, as follows:

[0053] The basic scene information for reinforcement learning input states includes a top-down view of the scene (RGB image, semantic information, etc.), and the test vehicle's operating state information includes the test vehicle's position coordinates, speed, and direction of travel.

[0054] Based on the acquired basic scene information and the operating status information of the tested vehicle, reinforcement learning is performed to generate the initial state of dynamic elements, as follows:

[0055] Step 2.1: Generate prior map information. Based on the basic scene information and the operating status information of the vehicle under test, areas with a high and low probability of generating safety-critical scenarios can be determined. Since the vehicle under test is traveling forward on the map, the location for generating dynamic elements also needs to be in front of the vehicle under test to increase the probability of generating safety-critical scenarios. At the same time, to ensure the realism of the scene, a section directly in front of the vehicle under test is marked as a non-generable area, and areas such as buildings and flower beds are listed as non-generable areas.

[0056] Step 2.2: Extract depth, RGB, and semantic information features from the top-down view of the basic scene using a convolutional neural network. The convolutional neural network consists of two convolutional layers: the first convolutional layer has a kernel size of 3*3*1, and the second convolutional layer has a kernel size of 2*2*1. Two pooling layers are also included, each with a size of 2*2 and a stride of 1. After the convolutional operations, the ReLU activation function is used to provide the network with non-linear modeling capabilities. Finally, bilinear interpolation is used to resize the image to 128*256 pixels.

[0057] Step 2.3: Using a fully connected layer, combining prior map information and basic scene top-down view features, generate initial state information for dynamic elements, including initial position and motion state information. The output of the fully connected layer is a 128*256 matrix, where each point corresponds to a generated point on the map, specifically representing the probability of the location of the dynamic element in the safety-critical scenario.

[0058] Step 3: Perform reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of the dynamic elements, as follows:

[0059] Step 3.1: Using a convolutional neural network, extract features from a specific square region centered on the location of the dynamic feature, with sides of 25 pixels. The convolutional neural network also consists of two convolutional layers: the first convolutional layer has a kernel size of 3*3*1, and the second convolutional layer has a kernel size of 2*2*1. Two pooling layers are also used, each with a size of 2*2 and a stride of 1. After the convolutional operation, the ReLU activation function is used to provide the network with non-linear modeling capabilities. Finally, the image is resized to 25*25 pixels.

[0060] Step 3.2: Analyze the first 10 historical action sequences of the dynamic element using LSTM. LSTM is a neural network capable of remembering both short-term and long-term information; it is a special type of RNN. Compared to ordinary RNNs, LSTM performs better when processing longer sequences. Therefore, using LSTM can avoid overly differentiated action sequences for dynamic elements, such as the dynamic element moving left forward in the previous time step and right backward in the next time step, ensuring that the action sequences of dynamic elements are more consistent with the real world.

[0061] Step 3.3: By combining feature extraction, historical action analysis, and the actions of the tested vehicle with a fully connected layer, the action of the dynamic element at the next time step is generated. To simplify the problem, the moving speed of the dynamic element is fixed, and the network output action is the moving direction of the dynamic element. The moving direction of the dynamic element is represented by discrete values, namely, forward, backward, left, right, left-front, left-back, right-front, right-back, and a fixed 9 directions. The output will be the probability of choosing these 9 directions; the higher the probability, the more likely the dynamic element will move in that direction under certain circumstances.

[0062] Step 3.4: The tested vehicle generates its position for the next time step through environmental perception, planning and decision-making, and control execution.

[0063] Step 3.5: Repeat steps 3.1 to 3.4 until the dynamic element collides with the vehicle under test, or until all time steps are completed.

[0064] Step 4: Based on the action sequence of the dynamic elements, obtain the feedback information of the interaction between the dynamic elements and the environment, as follows:

[0065] Based on the action sequence of the dynamic elements, at each time step, interactive feedback information is generated based on the interaction between the dynamic elements, the tested vehicle, and the environment. This feedback information includes whether a collision occurred between the dynamic elements and the tested vehicle, and whether the dynamic elements produced non-compliant actions.

[0066] Step 5: Adjust the network parameters of the policy network based on the feedback information from the interaction until the feedback information meets the expectations, thus obtaining the key security scenario, as follows:

[0067] Each round contains several time steps, and at each time step, an interactive feedback is generated. Based on the interactive feedback, reinforcement learning methods are used to encourage dynamic elements to produce excellent actions and punish dynamic elements to produce poor actions by rewarding or punishing them, until the average feedback in each round gradually stabilizes and reaches the expected standard.

[0068] Furthermore, the reinforcement learning method described herein is as follows:

[0069] The optimization is performed using the policy gradient method of reinforcement learning. The policy in the policy gradient is represented by a network. By inputting different observations into the policy network, the policy network outputs different actions. The policy network determines the output of the action through the parameters in the policy network. The parameters of the policy network are trained to maximize the reward of the dynamic elements.

[0070] Furthermore, the parameters of the training policy network are configured to maximize the reward for dynamic elements, as detailed below:

[0071] In generating key safety scenarios for autonomous driving using reinforcement learning, the scenario is dynamic. For example, in a 60-second scenario from 8:00 to 8:01, assuming reinforcement learning generates 10 actions per second for different input states, the entire scenario contains 600 time steps. Within a certain time step, the state can be represented by s. t To represent, an action can be represented by an a. t To represent, use Υ t Let t represent the state and action at time t, and Υ represent the state and action at all time steps, as shown below:

[0072] Υ={s1,a1,s2,a2…,s t ,a t}

[0073] Because the environment is uncertain, within a time step t-1, the environment is in the same action (a t-1 There will be a probability (p) to generate different states s.t The agent adjusts the state s according to its own policy network. t There will be a probability (p) θ Different actions are produced. t Let the parameters of the policy network be θ, therefore from Υ t-1 Generate Υ t The probability is:

[0074]

[0075] An agent receives different rewards for performing different actions in different states; therefore, at each time step, there is a reward denoted as R(Υ). t Let R represent the sum of all rewards obtained in the entire scenario. In reinforcement learning, the goal of the policy network θ is to obtain the maximum reward R. Because R is a random variable, it will produce different values ​​depending on the policy θ and the environment. θ The expected value of the reward is expressed by the following formula:

[0076]

[0077] Finally, gradient ascent is used to maximize the reward. The gradient of the reward R in an environment Υ is expressed as:

[0078]

[0079] Furthermore, the specific key security scenarios described are as follows:

[0080] During the interaction between the tested vehicle and dynamic elements, parameters such as their position, speed, and direction of movement are recorded at each time step. After training, the collected parameters are loaded into the Carla simulation simulator to visualize the interaction process between the tested vehicle and dynamic elements in the environment at each time step. This provides an intuitive representation of the generated safety-critical scenarios, allowing for a determination of whether the expected standards have been met. Scenarios that meet the expected standards are then retained.

[0081] Example 1

[0082] Combination Figure 1 This embodiment provides a method for generating key safety scenarios for autonomous driving based on deep reinforcement learning. The reinforcement learning method is based on policy gradient. The dataset was collected in the Carla simulation simulator. The meaning of each symbol is shown in Table 1.

[0083] Table 1 Symbol Description

[0084] Υ Scene composition t Time step action action state state reward award p State transition equation episode round θ Parameters of the policy network w Image width h Image height Fc Fully connected layer LSTM Long Short-Term Memory Artificial Neural Network pos Position coordinates v speed

[0085] Combined with Table 1 Figure 1This embodiment presents a method for generating key safety scenarios for autonomous driving based on deep reinforcement learning, comprising the following steps:

[0086] Step 1: In the Carla simulation simulator, collect basic scene information for autonomous driving, as follows:

[0087] During the process of collecting basic autonomous driving scenario data, the information is collected by using the built-in vehicle to drive in the town in the Carla 0.8.4 simulation simulator. The surrounding environment data is collected once after a certain time step. The CameraDepth depth camera collects the depth information of the surrounding environment, the CameraRGB conventional camera collects the image of the environment, and the CameraSeg semantic camera classifies different objects in the environment with different colors. The size of each collected image is 125*256, where 125 is h, representing the height, and 256 is w, representing the width. At the same time, the position coordinates pos of other dynamic elements during the built-in vehicle's driving in the town are recorded.

[0088] Step 2: Based on the obtained basic scene information and the operating status information of the tested vehicle, reinforcement learning is performed to generate the initial state of dynamic elements, as follows:

[0089] The basic scene information for reinforcement learning input states includes a top-down view of the scene (RGB image, semantic information, etc.), and the test vehicle's operating state information includes the test vehicle's position coordinates, speed, and direction of travel.

[0090] Combination Figure 2 Based on the obtained basic scene information and the operating status information of the tested vehicle, reinforcement learning is performed to generate the initial state of dynamic elements, as follows:

[0091] Step 2.1: Generate prior map information. Based on the basic scene information and the operating status information of the vehicle under test, areas with a high and low probability of generating safety-critical scenarios can be determined. Since the vehicle under test is traveling forward on the map, the location for generating dynamic elements also needs to be in front of the vehicle under test to increase the probability of generating safety-critical scenarios. At the same time, to ensure the realism of the scene, a section directly in front of the vehicle under test is marked as a non-generable area, and areas such as buildings and flower beds are listed as non-generable areas.

[0092] Step 2.2: Extract depth, RGB, and semantic information features from the top-down view of the basic scene using a convolutional neural network. The convolutional neural network consists of two convolutional layers: the first convolutional layer has a kernel size of 3*3*1, and the second convolutional layer has a kernel size of 2*2*1. Two pooling layers are also included, each with a size of 2*2 and a stride of 1. After the convolutional operations, the ReLU activation function is used to provide the network with non-linear modeling capabilities. Finally, bilinear interpolation is used to resize the image to 128*256 pixels.

[0093] Step 2.3: Through the fully connected layer Fc, combined with prior map information and basic scene top view feature information, the initial state information of dynamic elements is generated, including initial position and motion state information. The output of the fully connected layer is the probability of the location of the dynamic element.

[0094] Step 3: Perform reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of the dynamic elements, and combine it with... Figure 3 The details are as follows:

[0095] Step 3.1: Extract features from a specific square region centered on the location of the dynamic feature, with sides of 25 pixels, using a convolutional neural network. The convolutional neural network also consists of two convolutional layers: the first convolutional layer has a kernel size of 3*3*1, and the second convolutional layer has a kernel size of 2*2*1. Two pooling layers are also used, each with a size of 2*2 and a stride of 1. After the convolutional operation, the ReLU activation function is used to provide the network with non-linear modeling capabilities. Finally, the image is resized to 25*25 pixels.

[0096] Step 3.2: Analyze the first 10 historical action sequences of the dynamic element using LSTM. LSTM is a neural network capable of remembering both short-term and long-term information; it is a special type of RNN. Compared to ordinary RNNs, LSTM performs better when processing longer sequences. Therefore, using LSTM can avoid overly differentiated action sequences for dynamic elements, such as the dynamic element moving left forward in the previous time step and right backward in the next time step, ensuring that the action sequences of dynamic elements are more consistent with the real world.

[0097] Step 3.3: By combining feature extraction, historical action analysis, and the actions of the tested vehicle with a fully connected layer, the action of the dynamic element in the next time step is generated. To simplify the problem, the moving speed of the dynamic element is fixed, and the network output action is the moving direction of the dynamic element. The moving direction of the dynamic element is represented by discrete values, namely, forward, backward, left, right, left-forward, left-back, right-forward, right-back, and a fixed 9 directions. The output will be the selection probability of these 9 directions; the higher the probability, the more likely the dynamic element will move in that direction under certain circumstances.

[0098] Step 3.4: The tested vehicle generates its position for the next time step through environmental perception, planning and decision-making, and control execution.

[0099] Step 3.5: Repeat steps 3.1 to 3.4 until the dynamic element collides with the vehicle under test, or until all time steps are completed.

[0100] Step 4: Based on the action sequence of the dynamic elements, obtain the feedback information of the interaction between the dynamic elements and the environment, as follows:

[0101] Based on the action sequence of the dynamic elements, at each time step, interactive feedback information is generated based on the interaction between the dynamic elements, the tested vehicle, and the environment. This feedback information includes whether a collision occurred between the dynamic elements and the tested vehicle, and whether the dynamic elements produced non-compliant actions.

[0102] Step 5: Adjust the network parameters of the policy network based on the feedback information from the interaction until the feedback information meets the expectations, thus obtaining the key security scenario, as follows:

[0103] Each round contains several time steps, and at each time step, an interactive feedback is generated. Based on the interactive feedback, reinforcement learning methods are used to encourage dynamic elements to produce excellent actions and punish dynamic elements to produce poor actions by rewarding or punishing them, until the average feedback in each round gradually stabilizes and reaches the expected standard.

[0104] Furthermore, combined Figure 4 The reinforcement learning method using policy gradients trains the parameters of the policy network to maximize the reward for dynamic elements, as detailed below:

[0105] The optimization is performed using the policy gradient method of reinforcement learning. The policy in the policy gradient is represented by a network. By inputting different observations into the policy network, the policy network will output different actions. The policy network determines the output of the action through the parameters in the network. Therefore, the main goal is to train these network parameters to enable the agent to obtain the most reward.

[0106] In generating key safety scenarios for autonomous driving using reinforcement learning, the scenario is dynamic; at a certain time step, the state can be represented by s. t To represent, an action can be represented by an a. t To represent, use Υ t Let t represent the state and action at time t, and Υ represent the state and action at all time steps, as shown below:

[0107] Υ={s1,a1,s2,a2…,s t ,at}

[0108] Because the environment is uncertain, within a time step t-1, the environment is in the same action (a t-1 There will be a probability (p) to generate different states s. t The agent adjusts the state s according to its own policy network. t There will be a probability (p) θ Different actions are produced. t Let the parameters of the policy network be θ, therefore from Υ t-1 Generate Υ t The probability is:

[0109]

[0110] An agent receives different rewards for performing different actions in different states; therefore, at each time step, there is a reward denoted as R(Υ). t Let R represent the sum of all rewards obtained in the entire scenario. In reinforcement learning, the goal of the policy network θ is to obtain the maximum reward R. Because R is a random variable, it will produce different values ​​depending on the policy θ and the environment. θ The expected value of the reward is expressed by the following formula:

[0111]

[0112] Finally, gradient ascent is used to maximize the reward. The gradient of the reward R in an environment Υ is expressed as:

[0113]

[0114] Furthermore, the specific key security scenarios described are as follows:

[0115] During the interaction between the tested vehicle and dynamic elements, parameters such as their position, speed, and direction of movement are recorded at each time step. After training, the collected parameters are loaded into the Carla simulation simulator to visualize the interaction process between the tested vehicle and dynamic elements in the environment at each time step. This provides an intuitive representation of the generated safety-critical scenarios, allowing for a determination of whether the expected standards have been met. Scenarios that meet the expected standards are then retained.

[0116] This embodiment was tested on a Linux platform with a 2.30GHz Intel Core i7-12700H processor and 16.0GB RAM. The operating system was Ubuntu 18.04, and the experimental environment consisted of Python version 3.7.7 and TensorFlow version 1.15.4.

[0117] Furthermore, this invention collected 120 different scenarios in the CARLA simulation software, each scenario containing 200 frames of scene information. The model was trained on 100 of these basic scenarios, and tested and validated on the remaining 20 scenarios. During training, each round contained 100 basic scenarios, with each scenario generated 10 times. If a collision occurred between the generated scenario and the test vehicle, it indicated that the generated scenario was valid and a safety-critical scenario. After 600,000 training iterations, the average collision rate reached approximately 90%, and the average collision rate during testing and validation exceeded 80%. This demonstrates that this invention is applicable to most basic scenarios.

[0118] This invention proposes a method for generating safety-critical scenarios for autonomous driving based on deep reinforcement learning. It combines the feature extraction capabilities of convolutional neural networks with the decision-making capabilities of reinforcement learning, adding a dynamic element to the basic scenario. This dynamic element includes its initial state information and actions at each time step. This generates a safety-critical scenario to verify the safety and robustness of the vehicle system under test. This method is applicable to most basic scenarios; therefore, combining it with different basic scenarios can effectively improve the diversity of generated safety-critical scenarios. Furthermore, the basic scenarios are collected from the highly realistic Carla simulation simulator, ensuring the realism of the generated safety-critical scenarios.

Claims

1. A method for generating key safety scenarios for autonomous driving based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: In the Carla simulation simulator, collect basic scene information for autonomous driving; Step 2: Based on the obtained basic scene information and the operating status information of the tested vehicle, perform reinforcement learning to generate the initial state of dynamic elements; Step 3: Perform reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of dynamic elements; Step 4: Obtain feedback information on the interaction between dynamic elements and the environment based on the action sequence of dynamic elements; Step 5: Adjust the network parameters of the policy network based on the feedback information from the interaction until the feedback information from the interaction reaches the expected level, and obtain the key security scenario; Step 5 involves adjusting the network parameters of the policy network based on interactive feedback information until the interactive feedback information meets expectations, as detailed below: Each round contains several time steps, and each time step generates interactive feedback information. Based on the interactive feedback information, reinforcement learning methods are used to encourage dynamic elements to produce excellent actions and punish dynamic elements to produce poor actions by rewarding or punishing them, until the average feedback information of each round gradually stabilizes and reaches the expected standard. The reinforcement learning method described herein is as follows: The optimization is performed using the policy gradient method of reinforcement learning. The policy in the policy gradient is represented by a network. By inputting different observations into the policy network, the policy network outputs different actions. The policy network determines the output of the action through the parameters in the policy network. The parameters of the policy network are trained to make the dynamic elements obtain the most reward. The parameters of the training policy network are designed to maximize the reward for dynamic elements, as detailed below: Reinforcement learning is used to study how an agent can maximize the reward it can obtain in an uncertain environment. The process of reinforcement learning is actually the interaction between the agent and the environment. The agent obtains the state in the environment and then outputs an action based on its own decision. Reinforcement learning is used to generate key safety scenarios for autonomous driving. These scenarios are dynamic, specifically a 60-second scenario from 8:00 to 8:

01. Assuming reinforcement learning generates 10 actions per second for different input states, the entire scenario contains 600 time steps. Within each time step, the state is represented by... To represent, action is used To represent, using Representing the state and action at time t, using The state and action for the entire time step are represented as follows: = ｛ ， ， ， …, , ｝ Because the environment is uncertain, within a time step t-1, the environment is performing the same actions. There is a probability p that different states will be generated. The agent adjusts the state according to its own policy network. There is a probability Produce different actions The parameters of the policy network are set to θ, therefore from generate The probability is: An agent receives different rewards for performing different actions in different states; therefore, at each time step, a reward is set as follows: Let R represent the sum of all rewards obtained in the entire scenario. In reinforcement learning, the goal of the policy network θ is to obtain the maximum reward R. Because R is a random variable, its value varies depending on the policy θ and the environment. The expected value of the reward is expressed by the following formula: Finally, gradient ascent is used to maximize the reward in an environment. The gradient of the reward R is expressed as:

2. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that, Step 1 involves collecting basic scene information for autonomous driving in the Carla simulation simulator, as detailed below: In the Carla simulation simulator, the built-in vehicle drives through the town, and uses the vehicle's camera sensors and depth sensors to collect basic scene information at fixed time intervals, including image data, dynamic feature information and static feature information.

3. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that, The basic scene information of the reinforcement learning input state mentioned in step 2 includes a top view of the scene; the test vehicle's operating state information includes the test vehicle's position coordinates, driving speed, and driving direction.

4. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that, Step 2 involves performing reinforcement learning based on the acquired basic scene information and the operating status information of the tested vehicle to generate the initial state of the dynamic elements, as detailed below: Step 2.1: Generate prior map information. Based on the basic scene information and the operating status information of the vehicle under test, determine the areas with higher and lower probabilities in generating safety-critical scenes, and list buildings and flower bed areas as non-generable areas. Step 2.2: Extract features from the top-down view of the basic scene using a convolutional neural network; Step 2.3: Through the fully connected layer, combined with prior map information and scene top view feature information, generate the initial state information of dynamic elements, including initial position and motion state information.

5. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 4, characterized in that, Step 3 involves performing reinforcement learning on the interaction between dynamic elements and the environment to obtain the action sequence of the dynamic elements, as detailed below: Step 3.1: Using a convolutional neural network, extract features from a specific square region centered on the location of the dynamic element to prevent the dynamic element from deviating from the road or colliding with static elements, thereby generating a logically sound safety-critical scenario. Step 3.2: Use LSTM to analyze the historical action sequences of dynamic elements to avoid dynamic elements generating past differentiated action sequences and ensure that the action sequences of dynamic elements are consistent with the real world. Step 3.3: By combining feature extraction, historical action analysis, and the actions of the tested vehicle with a fully connected layer, the action of the dynamic element at the next time step is generated. Step 3.4: The tested autonomous vehicle generates its position for the next time step through environmental perception, planning and decision-making, and control execution. Step 3.5: Repeat steps 3.1 to 3.4 until the dynamic element collides with the vehicle under test, or until all time steps are completed, to obtain the action sequence of the dynamic element.

6. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 4, characterized in that, Step 4 describes obtaining feedback information on the interaction between dynamic elements and the environment based on the action sequence of dynamic elements, as detailed below: Based on the action sequence of the dynamic elements, at each time step, interactive feedback information is generated based on the interaction between the dynamic elements, the tested vehicle, and the environment. This feedback information includes whether a collision occurred between the dynamic elements and the tested autonomous vehicle, and whether the dynamic elements produced non-compliant actions.

7. The method for generating key safety scenarios for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that, The key security scenarios described in step 5 are as follows: The system visualizes the interaction between the tested vehicle and dynamic elements at each time step in the generated safety-critical scenario, providing an intuitive representation of the generated safety-critical scenario to determine whether it meets the expected standards, and retaining the safety-critical scenario that meets the expected standards.

Images