Vehicle danger cut-in strategy construction method for automatic driving test

Through the deep reinforcement learning algorithm based on TD3, the vehicle hazardous entry strategy model is constructed, which solves the problem of low efficiency in vehicle entry scenario construction in virtual tests, and realizes efficient safety testing of autonomous driving vehicles. The model has independent decision-making and interaction capabilities, which is suitable for Diverse traffic flow scenarios.

CN119005015APending Publication Date: 2024-11-22XIAN AERONAUTICAL UNIV

Patent Information

Application Number
CN202411476234.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-22
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art is difficult to build challenging and autonomous decision-making vehicle entry scenarios in virtual testing environments, resulting in low efficiency and high cost of safety testing of autonomous vehicles.

Method used

The vehicle risk entry strategy model is constructed using a deep reinforcement learning algorithm based on TD3. By defining state representation and reward functions, the vehicle cutting strategy model with independent decision-making and interaction capabilities can be trained, so that it can dangerously enter the front of the vehicle under test in a virtual environment. , simulate rear-end collision risk.

Benefits of technology

It realizes the construction of challenging vehicle entry scenarios in a virtual test environment, improves the efficiency and cost-effectiveness of safety testing of autonomous vehicles, and the model has strong generalization capabilities and is suitable for different traffic flow scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005015A_ABST
    Figure CN119005015A_ABST
Patent Text Reader

Abstract

The invention discloses a vehicle danger cut-in strategy construction method for an automatic driving test, and belongs to the technical field of automatic driving, and the technical key points are that the method comprises the following steps: 1, building a simulation scene needed for vehicle danger cut-in strategy model training; 2, defining state representation of the vehicle danger cut-in strategy model, wherein the state representation is composed of a vehicle, nearby vehicles, a target point and related characteristics of a target vehicle; and 3, defining a reward function. The reward function comprises danger cut-in instant reward, self-vehicle danger cut-in guiding reward and improper behavior punishment; 4, constructing a vehicle danger cut-in strategy model, wherein the strategy model constructs vehicle danger cut-in based on TD3; 5, training a vehicle danger cut-in strategy model; and 6, after model training in the step 5 is completed, a vehicle danger cut-in strategy model is obtained, and the model has the advantages of being high in generalization ability and capable of being applied to different traffic flow test scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for constructing a vehicle danger cut-in strategy for autonomous driving testing. Background Art

[0002] Autonomous Vehicles (AVs) provide new solutions to solve the problems of traffic congestion, frequent accidents and energy shortages existing in traditional vehicles, and have become the focus of research in recent years. In an autonomous driving environment, the driving subject of the vehicle changes from the driver to the machine. Therefore, before the actual deployment of AVs, it is crucial to conduct safety-oriented performance tests to ensure their safety.

[0003] Testing in an open road environment is a common method for AVs testing. Since dangerous scenarios that challenge the safety of AVs in an open road environment are extremely rare, AVs need to travel billions of miles to prove that their safety is no less than that of human drivers. Considering many aspects such as testing cost, efficiency, and safety, open road testing can no longer meet the requirements of autonomous vehicle technology research and development and iteration. Virtual testing has the advantages of flexible scenario configuration, low testing cost, and high repeatability. Under virtual conditions, dangerous scenarios with a small probability of occurring on open roads can be constructed. Virtual testing has become the main method for AVs testing.

[0004] If AVs can safely deal with dangerous situations, their safety can usually be guaranteed in other situations. Therefore, virtual testing of AVs based on dangerous scenarios is conducive to shortening the test cycle and realizing accelerated testing of AVs. In order to give full play to the advantages of virtual testing, constructing dangerous test scenarios that are challenging for AVs has become an important research topic.

[0005] The vehicle cut-in scenario is a typical potentially dangerous driving scenario. Improper vehicle cut-in can easily cause a rear-end collision with the vehicle behind. In 2015, 4.9% of the traffic accidents in my country were caused by improper vehicle cut-in. Constructing a dangerous vehicle cut-in virtual scenario is of great significance for AVs safety testing. Traditional methods use methods such as clustering of natural driving data, Monte Carlo simulation or probability sampling to generate vehicle cut-in dangerous test scenarios. The traffic vehicles in the generated scenarios drive according to preset trajectories, do not have autonomous decision-making capabilities, and cannot respond to changes in the AVs system under test in a timely manner, which is different from the real environment.

[0006] The present invention provides a method for constructing a vehicle dangerous cut-in strategy based on TD3 (Twin Delayed Deep Deterministic Policy Gradients). Compared with traditional methods, this method can construct a vehicle control strategy model with both dangerous cut-in capabilities and interactive capabilities. The model can be used to control traffic vehicles to dangerously cut in front of a tested vehicle, making the tested vehicle face the risk of rear-end collision, thereby testing the safety of the tested AVs system. At the same time, the model constructed by this method can respond to dynamic changes of the tested AVs system and the driving environment in a timely manner. Summary of the invention

[0007] In view of the deficiencies in the prior art, an embodiment of the present invention aims to provide a method for constructing a vehicle danger cut-in strategy for autonomous driving testing, so as to solve the problems in the above-mentioned background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions: A method for constructing a vehicle danger entry strategy for autonomous driving testing comprises the following steps: Step 1: Build a simulation scenario for training the vehicle dangerous entry strategy model; Step 2: Define the state representation of the vehicle dangerous entry strategy model, which is composed of the relevant features of the ego vehicle, nearby vehicles, target points and target vehicles; Step 3: Define the reward function. The reward function includes the immediate reward for dangerous entry, the reward for guiding the ego car to dangerous entry, and the penalty for improper behavior; Step 4: Construct a vehicle danger cut-in strategy model. The strategy model is constructed based on TD3 to realize vehicle danger cut-in; Step 5: Train the vehicle dangerous entry strategy model; Step 6: After completing the model training described in step 5, the vehicle danger entry strategy model is obtained and applied to the safety performance test of the AVs system.

[0009] As a further solution of the present invention, the vehicle characteristics in step 2 include longitudinal vehicle speed , lateral speed , direction , Lane offset distance .

[0010] As a further solution of the present invention, the immediate reward for dangerous entry in step 3 is , .

[0011] As a further solution of the present invention, the step 3 guides the vehicle to enter the dangerous reward .

[0012] As a further solution of the present invention, the improper behavior penalty in step 3 is , the total reward function ; in, , and for , and The weight coefficients corresponding to the three rewards.

[0013] As a further solution of the present invention, the strategy model in step 4 is composed of 6 neural networks, including two state action value networks , And the corresponding two target action value networks , , policy network and target strategy network .

[0014] As a further solution of the present invention, the strategy network is initialized in step 5 for , initialize the state action value network , The parameters are and , initialize the target action value network , Parameters , and , Similarly, initialize the replay buffer.

[0015] In summary, compared with the prior art, the embodiments of the present invention have the following beneficial effects: 1. The present invention adopts a deep reinforcement learning algorithm to train a vehicle dangerous cut-in strategy model with autonomous decision-making and interactive capabilities. By proposing a targeted reward function, the constructed model has dangerous cut-in capabilities and interactive capabilities. It can start from any initial position and eventually cut dangerously to the front of the lane where the vehicle under test is located, causing a greater risk of rear-end collision to the vehicle under test, thereby testing the safety of the AVs system.

[0016] 2. The present invention is trained in scenarios containing complex and diverse traffic flows. The trained model has strong generalization ability and can be applied to test scenarios with different traffic flows.

[0017] 3. The vehicle danger entry strategy model constructed by the present invention can be implanted into any autonomous driving virtual test scenario and system, and is easy to promote and use.

[0018] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the extraction of nearby vehicles and target points in an embodiment of the invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0022] In one embodiment, a method for constructing a vehicle hazard cut-in strategy for autonomous driving testing is described in detail in the following sections. Figure 1 , including the following steps: Step 1: Build a simulation scenario for training the vehicle dangerous entry strategy model; Step 2: Define the state representation of the vehicle dangerous entry strategy model, which is composed of the relevant features of the ego vehicle, nearby vehicles, target points and target vehicles; Step 3: Define the reward function. The reward function includes the immediate reward for dangerous entry, the reward for guiding the ego car to dangerous entry, and the penalty for improper behavior; Step 4: Construct a vehicle danger cut-in strategy model. The strategy model is constructed based on TD3 to realize vehicle danger cut-in; Step 5: Train the vehicle dangerous entry strategy model; Step 6: After completing the model training described in step 5, the vehicle danger entry strategy model is obtained and applied to the safety performance test of the AVs system.

[0023] For further information, see Figure 1 In step 2, the vehicle characteristics include longitudinal speed , lateral speed , direction , Lane offset distance .

[0024] For further information, see Figure 1 , the immediate reward for dangerous entry in step 3 is , , It refers to the time when the vehicle cuts in. The collision time with the target vehicle, It refers to the time when the vehicle cuts in. The relative speed to the target vehicle.

[0025] For further information, see Figure 1 , in step 3, the reward for guiding the vehicle to enter dangerously ,in, It refers to the distance reward between the vehicle and the target point. It refers to the collision time discount coefficient between the ego vehicle and the target vehicle.

[0026] For further information, see Figure 1 , the penalty for improper behavior in step 3 , the total reward function ; in, , and for , and The weight coefficients corresponding to the three rewards.

[0027] For further information, see Figure 1 The strategy model in step 4 is composed of 6 neural networks, including two state-action-value networks , And the corresponding two target action value networks , , policy network and target strategy network .

[0028] For further information, see Figure 1 , in step 5, the strategy network is initialized for , initialize the state action value network , The parameters are and , initialize the target action value network , Parameters , and , Similarly, initialize the replay buffer.

[0029] In this embodiment, the present invention provides a method for constructing a vehicle dangerous entry strategy for AVs testing. The vehicle dangerous entry strategy model constructed by this method can start from any initial position and eventually dangerously cut in front of the lane where the vehicle to be tested is located, causing a greater risk of rear-end collision to the vehicle to be tested, thereby realizing the test of the safety of the AVs system to be tested.

[0030] The method for constructing a vehicle hazard entry strategy for AVs testing includes the following steps: Step 1: Build a simulation scenario for training the vehicle dangerous cut-in strategy model. In order to improve the generalization ability of the model, in the simulation scenario, the spacing, speed and position of the traffic vehicles in the scene are randomly initialized to generate complex and diverse traffic flows. In the simulation scenario, arbitrarily select a traffic vehicle as the ego vehicle and another traffic vehicle as the target vehicle. The ego vehicle is controlled by the vehicle dangerous cut-in strategy model. During the model training process, at the beginning of each round, the ego vehicle starts from the starting position. When it enters the lane where the target vehicle is located and is in front of the target vehicle, it is considered that the ego vehicle has completed the cut-in. When the ego vehicle successfully cuts in front of the target vehicle and the TTC value at the time of cut-in is 0~6s, it is considered that the ego vehicle has successfully cut in dangerously.

[0031] Step 2: Define the state representation of the vehicle danger entry strategy model. The state representation used in the present invention is composed of the characteristics of the vehicle, nearby vehicles, target points and target vehicles. Among them, the characteristics of the vehicle include the longitudinal speed , lateral speed , direction , Lane offset distance ; Nearby vehicle features include the features of the six vehicles closest to the vehicle. These vehicles are located in the same lane and in front and behind the left and right adjacent lanes of the vehicle. The Euclidean distance between the vehicle and the ego vehicle , Horizontal distance and longitudinal distance ; The target point refers to a preset point directly in front of the lane where the target vehicle is located. The characteristics of the target point and the target vehicle include the distance between the vehicle and the set target point. , the longitudinal distance between the ego vehicle and the target vehicle , Horizontal distance , relative speed , .in, , .

[0032] Step 3: Define the reward function. The reward function is the core of reinforcement learning. Designing a suitable reward can accelerate the convergence of the model and improve the performance of the model. The reward function used in the present invention includes three parts: immediate reward for dangerous entry, reward for guiding the vehicle to dangerous entry, and penalty for improper behavior.

[0033] 1. Dangerous entry instant reward The time when the self-vehicle cuts into the lane where the target vehicle is and is in front of the target vehicle is ,like s, a larger reward is given to the agent; if If it does not fall into the above range, the relative speed between the self-vehicle and the target vehicle will be used. Determine the reward value, The larger the value, the higher the risk of the target car rear-ending the self-vehicle, and the reward value increases accordingly. Specifically, as shown in formula (1): Formula (1) 2. Guide the car to enter the dangerous reward In order to guide the intelligent agent to control the ego vehicle to drive in front of the target vehicle, a virtual target point is set at a distance of 10m in front of the lane where the target vehicle is located. Design the distance reward as shown in formula (2) ,along with Reduce, The closer the distance between the vehicle and the target point is, the higher the reward value is. In fact, it encourages the intelligent agent to control the vehicle to drive to the target point and arrive in front of the target vehicle.

[0034] Formula (2) In the formula, For The coefficient used for normalization.

[0035] Since reinforcement learning training aims to maximize the cumulative reward, setting only the distance reward will cause the agent to drive in front of the target car as fast as possible to collect more rewards, which will cause the vehicle to be faster than the target car when cutting in front of the target car, and fail to meet the requirements of dangerous cutting. In order to solve this problem, the TTC discount coefficient is defined , calculate its value according to formula (3). Obviously, the smaller the TTC value, the When the distance between the vehicle and the target point is far, The value is set to 1.

[0036] Formula (3) In the formula, is the coefficient for normalizing TTC.

[0037] Finally, the reward item for guiding the ego vehicle to cut in dangerously is calculated by formula (4): .

[0038] Formula (4) 3. Penalties for improper behavior In order to prevent the ego vehicle from engaging in improper behaviors such as driving off the road, reversing, or actively colliding with traffic vehicles during driving, the improper behaviors of the ego vehicle are punished by setting a negative reward value. Since the ego car can easily cause the target car to rear-end when it cuts in dangerously, in order not to affect the agent's exploration of the environment, no penalty is given when the ego car is rear-ended by a traffic car.

[0039] Finally, the reward function adopted by the present invention is the linear superposition of the above three parts of rewards, as shown in formula (5).

[0040] Formula (5) in, , and for , and The weight coefficients corresponding to the three rewards.

[0041] Step 4: Construct a vehicle danger cut-in strategy model. The present invention constructs a vehicle danger cut-in strategy model based on TD3. The constructed model consists of 6 neural networks, including two state action value networks , And the corresponding two target action value networks , , policy network and target strategy network Among them, the policy network The role is to adjust the input status ( )Select Action( ) as output, so that the ego vehicle can interact with the established vehicle cut-in environment; state action value network The role of the state action pair generated by the interaction between the vehicle and the environment is The target policy network and the target state action value network directly copy the corresponding policy network and action value network structure parameters, and update the parameters through soft update, in order to improve the stability of deep reinforcement learning model training.

[0042] Step 5: Train the vehicle dangerous entry strategy model. First, initialize the strategy network for , initialize the state action value network , The parameters are and , initialize the target action value network , Parameters , and , Similarly, initialize the replay buffer.

[0043] Then, repeat the following steps Second-rate: (1) Start and run the simulation scenario described in step 1, and extract the initial state of the vehicle according to the method described in step 2, denoted as .

[0044] (2) In the current strategy Select the action and add noise, as shown in formula (6).

[0045] Formula (6) in, , are the mean and variance of the normal distribution, which are 0 and 0.1 respectively. and is the upper and lower limits of the action value, function The action after adding noise is intercepted so that it is in interval.

[0046] (3) Execute action , extract the new state of the vehicle according to the method described in step 2 , calculate the reward according to the method described in step 3 ,Will( ) is stored in the replay buffer.

[0047] (4) The current state Updated to .

[0048] (5) Randomly extract a number of of small batch samples, for each sample ( ), and do the following calculation: Formula (7) Formula (8) (6) Calculate the action value network using the following formula: , loss.

[0049] Formula (9) (7) Update the action value network by minimizing the above loss , Parameters.

[0050] Formula (10) (8) Every Step 2, use the following formula to calculate the policy gradient. Use the policy gradient to update the policy network Parameters.

[0051] Formula (11) (9) Using the soft update method, the target strategy network is updated using the following formula: and the target state action value network , Parameters. is the soft update coefficient.

[0052] Formula (12) Formula (13) Step 6: After completing the model training described in step 5, the vehicle dangerous entry strategy model is obtained, which can be further applied to the safety performance test of the AVs system.

[0053] This example builds a vehicle cut-in simulation environment based on the SMARTS platform. SMARTS is a virtual simulation platform for multi-agent reinforcement learning and autonomous driving research, developed by Huawei Noah's Ark Laboratory. This example builds a vehicle cut-in environment based on the SMARTS platform. The length of the lane is 200m. The speed of the background vehicle in the scene is controlled by SUMO. Specifically, the background vehicle will combine the lane speed limit, the maximum expected speed, and the speed value output by the built-in following model of SUMO, and select the lowest value as the target speed. The ego vehicle is controlled by the vehicle dangerous cut-in strategy described above.

[0054] 1. In order to be closer to the ever-changing traffic flow density in the real traffic environment, three training scenarios with different flow levels were generated, corresponding to low, medium and high flow. The traffic flow on a single lane in these three scenarios was set to 1200veh / h, 1800veh / h and 2400veh / h respectively. The initial spacing, speed and position of the traffic vehicles in the scene were randomly generated, and the three scenarios contained a total of 1200 different initialized traffic flows.

[0055] 2. At the beginning of each round of interaction with the environment using the dangerous vehicle cut-in strategy, any vehicle in the traffic flow is randomly selected as the target vehicle. When the ego vehicle starts from the starting position, enters the lane where the target vehicle is located and is in front of the target vehicle, the ego vehicle is considered to have completed the cut-in. When the ego vehicle successfully cuts in front of the target vehicle and the TTC value at the time of cut-in is 0~6s, the ego vehicle is considered to have successfully cut in.

[0056] Step 2: Determine the state representation of the vehicle danger entry strategy model. The state representation used in this example consists of three parts, as shown in Table 1. The first part is the characteristics of the vehicle, which has a total of 4 items; the second part is the characteristics of nearby vehicles. In order to extract the most relevant information for the decision of the vehicle, the surrounding vehicle information provided by SMARTS is filtered, and only the six vehicles closest to the vehicle are extracted. The distance between each vehicle and the vehicle is extracted. , longitudinal distance and lateral distance Features, such as Figure 1 As shown in the figure, a total of 18 features are extracted; the third part is the target vehicle and target point features. Since the target vehicle may not be included in the six nearby vehicles closest to the vehicle, the target vehicle information needs to be extracted additionally. The target point (indicated by a star mark) refers to a preset point 10m in front of the lane where the target vehicle is located. Since the TTC calculation result has a wide range of values, in order to reduce the search space of parameters in model training, the TTC calculation value is limited to the range of 0~20s.

[0057] Table 1 Status indication

[0058] Step 3: Determine the reward function. The reward function in this example includes three parts: immediate reward for dangerous entry, reward for guiding the ego vehicle to dangerous entry, and penalty for improper behavior.

[0059] 1. Dangerous entry instant reward The time when the self-vehicle cuts into the lane where the target vehicle is and is in front of the target vehicle is ,like s, a larger reward is given to the agent; if If it does not fall into the above range, the relative speed between the self-vehicle and the target vehicle will be used. Determine the reward value, The larger the value, the higher the risk of the target car rear-ending the self-vehicle, and the reward value increases accordingly. Specifically, as shown in formula (14): Formula (14) 2. Guide the car to enter the dangerous reward In order to guide the intelligent agent to control the ego vehicle to drive in front of the target vehicle, a virtual target point is set at a distance of 10m in front of the lane where the target vehicle is located. Design the distance reward as shown in formula (15) ,along with Reduce, Increases, that is, the closer the distance between the vehicle and the target point is, the higher the reward value is.

[0060] Formula (15) In the formula, For The coefficient used for normalization is 50.

[0061] Since reinforcement learning training aims to maximize the cumulative reward, setting only the distance reward will cause the agent to drive in front of the target car as fast as possible to collect more rewards, which will cause the vehicle to be faster than the target car when cutting in front of the target car, and fail to meet the requirements of dangerous cutting. To solve this problem, this example defines the TTC discount coefficient , when the distance between the vehicle and the target point is less than 5m, the value is calculated according to formula (16). Obviously, the smaller the TTC value, the When the vehicle is far away from the target point, The value is set to 1.

[0062] Formula (16) In the formula, This is the coefficient for normalizing TTC, and in this example, the value is 20. If TTC is a negative value, 20 is uniformly substituted into the above formula for calculation.

[0063] Finally, the reward item for guiding the ego vehicle to cut in dangerously is calculated by formula (17): .

[0064] Formula (17) 3. Penalties for improper behavior In order to prevent the vehicle from performing improper behaviors such as driving off the road, reversing, or actively colliding with traffic vehicles during driving, this example sets the improper behavior penalty items shown in Table 2 Since the ego car can easily cause the target car to rear-end when it cuts in dangerously, in order not to affect the agent's exploration of the environment, no penalty is given when the ego car is rear-ended by a traffic car.

[0065] Finally, the reward function used in this example is the linear superposition of the above three parts of the reward, as shown in formula (18).

[0066] Formula (18) in, , and for , and The weight coefficients corresponding to the three rewards. After repeated experiments, the weight coefficients of the three rewards are all set to 1 in this example.

[0067] Table 2 Penalties for improper behavior of the vehicle

[0068] Step 4: This example builds a vehicle danger entry strategy model based on the deep reinforcement learning algorithm TD3. The model consists of 6 neural networks, including two state action value networks , And the corresponding two target action value networks , , policy network and target strategy network Among them, the policy network The role is to adjust the input status ( )Select Action( ) as output, so that the agent can interact with the established vehicle cut-in environment; state action value network The role of the state-action pair generated by the interaction between the agent and the environment is The target policy network and the target state action value network directly copy the corresponding policy network and action value network structure parameters, and update the parameters through soft update, in order to improve the stability of deep reinforcement learning model training.

[0069] In order to enhance the agent’s exploration of the environment, noise sampled from a normal distribution is added to the actions output by the TD3 main policy network, as shown in Equation (19).

[0070] Formula (19) In the formula, , are the mean and variance of the normal distribution, which are 0 and 0.1 respectively. and The upper and lower limits of the action value. In this example, the action specifically includes acceleration and yaw rate. The acceleration range is set to -5~5m / s2, and the yaw rate range is set to -0.5~0.5rad / s. When the action value exceeds the upper or lower limit of the set interval, the corresponding upper and lower limits are taken. The function intercepts the action after adding noise, making it interval.

[0071] Step 5: Train the vehicle dangerous entry strategy model. After grid search, the six neural networks in the TD3 vehicle dangerous entry strategy model constructed in this example all use a two-layer fully connected neural network structure, with 400 and 300 neurons respectively, and Tanh as the activation function. The hyperparameter values ​​used are shown in Table 3.

[0072] Table 3 TD3 hyperparameter values

[0073] First, initialize the policy network for , initialize the state action value network , The parameters are and , initialize the target action value network , Parameters , and , Similarly, initialize the replaybuffer.

[0074] Then, repeat the following steps Second-rate: (1) Initialize and run the simulation scenario described in step 1. Use the method described in step 2 to extract the initial state of the vehicle, which is recorded as .

[0075] (2) In the current strategy Select Action and add noise to the action according to step 2.

[0076] (3) Execute action , extract the new state of the vehicle according to the method described in step 2 , calculate the reward according to the method described in step 3 ,Will( ) is stored in the replay buffer.

[0077] (4) The current state Updated to .

[0078] (5) Randomly extract a number of of small batch samples, for each sample ( ), and do the following calculation: Formula (20) Formula (21) (6) Calculate the action value network using the following formula: , loss.

[0079] Formula (22) (7) Update the action value network by minimizing the above loss , Parameters.

[0080] Formula (23) (8) Every Step 2, use the following formula to calculate the policy gradient. Use the policy gradient to update the policy network Parameters.

[0081] Formula (24) (9) Using the soft update method, the target strategy network is updated using the following formula: and the target state action value network , Parameters. is the soft update coefficient.

[0082] Formula (25) Formula (26) Step six, after completing the model training described in step five, obtain the vehicle dangerous cut-in strategy model. In order to use the vehicle dangerous cut-in strategy model constructed in this example for AVs safety performance testing, a three-lane simulation environment is established, in which a multi-agent test environment containing two agents is constructed, wherein one agent adopts the intelligent driving strategy to be tested to control the vehicle to be tested, and the other agent controls the dangerously cutting-in traffic vehicle through the trained vehicle dangerous cut-in strategy, with the vehicle to be tested as the target vehicle. Other traffic vehicles in the environment are controlled by the built-in traffic vehicle model of SMARTS. During the test, the traffic test vehicle controlled by the vehicle dangerous cut-in strategy will actively and dangerously cut in front of the vehicle to be tested, causing the vehicle to be tested to have a rear-end collision risk, which can play a role in safety testing the control strategy of the vehicle to be tested.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a vehicle danger entry strategy for autonomous driving testing, characterized in that: The following steps are involved: Step 1: Build a simulation scenario for training the vehicle dangerous entry strategy model; Step 2: Define the state representation of the vehicle dangerous entry strategy model, which is composed of the relevant features of the ego vehicle, nearby vehicles, target points and target vehicles; Step 3: Define the reward function, which includes the immediate reward for dangerous entry, the reward for guiding the ego car to dangerous entry, and the penalty for improper behavior; Step 4: Construct a vehicle danger cut-in strategy model. The strategy model is constructed based on TD3 to realize vehicle danger cut-in; Step 5: Train the vehicle dangerous entry strategy model; Step 6: After completing the model training described in step 5, the vehicle dangerous entry strategy model is obtained, and the strategy model is used for the safety performance test of the AVs system.

2. The method for constructing a vehicle danger cut-in strategy for autonomous driving testing according to claim 1, characterized in that: The vehicle characteristics in step 2 include longitudinal vehicle speed , lateral speed , direction , Lane offset distance .

3. The method for constructing a vehicle danger entry strategy for autonomous driving testing according to claim 2, characterized in that: The immediate reward for risk entry in step 3 is , , It refers to the time when the vehicle cuts in. The collision time with the target vehicle, It refers to the time when the vehicle cuts in. The relative speed to the target vehicle.

4. The method for constructing a vehicle danger entry strategy for autonomous driving testing according to claim 3, characterized in that: In step 3, the reward for guiding the vehicle to enter dangerously ,in, It refers to the distance reward between the vehicle and the target point. It refers to the collision time discount coefficient between the ego vehicle and the target vehicle.

5. The method for constructing a vehicle dangerous entry strategy for autonomous driving testing according to claim 4, characterized in that: Penalties for inappropriate behavior in step 3 , the total reward function ; in, , and for , and The weight coefficients corresponding to the three rewards.

6. The method for constructing a vehicle danger cut-in strategy for autonomous driving testing according to claim 5, characterized in that: The strategy model in step 4 is composed of 6 neural networks, including two state-action-value networks: , And the corresponding two target action value networks , , policy network and target strategy network .

7. The method for constructing a vehicle dangerous entry strategy for autonomous driving testing according to claim 6, characterized in that: Initialize the policy network in step 5 for , initialize the state action value network , The parameters are and , initialize the target action value network , Parameters , and , Similarly, initialize the replaybuffer.

Citation Information

Patent Citations

  • Automatic driving lane changing decision control method based on rule fusion reinforcement learning

    CN115257745A

  • Intelligent vehicle driving method based on reinforcement learning

    CN117270394A

  • Method for generating dynamic cut-in behavior of traffic participant in automatic driving scene

    CN117609093A

  • Highly humanoid safe-driving decision-making method for autonomous commercial vehicle

    WO2023155231A1

Cited By

  • Multi-vehicle driving control method and system based on AI

    CN119200673A

  • Vehicle danger cut-in strategy construction method based on aerial view input

    CN119272639A