Dynamic behavior generation method for multiple autonomous driving vehicles based on H-PPO hierarchical algorithm

Abstract

The application discloses a kind of dynamic behavior generation method of multiple automatic driving vehicles based on H-PPO hierarchical algorithm, comprising: 1, simulating intersection vehicle intersection road condition, constructs simulation scene, and carries out dynamics modeling to automatic driving vehicle;2, according to environmental information, construct state quantity, mixed action space;3, encourage the intersection game between the driving vehicle and the measured vehicle by setting reward function;4, construct deep neural network, initialize deep reinforcement learning parameters, train deep neural network, produce for the measured vehicle intersection left turn working condition, make multiple straight automatic driving vehicles decision and action are layered output by neural network, make it have game behavior, to test the response capability of the measured vehicle to the edge scene of small probability, high risk.The application uses deep reinforcement learning and game theory to generate the game intersection scene encountered by the measured vehicle at the intersection, to provide a new way to build automatic driving scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of driving safety evaluation of unmanned vehicles at intersections, in particular to a dynamic behavior generation method for multiple autonomous vehicles based on H-PPO hierarchical algorithm. BACKGROUND

[0002] For intelligent vehicles, autonomous vehicles are an important part of the driving environment. Different driving styles of autonomous vehicles in different scenarios have a profound impact on the safety and reliability of intelligent vehicles. Unmanned driving has been widely used in relatively simple scenarios such as various mining areas and industrial parks, but for relatively complex driving scenarios such as actual highways, environmental factors such as obstacles, traffic signs, ground road conditions and other vehicles need to be considered, which brings many uncertain dangers to the landing of unmanned driving. How to determine the reliability of unmanned vehicles and find their defects has become an urgent need. The existing scenario generation method is artificially built, and the movement of other road users is determined before running, which cannot actively and deeply explore the deficiencies of unmanned vehicles. SUMMARY

[0003] In order to overcome the shortcomings of the prior art, the present application provides a dynamic behavior generation method for multiple autonomous vehicles based on H-PPO hierarchical algorithm, which can generate autonomous vehicles with game behavior, and play with the left turn behavior of the measured vehicle at the intersection, so as to explore the deficiencies of autonomous vehicles at the intersection, make the vehicle pass safely and efficiently during driving, and provide protection for the safe navigation of intelligent vehicles.

[0004] In order to achieve the above invention purpose, the following technical scheme is adopted:

[0005] The dynamic behavior generation method for multiple autonomous vehicles based on H-PPO hierarchical algorithm has the characteristics that it is applied to the automatic driving scene of preventing the test vehicle ego on the opposite lane from turning left at the intersection by the front and rear two autonomous vehicles agent1 and agent2 driving on the same lane, and includes the following steps:

[0006] Step 1, obtaining the upper limit a max and the lower limit a min of the acceleration a agent1 of the autonomous vehicle agent1 located in front agent2 and the acceleration a agent1 of the autonomous vehicle agent2 located in front agent2 , and taking a agent1 and a agent2 as an action parameter;

[0007] Step 2, establish a vehicle Frenet coordinate system with the center of the vehicle itself as the origin, the direction in which the vehicle travels along the road as the longitudinal direction, and the direction perpendicular to the road as the transverse direction;

[0008] According to the environmental information collected by the autonomous vehicle, and combined with the dynamics model, a state parameter s=(x agent1 -x ego ,y agent1 -y egi ,v agent1 -v ego ,a agent1 -a ego ,x agent2 -x ego ,y agent2 -y ego ,v agent2 -v ego ,a agent2 -a ego ,θ ego ) is constructed, wherein x agent1 represents the transverse coordinate of the autonomous vehicle agent1 located in front, x agent2 represents the transverse coordinate of the autonomous vehicle agent2 located in back, y agent1 represents the longitudinal coordinate of the autonomous vehicle agent1 located in front, y agent2 represents the longitudinal coordinate of the autonomous vehicle agent2 located in back, x ego represents the transverse coordinate of the measured vehicle ego, y ego represents the longitudinal coordinate of the measured vehicle ego, v agent1 represents the speed of the autonomous vehicle agent1 located in front, v agent2 represents the speed of the autonomous vehicle agent2 located in back, v ego represents the speed of the measured vehicle ego, a agent1 represents the acceleration of the autonomous vehicle agent1 located in front, a agent2 represents the acceleration of the autonomous vehicle agent2 located in back, and θ ego represents the heading angle of the test vehicle ego;

[0009] Step 3, construct a reward function composed of four components for the game between the front and back two autonomous vehicles agent1, agent2 and the test vehicle ego on the same lane;

[0010] Step 4, construct a deep neural network, including: a policy network and an evaluation network, wherein the policy network includes: a continuous action network and a discrete action network;

[0011] obtain a historical state parameter set S = {s i |i = 1, 2, …, n}, wherein s i represents the state of the i-th step, and n represents the upper limit of the experience pool capacity;

[0012] The historical state parameter set S is input into the policy network for processing to obtain an action parameter set A = {a i |i = 1, 2, …, n}, wherein a i represents the action of the i-th step, and a i = {c i , d i | c i ∈ C, d i ∈ D}; C = {c i |i = 1, 2, …, n} is a continuous action parameter, and D = {d i |i = 1, 2, …, n} is a discrete action parameter, c i represents the action parameter of the i-th step of the two autonomous vehicles agent1 and agent2, and d i represents the decision parameter of whether the two autonomous vehicles agent1 and agent2 enter the intersection at the i-th step;

[0013] (s i , a i , s i+1 , R i ) is stored in the experience pool as the i-th sample; wherein s i+1 represents the state of the i+1-th step;

[0014] Step 5, based on the historical state parameter set S and the action parameter A, the deep neural network is trained by using the H-PPO algorithm, and a reward function is calculated to obtain an optimal left-turn prevention model;

[0015] Step 6, the state parameter s i of the i-th step of the current scene is input into the optimal left-turn prevention model, and the optimal action parameter c * i of the i-th step of the current scene is output. * i , as the acceleration and action decision of the two autonomous vehicles agent1 and agent2 at the i-th step, to prevent the test vehicle ego from changing lanes at the i-th step.

[0016] The multi-autonomous vehicle dynamic behavior generation method based on the H-PPO hierarchical algorithm has the characteristics that the step 3 comprises:

[0017] Step 3.1: constructing the reward function R1 of the first part by using formula (5):

[0018]

[0019] In formula (5), s begin is the starting position of the vehicle ego entering the intersection, s end is the position where the vehicle ego leaves the intersection, s1 ego is the state of the tested vehicle ego before entering the intersection, s2 ego is the state of the vehicle ego just entering the intersection, s3 ego is the state of the tested vehicle ego when it is in the middle of the intersection, s4 ego is the state of the tested vehicle ego when it leaves the intersection; d agent1 represents the decision action of the autonomous vehicle agent 1, d agent2 Represents the decision action of the autonomous vehicle agent2;

[0020] Step 3.2: Use formula (6) to construct the reward function R2 of the second part:

[0021]

[0022] In formula (6), a set is a gaming indicator;

[0023] Step 3.3: Use formula (7) to construct the reward function R3 of the third part:

[0024]

[0025] In formula (7), min{a agent1 ,a agent2} is to take a agent1 with a agent2 The number with the smaller median value, max{a agent1 ,a agent2} is to take a agenet1 with a agenet2 The number with the larger median value;

[0026] Step 3.4: Use formula (8) to construct the reward function R4 of the fourth part:

[0027]

[0028] In formula (8), T c Indicates the collision risk factor, V p represents the speed of the autonomous driving vehicle agent1 or agent2 that collides with the vehicle under test ego, E ego To define the collision caused by the ego of the tested vehicle, E agentTo define an autonomous vehicle agent 1 or agent 2 caused collision.

[0029] The step 4 comprises:

[0030] The evaluation network is composed of two full connection layers and an evaluation output layer, and the activation function between all full connection layers is a Relu function; the parameters of the evaluation network are denoted as The input of the evaluation network is the normalized current state, and the output is the value of the current state;

[0031] The continuous action network outputs the neural network of the mean part and the neural network of the variance part, and the discrete action network outputs the probability value of executing each action; wherein the neural network of the mean part of the continuous network output comprises: two full connection layers and a mean output layer, and the activation function of the mean output layer is a tanh function; the neural network of the variance part comprises: two full connection layers and a variance output layer, and the activation function of the variance output layer is a softplus activation function; the activation function between all full connection layers is a Relu function, and the discrete action network comprises a full connection layer, which outputs the probability value of executing each discrete action;

[0032] The parameters of the continuous action network are denoted as θ c , the parameters of the discrete action network are θ d , and the input of the continuous action network and the discrete action network is the normalized current state; the output of the continuous action network is the current action a c ~N(ε, std), and the output of the discrete action network is the current decision a d ∈{d i |i=1,2,…,n}; wherein ε and std represent the mean and variance respectively; N represents Gaussian distribution, and ~ represents compliance;

[0033] The training of the policy network is defined as π θ , the environment interaction parameter of the policy network is defined as θ', and the environment interaction strategy of the policy network is π θ' .

[0034] The step 5 comprises:

[0035] Step 5.1: extract any ith sample from the experience pool, and calculate the estimated value of the ith sample by using the advantage function estimator shown in formula (9)

[0036]

[0037] In formula (9), V φ (s i ) is s in the ith samplei The state value obtained after inputting the value network, V φ (s i+1 ) is to convert s in the i-th sample i+1 The state value obtained after inputting the value network; γ is the reward discount coefficient;

[0038] Step 5.2: Define the state quantity s i The strategy π under θ Select action a in step i i The probability of π(a i |s i ,θ i ), in the state quantity s i The strategy π under θ' Select action a in step i i The probability of π(a i |s i ,θ' i ), define π(a i |s i ,θ i ) and π(a i |s i ,θ' i ) is r(θ i );θ i Represents the network parameters of the policy network at step i;

[0039] Step 5.3: Use formula (10) to get the network parameters θ of the policy network at step i+1 i+1 :

[0040]

[0041] In formula (10), is the gradient operator of the policy network parameter θ, clip(r(θ i ),1-ε,1+ε) is the range limiting function, which is r(θ i ) is limited to the range of 1-ε to 1+ε, where ε is the set threshold. For expectations;

[0042] Step 5.4: Use formula (11) to get the network parameters of the value network at step i+1

[0043]

[0044] In formula (11), is the value network parameter The gradient operator, is the network parameter of the value network at step i;

[0045] Step 5.5: After i+1 is assigned to i, return to step 5.1 for sequential execution until the maximum number of steps max is reached, forming a parameter set of the optimal lane-changing prevention model.

[0046] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the dynamic behavior generation method, and the processor is configured to execute the program stored in the memory.

[0047] The computer-readable storage medium of the present application has a computer program stored thereon, characterized in that the computer program, when executed by a processor, performs the steps of the dynamic behavior generation method.

[0048] Compared with the prior art, the beneficial effects of the present application are reflected in:

[0049] 1. The present application trains an autonomous vehicle using a deep reinforcement learning H-PPO hierarchical algorithm, combines game theory to design a reward function, and can generate an autonomous vehicle with game characteristics while ensuring that the traffic flow scenario can better match the actual driving.

[0050] 2. Compared with traditional scene generation methods, the present application introduces the exploration and development advantages of the H-PPO deep reinforcement hierarchical learning algorithm. In the case of traditional methods falling into local optimization, the present application can learn how to escape from local stable points. Compared with traditional reinforcement learning algorithms, the present application can effectively combine discrete actions and continuous actions for training, so that the autonomous vehicle can continuously explore and optimize the game actions and decisions taken according to the performance of the measured vehicle, resulting in a more comprehensive scene than traditional methods.

[0051] 3. The present application designs a reward function based on game theory, uses the trend of deep reinforcement learning to pursue maximum rewards, and encourages the autonomous vehicle to generate games with the measured vehicle. Thus, in the case of the intelligent vehicle turning left at the intersection, the present application can gradually generate marginal and low-probability traffic scenarios based on the behavior of the measured vehicle, thereby solving the problem that the autonomous vehicle cannot master reasonable game strength and cannot adjust its action style based on the effects of previous scenarios in traditional scene generation methods. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The present application is a scene schematic diagram;

[0053] Figure 2 The present application is a strategy-evaluation network structure diagram. DETAILED DESCRIPTION

[0054] In this embodiment, a method for generating dynamic behaviors of multiple autonomous vehicles based on the H-PPO hierarchical algorithm can train autonomous vehicles with game-like behavior in a straight-ahead intersection scenario and test the reliability of the tested vehicles in this scenario. Specifically, the method is performed in the following steps:

[0055] Step 1: Build a simulation environment for left-turn conditions at an intersection. Define the two autonomous driving vehicles in the simulation environment as agent1 and agent2, and the test vehicle as the ego vehicle. Figure 1 As shown in the figure, the two agent cars and the ego car are on both sides of the intersection and driving in opposite directions. A dynamic model is established for the tested vehicle ego and the two autonomous driving vehicles agent1 and agent2 in front and behind. According to the dynamic model, the acceleration selection range of the two autonomous driving vehicles agent1 and agent2 in front and behind is obtained, that is, the upper limit a max and lower limit a min .

[0056] Step 2: Based on the collected environmental information, the state space S and action space A are constructed in conjunction with the dynamics model;

[0057] Establish a Frenet coordinate system with the center of the vehicle as the origin, the direction of the vehicle traveling along the road as the longitudinal direction, and the direction perpendicular to the road as the transverse direction;

[0058] According to the collected environmental information of the autonomous driving vehicle and combined with the dynamic model, the state parameter s=(x agent1 -x ego ,v agent1 -y ego ,v agent1 -y ego ,a agent1 -a ego ,v agent2 -x ego ,y agent2 -y ego ,v agent2 -v ego ,a agent2 -a ego ,θ ego ), where x agent1 represents the lateral coordinate of the autonomous driving vehicle in front, x agent2 Represents the lateral coordinate of the autonomous driving vehicle at the rear, y agent1 Indicates the longitudinal coordinate of the autonomous driving vehicle in front, y agent2 represents the longitudinal coordinate of the autonomous driving vehicle at the rear, x ego Indicates the lateral coordinate of the vehicle being tested, y ego Indicates the longitudinal coordinate of the vehicle under test, v agent1represents the speed of the autonomous vehicle in front, v agent2 represents the speed of the autonomous vehicle behind, v ego Indicates the speed of the vehicle being tested, a agent1 represents the acceleration of the autonomous vehicle in front, a agent2 represents the acceleration of the autonomous vehicle behind, θ ego represents the heading angle of the test vehicle;

[0059] Step 3: Encourage the game between the autonomous vehicle and the test vehicle by setting an appropriate reward function;

[0060] Step 3.1: Construct the reward function R1 for the first part:

[0061] The first part of the reward function R1 is given by the relationship between the horizontal and vertical distances of the three vehicles. When the first autonomous driving vehicle is located in the intersection, it receives a penalty, a reward value, a maximum reward value, and a penalty according to the four situations of the tested vehicle not entering the intersection, entering the intersection, being in the middle of the intersection, and having passed the intersection. The second autonomous driving vehicle also faces the above four situations of the tested vehicle, but the reward value is given by the relationship between penalty, penalty, reward, and penalty. The specific relationship is shown in formula (1).

[0062]

[0063] In formula (1), s begin is the starting position of the vehicle entering the intersection, s end is the position where the tested vehicle leaves the intersection, s ego1 The vehicle under test does not enter the intersection, s ego2 is the state of the tested vehicle just entering the intersection, s ego3 The vehicle under test is located in the middle of the intersection, s ego2 The vehicle under test has passed through the intersection;

[0064] Step 3.2: Use formula (2) to construct the reward function R2 of the second part:

[0065] The reward function R2 in the second part is composed of the acceleration of the tested vehicle, which prevents the tested vehicle from turning left and is expected to cause the tested vehicle to perform a sudden acceleration or deceleration when turning left. That is, a positive reward is given if the absolute value of the tested vehicle's acceleration is greater than or less than a certain value, and the reward increases as the absolute value of the acceleration increases.

[0066]

[0067] In formula (2), a setTo identify whether the tested vehicle has taken a sufficiently strong stress response, the game indicator is: when the acceleration of the tested vehicle is greater than a set When the autonomous vehicle is given more rewards to encourage a more intense gaming scenario;

[0068] Step 3.3: Use formula (3) to construct the reward function R3 of the third part:

[0069] The acceleration values ​​of the two autonomous vehicles are restricted to better match the state of the vehicles during acceleration. If the acceleration of the autonomous vehicle exceeds a certain value, a penalty will be imposed.

[0070]

[0071] In formula (7), min{a agent1 ,a agent2} is to take a agent1 with a agent2 The number with the smaller median value, max{a agent1 ,a agent2} is to take a agent1 with a agent2 The number with the larger median value;

[0072] Step 3.4: Use formula (8) to construct the reward function R4 of the fourth part:

[0073] If a collision occurs, the cause must be determined. If the collision is caused by an unreasonable left turn by the tested vehicle, the generated scenario indicates that the safety of the intelligent driving vehicle is insufficient, and the fourth part of the reward function will be rewarded with a positive value, R4. If the collision is caused by abnormal acceleration or deceleration of the autonomous vehicle, the scenario generation is considered unreasonable, and a penalty of negative value, R4, is applied to reduce the occurrence of such situations.

[0074]

[0075] In formula (4), δ c Indicates the collision risk factor, V p represents the speed of the autonomous vehicle that collides with the vehicle under test, E ego To define the cause of the collision caused by the tested vehicle, E agent To define the causes of collisions caused by autonomous vehicles.

[0076] Step 4: Initialize the neural network parameters and build the policy network and value network:

[0077] Step 4.1: Initialize the decay coefficient to γ ​​and the capacity of the experience replay pool to n; discretize the highway ramp scene time by time interval Δt;

[0078] Step 4.2: Construct the policy network and value network and initialize the network weight parameters and bias parameters:

[0079] The policy network consists of two fully connected networks, which output the Gaussian distribution of the mean and variance of continuous actions respectively. It also contains a discrete action network, which outputs the discrete data represented by the execution strategy. The value network outputs the value function of the current state, with the structure as follows: Figure 2 As shown;

[0080] When different strategies are applied to policy gradient learning, the policy network not only interacts with the environment to generate training data, but also uses sample training to update network parameters. All parameters of the policy network during training are denoted as θ, π θ is the strategy when being trained; all parameters of the strategy network when interacting with the environment are recorded as θ′, π θ′ strategies for interacting with the environment;

[0081] Step 4.3: The policy network outputs the action parameters of the autonomous vehicle and generates samples after interacting with the environment and stores them in the experience pool;

[0082] Step 4.3.1: The process of the policy network getting the action is to convert the state parameter s of the i-th step i Input the input layer and use equations (5) and (6) to output the action parameter a of step i i The corresponding Gaussian distribution mean mu i and variance std i :

[0083] mu i =σ(W2×Relu(W1×s i +b1)+b2) (5)

[0084]

[0085] In formula (5) and formula (6), W1 is the weight array from the input layer to the hidden layer of the policy network, b1 is the bias array from the input layer to the hidden layer of the policy network, W2 and W3 are the weight arrays from the hidden layer to the output layer of the policy network, b2 and b3 are the bias arrays from the hidden layer to the output layer of the policy network, and θ′={W1,W2,W3,b1,b2,b3}; Relu is the activation function; σ is the tanh function;

[0086] Step 4.3.2: Calculate the action parameter a of step i using formula (7) i Take any value a within the action space x The corresponding probability p(a x ):

[0087]

[0088] According to the action space probability distribution p(a x ) Randomly get the action parameter a of step i i ;

[0089] Define the state parameter s at step i i The policy π of the policy network when it is trained θ Select the action parameter a of step i i The probability of π(a i |s i ,θ); the state parameter s at step i i The strategy π of the policy network when interacting with the environment θ' Select the action parameter a of step i i The probability of π(a i |s i ,θ');

[0090] Step 4.3.3: The autonomous vehicle executes the action parameter a in step i i , and according to the reward function of step 3, calculate the reward r of step i i ; After the time interval Δt, calculate the state parameter s of the autonomous driving vehicle at step i+1 i+1 ; will (s i ,a i ,s i+1 ,r i ) is stored as a sample in the experience pool;

[0091] Step 4.4: If the number of samples in the experience pool reaches the upper limit n, proceed to step 4.5; otherwise, assign i + 1 to i and return to step 4.3;

[0092] Step 4.5: Randomly extract a batch of samples from the experience pool to train the policy network and value network; calculate the gradient of the policy network corresponding to any i-th sample Gradient J with value network i (ω);

[0093] Step 4.5.1: Set the s of the i-th sample i and s i+1 are input to the value network, and the value network outputs the state parameter s i The state value V ω (s i ) and s i+1 The state value V ω (s i+1 ), and thus use formula (8) to calculate the advantage function estimate

[0094] In formula (8), r i+1 represents the i+1 step reward, represents the expected value of the environment interaction strategy π θ' ;

[0095] Step 4.5.2: Calculate the ratio r i (θ) of π(a i |s i , θ) and π(a i |s i , θ') of the i-th sample, so as to calculate the gradient of the i-th sample in the policy network

[0096]

[0097] In formula (9), ε is a hyperparameter, and clip(r(θ), 1-ε, 1+ε) represents an operation function for limiting r i (θ) in the range of [1-ε, 1+ε];

[0098] Step 4.5.3: Calculate the value network gradient J i (ω) of the i-th step sample by formula (10):

[0099]

[0100] In formula (10), ▽ ω V(s i ) is the gradient of the state value V ω (s i+1 ), and (r i+1 +rV ω (s i+1 )-V ω (s i ) is the TD error, which indicates that the future selection tendency of the current action should be strengthened if the error is positive, and the future selection tendency of the current action should be weakened if the error is negative;

[0101] Step 4.6: Based on the stochastic gradient descent method, the gradients of the policy network and the value network corresponding to all samples are optimized and solved to update the parameters θ of the policy network and the parameters ω of the value network; so as to obtain the optimal left-turn-prevention model containing the optimal parameters θ * of the policy network and the optimal parameters ω * of the value network;

[0102] Step 4.7: input the state parameter s i of the current scene at the i-th step into the optimal left-turn-prevention model, and output the optimal action parameter c * iThe optimal decision parameter d of the i-th step * i As two autonomous vehicles agent1, agent2 make acceleration and action decisions at the i-th step to prevent the test vehicle ego from changing lanes at the i-th step, thereby controlling the autonomous vehicles to game with the intelligent vehicle under the left-turn working condition at the intersection.

[0103] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0104] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to execute the steps of the above method.

Claims

1. A method for generating dynamic behaviors of multiple autonomous driving vehicles based on the H-PPO hierarchical algorithm, characterized in that: This method is used in an autonomous driving scenario where two autonomous vehicles (agent 1 and agent 2) traveling in the same lane prevent the test vehicle ego in a different lane from turning left at an intersection. It includes the following steps: Step 1: Get the acceleration of the autonomous driving vehicle agent 1 in front and the acceleration of the autonomous vehicle agent2 located behind Upper limit and lower limit , and and As an action parameter; Step 2: Establish a Frenet coordinate system for the vehicle, with the center of the vehicle as the origin, the direction of the vehicle traveling along the road as the longitudinal direction, and the direction perpendicular to the road as the transverse direction. Based on the environmental information collected by the autonomous vehicle and combined with the dynamic model, a state parameter is constructed ,in, represents the lateral coordinate of the autonomous driving vehicle agent1 in front, represents the lateral coordinate of the autonomous driving vehicle agent2 located behind, represents the longitudinal coordinate of the autonomous driving vehicle agent1 in front, represents the longitudinal coordinate of the autonomous driving vehicle agent2 located behind, Indicates the horizontal coordinate of the vehicle ego under test, Indicates the longitudinal coordinate of the vehicle under test ego, represents the speed of the autonomous driving vehicle agent1 in front, represents the speed of the autonomous driving vehicle agent2 located behind, Indicates the speed of the vehicle ego being tested, represents the acceleration of the autonomous driving vehicle agent1 in front, represents the acceleration of the autonomous driving vehicle agent2 located behind, Indicates the heading angle of the test vehicle ego; Step 3: Construct a reward function consisting of four components for the game between the two autonomous vehicles (agent 1 and agent 2) in the same lane and the test vehicle (ego). Step 4: Construct a deep neural network, including a policy network and an evaluation network, wherein the policy network includes a continuous action network and a discrete action network; Get the historical status parameter set ,in, Indicates the The status of the step, n Indicates the upper limit of the experience pool capacity; Set the historical state parameter set Input into the policy network for processing to obtain the action parameter set ,in, Indicates the Step action, and ; is the continuous action parameter, is the discrete action parameter, Indicates that two autonomous driving vehicles agent1 and agent2 are in the The action parameters of the step, Indicates that two autonomous driving vehicles agent1 and agent2 are in the The decision parameter of whether to enter the intersection; Will As the first i samples are stored in the experience pool; among them, Indicates the state of step i+1; Step 5: Based on the historical state parameter set and action parameters , using the H-PPO algorithm to train the deep neural network and calculate the reward function to obtain the optimal left-turn prevention model; Step 6: Set the current scene in Step status parameters Input the optimal left-turn prevention model and output the current scenario in the The optimal action parameters of the step With the The optimal decision parameters for the first step , as two autonomous driving vehicles agent1 and agent2 in the Step acceleration and action decision to prevent the test vehicle ego from Change lanes.

2. The method for generating dynamic behaviors of multiple autonomous driving vehicles based on the H-PPO hierarchical algorithm according to claim 1, characterized in that: The step 3 includes: Step 3.1: Use Equation (5) to construct the reward function of the first part : (5) In formula (5), is the starting position of the tested vehicle ego entering the intersection, is the position where the tested vehicle ego leaves the intersection, is the state of the tested vehicle ego before entering the intersection, is the state of the vehicle ego just entering the intersection, is the state of the tested vehicle ego when it is in the middle of the intersection, is the state of the tested vehicle ego when it leaves the intersection; represents the decision action of the autonomous vehicle agent 1, Represents the decision action of the autonomous vehicle agent2; Step 3.2: Use Equation (6) to construct the reward function of the second part : (6) In formula (6), is a gaming indicator; Step 3.3: Use Equation (7) to construct the reward function of the third part : (7) In formula (7), To obtain and The number with the smaller median value, To obtain and The number with the larger median value; Step 3.4: Use Equation (8) to construct the reward function of the fourth part : (8) In formula (8), Indicates collision risk factors, Indicates the speed of the autonomous driving vehicle agent1 or agent2 that collides with the vehicle under test ego, To define the collision caused by the ego of the tested vehicle, To define the collision caused by the autonomous vehicle agent1 or agent2.

3. The method for generating dynamic behaviors of multiple autonomous driving vehicles based on the H-PPO hierarchical algorithm according to claim 1, characterized in that: The step 4 includes: The evaluation network consists of two fully connected layers and an evaluation output layer, and the activation functions between all fully connected layers are ReLU functions; let the parameters of the evaluation network be , the input of the evaluation network is the normalized current state, and the output is the value of the current state; The continuous action network outputs a neural network of the mean part and a neural network of the variance part, and the discrete action network outputs the probability value of executing each action; wherein, the neural network of the continuous network outputs the mean part includes: two layers of fully connected layers and a mean output layer, and the activation function of the mean output layer is a tanh function; the neural network of the output variance part includes: two layers of fully connected layers and a variance output layer, and the activation function of the variance output layer is a softplus activation function; the activation function between all fully connected layers is a Relu function, and the discrete action network includes a fully connected layer, which outputs the probability value of each discrete action execution; Let the parameters of the continuous action network be , the parameters of the discrete action network are The input of the continuous action network and the discrete action network are both normalized current states, and the output of the continuous action network is the current action , the output of the discrete action network is the current decision ; Among them, ε and std represent mean and variance respectively; N represents Gaussian distribution, and ~ represents compliance; Define the training of the policy network as , define the environment interaction parameters of the policy network as , the environmental interaction strategy of the policy network is .

4. The method for generating dynamic behaviors of multiple autonomous driving vehicles based on the H-PPO hierarchical algorithm according to claim 3, characterized in that: The step 5 comprises: Step 5.1: Draw any i samples, and thus the advantage function estimator shown in formula (9) is used to calculate the i The estimated value of the sample : (9) In formula (9), For the i In the sample The state value obtained after inputting the evaluation network, To be the first i In the sample The state value obtained after inputting the evaluation network; is the reward discount coefficient; Step 5.2: Define the state The strategy below Select Step action The probability of , in the state quantity The strategy below Select Step action The probability of ,definition and The ratio is ; Represents the network parameters of the policy network at step i; Step 5.3: Use formula (10) to get the network parameters of the policy network at step i+1 : (10) In formula (10), are the parameters of the policy network The gradient operator, is the range limiting function, which is to Restricted to arrive Within the range, is the set threshold, For expectations; Step 5.4: Use formula (11) to get the network parameters of the evaluation network at step i+1 : (11) In formula (11), To evaluate network parameters The gradient operator, To evaluate the network parameters of the network at step i; Step 5.5: After assigning i+1 to i, return to step 5.1 and execute sequentially until the maximum number of steps max is reached, forming the parameter set of the optimal lane change prevention model.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the dynamic behavior generation method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic behavior generation method according to any one of claims 1 to 4 are executed.

Images