An automatic driving optimization control method based on adversarial reinforcement learning

Abstract

The application relates to an automatic driving optimization control method based on adversarial reinforcement learning, which comprises the following steps: designing key scene factors of an adversarial agent control part to construct an adversarial scene; training a vehicle by using the adversarial scene to obtain an optimized driving algorithm; and applying the optimized driving algorithm to a vehicle controller to control the driving state of the vehicle. Compared with the prior art, the application designs a first-generation driving strategy of reinforcement learning for the vehicle in a given scene, adopts an incomplete zero-sum game to construct an adversarial process, generates a plurality of adversarial agents to perform adversarial operations on the driving algorithm of the vehicle, and continuously iteratively optimizes the robustness of the driving algorithm of the vehicle. Therefore, the automatic driving algorithm with enhanced robustness is obtained through iterative optimization based on the adversarial reinforcement learning mode, so that the efficiency and scene robustness of the automatic driving control are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving optimization control method based on adversarial reinforcement learning. BACKGROUND

[0002] Automatic driving technology helps to reduce environmental pollution, improve traffic congestion and reduce traffic accidents, and is a promising research direction at present. Driving algorithms represented by deep reinforcement learning have become a research hotspot in the field of automatic driving in recent years, but a large number of studies have shown that the robustness of the reinforcement learning driving algorithm in scene migration is difficult, and the performance is poor after migration to unknown scenes.

[0003] The current main solution is to retrain the algorithm in a new adversarial scene, that is, to find adversarial scenes that will make the driving vehicle have poor performance through data-driven or mathematical optimization methods, and then train the driving algorithm in the adversarial scene to obtain better robustness and portability. Among them, the data-driven method is to sample adversarial scenes in natural traffic data streams through a probability distribution model, which can obtain more realistic training scenes; and the mathematical optimization method is to parameterize the scene, optimize the parameters through mathematical modeling and intelligent algorithms, and obtain the adversarial scene, which has high testing efficiency and low cost.

[0004] As can be seen, in the previous self-learning driving algorithm optimization method, the above two types of solutions have certain defects: the data-driven method needs to obtain a large amount of real traffic data, and the proportion of effective adversarial scenes searched is low and time-consuming, resulting in high cost of obtaining a high-robustness reinforcement learning driving algorithm; the mathematical optimization method sets up a zero-sum game adversarial agent control scene parameter, trains the adversarial agent to interfere with the driving process of the vehicle based on deep reinforcement learning, and then fixes the strategy of the adversarial agent to train the self-learning driving algorithm of the vehicle, and iteratively optimizes the robustness of the self-learning driving algorithm of the vehicle. The solution can effectively reduce the optimization cost of the reinforcement learning driving algorithm, but since it designs a one-to-one zero-sum game adversarial learning process, it will lead to aggressive adversarial behavior of the adversarial agent, which does not conform to the reality of traffic, and the adversarial strategy of a single adversarial agent is limited, which will gradually fall into a local optimum, and it is also difficult to provide a rich retraining environment, which can only improve the robustness of the driving algorithm in a specific adversarial scene. Ultimately, when the driving algorithm is applied to control the vehicle, it cannot solve the decision planning problem of the vehicle in complex scenes, and cannot adapt well to a variety of different driving scenes. SUMMARY

[0005] The present application aims to overcome the defects of the prior art and provides an automatic driving optimization control method based on adversarial reinforcement learning, which is based on adversarial reinforcement learning to iteratively optimize the robustness-enhanced automatic driving algorithm, thereby improving the efficiency and scene richness of automatic driving control.

[0006] The object of the present application can be achieved by the following technical solution: an automatic driving optimization control method based on adversarial reinforcement learning, comprising the following steps:

[0007] S1, design the key scene factors of the adversarial agent control part to construct an adversarial scene;

[0008] S2, train the vehicle using the adversarial scene to obtain an optimized driving algorithm;

[0009] S3, apply the optimized driving algorithm to the vehicle controller to control the driving state of the vehicle.

[0010] Further, the step S1 is specifically achieved by generating a plurality of non-zero-sum game adversarial agents, which are in confrontation with the ego vehicle, and the confrontation target of the ego vehicle and the adversarial agents is:

[0011]

[0012] Wherein, μ ego and μ sat are the policy functions of the ego agent and the scene agent respectively, is the optimization target, G t is the cumulative reward, S t is the environmental state random variable, γ is the reward discount, R n is the reward random variable provided by the environment, which is related to the environment, the ego agent, the scene agent and other scene elements.

[0013] Further, the optimization target of the policy function μ ego of the ego agent is to obtain the maximum cumulative reward of the optimized policy function; and the optimization target of the scene agent is to minimize the cumulative reward of the optimized policy function in order to confront the ego agent and make poor decisions.

[0014] Further, the reward function of the scene vehicle is

[0015]

[0016] Wherein, is the reward of the adversarial agent, is the reward of the ego vehicle, and R regis a regularization term optimization target, used to reward the adversarial agent to produce a more realistic scenario, and alpha is an adversarial control factor that controls the degree of adversality of the scenario, the larger the alpha, the greater the proportion of the adversarial term, the stronger the adversarial behavior of the agent, and vice versa, the greater the proportion of the regularization term, the more rational the behavior of the agent, and the more realistic the generated scenario.

[0017] Further, the step S1 specifically comprises the following steps:

[0018] S11, using the TD3 deep reinforcement learning algorithm of actor-critic architecture to design the adversarial agent, and designing the state value s i , an optional environment configuration action a i , the TD3 algorithm includes an action network μ(s i |θ μ ) and two policy networks and and their corresponding target networks μ'(s i |θ μ′ ), and

[0019] S12, the training steps of the adversarial agent are:

[0020] Randomly initialize the TD3 network θ μ , and and assign the parameters to the corresponding policy network, i.e. θ μ′ = θ μ′ ,

[0021] Obtain the environment state s i required by the scenario agent;

[0022] Design the neural network structure μ(s i |θ μ ) of the TD3 deep reinforcement learning, and output the optional environment configuration action according to the state value s i :

[0023]

[0024] where c is the symmetric truncation threshold of noise, a Low and a High are the truncation thresholds of actions;

[0025] After applying the action a i , the next environment state s i+1 is obtained, and the reward r i ' of each step is obtained according to the non-zero-sum game reward function ;

[0026] At this point, a single-step experience (s i ,a i ,s i+1 ,r i ) is obtained, and the experience is put into the experience pool D;

[0027] N experiences are sampled from the experience pool D, and the cost functions of the three networks are calculated:

[0028]

[0029]

[0030]

[0031] where y i is the target value, and the calculation method is:

[0032]

[0033] The gradient descent method is used to update the network parameters:

[0034]

[0035]

[0036]

[0037] where α μ and α Q are the learning rates of the actor network and the critic network, respectively;

[0038] The target network parameters θ μ′ are updated every fixed step d:

[0039] θ μ +(1-τ)θ μ′

[0040]

[0041] where τ∈(0,1) is the target network update proportion;

[0042] The above steps are repeated until the set step or the reward reaches the set value, and the training is stopped;

[0043] S13, test the success rate of the vehicle in the generated adversarial scene, if it is within the set threshold range, put the adversarial scene generated by the generated adversarial agent into the scene library L, otherwise discard the scene;

[0044] S14, repeat steps S11-S13 until the desired number N of scenes is generateds adversarial scenarios.

[0045] Further, the step S2 specifically provides the generated plurality of adversarial scenarios to the ego vehicle for training, and samples a different scenario each round of training of the ego vehicle to provide to the ego vehicle for training.

[0046] Further, the step S2 uses a priority success rate method to filter training scenarios from the scenario library, assuming there are n scenarios, the success rate of the ego vehicle in the n scenarios is counted in real time during the training process, and the scenarios are sorted from low to high according to the success rate, and the scenario S i rank(S i ) is recorded as rank(S i ).

[0047]

[0048] where δ∈Z is a random disturbance, and the normalized sampling probability P(i) of the scenario i is

[0049]

[0050] Further, the step S2 specifically includes the following process:

[0051] S21, first train the initial first generation reinforcement learning driving algorithm of the ego vehicle, and then initialize the arrays T[N] and A[N] to record the usage frequency of the N scenarios in the scenario library L and the success rate of the ego vehicle in the scenarios, respectively;

[0052] S22, sort the scenarios from low to high according to A[N] to obtain the rank(S i ) of each S i , i = 0, 1,..., N, and calculate the sampling probability:

[0053]

[0054] where δ∈Z is a random disturbance;

[0055] S23, sample the scenario S i according to the calculated probability value;

[0056] S24, the ego vehicle completes a round of training in the scenario S i .

[0057] S25, update the success rate of the scenario S i , and update the usage frequency T[i] = T[i] + 1 of the scenario S i .

[0058] S26, repeat steps S22-S25 until the success rate of the vehicle in each scenario reaches the desired value, and thus obtain the driving algorithm of the next generation of the vehicle;

[0059] S27, repeat the next round of optimization until the desired number of optimizations N is reached opt .

[0060] Further, the specific process of training the initial first generation of reinforcement learning driving algorithm of the vehicle in step S21 is:

[0061] 1. Select a deep reinforcement learning algorithm according to the task scenario, design state value s i , optional environment configuration action a i , and reward function R ego ;

[0062] 2. Select a training scenario and place the vehicle in the scenario for training;

[0063] 3. Obtain the state s i of the vehicle and the environment;

[0064] 4. The driving algorithm outputs the optional environment configuration action a i according to the state value s i ;

[0065] 5. Calculate the reward value r ego of the current environment configuration action according to the reward function R i , the state s i+1 after executing the action a i , the higher the reward obtained by the reinforcement learning agent of the vehicle, the more beneficial it is to complete the task;

[0066] 6. Repeat steps 3-5 until the vehicle reaches the expected success rate in the scenario or the maximum training step length.

[0067] Further, the success rate of the scenario S i in step S25 is specifically:

[0068]

[0069] Wherein, A[i] is the success rate corresponding to the scenario S i .

[0070] Compared with existing technologies, this invention proposes an autonomous driving algorithm optimization method based on adversarial reinforcement learning to overcome the robustness problem of reinforcement learning algorithms and the shortcomings of low efficiency and low scenario richness of two existing optimization techniques. In a given scenario, a first-generation reinforcement learning driving strategy is designed for the vehicle. Then, an adversarial process is constructed using an incomplete zero-sum game, generating multiple adversarial agents to compete against the vehicle's driving algorithm, continuously iterating and optimizing the robustness of the driving algorithm. The method proposed in this invention is universally applicable to learning-based driving algorithms, represented by reinforcement learning, and is not limited to overtaking and lane-changing scenarios. It can also be applied to various behavioral decisions of autonomous vehicles, such as parking, turning at intersections, and merging / exiting at highway intersections. By designing a highly robust reinforcement learning driving algorithm, the driving algorithm has better adaptability when transitioning between scenarios, thereby effectively improving the efficiency and scenario richness of autonomous driving control. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0072] Figure 2 This is a schematic diagram of a lane-changing overtaking scenario in the embodiment;

[0073] Figure 3 This is a schematic diagram illustrating the impact of the value of the adversarial control factor on the generated scene in the embodiment;

[0074] Figure 4a This is a schematic diagram of the return data for the vehicle during the robustness improvement phase in the embodiment.

[0075] Figure 4b This is a schematic diagram illustrating the success rate and failure rate of the vehicle during the robustness improvement phase in this embodiment.

[0076] Figures 5a-5d This example illustrates the performance of the vehicle's intelligent agent before and after optimization in different scenarios.

[0077] Figure 6 This is a schematic diagram comparing the success rates of different generations of driving algorithms in various test scenarios in the embodiments. Detailed Implementation

[0078] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0079] Example

[0080] like Figure 1 As shown, an autonomous driving optimization control method based on adversarial reinforcement learning includes the following steps:

[0081] S1. Design key scenario factors for the control part of the adversarial intelligent agent to construct adversarial scenarios;

[0082] S2, training the vehicle using the adversarial scene to obtain an optimized driving algorithm;

[0083] S3, applying the optimized driving algorithm to the vehicle controller to control the driving state of the vehicle.

[0084] The embodiment applies the above technical solution, and the main content includes:

[0085] I. Scene generation

[0086] To solve the robustness problem of the reinforcement learning driving algorithm in scene migration, the technical solution proposes an automatic driving algorithm design method based on adversarial reinforcement learning, the core of which is to design an adversarial agent to control the key scene factors of the scene. By generating such multiple non-zero-sum game adversarial agents to fight the host vehicle, the robustness of the host vehicle driving algorithm to different scenes is improved. The adversarial goal of the host vehicle and the adversarial agent is

[0087]

[0088] Where μ ego and μ sat are the policy functions of the host agent and the scene agent respectively, is the optimization goal, G t is the cumulative reward, S t is the environment state random variable, γ is the reward discount, and R n is the reward random variable provided by the environment, which is related to the environment, the host agent, the scene agent and other scene elements. After the host agent makes a decision, the cumulative reward G t obtained is higher, which means that the decision made is better, and the agent will select the action corresponding to the maximum state-action value to obtain the maximum cumulative reward. Therefore, the optimization goal of the policy function μ ego of the host agent is to optimize the policy function to obtain the maximum cumulative reward. The scene agent optimizes the policy function to minimize the cumulative reward in order to counter the host vehicle, so that the host agent makes a poor decision.

[0089] When the policy of the host vehicle and the scene agent both use reinforcement learning algorithm, the reward function of the scene vehicle can be derived from formula (1) as

[0090]

[0091] Where is the reward of the adversarial agent, For the reward of the ego vehicle, it can be seen that the reward of the adversarial agent is just the opposite of the ego vehicle, which is a zero-sum game process. In order to avoid aggressive confrontation behavior, a relaxation reward term is introduced in formula (2) to construct an incomplete zero-sum game confrontation process, and the reward function is

[0092]

[0093] wherein R reg is a regularization term optimization goal, which is used to reward the adversarial agent to produce a more realistic scenario, such as making the dynamic environment vehicle not intentionally collide with the ego vehicle or the curvature of the static curve not too large, etc. Alpha is an adversarial control factor that controls the degree of confrontation in the scene, so that the adversarial agent produces different confrontation behaviors, and then identifies the multi-class weaknesses of the driving algorithm. The greater the alpha, the greater the proportion of the confrontation term, and the stronger the adversarial behavior of the agent. Conversely, the greater the proportion of the regularization term, the more rational the behavior of the agent, and the more realistic the generated scene.

[0094] After that, a suitable deep reinforcement learning algorithm is selected to design and train the decision function μ sat of the adversarial agent, which can be done. After optimization, scenes that are too aggressive or moderate need to be removed. The former is not solvable for the ego vehicle driving algorithm, and the latter is too low in difficulty and lacks training value. For example, only the scenes that make the success rate of the ego vehicle task between 10% and 90% are added to the scene library, and the others are discarded.

[0095] By selecting different network initialization parameters and the value of the adversarial control factor alpha, each adversarial agent can be trapped in different local optimal places and exhibit different confrontation strategies to provide a rich confrontation environment for the ego vehicle. In addition, the training step of these adversarial agents should not be too long, usually less than one order of magnitude of the ego vehicle, which can improve the efficiency of scene generation and avoid the strategy function of the adversarial agent tending to the global optimum.

[0096] II. Robustness improvement

[0097] The main task of the robustness improvement stage is to provide multiple adversarial scenes obtained in the scene generation stage to the ego vehicle for training. A different scene is sampled for the ego vehicle to train in each round of training, which improves the robustness of the ego vehicle in different scenes.

[0098] In order to improve the efficiency of training, low success rate scenes should be used more, so the priority success rate method is used to select training scenes from the scene library. Specifically, there are a total of n scenes, and the task success rate of the ego vehicle in n scenes is calculated in real time during the training process. According to the task success rate from low to high, the ranking of scene S i is recorded as rank(S i ), then S iThe sampled probability p(i) is

[0099]

[0100] where δ∈Z is a random perturbation. The normalized sampling probability P(i) of scene i is

[0101]

[0102] III. Iterative optimization

[0103] The scene generation and robustness improvement are repeated continuously to obtain the reinforcement learning driving algorithm of the ego vehicle of multiple generations. The driving algorithm optimized multiple times can better cope with different scenes and has better robustness when the scene is transferred.

[0104] The implementation steps of the embodiment are mainly divided into three parts, which are described in detail as follows:

[0105] Step 1, training the first generation of reinforcement learning driving algorithm of the ego vehicle

[0106] 1. According to the task scene, select a suitable deep reinforcement learning algorithm, design the state value s i , the optional environment configuration action a i , and the reward function R ego ;

[0107] 2. Select a suitable training scene and place the ego vehicle in the scene for training;

[0108] 3. Obtain the state s i of the vehicle and the environment;

[0109] 4. The driving algorithm outputs the optional environment configuration action a i according to the state value s i ;

[0110] 5. Calculate the reward value r ego of the current environment configuration action according to the reward function R i , and the state s i+1 after executing the action a i . The higher the reward obtained by the ego vehicle reinforcement learning agent, the more beneficial it is to complete the task;

[0111] 6. Repeat steps 3-5 until the ego vehicle reaches the expected success rate or the maximum training step length in the scene.

[0112] Step 2, scene generation

[0113] 1. Design an adversarial agent using the TD3 deep reinforcement learning algorithm with an actor-critic architecture (other suitable deep reinforcement learning algorithms can be selected according to the specific scene), and design the state value si Optional environment configuration action a i The TD3 algorithm includes an action network μ(s) i |θ μ ) and two policy networks and and their corresponding target networks μ′(s) i |θ μ′ ), and The training steps for adversarial agents are as follows:

[0114] 1.1. Randomly initialize the TD3 network θ μ , and And assign the parameters to the corresponding policy network, i.e., θ. μ′ =θ μ′ ,

[0115] 1.2. Obtaining the environmental state s required by the scene's intelligent agent i ;

[0116] 1.3. Design of the neural network structure μ(s) for TD3 deep reinforcement learning i |θ μ According to the state value s i Output optional environment configuration actions

[0117]

[0118] Where c is the symmetric cutoff threshold for noise, a Low and a High This is the truncation threshold for actions. The number of nodes in the input layer of the neural network is determined by the number of attributes of the input state, and the number of nodes in the output layer is determined by the number of actions that can be configured in the environment. The hidden layers are used to extract environmental features and make decisions, and their structure and parameters need to be adjusted according to factors such as environmental complexity and computing power.

[0119] 1.4. Applying action a i Then the next environment state s is obtained. i+1 The reward r for each step i ′ is based on the reward function of a non-zero-sum game. This was obtained. Thus, single-step experience (s) was gained. i ,a i ,s i+1 ,r i Add this experience to experience pool D;

[0120] 1.5. Sample N experiences from the experience pool D and calculate the cost functions of the three networks.

[0121]

[0122]

[0123]

[0124] where y i is the target value, calculated as

[0125]

[0126] 1.6. Update network parameters using gradient descent method

[0127]

[0128]

[0129]

[0130] where α μ and α Q are the learning rates of actor network and critic network respectively;

[0131] 1.7. Update target network parameters every fixed steps d

[0132] θ μ′ = τθ μ + (1 - τ)θ μ′

[0133]

[0134] where τ ∈ (0, 1) is the target network update proportion;

[0135] 1.8. Repeat steps 1.2-1.6 until a set step size (usually less than one order of magnitude of the training step size of the ego vehicle) is reached or the reward reaches a set value, then stop training;

[0136] 2. Test the success rate of the ego vehicle in the generated adversarial scenarios, if it is between 10% and 90%, put the adversarial scenarios generated by the adversarial agent into the scenario library L, otherwise discard the scenario;

[0137] 3. Repeat steps 1-2 until the desired number N s of adversarial scenarios is generated.

[0138] Step3, Robustness improvement

[0139] 1. Initialize arrays T[N] and A[N] to record the number of times each of the N scenarios in the scenario library L is used and the success rate of the ego vehicle in the scenario respectively;

[0140] 2. According to the order of A[N] from low to high, get each S i rank(S i ), i = 0, 1,..., N, and calculate the sampling probability

[0141]

[0142] Random disturbance δ ∈ Z takes a small range around 0, for example {-2, -1, 0, 1, 2};

[0143] 3. According to the calculated probability value, sample the scene S i ;

[0144] 4. The vehicle completes a round of training in the scene S i ;

[0145] 5. Update the success rate of scene S i ;

[0146]

[0147] Update the usage times T[i] = T[i] + 1 of scene S i ;

[0148] 6. Repeat steps 2-5 until the success rate of the vehicle in each scene reaches the expected value, and thus obtain the driving algorithm of the next generation of the vehicle;

[0149] 7. Repeat Step 2 for the next round of optimization until the desired number of optimizations N opt is reached.

[0150] To verify the effectiveness of the present technical solution, an overtaking algorithm is designed in this embodiment and simulated in Carla.

[0151] The scene is set up as shown in Figure 2 . For this logical scene, the static environment includes two adjacent straight lanes and other irrelevant houses and trees, and the dynamic elements are the front and adjacent vehicles controlled by the scene agent, the two red vehicles drive slowly and have a certain distance, and the vehicle tries to change lanes at the right time to overtake the two vehicles. This embodiment simplifies the scene appropriately, and sets that the adjacent and front vehicles have no lane changing behavior and only drive in their current lane. The logical scene setting is shown in the table, involving vehicle initial position, speed and other parameters and related motion constraints. The initial speed of the vehicle is set to conform to the Gaussian distribution (5, 1) m / s, and the initial speed of the adjacent and front vehicles conforms to the Gaussian distribution (3,1) m / s. According to the traffic regulations of the People's Republic of China, a three-second distance should be maintained when driving on normal roads, and the speed of the vehicle is (5,1) m / s, the initial longitudinal distance between the vehicle and the front vehicle is set to be a Gaussian distribution (21,2.0) m, which can ensure that more than 99.73% of the initial conditions meet the regulations. The initial longitudinal distance between the vehicle and the adjacent vehicle is set to be (3,2.0) m. To ensure the comfort of the motion process, the speed range of each vehicle is set to be 0-10 m / s, and the acceleration is not more than 1 m / s 2 .

[0152] The vehicle driving algorithm is set, and the vehicle adopts a hierarchical control structure. The upper decision layer uses the PPO deep reinforcement learning algorithm, and the lower control layer uses the PID algorithm.

[0153] The state space of the decision layer is the continuous three-frame laser radar point cloud with a height of 281px and a width of 141px, as well as the vehicle speed v ego , angular velocity w ego data. The action space is divided into 9 discrete actions as shown in Table 1.

[0154] Table 1 Vehicle agent action space

[0155]

[0156] The PPO network structure is shown in Table 2.

[0157] Table 2 Network structure of vehicle PPO algorithm

[0158]

[0159] a The value of the actor and critic network structure is a tuple, where the elements from left to right represent the number of neurons in each hidden layer from front to back in the network

[0160] The reward is designed as

[0161]

[0162] The vehicle successfully changes lanes and overtakes from the middle of the two vehicles and receives a reward, and is punished if a collision occurs. However, the reward and punishment returns are too sparse. In order to better train, three dense returns are introduced. The default item encourages the vehicle to move faster, which is a description of the vehicle's driving behavior in the functional scenario. In addition, when the longitudinal distance between the vehicle and the remaining vehicles in the same lane is too small or the collision time is too small, a certain penalty is given to improve safety.

[0163] The lower controller of the ego vehicle adopts a PID controller, which can execute the decision actions such as constant speed, acceleration, deceleration and lane changing given by the decision layer.

[0164] The key parameters of the scenario agent are set to be the speeds of the two red cars, so the adversarial agent hinders the ego vehicle from overtaking by controlling the speeds of the two red cars, trying to make the ego vehicle make a wrong decision, and also needs to keep a proper distance between the two red cars, otherwise the success rate of the ego vehicle task will be 0. The state input of the adversarial agent is a ten-dimensional vector:

[0165] (Δx lead ,Δy lead ,Δyaw lead ,Δv x,lead ,Δv y,lead ,Δx near ,Δy near ,Δyaw near ,Δv x,near ,Δv y,near )

[0166] Each element represents the longitudinal distance, lateral distance, yaw angle, longitudinal speed and lateral speed of the front vehicle relative to the ego vehicle, and the longitudinal distance, lateral distance, yaw angle, longitudinal speed and lateral speed of the adjacent vehicle relative to the ego vehicle. According to the logical scenario, the front vehicle and the adjacent vehicle have no lane changing behavior, only the longitudinal speed changes. Since the TD3 deep reinforcement learning algorithm is used to deal with continuous input and continuous output decision problems, there is no need to discretize the action. The action of the scenario agent is a table of binary continuous output.

[0167] Table 3 Action setting of adversarial agent

[0168]

[0169] The TD3 scenario agent accepts a ten-dimensional state vector input, outputs a binary continuous action according to the state vector, which is the speed increment of the front vehicle and the ego vehicle respectively, and transmits the action to the lower PID controller of the scenario vehicle for distributed control. The key parameters of the TD3 scenario agent are shown in Table 4.

[0170] Table 4 Key parameter settings of TD3 network

[0171]

[0172] a The value of the aclor and critic network structure is a tuple, where the elements from left to right represent the number of neurons in each hidden layer from front to back in the network

[0173] The reward function of the scenario agent is where For the reward function of the ego vehicle in equation (3), the regularization term is

[0174]

[0175] The regularization term is used to relax the adversarial behavior, so that the driving behavior of the scenario vehicle is more consistent with the functional scenario and the logical scenario, but not too aggressive. Since the functional scenario is that the ego vehicle tries to pass the two scenario vehicles by slow driving, setting the scenario vehicle speed lower than 6 m / s provides a positive reward, encouraging the scenario agent to control the scenario vehicle to drive slowly. The acceleration term encourages the vehicle to drive smoothly, ensuring the comfort of the vehicle. The collision term punishes the vehicle that is necessary for normal driving. Considering that the scenario is too simple when the distance between the scenario vehicles is too far, and it will lead to the scenario being incomprehensible to the ego vehicle when the distance is too close, the distance between the scenario vehicles is expected to be in a suitable interval, and a density reward based on the expected distance is set, where the expected distance 16 m is a suitable value found through experiments.

[0176] Simulation results: After one iteration optimization, the second generation driving algorithm of the ego vehicle can be obtained. By setting a∈[0.5, 1.0] in the scenario generation stage, a series of adversarial scenarios are obtained, and the success rate and failure rate of the ego vehicle in these scenarios are as shown in Figure 3 When a∈[0.7, 0.9], the success rate of the ego vehicle is between 10% and 90%, and these scenarios can be put into the scenario library for training of the ego vehicle.

[0177] In the robustness improvement stage, the reward of the ego vehicle and the success rate and failure rate are as shown in Figure 4a and 4b From the figure, it can be seen that as the retraining proceeds, the reward function of the ego vehicle is constantly rising, and better performance is obtained, and the success rate of the ego vehicle in various scenarios in the scenario library is constantly rising, and finally reaches about 90%, and the second generation driving algorithm is obtained.

[0178] Here, the scenarios of a=0.8 and a=0.9 are selected to show the performance of the driving algorithm before and after optimization, as shown in Figures 5a-5d The first generation driving algorithm is too late to change lanes in the adversarial scenario, resulting in a collision, while the second generation driving algorithm overcomes the defect of slow lane changing and can find the right opportunity to change lanes, making the correct decision to change lanes and overtake.

[0179] This embodiment has a total of 3 rounds of iteration optimization, and the first to fourth generation driving algorithms are obtained, which are tested in 8 test scenarios, and each scenario is tested 60 times, and the results are as shown in Figure 6 The specific statistical data is shown in Table 5.

[0180] Table 5 Performance of each generation of driving algorithm in test scenarios

[0181]

[0182] With the increase of the number of iterations, the lane changing time of the ego vehicle is constantly reduced, the average success rate is constantly increased, and the standard deviation of the success rate is constantly reduced. The average performance of the reinforcement learning driving algorithm and the robustness to the scene are effectively improved.

[0183] In summary, for autonomous vehicles, reinforcement learning driving algorithms can effectively solve the decision planning problem in complex scenarios and avoid manual design. However, reinforcement learning driving algorithms have robustness problems when the scene is transferred, and they are not very adaptable to unknown scenes. The technical solution designs an automatic driving algorithm design method based on adversarial reinforcement learning. The core of this method is to design a non-zero-sum game adversarial process to adaptively generate multiple adversarial scenes for the ego vehicle driving algorithm to retrain, and to continuously improve the robustness of the driving algorithm through iterative optimization, so that the algorithm has better adaptability to different scenes. The case of this solution applies the design method to design a lane changing and overtaking driving algorithm and simulates it in Carla. The algorithm after multiple iterations can achieve good performance in different test scenarios. This design method is universal for learning-based driving algorithms represented by reinforcement learning, and is not limited to overtaking and lane changing scenarios. It can also be applied to various behavior decisions of autonomous vehicles such as parking, intersection turning, and highway merging and merging, to design reinforcement learning driving algorithms with high robustness, so that the driving algorithm has better adaptability when the scene is transferred.

Claims

1. An autonomous driving optimization control method based on adversarial reinforcement learning, characterized in that, Includes the following steps: S1. Design key scenario factors for the control part of the adversarial intelligent agent to construct adversarial scenarios; S2. Train the vehicle using adversarial scenarios to obtain an optimized driving algorithm; S3. Apply the optimized driving algorithm to the vehicle controller to control the vehicle's driving status accordingly; Step S1 specifically involves generating multiple non-zero-sum game adversarial agents, which then engage in adversarial combat against the vehicle. The objective of the adversarial combat between the vehicle and the adversarial agents is: Where, μ ego and μ sat These are the policy functions for the vehicle-specific intelligent agent and the scene-specific intelligent agent, respectively. It is the optimization objective, G t To accumulate returns, S t Let R be a random variable representing the environmental state, γ be the reward discount, and R be a random variable representing the environmental state. n The random variable of the reward provided to the environment is related to the environment, the vehicle intelligent agent, the scene intelligent agent, and other scene elements; The policy function μ of the vehicle's intelligent agent ego The optimization objective is to maximize the accumulated reward of the optimization policy function; the optimization objective of the scenario agent in order to counter the vehicle is to minimize this accumulated reward by optimizing the policy function. Step S2 specifically involves providing the generated multiple adversarial scenarios to the vehicle for training, and sampling a different scenario for the vehicle to train in each round of training. Step S2 uses a success rate-priority method to select training scenarios from the scenario library. Assuming there are n scenarios in total, the vehicle's task success rate in these n scenarios is calculated in real-time during training. Scenarios are then sorted from lowest to highest success rate and denoted as Sscenario. i The ranking is rank(S) i ), then S i The probability p(i) of being sampled is Where δ∈Z represents random perturbation, and the normalized sampling probability P(i) of scene i is...

2. The autonomous driving optimization control method based on adversarial reinforcement learning according to claim 1, characterized in that, The reward function for the vehicle in the scenario is: in, To counteract the rewards for intelligent agents, As a reward for this car, R reg α is the optimization objective of the regularization term, used to reward adversarial agents for generating scenarios that are more realistic. α is an adversarial regulation factor that controls the degree of adversarial interaction in the scenario. The larger α is, the greater the proportion of adversarial terms and the stronger the adversarial behavior of the agent. Conversely, the larger the proportion of regularization terms, the more rational the behavior of the agent and the more realistic the generated scenario.

3. The autonomous driving optimization control method based on adversarial reinforcement learning according to claim 2, characterized in that, Step S1 specifically includes the following steps: S11. Design an adversarial agent using the TD3 deep reinforcement learning algorithm with an actor-critc architecture, and design the state value s. i Optional environment configuration action a i The TD3 algorithm includes an action network μ(s) i |θ μ ) and two policy networks and and their corresponding target networks μ′(s) i |θ μ′ ), and S12. The training steps for the adversarial agent are as follows: Randomly initialize TD3 network θ μ , and And assign the parameters to the corresponding policy network, i.e., θ. μ′ =θ μ′ , The environmental state s required for the scene-based intelligent agent to obtain the scene-based intelligent agent i ; Design the neural network structure μ(s) for TD3 deep reinforcement learning. i |θ μ According to the state value s i Output optional environment configuration actions: Where c is the symmetric cutoff threshold for noise, a Low and a High This is the cutoff threshold for the action; Apply action a i Then the next environment state s is obtained. i+1 The reward r for each step i ′ is based on the reward function of a non-zero-sum game. Obtained; This yields single-step experience (s) i ,a i ,s i+1 ,r i Add this experience to experience pool D; Sample N experiences from the experience pool D and calculate the cost functions for the three networks: Where y i The target value is calculated as follows: Update network parameters using gradient descent: Where α μ and α Q These are the learning rates of the actor network and the critic network, respectively. Update the target network parameters every fixed step d. i μ′ =tθ μ +(1-τ)θ μ′ Where τ∈(0,1) is the target network update ratio; Repeat the above steps until the set step size or reward value is reached, then stop training. S13. Test the success rate of this vehicle in the generated adversarial scenario. If it is within the set threshold range, put the adversarial scenario built by the generated adversarial agent into the scenario library L; otherwise, discard the scenario. S14. Repeat steps S11 to S13 until the desired number N is generated. s The confrontation scenario.

4. The autonomous driving optimization control method based on adversarial reinforcement learning according to claim 1, characterized in that, Step S2 specifically includes the following process: S21. First, train the initial first-generation reinforcement learning driving algorithm of this vehicle, and then initialize arrays T[N] and A[N] to record the number of times N scenarios in the scenario library L are used and the success rate of this vehicle in the scenario. S22. Based on the sorting of each scene from low to high according to A[N], obtain each S i Scene rank(S) i Given i = 0, 1, ..., N, calculate the sampling probability: Wherein, the random perturbation δ∈Z; S23. Sample scene S based on the calculated probability value. i ; S24, This vehicle is in scenario S i Complete one round of training in the middle; S25, Update Scene S i The success rate, and update scenario S i The number of times it is used, T[i] = T[i] + 1; S26. Repeat steps S22 to S25 until the success rate of this vehicle in each scenario reaches the expected value, thus obtaining the next generation driving algorithm of this vehicle. S27. Repeat the next round of optimization until the desired number of optimizations N is reached. opt until.

5. The autonomous driving optimization control method based on adversarial reinforcement learning according to claim 4, characterized in that, The specific process of training the initial first-generation reinforcement learning driving algorithm for this vehicle in step S21 is as follows: S211. Select a deep reinforcement learning algorithm based on the task scenario and design the state value s. i Optional environment configuration action a i and reward function R ego ; S212. Select a training scenario and place the vehicle in that scenario for training. S213, Obtain vehicle and environmental status s i ; S214, The driving algorithm is based on the state value s i Output optional environment configuration action a i ; S215. According to the reward function R ego Calculate the reward value r of the current environment configuration action. i Execute action a i The state after s i+1 The more beneficial it is to completing the task, the higher the reward the reinforcement learning agent in this vehicle will receive. S216. Repeat steps 3 to 5 until the vehicle achieves the expected success rate or maximum training step length in the scenario.

6. The autonomous driving optimization control method based on adversarial reinforcement learning according to claim 4, characterized in that, In step S25, scenario S i The specific success rate is as follows: Where A[i] represents scene S i The corresponding success rate.

Images