Large-scale vehicle control method based on multi-agent deep reinforcement learning

Abstract

The invention discloses a large-scale vehicle control method based on multi-agent deep reinforcement learning, and the method comprises the steps: defining a continuous control signal corresponding to a discrete control instruction, setting a hyper-parameter, building a model, carrying out the model training of a small-scale environment, updating a network weight, carrying out the soft updating of a target network, and carrying out the iterative updating. And finally, deploying the iterated model to a large-scale highway lane changing and overtaking control scene to realize the control of a large-scale vehicle. According to the method, a multi-agent decision-making process is decomposed into a plurality of single-agent decision-making processes, so that the number of parameters is effectively reduced. And then, a discrete soft actor-commissioner (Di score Soft Actor-Cr it ic, DSAC) algorithm is improved, and the stability of value network training is improved. Meanwhile, the limited influence range of the vehicles in the traffic scene is fully considered, a reward scheme based on the elliptic neighborhood is provided, the vehicles which influence each other are encouraged to form cooperative combined actions, computing resources are reduced, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent traffic control, and mainly relates to a large-scale vehicle control method based on multi-agent deep reinforcement learning. Background Art

[0002] Autonomous driving technology is expected to revolutionize traditional transportation modes and significantly improve the efficiency and safety of transportation. Autonomous driving is divided into two modules: perception and decision-making. In the decision-making module, in typical scenarios of autonomous driving, lane changing and overtaking are the most basic and common behaviors. Due to their interactive complexity and uncertainty, they have become one of the core issues that need to be urgently addressed in autonomous driving decision-making.

[0003] Deep reinforcement learning provides an end-to-end intelligent control solution for autonomous driving technology and has been proven to have great potential. However, deep reinforcement learning algorithms are mainly designed for single-vehicle control, while vehicle control often involves interaction and synergy with other vehicles, aiming to ultimately jointly improve the efficiency and safety of transportation. Single-vehicle control will cause the vehicle to focus only on its own interests and ignore the collaboration with surrounding vehicles. As a result, the controller will produce self-centered and overly aggressive driving behaviors, which may lead to serious traffic conditions such as traffic congestion and even collisions. If large-scale vehicle control is regarded as a separate centralized controller system, traditional deep reinforcement learning algorithms often face the dimensionality disaster of state and action space, and cannot train an effective model within a reasonable time. Classic multi-agent reinforcement learning algorithms also face limitations when dealing with large-scale vehicle control problems. Since the global state contains information about all vehicles, the data is redundant and has a high dimensionality, it is difficult for neural networks to extract truly useful information, which poses a huge challenge to collaborative decision-making among multiple agents. Therefore, the development of large-scale vehicle control technology has become an urgent problem to be solved. Summary of the invention

[0004] The present invention is aimed at the problems existing in the prior art, and proposes a large-scale vehicle control method based on multi-agent deep reinforcement learning. After defining the discrete control instructions corresponding to continuous control signals, hyperparameter setting, model building, model training in a small-scale environment, network weight update, target network soft update and iterative update steps, the iterative model is finally deployed to a large-scale highway lane change and overtaking control scene to achieve large-scale vehicle control. The method of the present invention decomposes the multi-agent decision-making process into multiple single-agent decision-making processes, effectively reducing the number of parameters. Then, the Discrete Soft Actor-Critic (DSAC) algorithm is improved to improve the stability of value network training. At the same time, the limited influence range of vehicles in traffic scenes is fully considered, and a reward scheme based on elliptical neighborhoods is proposed to encourage vehicles that influence each other to form cooperative joint actions, reduce computing resources and improve training efficiency.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a large-scale vehicle control method based on multi-agent deep reinforcement learning, comprising the following steps:

[0006] S1, define the mapping relationship between discrete control instructions and continuous control signals: define the mapping relationship between the discrete action instructions provided by the deep reinforcement learning controller and the continuous control signals at the bottom layer of the controlled vehicle;

[0007] S2, hyperparameter setting: determine the ellipse neighborhood expression and hyperparameters, and define the reward function expression;

[0008] S3, model building: building an improved discrete soft actor-critic (DSAC) model, the model includes 1 policy network, 2 principal value networks and 2 target value networks, wherein the 2 principal value networks and the 2 target value networks both introduce layer normalization (LayerNorm) modules between layers;

[0009] S4, model training in a small-scale environment: training in a small-scale environment, each vehicle makes decisions based on its own observation sequence, performs joint actions to interact with the environment to obtain its own observations, actions, and reward values, and stores the data pairs in a buffer;

[0010] S5, network weight update: obtain observation, action and reward value pairs from the buffer, calculate the loss value of the value network and the policy network, and use the gradient descent algorithm to update the network weights. All agents share a set of network parameters;

[0011] S6, target network soft update: soft update the two target networks respectively according to the weights of the two main value networks obtained in step S5;

[0012] S7, iterative calculation: repeat steps S4-S6 until the early stopping condition is met or the training time ends;

[0013] S8, large-scale environment deployment: deploy the model after the iteration of step S7 to a large-scale highway lane change and overtaking control scenario to achieve large-scale vehicle control.

[0014] As an improvement of the present invention, in step S1, the kinematic model of the i-th vehicle is considered to be:

[0015]

[0016] β i (t) = tan -1 (1 / 2tan(δ i (t))

[0017] Among them, x i (t) and y i (t) is the horizontal and vertical coordinates of vehicle i in the Frenet coordinate system at time t. The horizontal axis is parallel to the lane direction, and the vertical axis is perpendicular to the lane direction. i (t) is the speed of car i at time t, ψ i (t) is the heading angle of vehicle i at time t, β i (t) is the slip angle of vehicle i at the center of gravity at time t, l i is the length of car i; a i (t) and δ i (t) are the acceleration and front wheel steering angle signals of vehicle i at time t, which are continuous control signals;

[0018] Let the control period of the discrete action command provided by deep reinforcement learning be h. In t∈[kh, (k+1)h), the acceleration control signal a i (t) is obtained by the following proportional controller:

[0019]

[0020] in, is the expected speed of car i at time k, which is obtained by mapping the discrete control instructions of car i at time k. p are the preset controller gains that are the same for all vehicles;

[0021] In t∈[kh, (k+1)h), the front wheel steering control signal δ i The specific expression of (t) is:

[0022]

[0023] ψ r，i (t) = ψL (t)+Δψ r，i (t),

[0024]

[0025] v lat,r,i (t) = -K p,lat Δ lat.i (t)

[0026] Δ lat,i (t) = y i (t)-y i,r (k)

[0027] Among them, ψ i (t) is the heading angle of vehicle i at time t, ψ L (t) is the lane direction, v lat,r,i (t) is the lateral velocity command of vehicle i, Δψ r，i (t) is the required heading change of vehicle i at time t, K p,lat and K p，ψ is the preset controller gain, Δ lat,i (t) is the lateral position of vehicle i relative to the centerline of the lane, y i，r (k) is the lateral coordinate of the centerline of the target lane of vehicle i at time k, which is specified by the discrete control instruction.

[0028] As an improvement of the present invention, in step S2, two ellipses are used to describe the neighborhood of vehicle i, and the position is defined at (x i ,y i ) The neighborhood of vehicle i is as follows

[0029]

[0030] in, Focus on the front of vehicle i, defined as

[0031]

[0032] Among them, a and b1 are the front ellipse parameters, a represents the focus range in the lane direction, and b1 represents the focus range in front of the lane; Focusing on the rear of vehicle i, it is defined as

[0033]

[0034] Among them, b2 is the rear ellipse parameter, which represents the focus range behind the lane.

[0035] As an improvement of the present invention, in step S2, the reward function expression of vehicle i is as follows:

[0036]

[0037] where α i Indicates whether vehicle i is on the road, α i =1 means the vehicle is driving normally on the road, otherwise, α i =0, It is an incentive for vehicles to travel at high speed. It is a reward for headway distance for safety reasons. It is a penalty for collision.

[0038] As another improvement of the present invention, in step S2, the incentive for the vehicle to travel at high speed Specifically:

[0039]

[0040] Among them, w1 is the corresponding weight, v i cos(ψ) is the speed of the vehicle along the lane, v max is a fixed value, indicating the maximum speed of the vehicle;

[0041] Bonus for headway for safety reasons Specifically:

[0042]

[0043] Among them, d i represents the headway of vehicle i;

[0044] Penalty for collision Specifically:

[0045]

[0046] The reward ri obtained by each agent is defined as the sum of its own reward and the rewards of other vehicles in the neighborhood, that is,

[0047]

[0048] in, is the weight factor.

[0049] As another improvement of the present invention, in step S3, the expression of the main value network is:

[0050]

[0051] in, is the principal value network, j = 1, 2, w represents the weight of the last linear layer, symbol represents function composition, Represents a single-layer network; the target network has the same structure as the main network.

[0052] As another improvement of the present invention, in step S5, a small batch of observation, action and reward value pairs, i.e., b, is obtained from buffer B, and the loss value is calculated according to the following formula; for the principal value network

[0053]

[0054] in

[0055]

[0056] γ is the discount factor and β is the entropy regularization coefficient; therefore, the parameter update is as follows:

[0057]

[0058] Among them, α Q is the learning rate of the principal value network; the loss value of the policy network is calculated as:

[0059]

[0060] The policy network is updated as follows

[0061]

[0062] α π is the learning rate of the policy network, and the loss value of the entropy regularization coefficient is calculated as follows

[0063]

[0064] in, is the target entropy, and the entropy regularization coefficient β is updated as follows

[0065]

[0066] As another improvement of the present invention, in step S6, the target network performs the soft update step according to the main value network as follows:

[0067]

[0068] Where τ is the soft update weight.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The method of the present invention decomposes the decision-making process of multiple agents into the decision-making process of multiple single agents, effectively reducing the number of parameters, state and action space dimensions, and providing a feasible solution for realizing large-scale vehicle control.

[0071] (2) The method of the present invention proposes the concept of elliptical neighborhood, which distinguishes the influence range of vehicles in the direction perpendicular to the lane and parallel to the lane, and accurately describes the range of affected vehicles based on lane attributes rather than Euler distance.

[0072] (3) The method of the present invention enhances the stability of value network training and accelerates the training of the value network by introducing a layer normalization module into the value network.

[0073] (4) The method of the present invention is trained in a small-scale scenario and tested in a large-scale scenario, which ensures the performance of the controller while saving training cost and training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a work flow chart of the method of the present invention;

[0075] Figure 2 It is an intuitive display of the range of the ellipse neighborhood;

[0076] Figure 3 It is a training block diagram of the model algorithm of the present invention;

[0077] Figure 4 It is a comparison diagram of the training curves of the method of the present invention and the existing algorithm;

[0078] Figure 5 This is a comparison chart of training curves for several different neighborhood definition methods;

[0079] Figure 6 It is a comparison chart of the training curves of the improved DSAC algorithm and the ordinary DSAC algorithm. DETAILED DESCRIPTION

[0080] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0081] Example 1

[0082] A large-scale vehicle control method based on multi-agent deep reinforcement learning aims to propose a multi-agent reinforcement learning technology that can be used for large-scale vehicle control, encourage collaboration between vehicles, and jointly improve safety and transportation efficiency. Figure 1 As shown, the following steps are included:

[0083] Step S1, define the mapping relationship between discrete control instructions corresponding to continuous control signals: define the mapping relationship between discrete action instructions provided by the deep reinforcement learning controller and the underlying continuous control signals of the controlled vehicle.

[0084] Assuming that all vehicles are homogeneous, that is, all vehicles follow the same mapping rules and parameters, define the discrete action instructions provided by the deep reinforcement learning controller corresponding to the underlying control signals of the vehicle. Consider the kinematic model of the i-th vehicle as:

[0085]

[0086] β i (t) = tan -1 (1 / 2tan(δ i (t)) (5)

[0087] Among them, x i (t) and y i (t) is the horizontal and vertical coordinates of vehicle i in the Frenet coordinate system at time t. The horizontal axis is parallel to the lane direction, and the vertical axis is perpendicular to the lane direction. i (t) is the speed of car i at time t, ψ i (t) is the heading angle of vehicle i at time t, β i (t) is the slip angle of vehicle i at the center of gravity at time t, l i is the length of car i; a i (t) and δ i (t) are the acceleration and front wheel steering angle signals of vehicle i at time t, which are continuous control signals;

[0088] Let the control period of the discrete action command provided by deep reinforcement learning be h. In t∈[kh, (k+1)h), the acceleration control signal a i (t) is obtained by the following proportional controller:

[0089]

[0090] in, is the expected speed of car i at time k, which is obtained by mapping the discrete control instructions of car i at time k. p are the preset controller gains that are the same for all vehicles;

[0091] In t∈[kh, (k+1)h), the front wheel steering control signal δ i The specific expression of (t) is:

[0092]

[0093] ψ r，i (t) = ψ L (t)+Δψ r，i (t) (9)

[0094]

[0095] v lat，r，i (t) = -K p，lat Δ lat，i (t) (11)

[0096] Δ lat，i (t) = y i (t)-y i，r (k) (12)

[0097] Among them, ψ i (t) is the heading angle of vehicle i at time t, ψ L (t) is the lane direction, v lat，r，i (t) is the lateral velocity command of vehicle i, Δψ r，i (t) is the required heading change of vehicle i at time t, K p，lat and K p，ψ is the preset controller gain, Δ lat，i (t) is the lateral position of vehicle i relative to the centerline of the lane, y i，r (k) is the lateral coordinate of the centerline of the target lane of vehicle i at time k, which is specified by the discrete control command;

[0098] The discrete controller is trained through deep reinforcement learning, which includes five action instructions: left lane change (0), hold (1), right lane change (2), accelerate (3), and decelerate (4). These five instructions correspond to the above reference values. The horizontal coordinate is defined as positive to the right, and the horizontal coordinate of the center line of the leftmost lane is 0. When the instruction is to change lanes, the horizontal coordinate y of the center line of the target lane is i , r(k) will change according to the instruction requirements; at control time k, the instruction changes lane left (0) so that y i , r(k)=max{0,y r (k-1)-L}, where L is the lane width; command right lane change (2) so that y i，r (k) = min{(n-1)*L, y i，r (k-1)+L}, where n is the number of lanes; the acceleration (3) command increases the expected speed, i.e. Among them, v max，i is the maximum speed of car i, is the amplitude of each change of the preset desired speed, and these two values ​​are different for each vehicle; accordingly, the deceleration (4) instruction reduces the desired speed, that is, Similarly, keeping (1) instruction means y i，r (k) = y i，r (k-1) and

[0099] Step S2, hyperparameter setting: determine the elliptical neighborhood expression and hyperparameters, and define the reward function expression. First, define the neighborhood of the vehicle, and then, based on the neighborhood, define the reward that each vehicle can get for performing a certain action under the current observation, which specifically includes the following two steps:

[0100] Step S21: Use two ellipses to describe the neighborhood of vehicle i. Vehicle j in the neighborhood is considered to be affected by the control action of vehicle i and therefore participates in the reward calculation of vehicle i. Use i to represent vehicle i and define the position at (x i ,y i ) The neighborhood of vehicle i is as follows

[0101]

[0102] in, Focus on the front of vehicle i, defined as

[0103]

[0104] Among them, a and b1 are the front ellipse parameters, a represents the focus range in the lane direction, and b1 represents the focus range in front of the lane; Focusing on the rear of vehicle i, it is defined as

[0105]

[0106] Among them, b2 is the back ellipse parameter, which indicates the focus range behind the lane. The intuitive representation of the neighborhood is as follows Figure 2 As shown in Figure 2, compared with the simple Euclidean distance, the elliptical neighborhood can distinguish the influence ranges perpendicular to the lane direction and parallel to the lane direction.

[0107] Step S22: Define the reward function expression of vehicle i. First, based on the physical information and control target of the vehicle itself, define the reward value obtained by the vehicle's own information as follows:

[0108]

[0109] where α i Indicates whether vehicle i is on the road, α i =1 means the vehicle is driving normally on the road, otherwise, α i =0, when the vehicle is running normally, the reward includes three items, the first of which is the incentive for the vehicle to run at high speed:

[0110]

[0111] Among them, w1 is the corresponding weight, v i cos(ψ) is the speed of the vehicle along the lane, v maxIt is a fixed value, indicating the maximum speed of the vehicle. Although different vehicles may not be able to reach the maximum speed due to different performance, Still and v i It is linearly proportional, so it can effectively encourage autonomous driving vehicles to drive at high speed; the second is the reward for the headway for safety reasons:

[0112]

[0113] Among them, d i represents the headway of vehicle i; the third term is the penalty for collision

[0114]

[0115] In order to promote cooperative actions between vehicles, the reward ri obtained by each agent is defined as the sum of its own reward and the rewards of other vehicles in the neighborhood, that is,

[0116]

[0117] in, is the weight factor.

[0118] Step S3, model building: building an improved discrete soft actor-critic (DSAC) model, the model includes 1 policy network, 2 principal value networks and 2 target value networks, wherein the 2 principal value networks and the 2 target value networks both introduce layer normalization (LayerNorm) modules between layers.

[0119] Decompose multi-agent decision making into single-agent decision making, and build an improved discrete soft actor-critic model for each agent, including a policy network π θ , 2 main value networks and 2 target value networks The method of the present invention improves the structure of the value network, wherein the two main value networks and the two target value networks both introduce layer normalization (LayerNorm) modules between layers, and the expression of the single-layer network is:

[0120]

[0121] Among them, the symbol represents function composition; therefore, the expression of the principal value network is

[0122]

[0123] w represents the weight of the last linear layer. The structure of the target network is the same as that of the main network. The intuitive algorithm block diagram is as follows Figure 3 As shown, the dotted line represents the data flow direction.

[0124] Step S4, model training in a small-scale environment: training is performed in a small-scale environment, where each vehicle makes decisions based on its own observation sequence, performs joint actions to interact with the environment to obtain its own observations, actions, and reward values, and stores the data pairs in a buffer.

[0125] Trained in a small-scale vehicle decision environment, at each training step, a sequential decision is made for each vehicle, and the observation o i Next, vehicle i performs action a i ~π θ (o i ), perform joint actions after sequential decision making, interact with the environment, transfer to the next state and obtain each vehicle i and the next observation o′ i And calculate the reward r i , all data pairs (o i , a i , r i , o′ i ) are stored in buffer B.

[0126] Step S5, network weight update: obtain observation, action and reward value pairs from the buffer, calculate the loss values ​​of the value network and the policy network, and use the gradient descent algorithm to update the network weights. All agents share a set of network parameters.

[0127] Get a small batch of observations, actions, and reward value pairs from buffer B, that is, b, and calculate the loss value according to the following formula; for the main value network

[0128]

[0129] in

[0130]

[0131] γ is the discount factor and β is the entropy regularization coefficient; therefore, the network parameters are updated as follows:

[0132]

[0133] Among them, α Q is the learning rate of the principal value network; the loss value of the policy network is calculated as:

[0134]

[0135] The policy network is updated as follows

[0136]

[0137] α πis the learning rate of the policy network. Similarly, the loss value of the entropy regularization coefficient is calculated as follows

[0138]

[0139] in, is the target entropy, then, the entropy regularization coefficient β is updated as follows

[0140]

[0141] Step S6, soft update of the target network: soft update is performed on the two target networks respectively according to the weights of the two main value networks obtained in step S5.

[0142] The target network performs soft update steps according to the main value network as follows:

[0143]

[0144] Where τ is the soft update weight.

[0145] Step S7, iterative calculation: repeat steps S4-S6 until the early stopping condition is met or the training time ends.

[0146] Step S8, large-scale environment deployment: deploy the model after the iteration of step S7 to a large-scale highway lane change and overtaking control scenario to achieve large-scale vehicle control.

[0147] Example 2

[0148] This embodiment is based on the open source highway_env environment and includes a four-lane lane change and overtaking scenario. The scenario includes a 2000-meter road section where vehicles arrive continuously from the upstream entrance, and the arriving vehicles satisfy the Poisson distribution. Based on the statistical data of real-world traffic data, the maximum expected speed of each vehicle is sampled from the Gamma distribution and limited to between 20m / s and 35m / s, with a variable of 5m / s for each expected speed. The vehicle observations include the positions and speeds of other vehicles and the self-vehicle in the left and right lanes, 120 meters in front and 10 meters behind. The observation and control frequencies are both 1Hz. There are 20 controlled vehicles in the training environment, and a total of 1000 vehicles are generated in the testing environment.

[0149] 1) Define the discrete control instructions corresponding to the continuous control signals

[0150] According to equations (1)-(12), the mapping between discrete action instructions and continuous control signals is established. The parameters related to the mapping are: K p =1.87, K p，lat =1.67 and K p，ψ=5, all controlled vehicles remain consistent. The speed setting value is divided into three levels within this range. When the discrete action gives an acceleration instruction, the speed setting value is increased by one level until the maximum value. When the deceleration instruction is given, the speed setting is reduced by one level until the minimum value. Considering that different vehicles have different performances, the maximum expected speed of each controlled vehicle is sampled from the Gamma distribution and limited to between 20m / s and 35m / s, with a variable of 5m / s each time. The observation and control frequencies are both 1Hz.

[0151] 2) Hyperparameter setting

[0152] The parameters of the elliptical neighborhood of (13)-(15) are a=4, b1=10, b2=30, and the reward function related weights of (16)-(20) are w=0.5, w1=1, W2=0.01 and W3=5; the policy network uses a two-layer fully connected network with 512-128 neurons in the hidden layer, and the two principal value networks also use two-layer fully connected networks with 512-128 neurons in the hidden layer. A normalization module is inserted between the fully connected layers, and the target network has the same structure as the main network; (21)-(30) involve parameters for deep reinforcement learning training, with the batch size set to 256 and the training learning rate α π =α Q =α β =0.001, target entropy The soft update weight τ=0.01.

[0153] 3) Initialization

[0154] The initialization weights of all networks are normally distributed with a mean of 0 and a standard deviation of 0.1, and the initialization value of the bias is 0.01.

[0155] 4) Training in small-scale scenarios, network weight update and target network soft update. At each decision moment, each vehicle makes a decision based on its own observation sequence, then obtains a joint action, interacts with the environment and obtains the immediate reward of each vehicle according to formula (20), and stores the corresponding observation-action-reward pairs in the buffer. At each training moment, a small batch of data is extracted from the buffer, and the value network parameters, policy network parameters and entropy regularization coefficients are updated according to formulas (25), (27) and (29), respectively, and then the target network is soft updated according to formula (30).

[0156] 5) Verification in large-scale scenarios. Compare the proposed large-scale vehicle control method based on multi-agent deep reinforcement learning with existing algorithms. To ensure the consistency of the standard, we use the reward corresponding to the state of each vehicle The average value of is used as the evaluation indicator, that is, the overall average reward is defined as

[0157]

[0158] Among them, n is the number of agents, which is used as an evaluation indicator.

[0159] The comparison results are as follows Figure 4 As shown in the figure, the ordinate is the overall average reward, and the abscissa is the number of training rounds. Compared with the last 100 rounds, the proposed method obtains an overall average reward of 0.7592±0.0398, while the existing algorithm obtains an overall average reward of 0.6453±0.04339, an average increase of 17.65%.

[0160] Comparing the performance of the algorithm under different neighborhood definitions, the results are as follows Figure 5 As shown in the figure. Greedy rewards overemphasize self-interest and are prone to aggressive behavior. Global rewards focus on a large amount of information irrelevant to the controlled vehicle, weakening self-interest and not having a good training effect. The circular neighborhood reward is centered on the controlled vehicle and defines the neighborhood by Euler distance. It takes into account the neighboring vehicles that may be affected and encourages coordinated actions. Compared with the first two, the effect is better. The elliptical neighborhood can distinguish vehicles in different directions with different influence ranges, which is more suitable for traffic scenarios and achieves the best performance.

[0161] Figure 6 The effectiveness of the improved DSAC algorithm was verified. By introducing the layer normalization module and comparing the last 100 rounds, the overall average reward increased from 0.6108±0.05146 to 0.7592±0.0398, and the average reward increased by 24.3%.

[0162] In summary, the method of the present invention decomposes the decision-making process of multiple agents into the decision-making process of multiple single agents, greatly reducing the number of parameters and the dimensions of actions and observations. At the same time, in order to avoid aggressive and selfish driving behaviors of vehicles, it encourages cooperation between vehicles. The reward value of a vehicle is not only determined by its own state, but also related to the state of other vehicles in the neighborhood. In order to further fit the traffic scene, the neighborhood of the vehicle is formed by a combination of two ellipses with different front and rear major axes centered on the vehicle. In order to further reduce computing resources and improve training efficiency, the method of the present invention adopts a strategy of small-scale scene training and large-scale scene testing. The performance and efficiency exceed the existing algorithms, and it is more efficient and practical.

[0163] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A large-scale vehicle control method based on multi-agent deep reinforcement learning, characterized in that: The steps include: S1, define the mapping relationship between discrete control instructions and continuous control signals: define the mapping relationship between the discrete action instructions provided by the deep reinforcement learning controller and the continuous control signals at the bottom layer of the controlled vehicle; S2, hyperparameter setting: determine the ellipse neighborhood expression and hyperparameters, and define the reward function expression; S3, model building: building an improved Discrete Soft Actor-Critic (DSAC) model, the model includes 1 policy network, 2 principal value networks and 2 target value networks, wherein the 2 principal value networks and the 2 target value networks both introduce layer normalization (LayerNorm) modules between layers; S4, model training in a small-scale environment: training in a small-scale environment, each vehicle makes decisions based on its own observation sequence, performs joint actions to interact with the environment to obtain its own observations, actions, and reward values, and stores the data pairs in a buffer; S5, network weight update: obtain observation, action and reward value pairs from the buffer, calculate the loss value of the value network and the policy network, and use the gradient descent algorithm to update the network weights. All agents share a set of network parameters; S6, target network soft update: soft update the two target networks respectively according to the weights of the two main value networks obtained in step S5; S7, iterative calculation: repeat steps S4-S6 until the early stopping condition is met or the training time ends; S8, large-scale environment deployment: deploy the model after the iteration of step S7 to a large-scale highway lane change and overtaking control scenario to achieve large-scale vehicle control.

2. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, the kinematic model of the i-th vehicle is considered to be: Among them, x i (t) and y i (t) is the horizontal and vertical coordinates of vehicle i in the Frenet coordinate system at time t. The horizontal axis is parallel to the lane direction, and the vertical axis is perpendicular to the lane direction. i (t) is the speed of car i at time t, ψ i (t) is the heading angle of vehicle i at time t, β i (t) is the slip angle of vehicle i at the center of gravity at time t, l i is the length of car i; a i (t) and δ i (t) are the acceleration and front wheel steering angle signals of vehicle i at time t, which are continuous control signals; Let the control period of the discrete action command provided by deep reinforcement learning be h. In t∈[kh, (k+1)h), the acceleration control signal a i (t) is obtained by the following proportional controller: in, is the expected speed of car i at time k, which is obtained by mapping the discrete control instructions of car i at time k. p are the preset controller gains that are the same for all vehicles; In t∈[kh, (k+1)h), the front wheel steering control signal δ i The specific expression of (t) is: ψ r，i (t)=ψ L (t)+Δψ r，i (t), v lat，r，i (t)=-K p，lat Δ lat，i (t) Δ lat，i (t)=y i (t)-y i，r (k) Among them, ψ i (t) is the heading angle of vehicle i at time t, ψ L (t) is the lane direction, v lat，r，i (t) is the lateral velocity command of vehicle i, Δψ r，i (t) is the required heading change of vehicle i at time t, K p，lat and K p，ψ is the preset controller gain, Δ lat，i (t) is the lateral position of vehicle i relative to the centerline of the lane, y i，r (k) is the lateral coordinate of the centerline of the target lane of vehicle i at time k, which is specified by the discrete control instruction.

3. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 2, characterized in that: In step S2, two ellipses are used to characterize the neighborhood of vehicle i, and the position is defined at (x i ,y i ) The neighborhood of vehicle i is as follows in, Focus on the front of vehicle i, defined as Among them, a and b1 are the front ellipse parameters, a represents the focus range in the lane direction, and b1 represents the focus range in front of the lane; Focusing on the rear of vehicle i, it is defined as Among them, b2 is the rear ellipse parameter, which represents the focus range behind the lane.

4. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 3, characterized in that: In step S2, the reward function expression of vehicle i is as follows: where α i Indicates whether vehicle i is on the road, α i =1 means the vehicle is driving normally on the road, otherwise, α i =0, It is an incentive for vehicles to travel at high speed. It is a reward for headway distance for safety reasons. It is a penalty for collision.

5. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 4, characterized in that: In step S2, the incentive for the vehicle to travel at high speed Specifically: Among them, w1 is the corresponding weight, ν i cos(ψ) is the speed of the vehicle along the lane, v max is a fixed value, indicating the maximum speed of the vehicle; Bonus for headway for safety reasons Specifically: Among them, d i represents the headway of vehicle i; Penalty for collision Specifically: The reward r received by each agent i is defined as the sum of its own reward and the rewards of other vehicles in the neighborhood, that is, in, is the weight factor.

6. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In step S3, the expression of the main value network is: in, is the principal value network, j = 1, 2, w represents the weight of the last linear layer, symbol represents function composition, Represents a single-layer network; the target network has the same structure as the main network.

7. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: In step S5, a small batch of observation, action and reward value pairs, i.e., b, is obtained from buffer B, and the loss value is calculated according to the following formula; for the principal value network in γ is the discount factor and β is the entropy regularization coefficient; therefore, the parameter update is as follows: Among them, α Q is the learning rate of the principal value network; the loss value of the policy network is calculated as: The policy network is updated as follows α π is the learning rate of the policy network, and the loss value of the entropy regularization coefficient is calculated as follows in, is the target entropy, and the entropy regularization coefficient β is updated as follows 8. The large-scale vehicle control method based on multi-agent deep reinforcement learning as claimed in claim 7, characterized in that: In step S6, the target network performs soft update steps according to the main value network as follows: Where τ is the soft update weight.

Images