Cooperative roadway control method for complex intersections using multi-agent association reinforcement learning

Abstract

The present invention proposes a roadway cooperative control framework with a roadside static processing module and a vehicle-side dynamic processing module, supplements road history information with roadside advantage, proposes a federated reinforcement learning algorithm FTD3 for connecting a reinforcement learning module and a federated learning module, and discloses a roadway cooperative control system and method using multi-agent federated reinforcement learning at complex intersections, in which the algorithm transfers only neural network parameters, not vehicle-side data, to protect privacy. In the algorithm, only some neural networks are selected for aggregation to reduce communication overhead, networks that produce relatively small Q values ​​are selected for aggregation to prevent overfitting, and a deep combination of federated learning and reinforcement learning is realized, and the RSU neural network participates in aggregation but does not participate in training, and only the aggregated shared model is used for updating, not the experience generated on the vehicle side. Vehicle-side privacy is protected, the convergence of the neural network is slowed down, and only some neural networks are selected to participate in aggregation, reducing the network aggregation cost.

Description

[Technical field]

[0001] The present invention relates to the field of traffic transportation, and more particularly to a system and method for roadway cooperative control using multi-agent association reinforcement learning at complex intersections. [Background technology]

[0002] In recent years, research into autonomous driving has progressed more and more. However, the intelligence of a single vehicle has significant limitations, and its limited sensing range and calculation capabilities may affect decision-making in complex traffic situations. Blindly increasing the cost and increasing the performance of a single vehicle is not a panacea; in contrast, sensing coordination and shifting the calculation load are more realistic. Roadway coordination technology not only makes vehicles intelligent, but also installs sensing sensors on the roadside, and at the same time provides data to vehicles after the roadside unit completes calculation, thereby reducing the burden on single vehicles and helping vehicles complete automatic driving. However, in the current stage of roadway coordination technology, complex traffic situations and redundant traffic information make it difficult to extract effective information, and the communication overhead is very large, directly leading to the problem that the control effect is difficult to reach expectations. And information asymmetry due to privacy consciousness is also becoming a major bottleneck in roadway coordination.

[0003] Federated learning is a distributed collaboration method that allows multiple partners to train data and build a shared model, and protects vehicle-side privacy through a special learning framework, training method, and transfer principle, providing a safer learning environment and collaboration process. In addition, reinforcement learning can optimize the vehicle's control policy and embody altruism in ensuring safety when facing a complex driving environment through a composite reward function and trial-and-error training method. Federated reinforcement learning is a combination of federated learning and reinforcement learning, and uses the distributed multi-agent training framework of federated learning to perform cooperative training, protects privacy through the communication characteristics of transmitting network parameters instead of training data, greatly reduces communication overhead, and is a training method that improves policies through trial and error in line with reinforcement learning, showing great potential in the field of autonomous driving. However, there are problems with existing federated reinforcement learning algorithms, such as federated reinforcement learning being demanding on network aggregation settings and the incompatibility between the two being shown in multi-network algorithms, resulting in unstable network convergence, poor training effect, and large network overhead. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a system and method for roadway cooperative control using multi-agent associative reinforcement learning at complex intersections, which guides training by roadside advantage, realizes vehicle-side roadside cooperative sensing, cooperative training, cooperative evaluation, and realizes true roadway cooperative control. And the proposed FTD3 algorithm improves the algorithm from multiple perspectives by combining associative learning and reinforcement learning, speeding up convergence, improving the convergence level, and reducing communication costs while protecting vehicle-side privacy.

[0005] In this invention, the technical solution of the roadway coordination control system based on multi-agent associative reinforcement learning includes two main contents: a roadway coordination framework including a roadside static processing module, a simulation environment and sensor module, and a vehicle dynamic processing module, and an FTD3 algorithm including a reinforcement learning module and an associative learning module.

[0006] In the roadway coordination framework, the main purpose is to synthesize coordination state quantities for training, among which the roadside static processing module is for obtaining static road information, and separately extracting lane centerline information from it as a static matrix and transmitting it to the vehicle dynamic processing module.

[0007] The simulation environment Carla is used for the interaction between the agent and the environment, and the sensor module is used for acquiring the vehicle dynamic state quantity, and the collision sensor and the sensor for detecting whether the wheels are on the road markings can detect and record two types of events, collision and line pressing. The navigation satellite sensor can acquire the vehicle's position information, and the speed information can be acquired by the position of two frames. The inertial sensor can acquire the acceleration information and orientation of the vehicle. The specific interaction process is to use the sensor to capture the state quantity of the agent's location, and then the neural network outputs the control quantity based on the state quantity, and finally the control quantity is given to the simulation environment Carla for execution, and this is repeated.

[0008] The vehicle dynamic processing module is used to synthesize the coordination state matrix information, and clips the static matrix acquired by the roadside static processing module according to the vehicle position information, and clips it into a 56x56 matrix centered on the smart car's center of gravity. Then, the coordination state quantity is synthesized by stacking the matrices and sensor information of two consecutive frames, and transmitted to the reinforcement learning module.

[0009] The main objective of the FTD3 algorithm is to output a control amount based on the cooperative state matrix. The reinforcement learning module is used to output a control policy and is described by a Markov decision process. In a Markov decision process, the state at the next time is related only to the current state, not to the previous state. The state sequence Markov chain constructed under this premise is the basis of the reinforcement learning module of the present invention. The reinforcement learning module includes three small modules: a neural network module, a reward function module, and a network training module.

[0010] The neural network module extracts the features input into the coordination state matrix, outputs the control amount based on the features, and provides it to the simulation environment for execution. In addition to the director network and two critic networks that the conventional TD3 algorithm has, the single agent in FTD3 also has their respective target networks. The six neural network structures are completely the same except for the output layer, which uses one convolutional layer and four fully connected layers to extract and integrate features, and for the director network, the output layer maps to [-1, 1] via a tanh activation function. As shown in Figure 1, the neural network represents the steering wheel control amount in the CARLA simulator as a t1 and output as a t2 is divided into [-1,0] and [0,1], which represent the brake and accelerator control amounts, respectively. For the critic network, no activation function is used in the output layer, and the evaluation value is output as is.

[0011] The reward function module determines whether the output value of the neural network module is good or bad depending on the new state reached by the execution of the action, and guides the learning of the network training module. lateral and the vertical reward function r longitudinal Considering these two points,

number

number

number

[0012] The network training module is mainly for training the neural network in the neural network module according to the setting method, and according to the guidance of the reward function module, the director network and the critic network update the parameters by backpropagation, and all the target networks update the parameters by soft update, thus achieving the training purpose and finding the optimal solution that maximizes the cumulative gain in a specific state. After sampling in small batches from the experience pool, calculate the target function y;

number

number

number

number

number

number

number

number

[0013] The federated learning module is mainly for obtaining the trained neural network parameters by the training module, aggregating the shared model parameters, and sending the shared model parameters to the agents for local updating. The federated learning module includes two small modules: a network parameter module and an aggregation module.

[0014] The network parameter module is used to obtain each neural network parameter before the aggregation begins and upload the parameters to the aggregation module for aggregating the shared model parameters, and is used to obtain the shared model parameters after the aggregation is completed and send the parameters to each agent for local updating.

[0015] The aggregation module aggregates each neural network parameter uploaded by the network parameter module according to an aggregation interval into a shared model parameter by using a parameter averaging method;

number

[0016] In short, the FTD3 algorithm is for connecting the reinforcement learning module and the federated learning module, and the algorithm transfers only neural network parameters, not vehicle-side data, to preserve privacy; the algorithm selects only a part of neural networks for aggregation to reduce communication overhead; the algorithm selects networks with relatively small Q values ​​for aggregation to prevent overfitting.

[0017] The technical solution of the roadway cooperative control method based on multi-agent association reinforcement learning of the present invention includes the following steps:

[0018] Step 1: construct a roadway coordination framework in a simulation environment, and use a roadside static processing module and a vehicle dynamic processing module to synthesize a coordination state quantity for reinforcement learning. The roadside static processing module is used to divide the roadside unit RSU aerial photo information into two types: static (road, lane, lane centerline) and dynamic (intelligent connected car), and the lane centerline extracted separately from the static information is the basis for the reinforcement learning coordination state quantity, and the dynamic information is the basis for state quantity clipping. The vehicle dynamic processing module clips the static matrix obtained by the roadside static processing module according to the vehicle position information, respectively, and the clipped 56x56 matrix is ​​the sensing range of a single vehicle, covering a physical space of about 14mx14m. To obtain more comprehensive dynamic information, the dynamic information is stacked using two consecutive frames. The dynamic processing module stacks the clipped static matrix and the stacked dynamic information to synthesize the coordination state quantity for FTD3.

[0019] Step 2: The control method is determined by Markov decision process The Markov decision process is described as a tuple (S, A, P, R, γ), S represents a state set, and the cooperative state quantity output according to the roadway cooperation framework in the present invention is composed of two parts of matrices. First, the cooperative sensing matrix, which is acquired by the proposed vehicle dynamic processing module, includes static road information, dynamic vehicle speed, position information, and implicit information such as the distance of vehicle acceleration from the lane center line, traveling direction, azimuth deviation, etc., and integrates the features through a convolution layer and a fully connected layer. Second, the sensor information matrix at the current time includes the speed information, orientation, and acceleration information acquired and calculated by the vehicle-side sensor; A represents an action set, which corresponds to the vehicle accelerator and steering wheel control amount in the present invention, P represents the state transition equation p:S×A→P(S), and for each state-action pair (s,a)∈S×A, there is a probability distribution that represents the probability of transitioning to a new state when action a is taken in state s.

number

number

[0020] Markov Decision process The solution is that the maximum total return is π * :=argmax θ η(π θ ), the objective is to find a policy π:S→A such that A = S ≠ A . In this invention, the coordination state quantity is output according to the roadway coordination framework, and then the corresponding optimal control policy of the coordination state matrix is ​​output by the FTD3 algorithm.

[0021] In step 3, the FTD3 algorithm is constructed, which mainly consists of two parts: a reinforcement learning module and a federated learning module. decision process The elements (S, A, P, R, γ) in the model constitute a reinforcement learning module, and the network parameter module and the aggregation module constitute a federated learning module. Each agent has a directing network and two critic networks, as well as their respective target networks, i.e., a total of six neural networks.

[0022] Step 4: In the simulation environment, interactive training is performed, and the training process includes two stages: free search and sampling learning. In the free search stage, the policy noise of the algorithm is increased to generate random behavior. In the entire training process, the roadway coordination framework captures and synthesizes the coordination state, and then the FTD3 algorithm inputs the coordination state and outputs a noise-added behavior. After the behavior is executed, the roadway coordination framework captures a new state, and finally the reward function module judges whether the behavior is good or bad. This tuple consisting of the state, behavior, next state, and reward function is an experience, and the randomly generated experience samples are stored in the experience pool. When the number of experiences is 3000 or more, the training transitions to the sampling learning stage. Samples are extracted from the experience pool in small batches and learned according to the training method of the FTD3 network training module, and the policy noise is attenuated with the increase in the learning degree.

[0023] Step 5: obtain each neural network parameter through the network parameter module in federated learning, and upload the parameters to an aggregation module of the roadside unit RSU. The aggregation module aggregates each neural network parameter uploaded by the network parameter module according to an aggregation interval by a parameter averaging method to obtain a shared model parameter.

[0024] In step 6, the shared model aggregated by the network parameter module in the federated learning is sent to the vehicle side for model updating, and this is repeated until the network converges.

[0025] Preferably, in step 2, a cooperation state matrix having a size of (56*56*1) for the cooperation state amount and a sensor information matrix having a size of (3*1) are used.

[0026] Preferably, in step 3, the neural network model structure used in the director network in the FTD3 algorithm is composed of one convolutional layer and four fully connected layers, and in the last layer of the network, a tanh activation function is used to map the output to the [-1, 1] interval, and a relu activation function is used in the other layers. The critic network similarly uses one convolutional layer and four fully connected layers, and in the last layer of the network, a relu activation function is used in the other layers, and in addition to directly outputting and evaluating the Q value without using an activation function.

[0027] Preferably, in step 4, in the training network process, the learning rates selected for the director and critic networks are both 0.0001, the policy noise is 0.2, the delay update parameter is 2, the discount factor γ is 0.95, and the target network update weight tau is 0.995.

[0028] Preferably, in step 4, the maximum capacity of the experience pool is selected as 10000, and the number of minibatches extracted from the experience pool is 128.

[0029] Preferably, in step 5, the neural network used in the roadside unit RSU participates in the aggregation but does not participate in the training, and only some neural networks (direction network, target network of the directing network, critic target network that produces more small Q value) are selected to participate in the aggregation. Regarding the selection of the critic target network, for example, when the samples are extracted with a minibatch of 128, each of the two critic target networks scores 128 samples, and the one that produces a smaller Q value and has more than 64 samples is selected to participate in the aggregation.

[0030] Beneficial effects of the present invention include: (1) The present invention uses a roadway cooperative control framework with a roadside static processing module and a vehicle dynamic processing module. For problems where feature extraction is difficult, an innovative cooperative state quantity is constructed with roadside dominance, reducing the difficulty of training. This framework realizes vehicle-side and roadside cooperative sensing, cooperative training, and cooperative evaluation, and realizes roadway cooperative control in the true sense, providing a new idea for roadway cooperation. (2) The present invention improves on the prior art problems in many ways by using the proposed FTD3 algorithm. For the user privacy problem, FTD3 only transmits neural network parameters, not vehicle-side samples, to protect privacy. For the problem of extremely high communication overhead, FTD3 selects and aggregates only a portion of the networks, reducing communication costs. For the problem of overfitting, FTD3 aggregates only neural networks that generate small Q values ​​by filtering. Unlike the hard connection between traditional associative learning and reinforcement learning, the deep combination of the two is realized. [Brief description of the drawings]

[0031] [Figure 1] 1 is a roadway coordination framework proposed in the present invention. [Diagram 2] FIG. 2 is a schematic diagram of a cooperative sensing system according to the present invention; [Diagram 3] 1 is a neural network structure used in the present invention. [Figure 4] 1 is the framework of the FTD3 algorithm proposed in this invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The technical solutions of the present invention are described in detail below with reference to the drawings, but the present invention is not limited thereto.

[0033] The present invention provides a road coordination control framework and FTD3 algorithm based on federated reinforcement learning to realize multi-vehicle control in surrounding island situations, which specifically includes the following steps:

[0034] (1) A roadway cooperative control framework is established in the CARLA simulator, which includes a camera-equipped RSU and a multi-sensor smart car as shown in Figure 1, and the corresponding roadside static processing module and car-side dynamic processing module are initialized and cooperative sensing is established as shown in Figure 2. The multiple types of sensors used are the basis for obtaining the vehicle dynamic state quantity, among which the collision sensor and the sensor that detects whether the wheels are on the road marking line can detect and record two types of events, namely collision and line pressing. The navigation satellite sensor can obtain the position information of the vehicle, and the speed information can be obtained by the position of two frames. The inertial sensor can obtain the acceleration information and orientation of the vehicle.

[0035] (2) As shown in Figure 3, the FTD3 algorithm is constructed and a neural network is assigned to the agent. The input, output, and reward function of the network are specified. The input is the cooperative state quantity, which is composed of two matrices. First, the cooperative sensing matrix is ​​obtained by the submitted vehicle-side dynamic processing module and includes static road information, dynamic vehicle speed, position information, and implicit information such as the distance from the lane center line of the vehicle acceleration, the traveling direction, and the azimuth deviation. Second, the sensor information matrix at the current time includes the speed information, orientation, and acceleration information obtained and calculated by the vehicle-side sensor. The two matrices are extracted and integrated by the corresponding convolution layer and fully connected layer, respectively.

[0036] The output of the neural network module is combined with the vehicle control method in the Carla simulator, and the output layer of the neural network module is mapped to [-1, 1] after passing through the tanh activation function, as shown in Figure 1. t1 represents the amount of steering control in the CARLA simulator, and a t2 is divided into [-1,0] and [0,1], which represent the brake and accelerator control amounts, respectively.

[0037] Regarding the setting of the reward function, from two perspectives, horizontal and vertical, the reward function judges whether the action performed by the smart car is good or bad, and guides the training, i.e.

number

number

number

[0038] (4) Based on the OpenDD real driving dataset, random positions and initial velocities are obtained and combined with random noise to allow the reinforcement learning agent to generate experience by interacting with the simulation environment, which is then stored in a pre-defined experience pool.

[0039] (5) After the experience pool is full, the system extracts a minibatch from the experience pool and applies gradient descent to train the network. The parameters used in training are: the learning rates selected for the director and critic networks are both 0.0001, the policy noise is 0.2, the delay update parameter is 2, the discount coefficient γ is 0.95, the target network update weight tau is 0.995, the maximum capacity of the experience pool is selected as 10000, and the minibatch extracted from the experience pool is 128. The specific algorithm flow is as follows: After sampling in small batches from the experience pool, calculate the target function y,

number

number

number

number

number

number

number

[0040] [Table 1-1] [Table 1-2]

[0041] Regarding the initialization process,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0042] The specific flow is as follows: the agent's neural network and experience pool are randomly initialized, and when the experience pool samples are less than 3000, it enters the random search process. The vehicle dynamic information is acquired by the smart car sensor, the roadside static module acquires static road information, the vehicle side dynamic module clips the road information into a 56×56 matrix centered on the smart car's center of gravity, and then the matrix and sensor information of two consecutive frames are stacked to synthesize the cooperative state quantity. The neural network module outputs the steering wheel and accelerator control amount with normal distribution noise based on the state quantity, and provides it to the simulation environment for execution. The vehicle dynamic information is acquired again by the smart car sensor, the roadside static module acquires static road information, the vehicle side dynamic module clips the road information into a 56×56 matrix centered on the smart car's center of gravity, and then the matrix and sensor information of two consecutive frames are stacked to generate the cooperative state quantity of the next time, and the reward function module acquires a specific reward value based on the new state quantity. The cooperative state quantity, control amount, reward, and cooperative state quantity of the next time are stored in the experience pool according to the tuple. When the number of experiences in the experience pool is 3000 or more, the normal distribution noise starts to decay and the training phase begins. Samples are extracted from the experience pool with the minimum lot size to perform learning, and the direction network and critic network are trained by gradient descent, while other target networks are trained by soft update. According to the aggregation interval, the network parameter module obtains the direction network, the target network of the direction network, and the critic target network parameters that produce more and smaller Q values ​​before the start of aggregation, and uploads the parameters to the aggregation module for aggregation of the shared model parameters. After the aggregation is completed, the network parameter module obtains the shared model parameters again and sends the parameters to each agent for local update. In this manner, it is repeated until the network converges.

[0043] (6) Feasibility analysis: The proposed federated reinforcement learning control method can still perform well even in a delayed communication environment. This is mainly due to the algorithm characteristics of only transferring neural network parameters and the algorithm settings that select individual networks to participate in the aggregation. These advantages allow it to operate in existing Wi-Fi and 4G environments with high communication requirements, and its application scenarios are broader.

[0044] In summary, the roadway cooperative control framework proposed in this invention, which uses roadside static processing module and vehicle-side dynamic processing module, constructs innovative cooperative state quantity and reward function by roadside advantage, realizes vehicle-side roadside cooperative sensing, cooperative training, and cooperative evaluation, and realizes roadway cooperative control in a true sense. And then, a federated reinforcement learning algorithm FTD3 is proposed, which improves the algorithm performance in three ways and realizes a deep combination of federated learning and reinforcement learning, that is, the RSU neural network participates in aggregation but does not participate in training, and updates only using the aggregated shared model, not the experience generated on the vehicle side. It protects the privacy of the vehicle side, slows down the convergence of the neural network, selects only some neural networks to participate in aggregation, reduces the network aggregation cost, selects and aggregates target networks that produce more and smaller Q values, and prevents overestimation. The proposed FTD3 algorithm is different from the hard connection between federated learning and reinforcement learning, and realizes a deep combination of the two.

[0045] The series of detailed descriptions listed above are merely specific descriptions for possible embodiments of the present invention, and they do not limit the protection scope of the present invention, and any equivalent method or modification that does not deviate from the technology of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for roadway cooperative control at a complex intersection using multi-agent association reinforcement learning, comprising the steps of: Step 1: construct a roadway coordination framework in a simulation environment, and use a roadside static processing module and a vehicle-side dynamic processing module to synthesize a coordination state quantity for reinforcement learning; use the roadside static processing module to divide the roadside unit RSU aerial photo information into two types: static (road, lane, lane centerline) and dynamic (intelligent connected car); the lane centerline extracted separately from the static information is the basis for the reinforcement learning coordination state quantity, and the dynamic information is the basis for state quantity clipping; the vehicle-side dynamic processing module clips the static matrix obtained by the roadside static processing module according to the vehicle position information and coordinate transformation; the clipped 56×56 matrix is ​​the detection range of a single vehicle, covering a physical space of 14m×14m; and stack the dynamic information using two consecutive frames to obtain more comprehensive dynamic information; and the dynamic processing module stacks the clipped static matrix and the stacked dynamic information to synthesize a coordination state quantity for FTD3; Step 2: model the control process as a Markov decision process, the Markov decision process being described by a tuple (S, A, P, R, γ); S represents a state set, and the cooperative state quantity output according to the roadway cooperation framework is composed of two parts of matrices. First, for the cooperative sensing matrix, the cooperative sensing matrix obtained by the submitted vehicle-side dynamic processing module includes static road information, dynamic vehicle speed, position information, and implicit information of vehicle acceleration, distance from the lane center line, heading direction, and azimuth deviation, and integrates the features through a convolution layer and a fully connected layer. Secondly, for the sensor information matrix at the current time, it includes speed information, orientation, and acceleration information obtained and calculated by the vehicle-side sensor; A represents an action set, which corresponds to the vehicle accelerator and steering wheel control amount, P represents the state transition equation p: S×A→P(S), and for each state-action pair (s, a) ∈ S×A, there is a probability distribution that represents the probability of transitioning to a new state after taking action a in state s. [0,54] having R represents the reward function R: S × S × A → R, and R(s ｔ＋１ , s ｔ , a ｔ ) is the original state s ｔ From the new state s ｔ＋１ After moving to, the obtained return is represented, and the reward function defines whether the action was performed or not. γ denotes the discount factor, γ∈[0,1], and the total return [0.55] The purpose of this is to calculate The solution to the Markov decision process is that the maximum total return is π ＊ :=argmax θ η (π θ ), that is, according to the coordination state quantity output by the roadway coordination framework, output the corresponding optimal control policy of the coordination state matrix by the FTD3 algorithm; Step 3: Design an FTD3 algorithm, which includes a reinforcement learning module and an association learning module, in which the reinforcement learning module is configured by the elements (S, A, P, R, γ) in the Markov decision process, and the association learning module is configured by the network parameter module and the aggregation module; Step 4: In the simulation environment, interactive training is performed, and the training process includes two stages: free search and sampling learning. In the free search stage, the policy noise of the algorithm is increased and random actions are generated. In the entire training process, the coordination state is captured and synthesized by the roadway coordination framework, and then the coordination state is input by the FTD3 algorithm and an action with noise is output. After the action is performed, a new state is captured by the roadway coordination framework. Finally, the reward function module judges whether the action is good or bad. The tuple consisting of the state, action, next state, and reward function is an experience. The randomly generated experience samples are stored in an experience pool. When the number of experiences meets a predetermined condition, the training transitions to the sampling learning stage, and samples are extracted from the experience pool in small batches and learned according to the training method of the FTD3 network training module. The policy noise is attenuated with the increase in the learning degree. Step 5: obtaining each neural network parameter through the network parameter module in the federated learning, and uploading the parameters to the aggregation module, and the aggregation module aggregates each neural network parameter uploaded by the network parameter module according to an aggregation interval by a parameter averaging method to obtain a shared model parameter; Step 6: The shared model aggregated by the network parameter module in the federated learning is sent to the vehicle side for model update, and the process is repeated until the network converges. This paper presents a method for cooperative roadway control at complex intersections using multi-agent association reinforcement learning.

2. In step 2, a cooperation state matrix having a size of (56*56*1) of the cooperation state amount and a sensor information matrix having a size of (3*1) are 2. The method for roadway cooperative control at a complex intersection using multi-agent association reinforcement learning according to claim 1.

3. In step 3, the neural network model structure used in the performance network in the reinforcement learning module in the FTD3 algorithm protects one convolutional layer and four fully connected layers, and in the last layer of the network, a tanh activation function is used to map the output to the [-1, 1] interval, and a relu activation function is used in the other layers. Similarly, the critic network uses one convolutional layer and four fully connected layers, and in the last layer of the network, a relu activation function is used in the other layers, and in addition to directly outputting and evaluating the Q value without using an activation function.

2. The method for roadway cooperative control at a complex intersection using multi-agent association reinforcement learning according to claim 1.

4. In step 4, in the training network process, the learning rates selected for the director and critic networks are both 0.0001, the policy noise is 0.2, the delay update parameter is 2, the discount coefficient γ is 0.95, the target network update weight tau is 0.995, the maximum capacity of the experience pool is selected as 10000, and the minibatch extracted from the experience pool is 128.

2. The method for roadway cooperative control at a complex intersection using multi-agent association reinforcement learning according to claim 1.

5. In step 5, the six neural networks used in the agent RSU participate in the aggregation but do not participate in the training, and only some of the neural networks are selected to participate in the aggregation, and the target networks that produce the most small Q values ​​are selected for aggregation.

2. The method for roadway cooperative control at a complex intersection using multi-agent association reinforcement learning according to claim 1.

Images