Distributed multi-agent traffic signal cooperative control method

Abstract

The invention provides a distributed multi-agent traffic signal cooperative control method, and belongs to the field of intelligent traffic control. The method comprises the steps of performing initialization operation on a road network environment, constructing an adjacency list according to a topological structure of an urban road network, and regarding each traffic signal lamp control intersection in the urban road network as an independent intelligent agent; for each intelligent agent, acquiring a local observation state of the intersection where the intelligent agent is located; performing action selection through sampling according to the action probability distribution; sending the selected action to the traffic simulator by using the intelligent agent, updating the state of the traffic signal lamp, and calculating and returning the intelligent agent information of the next time step by using the traffic simulator; sending the action probability distribution to all neighbor agents; and parameter updating is carried out until a preset training round is reached, and cooperative control of the traffic signals is completed. By adopting the distributed multi-agent reinforcement learning framework, the exploration efficiency is improved, and the collaboration among the agents is promoted.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent traffic control, and in particular relates to a distributed multi-agent traffic signal collaborative control method. Background Art

[0002] Urban traffic congestion has become a serious challenge faced by major cities around the world. Traditional traffic signal control methods, such as timed control and sensor control, are unable to cope with complex and changing traffic flows. Timed control relies on a pre-set fixed timing scheme and cannot be dynamically adjusted according to real-time traffic conditions, resulting in low traffic efficiency at intersections, especially when traffic demand fluctuates greatly. Although sensor control can make certain adjustments based on the information from vehicle detectors, its adjustment range is limited and it lacks a macro grasp of the overall traffic conditions of the road network.

[0003] With the rise of artificial intelligence and reinforcement learning technology, more and more research is devoted to applying reinforcement learning to traffic signal control. The single-agent reinforcement learning method treats a single intersection as an independent agent and optimizes the signal timing of the intersection through learning. However, the urban traffic network is a complex system with close interdependence between intersections. The single-agent method is difficult to capture this global correlation and is prone to the phenomenon of "treating the symptoms rather than the root cause", that is, optimizing a single intersection may worsen the traffic conditions of adjacent intersections.

[0004] In order to overcome the limitations of the single-agent method, the multi-agent reinforcement learning (MARL) method came into being. The multi-agent reinforcement learning (MARL) method models each intersection as an independent agent and optimizes the traffic performance of the entire road network through collaboration and competition between agents. At present, some multi-agent reinforcement learning (MARL) methods have been applied to traffic signal control, such as: Independent Learning: Each agent learns its own strategy independently, completely ignoring the existence of other agents. The advantage of this method is that it is simple to implement, but due to the lack of coordination between agents, it usually leads to poor overall performance.

[0005] Centralized Learning: Use a central controller to learn the joint strategy of all agents. This method can theoretically fully consider the mutual influence between agents, but it has serious problems of high computational complexity and poor scalability, making it difficult to apply to large-scale road networks.

[0006] Value Decomposition Networks (VDN)-based method: decompose the global Q-value function into the sum of the Q-value functions of each agent. This method alleviates the computational pressure of centralized learning to a certain extent, but its value function decomposition method limits the expressive power of the model.

[0007] Attention-based methods (Multi-Agent Attention, MAA): The attention mechanism is used to selectively focus on the state and action information of other agents. This method can improve the efficiency of collaboration between agents, but the design and training of the attention mechanism is relatively complex and has a high computational overhead.

[0008] Although the above multi-agent reinforcement learning (MARL) method has improved the effect of traffic signal control to a certain extent, there are still the following problems to be solved: Insufficient exploration: Reinforcement learning needs to strike a balance between exploration (trying new actions) and exploitation (selecting the best known actions). Traditional multi-agent reinforcement learning (MARL) methods usually adopt ε-greedy or Boltzmann exploration strategies, which have low exploration efficiency and easily lead to the algorithm converging to the local optimal solution prematurely, and failing to find the global optimal signal control strategy.

[0009] Difficulty in collaboration: Effective collaboration between agents is the key to achieving excellent performance in multi-agent reinforcement learning (MARL). However, in complex traffic environments, it is difficult for agents to spontaneously establish an effective collaboration mechanism, especially in the absence of explicit communication. How to design a mechanism to promote collaboration between agents is an important research direction in the field of multi-agent reinforcement learning (MARL).

[0010] Poor scalability: Many existing multi-agent reinforcement learning (MARL) methods have a sharp increase in computational complexity when the number of agents increases, making them difficult to apply to large-scale, high-density urban road networks. How to reduce computational complexity and communication overhead while ensuring algorithm performance is the key to the practical application of multi-agent reinforcement learning (MARL) methods. Summary of the invention

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention provides a distributed multi-agent traffic signal collaborative control method to solve the following problems: insufficient exploration problem: the exploration strategy of the existing reinforcement learning method (such as ε-greedy) is inefficient, which can easily cause the algorithm to fall into the local optimum and fail to find the global optimal signal control strategy; difficult collaboration problem: in a multi-agent environment, it is difficult for agents to spontaneously establish an effective collaboration mechanism, resulting in the inability to maximize the improvement of overall traffic efficiency; scalability problem: the computational complexity of many existing multi-agent reinforcement learning methods will increase sharply when the number of agents increases, making it difficult to apply them to large-scale road networks.

[0012] In order to achieve the above objectives, the technical solution adopted by the present invention is: a distributed multi-agent traffic signal collaborative control method, comprising the following steps: S1, initialization stage: Initialize the urban road network environment, and build an adjacency table according to the topological structure of the urban road network. Each traffic light-controlled intersection in the urban road network is regarded as an independent intelligent agent, and each intelligent agent is initialized. Among them, the intelligent agents share local information through the adjacency table; S2, environment interaction stage: each agent independently obtains the local observation state of the intersection where it is located; S3, decision-making stage: based on the local observation state obtained by each agent, the action probability distribution is obtained, and the agent selects the action through sampling according to the action probability distribution, where the action probability distribution represents the probability of the agent selecting each available traffic signal phase; S4, execution phase: Each agent sends the selected action to the traffic simulator, updates the state of the traffic lights and simulates the movement of vehicles, and uses the traffic simulator to calculate and return the agent information of the next time step, where the agent information includes the reward of each agent and the next local observation state of each agent; S5, information sharing stage: each agent sends the action probability distribution to all its neighboring agents, where each agent only shares the action probability distribution, and the neighboring agents are determined by the adjacency list; S6, learning phase: according to the action probability distribution obtained in S3, the reward and the next local observation state of the traffic simulator obtained in S4, and the action probability distribution received from the neighboring agents obtained in S5, the parameters are updated, and the Actor network, the Critic network, and the adaptive temperature coefficients alpha and beta are updated by calculating the individual entropy and the joint entropy. The individual entropy is used to measure the randomness of the action selection of a single agent, and the joint entropy is used to measure the joint distribution of actions between neighboring agents. S7, loop iteration: Repeat S2 to S6 until the preset training rounds are reached. The agent learns to obtain the optimal signal control decision under different traffic conditions, and works with all its neighboring agents to complete the coordinated control of traffic signals.

[0013] Furthermore, the agent in S1 includes: Actor network, which is used to generate action probability distribution based on the local observation state observed by the agent, where the local observation state includes the lane queue length, average vehicle speed and number of vehicles at each entrance of the current intersection, as well as the phase state of the current traffic light and the duration of the phase; The Critic network is used to evaluate the value of the current state based on the local observation state observed by the agent, that is, to predict the long-term cumulative return of the current state; Optimizer, used to update the parameters of the Actor network, Critic network, and adaptive temperature coefficients alpha and beta; The entropy regularization module is used to calculate the individual entropy according to the action probability distribution output by the Actor network, and calculate the joint entropy based on each agent sending the action probability distribution to all neighboring agents determined by the adjacency table at each time step, wherein the adaptive temperature coefficients alpha and beta are used to adjust the strength of the individual entropy regularization and the joint entropy regularization respectively. The agent identification and adjacency relationship module is used to set a unique identifier for each agent and set the identifiers of all neighboring agents for each agent according to the adjacency table.

[0014] Furthermore, the expression of the individual entropy is as follows: H(π(·|s))=-Σπ(a|s)*log(π(a|s)) Among them, H(π(·|s)) represents individual entropy, H() represents the operation of finding entropy, π(a|s) represents the probability of taking action a in state s, and Σ represents the sum of all actions a.

[0015] Furthermore, the calculation process of the joint entropy is as follows: A1. Obtain the action probability distribution of the current agent output by the Actor network, and obtain the action probability distribution of all neighboring agents of the current agent according to the adjacency list; A2, perform outer product operation on the action probability distribution obtained in A1; A3, reshape the tensor obtained in A2, and calculate the reshaped tensor to complete the calculation of joint entropy. The reshaped tensor becomes a two-dimensional tensor with a shape of (batch_size, A). , where each row contains the probabilities of all joint actions in the batch, batch_size represents the batch size, A represents the number of all joint action combinations, and action_dim_N represents the size of the action space of the Nth agent.

[0016] Furthermore, the expression of the joint entropy is as follows: H_joint=-ΣP*log(P) Among them, H_joint represents the joint entropy and P represents the joint probability distribution.

[0017] Furthermore, the outer product operation is specifically as follows: Expand the probability distribution of the agent's actions in sequence; Multiply the expanded tensors to get a shape of The outer product operation is completed on the tensor of , where each element in the obtained tensor represents the probability of a set of joint actions, and action_dim_N represents the size of the action space of the Nth agent.

[0018] Furthermore, the updated expression of the adaptive temperature coefficient alpha is as follows: L_alpha=-log(alpha)*(H(π(·|s)+target_entropy_individual) Wherein, L_alpha represents the loss function value of the adaptive temperature coefficient alpha, alpha represents the adaptive temperature coefficient used to adjust the individual entropy regularization strength, H(π(·|s)) represents the individual entropy, target_entropy_individual represents the preset target individual entropy, and the preset target individual entropy is set to the size of the negative action space, and log(alpha) represents the logarithm of the adaptive temperature coefficient alpha; The updated expression of the adaptive temperature coefficient beta is as follows: L_beta=-log(beta)*(H(π_joint)+target_entropy_joint) Among them, L_beta represents the loss function value of the adaptive temperature coefficient beta, beta represents the adaptive temperature coefficient used to adjust the strength of the joint entropy regularization, log(beta) represents the logarithm of the adaptive temperature coefficient beta, H(π_joint) represents the joint entropy, target_entropy_joint represents the preset target joint entropy, and the preset target joint entropy is set to the negative size of the action space * the number of joint agents.

[0019] Furthermore, the loss function of the Critic network is expressed as follows: L_critic=(R+γ*V(s')-V(s))^2 Where L_critic represents the loss function of the Critic network, R represents the immediate reward, γ represents the discount factor, V(s) and V(s') represent the value estimates of the current state and the next state respectively; The expression of the loss function of the Actor network is as follows: L_actor=-(log(π(a|s))*A(s, a)+α*H(π(·|s))+β*H(π_joint)) Where L_actor represents the loss function of the Actor network, π(a|s) represents the probability of taking action a in state s, A(s, a) represents the advantage function, H(π(·|s)) represents the individual entropy, H(π_joint) represents the joint entropy, and α and β represent the adaptive temperature coefficients of the individual entropy and joint entropy, respectively.

[0020] The beneficial effects of the present invention are: (1) This paper adopts a distributed multi-agent reinforcement learning framework, treating each traffic light-controlled intersection in the urban road network as an independent agent. Agents share limited local information (only action probability distribution) through a predefined adjacency list. Each agent learns based on the A2C (AdvantageActor-Critic) algorithm framework, and innovatively introduces adaptive individual entropy regularization and adaptive joint entropy regularization mechanisms to improve exploration efficiency and promote collaboration between agents.

[0021] (2) Higher exploration efficiency: This invention introduces individual entropy regularization to encourage the agent to explore diverse action strategies, thus avoiding the algorithm from converging to a suboptimal solution too early. The adaptive temperature coefficient alpha can dynamically adjust the regularization strength according to the entropy value of the current strategy, so that the agent can achieve a better balance between exploration and exploitation, thereby exploring the state-action space more effectively.

[0022] (3) Stronger collaboration capability: The most prominent advantage of the present invention is the introduction of joint entropy regularization. Joint entropy regularization encourages adjacent agents to explore diverse joint actions, that is, it encourages them to take different action combinations, which is completely different from the traditional method of encouraging individual agents to explore independently. By maximizing the joint entropy, the agents not only have to explore their own action space, but also consider each other's action choices, thereby avoiding the situation where "all agents choose the same action" or "adjacent agents take conflicting actions". The adaptive temperature coefficient beta can dynamically adjust the regularization strength according to the entropy value of the current joint strategy, making the collaboration between agents more effective and flexible. The joint entropy calculation method (based on the outer product) proposed in the present invention can accurately measure the entropy of the joint distribution of the actions of adjacent agents, providing key technical support for achieving effective collaboration.

[0023] (4) Better scalability: The present invention adopts a fully distributed learning framework. Each agent only needs to learn based on its own local observation information and information from neighboring agents. No global information is required, and no central controller is required. This distributed architecture greatly reduces the computational complexity and communication overhead, allowing the system to be easily expanded to large-scale, high-density urban road networks.

[0024] (5) Unification of theoretical guarantee and practical performance: Many existing multi-agent reinforcement learning methods lack theoretical convergence guarantee, while the present invention is based on the A2C (Advantage Actor-Critic) algorithm framework and introduces entropy regularization, which can theoretically guarantee the convergence of the algorithm (or at least local convergence). At the same time, a large number of simulation experimental results show that the present invention has also achieved excellent performance in actual traffic signal control tasks, which is significantly better than traditional traffic signal control methods and other existing multi-agent reinforcement learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0026] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0027] Example like Figure 1 As shown, the present invention provides a distributed multi-agent traffic signal collaborative control method, and its implementation method is as follows: S1, initialization stage: Initialize the urban road network environment, and build an adjacency table according to the topological structure of the urban road network. Each traffic light-controlled intersection in the urban road network is regarded as an independent intelligent agent, and each intelligent agent is initialized. Among them, the intelligent agents share local information through the adjacency table; In this embodiment, the urban road network environment (traffic simulator) is initialized, an adjacency table is constructed according to the topological structure of the urban road network (defining the connection relationship between agents), and each traffic light-controlled intersection is regarded as an independent agent. Each agent is initialized, including initializing its Actor network, Critic network, optimizer, target entropy and other parameters. Local information is shared between agents through the adjacency table.

[0028] In this embodiment, the agent includes: Actor network, which is used to generate action probability distribution based on the local observation state observed by the agent, where the local observation state includes the lane queue length, average vehicle speed and number of vehicles at each entrance of the current intersection, as well as the phase state of the current traffic light and the duration of the phase; The Critic network is used to evaluate the value of the current state based on the local observation state observed by the agent, that is, to predict the long-term cumulative return of the current state; in this embodiment, the value output by the Critic network is applied in the S6 learning stage: calculating the TD error; serving as the basis for updating the parameters of the Critic network (the goal is to reduce the TD error); calculating the advantage function, which indirectly affects the update of the Actor network. In this embodiment, TD error = TD target value - current value, that is: TD error = TD_target - V(state).

[0029] Optimizer, used to update the parameters of the Actor network, Critic network, and adaptive temperature coefficients alpha and beta; The entropy regularization module is used to calculate the individual entropy according to the action probability distribution output by the Actor network, and calculate the joint entropy based on each agent sending the action probability distribution to all neighboring agents determined by the adjacency table at each time step, wherein the adaptive temperature coefficients alpha and beta are used to adjust the strength of individual entropy regularization and joint entropy regularization respectively, wherein the calculation process of the joint entropy is as follows: A1. Obtain the action probability distribution of the current agent (the current here does not refer to the current in the sense of time, because it is a multi-agent environment, so it refers to the individual) output by the Actor network, and obtain the action probability distribution of all neighboring agents of the current agent according to the adjacency list; A2. Perform an outer product operation on the action probability distribution obtained in A1. The outer product operation is as follows: Expand the probability distribution of the agent's actions in sequence; Multiply the expanded tensors to get a shape of The outer product operation is completed, where each element in the obtained tensor represents the probability of a set of joint actions, and action_dim_N represents the size of the action space of the Nth agent; A3, reshape the tensor obtained in A2, and calculate the reshaped tensor to complete the calculation of joint entropy. The reshaped tensor becomes a two-dimensional tensor with a shape of (batch_size, A). , where each row contains the probability of all joint actions in the batch, batch_size represents the batch size, which usually refers to the number of data samples processed in one training iteration in deep learning and reinforcement learning, A represents the number of all joint action combinations, and A defines the dimension of the joint action space. The joint action space refers to the space composed of all possible action combinations of all agents, and action_dim_N represents the size of the action space of the Nth agent.

[0030] The agent identification and adjacency relationship module is used to set a unique identifier for each agent and set the identifiers of all neighboring agents for each agent according to the adjacency table; In this embodiment, the present invention adopts a fully distributed architecture, where an intelligent agent is deployed at each intersection. The intelligent agents communicate with each other through message passing. There is no central controller, and each intelligent agent is responsible for controlling the traffic lights at an intersection.

[0031] In this embodiment, the urban road network environment (traffic simulator) and adjacency table are initialized; the Actor network, Critic network, target individual entropy, target joint entropy, adaptive temperature coefficients alpha and beta, and optimizer (Adam) of each intelligent agent are initialized.

[0032] In this embodiment, the main components of the agent include: (1) Actor network (PolicyNet): Its input is the local observation state of the agent. The local observation state may include: the lane queue length of each entrance lane of the current intersection; the average speed of vehicles at each entrance lane of the current intersection; the number of vehicles at each entrance lane of the current intersection; the phase state of the current traffic light (for example, red light, green light, yellow light); the duration of the current phase, and other relevant state features may also be added according to the actual application scenario.

[0033] The output of the Actor network (PolicyNet) is: the probability distribution of the agent on each optional action. In the present invention, the action is a discrete signal phase selection, for example, the phases such as "east-west straight", "east-west left turn", "north-south straight", "north-south left turn" can be selected.

[0034] The network structure of the Actor network: It uses a multi-layer perceptron (MLP) structure, which includes an input layer, multiple hidden layers, and an output layer. The activation function uses ReLU (Rectified Linear Unit). The output layer uses the Softmax function to ensure that the output is a legal probability distribution.

[0035] (2) Critic network (ValueNet): The input of the Critic network is: the local observation state of the agent. The local observation state may include: the lane queue length of each entrance lane of the current intersection; the average speed of vehicles at each entrance lane of the current intersection; the number of vehicles at each entrance lane of the current intersection; the phase state of the current traffic light (for example, red light, green light, yellow light); the duration of the current phase; other relevant state features may also be added according to the actual application scenario.

[0036] The output of the Critic network is: the value estimate of the state, that is, the prediction of the long-term cumulative return of the current state.

[0037] The network structure of the Critic network: It adopts the MLP structure, which includes an input layer, multiple hidden layers and an output layer, and the activation function uses ReLU.

[0038] (3) Entropy regularization module includes: 1. Calculate individual entropy: Calculate the individual entropy of the current strategy based on the action probability distribution output by the Actor network. Individual entropy measures the randomness of the action selection of a single agent. The formula is as follows: H(π(·|s))=-Σπ(a|s)*log(π(a|s)) Among them, H(π(·|s)) represents individual entropy, H() represents the operation of finding entropy, π(a|s) represents the probability of taking action a in state s, and Σ represents the sum of all actions a.

[0039] More specifically, H() is a function that takes as input an action probability distribution (in this case, the policy π(·|s)) and outputs the entropy of this action probability distribution (a number), mapping an action probability distribution to a number (entropy) that represents its degree of uncertainty.

[0040] The purpose of calculating individual entropy is: individual entropy regularization encourages the agent to explore different actions and avoid converging to suboptimal strategies too early.

[0041] 2. Calculate joint entropy: Joint entropy measures the entropy of the joint distribution of actions between adjacent agents. Unlike individual entropy, which encourages a single agent to explore different actions, joint entropy encourages adjacent agents to explore different combinations of actions, thereby promoting collaboration between them and avoiding situations where "everyone chooses the same action" or "conflicting actions". The joint entropy calculation method is explained in detail as follows: Probability tensor construction: For each agent, first obtain its own action probability distribution (output by the Actor network, shape (batch_size, action_dim)). Then, obtain the action probability distribution of all its neighboring agents based on the adjacency list. batch_size represents the batch size, and action_dim represents the dimension of the action space, that is, the number of different actions that each agent can take.

[0042] Outer Product calculation: Perform an outer product operation on the action probability distribution. The outer product operation is the key step in calculating the joint entropy of the present invention. Specifically, assuming that the total number of the current agent and its neighboring agents is N, perform the following operations: expand the action probability distribution of the first agent (shape is (batch_size, action_dim_1)) to (batch_size, action_dim_1, 1, 1, ...) (N-1 1s); expand the action probability distribution of the second agent (shape is (batch_size, action_dim_2)) to (batch_size, 1, action_dim_2, 1, ...); and so on, expand the action probability distribution of the Nth agent to (batch_size, 1, 1, ..., action_dim_N); then, multiply these expanded tensors. Due to the broadcast mechanism of PyTorch, this multiplication operation actually calculates the outer product of these probability distributions, and obtains a tensor of shape (batch_size, action_dim_1, action_dim_2,..., action_dim_N). Each element in this tensor represents the probability of a corresponding set of joint actions (that is, a combination of a specific action taken by each agent), and action_dim_N represents the size of the action space of the Nth agent.

[0043] Reshape: Reshape the tensor obtained in the previous step into (batch_size, action_dim_1 *action_dim_2 * ,...,* action_dim_N). In this way, each row represents the probability of a joint action of all relevant agents (the current agent and its neighbors) in a batch of data.

[0044] Entropy calculation: Finally, the entropy is calculated for the reshaped tensor, and the formula is as follows: H_joint=-ΣP*log(P) Among them, H_joint represents joint entropy, P represents joint probability distribution, and represents the probability value of different joint action combinations. This formula calculates the weighted sum of the probability of all joint actions and their logarithms, and then takes the negative value to get the joint entropy.

[0045] The calculation process of joint entropy can be seen as combining the action probability distributions of multiple agents into a joint probability distribution, and then calculating the entropy of this joint distribution. The outer product operation is the key to achieving this combination. It "expands" multiple independent probability distributions into a high-dimensional tensor, where each element represents the probability of a possible joint action. The reshape operation "flattens" this high-dimensional tensor into a two-dimensional tensor to facilitate subsequent entropy calculations. Finally, according to the entropy formula, the entropy of this joint probability distribution is calculated as an indicator to measure the diversity of collaborative actions of adjacent agents.

[0046] Calculate the adaptive temperature coefficient: Individual entropy and joint entropy correspond to two learnable adaptive temperature coefficients alpha and beta, respectively, which are used to adjust the strength of entropy regularization. These two temperature coefficients are another key innovation of the present invention. They are not fixed hyperparameters, but are adaptively adjusted by gradient descent.

[0047] Update of adaptive temperature coefficient alpha: The update goal of adaptive temperature coefficient alpha is to make individual entropy close to a preset target individual entropy value (target_entropy_individual), the formula is as follows: L_alpha=-log(alpha)*(H(π(·|s)+target_entropy_individual) Among them, L_alpha represents the loss function value of the adaptive temperature coefficient alpha, alpha represents the adaptive temperature coefficient used to adjust the individual entropy regularization strength, H(π(·|s)) represents the individual entropy, target_entropy_individual represents the preset target individual entropy, and the preset target individual entropy is set to the size of the negative action space, and log(alpha) represents the logarithm of the adaptive temperature coefficient alpha.

[0048] Adaptive temperature coefficient alpha update process: Clear the old gradient: Use the optimizer to reset the calculated adaptive temperature coefficient alpha parameter gradient to zero; Calculate the new gradient: According to the adaptive temperature coefficient alpha loss, automatically calculate the direction and magnitude (gradient) of the adaptive temperature coefficient alpha parameter that needs to be adjusted; Update the adaptive temperature coefficient alpha: Use the Adam optimizer to automatically adjust the adaptive temperature coefficient alpha parameter according to the calculated gradient (actually adjust the logarithm of the adaptive temperature coefficient alpha).

[0049] Update of adaptive temperature coefficient beta: The update goal of adaptive temperature coefficient beta is to make the joint entropy close to a preset target joint entropy value (target_entropy_joint). The formula is as follows: L_beta=-log(beta)*(H(π_joint)+target_entropy_joint) Among them, L_beta represents the loss function value of the adaptive temperature coefficient beta, beta represents the adaptive temperature coefficient used to adjust the strength of the joint entropy regularization, log(beta) represents the logarithm of the adaptive temperature coefficient beta, H(π_joint) represents the joint entropy, target_entropy_joint represents the preset target joint entropy, and the preset target joint entropy is set to the negative size of the action space * the number of joint agents.

[0050] Update of adaptive temperature coefficient beta: Clear the old gradient: Use the optimizer to reset the calculated adaptive temperature coefficient beta parameter gradient to zero; Calculate the new gradient: According to the adaptive temperature coefficient beta loss, automatically calculate the direction and magnitude (gradient) of the adaptive temperature coefficient beta parameter adjustment; Update the adaptive temperature coefficient beta: Use the Adam optimizer to automatically adjust the adaptive temperature coefficient beta parameter according to the calculated gradient (actually adjust the logarithm of the adaptive temperature coefficient beta).

[0051] (4) Optimizer: The Adam optimizer is used to update the parameters of the Actor network, the Critic network, and the two adaptive temperature coefficients (alpha and beta). Adam is a commonly used stochastic gradient descent optimization algorithm with the advantage of adaptive learning rate.

[0052] In this embodiment, the adjacency list is used to define the communication topology between agents. It is a dictionary, the key is the name (or ID) of the agent, and the value is a list containing the names (or IDs) of all agents adjacent to the agent. The adjacency list specifies which agents can exchange information. In the present invention, the adjacency list is predefined based on the actual topology of the road network, that is, the agents corresponding to adjacent intersections are neighbors.

[0053] S2, environment interaction stage: each agent independently obtains the local observation state of the intersection where it is located, such as lane queue length, average vehicle speed, etc.; In this embodiment, for each agent, the current local observation state is obtained. According to the action probability distribution output by the Actor network, an action (signal phase) is sampled, the action is applied to the environment (traffic simulator), and the next state, reward, termination, and other information are obtained.

[0054] In this embodiment, at the beginning of each simulation time step, each agent independently observes the local state of the intersection where it is located. The local state information is the basis for the agent to make decisions, and usually includes: the queue length of each entrance lane; the average speed of vehicles on each entrance lane; the number of vehicles on each entrance lane; the state of the current signal phase; the duration of the current phase; and other information related to traffic conditions.

[0055] S3, decision-making stage: based on the local observation state obtained by each agent, the action probability distribution is obtained, and the agent selects the action through sampling according to the action probability distribution, where the action probability distribution represents the probability of the agent selecting each available traffic signal phase; In this embodiment, each agent obtains an action probability distribution through the Actor network based on its local observation state. The action probability distribution represents the probability of the agent selecting each available traffic signal phase (e.g., east-west straight, north-south left turn, etc.). Then, the agent samples according to the action probability distribution and selects a specific action (i.e., signal phase). That is, each agent inputs the collected local observation state into its Actor network, and the Actor network outputs an action probability distribution after processing. The action probability distribution represents the probability of the agent selecting each available signal phase. The agent selects a specific action (signal phase) through sampling according to the action probability distribution, and the agent records the action probability distribution this time.

[0056] S4, execution phase: Each agent sends the selected action to the traffic simulator, updates the state of the traffic lights and simulates the movement of vehicles, and uses the traffic simulator to calculate and return the agent information of the next time step, where the agent information includes the reward of each agent and the next local observation state of each agent; In this example, each agent sends the selected action (signal phase) to the traffic simulator. The simulator updates the state of the traffic light based on the action and simulates the movement of vehicles. The traffic simulator calculates and returns the agent information for the next time step, including each agent's reward (e.g., negative waiting time) and the next local observation state (next_state).

[0057] In this embodiment, in the execution phase S4, the results obtained after the agent interacts with the traffic simulator (the reward reward of each agent and the next local observation state next_state) are applied to the learning phase S6. The specific application method is as follows: Using the agent's reward and the next local observation state next_state, as well as a discount factor (gamma), calculate the "target value" (TD target), which represents an estimate of future cumulative rewards. Use the TD target calculated using the reward to subtract the value of the current state to get the advantage function and TD error. The advantage function is used to update the Actor network, and the TD error is used to update the Critic network.

[0058] S5, information sharing stage: each agent sends the action probability distribution to all its neighboring agents, where each agent only shares the action probability distribution, and the neighboring agents are determined by the adjacency list; In this embodiment, each agent sends its action probability distribution (i.e., the output of the Actor network in stage S3) to all its neighboring agents (neighbors are determined based on the adjacency table constructed in stage S1). Agents only share action probability distributions, and do not share other information such as observation status.

[0059] In this embodiment, the information sharing result of S5 (the action probability distribution of neighboring agents) is used to calculate the joint entropy between neighboring agents in the S6 learning phase. As a regularization method, the joint entropy is incorporated into the loss function of the Actor network to encourage cooperation between neighboring agents, avoid them all taking the same action or conflicting actions, and ultimately improve the performance of the entire system.

[0060] In this embodiment, agents share information through message passing. Specifically, each agent sends the action probability distribution output by its Actor network to all its neighboring agents (determined according to the adjacency list) at each time step. After receiving this information, the neighboring agents use it to calculate the joint entropy. The present invention does not share the observed state of the agent, but only shares the action probability distribution, which helps to reduce communication overhead and protect the local privacy of the agent.

[0061] S6, learning phase: according to the action probability distribution obtained in S3, the reward and the next local observation state of the traffic simulator obtained in S4, and the action probability distribution received from the neighboring agents obtained in S5, the parameters are updated, and the Actor network, the Critic network, and the adaptive temperature coefficients alpha and beta are updated by calculating the individual entropy and the joint entropy. The individual entropy is used to measure the randomness of the action selection of a single agent, and the joint entropy is used to measure the joint distribution of actions between neighboring agents. In this embodiment, each agent receives: the reward and the next local observation state from the traffic simulator and the action probability distribution from all its neighboring agents.

[0062] Each agent uses the received information to update its internal Actor network, Critic network, and adaptive temperature coefficients alpha and beta. The update process is as follows: First, data preparation: Prepare the received data (which means converting the information into tensors); estimate the value V(state) of the current state, and estimate the value V(next_state) of the next state; calculate the TD target value (TD_target); calculate the TD error (Temporal Difference Error) and the Advantage function (which represents the advantage of taking a specific action compared to the average level). The Advantage function and the TD error are numerically the same here, that is, both equal TD_target - V(state), but their usage and meanings are different, so two terms are used for distinction.

[0063] The expression of the Advantage function is as follows: Advantage = TD_target - V(state) Among them, V(state) represents the estimate of the value of the current state state by the Critic network, and TD_target represents an estimated value of the future cumulative reward, that is, combining the immediate reward reward and the estimated value of the future expected reward V(next_state) to improve the evaluation of the value of the current state.

[0064] The expression of the target value TD_target is as follows: TD_target = reward + gamma * V(next_state) * (1 - done) Among them, reward represents the immediate reward obtained at the current time step, gamma represents the discount factor, which represents the importance of future rewards (0 < gamma <= 1), V(next_state) represents the estimate of the value of the next state by the Critic network, and done represents a boolean value indicating whether the current episode has ended (1 means ended, 0 means not ended); Then, calculate the Individual Entropy. The calculation process is as follows: According to the action probability distribution output by the current agent's Actor network, calculate the individual entropy of the current policy. The individual entropy measures the randomness (or uncertainty) of the action selection of a single agent.

[0065] Next, calculate the Joint Entropy. The calculation process is as follows: If the current agent has neighbor agents, then calculate the joint entropy; obtain the action probability distributions of all neighbor agents; also add the action probability distribution of the current agent itself; calculate the joint entropy of these probability distributions. The joint entropy measures the diversity (or uncertainty) of the joint actions of adjacent agents; update the target joint entropy.

[0066] Then calculate the loss function, the calculation process is as follows: Critic network loss: measures the accuracy of the critic network’s estimate of the state value; Actor network loss: It contains three main parts: (1) Encouraging agents to take actions with higher advantages (i.e., advantage function term); (2) Individual entropy regularization term: Encouraging agents to explore diverse actions. By maximizing individual entropy, agents will not take only one action too early, but will be more likely to try various actions; (3) Joint entropy regularization term: Encouraging adjacent agents to explore diverse joint actions, promoting cooperation between them, and avoiding all agents taking the same action or taking conflicting actions. Among them, the adaptive temperature coefficient alpha loss is used to adjust the adaptive temperature coefficient alpha so that the individual entropy is close to the preset target individual entropy. The adaptive temperature coefficient beta loss is used to adjust the adaptive temperature coefficient beta so that the joint entropy is close to the preset target joint entropy.

[0067] Finally, parameter update: update the parameters of the Actor network, Critic network, and adaptive temperature coefficients alpha and beta respectively. The update process includes the following three steps: Gradient clearing: clear the previous gradient information; Back propagation: calculate the gradient of each parameter according to the loss function; Parameter update: adjust the parameter value according to the gradient.

[0068] In this embodiment, the loss is calculated as follows: For each agent: Calculate the loss of the Critic network: Use the square of the TD error (Temporal DifferenceError) as the loss function, the formula is as follows: L_critic=(R+γ*V(s')-V(s))^2 Among them, L_critic represents the loss function of the Critic network, R represents the immediate reward, γ represents the discount factor, V(s) and V(s') represent the value estimates of the current state and the next state respectively.

[0069] Calculate the loss of the Actor network: It includes three parts: Advantage Function, individual entropy regularization term and joint entropy regularization term. The formula is as follows: L_actor=-(log(π(a|s))*A(s, a)+α*H(π(·|s))+β*H(π_joint)) Where L_actor represents the loss function of the Actor network, π(a|s) represents the probability of taking action a in state s, A(s, a) represents the advantage function, H(π(·|s)) represents the individual entropy, H(π_joint) represents the joint entropy, and α and β represent the adaptive temperature coefficients of the individual entropy and joint entropy, respectively.

[0070] Calculate the loss of the adaptive temperature coefficient alpha: L_alpha=-log(alpha)*(H(π(·|s)+target_entropy_individual) Among them, L_alpha represents the loss function value of the adaptive temperature coefficient alpha, alpha represents the adaptive temperature coefficient used to adjust the individual entropy regularization strength, H(π(·|s)) represents the individual entropy, target_entropy_individual represents the preset target individual entropy, and the preset target individual entropy is set to the size of the negative action space, and log(alpha) represents the logarithm of the adaptive temperature coefficient alpha.

[0071] Calculate the loss of the adaptive temperature coefficient beta: L_beta=-log(beta)*(H(π_joint)+target_entropy_joint) Among them, L_beta represents the loss function value of the adaptive temperature coefficient beta, beta represents the adaptive temperature coefficient used to adjust the strength of the joint entropy regularization, log(beta) represents the logarithm of the adaptive temperature coefficient beta, H(π_joint) represents the joint entropy, target_entropy_joint represents the preset target joint entropy, and the preset target joint entropy is set to the negative size of the action space * the number of joint agents.

[0072] In this embodiment, the purpose of individual entropy regularization is to encourage each agent to independently explore more possible actions and avoid premature convergence to suboptimal strategies. The mechanism is as follows: by adding individual entropy to the Actor network loss function, the agent tends to maximize the randomness of its action selection while maximizing the cumulative reward.

[0073] The adaptive temperature coefficient alpha is used to control the strength of individual entropy regularization. The larger the adaptive temperature coefficient alpha, the greater the encouragement for individual entropy, and the more the agent tends to explore; the smaller the adaptive temperature coefficient alpha, the smaller the encouragement for individual entropy, and the more the agent tends to use the learned knowledge. The adaptive temperature coefficient alpha will be adjusted automatically.

[0074] In this embodiment, the purpose of joint entropy regularization is to encourage neighboring agents to explore different combinations of joint actions, promote effective collaboration between them, and avoid situations where "everyone chooses the same action" or "actions conflict with each other". Its mechanism is to add joint entropy to the Actor network loss function, so that the agent tends to maximize the diversity of its joint actions with neighboring agents while maximizing the cumulative reward.

[0075] The role of the adaptive temperature coefficient beta is to control the strength of the joint entropy regularization. The larger the adaptive temperature coefficient beta is, the greater the encouragement for the joint entropy, and the more agents tend to take different action combinations; the smaller the adaptive temperature coefficient is, the smaller the encouragement for the joint entropy is. The adaptive temperature coefficient will be adjusted automatically.

[0076] In this embodiment, the adaptive temperature coefficients alpha and beta are not fixed, but are automatically adjusted. The advantage is that the exploration and utilization are dynamically balanced: the adaptive adjustment of the adaptive temperature coefficient alpha enables the agent to conduct more exploration in the early stage of training, and gradually reduce exploration in the later stage of training. Dynamic balance between individuals and collaboration: the adaptive adjustment of the adaptive temperature coefficient beta enables the agent to adjust the degree of collaboration according to the current learning state.

[0077] S7, loop iteration: Repeat S2 to S6 until the preset training rounds are reached to complete the coordinated control of traffic signals.

[0078] In this embodiment, the process from S2 to S6 is repeated, and environmental interaction, decision-making, execution, information sharing and learning are continuously performed until the preset training rounds are reached (or other stopping conditions are met). Through this cyclic iterative process, the agent gradually learns how to make the best signal control decision under different traffic conditions, and works in collaboration with other agents to achieve collaborative control of the entire traffic network.

[0079] In summary, the beneficial effects of the present invention are: Improve exploration efficiency: By introducing the adaptive dual regularization mechanism of individual entropy and joint entropy, the intelligent agent is encouraged to explore diverse behavioral strategies, thereby improving the algorithm's exploration efficiency and global optimization ability; Enhanced agent collaboration: By combining entropy regularization and an adjacency table-based information sharing mechanism, we promote collaboration among agents, enabling them to jointly optimize the overall traffic performance of the road network rather than just the local optimum of a single intersection. Improve scalability: The distributed learning framework and local communication mechanism are used to reduce the computational complexity and communication overhead of the algorithm, making the system applicable to large-scale, high-density urban road networks.

Claims

1. A distributed multi-agent traffic signal collaborative control method, characterized in that: The following steps are involved: S1, initialization stage: Initialize the urban road network environment, and build an adjacency table according to the topological structure of the urban road network. Each traffic light-controlled intersection in the urban road network is regarded as an independent intelligent agent, and each intelligent agent is initialized. Among them, the intelligent agents share local information through the adjacency table; S2, environment interaction stage: each agent independently obtains the local observation state of the intersection where it is located; S3, decision-making stage: based on the local observation state obtained by each agent, the action probability distribution is obtained, and the agent selects the action through sampling according to the action probability distribution, where the action probability distribution represents the probability of the agent selecting each available traffic signal phase; S4, execution phase: Each agent sends the selected action to the traffic simulator, updates the state of the traffic lights and simulates the movement of vehicles, and uses the traffic simulator to calculate and return the agent information of the next time step, where the agent information includes the reward of each agent and the next local observation state of each agent; S5, information sharing stage: each agent sends the action probability distribution to all its neighboring agents, where each agent only shares the action probability distribution, and the neighboring agents are determined by the adjacency list; S6, learning phase: according to the action probability distribution obtained in S3, the reward and the next local observation state of the traffic simulator obtained in S4, and the action probability distribution received from the neighboring agents obtained in S5, the parameters are updated, and the Actor network, the Critic network, and the adaptive temperature coefficients alpha and beta are updated by calculating the individual entropy and the joint entropy. The individual entropy is used to measure the randomness of the action selection of a single agent, and the joint entropy is used to measure the joint distribution of actions between neighboring agents. S7, loop iteration: Repeat S2 to S6 until the preset training rounds are reached. The agent learns to obtain the optimal signal control decision under different traffic conditions, and works with all its neighboring agents to complete the coordinated control of traffic signals.

2. The distributed multi-agent traffic signal cooperative control method according to claim 1 is characterized in that: The intelligent agent in S1 includes: Actor network, which is used to generate action probability distribution based on the local observation state observed by the agent, where the local observation state includes the lane queue length, average vehicle speed and number of vehicles at each entrance of the current intersection, as well as the phase state of the current traffic light and the duration of the phase; The Critic network is used to evaluate the value of the current state based on the local observation state observed by the agent, that is, to predict the long-term cumulative return of the current state; Optimizer, used to update the parameters of the Actor network, Critic network, and adaptive temperature coefficients alpha and beta; The entropy regularization module is used to calculate the individual entropy according to the action probability distribution output by the Actor network, and calculate the joint entropy based on each agent sending the action probability distribution to all neighboring agents determined by the adjacency table at each time step, wherein the adaptive temperature coefficients alpha and beta are used to adjust the strength of the individual entropy regularization and the joint entropy regularization respectively. The agent identification and adjacency relationship module is used to set a unique identifier for each agent and set the identifiers of all neighboring agents for each agent according to the adjacency table.

3. The distributed multi-agent traffic signal collaborative control method according to claim 2 is characterized in that: The expression of the individual entropy is as follows: H(π(·|s))=-Σπ(a|s)*log(π(a|s)) Among them, H(π(·|s)) represents individual entropy, H() represents the operation of finding entropy, π(a|s) represents the probability of taking action a in state s, and Σ represents the sum of all actions a.

4. The distributed multi-agent traffic signal cooperative control method according to claim 2 is characterized in that: The calculation process of the joint entropy is as follows: A1. Obtain the action probability distribution of the current agent output by the Actor network, and obtain the action probability distribution of all neighboring agents of the current agent according to the adjacency list; A2, perform outer product operation on the action probability distribution obtained in A1; A3, reshape the tensor obtained in A2, and calculate the reshaped tensor to complete the calculation of joint entropy. The reshaped tensor becomes a two-dimensional tensor with a shape of (batch_size, A). , where each row contains the probabilities of all joint actions in the batch, batch_size represents the batch size, A represents the number of all joint action combinations, and action_dim_N represents the size of the action space of the Nth agent.

5. The distributed multi-agent traffic signal cooperative control method according to claim 4 is characterized in that: The expression of the joint entropy is as follows: H_joint=-ΣP*log(P) Among them, H_joint represents the joint entropy and P represents the joint probability distribution.

6. The distributed multi-agent traffic signal cooperative control method according to claim 4 is characterized in that: The outer product operation is specifically as follows: Expand the probability distribution of the agent's actions in sequence; Multiply the expanded tensors to get a shape of The outer product operation is completed on the tensor of , where each element in the obtained tensor represents the probability of a set of joint actions, and action_dim_N represents the size of the action space of the Nth agent.

7. The distributed multi-agent traffic signal cooperative control method according to claim 2 is characterized in that: The updated expression of the adaptive temperature coefficient alpha is as follows: L_alpha=-log(alpha)*(H(π(·|s)+target_entropy_individual) Wherein, L_alpha represents the loss function value of the adaptive temperature coefficient alpha, alpha represents the adaptive temperature coefficient used to adjust the individual entropy regularization strength, H(π(·|s)) represents the individual entropy, target_entropy_individual represents the preset target individual entropy, and the preset target individual entropy is set to the size of the negative action space, and log(alpha) represents the logarithm of the adaptive temperature coefficient alpha; The updated expression of the adaptive temperature coefficient beta is as follows: L_beta=-log(beta)*(H(π_joint)+target_entropy_joint) Among them, L_beta represents the loss function value of the adaptive temperature coefficient beta, beta represents the adaptive temperature coefficient used to adjust the strength of the joint entropy regularization, log(beta) represents the logarithm of the adaptive temperature coefficient beta, H(π_joint) represents the joint entropy, target_entropy_joint represents the preset target joint entropy, and the preset target joint entropy is set to the negative size of the action space * the number of joint agents.

8. The distributed multi-agent traffic signal cooperative control method according to claim 2 is characterized in that: The expression of the loss function of the Critic network is as follows: L_critic=(R+γ*V(s')-V(s))^2 Where L_critic represents the loss function of the Critic network, R represents the immediate reward, γ represents the discount factor, V(s) and V(s') represent the value estimates of the current state and the next state respectively; The expression of the loss function of the Actor network is as follows: L_actor=-(log(π(a|s))*A(s, a)+α*H(π(·|s))+β*H(π_joint)) Where L_actor represents the loss function of the Actor network, π(a|s) represents the probability of taking action a in state s, A(s, a) represents the advantage function, H(π(·|s)) represents the individual entropy, H(π_joint) represents the joint entropy, and α and β represent the adaptive temperature coefficients of the individual entropy and joint entropy, respectively.

Images