Personalized multi-agent reinforcement learning traffic control method based on hyperpolicy network

Abstract

The invention provides a personalized multi-agent reinforcement learning traffic control method based on a hyperpolicy network, and belongs to the technical field of traffic signal control. Constructing a reinforcement learning traffic signal control model, wherein the reinforcement learning traffic signal control model comprises a strategy network, a value network and a hyperstrategy network; the strategy network outputs action distribution, selects actions through probability sampling, and controls traffic signals; the value network outputs a value function; the hyper-policy network outputs a hyper-action; obtaining a joint value function through a super-action and value function dot product; the value network evaluates the strategy network through a joint value function; the reinforcement learning traffic signal control model performs model training through a loss function, including updating a value network through a mean square error; and the strategy network is updated by limiting the change range of new and old strategies through gradient clipping. According to the method, all agents share parameters, so that the training efficiency of the model is improved, and the generalization ability of the strategy is improved; the training efficiency is ensured, meanwhile, the individuation of the intersection is considered, the performance is improved, and the training cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic signal control, and in particular to a personalized multi-agent reinforcement learning traffic control method based on a super strategy network. Background Art

[0002] Traffic signal control (TSC) is essentially a multi-agent problem, in which each intersection can be regarded as an independent agent. In complex traffic systems, the coordination mechanism between agents is particularly critical, which not only needs to consider the dynamic impact of adjacent intersections, but also needs to accurately grasp the traffic flow characteristics of the same intersection at different time periods. Given that real-world traffic networks often contain a large number of intersections, these intersections cannot be simply treated uniformly or considered in isolation. In recent years, adaptive traffic signal control (ATSC) has made significant progress with the development of multi-agent reinforcement learning (MARL). In traffic scenarios with multiple intersections, MARL regards each intersection as an agent and optimizes traffic signal control strategies through learning and real-time decision-making. Multi-agent reinforcement learning (MARL) aims to jointly train a group of agents to handle specific tasks in a shared environment. Existing multi-agent reinforcement learning mainly focuses on optimizing feature extraction techniques, improving agent modeling, and designing hierarchical cooperation strategies. Therefore, how to effectively extend multi-agent reinforcement learning to scenarios involving a large number of agents, especially in complex traffic signal control (TSC), has been neglected.

[0003] The common technique for coordinating and controlling large-scale traffic signal network training using multi-agent systems is parameter sharing, where agents share network parameters. Under similar environmental conditions between many intersections, agents can find similar observation and reward functions, and this sharing is very effective. However, although parameter sharing can speed up the training of large-scale agents, treating each agent indiscriminately will lose some information, thereby ignoring the personalization of intersections and failing to learn the optimal strategy. As the scale of traffic gradually increases, increasing the scale of network parameters is a natural choice. However, simply increasing the scale of the network does not necessarily improve the generalization ability of the strategy, and the deployment cost of actual traffic signal control equipment will also increase. In real scenarios, the differences in traffic environments are still very obvious. Therefore, existing technologies also try to promote the diversity of agents by training multiple strategies in groups. This selective sharing method can benefit from parameter sharing and take into account the diversity of agents, but it is highly dependent on whether the agents can be stably and correctly divided, lacks stability, and does not take into account the preferences of each agent.

[0004] Therefore, a traffic signal control method based on multi-agent reinforcement learning is needed that can consider intersection personalization and exploit the efficiency of parameter sharing. Summary of the invention

[0005] In view of this, the present invention provides a personalized multi-agent reinforcement learning traffic control method based on a super strategy network, which can take into account both model training efficiency and intersection model personalization.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A personalized multi-agent reinforcement learning traffic control method based on a super-strategy network, including:

[0008] Constructing a reinforcement learning traffic signal control model; the model includes: a strategy network, a value network, and a super strategy network;

[0009] The policy network outputs an action distribution, selects actions through probability sampling, and controls traffic signals;

[0010] The value network outputs a value function; the super-strategy network outputs a super-action;

[0011] The joint value function is obtained by dot product of the super action and the value function; the value network evaluates the strategy network through the joint value function;

[0012] The reinforcement learning traffic signal control model is trained by a loss function;

[0013] The training model process includes:

[0014] Update the value network by the mean square error;

[0015] Update the policy network by limiting the range of changes between the old and new policies through gradient clipping.

[0016] Furthermore, the policy network outputs an action distribution, selects an action through probability sampling, and controls traffic signals, including:

[0017] Enter local observation history

[0018] Using observation history to obtain embedded features through multi-layer perceptron

[0019] The temporal features of the intersection are extracted through the GRU unit, and the hidden features are output

[0020] The hidden features generate action probability distribution through a linear layer;

[0021] The action is selected by sampling the action probability to control the traffic signal.

[0022] Furthermore, the super-strategy network outputs a super-action, including: generating a super-action through a linear layer using hidden features

[0023]

[0024] Among them, W e is the weight matrix, b e is the bias vector, is the cyclic hidden feature of intersection i at time t output by the GRU unit, and ⊕ is the concatenation operation.

[0025] Furthermore, the value network outputs a value function, including:

[0026] The embedding features of each intersection are obtained through the embedding layer

[0027] The embedding features are transformed into The information of its neighbors is integrated according to importance; the integrated embedded features are passed through a linear layer to estimate the multidimensional value output for each intersection model

[0028] Furthermore, the policy network and the value network share an embedding layer.

[0029] Further, obtaining the joint value estimate by performing a dot product of a super action and a value function includes:

[0030] Using Super Action express Probability distribution of the data items in , and calculate joint value estimates for each intersection;

[0031] The multi-valued function and the hyper-action have the same dimension k, and the joint value estimate is generated through the dot product:

[0032]

[0033] Among them, O t is the global observation, G is the graph network structure of the intersection, is the cycle characteristic of the previous moment, i is the intersection subscript, For multidimensional valued function estimation, For super action, is the local observation of intersection i.

[0034] Furthermore, the value network is updated by the mean square error:

[0035]

[0036] in, represents the value network parameters, represents the joint value function, N represents the number of intersections, T represents the time step, represents the reward of intersection i at time t, and γ represents the discount factor.

[0037] Furthermore, the value network evaluates the policy network through a joint value function, including introducing an entropy regularization term for the super-action distribution, expressed as in represents the parameterized by θ

[0038] Furthermore, the updating of the strategy network by limiting the variation range of the new and old strategies through gradient clipping includes:

[0039]

[0040] in, is importance sampling, π θ is a parameterized policy network, is the advantage function, is the action of intersection i at time t, ε is a hyperparameter, and λ is the regularization coefficient.

[0041] Advantages and positive effects of the present invention:

[0042] The present invention adopts a centralized training and decentralized execution (CTDE) mode, in which all agents share parameters; it improves the training efficiency of the model and improves the generalization ability of the strategy;

[0043] The centralized critic network takes global observations as input to estimate the travel time of each intersection, and its output is a vector containing multiple value functions. Multiple value functions are estimated and combined with the super actions output by the super policy network to provide different value functions for all intersections, thereby guiding the learning of personalized strategies.

[0044] The Actor Policy Network takes local observations as input, captures the temporal characteristics of intersections, and outputs action probabilities and super-actions; super-actions are k-dimensional probability distributions that are different for each intersection; it inherits the advantages of parameter sharing, ensures training efficiency while considering the personalization of intersections, improves performance and reduces training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 This is a comparison result of the convergence speed of each model based on the data set in the embodiment of the present invention;

[0047] Figure 2 The component ablation result based on the Jinan data set in an embodiment of the present invention;

[0048] Figure 3 The performance comparison results of five reinforcement learning methods based on a 4×4 synthetic data set at four parameter scales in an embodiment of the present invention are shown;

[0049] Figure 4 This is a framework diagram of an embodiment of the present invention;

[0050] Figure 5 This is a diagram of the model training process of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] The present invention provides a personalized multi-agent reinforcement learning traffic control method based on a super-strategy network, which is conceived as Hyper-Action Multi-Agent Proximal Policy Optimization (HAMA-PPO). Based on PPO, the CTDE paradigm is adopted to train agents to learn coordination strategies; a shared strategy network equipped with super-actions and multiple value estimates is used to provide personalized strategies for multiple intersections with non-independent and identically distributed (non-iid) observations, thereby providing a flexible trade-off between data efficiency and strategy performance.

[0054] Specifically, the actor policy network outputs action distribution, while the other part outputs super actions through the super policy network to achieve diversity in value function estimation. At the same time, the critic evaluation network uses global data to train a set of value functions, providing different value function estimates for each intersection to achieve joint value estimation, thereby guiding the learning of personalized strategies. Its effectiveness is verified on real and synthetic road network traffic.

[0055] The actual process is as follows: In a traffic signal control system, N local agents with shared parameters make decisions based on their own independent actors. They take local observation history as input, extract the temporal characteristics of intersections through GRU, and output action distribution and super-action. The centralized critic network is shared by all agents and estimates the reward of each intersection by accessing global information. Considering that the estimate of a single value function may be inaccurate, the critic outputs a set of value functions, which are dot-producted with the super-actions output by the local actor to obtain a joint value estimate. Parameters are updated by comparing the differences in individual true rewards. Finally, in order to capture the spatiotemporal characteristics of the observations of each intersection in the traffic network, the critic adopts a graph attention network (GAT) to capture the dynamic influence of surrounding intersections.

[0056] Combination Figure 4 As shown in the module structure diagram, the present invention includes an actor decision network and a critic value network, which are as follows:

[0057] Shared actor network with super policy network, specifically:

[0058] In the traffic signal control problem, historical observations are very important. Therefore, GRU is used to process long sequence data to capture the temporal dependencies of intersections.

[0059] Local observation of the agent Input actor network; use multi-layer perceptron to transform k-dimensional raw observations Embedded into the m-dimensional latent space to obtain observation embedding; output hidden features through the GRU unit Hidden features are passed through a linear layer to generate action probability distribution and super action Action probability distribution selects an action through probability sampling There are eight combinations of action phases, obtained through probability sampling The signal phase of the traffic simulator at the next moment is set to Continue for ten seconds, then calculate the next move.

[0060] In this implementation, the clip function is used to limit the difference before and after the strategy update. Therefore, the loss function of the strategy network output action probability distribution in this embodiment is:

[0061]

[0062] in, is importance sampling, π θ is a parameterized policy network, is the advantage function, is the action of intersection i at time t, ε is a hyperparameter, γ represents the discount factor, is the cycle characteristic of the previous moment.

[0063] Compared with a single value function, training a set of value functions can match different value functions for different intersections. This enables generating different value estimates for intersections to guide personalized learning of strategies; selecting one from a limited set of value functions as an estimate of the current state is not the most accurate. Therefore, in this implementation, several value functions are allowed to jointly guide the update of the strategy network. Specifically:

[0064] Adopt the idea of ​​hypernetwork; take hidden features and intersection index as input, generate unique weights for each value function, i.e. hyperaction In this embodiment, As a probability distribution of value functions, rather than a multi-dimensional action. As a probability distribution of value functions, we can obtain an infinite number of combinations of value functions, freeing the solution from constraints. The specific formula is as follows:

[0065]

[0066] Among them, W e and b e are the weight matrices and bias vectors, is the cyclic feature of intersection i at time t-1 output by the GRU unit, and ⊕ is the concatenation operation.

[0067] This model uses a centralized training method to train the actor network, that is, the local observations of all agents are integrated through parameter sharing to jointly train the actor network.

[0068] In order to prevent the problem that some value functions are updated slowly due to individual weights being too small, this implementation adds an entropy regularization term to the hyper-action distribution The hyperaction is used to estimate the importance of each estimated value. In order to effectively update the value function of each dimension, an entropy regularization term is introduced to narrow the gap between hyperactions, make the update of the value function more uniform, and enable all value functions to be effectively learned. represents the parameterized by θ

[0069] The loss function of the actor network after training is updated to:

[0070]

[0071] The personalized critic network trained centrally is as follows:

[0072] The centrally trained critic network requires global information. In this implementation, the global information is composed of the local observations of all agents and the adjacency between graph nodes. The local observation information of all agents is obtained by the embedding layer. The actor and critic share an observation embedding layer, which not only reduces the number of parameters in the network, but also promotes better information sharing and complementarity between them, improving the generalization ability and performance of the entire system. The graph attention network (GAT) is used to process the structural information of the graph and extract the spatial features between the intersections.

[0073] The specific process is: all the observations of the agents are passed through the embedding layer to obtain the embedding features of each intersection The embedded features integrate neighbor information according to importance through two layers of GAT, and output multiple value functions through the linear layer.

[0074] As the difference in traffic conditions at intersections increases, the weight The difference between them will also increase, that is, the probability distribution of the value function will change, resulting in a proportional change in the value function in the strategy update, thereby achieving personalized learning of the strategy.

[0075] Specifically: Each intersection has its value function vector, which represents multiple reward estimates for the current state.

[0076] express The probability distribution of the data items in .

[0077] The multi-valued function and the super-action have the same dimension k, and the final joint value function is generated through the dot product, providing precise and comprehensive guidance for optimizing the policy network. This set of value functions acts as an expert, allowing these experts to learn in parallel and jointly guide the update of the policy network.

[0078] The joint value estimate is expressed as:

[0079]

[0080] Among them, O t is the global observation, G is the graph network structure of the intersection, is the cycle characteristic of the previous moment, i is the intersection subscript, For multidimensional valued function estimation, For super action, is the local observation of intersection i.

[0081] The loss function of the Critic network is expressed as:

[0082]

[0083] in, represents the value network parameters, represents the joint value function, N represents the number of intersections, T represents the time step, represents the reward of intersection i at time t, and γ represents the discount factor.

[0084] The specific model algorithm of this implementation is:

[0085]

[0086] The effect of this method is described with specific application examples:

[0087] In this embodiment, a microscopic traffic simulation simulator CityFlow is used for experiments, which can simulate the driving behavior of vehicles on urban roads in real time and support flexible definition of road networks and traffic flows based on synthetic data and real data.

[0088] In this embodiment, four sets of data, including two real traffic road networks (Jinan 3×4 and New York 16×3) and two synthetic road networks (4×4 and 10×10), are used to evaluate the performance of the model.

[0089] The traffic data distribution statistics of the four data sets are shown in Table 1.

[0090] Table 1

[0091]

[0092] The performance improvement of this model is demonstrated through three groups of control experiments, specifically:

[0093] Experiment 1: Comparative experiment to evaluate the convergence speed of the model;

[0094] HAMA-PPO is compared with four state-of-the-art reinforcement learning methods (Colight, PNC-HDQN, MPLight, and MA2C).

[0095] like Figure 1 The convergence speed of each reinforcement learning method (PNC-HDQN, MA2C, MPLight, Colight and HAMA-PPO) based on different datasets during training, where

[0096] a. Convergence speed comparison results of various models based on Jinan dataset;

[0097] b. Convergence speed comparison results of each model based on the New York dataset;

[0098] c. Convergence speed comparison results of each model based on 4×4 synthetic dataset;

[0099] d. Convergence speed comparison results of each model based on 10×10 synthetic dataset.

[0100] The following conclusions can be drawn: 1) MA2C and PNC-HDQN train one agent for each intersection, and the performance of the two methods is very similar. This completely heterogeneous algorithm takes a long time to converge, and has not converged after 200 episodes;

[0101] 2) Colight,shows a faster convergence speed in the early stage of training, and can effectively utilize the information of the,traffic network to quickly find a more reasonable strategy, but it cannot learn the,optimal strategy.

[0102] 3) HAMA-PPO has better performance in both convergence speed and final performance. Therefore, this method can learn a better strategy method with excellent convergence speed.

[0103] Experiment 2: Ablation experiment, to evaluate the effectiveness of the module;

[0104] The hyperaction proposed in this implementation is used to capture the preference of the intersection, and the entropy is to make the update of the value function more uniform, so that all value functions can be effectively learned. In order to verify the effectiveness of hyperaction and weighted entropy, an ablation experiment was conducted, and the agents of all methods shared parameters. The model of the embodiment of the present invention is PPO+hyper+Entrony, and the control group PPO+hyper only includes the hypernetwork, and PPO removes both. The results are shown in Figure 2. Figure 2 As shown in (Jinan dataset), the performance of adding hypernetwork and weight entropy is the best, and the convergence speed is faster than PPO+hyper.

[0105] Experiment 3: Ablation experiment to evaluate the scale of neural network parameters;

[0106] The model in this embodiment combines the advantages of shared parameter and non-shared parameter algorithms. Parameter sharing can improve the efficiency of model training, but it cannot take into account the characteristics of each intersection. Non-parameter sharing trains a set of parameters for each agent, but when there are many intersections in the environment, the parameter scale will be very large.

[0107] like Figure 3 The performance of five reinforcement learning methods at four parameter scales (4×4 synthetic data set) shows that the model of the present invention has the best effect among the five models, specifically:

[0108] This model retains the advantage of parameter sharing and can achieve good results even with fewer parameters. When the parameter scale is small, all algorithms cannot learn the optimal strategy because the neural network cannot fully characterize the characteristics of the intersection. As the number of neural network parameters increases, the performance of the shared parameter algorithms (HAMA-PPO, Colight, MPLight) shows a trend of first improving and then stabilizing. Even if the number of parameters is large, the performance of HAMA-PPO will decrease, which is a problem of overfitting. Increasing the weight entropy coefficient can appropriately alleviate the overfitting of the neural network.

[0109] Experimental results show that the algorithm effectively improves the performance of large-scale road networks and reduces the cost of decision-making and deployment. Therefore, the innovation of the algorithm is meaningful.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A personalized multi-agent reinforcement learning traffic control method based on a super-strategy network, characterized in that: include: Constructing a reinforcement learning traffic signal control model; the model includes: a strategy network, a value network, and a super strategy network; The policy network outputs an action distribution, selects actions through probability sampling, and controls traffic signals; The value network outputs a value function; the super-strategy network outputs a super-action; The joint value function is obtained by dot product of the super action and the value function; the value network evaluates the strategy network through the joint value function; The reinforcement learning traffic signal control model is trained by a loss function; The training model process includes: Update the value network by the mean square error; Update the policy network by limiting the range of changes between the old and new policies through gradient clipping.

2. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The policy network outputs an action distribution, selects actions through probability sampling, and controls traffic signals, including: Enter local observation history Using observation history to obtain embedded features through multi-layer perceptron The temporal features of the intersection are extracted through the GRU unit, and the hidden features are output The hidden features generate action probability distribution through a linear layer; The action is selected by sampling the action probability to control the traffic signal.

3. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 2 is characterized in that: The super-strategy network outputs a super-action, including: hidden features generate super-actions through linear layers Among them, W e is the weight matrix, b e is the bias vector, is the cyclic hidden feature of intersection i at time t output by the GRU unit, It is a splicing operation.

4. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The value network outputs a value function, including: The embedding features of each intersection are obtained through the embedding layer The embedding features are transformed into Integrate with its neighbors information according to importance; The integrated embedded features are passed through a linear layer to estimate the multi-dimensional value output for each intersection model.

5. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The policy network and the value network share an embedding layer.

6. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The step of obtaining a joint value estimate by using a dot product of a super action and a value function includes: Using Super Action express Probability distribution of the data items in , and calculate joint value estimates for each intersection; The multi-valued function and the hyper-action have the same dimension k, and the joint value estimate is generated through the dot product: Among them, O t is the global observation, G is the graph network structure of the intersection, is the cycle characteristic of the previous moment, i is the intersection subscript, For multidimensional valued function estimation, For super action, is the local observation of intersection i.

7. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The value network is updated by the mean square error: in, represents the value network parameters, represents the joint value function, N represents the number of intersections, T represents the time step, represents the reward of intersection i at time t, and γ represents the discount factor.

8. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The value network evaluates the policy network through a joint value function, including introducing an entropy regularization term for the super-action distribution, expressed as in represents the parameterized by θ 9. The personalized multi-agent reinforcement learning traffic control method based on super strategy network according to claim 1 is characterized in that: The updating of the strategy network by limiting the variation range of the new and old strategies through gradient clipping includes: in, is importance sampling, π θ is a parameterized policy network, is the advantage function, is the action of intersection i at time t, ε is a hyperparameter, and λ is the regularization coefficient.

Images