A method and system for controlling multiple traffic light signals based on multi-agent action-dependent reinforcement learning
By constructing an action dependency graph and an action dependency policy neural network, the problem of unmodeled action dependencies in multi-traffic light signal control is solved, achieving collaborative optimization of traffic light signal control and improving the overall traffic efficiency and consistency of the traffic network.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-02
AI Technical Summary
Existing multi-traffic light signal control methods fail to effectively model the action dependencies between traffic lights, leading to independent decision-making conflicts and affecting overall traffic operation efficiency.
By constructing an action-dependent graph and an action-dependent policy neural network, the signal decision-making dependencies between traffic lights are clarified, and a multi-agent reinforcement learning algorithm is used to optimize the control strategy, ensuring that the traffic lights are collaboratively optimized in the orderly decision-making process.
It effectively avoids conflicts between traffic lights, improves the traffic efficiency and signal control consistency of the road network, reduces fluctuations in vehicle queue length, and enhances the operational performance of complex road networks.
Smart Images
Figure CN122135580A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of intelligent traffic control and artificial intelligence, and in particular provides a method and system for controlling multiple traffic light signals based on multi-agent action-dependent reinforcement learning. Background Technology
[0002] With the continuous growth of urban motor vehicle ownership, traffic congestion has become increasingly prominent. Traffic signal control, as an important means of urban road traffic management, directly affects road traffic efficiency, vehicle delay time, and energy consumption levels through its control strategies. Existing traffic light control methods mainly include timing control, inductive control, and adaptive control methods based on optimization or learning.
[0003] Traditional timing control methods typically rely on pre-setting signal timing schemes based on historical traffic flow data, making it difficult to cope with the time-varying and random nature of traffic flow and limiting their effectiveness in complex road networks. While inductive control methods can adjust signals to some extent based on real-time detected traffic conditions, their decision-making logic largely depends on manually configured rules, lacking the ability to comprehensively model the global traffic situation and hindering collaborative optimization among multiple intersections. In multi-traffic light signal control scenarios, the signal decisions of each traffic light are coupled in time and space; the signal selection at one intersection often directly affects the traffic conditions and decision results of adjacent intersections. Existing multi-traffic light reinforcement learning signal control technologies often employ independent decision-making for each traffic light or synchronous decision-making based on the global state, failing to explicitly model the decision-making sequence and action dependencies between traffic lights. This can easily lead to multiple traffic lights making conflicting or uncoordinated control decisions at the same time, thus affecting the overall traffic operation efficiency.
[0004] In recent years, reinforcement learning technology has been gradually introduced into the field of traffic signal control. By modeling single or multiple traffic lights as intelligent agents, they autonomously learn signal control strategies using interactive data, thus improving the adaptability of traffic control to some extent. However, existing reinforcement learning-based multi-traffic light control methods still treat multiple traffic lights as independent intelligent agents, learning control strategies separately, ignoring the mutual influence between adjacent or related intersections in signal decisions. This can easily lead to suboptimal strategies or even traffic conflicts (CN115631638A, CN113223305A). Although some methods (CN116137103A, CN118247977A) have introduced graph structure features and combined them with deep reinforcement learning to achieve multi-traffic light signal control, these schemes focus on graph representation and feature extraction of traffic state information, still mainly relying on parallel decision-making by multiple agents. They only focus on the sharing of traffic state information and rarely characterize the dependencies between the decision actions of different traffic lights, thus limiting the expressive power and control performance of the strategy.
[0005] Therefore, there is an urgent need for a traffic light signal control method that can effectively model the action dependencies between multiple traffic lights while ensuring scalability, and achieve collaborative optimization through multi-agent reinforcement learning, so as to improve the overall operating efficiency under complex traffic networks. Summary of the Invention
[0006] To address the aforementioned problems, this invention provides a multi-traffic light signal control method and system based on multi-agent action-dependent reinforcement learning. This transforms the original independent or synchronous multi-traffic light decision-making process into an ordered decision-making process with clear dependency constraints, thereby avoiding conflicts caused by independent decision-making of multiple traffic lights from the perspective of control mechanism and improving the overall traffic efficiency of the road network.
[0007] To achieve the above objectives, in a first aspect, the present invention provides a multi-traffic light signal control method based on multi-agent action-dependent reinforcement learning, comprising the following steps:
[0008] S1. Construct a traffic network map for the traffic network. ,in, This represents the set of nodes formed by intersections. This represents the set of edges formed by the roads connecting the intersections.
[0009] S2. Based on the traffic network map Building an action dependency graph The action dependency graph is a directed acyclic graph, wherein, This represents the set of nodes formed by intersections. This represents the set of edges representing the signal dependencies between traffic lights; if This means that traffic light j needs to consider the current signal action of traffic light i when making signal decisions.
[0010] S3. Based on the action dependency graph Initialize the action-dependent policy neural network, which is used to jointly model traffic state information and action dependency information under the constraints of the action dependency graph, and output the control strategy of each traffic light.
[0011] S4. The action-dependent policy neural network is trained using a reinforcement learning algorithm to obtain policy parameters that satisfy the traffic cooperative control objective.
[0012] S5. During actual operation, each traffic light acquires traffic status information at the intersection where it is located.
[0013] S6. Using the trained action-dependent policy neural network, each traffic light generates its corresponding traffic signal control action sequentially according to the dependency order determined by the action dependency graph, based on the current traffic state information of the intersection and the current signal action of the traffic light it depends on. After all traffic lights have generated their control actions, they execute their actions together.
[0014] In a preferred embodiment, the action dependency graph in step S2 It is generated through a method for generating sparse graphs, including the following steps:
[0015] S2.1 Predetermine the decision order for traffic lights, and number the traffic lights as 1, 2, ..., n according to the decision order.
[0016] S2.2 Initialize the set of action-dependent edges It is an empty set.
[0017] S2.3 According to The order, for the i-th traffic light, determines the positions of nodes i to n in the traffic network graph. The set of all neighboring nodes (excluding i to n themselves) is denoted as . and the set This is the set of nodes that the i-th traffic light depends on for its actions; for any There will be a directed edge Add to the action dependency edge set middle.
[0018] The action-dependent policy neural network in step S3 includes a state encoding module, an action information encoding module, and a policy decision module.
[0019] The state encoding module is used to represent the traffic state information of each traffic light at the intersection; the action information encoding module is used to represent the current signal action of the traffic light that has a dependency relationship with the current traffic light in the action dependency graph; and the strategy decision module is used to fuse the state features and action features and output the control action or action probability distribution of the corresponding traffic light.
[0020] In a preferred embodiment, the state encoding module, action information encoding module, and strategy decision-making module can be implemented using a multilayer perceptron network structure. By methods including but not limited to concatenating or weighting state features and action features, joint modeling of traffic state information and action dependency information can be achieved.
[0021] In a preferred embodiment, step S4 employs a multi-agent reinforcement learning algorithm based on proximal policy optimization (MAPPO) to centrally train the action-dependent policy neural network. The training process includes the following steps:
[0022] S4.1 Initialize the parameters of the policy network (Actor) and the centralized value evaluation network (Critic) for each traffic light, and set the hyperparameters required for training, including the learning rate, discount factor, and policy update threshold.
[0023] S4.2 In each round of training, each traffic light acts as an independent agent, generating traffic signal control actions through its own Actor network based on the current traffic state information of the intersection and the current signal actions of the traffic lights it depends on in the action dependency graph, and applying the control actions to the traffic simulation environment.
[0024] The traffic simulation environment described in S4.3 updates the traffic status according to the control actions of each traffic light, and returns the corresponding instant reward and the traffic status information for the next moment to each traffic light. The instant reward is used to measure indicators such as traffic efficiency, vehicle queue length or delay time.
[0025] In the centralized training phase, S4.4 utilizes the Critic network to evaluate the global traffic state information and the joint actions of all traffic lights, and calculates the policy loss function for each traffic light based on the near-end policy optimization method; for the i-th traffic light, its policy loss function is defined as:
[0026]
[0027] in, This represents the policy network parameters for the i-th traffic light. This represents the estimated dominance function value for the corresponding traffic light at time $t$. Update the threshold for the preset strategy. The ratio of the probabilities of the new and old strategies is defined as:
[0028]
[0029] in, This represents the control action taken by the i-th traffic light at time t. This indicates the traffic status information at the corresponding intersection. Let represent the set of actions of the traffic light at time t that has a dependency relationship with the i-th traffic light in the action dependency graph.
[0030] The advantage function The calculation method is as follows, estimated by the Critic network:
[0031]
[0032] in, This represents the cumulative reward of the i-th traffic light at time t. Represents the Critic network. Its network parameters, the Critic network is based on global traffic state As input.
[0033] S4.5 Based on the policy loss function, the near-end policy optimization method is used to update the Actor network parameters of each traffic light to limit the update range between the old and new policies and improve the stability of the training process.
[0034] The Critic network described in S4.6 updates by minimizing the value function loss, which is defined as:
[0035]
[0036] S4.7 Repeat steps S4.2 to S4.6 until the preset convergence condition is met or the maximum number of training rounds is reached, thus completing the training of the action-dependent policy neural network.
[0037] After training is completed, only the Actor network corresponding to each traffic light is retained for online signal control decision-making. The Critic network does not participate in traffic signal control during the actual deployment phase.
[0038] To achieve the above objectives, in a second aspect, the present invention provides a multi-traffic light signal control system based on multi-agent action-dependent reinforcement learning, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-traffic light signal control method according to any one of claims 1 to 8; and / or, it includes a readable storage medium storing a computer program thereon, wherein the computer program, when executed by the processor, implements the multi-traffic light signal control method according to any one of claims 1 to 8.
[0039] Compared with existing technologies, the multi-traffic light signal control method and system based on multi-agent action-dependent reinforcement learning provided by this invention has at least the following technical advantages:
[0040] Firstly, by introducing an action dependency graph to explicitly model the decision-making relationships among multiple traffic lights, each traffic light follows dependency constraints and decision-making order during the signal decision-making process, avoiding multiple traffic lights making conflicting control actions at the same time, thereby improving the consistency and coordination of signal control at adjacent intersections.
[0041] Secondly, since traffic lights can comprehensively consider the signal actions of the intersections when making decisions, this invention can suppress the spread of traffic congestion in the road network while ensuring the operational efficiency of local intersections, effectively reducing the fluctuation range of vehicle queue length at intersections, thereby improving the overall operational performance of multi-intersection traffic systems.
[0042] Thirdly, under complex road network conditions, this invention transforms the coordinated control problem of multiple traffic lights into a sequential decision-making problem constrained by action dependence. Compared with decision-making methods based solely on local or global states, this is more conducive to reflecting the temporal influence relationship between multiple intersections, making the reinforcement learning training process more stable, and the obtained traffic signal control strategy exhibits better robustness in actual operation. Attached Figure Description
[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is a flowchart of a multi-traffic light signal control method based on multi-agent action-dependent reinforcement learning.
[0045] Figure 2 This is a schematic diagram of a 3x3 traffic network.
[0046] Figure 3 It is a schematic diagram of the traffic network showing the connection relationships.
[0047] Figure 4 It is an action dependency graph constructed based on the transportation network.
[0048] Figure 5 This is a schematic diagram of the agent policy network structure. Detailed Implementation
[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0050] This invention relates to the fields of intelligent traffic control and artificial intelligence, and in particular to a method for collaborative optimization control of traffic signals at multiple intersections by utilizing a multi-agent reinforcement learning algorithm and constructing action dependencies between traffic lights.
[0051] In multi-traffic light signal control scenarios, the signal decisions of each traffic light are coupled in time and space. The signal selection at one intersection often directly affects the traffic status and decision results of adjacent intersections. Existing multi-traffic light reinforcement learning signal control technologies mostly adopt independent decision-making for each traffic light or synchronous decision-making based on the global state. They do not explicitly model the decision-making sequence and action dependencies between traffic lights, which can easily lead to multiple traffic lights making conflicting or uncoordinated control decisions at the same time, thus affecting the overall traffic operation efficiency.
[0052] To address the aforementioned problems, this invention provides a multi-traffic light signal control method based on multi-agent action-dependent reinforcement learning. By constructing an action dependency graph, the dependency relationships and decision order of different traffic lights in the signal decision-making process are clarified. This allows traffic lights to consider not only the traffic state information of their own intersections but also the control actions of already decided traffic lights as conditional inputs for joint decision-making. This approach transforms the originally independent or synchronous multi-traffic light decision-making process into an ordered decision-making process with clear dependency constraints, avoiding conflicts arising from independent decision-making by multiple traffic lights from a control mechanism perspective, thereby improving the overall traffic efficiency of the road network.
[0053] like Figures 1 to 5 As shown, the present invention provides a multi-traffic light signal control method based on multi-agent action-dependent reinforcement learning, the steps of which include:
[0054] S1: As Figure 2 As shown in the example, this method is applied to a traffic network with a 3×3 (three horizontal and three vertical) structure, comprising nine intersections, each with a traffic light. The traffic lights are numbered sequentially from left to right and top to bottom as Traffic Light 0 to Traffic Light 8.
[0055] according to Figure 3 The road connections shown are used to construct a traffic network map. , where the set of nodes Represents each intersection, edge set It indicates the road connection relationship between intersections in a transportation network.
[0056] S2: In this embodiment, based on the traffic network map Construct an action dependency graph according to the aforementioned preferred implementation method. This is used to describe the action dependencies of traffic lights during the signal decision-making process. The decision-making order of the traffic lights is predetermined as traffic light 0 to traffic light 8, and action dependencies are constructed sequentially according to this order. Based on the traffic network topology, the action dependency sets corresponding to each traffic light can be obtained as follows: For any traffic light i, if Then add directed edges to the action dependency graph. This indicates that traffic light i needs to consider the current signal action of traffic light j when making signal decisions. Therefore, we get... Figure 3 The action dependency graph structure is shown.
[0057] S3: In this embodiment, each traffic light is considered as an intelligent agent. For the i-th traffic light, its traffic state at time t is... Defined as:
[0058]
[0059] in, This indicates the lane density of each approach lane at the intersection. This indicates the length of the queue for vehicles in each entrance lane. This indicates the current phase state of the traffic light.
[0060] In this embodiment, each intersection has two approach lanes in each direction, for a total of eight approach lanes. Lane density Defined as the ratio of the total number of vehicles in a lane to the lane length, corresponding to an 8-dimensional feature; vehicle queue length. This represents the total number of vehicles waiting for passage in each entrance lane, corresponding to an 8-dimensional feature. Traffic lights have a total of 4 signal phases; the current signal phase... It is represented using a 4-dimensional one-hot encoding method.
[0061] Traffic light control actions This indicates the signal selected by the traffic light at time t.
[0062] Construct an action-dependent policy neural network for the i-th traffic light, whose policy function is expressed as:
[0063]
[0064] in, Let be the parameters of the i-th traffic light policy network. Represents the action dependency set The set of actions of each traffic light at time t.
[0065] like Figure 4 As shown, the action-dependent policy neural network includes the following modules:
[0066] Status information encoding module: used for traffic status Feature extraction is performed to obtain the state feature vector. ;
[0067] Action information encoding module: used for actions that depend on traffic lights. Encode to obtain action feature vectors ;
[0068] Strategy Decision Module: Used to fuse state features and action features, and output the action probability distribution.
[0069] In this embodiment, state features and action features are fused by splicing:
[0070]
[0071] The data is then input into the strategy decision module to output the probability distribution of the traffic signal actions corresponding to the traffic light.
[0072] S4: In this embodiment, a multi-agent reinforcement learning algorithm based on proximal policy optimization is used to train the action-dependent policy neural network in a concentrated manner to optimize the policy parameters of multi-traffic light signal control.
[0073] During training, an instant reward function is defined for each traffic light agent to characterize the impact of the current signal control action on the traffic operation state.
[0074] In this embodiment, the instantaneous reward of the i-th traffic light at time t Defined as a negative sum of the queue lengths of all approach lanes at the intersection where the traffic light is located, i.e.:
[0075]
[0076] in, Let i represent the set of approach lanes at the i-th intersection. This represents the length of the queue of vehicles in lane l at time t.
[0077] Before training begins, initialize the hyperparameters required for multi-agent reinforcement learning training, including:
[0078] Discount factor, used to balance immediate rewards with long-term cumulative returns;
[0079] Learning rates of policy networks and value evaluation networks;
[0080] The policy update threshold is used to limit the magnitude of policy updates.
[0081] The time step length of each training round and the maximum number of training rounds;
[0082] Small batch sample size, used for random sampling during the parameter update phase.
[0083] In each training round, each traffic light agent generates control actions through its own policy network according to the decision order provided by its number, based on the current traffic state and the dependent actions specified in the action dependency graph, and applies the control actions to the traffic simulation environment.
[0084] After executing the control action, the traffic simulation environment returns the traffic state information and corresponding instant reward for the next moment. During training, the interaction data collected at each moment is stored in time series format, including traffic state, control action, instant reward, and traffic state for the next moment, for subsequent policy updates.
[0085] In this embodiment, the interaction data is stored in units of a complete simulation, and after a preset condition is met, random sampling following a uniform distribution is performed from the stored data to construct a training sample set for network updates.
[0086] During the intensive training phase, a centralized value assessment network is constructed. Its input includes global traffic state information and the joint actions of all traffic lights, which is used to estimate the state value function under the current policy. ,in, This represents the global traffic state at time t. These are the parameters for the value assessment network.
[0087] Based on the aforementioned value assessment results, calculate the estimated value of the advantage function of the i-th traffic light at time t:
[0088]
[0089] in, Let represent the cumulative reward of the i-th traffic light at time t.
[0090] Based on the estimated advantage function, the policy network parameters of each traffic light are updated using a near-end policy optimization method. For the i-th traffic light, its policy loss function is defined as:
[0091]
[0092] in, This represents the ratio of the probabilities of the new and old strategies.
[0093] Meanwhile, the parameters of the centralized valuation network are updated by minimizing the following value function loss:
[0094]
[0095] The process of interactive data acquisition, advantage function calculation, and network parameter update is repeated until the preset convergence condition is met or the maximum number of training rounds is reached, thus completing the training of the policy neural network. After training, only the policy networks corresponding to each traffic light are retained for actual traffic signal control decisions; the centralized value assessment network does not participate in the signal control process during the actual deployment phase.
[0096] S5: During actual operation, at time t, each traffic light collects traffic status information of the corresponding intersection through traffic detection devices installed at the intersection.
[0097] Specifically, traffic lights 0 through 8 obtain the traffic status of their respective intersections at time t. to The traffic status information includes lane density, vehicle queue length, and current signal phase for each entrance lane.
[0098] S6: At time t, according to the decision order determined by the numbering sequence of the traffic lights, each traffic light generates and executes traffic signal control actions in sequence.
[0099] Specifically, traffic light 0, as a dependency-free node, is primarily based on its own traffic state. Traffic signal control actions are generated through a corresponding action-dependent policy neural network. Subsequently, for traffic light numbered i, its own traffic status is obtained. Based on its action dependency set Further obtain the control actions that the traffic lights have generated at time t in the action dependency set. The dependent actions, along with the traffic state itself, are input into the corresponding action-dependent policy neural network to generate traffic signal control actions. Once all traffic lights have generated their control actions, they will execute their respective actions together.
[0100] This invention provides a multi-traffic light signal control method based on multi-agent action-dependency reinforcement learning. By constructing a traffic network map and further generating an action dependency graph, the method clarifies the action dependencies among multiple traffic lights during the signal decision-making process. This enables traffic lights to comprehensively utilize their own traffic state and the control action information of previously decided traffic lights within a predetermined decision-making sequence. Based on this, a policy neural network capable of simultaneously processing traffic state information and action dependency information is constructed, and reinforcement learning is used to optimize the traffic signal control strategy. By introducing an action dependency modeling mechanism, the method effectively reduces conflicts and inconsistencies caused by simultaneous decision-making by multiple traffic lights, making signal control at adjacent intersections more coordinated. This results in a stable reduction in vehicle queue length and improved overall coordination of traffic operation at multiple intersections under complex road network conditions.
[0101] In some embodiments, this invention provides a multi-traffic light signal control system based on multi-agent action-dependent reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned method. A computer-readable storage medium storing the computer program is also provided. When the computer program is executed by the processor, it implements the multi-traffic light signal control method. This invention provides a multi-traffic light signal control system based on multi-agent action-dependent reinforcement learning. By constructing an action dependency graph, it clarifies the dependencies and decision order of different traffic lights in the signal decision-making process. This allows traffic lights to consider not only the traffic state information of their own intersections but also the control actions of already decided traffic lights as conditional inputs for joint decision-making. In this way, the originally independent or synchronous multi-traffic light decision-making process is transformed into an ordered decision-making process with clear dependency constraints. From a control mechanism perspective, this avoids the conflict problems caused by independent decision-making by multiple traffic lights, thereby improving the overall traffic efficiency of the road network.
[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.
Claims
1. A method for controlling multiple traffic light signals based on multi-agent action-dependent reinforcement learning, characterized by the following steps: include: S1. Construct a traffic network map ;in, This represents the set of nodes formed by intersections. This represents the set of edges formed by roads between intersections; S2. Based on the aforementioned traffic network map Building an action dependency graph The action dependency graph is a directed acyclic graph used to describe the action dependencies of traffic lights during the signal decision-making process; wherein, if This means that traffic light i needs to consider the control action of traffic light j when making signal control decisions; S3. Based on the action dependency graph A neural network structure for action-dependent policies is designed for each traffic light. The action-dependent policy neural network is used to jointly model traffic state information and control actions dependent on traffic lights under action-dependent constraints, and output traffic signal control actions or action probability distributions. S4. The action-dependent policy neural network is trained using a reinforcement learning algorithm to obtain traffic signal control policy parameters; S5. During actual operation, each traffic light obtains traffic status information of the intersection where it is located; S6. According to the decision order determined by the action dependency graph, each traffic light generates and executes the corresponding traffic signal control action in sequence based on its own traffic state information and the control actions of the traffic lights it depends on in the action dependency graph.
2. The multi-traffic light signal control method according to claim 1, characterized in that, In step S2, an action dependency graph is constructed. include: Determine the decision order for traffic lights, and number each traffic light according to the decision order; For any traffic light, determine the set of traffic lights it depends on based on its adjacency relationship in the traffic network graph, and add directed edges to the action dependency graph accordingly.
3. The multi-traffic light signal control method according to claim 1, characterized in that, The action dependency graph It is generated through a method for generating sparse graphs, including the following steps: S2.1 Predetermine the decision order for traffic lights, and number the traffic lights as 1, 2, ..., n according to the decision order; S2.2 Initialize the set of action-dependent edges It is an empty set; S2.3 According to The order, for the i-th traffic light, determines the positions of nodes i to n in the traffic network graph. The set of all neighboring nodes (excluding i to n themselves) is denoted as . and the set This is the set of nodes that the i-th traffic light depends on for its actions; for any There will be a directed edge Add to the action dependency edge set middle.
4. The multi-traffic light signal control method according to claim 1, characterized in that, The policy function corresponding to the action-dependent policy neural network is expressed as follows: ,in, This represents the traffic status information of the i-th traffic light. This represents the set of control actions for the traffic light that has a dependency relationship with the i-th traffic light in the action dependency graph.
5. The multi-traffic light signal control method according to claim 1, characterized in that, The action-dependent policy neural network includes: The status information encoding module extracts features from traffic status information; The action information encoding module extracts features from control actions that rely on traffic lights; and The strategy decision-making module processes the fused state and action features and outputs traffic signal control actions.
6. The multi-traffic light signal control method according to claim 1, characterized in that, The traffic state information for each traffic light during the reinforcement learning process includes at least one of the following or a combination thereof: Lane density information for each approach lane at the intersection; Information on the length of vehicle queues at each approach lane of the intersection; as well as Current traffic light signal phase status information; The traffic state information is used as the state input for the action-dependent policy neural network.
7. The multi-traffic light signal control method according to claim 1, characterized in that, The control actions of each traffic light during the reinforcement learning process are represented as traffic signal control actions, which include at least one of the following: Traffic signal phase selection; and Adjustment of traffic signal phase duration; The control action is used as the state output or the output after random sampling of the action-dependent policy neural network.
8. The multi-traffic light signal control method according to claim 1, characterized in that, The MAPPO multi-agent reinforcement learning algorithm based on proximal policy optimization is used to train action-dependent policy neural networks. The training process includes the following steps: S4.1 Initialize the parameters of the policy network Actor and the centralized value evaluation network Critic for each traffic light, and set the hyperparameters required for training, including learning rate, discount factor and policy update threshold. S4.2 In each round of training, each traffic light acts as an independent intelligent agent. Based on the current traffic state information of the intersection and the current signal action of the traffic lights it depends on in the action dependency graph, it generates traffic signal control actions through its own Actor network and applies the control actions to the traffic simulation environment. The traffic simulation environment described in S4.3 updates the traffic status according to the control actions of each traffic light, and returns the corresponding instant reward and the traffic status information at the next moment to each traffic light. The instant reward is used to measure indicators such as traffic efficiency, vehicle queue length or delay time. During the intensive training phase, S4.4 utilizes the Critic network to evaluate the global traffic state information and the joint actions of all traffic lights, and calculates the policy loss function corresponding to each traffic light based on the near-end policy optimization method. S4.5 Based on the policy loss function, the near-end policy optimization method is used to update the Actor network parameters of each traffic light to limit the update range between the old and new policies and improve the stability of the training process. The Critic network described in S4.6 updates by minimizing the value function loss, which is defined as: ； S4.7 Repeat steps S4.2 to S4.6 until the preset convergence condition is met or the maximum number of training rounds is reached, thus completing the training of the action-dependent policy neural network.
9. The multi-traffic light signal control method according to claim 8, characterized in that, For the i-th traffic light, its policy loss function is defined as: ； in, This represents the policy network parameters for the i-th traffic light. This represents the estimated dominance function value for the corresponding traffic light at time $t$. Update the threshold for the preset strategy; The ratio of the probabilities of the new and old strategies is defined as: ； in, This represents the control action taken by the i-th traffic light at time t. This indicates the traffic status information at the corresponding intersection. Let represent the set of actions of the traffic light that has a dependency relationship with the i-th traffic light at time t in the action dependency graph; The advantage function The calculation method is as follows, estimated by the Critic network: ； in, This represents the cumulative reward of the i-th traffic light at time t. Represents the Critic network. Its network parameters, the Critic network is based on global traffic state As input.
10. A multi-traffic light signal control system based on multi-agent action-dependent reinforcement learning, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-traffic light signal control method according to any one of claims 1 to 8; and / or includes a readable storage medium storing a computer program thereon, wherein the computer program, when executed by the processor, implements the multi-traffic light signal control method according to any one of claims 1 to 8.