A Multi-Agent Traffic Signal Control Method Based on Autoregressive Strategy

Abstract

This invention discloses a multi-agent traffic signal control method based on an autoregressive strategy, belonging to the field of intelligent traffic control technology. Addressing the shortcomings of existing multi-agent reinforcement learning methods, such as insufficient collaborative mechanisms, difficulty in characterizing decision dependencies between agents, and unstable training processes, this invention introduces an autoregressive strategy structure. This structure decomposes the joint multi-agent strategy into conditional probabilities arranged according to decision order, enabling explicit modeling of decision dependencies between agents. An advantage function is constructed, and an Actor-Critic framework is used for joint updating of the policy and value functions. A stability constraint term based on KL divergence is introduced to limit the policy update magnitude, thereby improving training stability and convergence performance in non-stationary multi-agent environments. This invention can achieve efficient and stable signal cooperative control in large-scale traffic networks, significantly improving traffic efficiency and reducing average queue length, and possesses good scalability and engineering application value.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic control technology, specifically relating to a multi-agent traffic signal control method and system based on an autoregressive strategy. Background Technology

[0002] Traffic signal control systems, as one of the core means of urban traffic management, directly determine the efficiency of road network traffic due to the quality of their control strategies. How to achieve efficient and coordinated control of traffic lights at numerous intersections in a large-scale urban road network has become a key technical challenge that urgently needs to be solved in the field of intelligent transportation. In recent years, deep reinforcement learning technology has demonstrated powerful capabilities in sequential decision-making problems and has been widely studied and applied in the field of traffic signal control.

[0003] The prior art solutions related to this invention include: CN115472023B A method and device for controlling intelligent traffic lights based on deep reinforcement learning; CN115762199A A method for controlling traffic lights based on deep reinforcement learning and inverse reinforcement learning; CN116137103B A method for controlling large-scale traffic light signals based on primitive learning and deep reinforcement learning; CN118553103A A method for dissipating traffic congestion based on multi-agent reinforcement learning.

[0004] While the aforementioned technical solutions have promoted the development of intelligent traffic signal control technology to varying degrees, they still have the following shortcomings in practical applications: Existing technologies mostly adopt single-agent or weakly cooperative multi-agent modeling methods based on deep reinforcement learning. For example, in CN115472023B, although the agents at each intersection introduce neighbor information, they still rely on independent Q-networks for local decision-making, lacking structured modeling of the decision-making dependencies among multiple agents. While methods such as CN116137103B aggregate neighbor information through graph structures, their collaborative mechanisms remain at the information level rather than the decision level, failing to effectively model the decision-making order and action dependencies among agents, resulting in limited collaborative effects and difficulty in achieving global optimal control. At the same time, due to the high dynamism and randomness of the traffic environment, multi-agent scenarios further exacerbate environmental non-stationarity. For example, although CN115762199A introduces inverse reinforcement learning to improve the rationality of reward design, it is highly dependent on expert data and easily introduces additional instability in complex multi-agent environments, making existing methods generally suffer from unstable training and convergence difficulties, making it difficult to promote and apply them in large-scale traffic networks.

[0005] Existing technologies generally suffer from two major challenges: poor collaborative effects due to insufficient collaborative mechanisms, and unstable training due to environmental non-stationarity. How to perform refined modeling of decision-making dependencies among multiple agents, achieving a leap from "information collaboration" to "decision collaboration," and improving global collaborative control effectiveness while ensuring training stability, is a technical problem that urgently needs to be solved by those skilled in the art.

[0006] The key technical points of this invention are: by deeply analyzing the cooperative relationship between multiple agents, it classifies them into implicit cooperation and explicit cooperation, and constructs a unified multi-agent cooperative decision-making model; on this basis, it innovatively introduces an autoregressive policy structure, decomposing the joint policy into a series of conditional probabilities to explicitly model the sequential decision-making dependencies between agents; at the same time, it uses a local value function based on local observation and preceding actions to replace the global value function, so as to reduce computational complexity, improve the scalability of the system and the training stability in non-stationary environments. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a multi-agent traffic signal control method based on an autoregressive strategy. This method can explicitly model the sequential decision-making dependencies among multiple agents and improve the global collaborative control effect while ensuring training stability. It is applicable to the collaborative signal control of large-scale urban traffic networks.

[0008] The first technical solution adopted in this invention is: a multi-agent traffic signal control method based on an autoregressive strategy, specifically including the following steps:

[0009] S1. Construct a traffic signal control problem model, treating each intersection in the traffic network as an agent, and the set of all agents as a multi-agent system; at the discrete control time t, the system state is denoted as... The local observation of the i-th agent is denoted as i takes values ​​from 1 to N, where N is the number of agents; joint action Recorded as:

[0010] S2. Based on the spatial relationship of each intersection in the traffic network, the direction of road connection, the priority of the main road or the real-time traffic pressure, determine the decision order of each agent at time t, and decompose the joint strategy into an autoregressive strategy so that the actions of subsequent agents depend on the action information of the preceding agents.

[0011] S3. Based on each agent's local observations, neighborhood traffic state characteristics, and previous agent action information, construct the conditional local action value function and conditional local state value function for each agent.

[0012] S4. Calculate the advantage function of each agent based on the conditional local action value function and the conditional local state value function, and update the policy parameters and value function parameters of each agent using reinforcement learning methods.

[0013] S5. Add a stability constraint term during the policy update process to limit the change range between two adjacent policy updates, so as to ensure monotonic improvement and convergence stability during training.

[0014] S6. When the preset convergence condition is met, training stops and the trained strategy is deployed to each traffic intersection controller. Each agent outputs traffic signal control actions in sequence according to the decision order based on real-time observation, thereby realizing coordinated traffic signal control.

[0015] Furthermore, the local observation includes information reflecting the current traffic operation status of the intersection, including traffic flow status information, signal control status information, traffic operation efficiency information, and traffic association information of adjacent intersections; the agent's action is to adjust the traffic signal control strategy, including signal phase selection, phase duration adjustment, and multi-phase timing allocation.

[0016] Furthermore, the decision-making order of the agents is determined based on the traffic network topology or real-time traffic operation status, and the joint strategy is represented in an autoregressive form:

[0017] ,

[0018] in, This represents the ID of the k-th agent that makes a decision after sorting at time t. Indicates the joint policy parameters, This represents the policy parameters of the i-th agent. The conditional input for the i-th agent includes local observations, neighborhood traffic features, and action information of the preceding agent.

[0019] By employing the aforementioned autoregressive strategy, the joint decision-making process is transformed into a sequential conditional decision-making process, thereby explicitly modeling the decision dependencies among multiple agents.

[0020] Furthermore, the decision-making order can be a fixed order or a dynamic order; when a dynamic order is used, the intersections are ranked according to a traffic pressure index, which is defined as:

[0021] ,

[0022] in, This represents the queue length statistics at the i-th intersection. This represents the statistical measure of waiting delay. This represents the lane occupancy rate statistic. This represents a statistic indicating the arrival rate or saturation of traffic flow. These are non-negative weighting coefficients;

[0023] Furthermore, in step S3, the conditional local action value function and conditional local state value function of the i-th agent are constructed as follows:

[0024] ,

[0025] ,

[0026] in, Represents the parameters of the value network. As a discount factor, Indicates the time when the i-th agent is at time... The obtained local reward. The inputs to the conditional local action value function and the conditional local state value function do not use the global joint state, but instead use local observations, neighborhood context features, and previous agent action information related to the current agent, thereby reducing the input dimensionality and computational complexity.

[0027] Furthermore, the local reward function of the i-th agent in step S3 is defined as:

[0028] ,

[0029] in, Let represent the comprehensive traffic operation cost function of the i-th intersection at time t, which is used to characterize the traffic operation status of the intersection.

[0030] Furthermore, the advantage function of the i-th agent in step S4 is defined as:

[0031] ,

[0032] The objective function for updating the policy parameters in step S4 is:

[0033] ,

[0034] The loss function for updating the value network parameters is:

[0035] ,

[0036] in, For batch sample size, Accumulate returns towards the target.

[0037] Furthermore, the stability constraint term in step S5 is represented by the KL divergence between the policy distributions before and after the update, satisfying:

[0038] ,

[0039] in, This represents the policy parameters before the i-th agent updates. This indicates the corresponding update threshold;

[0040] The convergence constraint term in step S5 is characterized by the relationship between performance difference and policy gradient norm, satisfying:

[0041] ,

[0042] in, Indicates the performance of the optimal strategy. Indicates the current policy performance. The value is a positive constant; the KL divergence constraint and convergence constraint limit the policy parameter update amplitude, thereby improving the training stability in a multi-agent non-stationary environment.

[0043] Furthermore, the convergence criterion in step S5 includes at least one of the following:

[0044] (1) The change in strategy parameters between two consecutive iterations satisfies

[0045] ;

[0046] (2) The change in the value function loss between two adjacent iterations satisfies

[0047] ;

[0048] (3) The cumulative change in return between two adjacent iterations satisfies

[0049] ;

[0050] (4) The sum of the policy gradient norms of all agents satisfies

[0051] ;

[0052] in, The preset threshold;

[0053] When any of the aforementioned convergence criteria is met, the training ends and the trained traffic signal control strategy is output.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The multi-agent joint strategy is decomposed into an autoregressive conditional probability product, which explicitly models the sequential decision dependency relationship between agents. This upgrades the "information-level collaboration" in traditional methods to "strategy-level collaboration", effectively solving the problems of limited collaboration effect and easy trapping in local optima in existing technologies.

[0056] (2) By adopting a local value function based on local observation and preceding action information, the input dimension and fitting difficulty of the value network are significantly reduced, and the scalability of the algorithm in large-scale transportation networks is improved.

[0057] (3) Introducing stable constraints such as KL divergence constraints during the policy update process effectively suppresses the training oscillation problem caused by environmental non-stationarity in a multi-agent environment, ensuring the convergence stability and monotonic improvement of the algorithm. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall process of a multi-agent traffic signal control method based on an autoregressive strategy according to the present invention.

[0059] Figure 2 This is a schematic diagram of the traffic network topology and multi-agent system constructed in an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of the autoregressive strategy decomposition and decision order mechanism in an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the training framework based on the Actor-Critic architecture used in this embodiment of the invention.

[0062] Figure 5 This is a comparison chart of the average queue length convergence curves in a simulation environment for the present invention. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of protection of the invention.

[0064] like Figure 1 As shown, this embodiment provides a multi-agent traffic signal control method based on an autoregressive strategy, aiming to solve the problems of poor signal coordination and unstable training at multiple intersections in existing technologies. The method includes the following steps:

[0065] S1. Construct a model for the traffic signal control problem.

[0066] This step involves mathematically abstracting and formalizing the practical traffic control problem. First, a real urban traffic network (such as...) is considered. Figure 2 Modeling is performed as shown in the figure. Figure 2The illustrated typical regional road network includes a main road and several intersecting branch roads, totaling 9 intersections. Each of these 9 intersections is considered an independent agent (Agent 1 to Agent 9), and their collection constitutes a multi-agent system. This system operates at discrete time steps (i.e., control moments, such as every 10 or 15 seconds) t.

[0067] At each control time t, the system possesses a global state. It contains all information about the position, speed, and traffic light status of all vehicles in the road network. However, in a real system, it is impossible for any intersection controller to obtain the global state. Therefore, a local observation is defined for each agent i. This local observation is an abstraction of the actual sensor data that can be obtained at this intersection. Specifically, It may include one or more of the following types of information:

[0068] (1) Traffic flow status information: the number of vehicles queuing in each lane of the intersection, queue length, average vehicle speed, etc. This data can be obtained through roadside sensors such as geomagnetic coils, cameras or radar.

[0069] (2) Signal control status information: the current signal phase being executed, the green light duration of the current phase, etc.

[0070] (3) Traffic operation efficiency information: the number of vehicles passing through the intersection in the past period, the average waiting time of vehicles, or the number of stops, etc.

[0071] (4) Traffic information related to adjacent intersections: estimated number of vehicles traveling from upstream adjacent intersections to this intersection, and current congestion level of downstream adjacent intersections.

[0072] After acquiring local observations, each agent i needs to output an action. This involves adjusting the traffic lights it controls. This isn't simply a matter of "changing the lights" or "keeping them still," but rather an adjustment to the traffic signal control strategy, which can specifically include:

[0073] (1) Selection of signal phase: to determine which direction of vehicles can pass in the next time period, one can select from a preset set of phases {east-west straight, east-west left turn, north-south straight, north-south left turn}.

[0074] (2) Adjustment of signal phase duration: After selecting a phase, determine how long the phase will last, for example, a value between 10 seconds and 60 seconds.

[0075] (3) Multi-phase timing allocation: determines the execution order and duration allocation of each phase within a complete signal cycle.

[0076] The actions performed by all agents at the same time t are combined to form a joint action. This step, through the above definitions, clarifies the agent's local observations and actions, laying the foundation for subsequent collaborative decision-making.

[0077] S2. Determine the decision order and construct an autoregressive strategy.

[0078] This step is one of the core innovations of this invention, aiming to solve the problem of how to coordinate. Traditional methods require all agents to make decisions simultaneously, leading to difficulties in coordination. This invention introduces a sequential decision-making mechanism and defines how to determine the decision-making order.

[0079] First, determine the decision-making order at each time t based on the actual situation of the road network. This order can be a fixed order or a dynamic order.

[0080] (1) Fixed sequence: Suitable for scenarios with a clear road network structure and distinct primary and secondary elements. For example, in Figure 2 In this system, the decision-making order can be defined as follows: first, the intersections on the main road (agents 1, 2, and 3) make decisions from upstream to downstream, and then the intersections on the side roads (agents 4-9) make decisions. This order conforms to the natural laws of traffic flow, that is, traffic on the main road has priority.

[0081] (2) Dynamic Sequence: Suitable for scenarios with complex traffic flow changes. Based on the traffic pressure index of each intersection at time t... Traffic pressure is ranked from highest to lowest, with intersections experiencing the greatest traffic pressure given priority. (Traffic pressure index) The calculation formula is:

[0082] ,

[0083] in, (Queue length) (Delay time) (Lane occupancy rate) (Traffic saturation) is a statistical measure that reflects the degree of congestion at an intersection from different dimensions. These are non-negative weighting coefficients.

[0084] After determining the decision order, the joint policy of all agents is decomposed into an autoregressive policy form, such as... Figure 3 As shown. Its mathematical expression is:

[0085] ,

[0086] Specifically, the entire system should take coordinated actions. The probability, in order The conditional probability of the first agent taking its action is multiplied by the conditional probability of the second agent taking its action given the first agent's action, and so on, until the last agent. This is analogous to a chain of decision-making processes, rather than individual actions.

[0087] To enable this chain of decisions, a special conditional input was designed for each agent. .

[0088] in, It is its own local observation; It is the traffic context features it draws from neighboring intersections; most importantly, It represents the set of actions of all agents preceding the i-th agent in the current decision-making order.

[0089] Will As input, it ensures that subsequent agents can clearly understand the intentions of preceding agents when making decisions. For example, when making decisions, the agent at the side intersection already knows whether the upstream main road intersection has chosen to allow a large flow of traffic, and can thus proactively choose to cooperate in allowing or controlling traffic flow.

[0090] S3. Construct a conditional local value function.

[0091] In reinforcement learning, a value function is used to evaluate how "good" it is to take a certain action in a given state. In this step, a conditional local value function is constructed for each agent to evaluate its performance when receiving conditional input. Then, take action. The "good" or "bad".

[0092] ,

[0093] ,

[0094] Among them, local rewards Defined as the negative value of the overall traffic operation cost at this intersection, i.e. .For example, This represents the weighted sum of queue lengths and waiting delays for all vehicles entering the intersection. Higher costs result in lower rewards; the agent's goal is to maximize the reward (i.e., minimize the cost).

[0095] S4. Calculate the advantage function and update the policy and value network parameters.

[0096] This step is the core training process of reinforcement learning, employing an Actor-Critic architecture (such as...). Figure 4As shown in the figure, Actor refers to the policy network and Critic refers to the value network.

[0097] First, calculate the advantage function for each agent. :

[0098] ,

[0099] Then, the advantage function is used to guide the update of the policy network parameters, and its objective function is:

[0100] ,

[0101] The meaning of this formula is: if the dominance value of an action... If positive, adjust the strategy network parameters. To increase in the current state The probability of choosing this action Conversely, it reduces the probability.

[0102] Simultaneously, the value network needs to be updated to make its estimation of state values ​​more accurate. This is achieved by minimizing the following loss function:

[0103] ,

[0104] S5. Add stability constraints to the policy update.

[0105] In multi-agent environments, a common problem is "training oscillation": a change in one agent's policy can disrupt the policies already learned by other agents, causing all agents' policies to fluctuate repeatedly and fail to converge. This step introduces constraints to ensure a "smooth ride" in the training process.

[0106] The KL divergence between the policy distributions before and after the update is used as a stability constraint.

[0107] ,

[0108] In layman's terms, KL divergence measures the performance of the old and new strategies in the same state. The output is the "similarity" of the probability distribution of actions. This constraint requires that after each update, the new policy cannot differ too much from the old policy, and the update magnitude must be controlled within a threshold. Within the defined trust domain.

[0109] S6. Determine if convergence has occurred and deploy the application.

[0110] During training, it is necessary to determine when the algorithm has learned a sufficiently good policy. This method sets several convergence criteria, and training can be stopped as long as one of them is met:

[0111] (1) The policy network parameters hardly change anymore: ;

[0112] (2) The prediction error of the value network is already very small and stable: ;

[0113] (3) The cumulative rewards obtained by the system no longer increase: ;

[0114] (4) The sum of the policy gradient norms of all agents is small enough, which means that the policy has reached a stable point: ,in This is a preset threshold.

[0115] Once training is complete, the trained policy network (Actor network) is deployed to the signal controllers at each intersection in the actual road network. During runtime, each agent no longer needs global information or the complex calculations required during training. It only needs to quickly calculate its optimal signal control action based on real-time local observations and a preset decision sequence, thereby achieving real-time, decentralized, collaborative traffic signal control.

[0116] like Figure 5 As shown, the convergence curves of the proposed Autoregressive Multi-Agent Method (AR-MAAC) and existing methods (MA2C, IQL, and Fixed-Time) in a simulation environment, as well as the average queue length as a function of training rounds, are compared. The significant advantages of this invention are evident.

[0117] (1) In the early stage of training (about the first 30 rounds), the average queue length of the method of this invention dropped rapidly from about 6-7 vehicles to less than 2 vehicles, showing a significantly faster convergence speed. In contrast, the MA2C method took more than 50 rounds to gradually stabilize at about 4-5 vehicles, the IQL method decreased slowly and always remained at about 22 vehicles, while the Fixed-Time method, due to its lack of adaptive optimization ability, always maintained an average queue length at a high level (about 46-48 vehicles), with basically no significant improvement.

[0118] (2) In terms of convergence stability, the curve of the method of the present invention has the smallest fluctuation and remains basically stable after convergence, which shows that it has better training stability in multi-agent non-stationary environment, while MA2C and IQL methods still have some oscillations during training.

[0119] (3) In terms of the final control effect, the average queue length after convergence of the method of the present invention is about 1–2 vehicles, which is significantly better than MA2C (about 4–5 vehicles), IQL (about 22 vehicles) and Fixed-Time (about 47 vehicles), indicating that it can more effectively reduce traffic congestion and improve the efficiency of road network traffic.

[0120] In summary, this invention achieves a comprehensive improvement in convergence speed, training stability, and control performance by introducing an autoregressive strategy structure to explicitly model the decision dependencies between multiple agents and combining it with a stability constraint mechanism.

[0121] The above description is merely one specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-agent traffic signal control method based on an autoregressive strategy, characterized in that, The specific implementation of this method is as follows: S1. Construct a traffic signal control problem model, treating each intersection in the traffic network as an agent, and the set of all agents as a multi-agent system; at the discrete control time t, the system state is denoted as... The local observation of the i-th agent is denoted as i takes values ​​from 1 to N, where N is the number of agents; joint action Recorded as: ; S2. Based on the spatial relationship of each intersection in the traffic network, the direction of road connection, the priority of the main road or the real-time traffic pressure, determine the decision order of each agent at time t, and decompose the joint strategy into an autoregressive strategy so that the actions of subsequent agents depend on the action information of the preceding agents. S3. Based on each agent's local observations, neighborhood traffic state characteristics, and previous agent action information, construct the conditional local action value function and conditional local state value function for each agent. S4. Calculate the advantage function of each agent based on the conditional local action value function and the conditional local state value function, and update the policy parameters and value function parameters of each agent using reinforcement learning methods. S5. Add a stability constraint term during the policy update process to limit the change range between two adjacent policy updates, so as to ensure monotonic improvement and convergence stability during training. S6. When the preset convergence condition is met, training stops and the trained strategy is deployed to each traffic intersection controller. Each agent outputs traffic signal control actions in sequence according to the decision order based on real-time observation, thereby realizing coordinated traffic signal control.

2. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The local observation of the i-th agent in step S1 This includes information reflecting the current traffic status at the intersection, and the actions of the i-th agent. This includes adjustments to traffic signal control strategies.

3. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The autoregressive strategy described in step S2 is expressed as follows: ， in, This represents the ID of the k-th agent that makes a decision after sorting at time t. Indicates the joint policy parameters, This represents the policy parameters of the i-th agent. This represents the conditional input for the i-th agent; The conditional input Represented as: ， in, This represents the contextual features extracted from the traffic conditions of the surrounding area. This represents the set of actions of all agents preceding the i-th agent in the current decision-making order; The autoregressive strategy transforms the direct search of the original joint action space into a conditional action search in a predetermined order.

4. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The decision-making order described in step S2 can be a fixed order or a dynamic order; When a fixed sequence is used, decisions are made according to the order of main roads to branch roads, upstream to downstream, or preset intersection numbers in the traffic network. When a dynamic sequence is adopted, the decision-making order is determined based on the traffic pressure index of each intersection at time t, wherein the traffic pressure index is defined as: ， in, This represents the queue length statistics at the i-th intersection. This represents the statistical measure of waiting delay. This represents the lane occupancy rate statistic. This represents a statistic indicating the arrival rate or saturation of traffic flow. These are non-negative weighting coefficients; The decision-making order of the agent at the current control moment is obtained by sorting the traffic pressure indicators of each intersection from largest to smallest.

5. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, In step S3, the conditional local action value function and conditional local state value function of the i-th agent are constructed as follows: ， ， in, Represents the parameters of the value network. As a discount factor, Indicates the time when the i-th agent is at time... The partial reward obtained; In step S3, the inputs to the conditional local action value function and the conditional local state value function do not use the global joint state, but instead use local observations, neighborhood context features, and previous agent action information related to the current agent, thereby reducing the input dimension and computational complexity.

6. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 5, characterized in that, The local reward function of the i-th agent in step S3 is defined as follows: ， in, Let represent the comprehensive traffic operation cost function of the i-th intersection at time t, which is used to characterize the traffic operation status of the intersection.

7. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The advantage function of the i-th agent in step S4 is defined as follows: ， The objective function for updating the policy parameters in step S4 is: ， The loss function for updating the value network parameters is: ， in, For batch sample size, Accumulate returns towards the target.

8. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The stability constraint term in step S5 is represented by the KL divergence between the policy distributions before and after the update, satisfying: ， in, This represents the policy parameters before the i-th agent updates. This indicates the corresponding update threshold; The convergence constraint term in step S5 is characterized by the relationship between performance difference and policy gradient norm, satisfying: ， in, Indicates the performance of the optimal strategy. Indicates the current policy performance. It is a positive number; By using the KL divergence constraint and convergence constraint, the magnitude of policy parameter updates is limited, thereby improving training stability in non-stationary multi-agent environments.

9. The multi-agent traffic signal control method based on an autoregressive strategy according to claim 1, characterized in that, The convergence criterion in step S5 includes at least one of the following: (1) The change in strategy parameters between two consecutive iterations satisfies ； (2) The change in the value function loss between two adjacent iterations satisfies ； (3) The cumulative change in return between two adjacent iterations satisfies ； (4) The sum of the policy gradient norms of all agents satisfies ； in, The preset threshold; When any of the aforementioned convergence criteria is met, the training ends and the trained traffic signal control strategy is output.

10. A multi-agent traffic signal control system based on an autoregressive strategy that implements the method as described in any one of claims 1-9, characterized in that, include: The status acquisition module is used to obtain traffic status information at each intersection. The decision-making module is used to generate traffic signal control actions based on an autoregressive strategy; The training and update module is used to update policy parameters and determine the convergence status; The control execution module is used to output traffic signal control results; The system is used to perform the method according to any one of claims 1 to 9.

Images