Multi-depth reinforcement learning agent collaborative regional traffic signal control system and method based on green wave band theory

Abstract

The invention belongs to the field of intelligent traffic, and provides a multi-deep reinforcement learning agent cooperative regional traffic signal control system and method based on a green wave band theory, and the method comprises the following steps: (1) carrying out the overall architecture design of a multi-agent cooperative control system, a multi-center hierarchical multi-DDPG intelligent agent cooperative control system based on periodic control and trunk line green wave control is constructed; (2) state spaces, action spaces, reward functions and exploration modes of three types of agents in the hierarchical multi-agent control system are defined, and control algorithms are selected; and (3) performing multi-agent cooperative training optimization on the design by adopting four technologies of output standardization, Ornstein-Uhlenbeck exploration noise, experience playback and expert strategy, solving the problem that the system is difficult to converge when a single agent faces moving target learning, and timely responding to the dynamic change of the traffic flow through the agent system architecture design and the single agent design. And a signal timing scheme is optimized for real-time control, so that the driving delay at the intersection is reduced, and the passing efficiency at the intersection is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a regional traffic signal control system and method based on the collaboration of multiple deep reinforcement learning agents based on green band theory. Background Art

[0002] Due to population growth and urbanization, the number of cars in cities has increased rapidly and the level of motorization has continued to improve. The huge daily traffic volume has put pressure on the existing urban transportation infrastructure, leading to daily traffic congestion. The essence of traffic congestion is the imbalance between the road network capacity and traffic demand, and urban signal intersections are the bottleneck of the urban road network capacity, limiting the traffic efficiency of the urban road network. Scientific and reasonable regional signal control can significantly improve the traffic capacity of intersections in the road network, thereby improving the traffic efficiency of the road network and improving traffic congestion. Urban intelligent regional signal control treats intersections as a real-time controllable system, and continuously monitors, diagnoses, models, and controls intersections in real time. However, the traditional intersection signal control system based on a fixed timing scheme cannot adapt to the nonlinearity, randomness, fuzziness, and uncertainty of the traffic system.

[0003] Adaptive traffic signal control systems can respond to dynamic changes in traffic flow in a timely manner and optimize signal timing schemes for real-time control. However, existing adaptive traffic signal control systems have the following limitations: (1) Most of them simply replicate single-point control to multiple points to achieve regional signal control, and simplify the actual application scenarios. (2) Single agents face the problem of moving target learning, and agents may compete with each other, making it difficult for the control system to converge. (3) As time changes, model parameters or preset schemes may no longer be applicable. (4) Detector failures can easily affect the stability of the system.

[0004] As an unsupervised machine learning method, reinforcement learning can directly interact with the environment to learn control strategies. It does not require a predefined model of the environment, nor does it require human supervision and intervention. The agent obtains the state representation of the environment by observing the environment, and randomly selects actions from the predefined action set based on the policy function. After the agent performs the action, the environment will feedback the reward supervision signal to evaluate the effect of the agent's action. At the same time, the environment will be updated to the next state according to the state transition function. The agent will repeat the above process until the maximum cumulative reward is obtained at the end of a round. The signal control method based on deep reinforcement learning is a model-free, data-driven control method that can self-learn and improve its own strategy during the control process, and can achieve true closed-loop online control. However, there are still few studies on regional multi-point signal control based on deep reinforcement learning. In addition, it is difficult for traditional regional adaptive signal control methods to cover all situations through interaction with the environment for larger state spaces and action spaces, and it is difficult to directly model the state transition function of the environment based on the model-based reinforcement learning algorithm. Summary of the invention

[0005] In order to overcome the above shortcomings, the present invention aims to provide a regional traffic signal control system and method based on the green wave theory and multi-deep reinforcement learning agent collaboration. Through the agent system architecture design and single agent design, it can respond to the dynamic changes of traffic flow in a timely manner, optimize the signal timing scheme for real-time control, thereby reducing traffic delays at intersections and improving traffic efficiency at intersections. The present invention achieves the above purpose through the following technical means:

[0006] A regional traffic signal control system and method based on the green wave theory and multiple deep reinforcement learning agents, characterized by including agent system architecture design and single agent design;

[0007] The intelligent system architecture design refers to building a sub-center double-layer multi-agent system based on the design concept of green wave control, and realizing the coordination of the multi-agent system through the coordinated control of the top-level intelligent agent on the bottom-level intelligent agent and the coordination between the bottom-level intelligent agents;

[0008] The single-agent design refers to the three types of agents in the agent system interacting with the traffic environment in real time to learn control strategies. The three types of agents are coordination control agents, trunk intersection control agents, and non-trunk intersection control agents. The three types of agents all include algorithm selection, definition of action space, definition of perception state space, and definition of reward function.

[0009] Furthermore, the intelligent system architecture design includes two levels of control systems: trunk green wave coordination control system and non-trunk coordination control system, specifically:

[0010] The trunk green wave coordination control system is responsible for controlling and coordinating the intersection signal control parameters of the trunk line;

[0011] The non-main line coordinated control system is responsible for controlling the signal control of the intersections of the branches adjacent to the main line, and coordinating with the adjacent main line intersections;

[0012] Furthermore, the single agent design includes algorithm selection, definition of action space, definition of perception state space and definition of reward function, specifically:

[0013] The algorithm selection is based on the DDPG algorithm, which is based on the Actor-Critic framework and introduces the experience replay and target network technologies in DQN to optimize the convergence of the value function;

[0014] Furthermore, the DDPG algorithm process is specifically as follows:

[0015] Step 1 is to randomly initialize the policy network μ(s|θ μ ) and the value network Q(s,α|θ Q );

[0016] Step 2 is to initialize the target policy network and the target value network μ'(s|θ μ′ )=μ(s|θ μ ), Q'(s,α|θ Q ′)=Q(s,α|θ Q );

[0017] Step 3 is to initialize the replay buffer R and initialize the random ξ process as the exploration noise in the period

[0018] Step 4 is to receive the initial state s0 of the environment within the period;

[0019] Step 5 is to select behavior a at each time t in the cycle according to the policy network and exploration noise t =μ(s|θ μ )+ξ t ;

[0020] Step 6 is to execute a at each time t in the cycle t And get reward t and the next environment state s t+1 , while storing the experience trajectory data i ,a i ,r i ,s i+1 >To replay buffer R;

[0021] Step 7 is to extract the empirical trajectory data in batches from the replay buffer R according to the formula​

[0022]

[0023] y i =r i +Q'(s i+1 ,μ'(s i+1 |θ μ ′)|θ Q ′)

[0024] Update the value function network μ(s|θ μ );

[0025] Step 8 is based on the formula

[0026]

[0027] Update the policy network μ(s|θ μ );

[0028] Step 9 is to update the target network μ'(s|θ μ′ )=μ(s|θ μ ), Q'(s,α|θ Q′ )=μ(s|θμ).

[0029] Furthermore, the policy neural network (ActorNetwork) and value neural network (CriticNetwork) in the DDPG algorithm adopt a 5×10 fully connected network (5 hidden layers, 10 neurons in each layer), the hidden layer activation function is the elu function, and the output layer activation function is the hyperbolic tangent (tanh) function.

[0030] Furthermore, the exploration noise in the DDPG algorithm is subject to the random noise of the Ornstein-Uhlenbeck random process, specifically:

[0031] dx t = -θ(x t -μ)dt+σdW t

[0032] Where μ is the mean and θ is the normalization parameter, which defines the strength of regression to the mean. t is the Brownian motion process, and σ is the fluctuation coefficient.

[0033] Furthermore, the initial state s0 of the environment includes the perception state space definitions of three types of agents, specifically:

[0034] The definition of the perception state space of the coordinated control agent modifies the control of the phase difference to avoid the agent's perception state space dimension being too large. The formula is as follows

[0035]

[0036] in represents the mth intersection of the ith intersection j The average upstream arrival flow per second in a time step in the phase;

[0037] The perception state space of the trunk intersection control agent is defined with the maximum queue length of the phase and the arrival flow per second in the previous control cycle of the phase as state parameters. The specific perception state space is as follows:

[0038] S=(f1,f2,…,f m ,l1,l2,…,l m )

[0039] where f m It represents the average upstream arrival flow per second in the previous time step of the mth phase. m represents the maximum queue length of the mth phase in the previous time step;

[0040] The perception state space definition of the non-artery intersection control agent also uses the maximum queue length of the phase and the arrival flow per second of the previous control cycle of the phase as state parameters. The specific perception state space formula is the same as that of the artery intersection control agent, such as formula S = (f1, f2, ..., f m ,l1,l2,...,l m ).

[0041] Furthermore, the policy network output should be standardized, specifically:

[0042] The output of the policy model is normalized to [1,10], and the specific formula is as follows

[0043]

[0044] C=6c+40

[0045] where n i represents the i-th phase, and c represents the signal period.

[0046] Furthermore, the agent system selects action behaviors based on the exploration noise, specifically:

[0047] The action space definition of the coordinated control agent uses the correction amount of the ideal phase difference as the actual output parameter of the agent, as follows

[0048]

[0049] Where L i represents the distance between the ith intersection and the i+1th intersection, Vi represents the free flow speed between the i-th intersection and the i+1-th intersection;

[0050] The action space of the arterial intersection control agent is defined based on its own green signal ratio in each phase, and its formula is as follows

[0051] A=(g1,g2,…,g m )

[0052] where g m Indicates the relative proportion of the green light duration of the mth phase to the cycle;

[0053] The action space of the non-mainline intersection control agent is defined with its own period and phase green signal ratio as parameters, and its formula is as follows

[0054] A=(g1,g2,...,g m ,c)

[0055] where g m It indicates the relative proportion of the green light duration of the mth phase to the cycle; c indicates the signal cycle.

[0056] Furthermore, the DDPG algorithm in the single agent design defines the reward functions of the three types of agents respectively, specifically:

[0057] The reward function of the coordinated control agent is defined and updated with the signal control period as the time step, as follows

[0058]

[0059] where d kt represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane;

[0060] The reward function definition of the arterial intersection control agent uses a spatially weighted approach to calculate the reward of the arterial intersection, as follows

[0061]

[0062] where d kt The average single-vehicle delay per second in the control cycle of the k-th entrance lane during this time step. L n Indicates the distance between the current intersection and the nth intersection. Represents the reward information of the current time step of the nth intersection;

[0063] The reward function definition of the non-mainline intersection control agent only considers the delay of the control itself. The reward function formula is as follows

[0064]

[0065] where d kt It represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane.

[0066] The beneficial effects of the present invention are:

[0067] (1) The regional traffic signal control system and method based on the green wave theory and multi-deep reinforcement learning agent collaboration of the present invention absorbs the idea of ​​green wave control, takes period, phase difference, and green signal ratio as control parameters, and uses the DDPG algorithm to construct a multi-center hierarchical multi-agent collaborative control framework. The collaboration of multiple agents is achieved through the interaction of perception states between the bottom-level agents, the sharing of reward information, and the action linkage between the top-level agent and the bottom-level agent.

[0068] (2) The regional traffic signal control system and method based on the green band theory and the collaboration of multiple deep reinforcement learning agents described in the present invention adopt four optimization technologies, namely, experience replay, OU exploration noise, expert strategy, and strategy network output standardization, to improve the learning performance of the agent.

[0069] (3) The regional traffic signal control system and method based on the green band theory and coordinated by multiple deep reinforcement learning agents described in the present invention analyzes the factors affecting the multi-agent training effect, optimizes the convergence of the multi-agent system training by learning rate, selection of neural network activation function, batch learning sample size, output standardization, and agent reward definition method, and introduces expert strategies to significantly improve the convergence stability of the agent and the control effect at the final convergence.

[0070] (4) Compared with traditional timing control and induction control, the regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents described in the present invention can respond to the dynamic changes of traffic flow in a timely manner, reduce delays at road network intersections, increase operating speeds, and make the control process more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the system framework of a regional traffic signal control system and method based on the green wave theory and multi-deep reinforcement learning agent collaboration in an embodiment of the present invention;

[0072] Figure 2 is a schematic diagram of the collaborative relationship between three types of intelligent agents in an embodiment of the present invention;

[0073] Figure 3 Schematic diagram of a double-layer intelligent agent control system in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical means and advantages of the present invention more clearly explained, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention, and the protection scope of the present invention is not limited thereto:

[0075] like Figure 1 As shown, the present invention provides a regional traffic signal control system and method based on the green wave band theory and multi-deep reinforcement learning agent collaboration, including agent system architecture design and single agent design; the agent system architecture design includes two levels of control systems: trunk green wave coordination control system and non-trunk coordination control system; the single agent design includes algorithm selection, definition of action space, definition of perception state space and definition of reward function;

[0076] The trunk green wave coordination control system is responsible for controlling and coordinating the intersection signal control parameters of the trunk line;

[0077] The non-main line coordinated control system is responsible for controlling the signal control of the intersections of the branches adjacent to the main line, and coordinating with the adjacent main line intersections;

[0078] The algorithm selection is based on the DDPG algorithm, which is based on the Actor-Critic framework and introduces the experience replay and target network technologies in DQN to optimize the convergence of the value function.

[0079] In this embodiment, the algorithm selection is based on the DDPG algorithm, and the DDPG algorithm process is specifically as follows:

[0080] Step 1 is to randomly initialize the policy network μ(s|θ μ ) and the value network Q(s,α|θ Q );

[0081] Step 2 is to initialize the target policy network and the target value network μ'(s|θ μ′ )=μ(s|θ μ ), Q'(s,α|θ Q′ )=Q(s,α|θ Q );

[0082] Step 3 is to initialize the replay buffer R and initialize the random ξ process as exploration noise within the period. Step 4 is to receive the initial state s0 of the environment within the period.

[0083] Step 5 is to select behavior a at each time t in the cycle according to the policy network and exploration noise t =μ(s|θ μ )+ξ t ;

[0084] Step 6 is to execute a at each time t in the cycle t And get reward t and the next environment state s t+1 , while storing the experience trajectory data i ,a i ,r i ,s i+1 >To replay buffer R;

[0085] Step 7 is to extract the empirical trajectory data in batches from the replay buffer R according to the formula

[0086]

[0087] y i =r i +Q'(s i+1 ,μ'(s i+1 |θ μ′ )|θ Q′ )

[0088] Update the value function network μ(s|θ μ );

[0089] Step 8 is based on the formula

[0090]

[0091]

[0092] Update the policy network μ(s|θ μ );

[0093] Step 9 is to update the target network μ'(s|θ μ )=μ(s|θ μ ), Q'(s,α|θ Q′ )=μ(s|θ μ );

[0094] In this embodiment, the exploration noise in the DDPG algorithm obeys the random noise of the Ornstein-Uhlenbeck random process, specifically:

[0095] dx t = -θ(x t -μ)dt+σdW t

[0096] Where μ is the mean and θ is the normalization parameter, which defines the strength of regression to the mean. t is the Brownian motion process, σ is the fluctuation coefficient;

[0097] ​In this embodiment, the policy network output in the DDPG algorithm should be standardized, specifically:

[0098] The output of the policy model is normalized to [1,10], and the specific formula is as follows

[0099]

[0100] C=6c+40

[0101] where n i represents the i-th phase, and c represents the signal period.

[0102] In this embodiment, if Figure 2 The trunk green wave coordinated control system includes the perception state space definition, action space definition and reward function definition of the trunk single agent, specifically:

[0103] The definition of the perception state space of the coordinated control agent modifies the control of the phase difference to avoid the agent's perception state space dimension being too large. The formula is as follows

[0104]

[0105] in represents the mth intersection of the ith intersection j The average upstream arrival flow per second in a time step in the phase;

[0106] The perception state space of the trunk intersection control agent is defined with the maximum queue length of the phase and the arrival flow per second in the previous control cycle of the phase as state parameters. The specific perception state space is as follows:

[0107] S=(f1,f2,…,f m ,l1,l2,…,l m )

[0108] where f m It represents the average upstream arrival flow per second in the previous time step of the mth phase. m represents the maximum queue length of the mth phase in the previous time step;

[0109] The action space definition of the coordinated control agent uses the correction amount of the ideal phase difference as the actual output parameter of the agent, as follows

[0110]

[0111] Where L i represents the distance between the ith intersection and the i+1th intersection, V i represents the free flow speed between the i-th intersection and the i+1-th intersection;

[0112] The action space of the arterial intersection control agent is defined based on its own green signal ratio in each phase, and its formula is as follows

[0113] A=(g1,g2,…,g m )

[0114] where g m Indicates the relative proportion of the green light duration of the mth phase to the cycle;

[0115] The reward function of the coordinated control agent is defined and updated with the signal control period as the time step, as follows

[0116]

[0117] where d kt represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane;

[0118] The reward function definition of the arterial intersection control agent uses a spatially weighted approach to calculate the reward of the arterial intersection, as follows

[0119]

[0120] where d kt The average single-vehicle delay per second in the control cycle of the k-th entrance lane during this time step. L n Indicates the distance between the current intersection and the nth intersection. Represents the reward information of the current time step of the nth intersection;

[0121] In this embodiment, the non-mainline collaborative control system includes the perception state space definition, action space definition and reward function definition of the non-mainline agent, specifically:

[0122] The perception state space definition of the non-artery intersection control agent also uses the maximum queue length of the phase and the arrival flow per second of the previous control cycle of the phase as state parameters. The specific perception state space formula is the same as that of the artery intersection control agent, such as formula S = (f1, f2, ..., f m ,l1,l2,…,l m );

[0123] where f m It represents the average upstream arrival flow per second in the previous time step of the mth phase. m represents the maximum queue length of the mth phase in the previous time step;

[0124] The action space of the non-mainline intersection control agent is defined with its own period and phase green signal ratio as parameters, and its formula is as follows

[0125] A=(g1,g2,…,g m ,c)

[0126] where g m It indicates the relative proportion of the green light duration of the mth phase to the period; c indicates the signal period;

[0127] The reward function definition of the non-mainline intersection control agent only considers the delay of the control itself. The reward function formula is as follows

[0128]

[0129] where d kt It represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane.

Claims

1. A regional traffic signal control system and method based on the green band theory and multi-deep reinforcement learning agent collaboration, characterized in that: Includes agent system architecture design and single agent design; The intelligent system architecture design refers to building a sub-center double-layer multi-agent system based on the design concept of green wave control, and realizing the coordination of the multi-agent system through the coordinated control of the top-level intelligent agent on the bottom-level intelligent agent and the coordination between the bottom-level intelligent agents; The single-agent design refers to the three types of agents in the agent system interacting with the traffic environment in real time to learn control strategies. The three types of agents are coordination control agents, trunk intersection control agents, and non-trunk intersection control agents. The three types of agents all include algorithm selection, definition of action space, definition of perception state space, and definition of reward function.

2. According to claim 1, a regional traffic signal control system and method based on the green wave theory and multiple deep reinforcement learning agents, characterized in that: The intelligent system architecture design includes two levels of control systems: trunk green wave coordination control system and non-trunk coordination control system, specifically: The trunk green wave coordination control system is responsible for controlling and coordinating the intersection signal control parameters of the trunk line; The non-main line coordinated control system is responsible for controlling the signal control of the intersections of the branches adjacent to the main line, and cooperates with the adjacent main line intersections.

3. According to claim 1, a regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents is characterized in that: The single agent design includes algorithm selection, definition of action space, definition of perception state space and definition of reward function, specifically: The algorithm selection is based on the DDPG algorithm, which is based on the Actor-Critic framework and introduces the experience replay and target network technologies in DQN to optimize the convergence of the value function.

4. According to claim 3, a regional traffic signal control system and method based on the green wave theory and multiple deep reinforcement learning agents are coordinated, characterized in that: The DDPG algorithm process is specifically as follows: Step 1 is to randomly initialize the policy network μ(s|θ μ ) and the value network Q(s,α|θ Q ); Step 2 is to initialize the target policy network and the target value network μ'(s|θ μ′ )=μ(s|θ μ ), Q'(s,α|θ Q′ )=Q(s,α|θ Q ); Step 3 is to initialize the replay buffer R and initialize the random ξ process as the exploration noise in the period Step 4 is to receive the initial state s0 of the environment within the period; Step 5 is to select behavior a at each time t in the cycle according to the policy network and exploration noise t =μ(s|θ μ )+ξ t ; Step 6 is to execute a at each time t in the cycle t And get reward t and the next environment state s t+1 , while storing the experience trajectory data i ,a i ,r i ,s i+1 >To replay buffer R;​ Step 7 is to extract the empirical trajectory data in batches from the replay buffer R according to the formula y i =r i +Q'(s i+1 ,m'(s i+1 |θ μ′ )|θ Q ) Update the value function network μ(s|θ μ ); Step 8 is based on the formula Update the policy network μ(s|θ μ ); Step 9 is to update the target network μ'(s|θ μ‘ )=μ(s|θ μ ), Q'(s,α|θ Q′ )=μ(s|θ μ ).

5. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 1, 3 or 4, characterized in that: The policy neural network (ActorNetwork) and value neural network (CriticNetwork) in the DDPG algorithm adopt a 5×10 fully connected network (5 hidden layers, 10 neurons in each layer), the hidden layer activation function is the elu function, and the output layer activation function is the hyperbolic tangent (tanh) function.

6. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 1, 3 or 4, characterized in that: The exploration noise in the DDPG algorithm described above obeys the random noise of the Ornstein-Uhlenbeck random process, specifically: dx t =-θ(x t -μ)dt+σdW t Where μ is the mean and θ is the normalization parameter, which defines the strength of regression to the mean. t is the Brownian motion process, and σ is the fluctuation coefficient.

7. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 1, 2, 3 or 4, characterized in that: The initial state s0 of the environment includes the perception state space definitions of three types of agents, specifically: The definition of the perception state space of the coordinated control agent modifies the control of the phase difference to avoid the agent's perception state space dimension being too large. The formula is as follows in represents the mth intersection of the ith intersection j The average upstream arrival flow per second in a time step in the phase; The perception state space of the trunk intersection control agent is defined with the maximum queue length of the phase and the arrival flow per second in the previous control cycle of the phase as state parameters. The specific perception state space is as follows: S=(f1,f2,…,f m ,l1,l2,…,l m ) where f m It represents the average upstream arrival flow per second in the previous time step of the mth phase. m represents the maximum queue length of the mth phase in the previous time step; The perception state space definition of the non-artery intersection control agent also uses the maximum queue length of the phase and the arrival flow per second of the previous control cycle of the phase as state parameters. The specific perception state space formula is the same as that of the artery intersection control agent, such as S = (f1, f2, ..., f m ,l1,l2,…,l m ).

8. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 1, 2, 3 or 4, characterized in that: The output of the policy network should be standardized, specifically: The output of the policy model is normalized to [1,10], and the specific formula is as follows C=6c+40 Where n i represents the i-th phase, and c represents the signal period.

9. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 1, 2, 3 or 4, characterized in that: The agent system selects action behaviors based on exploration noise, specifically: The action space definition of the coordinated control agent uses the correction amount of the ideal phase difference as the actual output parameter of the agent, as follows Where L i represents the distance between the ith intersection and the i+1th intersection, V i represents the free flow speed between the i-th intersection and the i+1-th intersection; The action space of the arterial intersection control agent is defined based on its own green signal ratio in each phase, and its formula is as follows A=(g1,g2,…,g m ) where g m Indicates the relative proportion of the green light duration of the mth phase to the cycle; The action space of the non-mainline intersection control agent is defined with its own period and phase green signal ratio as parameters, and its formula is as follows A=(g1,g2,…,g m ,c) where g m It indicates the relative proportion of the green light duration of the mth phase to the cycle; c indicates the signal cycle.

10. A regional traffic signal control system and method based on the green wave theory and coordinated by multiple deep reinforcement learning agents according to claim 3 or 4, characterized in that: The DDPG algorithm in the single agent design defines the reward functions of the three types of agents, specifically: The reward function of the coordinated control agent is defined and updated with the signal control period as the time step, as follows where d kt represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane; The reward function definition of the arterial intersection control agent uses a spatially weighted approach to calculate the reward of the arterial intersection, as follows where d kt The average single-vehicle delay per second in the control cycle of the k-th entrance during this time step. L n Indicates the distance between the current intersection and the nth intersection. Represents the reward information of the current time step of the nth intersection; The reward function definition of the non-mainline intersection control agent only considers the delay of the control itself. The reward function formula is as follows where d kt It represents the average single-vehicle delay per second in one control cycle at the nth intersection, mth phase, and kth entrance lane.

Images