Traffic signal control method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a traffic signal control method and device, electronic equipment and storage medium, belonging to the technical field of intelligent traffic control. The method comprises: obtaining a plurality of candidate intersections of a traffic network; according to the spatial proximity relationship between the plurality of candidate intersections, performing merging processing on the candidate intersections that meet the preset merging condition to obtain at least one leader set; according to the at least one leader set, partitioning the traffic network to obtain at least one control group; constructing a deep reinforcement learning network according to the at least one control group, and training the deep reinforcement learning network to enable the deep reinforcement learning network to learn a first coordination strategy and a second coordination strategy to obtain a control network; and using the control network to make decisions based on the real-time traffic state of the traffic network to obtain the signal control action of each intersection in the traffic network. The embodiments of the present application can reduce the complexity of joint decision-making.

Description

Technical Field

[0001] This application relates to the field of intelligent traffic control technology, and in particular to a traffic signal control method, device, electronic equipment and storage medium. Background Technology

[0002] Traffic signal control systems are key infrastructure for alleviating urban road congestion and improving road network efficiency. With the development of the Internet of Things (IoT) and vehicle-road cooperative technologies, signal control has evolved from fixed timing and sensor-based control to adaptive control driven by real-time data. In recent years, deep reinforcement learning, due to its powerful perception and decision-making capabilities, has been widely applied to adaptive signal control at single intersections and small-scale road networks, achieving dynamic timing optimization superior to traditional rule-based methods under complex traffic conditions.

[0003] However, existing traffic signal control methods based on deep reinforcement learning treat all intersections in the entire road network as a unified control object for joint decision-making. Although this can theoretically achieve global optimal coordination, the dimensions of the joint state space and action space increase exponentially with the number of intersections, resulting in excessively high training complexity. Summary of the Invention

[0004] The main objective of this application is to provide a traffic signal control method, device, electronic device, and storage medium that can reduce the complexity of joint decision-making.

[0005] To achieve the above objectives, a first aspect of this application provides a traffic signal control method, the method comprising: Obtain multiple candidate intersections in the traffic network; Based on the spatial proximity relationship between multiple candidate intersections, candidate intersections that meet the preset merging conditions are merged to obtain at least one leader set, wherein the leader set includes at least one of the candidate intersections. The traffic network is partitioned according to at least one set of leaders to obtain at least one control group. Each control group includes one set of leaders and one set of followers. The set of followers includes ordinary intersections in the traffic network, which are intersections in the traffic network other than candidate intersections in the set of leaders. A deep reinforcement learning network is constructed based on at least one of the control groups, and the deep reinforcement learning network is trained to learn a first coordination strategy and a second coordination strategy. The first coordination strategy is a coordination strategy for controlling candidate crossroads in each of the leader sets, and the second coordination strategy is a coordination strategy for controlling ordinary crossroads in the follower sets, thereby obtaining a control network. The control network is used to make decisions on the real-time traffic status of the traffic network, and to obtain the signal control actions for each intersection in the traffic network.

[0006] In some embodiments, the preset merging conditions include at least one of the following: The spatial distance between the two candidate intersections is less than a preset distance threshold; The traffic coupling strength between the two candidate intersections is greater than a preset coupling threshold. The traffic coupling strength is obtained based on the traffic demand, number of vehicles, and road segment length between the two intersections within a preset time period. The key movement directions between the two candidate intersections are the same or related, and the key movement direction is the vehicle movement direction that has the greatest impact on the signal timing or capacity of the intersection. The two candidate intersections are located in the same traffic corridor.

[0007] In some embodiments, partitioning the traffic network according to at least one set of leaders to obtain at least one control group includes: Using each set of leaders as the core point of each partition, traverse the ordinary intersections; Under the constraints that each ordinary intersection is assigned to only one control group, spatial connectivity is maintained among intersections within each control group, and each control group includes only one set of leaders, the comprehensive allocation cost of each ordinary intersection is determined based on the spatial compactness between the ordinary intersection and the set of leaders within the control group, the degree of traffic flow separation between the ordinary intersection and other intersections within the control group, the consistency of traffic load among each intersection within the control group, and the degree of balance in the number of intersections among each control group. Each of the ordinary intersections is assigned to a candidate control group that minimizes the overall allocation cost of the ordinary intersection, resulting in multiple control groups.

[0008] In some embodiments, constructing a deep reinforcement learning network based on at least one of the control groups and training the deep reinforcement learning network to learn a first coordination policy and a second coordination policy includes: During the learning phase of the second coordination strategy, the first coordination strategy of the leader set is fixed, and the second coordination strategy of the follower set is updated. During the learning phase of the first coordination strategy, candidate coordination strategies are generated based on the updated second coordination strategy. Based on the value comparison between the candidate coordination strategy and the current coordination strategy, the first coordination strategy of the leader set is updated.

[0009] In some embodiments, during the learning phase of the second coordination strategy, fixing the first coordination strategy of the leader set and updating the second coordination strategy of the follower set includes: With the first coordination strategy of the leader set fixed, obtain the state transition data generated by the signal phase action of each follower set executing the second coordination strategy; The state transition data is stored in the experience replay pool; Historical state transition data are sampled from the experience replay pool and calculated using a preset target value function to obtain the temporal difference target of each follower set under a given local observation state, the first coordination strategy of the leader set, and its own action conditions. The second coordination strategy of the follower set is updated by minimizing the deviation between the temporal difference objective and the value of the current follower action.

[0010] In some embodiments, updating the first coordination strategy of the leader set based on a value comparison between the candidate coordination strategies and the current coordination strategy includes: With the updated second coordination strategy fixed, the deep reinforcement learning network obtains the first global value evaluation result corresponding to the current coordination strategy and the second global value evaluation result corresponding to the candidate coordination strategy in each enhanced state. The enhanced state includes: the maximum pressure value of each signal phase level, the signal phase currently being executed at each intersection, the duration of the current phase, the maximum vehicle waiting time in each entrance direction, and the vehicle queue length in each entrance direction. If the value comparison result meets the preset conditions, the candidate coordination strategy is taken as the current coordination strategy, and the first coordination strategy of the leader set is updated according to the candidate coordination strategy. The preset conditions are that the second global value assessment result is greater than or equal to the first global value assessment result in each of the enhanced states, and is greater than the first global value assessment result in at least one of the enhanced states. If the value comparison result does not meet the preset conditions, the current coordination strategy remains unchanged.

[0011] In some embodiments, the control network includes a leader deep network and a follower deep network, wherein the leader deep network is used to learn a first coordination strategy for the leader set, and the follower deep network is used to learn a second coordination strategy for the follower set. The step of using the control network to make decisions based on the real-time traffic conditions of the traffic network to obtain the signal control actions for each intersection in the traffic network includes: The real-time traffic status of the traffic network at the current moment is obtained. The real-time traffic status includes the vehicle queue length in each direction of each intersection, the vehicle waiting time in each direction of each intersection, the currently executed signal phase and its duration. The real-time traffic status is input into the leader deep network to obtain multiple candidate coordination actions; Based on the candidate coordinated action and the traffic state of each follower set, the candidate signal phase action of each follower set under each candidate coordinated action is generated through the follower deep network. Through the leader deep network, each candidate coordination action is evaluated for value based on the real-time traffic status, each candidate coordination action, and the candidate signal phase actions of each follower set corresponding to each candidate coordination action. The candidate coordination action with the highest value is selected as the target coordination action at the current moment. The target coordination action is the signal phase control action of the intersection in the leader set. The candidate signal phase action of each follower set corresponding to the target coordination action is taken as the target signal phase action of each follower set at the current time, and the target signal phase action is the signal phase control action of the intersection in the follower set.

[0012] To achieve the above objectives, a second aspect of this application provides a traffic signal control device, the device comprising: The acquisition module is used to acquire multiple candidate intersections in the traffic network; The merging module is used to merge candidate intersections that meet preset merging conditions based on the spatial proximity relationship between multiple candidate intersections, to obtain at least one leader set, wherein the leader set includes at least one of the candidate intersections; A partitioning module is used to partition the traffic network according to at least one set of leaders to obtain at least one control group. Each control group includes a set of leaders and a set of followers. The set of followers includes ordinary intersections in the traffic network. Ordinary intersections refer to other intersections in the traffic network other than the candidate intersections in the set of leaders. A training module is configured to construct a deep reinforcement learning network based on at least one of the control groups and train the deep reinforcement learning network to learn a first coordination strategy and a second coordination strategy, wherein the first coordination strategy refers to a coordination strategy for controlling candidate crossroads in each of the leader sets, and the second coordination strategy refers to a coordination strategy for controlling ordinary crossroads in the follower sets, thereby obtaining a control network. The decision module is used to make decisions based on the real-time traffic conditions of the traffic network using the control network, and to obtain the signal control actions for each intersection in the traffic network.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0015] The traffic signal control method, device, electronic equipment, and storage medium proposed in this application acquire multiple candidate intersections in a traffic network and merge candidate intersections that meet preset merging conditions based on spatial proximity to obtain at least one leader set. Then, the traffic network is partitioned around the leader set to form at least one control group consisting of the leader set and a follower set including ordinary intersections. Subsequently, a deep reinforcement learning network is constructed based on the control group, and through hierarchical training, the deep reinforcement learning network learns a first coordination strategy for the leader set and a second coordination strategy for the follower set, respectively, to obtain the control network. Finally, the control network is used to determine and generate the signal control actions for each intersection in the traffic network based on real-time traffic conditions. This application decomposes a large-scale road network into multiple control groups with clear hierarchical relationships through candidate intersection merging and leader-guided partitioning, effectively reducing the dimensionality and training complexity of joint decision-making. Simultaneously, hierarchical training allows the leader set to focus on regional coordination and the follower set to focus on local response, balancing network-level coordination capabilities with the flexibility of local control. This achieves highly scalable, stable, and globally coordinated adaptive traffic signal control in large-scale heterogeneous road networks. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the traffic signal control method provided in the embodiments of this application; Figure 2This is a schematic diagram of the partitioning results of the first traffic network provided in the embodiments of this application; Figure 3 This is a schematic diagram of the zoning results of the second traffic network provided in the embodiments of this application; Figure 4 This is a schematic diagram of the traffic signal control device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Traffic signal control is a crucial component of urban road traffic management. It improves road efficiency, reduces vehicle delays, and alleviates traffic congestion by allocating right-of-way for different traffic flows at intersections. Traditional fixed timing methods typically rely on preset signal cycles and phase durations, offering the advantage of simple deployment but struggling to dynamically adjust to real-time traffic demand changes. Traditional adaptive control methods can adjust signal control strategies using detector data or local traffic conditions, but most rely on local control rules, manual parameter settings, or predefined coordination logic, making it difficult to fully characterize the traffic coupling relationships between different areas in large-scale road networks.

[0021] In recent years, deep reinforcement learning has been applied to traffic signal control. This type of method learns signal phase selection strategies from traffic states through neural networks, enabling adaptive control under complex traffic conditions. However, in large-scale traffic networks, if all intersections are treated as a unified control object for centralized joint control, the state space and action space grow rapidly with the number of intersections, leading to high training complexity, low sample efficiency, poor policy stability, and deployment difficulties. While decentralized control of each intersection as an independent agent offers better scalability, it easily overlooks the propagation relationships between adjacent roads, adjacent areas, and traffic bottlenecks, making effective network-level coordination difficult.

[0022] Existing multi-agent traffic signal control methods typically employ parameter sharing, graph attention, communication mechanisms, or region partitioning to enhance coordination capabilities, but several shortcomings remain: First, they lack a hierarchical coordination structure oriented towards candidate key intersections or bottleneck intersections; second, the control roles between ordinary intersections and candidate leader intersections are not clearly distinguished; third, there is a lack of unified integration between traffic network partitioning and deep reinforcement learning policy updates; and fourth, a stable acceptance mechanism based on the learning value function is lacking during the candidate high-level policy update process. Therefore, in large-scale heterogeneous traffic networks, a traffic signal control method that can balance scalability, regional coordination capabilities, and training stability is still needed.

[0023] Based on this, embodiments of this application provide a traffic signal control method, device, electronic device, and storage medium. Based on candidate leader intersections obtained from preset, user-specified, or external traffic engineering analysis, candidate leaders are merged to form a leader set. Further, the traffic network is partitioned based on the leader set, forming multiple control groups consisting of the leader set and follower sets. Local response strategies within the groups are learned through follower deep Q-networks, regional coordination strategies are learned through leader deep Q-networks, and candidate leader strategies are evaluated and accepted using a Q-value-based gating mechanism, thereby achieving hierarchical collaborative optimization of large-scale traffic signals.

[0024] The traffic signal control method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the traffic signal control method in this application is described.

[0025] The traffic signal control method provided in this application relates to the field of intelligent traffic control technology. The traffic signal control method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the traffic signal control method, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0028] Figure 1 This is an optional flowchart of the traffic signal control method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S100 to S500.

[0029] Step S100: Obtain multiple candidate intersections of the traffic network.

[0030] In this embodiment, multiple candidate intersections in the traffic network to be controlled are first obtained.

[0031] A transportation network refers to a road network consisting of multiple intersections and road segments connecting these intersections. Specifically, this transportation network can be a local road network within a city area, or a city-wide or intercity road network. For example, such a network could be the road network of a city's central business district, including twenty intersections and the various main and secondary arterial roads connecting these intersections; or it could be a city-wide road network covering expressways and surface roads, including hundreds of intersections.

[0032] A candidate intersection refers to a pre-determined intersection in a transportation network that has the potential to serve as a regional coordination node. Specifically, this candidate intersection can be one provided by a pre-configured network, one designated by the user based on practical management experience or specific control requirements, or one obtained through external traffic engineering analysis. External traffic engineering analysis can comprehensively consider factors such as intersection traffic flow, saturation, queue length, delay, congestion propagation risk, road network location, and the degree of impact on surrounding intersections. For example, an intersection where two main urban roads intersect, with an average daily traffic flow exceeding 50,000 vehicles, and located at a critical node connecting two commercial centers, can be identified as a candidate intersection through external traffic engineering analysis. Similarly, traffic management departments may directly designate several key intersections on a ring road as candidate intersections based on the needs of key control areas.

[0033] It should be noted that the candidate intersection set is a pre-given input, and this embodiment does not limit the unique screening method or fixed judgment threshold. In practical applications, it can be flexibly determined according to the road network structure, traffic operation status, and control requirements. The acquisition of multiple candidate intersections in the traffic network can be achieved by the electronic device reading a pre-configured candidate intersection list from local storage, by the electronic device receiving candidate intersection identification information input by the user, or by the electronic device receiving candidate intersection analysis results sent by an external traffic engineering analysis system.

[0034] While acquiring candidate intersections, the system can also obtain the traffic network topology, intersection set, road connectivity, signal phase set, and traffic state data. The traffic network topology refers to the spatial structure of the urban road network comprised of intersection nodes and the roads connecting them. Road connectivity includes information such as the starting and ending points of roads, road class, number of lanes, direction of travel, and turning restrictions. The signal phase set refers to the possible combinations of signal light colors that can be executed at each intersection; different intersections may have different signal phase designs based on their geometry and traffic flow characteristics. Traffic state data includes vehicle queue length, vehicle waiting time, current signal phase, phase duration, traffic pressure information, and traffic demand information. This data can be collected in real time by roadside detectors or generated by a traffic simulation platform.

[0035] Step S200: Based on the spatial proximity relationship between multiple candidate intersections, merge candidate intersections that meet the preset merging conditions to obtain at least one leader set, the leader set including at least one candidate intersection.

[0036] In this embodiment, based on the spatial proximity between candidate intersections, the main traffic direction, traffic demand intensity, and road network coupling relationship, a merging operation is performed on candidate intersections that meet the conditions to form one or more leader sets.

[0037] Spatial proximity refers to the degree of spatial proximity between candidate intersections and the resulting traffic association. Specifically, this spatial proximity can be characterized by one or more of the following: spatial distance between candidate intersections, road connectivity, key direction of movement relationship, and traffic coupling strength. For example, if two candidate intersections are 300 meters apart and directly connected by a main urban road, they can be considered to have a strong spatial proximity. Similarly, if two candidate intersections are 800 meters apart but located on the same traffic corridor with high traffic coupling strength, they can also be considered to have a spatial proximity.

[0038] Preset merging conditions refer to the criteria used to determine whether two candidate intersections should be merged into the same leader set. Specifically, these preset merging conditions can include one or more of the following: spatial distance conditions, traffic coupling strength conditions, key direction of movement conditions, and traffic corridor conditions. For example, a preset merging condition could be that the spatial distance between two candidate intersections is less than a preset distance threshold, which can be set according to the actual road network conditions, such as a value of 0.5 miles. Another example is that a preset merging condition could be that the traffic coupling strength between two candidate intersections is greater than a preset coupling threshold, which can be an adaptively adjusted value, i.e., dynamically determined based on the traffic characteristics of different road networks.

[0039] Merging refers to the operation of grouping two or more candidate intersections that meet preset merging conditions into the same set. Specifically, this merging process can adopt a transitive merging method, that is, when candidate intersection A and candidate intersection B meet the preset merging conditions, and candidate intersection A and candidate intersection C also meet the preset merging conditions, even if candidate intersection B and candidate intersection C do not meet the preset merging conditions, candidate intersection A, candidate intersection B, and candidate intersection C will still be merged into the same leader set. For example, if the spatial distance between candidate intersection A and candidate intersection B is 200 meters, meeting the preset merging conditions, and the spatial distance between candidate intersection A and candidate intersection C is 300 meters, also meeting the preset merging conditions, but the spatial distance between candidate intersection B and candidate intersection C is 600 meters, not meeting the preset merging conditions, then candidate intersection A, candidate intersection B, and candidate intersection C will still be merged into the same leader set.

[0040] A leader set is a set formed by merging one or more candidate intersections. This leader set serves as a seed node for traffic network zoning and hierarchical coordination control. Specifically, candidate intersections in a leader set have strong spatial proximity and traffic association relationships. Treating this leader set as a whole for regional coordination decisions can avoid repeatedly setting adjacent or strongly associated intersections as independent high-level coordination units, thereby reducing control structure redundancy and improving regional coordination capabilities. For example, a traffic network may contain ten candidate intersections. After merging, three leader sets are obtained: the first leader set contains three candidate intersections, the second leader set contains two candidate intersections, and the third leader set contains five candidate intersections.

[0041] Step S300: Based on at least one leader set, the traffic network is partitioned to obtain at least one control group. Each control group includes a leader set and a follower set. The follower set includes ordinary intersections in the traffic network, which are intersections in the traffic network other than the candidate intersections in the leader set.

[0042] In this embodiment, after obtaining the leader sets, the entire traffic network is partitioned using each leader set as the core, thereby obtaining one or more control groups. Partitioning refers to the process of assigning all intersections in the traffic network to different control groups. Specifically, the partitioning uses the leader sets as seed nodes and assigns ordinary intersections that are not set as leaders to the corresponding control groups. For example, if a traffic network includes fifty intersections, of which ten intersections are candidate intersections and have been merged into three leader sets, the partitioning process will assign the remaining forty ordinary intersections to control groups centered around the three leader sets.

[0043] A control group is a traffic control unit consisting of a set of leaders and a set of followers. Specifically, each control group includes a set of leaders and a set of followers corresponding to that set of leaders. Intersections within the same control group have strong traffic connections, while traffic coupling between different control groups is relatively weak. For example, a control group may include a set of leaders consisting of two candidate intersections and a set of followers consisting of eight ordinary intersections. The ten intersections within this control group are spatially adjacent and their traffic flows are interconnected.

[0044] A follower set refers to a group of ordinary intersections that act as followers in a hierarchical control structure, executing local signal control responses within the group given a leader's action or strategy. Specifically, the ordinary intersections in the follower set generate signal phase control actions based on local traffic conditions and the corresponding leader's action, without directly participating in area-level coordination decisions. For example, a follower set might include five ordinary intersections. After receiving a coordination action from the leader set, each of these five intersections generates its own signal phase control action based on its local traffic conditions.

[0045] A common intersection refers to any intersection in a traffic network other than the candidate intersections in the leader set. Specifically, common intersections are subordinate to traffic control groups, and their signal control actions are guided by the coordination strategies of the leader set. For example, in a traffic network with fifty intersections, ten of which are candidate intersections and have been merged into the leader set, the remaining forty intersections are common intersections. These common intersections are assigned to the corresponding control groups during the zoning process and form a follower set.

[0046] It should be noted that the above partitioning process can determine the comprehensive allocation cost of each ordinary intersection under the constraints of intra-group connectivity, unique leader affiliation, non-overlapping intersections, and each ordinary intersection belonging to only one control group. The comprehensive allocation cost is determined based on the spatial compactness between the ordinary intersection and the leader set within the control group, the degree of traffic flow separation between the ordinary intersection and other intersections within the control group, the consistency of traffic load among intersections within the control group, and the balance of the number of intersections among control groups. Each ordinary intersection is assigned to the candidate control group that minimizes its comprehensive allocation cost, resulting in multiple control groups. Intra-group connectivity means that intersections within the same control group remain spatially connected, i.e., any two intersections within the same control group can reach each other via road segments within the group; unique leader affiliation means that each control group includes only one leader set; and non-overlapping intersections mean that each ordinary intersection is assigned to only one control group, preventing the same ordinary intersection from belonging to multiple control groups simultaneously.

[0047] Step S400: Construct a deep reinforcement learning network based on at least one control group, and train the deep reinforcement learning network to learn a first coordination strategy and a second coordination strategy. The first coordination strategy refers to the coordination strategy for controlling candidate crossroads in each leader set, and the second coordination strategy refers to the coordination strategy for controlling ordinary crossroads in the follower set, thereby obtaining a control network.

[0048] In this embodiment, after the traffic network is partitioned into multiple control groups, a deep reinforcement learning network is constructed based on these control groups and trained. The goal of the training is to enable the deep reinforcement learning network to learn two coordination strategies: a first coordination strategy and a second coordination strategy.

[0049] Deep reinforcement learning networks are neural network models used to learn traffic signal control strategies. Using deep reinforcement learning for traffic signal control offers several advantages. Deep neural networks possess powerful nonlinear fitting capabilities, enabling them to extract effective features from high-dimensional, dynamically changing traffic state data and learn complex signal control strategies. Compared to traditional rule-based or mathematical model-based methods, deep reinforcement learning methods do not require manually designed complex control rules or precise traffic flow mathematical models. They can autonomously learn optimal control strategies through interaction with the environment, making them more suitable for complex and heterogeneous urban road network environments. Specifically, this deep reinforcement learning network can include a leader deep network for learning the coordination strategy of the leader set and a follower deep network for learning the local response strategy of the follower set. For example, this deep reinforcement learning network can employ a deep Q-network structure, which takes traffic states, leader actions, follower actions, and reward signals as input or training information, and outputs a Q-value for selecting signal phase control actions.

[0050] The first coordination strategy refers to the coordination strategy used to control candidate intersections in each leader set. Specifically, the first coordination strategy is used to generate group-level or area-level coordination actions based on the enhanced traffic conditions; that is, the leader set outputs signal phase control actions for the leader intersections according to the first coordination strategy. For example, if a leader set includes two candidate intersections, the first coordination strategy determines the coordination actions that the two candidate intersections should perform at the current decision time based on the current enhanced traffic conditions. The coordination actions may include the first candidate intersection implementing a green light for north-south traffic and the second candidate intersection implementing a green light for east-west traffic.

[0051] The second coordination strategy refers to a coordination strategy for controlling ordinary intersections within a follower set. Specifically, this second coordination strategy is used to generate signal phase control actions for ordinary intersections within the group based on local traffic conditions, given a leader's action or leader strategy. For example, if a follower set includes five ordinary intersections, after the leader set determines the current coordination action based on the current enhanced traffic conditions according to the first coordination strategy, the second coordination strategy outputs the signal phase control action that each of the five ordinary intersections should perform at the current time, based on the local traffic conditions of each intersection.

[0052] The aforementioned training refers to the process of collecting state transition samples through interaction with the environment and updating the parameters of the deep reinforcement learning network using methods such as experience replay, control networks, and temporal differential learning. Specifically, this training process can adopt a phased training approach, including a follower learning sub-stage and a leader learning sub-stage. In the follower learning sub-stage, the parameters of the first coordination policy of the leader set are fixed, and the leader set determines the leader action based on the current traffic state according to the first coordination policy. During this period, only the follower deep network is updated to learn the second coordination policy. In the leader learning sub-stage, the learned follower responses are fixed, candidate leader policies are generated, and the leader deep network is updated through a gating acceptance mechanism to learn the first coordination policy.

[0053] Specifically, the first coordination strategy is determined by the leader's deep network by comparing the global value assessment results of the current coordination strategy and candidate coordination decisions in the enhanced traffic state space, assuming that the local response strategies of the follower set have been updated and fixed. Only when a candidate strategy is not inferior to the current strategy in all observable enhanced states, and is strictly superior to the current strategy in at least one state, is the candidate strategy accepted and becomes the updated first coordination strategy, thus ensuring the stable improvement of the leader's strategy. The second coordination strategy, under the condition that the leader set's first coordination strategy is fixed, involves each follower set continuously interacting with the traffic environment, collecting state transition data and storing it in an experience replay pool. A temporal difference objective is constructed using a preset objective function, and the deep network parameters are repeatedly adjusted with the optimization direction of minimizing the deviation between this temporal difference objective and the current follower action value. Ultimately, this enables the follower set to take the signal phase action that maximizes its cumulative local gain based on the leader's action determined in the current state according to the fixed first coordination strategy and the local traffic state output.

[0054] A control network refers to a deep reinforcement learning network obtained after training. This control network includes a trained leader deep network and a trained follower deep network. Specifically, this control network can be deployed in a real-time traffic signal control environment to generate signal phases at each intersection and determine actions based on the current traffic conditions. For example, the traffic network is divided into multiple control groups. The trained control network includes three leader deep networks and three follower deep networks corresponding to the three control groups, with each control group having one leader deep network and one follower deep network. During the real-time control phase, this control network outputs signal phase control actions for each intersection in the three control groups based on the current traffic conditions.

[0055] Step S500: The control network makes decisions based on the real-time traffic status of the traffic network to obtain the signal control actions for each intersection in the traffic network.

[0056] In this embodiment, after the deep reinforcement learning network is trained and a control network is obtained, the traffic signal control device deploys the control network into the actual traffic signal control environment and uses the control network to process and make decisions on the real-time traffic status of the traffic network, thereby obtaining the signal control actions of each intersection in the traffic network.

[0057] Real-time traffic status refers to the operational status data of the traffic network at the current moment. Specifically, this real-time traffic status may include the queue length of vehicles in each direction at each intersection, the waiting time of vehicles in each direction at each intersection, the currently executing signal phase, and its duration. For example, the real-time traffic status of a certain intersection at the current moment may include a queue length of 50 meters for the northbound direction, a waiting time of 60 seconds, and the currently executing signal phase being a green light for the north-south direction that has lasted for 20 seconds.

[0058] The aforementioned decision-making process refers to the process of inputting real-time traffic conditions into a control network, which then outputs signal control actions for each intersection. Specifically, this decision-making process includes inputting real-time traffic conditions into a leader deep network to obtain candidate coordinated actions; generating candidate signal phase actions through a follower deep network based on the candidate coordinated actions and the traffic conditions of the follower set; evaluating the value of the candidate coordinated actions through the leader deep network; selecting the candidate coordinated action with the highest value as the target coordinated action; and using the candidate signal phase actions of the follower set corresponding to the target coordinated action as the target signal phase action.

[0059] Signal control actions refer to the phase control actions that traffic lights at an intersection should perform at the current moment. Specifically, this signal control action can be the selection of a traffic light phase, such as selecting a green light phase for the north-south direction or a green light phase for the east-west direction. In the reinforcement learning framework, signal control actions are interpreted as actions. For example, if the current signal control action at an intersection is a green light for the north-south direction, and this action is the same as the signal control action at the previous moment, then the current signal phase remains unchanged; if this action is different from the signal control action at the previous moment, then a signal switch is performed, and the switch process must satisfy transition phase constraints such as yellow light phase and all-red phase.

[0060] It should be noted that when the output signal phase control action is performed, preset signal safety constraints can be further satisfied. These signal safety constraints include one or more of the following: yellow light phase constraint, all-red phase constraint, minimum phase duration constraint, and executable phase set constraint. Specifically, the yellow light phase and all-red phase refer to the transition phase between two different green light phases, such as a three-second yellow light followed by a one-second all-red phase. The minimum phase duration means that the duration of the green light phase needs to consider the needs of pedestrians crossing the street; for example, the minimum green light time should not be less than fifteen seconds. The executable phase set constraint means that the set of phases that can be executed at the intersection depends on the intersection's topology; different intersections may have different phase designs.

[0061] This embodiment divides a large-scale traffic network into multiple control groups with clear hierarchical coordination relationships by merging candidate intersections and leader-guided partitioning. This effectively reduces the decision-making dimension and training complexity of large-scale traffic signal joint control, and improves the scalability of the method in multi-intersection road networks. Through a hierarchical structure of multiple leaders and multiple followers, the regional coordination capability of the leader set is combined with the local response capability of the follower set, enabling the traffic signal control process to take into account both local traffic conditions and regional traffic coupling relationships, thereby improving the cross-regional collaborative control effect. By using deep reinforcement learning to learn the first coordination strategy and the second coordination strategy respectively, it can handle high-dimensional and dynamically changing traffic state information. Compared with methods that rely on manual rules or discrete state tables, it is more suitable for complex heterogeneous road networks and large-scale traffic signal control scenarios.

[0062] In some embodiments, the preset merging conditions include at least one of the following: The spatial distance between two candidate intersections is less than a preset distance threshold; The traffic coupling strength between two candidate intersections is greater than a preset coupling threshold. The traffic coupling strength is obtained based on the traffic demand, number of vehicles, and road segment length between the two intersections within a preset time period. The key movement directions between two candidate intersections are the same or related. The key movement direction is the vehicle movement direction that has the greatest impact on the signal timing or capacity of the intersection. The two candidate intersections are located in the same traffic corridor.

[0063] In transportation networks, repeatedly designating adjacent or strongly related candidate intersections as independent high-level coordination units increases redundancy in the control structure and reduces the effectiveness of regional coordination. Therefore, it is necessary to merge candidate intersections that meet the criteria to form a leader set.

[0064] In this embodiment, spatial distance refers to the straight-line distance or path distance along the road network between two candidate intersections in geographic space. Specifically, this spatial distance can be calculated using the latitude and longitude coordinates of the two candidate intersections, or it can be calculated using the shortest path length in the road network. A preset distance threshold is a critical value used to determine whether two candidate intersections are spatially close enough. Specifically, this preset distance threshold can be set according to the actual road network conditions, such as 0.5 miles. When the spatial distance between two candidate intersections is less than this threshold, it indicates that they are geographically close and have a reasonable basis for merging.

[0065] Traffic coupling intensity refers to the degree of mutual influence between traffic flows at two intersections. Specifically, this traffic coupling intensity can be calculated based on the traffic demand, number of vehicles, and road segment length between the two intersections within a preset time period. For example, this traffic coupling intensity can be calculated using formula (1): (1); Where, d i→j This represents the traffic demand or number of vehicles traveling from intersection i to intersection j within a time window T. i→j This represents the length of the road segment in the corresponding direction between the two intersections. Therefore, the first term represents the traffic intensity per unit time and unit distance from intersection i to intersection j. The second term represents the traffic intensity in the opposite direction. Adding the two terms together gives the overall traffic coupling intensity between intersection i and intersection j. The magnitude of the traffic coupling intensity reflects the degree of interdependence of traffic flows between the two intersections.

[0066] A preset coupling threshold is a critical value used to determine whether the traffic coupling strength between two candidate intersections is sufficiently strong. Specifically, this preset coupling threshold can be an adaptively adjusted value, dynamically determined based on the traffic characteristics of different road networks, rather than using a fixed empirical value. When the calculated traffic coupling strength is greater than the preset coupling threshold, it indicates a strong traffic association between the two candidate intersections, and they should be merged into the same leader set. For example, in a city's central business district road network, due to high traffic volume, the preset coupling threshold can be set to a higher value, such as 0.5 vehicles per minute per meter; while in a city's suburban road network, due to lower traffic volume, the preset coupling threshold can be set to a lower value, such as 0.1 vehicles per minute per meter.

[0067] The critical direction of movement refers to the direction of vehicle movement at an intersection that has the greatest impact on signal timing or capacity. Specifically, this critical direction of movement can be determined by analyzing factors such as traffic flow, lane function allocation, and signal phase design at each approach of the intersection. For example, if the east-west direction is the main arterial direction of an intersection, and the through traffic flow in the east-west direction accounts for 70% of the total traffic flow at the intersection, then the critical direction of movement for this intersection is the east-west through traffic. Similarly, if the left-turn traffic flow at the north approach of an intersection is large and this left-turn direction has a significant impact on the intersection's capacity, then the critical direction of movement for this intersection is the left-turn at the north approach.

[0068] Having the same critical direction of movement means that the critical directions of movement of the two candidate intersections are consistent. For example, if the critical direction of movement for candidate intersection A is east-west straight traffic, and the critical direction of movement for candidate intersection B is also east-west straight traffic, then the critical directions of movement for these two candidate intersections are determined to be the same. Having related critical directions of movement means that the critical directions of movement for the two candidate intersections are not completely identical but are related. For example, if the critical direction of movement for candidate intersection A is east-west straight traffic, and the critical direction of movement for candidate intersection B is left-turning from the east entrance, since vehicles turning left from the east entrance eventually merge into the east-west straight traffic flow, then the critical directions of movement for these two candidate intersections are determined to be related. When the critical directions of movement for two candidate intersections are the same or related, the two candidate intersections are determined to meet the critical direction of movement condition.

[0069] A traffic corridor is a traffic channel consisting of one or more continuous roads that handles a large volume of traffic. Specifically, intersections within the same traffic corridor are continuous in the direction of traffic flow, with traffic flowing along the corridor. For example, a main road in a city runs east-west through the entire city, and all intersections on this main road are located within the same traffic corridor; similarly, an expressway in a city extends from east to west, and the numerous ramp intersections along this expressway are all located within the same traffic corridor. When two candidate intersections are located within the same traffic corridor, these two candidate intersections are deemed to meet the traffic corridor condition.

[0070] It should be noted that the aforementioned preset merging conditions may include one or more of the four conditions mentioned above. When the preset merging conditions include multiple conditions, a merging method can be adopted where merging occurs only when any one condition is met, or a merging method can be adopted where merging only occurs when all conditions are met. The specific settings can be based on the actual road network characteristics and control requirements. For example, the preset merging conditions can be set to either a spatial distance less than a preset distance threshold or a traffic coupling strength greater than a preset coupling threshold. That is, if two candidate intersections meet either the spatial distance condition or the traffic coupling strength condition, the two candidate intersections will be merged.

[0071] This embodiment sets preset merging conditions, including at least one of spatial distance conditions, traffic coupling strength conditions, key movement direction conditions, and traffic corridor conditions. This allows for a comprehensive assessment of the correlation between candidate intersections from multiple dimensions, ensuring that candidate intersections merged into the same leader set have strong traffic correlation. This provides a reasonable leader set basis for subsequent zoning and hierarchical coordination control, thereby improving the pertinence and effectiveness of regional coordination control.

[0072] In some embodiments, step S300 may include, but is not limited to, steps S310 to S330: Step S310: Using each leader set as the core point of each partition, traverse the ordinary intersections; Step S320: Under the constraints that each ordinary intersection is assigned to only one control group, the intersections within each control group maintain spatial connectivity, and each control group includes only one set of leaders, determine the comprehensive allocation cost of each ordinary intersection. The comprehensive allocation cost is determined based on the spatial compactness between the ordinary intersection and the set of leaders within the control group, the degree of traffic flow separation between the ordinary intersection and other intersections within the control group, the consistency of traffic load among each intersection within the control group, and the degree of balance in the number of intersections among each control group. Step S330: Assign each ordinary intersection to the candidate control group that minimizes the overall allocation cost of the ordinary intersection, resulting in multiple control groups.

[0073] In this embodiment, each leader set is used as the core point or seed point of the corresponding partition, and then all ordinary intersections in the traffic network are traversed sequentially. The core point is a reference point used to identify the center of the control group during the partitioning process. Specifically, each control group uses one leader set as its core point, and ordinary intersections are assigned around this core point. For example, if a traffic network includes three leader sets, then these three leader sets are used as the core points of three control groups, and all ordinary intersections in the traffic network are traversed to determine which control group each ordinary intersection should be assigned to.

[0074] Several constraints must be met when assigning control groups to ordinary intersections. First, each ordinary intersection can only be assigned to one control group and cannot belong to multiple control groups simultaneously, ensuring the mutual exclusivity of the zones. Second, intersections within each control group must maintain spatial connectivity; that is, any two intersections within the group must be reachable from each other via roads within the group, and enclaves cannot exist, ensuring the feasibility of coordinated control within the group. Third, each control group can contain only one set of leaders to ensure the clarity of the hierarchical control structure.

[0075] Under the premise of satisfying the above constraints, it is necessary to determine the comprehensive allocation cost of each ordinary intersection to each candidate control group. The comprehensive allocation cost refers to the overall cost incurred by an ordinary intersection being assigned to a candidate control group. It is an indicator that comprehensively measures multiple factors; the smaller the value, the more reasonable it is to assign the ordinary intersection to that control group. The comprehensive allocation cost is determined based on four factors: First, spatial compactness, i.e., the compactness of the spatial distance between the ordinary intersection and the leader set within the control group; the smaller the spatial distance, the higher the spatial compactness and the lower the comprehensive allocation cost. Second, the degree of traffic flow segmentation, i.e., the degree of traffic flow segmentation between the ordinary intersection and other intersections within the control group; the smaller the degree of traffic flow segmentation, the lower the comprehensive allocation cost. Third, the degree of traffic load consistency, i.e., the consistency of traffic load among each intersection within the control group; the higher the consistency of traffic load, the lower the comprehensive allocation cost. Fourth, the degree of balance in the number of intersections, i.e., the balance in the number of intersections among each control group; the more balanced the number of intersections among control groups, the lower the comprehensive allocation cost.

[0076] Specifically, the aforementioned comprehensive allocation cost can be calculated using a comprehensive zoning objective function, which includes a term for intra-group spatial compactness, an inter-group coupling segmentation, a term for traffic load homogeneity, and a term for group size balance. The comprehensive zoning objective function is shown in formula (2): (2); in, This indicates the compactness of the space within the group. This represents the coupled weighted normalized cut index. Indicators representing the homogeneity of traffic load This represents the group size balance index. In actual calculations, each objective item can be normalized to avoid different dimensions or numerical scales causing one objective item to dominate the overall objective. , , , These represent the weights of each part. This is because the magnitudes and ranges of each part are inconsistent, requiring normalization to an appropriate range. It is also used to adjust the importance of each part. Therefore, these are adaptively adjusted values.

[0077] For each ordinary intersection, the overall allocation cost of assigning it to each candidate control group is compared, and the control group with the lowest overall allocation cost is selected as the final assignment for that ordinary intersection. After all ordinary intersections have been assigned according to this principle, multiple traffic control groups are obtained. Through the above traversal and allocation process, multiple control groups can be obtained, each control group consisting of a leader set and a follower set.

[0078] This embodiment uses each leader set as the core point of each partition to traverse ordinary intersections. Under the condition of satisfying constraints, it determines the comprehensive allocation cost of each ordinary intersection and assigns each ordinary intersection to the candidate control group that minimizes the comprehensive allocation cost. This can make the intersections within the same control group have strong traffic correlation, while reducing unnecessary coupling between different control groups. This provides a reasonable partitioning basis for subsequent hierarchical coordinated control and improves the pertinence of traffic signal control and the regional coordination effect.

[0079] In some embodiments, step S400 may include, but is not limited to, steps S410 to S430: Step S410, during the learning phase of the second coordination strategy, fix the first coordination strategy of the leader set and update the second coordination strategy of the follower set; Step S420: During the learning phase of the first coordination strategy, candidate coordination strategies are generated based on the updated second coordination strategy. Step S430: Update the first coordination strategy of the leader set based on the value comparison results of the candidate coordination strategy and the current coordination strategy.

[0080] In this embodiment, the learning phase of the second coordination strategy refers to the stage during the training process used to learn the local response strategy of the follower set. Specifically, in this learning phase, the first coordination strategy of the leader set is fixed, that is, the coordination strategy of the leader set is maintained, and only the second coordination strategy of the follower set is updated. For example, with the first coordination strategy remaining unchanged, the leader set can determine the north-south traffic phase based on the traffic conditions at the current decision time, and determine the east-west traffic phase based on the changed traffic conditions at the next decision time. Under this fixed coordination strategy condition, the follower set learns and generates signal phase control actions for each ordinary intersection based on the local traffic conditions.

[0081] The first coordination strategy of fixing the leader set refers to keeping the coordination strategy parameters of the leader set unchanged. Specifically, this fixing method is complete fixing, that is, the coordination strategy parameters of the leader set remain unchanged throughout the learning phase of the second coordination strategy.

[0082] The second coordination strategy for updating the follower set involves collecting state transition samples through interaction with the environment and updating the parameters of the follower deep network using methods such as experience replay, control networks, and temporal difference learning. Specifically, this update process is achieved by minimizing the loss function of the follower deep network, which is the mean square error between the Q-value output by the follower deep network and the temporal difference objective.

[0083] For example, in the initial training phase, the leader policy adopts a random policy or a rule-based initial policy. In the follower learning sub-phase, the leader policy remains unchanged, and the follower deep network interacts with the traffic environment, collecting state transition samples and using these samples to update the parameters of the follower network. After several iterations, the follower set gradually learns the optimal response policy under the current leader policy.

[0084] In this embodiment, the learning phase of the first coordination strategy refers to the phase during the training process used to learn the coordination strategy of the leader set. Specifically, in this learning phase, the updated second coordination strategy is fixed or approximately fixed, that is, the local response strategy of the follower set is kept unchanged, and candidate leader strategies are generated.

[0085] Given a fixed follower response, candidate strategies are generated to replace the current leader coordination strategy. Specifically, these candidate strategies can be generated by perturbing, mutating, or adjusting the parameters of the current leader coordination strategy, or through a strategy optimization method, or based on experience or heuristic rules. The purpose of generating candidate coordination strategies is to explore potentially better leader strategies for comparison and selection with the current strategy.

[0086] For example, after the follower policy is updated, the leader learning sub-stage begins. At this point, based on the current leader policy, several candidate coordination policies are generated by adding random noise or perturbing the parameters. These candidate policies differ from the current policy but are not too far off, allowing for the search for a better policy within the neighborhood of the current policy.

[0087] In this embodiment, the value comparison result refers to the comparison of the action values ​​of the candidate coordination strategy and the current coordination strategy under each enhancement state. Specifically, this value comparison result is obtained by estimating the action values ​​of the candidate coordination strategy and the current coordination strategy under each enhancement state through the leader deep network. The candidate coordination strategy and the current coordination strategy are compared in value, and the decision on whether to update the first coordination strategy of the leader set is based on the comparison result. If the value of the candidate coordination strategy is better than the current coordination strategy, the candidate coordination strategy is used to update the current strategy; if the value of the candidate coordination strategy is not better than the current strategy, the current strategy remains unchanged. This value comparison-based strategy update method ensures that each strategy update does not lead to performance degradation, thereby improving the stability of the training process.

[0088] For example, a leader deep network is used to estimate the action values ​​of the current coordination strategy and candidate coordination strategies under various traffic conditions. If the value of the candidate strategy is not lower than that of the current strategy in all evaluated states, and is higher than that of the current strategy in at least some states, the candidate strategy is considered superior and is adopted as the new current strategy. Conversely, if the value of the candidate strategy is lower than that of the current strategy in some states, the candidate strategy is not accepted, and the current strategy remains unchanged.

[0089] This embodiment achieves a phased training approach by fixing the first coordination strategy of the leader set and updating the second coordination strategy of the follower set during the learning phase of the second coordination strategy. During the learning phase of the first coordination strategy, candidate coordination strategies are generated based on the updated second coordination strategy, and the first coordination strategy of the leader set is updated based on the value comparison results between the candidate coordination strategies and the current coordination strategy. This avoids the training instability caused by updating the leader strategy and the follower strategy simultaneously. At the same time, the strategy update mechanism based on value comparison reduces the performance degradation of strategy updates and improves the stability of the hierarchical reinforcement learning process.

[0090] In some embodiments, step S410 may include, but is not limited to, steps S411 to S414: Step S411: Under the first coordination strategy of the fixed leader set, obtain the state transition data generated by the signal phase action of each follower set executing the second coordination strategy; Step S412: Store the state transition data in the experience replay pool; Step S413: Sample historical state transition data from the experience replay pool, and calculate using a preset target value function to obtain the temporal difference target of each follower set under the given local observation state, the first coordination strategy of the leader set, and its own action conditions. Step S414: Update the second coordination policy of the follower set by minimizing the deviation between the temporal difference objective and the value of the current follower action.

[0091] In this embodiment, during the follower learning sub-phase, the first coordination strategy of the leader set is first fixed, keeping its parameters unchanged. Then, the follower set interacts with the traffic environment according to the current second coordination strategy, performs corresponding signal phase actions, and records the state transition data generated during the interaction.

[0092] State transition data refers to the data generated when the traffic environment state changes after a group of followers performs signal phase actions. Specifically, this state transition data can include information such as the local observation state before the transition, the leader's actions, the followers' actions, the local observation state after the transition, and the follower group's rewards. State transition data can be acquired through interaction between electronic devices and a traffic simulation environment or an actual traffic signal control system.

[0093] For example, after fixing the leader strategy, the follower deep network selects and executes the appropriate signal phase action based on the current local traffic state and the leader's actions. After executing this action, the state of the traffic environment changes, and indicators such as vehicle queue length and waiting time will change accordingly. At the same time, the environment will provide a reward signal, which reflects the control effect of the action, such as the reduction in queue length or the increase in the number of vehicles passing through. Recording the four-tuple of current state, executed action, reward value, and next state constitutes a state transition data.

[0094] In this embodiment, the experience replay pool refers to the data structure used to store state transition data. Specifically, the experience replay pool can adopt a first-in, first-out (FIFO) queue structure. When the amount of data in the experience replay pool reaches its upper limit, newly stored state transition data will replace the oldest stored state transition data. By storing state transition data in the experience replay pool, the temporal correlation between state transition data can be broken, making the state transition data sampled from the experience replay pool more independent and identically distributed, thereby improving training stability.

[0095] In this embodiment, a batch of historical state transition data is randomly sampled from the experience replay pool, and then these samples are calculated using a preset target value function to obtain the temporal difference target. Specifically, sampling can be performed using a uniform random sampling method or a priority sampling method, that is, sampling is performed according to the importance weight of the state transition data.

[0096] The preset target value function is the function used to calculate the temporal difference objective. Specifically, this target value function is constructed based on a follower deep network. The parameters of the follower deep network are obtained by periodically copying the current follower deep network parameters and remain unchanged over several training steps, thus constructing a stable temporal difference objective.

[0097] A temporal difference objective refers to the target Q-value of a set of followers given their local observed states, the first coordination policy of the leader set, and their own actions. It consists of the immediate reward gained at the current time step plus the discounted value for the next state. Calculating a temporal difference objective requires considering the given conditions, including the local observed states of the follower set, the first coordination policy of the leader set, and the followers' own actions. Since the leader policy is fixed, the leader's action at the next time step can be determined by the fixed leader policy based on the next state; therefore, it can be incorporated into the calculation of the temporal difference objective.

[0098] In this embodiment, the value estimate of the current follower value function for the current state-action pair is calculated. Then, the deviation between this value estimate and the temporal difference objective is calculated, typically using mean squared error as the loss function. This loss function is minimized using optimization methods such as gradient descent, thereby updating the parameters of the follower deep network and subsequently updating the second coordination policy of the follower set.

[0099] This embodiment acquires state transition data under a first coordination strategy with a fixed set of leaders, stores the state transition data in an experience replay pool, samples historical state transition data from the experience replay pool, calculates a temporal difference objective using a preset target value function, and updates the second coordination strategy of the follower set by the deviation between the minimum temporal difference objective and the value of the current follower action. This achieves stable learning of the local response strategy of the follower set. By breaking the temporal correlation of data and constructing a stable temporal difference objective through the experience replay pool and control network mechanism, the stability of the follower deep network training is improved.

[0100] In some embodiments, step S430 may include, but is not limited to, steps S431 to S433: Step S431: With the updated second coordination strategy fixed, the first global value evaluation result corresponding to the current coordination strategy and the second global value evaluation result corresponding to the candidate coordination strategy are obtained through a deep reinforcement learning network for each enhanced state. The enhanced state includes: the maximum pressure value of each signal phase level, the signal phase currently being executed at each intersection, the duration of the current phase, the maximum vehicle waiting time for each entrance direction, and the vehicle queue length for each entrance direction. Step S432: If the value comparison result meets the preset conditions, the candidate coordination strategy is used as the current coordination strategy, and the first coordination strategy of the leader set is updated according to the candidate coordination strategy. The preset conditions are that the second global value assessment result is greater than or equal to the first global value assessment result in each enhancement state, and is greater than the first global value assessment result in at least one enhancement state. Step S433: If the value comparison result does not meet the preset conditions, then the current coordination strategy remains unchanged.

[0101] In this embodiment, during the leader learning sub-stage, the second coordination policy, which was updated in the previous stage, is first fixed. Then, a deep reinforcement learning network is used to calculate the global value evaluation results of the current coordination policy and candidate coordination policies in each augmentation state. The global value evaluation result refers to the expected value of the cumulative reward that can be obtained by adopting a certain policy in the long run, and it reflects the quality of the policy.

[0102] The augmented state refers to the state used to construct an approximate local Markov state representation for each group of traffic lights. Specifically, when only ordinary local states are used, the state transition process formed by each control group based on local observations usually does not satisfy strict local Markov property. By introducing augmented states, the local Markov property of each group of traffic lights can be approximately recovered, thereby ensuring the effectiveness of deep reinforcement learning training.

[0103] The enhanced status includes the following information: First, the maximum pressure value for each signal phase level. The pressure value can usually be measured by the queue length or number of waiting vehicles in that phase direction. The maximum pressure value reflects the situation of the phase with the most intense traffic demand. Second, the signal phase currently in operation at each intersection, i.e., which direction's signal light is currently green. Third, the duration of the current phase, i.e., how long the current green light phase has been in operation. Fourth, the maximum vehicle waiting time for each entrance direction, i.e., the waiting time of the longest-waiting vehicle in each entrance direction. Fifth, the queue length for each entrance direction, i.e., the number of vehicles queuing and waiting in each entrance direction.

[0104] Specifically, after fixing the follower strategy, several representative augmentation states are selected as evaluation states. For each evaluation state, the current coordination strategy and candidate coordination strategies are input, and the corresponding global value estimate is calculated using a leader deep network. The value estimate corresponding to the current coordination strategy is called the first global value evaluation result, and the value estimate corresponding to the candidate coordination strategy is called the second global value evaluation result. By comparing these two results, it can be determined whether the candidate strategy is superior to the current strategy.

[0105] The first global value assessment result refers to the leader depth Q-value induced by the current coordination strategy under various enhancement states. Specifically, this first global value assessment result is calculated using a leader depth network under enhancement states, the joint actions of leaders induced by the current coordination strategy, and the joint responses of followers. The second global value assessment result refers to the leader depth Q-value induced by candidate coordination strategies under various enhancement states. Specifically, this second global value assessment result is calculated using a leader depth network under enhancement states, the joint actions of leaders induced by candidate coordination strategies, and the joint responses of followers.

[0106] In this embodiment, the value assessment results of the candidate coordination strategy and the current coordination strategy under each enhancement state are compared. If the comparison result meets preset conditions, the candidate coordination strategy is accepted, and the current strategy is updated using the candidate strategy. The preset conditions include two aspects: first, the value assessment result of the candidate strategy under each enhancement state is greater than or equal to the value assessment result of the current strategy, that is, the candidate strategy is no worse than the current strategy under any state; second, the value assessment result of the candidate strategy under at least one enhancement state is strictly greater than the value assessment result of the current strategy, that is, the candidate strategy is better than the current strategy under at least some states. Only when both conditions are met simultaneously is the candidate strategy accepted.

[0107] If the value comparison between the candidate coordination strategy and the current coordination strategy does not meet the above preset conditions, the candidate strategy should be rejected, and the current coordination strategy should remain unchanged. There are two situations where the preset conditions are not met: first, the candidate strategy's value is lower than the current strategy in some enhanced states, indicating that the candidate strategy may perform worse in certain scenarios, and accepting it may lead to a decrease in overall performance; second, the candidate strategy's value is equal to the current strategy in all states, meaning there is no improvement, and updating the strategy in this case is meaningless. In either case, the candidate strategy should be rejected, and the current strategy should remain unchanged.

[0108] It should be noted that the above preset conditions can be further enhanced by a gating threshold. Specifically, the second global value assessment result of the candidate coordination strategy under all enhanced states must be greater than or equal to the first global value assessment result minus the gating threshold, and the second global value assessment result under at least one enhanced state must be greater than the first global value assessment result plus the gating threshold. This gating threshold is an adaptively adjusted value used to control the strictness of acceptance of candidate coordination strategies. For example, if the gating threshold is set to 0.05, then the second global value assessment result of the candidate coordination strategy under all enhanced states must be greater than or equal to the first global value assessment result minus 0.05, and the second global value assessment result under at least one enhanced state must be greater than the first global value assessment result plus 0.05 to be accepted.

[0109] This embodiment effectively ensures the safety and stability of leader policy updates by introducing a state-level value comparison-based gating acceptance mechanism. A candidate policy is only accepted if it is not inferior to the current policy in all evaluation states and shows improvement in at least some states. This mechanism avoids performance degradation in policy updates, reduces policy oscillations during training, and improves the stability of the hierarchical reinforcement learning process.

[0110] In some embodiments, the control network includes a leader deep network and a follower deep network, wherein the leader deep network is used to learn a first coordination strategy for the leader set, and the follower deep network is used to learn a second coordination strategy for the follower set. Step S500 may include, but is not limited to, steps S510 to S550: Step S510: Obtain the real-time traffic status of the traffic network at the current moment. The real-time traffic status includes the vehicle queue length for each approach direction at each intersection, the vehicle waiting time for each approach direction at each intersection, the currently executed signal phase and its duration. Step S520: Based on the set of executable signal phases of each candidate intersection in the leader set, obtain multiple candidate coordination actions; Step S530: Using a follower deep network, generate candidate signal phase actions for each follower set under each candidate coordinated action, based on each candidate coordinated action and the traffic state of each follower set. Step S540: Through the leader deep network, the value of each candidate coordination action is evaluated based on the real-time traffic status, each candidate coordination action, and the candidate signal phase actions of each follower set corresponding to each candidate coordination action. The candidate coordination action with the highest value is selected as the target coordination action at the current moment. The target coordination action is the signal phase control action of the intersection in the leader set. Step S550: The candidate signal phase action of each follower set corresponding to the target coordination action is taken as the target signal phase action of each follower set at the current time. The target signal phase action is the signal phase control action of the intersection in the follower set.

[0111] In this embodiment, the leader deep network refers to a deep Q-network used to learn the coordination strategy of the leader set. Specifically, the leader deep network takes the enhanced traffic state, the joint actions of the leaders, and the joint responses of the followers as inputs, and outputs a value estimate of the corresponding leader actions. The follower deep network refers to a deep Q-network used to learn the local response strategy of the follower set. Specifically, the follower deep network takes the observable traffic state of the follower group, the leader actions, and the follower actions as inputs, and outputs a value estimate of the corresponding follower actions.

[0112] In a specific embodiment, the follower deep Q network can be expressed as formula (3): (3); The leader deep Q network can be represented by formula (4): (4); in, This indicates that the follower group can observe the local state. Indicates enhanced network status. Indicates the actions of the group leader. This indicates the coordinated actions of the followers within the group. and These respectively represent coordinated actions by leaders and coordinated actions by followers at the network level. and represents the parameters of the follower deep Q-network and the leader deep Q-network, respectively.

[0113] The follower learning sub-stage includes: fixing the current leader's strategy; collecting state transition samples generated by the traffic environment and storing the state transition samples in the experience replay pool; constructing the follower temporal difference objective using the follower deep network; updating the parameters of the follower deep Q network by minimizing the loss function of the follower deep Q network, thereby updating the local cooperative response strategy of the follower set; the follower temporal difference objective is shown in formula (5): (5); The loss function of the follower deep Q network is shown in Equation (6): (6); in, This indicates a reward for the follower group. To enhance the discount factor of learning, Indicates the local observation state at the next moment. This indicates the next leader's action induced by the fixed leader strategy. For the actions of the followers. Additionally... The control network parameters are obtained by periodically copying the current Q network parameters and remain unchanged within several update steps. They are used to construct a stable temporal difference objective. E represents the mathematical expectation.

[0114] In this embodiment, during the real-time traffic signal control phase, real-time traffic status data of the traffic network at the current moment is first acquired. This data can be collected by devices such as vehicle detectors, cameras, and radars deployed at various intersections, or it can be obtained through floating car data, vehicle-to-infrastructure (V2I) data, etc. Real-time traffic status is the basis for signal control decisions, and its accuracy and timeliness directly affect the control effect.

[0115] Real-time traffic conditions specifically include the following types of information: First, the queue length of vehicles at each intersection in each direction, i.e., the number of vehicles queuing and waiting to pass in each direction, which is an important indicator for measuring the degree of traffic congestion; second, the waiting time of vehicles at each intersection in each direction, i.e., the average or maximum waiting time of vehicles in each direction, reflecting the level of vehicle delays; and third, the currently executing signal phase and its duration, i.e., which direction is currently green and how long the green light phase has been running, which is an important basis for deciding whether to switch phases.

[0116] In this embodiment, the acquired real-time traffic status is input into the leader deep network, which then outputs multiple candidate coordination actions. A candidate coordination action refers to several alternative coordination actions output by the leader deep network based on the real-time traffic status. Specifically, the candidate coordination action can be a combination of signal phases executable by each candidate intersection in the leader set. For example, if a leader set includes two candidate intersections, and each candidate intersection can execute four signal phases, then the leader deep network outputs sixteen candidate coordination actions based on the real-time traffic status, with each candidate coordination action corresponding to a signal phase combination of the two candidate intersections.

[0117] It should be noted that the aforementioned candidate coordination actions can be all executable coordination actions output by the leader deep network based on real-time traffic conditions, or they can be a subset of executable coordination actions selected according to preset rules. For example, the leader deep network outputs all sixteen candidate coordination actions based on real-time traffic conditions; or, the leader deep network outputs the top four candidate coordination actions with higher value assessments based on real-time traffic conditions.

[0118] In this embodiment, for each candidate coordination action, the candidate action and the local traffic state of the follower set are input into the follower deep network. The follower deep network then generates the candidate signal phase action that the follower set should take under that candidate coordination action. In other words, for each candidate action of the leader, the followers will provide a corresponding optimal response action.

[0119] The traffic state of a follower set refers to the local traffic state of each ordinary intersection within the follower set. Specifically, this traffic state may include information such as the queue length, waiting time, currently active signal phase, and its duration for each approach direction at each ordinary intersection within the follower set.

[0120] Candidate signal phase actions refer to the signal phase actions output by the follower deep network under given candidate coordinated actions and the traffic states of the follower set. Specifically, for each candidate coordinated action, the follower deep network generates a signal phase action that maximizes the benefit of the follower set based on the candidate coordinated action and the traffic states of the follower set.

[0121] This step illustrates the hierarchical relationship between leaders and followers. When making decisions, leaders need to consider how followers will respond to their actions. By pre-calculating the follower responses to each candidate leader's actions, more comprehensive information can be provided for subsequent leader decisions.

[0122] In this embodiment, the leader deep network performs a comprehensive value assessment of each candidate coordination action based on real-time traffic conditions, each candidate coordination action, and the corresponding follower candidate signal phase actions. The value assessment considers not only the leader's own actions but also the followers' responses to those actions, as the final control effect is jointly determined by the actions of both the leader and the followers. After the assessment, the candidate coordination action with the highest value is selected as the target coordination action for the current moment. This target coordination action is the signal phase control action for each intersection in the leader set. Specifically, by comparing the Q values ​​of each candidate coordination action, the candidate coordination action with the highest Q value is selected as the target coordination action.

[0123] Value assessment refers to the process by which a leader deep network calculates the Q-value of a candidate coordinated action based on real-time traffic conditions, candidate coordinated actions, and the candidate signal phase actions of the corresponding follower set. The Q-value is used to evaluate the merits of a candidate coordinated action under the current traffic conditions.

[0124] In this embodiment, after the target coordination action is determined, the candidate signal phase actions of the followers corresponding to the target coordination action are taken as the target signal phase actions of the follower set at the current time. These target signal phase actions are the signal phase control actions of each ordinary intersection in the follower set.

[0125] It should be noted that although the above decision-making process is logically divided into two levels: leader decision-making and follower response, in actual execution, the signal control actions of the leader and followers are output and executed simultaneously. The leader has decision-making priorities; that is, the leader's actions will influence the followers' choices. However, this does not mean that there is a sequence in execution.

[0126] Furthermore, when actually executing signal control actions, preset signal safety constraints must be met to ensure traffic safety. These safety constraints include yellow light phase constraints, all-red phase constraints, minimum phase duration constraints, and phase feasibility constraints. The yellow light phase and all-red phase are transitional phases between two different green light phases, used to clear vehicles from the intersection and prevent conflicts between vehicles from different directions. The minimum phase duration refers to the shortest duration of the green light phase, taking into account pedestrian crossing needs. Phase feasibility constraints depend on the intersection's topology and phase design; different intersections may have different sets of executable phases.

[0127] Specifically, if the signal control action at the current moment differs from the signal control action at the previous moment, a signal switch is executed. During the switch, transition phase constraints such as the yellow light phase and the all-red phase must be met. If the signal control action at the current moment is the same as the signal control action at the previous moment, the current signal phase remains unchanged. For example, when the decision is to switch from a north-south green light to an east-west green light, the switch cannot be performed directly. Instead, a three-second yellow light followed by a one-second all-red transition phase must be executed before switching to the east-west green light. Furthermore, if the current north-south green light has only been executed for five seconds, while the minimum phase duration is specified as ten seconds, the switch cannot be performed immediately. The north-south green light must continue to be executed until the minimum duration requirement is met.

[0128] This embodiment achieves hierarchical real-time traffic signal control through the cooperation of a leader deep network and a follower deep network. The leader is responsible for regional coordination decisions, while the followers are responsible for local responses. The two cooperate with each other, which not only ensures the regional coordination effect but also reduces the decision complexity. During the decision-making process, the leader fully considers the followers' responses to each candidate action and selects the action plan with the highest comprehensive value, which can achieve a better overall control effect.

[0129] In one implementation method Figure 2 This is a schematic diagram of the partitioning results of the first transportation network. Figure 3 This is a schematic diagram of the zoning results for the second traffic network. In the two zoning result diagrams, intersection nodes of different colors represent different traffic control groups, star-shaped markers represent leader intersections or intersections in a leader set, and other circular nodes represent follower intersections. Road colors are used to represent road traffic demand or load intensity; different shades of color can visually reflect the distribution of traffic pressure on each road segment.

[0130] Figure 2 and Figure 3 This is used to illustrate the zoning effect of this application, which, based on given candidate leader intersections, forms traffic control groups through candidate leader merging, leader set construction, and follower intersection division. Figure 2and Figure 3 As can be clearly seen, after zoning, the intersections within each control group exhibit a compact geographical distribution, with clear boundaries between groups, verifying the effectiveness of the zoning method proposed in this application. Furthermore, the distinction between star and circular markers indicates that each control group has exactly one set of leaders, with the remaining intersections being follower intersections, reflecting the spatial layout characteristics of a multi-leader-multi-follower hierarchical structure. The distribution of road colors also reflects a certain homogeneity in traffic load within each control group, while there are differences in traffic load between different control groups. This aligns with the design intent of the traffic load homogeneity term in the zoning objective function.

[0131] It should be noted that, Figure 2 and Figure 3 The zoning results shown are only intended to illustrate the effect of the zoning method of this application. In actual applications, the zoning results will vary depending on the road network structure, traffic demand and control objectives, and are not limited to the specific layout shown in the attached figures.

[0132] To verify the effectiveness of the proposed method, comparative experiments were conducted in traffic network scenarios of different scales. The experiments compared the proposed method with various existing traffic signal control methods, evaluating the control effect from multiple dimensions.

[0133] The comparative experiments described above were conducted under different traffic network scenarios and evaluation settings. The training evaluation results reflect the model's performance during training, while the test results reflect the independent test performance of the control strategy after training. The evaluation metrics used in the experiments included average travel time, vehicle throughput, average distance traveled, and average speed.

[0134] Comparison Methods: Four representative comparison methods were selected for the experiment. The first is the fixed timing control method, a traditional traffic signal control method that operates according to a preset signal cycle and phase duration, unaffected by changes in traffic conditions. The second is the maximum pressure control method, a rule-based adaptive control method where each intersection selects the phase with the highest traffic pressure in each direction for traffic flow. The third is the CoLight method, a multi-agent reinforcement learning method based on graph attention mechanisms, which achieves coordination between intersections through communication mechanisms. The fourth is the MPLight method, a pressure-based multi-agent reinforcement learning method. The method used in this application is denoted as the NSCQ method.

[0135] Evaluation Metrics: The experiment used four evaluation metrics to comprehensively measure the control effectiveness. The first metric is average travel time, which represents the average time taken for vehicles to complete a journey; a smaller value indicates a shorter time required for vehicles to complete the journey, and thus a better control effect. The second metric is vehicle throughput, which represents the number of vehicles that complete passage within the evaluation period; a larger value indicates a stronger network service capacity. The third metric is average distance traveled, used to help interpret the average travel time metric and avoid judging control effectiveness solely based on travel time. The fourth metric is average speed, also used to assist in the evaluation, reflecting the distance-normalized traffic efficiency.

[0136] The experimental results are shown in Table 1: Table 1

[0137] As can be seen, the NSCQ method of this application exhibits excellent control performance in traffic networks of different sizes. In training evaluation, the NSCQ method has higher vehicle throughput than other comparative methods, and its average travel time is also at a lower level. In independent test scenarios, the NSCQ method achieves the highest vehicle throughput, the longest average distance traveled, and the highest average speed. This indicates that the method of this application can serve more vehicles and cover a longer travel distance within the same evaluation period, and improves the traffic efficiency of distance normalization.

[0138] Specifically, compared to fixed-timing methods, the proposed method dynamically adjusts signal timing based on real-time traffic conditions, significantly improving vehicle throughput and reducing average travel time. Compared to maximum pressure methods, the proposed method not only considers the local state of individual intersections but also the overall traffic situation of the region through a hierarchical coordination mechanism, thus exhibiting superior performance in network-level coordination. Compared to multi-agent reinforcement learning methods such as CoLight and MPLight, the proposed method's multi-leader, multi-follower hierarchical structure and gating acceptance mechanism better handle coordination problems in large-scale road networks, resulting in better control performance.

[0139] It should be noted that the above experimental results are merely illustrative, and the actual results may vary depending on factors such as road network structure, traffic demand, and parameter settings. However, overall, the experimental results verify the effectiveness and superiority of the method described in this application in large-scale traffic signal control.

[0140] This application's embodiments divide large-scale traffic networks into multiple traffic control groups with clear coordination relationships by merging candidate leader intersections and leader-guided partitioning. This reduces the decision-making dimensionality and training complexity of large-scale traffic signal joint control, improving the scalability of the method in multi-intersection road networks. Through a multi-leader-multi-follower hierarchical structure, it combines the regional coordination capabilities of candidate leader intersections with the local response capabilities of follower intersections, enabling the traffic signal control process to consider both local traffic conditions and regional traffic coupling relationships, thereby improving cross-regional collaborative control effectiveness. Employing a deep Q-network to learn the coordination strategies of the leader set and the local response strategies of the follower set respectively, it can handle high-dimensional, dynamically changing traffic state information. Compared to methods relying on manual rules or discrete state tables, it is more suitable for complex heterogeneous road networks and large-scale traffic signal control scenarios. The learning phase introduces candidate leader strategy evaluation and gating acceptance mechanisms. By comparing candidate strategies at the state level using the learned Q-value, it can reduce performance degradation in strategy updates, decrease policy oscillations during training, and improve the stability of the hierarchical reinforcement learning process. In the real-time traffic signal control phase, it can directly generate signal phase control actions for leader and follower intersections based on the current traffic state. It can also incorporate safety constraints such as yellow light, all-red light, minimum phase duration, and phase feasibility, making it easy to deploy on traffic simulation platforms, edge control devices, or actual traffic signal control systems. It can reduce computational and communication burdens while maintaining network-level coordination capabilities, making it suitable for urban traffic networks with different traffic demands, road network sizes, and intersection types. It helps improve traffic efficiency, vehicle throughput, and the engineering applicability of large-scale traffic signal control systems.

[0141] Please see Figure 4 This application also provides a traffic signal control device 600, which can implement the above-described traffic signal control method. The device includes: The acquisition module 10 is used to acquire multiple candidate intersections of the traffic network; The merging module 20 is used to merge candidate intersections that meet preset merging conditions based on the spatial proximity relationship between multiple candidate intersections, to obtain at least one leader set, the leader set including at least one candidate intersection; The partitioning module 30 is used to partition the traffic network according to at least one leader set to obtain at least one control group. Each control group includes a leader set and a follower set. The follower set includes ordinary intersections in the traffic network. Ordinary intersections refer to other intersections in the traffic network besides the candidate intersections in the leader set. Training module 40 is used to construct a deep reinforcement learning network based on at least one control group and train the deep reinforcement learning network so that the deep reinforcement learning network learns a first coordination policy and a second coordination policy. The first coordination policy is a coordination policy that controls the candidate crossroads in each leader set, and the second coordination policy is a coordination policy that controls the ordinary crossroads in the follower set, thereby obtaining a control network. The decision module 50 is used to make decisions based on the real-time traffic conditions of the traffic network using the control network, and to obtain the signal control actions for each intersection in the traffic network.

[0142] The specific implementation of this traffic signal control device is basically the same as the specific implementation of the traffic signal control method described above, and will not be repeated here.

[0143] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the traffic signal control method described above. This electronic device can be any smart terminal, including a tablet computer, an in-vehicle computer, or similar device.

[0144] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the traffic signal control method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the traffic signal control method described above.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] The traffic signal control method, device, electronic equipment, and storage medium provided in this application obtain multiple candidate intersections in a traffic network and merge candidate intersections that meet preset merging conditions based on spatial proximity to obtain at least one leader set. Then, the traffic network is partitioned around the leader set to form at least one control group consisting of the leader set and a follower set including ordinary intersections. Subsequently, a deep reinforcement learning network is constructed based on the control group, and through hierarchical training, the deep reinforcement learning network learns a first coordination strategy for the leader set and a second coordination strategy for the follower set, respectively, to obtain the control network. Finally, the control network is used to determine and generate the signal control action for each intersection in the traffic network based on real-time traffic conditions. This application decomposes a large-scale road network into multiple control groups with clear hierarchical relationships through candidate intersection merging and leader-guided partitioning, effectively reducing the dimensionality and training complexity of joint decision-making. Simultaneously, hierarchical training allows the leader set to focus on regional coordination and the follower set to focus on local response, balancing network-level coordination capabilities with the flexibility of local control. This achieves highly scalable, stable, and globally coordinated adaptive traffic signal control in large-scale heterogeneous road networks.

[0148] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0149] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0152] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0153] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A traffic signal control method, characterized in that, The method includes: Obtain multiple candidate intersections in the traffic network; Based on the spatial proximity relationship between multiple candidate intersections, candidate intersections that meet the preset merging conditions are merged to obtain at least one leader set, wherein the leader set includes at least one of the candidate intersections. The traffic network is partitioned according to at least one set of leaders to obtain at least one control group. Each control group includes one set of leaders and one set of followers. The set of followers includes ordinary intersections in the traffic network, which are intersections in the traffic network other than candidate intersections in the set of leaders. A deep reinforcement learning network is constructed based on at least one of the control groups, and the deep reinforcement learning network is trained to learn a first coordination strategy and a second coordination strategy. The first coordination strategy is a coordination strategy for controlling candidate crossroads in each of the leader sets, and the second coordination strategy is a coordination strategy for controlling ordinary crossroads in the follower sets, thereby obtaining a control network. The control network makes decisions based on the real-time traffic conditions of the traffic network to obtain the signal control actions for each intersection in the traffic network.

2. The method according to claim 1, characterized in that, The preset merging conditions include at least one of the following: The spatial distance between the two candidate intersections is less than a preset distance threshold; The traffic coupling strength between the two candidate intersections is greater than a preset coupling threshold. The traffic coupling strength is obtained based on the traffic demand, number of vehicles, and road segment length between the two intersections within a preset time period. The key movement directions between the two candidate intersections are the same or related, and the key movement direction is the vehicle movement direction that has the greatest impact on the signal timing or capacity of the intersection. The two candidate intersections are located in the same traffic corridor.

3. The method according to claim 1, characterized in that, The step of partitioning the traffic network according to at least one set of leaders to obtain at least one control group includes: Using each set of leaders as the core point of each partition, traverse the ordinary intersections; Under the constraints that each ordinary intersection is assigned to only one control group, spatial connectivity is maintained among intersections within each control group, and each control group includes only one set of leaders, the comprehensive allocation cost of each ordinary intersection is determined based on the spatial compactness between the ordinary intersection and the set of leaders within the control group, the degree of traffic flow separation between the ordinary intersection and other intersections within the control group, the consistency of traffic load among each intersection within the control group, and the degree of balance in the number of intersections among each control group. Each of the ordinary intersections is assigned to a candidate control group that minimizes the overall allocation cost of the ordinary intersection, resulting in multiple control groups.

4. The method according to claim 1, characterized in that, The step of constructing a deep reinforcement learning network based on at least one of the control groups and training the deep reinforcement learning network to learn a first coordination policy and a second coordination policy includes: During the learning phase of the second coordination strategy, the first coordination strategy of the leader set is fixed, and the second coordination strategy of the follower set is updated. During the learning phase of the first coordination strategy, candidate coordination strategies are generated based on the updated second coordination strategy. Based on the value comparison between the candidate coordination strategy and the current coordination strategy, the first coordination strategy of the leader set is updated.

5. The method according to claim 4, characterized in that, The step of fixing the first coordination strategy of the leader set and updating the second coordination strategy of the follower set during the learning phase of the second coordination strategy includes: With the first coordination strategy of the leader set fixed, obtain the state transition data generated by the signal phase action of each follower set executing the second coordination strategy; The state transition data is stored in the experience replay pool; Historical state transition data are sampled from the experience replay pool and calculated using a preset target value function to obtain the temporal difference target of each follower set under a given local observation state, the first coordination strategy of the leader set, and its own action conditions. The second coordination strategy of the follower set is updated by minimizing the deviation between the temporal difference objective and the value of the current follower action.

6. The method according to claim 4, characterized in that, The step of updating the first coordination strategy of the leader set based on a value comparison between the candidate coordination strategies and the current coordination strategy includes: With the updated second coordination strategy fixed, the deep reinforcement learning network obtains the first global value evaluation result corresponding to the current coordination strategy and the second global value evaluation result corresponding to the candidate coordination strategy in each enhanced state. The enhanced state includes: the maximum pressure value of each signal phase level, the signal phase currently being executed at each intersection, the duration of the current phase, the maximum vehicle waiting time in each entrance direction, and the vehicle queue length in each entrance direction. If the value comparison result meets the preset conditions, the candidate coordination strategy is taken as the current coordination strategy, and the first coordination strategy of the leader set is updated according to the candidate coordination strategy. The preset conditions are that the second global value assessment result is greater than or equal to the first global value assessment result in each of the enhanced states, and is greater than the first global value assessment result in at least one of the enhanced states. If the value comparison result does not meet the preset conditions, the current coordination strategy remains unchanged.

7. The method according to claim 1, characterized in that, The control network includes a leader deep network and a follower deep network. The leader deep network is used to learn the first coordination strategy of the leader set, and the follower deep network is used to learn the second coordination strategy of the follower set. The step of using the control network to make decisions based on the real-time traffic conditions of the traffic network to obtain the signal control actions for each intersection in the traffic network includes: The real-time traffic status of the traffic network at the current moment is obtained. The real-time traffic status includes the vehicle queue length in each direction of each intersection, the vehicle waiting time in each direction of each intersection, the currently executed signal phase and its duration. The real-time traffic status is input into the leader deep network to obtain multiple candidate coordination actions; Based on the candidate coordinated action and the traffic state of each follower set, the candidate signal phase action of each follower set under each candidate coordinated action is generated through the follower deep network. Through the leader deep network, each candidate coordination action is evaluated for value based on the real-time traffic status, each candidate coordination action, and the candidate signal phase actions of each follower set corresponding to each candidate coordination action. The candidate coordination action with the highest value is selected as the target coordination action at the current moment. The target coordination action is the signal phase control action of the intersection in the leader set. The candidate signal phase action of each follower set corresponding to the target coordination action is taken as the target signal phase action of each follower set at the current time, and the target signal phase action is the signal phase control action of the intersection in the follower set.

8. A traffic signal control device, characterized in that, The device includes: The acquisition module is used to acquire multiple candidate intersections in the traffic network; The merging module is used to merge candidate intersections that meet preset merging conditions based on the spatial proximity relationship between multiple candidate intersections, to obtain at least one leader set, wherein the leader set includes at least one of the candidate intersections; A partitioning module is used to partition the traffic network according to at least one set of leaders to obtain at least one control group. Each control group includes a set of leaders and a set of followers. The set of followers includes ordinary intersections in the traffic network. Ordinary intersections refer to other intersections in the traffic network other than the candidate intersections in the set of leaders. A training module is configured to construct a deep reinforcement learning network based on at least one of the control groups and train the deep reinforcement learning network to learn a first coordination strategy and a second coordination strategy, wherein the first coordination strategy refers to a coordination strategy for controlling candidate crossroads in each of the leader sets, and the second coordination strategy refers to a coordination strategy for controlling ordinary crossroads in the follower sets, thereby obtaining a control network. The decision module is used to make decisions based on the real-time traffic conditions of the traffic network using the control network, and to obtain the signal control actions for each intersection in the traffic network.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the traffic signal control method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the traffic signal control method according to any one of claims 1 to 7.

Images