Multi-intersection vehicle-road cooperative control method and device based on hierarchical reinforcement learning, medium
By deploying managers, traffic lights, and CAV agents in the traffic system and adjusting their behavior using global reward offsets, the coordination problem of traffic light and CAV trajectory planning in existing technologies is solved, achieving optimization and stability of global traffic flow.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-05-25
- Publication Date
- 2026-08-04
AI Technical Summary
In existing hierarchical reinforcement learning traffic control methods, there is a lack of flexible and efficient linkage mechanisms between managers and frontline workers. This makes it difficult for traffic light timing and CAV trajectory planning to work together under the same optimization objective, resulting in policy oscillations and failing to optimize global traffic flow.
By deploying manager agents, traffic light agents, and connected autonomous vehicle agents, the manager agent collects global traffic information and calculates global reward offset, which is then passed to the reward functions of the traffic light and CAV agents to adjust their behavior to achieve global optimization.
It achieves the co-evolution of traffic light timing and CAV trajectory planning, avoids system oscillations, achieves global optimization of regional traffic flow, and improves the flexibility and adaptability of the traffic system.
Smart Images
Figure CN122245129B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of cooperative control technology of intelligent transportation systems, and particularly relates to a multi-intersection vehicle-road cooperative control method, device, and medium based on hierarchical reinforcement learning. Background Technology
[0002] In recent years, with the continuous development of intelligent transportation systems, traffic flow optimization has become a key technology for solving traffic congestion and improving road efficiency. Especially with the gradual integration of connected automated vehicles (CAVs) into daily travel, traffic management is evolving from single vehicle control to large-scale vehicle-road cooperation.
[0003] Traditional traffic signal control methods often rely on pre-set periodic control or local adaptive strategies based on single intersections. While these methods improve traffic efficiency to some extent, they still face problems of local optima and response lag in complex and dynamically changing traffic environments.
[0004] Traditional methods lack global coordination across intersections and often rely on reactive adjustments based on limited local data, making it difficult to achieve real-time dynamic responses across the entire network. In particular, when faced with large-scale road network systems and the introduction of connected autonomous vehicles (CAVs), traditional methods often fail to effectively adjust traffic signals and vehicle trajectories in real time. Traffic light timing and vehicle trajectory planning are treated as two independent islands, thus limiting the upper limit of overall traffic flow optimization.
[0005] To address these challenges, hierarchical reinforcement learning methods have been increasingly applied to traffic flow optimization in recent years. By dividing the traffic control system into multiple layers of agents, hierarchical reinforcement learning enables effective coordination between global and local levels, enhancing the system's decision-making capabilities. Specifically, the manager agent in hierarchical reinforcement learning can dynamically adjust the collaboration among worker agents (such as traffic lights and CAVs) based on global traffic information, thereby optimizing the decision-making process for multiple intersections and vehicle-road cooperation. This method not only effectively solves the local optima problem in traditional traffic signal control but also achieves global traffic flow optimization through agent collaboration. Furthermore, the introduction of hierarchical reinforcement learning enables intelligent transportation systems to better cope with complex and dynamically changing traffic environments, improving the flexibility and adaptability of intelligent management.
[0006] However, in existing hierarchical reinforcement learning traffic control architectures, establishing a "flexible and efficient" linkage mechanism between managers and lower-level workers remains a technical bottleneck. Conventional multi-agent systems often lack fine-grained and unified guidance methods, easily leading to policy oscillations in the autonomous exploration of lower-level agents, making it difficult to achieve true synchronization between traffic light timing and CAV trajectory planning under the same optimization objective. Therefore, designing a more refined agent cooperative control architecture to break through the optimization ceiling of existing multi-intersection vehicle-road cooperation is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0007] To address the shortcomings of existing technologies, embodiments of the present invention provide a multi-intersection vehicle-road cooperative control method, device, and medium based on hierarchical reinforcement learning.
[0008] In a first aspect, embodiments of the present invention provide a multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning, the method comprising:
[0009] Obtain the intersection nodes and road connection relationships of the traffic network to be analyzed, thereby constructing a road network model; deploy a manager intelligent agent, several traffic light intelligent agents, and several connected autonomous vehicle intelligent agents at each intersection;
[0010] Each traffic light intelligent agent collects the current traffic status at the intersection, and each connected autonomous vehicle intelligent agent collects the current traffic flow status on the road.
[0011] The manager agent at each intersection calculates the global reward offset based on the shared global traffic state and traffic flow state, and passes the global reward offset to the reward function of the corresponding traffic light agent and the connected autonomous vehicle agent.
[0012] Each traffic light agent and each connected autonomous vehicle agent adjusts the traffic light phase and the trajectory planning of the connected autonomous vehicle respectively based on the global reward offset.
[0013] In a second aspect, embodiments of the present invention provide an electronic device, comprising:
[0014] At least one processor; and
[0015] A memory communicatively connected to the at least one processor; wherein,
[0016] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the above-described multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning.
[0017] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning.
[0018] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning.
[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0020] This invention provides a multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning. By deploying a multi-level hierarchical decision architecture that includes a manager agent, a traffic light agent, and a connected autonomous vehicle agent, the manager agent collects global traffic information and makes hierarchical decisions. This method can effectively solve the problem of conflict between local optimization and global objectives while ensuring the real-time performance of local control at the lower level, and achieve global optimization of regional traffic flow.
[0021] Meanwhile, in this invention, the manager agent does not directly take over the underlying control. Instead, it calculates the global reward offset (ΔR) and passes it as a reward offset term to the traffic light agent and the connected autonomous vehicle agent. This dynamic adjustment mechanism enables the traffic light timing optimization and the connected autonomous vehicle's driving strategy to co-evolve under a common global goal. This allows the autonomous exploration and decision-making of each agent to naturally converge to the globally optimal direction, avoiding system oscillations caused by rigid instructions. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 A flowchart of a multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning provided in an embodiment of the present invention;
[0024] Figure 2 This is a topology diagram of the test road network provided in an embodiment of the present invention;
[0025] Figure 3 A schematic diagram of hierarchical reinforcement learning provided in an embodiment of the present invention;
[0026] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.
[0029] like Figure 1 As shown, this embodiment of the invention provides a multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning, the method comprising:
[0030] Step S1: Obtain the intersection nodes and road connection relationships of the traffic network to be analyzed, thereby constructing a road network model; deploy a manager intelligent agent, several traffic light intelligent agents, and several connected autonomous vehicle intelligent agents at each intersection.
[0031] In some embodiments, the intersection nodes and road connections of the traffic network to be analyzed are obtained. Since the layout of intersections and the road connections directly determine the direction and trajectory of traffic flow, the road composition of each intersection (including several entrance and exit roads) is clarified, thereby constructing a road network model.
[0032] In this example, based on the connection relationships between intersections, an adjacency matrix G={V,E} is used to store the road network; where V represents intersection nodes and E represents the road connection relationships between intersections. Through this adjacency matrix, the system can obtain the status information of adjacent intersections and establish cross-intersection communication links. Each lane is initialized based on preset road data (such as the number of lanes, speed limits, etc.).
[0033] The status data of the intersection node includes: a unique identifier for the current intersection, and identifiers of the surrounding neighboring intersections arranged in a preset directional order; if the current intersection has no neighboring intersections in a certain preset direction, a preset edge identifier is used as a substitute.
[0034] Combination Figure 2As shown, this method employed a 3×3 road network topology in the test, which represents a rectangular grid road network consisting of 9 intersections (i.e., nodes), numbered from 0 to 8. Black lines represent roads connecting the intersections. Except for edge intersections, internal intersections (such as node 4) are directly connected to the four intersections to the left, right, and up. Each intersection is equipped with an independent traffic light agent and a connected autonomous vehicle agent. Regarding state representation:
[0035] Current intersection status [ID]: The number in square brackets is the unique identifier (ID) of this intersection. For example, the ID of the center node is 4.
[0036] Adjacent intersection status: The adjacency relationship between this intersection and surrounding intersections is represented by a list [N, E, S, W]. The four numbers in the list correspond to the neighbor IDs [North, East, South, West] (clockwise or top right bottom left). For example, taking node 4 as an example, the adjacent intersection status is: [1, 5, 7, 3], where 1 is above (North), 5 is to the right (East), 7 is below (South), and 3 is to the left (West). The [ / ] in the adjacent intersection status indicates that there is no adjacent intersection in that direction, i.e., the edge of the road network. For example, taking node 0 as an example, the adjacent intersection status is: [ / , 1, 3, / ], where North ( / ) is above, East (1) is to the right connected to node 1, South (3) is to the bottom connected to node 3, and West ( / ) is to the left.
[0037] During the initialization phase, independent manager agents, traffic light agents, and connected autonomous vehicle (CAV) agents are deployed at each intersection. The traffic light agent determines the initial phase based on the initial traffic flow, the CAV agent performs preliminary trajectory planning based on the initial and target locations, and the manager agent collects preliminary data to formulate a global initial policy. The policy networks of all agents are initialized, and can employ random policies or heuristic rule-based configurations to support initial exploration.
[0038] It should be noted that this invention uses an adjacency matrix to systematically model the road network, clearly defining the intersection nodes and road connection relationships, providing fundamental support for information sharing and coordination across intersections. This method not only supports complex road network topologies with multiple nodes but can also be widely applied in real-world urban traffic environments with multiple intersections and lanes, providing strong data support and flexible decision-making basis for large-scale intelligent transportation systems.
[0039] In step S2, each traffic light agent collects the current traffic status of the intersection, and each connected autonomous vehicle agent collects the current traffic flow status of the road.
[0040] Specifically, each traffic light agent collects traffic data including the queue length of vehicles on each road at the current intersection, data from the nearest connected autonomous vehicle to the traffic light, and the current signal phase.
[0041] Each connected autonomous vehicle agent acquires traffic flow status including the current location, speed, and acceleration of the connected autonomous vehicle, the location, speed, and acceleration of its neighboring connected autonomous vehicles, and the phase of the traffic lights at the intersection.
[0042] In step S3, the manager agent at each intersection calculates the global reward offset based on the shared global traffic state and traffic flow state, and passes the global reward offset to the reward functions of the corresponding traffic light agent and the connected autonomous vehicle agent.
[0043] Furthermore, the manager agent integrates macroscopic traffic flow data, including intersection pressure (the difference between the number of exiting and entering vehicles), queue length, and CAV status, based on data provided by the traffic light agents at the current and adjacent intersections. The collected raw data needs to be normalized to construct a standardized state vector input. For example, vehicle speed is normalized according to... Normalize (where, , To maximize speed limits, traffic light phases are processed using one-hot encoding, while the traffic light states at adjacent intersections are fused using a weighted average method.
[0044] Specifically, to prevent congestion spillover effects from adjacent intersections caused by optimizing a single intersection, the traffic manager at each intersection will periodically share information such as traffic flow, vehicle queues, and delay times with the managers of adjacent intersections. In this scenario, an objective function is defined, expressed as follows:
[0045]
[0046] In the formula, Let be the queue length at the i-th intersection. Let i be the vehicle throughput at the i-th intersection. Let represent the energy consumption of all connected autonomous vehicles at the i-th intersection.
[0047] Furthermore, the manager agent at each intersection calculates the specific reward offset based on the objective function described above. For the i-th intersection, the global reward offset corresponding to the i-th intersection is determined by subtracting the vehicle queue length of the i-th intersection and the energy consumption of all connected autonomous vehicles at the i-th intersection from the vehicle throughput of the i-th intersection, as shown in the following expression:
[0048]
[0049] It should be noted that in multi-intersection collaborative scenarios, the manager's intelligent agent comprehensively considers factors such as vehicle queue length, vehicle throughput, and CAV energy consumption at each intersection to calculate the global reward function. This not only maximizes the capacity of each intersection and reduces traffic delays but also optimizes the energy use of the entire network, promoting low-energy traffic and meeting the global indicators of energy conservation and emission reduction. Furthermore, addressing the problem of the separation between CAV trajectory planning and traffic light decision-making in existing technologies, this invention does not directly take over the underlying control. Instead, it transmits the global reward offset to the reward functions of the corresponding traffic light agents and connected autonomous vehicle agents. The manager's intelligent agents communicate with each other, perceiving the overall traffic status and calculating the reward offset; the traffic light agents and connected autonomous vehicle agents spontaneously adjust their behavior to obtain higher rewards, thus achieving a high degree of collaborative control between "vehicles" and "roads" under the same optimization objective. This dynamic adjustment mechanism enables the traffic light timing optimization and CAV driving strategy to evolve collaboratively under a common global goal, allowing the autonomous exploration and decision-making of each agent to naturally converge to the global optimal direction, thus avoiding system oscillations caused by rigid instructions.
[0050] In step S4, each traffic light agent and each connected autonomous vehicle agent adjusts the traffic light phase and the trajectory planning of the connected autonomous vehicle according to the global reward offset.
[0051] Specifically, the manager agent adds the global reward offset to the reward functions of the traffic light agent and the connected autonomous vehicle agent in the target optimization direction;
[0052] The traffic light agent aims to maximize its own reward by adjusting the signal phase of the target optimization direction to a passable state and / or extending the green light time of the target optimization direction.
[0053] The connected autonomous vehicle's intelligent agent aims to maximize its own rewards by adjusting the vehicle's average speed, acceleration, and driving trajectory to match the green light window of the traffic flow and pass through the current intersection.
[0054] For example, combined Figure 3 The hierarchical reinforcement learning structure and collaborative workflow of the present invention will be described in detail. Figure 3 It intuitively demonstrates how the upper-level manager agent flexibly coordinates the specific behaviors of the lower-level worker agents (including traffic light agents and CAV agents) by issuing "reward offsets".
[0055] In real-world traffic networks, each managerial agent not only monitors the traffic conditions at its own intersection but also periodically exchanges key information such as traffic flow and queue length with managers at adjacent intersections. This cross-node communication mechanism ensures that the system's decisions are based on a "regional global perspective" rather than an isolated, single-point perspective.
[0056] like Figure 3 As shown in the diagram, different types of lines are used to distinguish the traffic flow adjustment strategies for different directions at the intersection. The solid blue line represents the strategy for adjusting east-west traffic flow, while the dashed yellow line represents the strategy for adjusting north-south traffic flow.
[0057] When the road network traffic conditions become unbalanced, for example, when the manager agent detects congestion in the east-west direction, the manager agent will generate a global reward offset ΔR that tends to alleviate congestion in the east-west direction, and will send the reward offset ΔR in real time to the traffic light agent in the center of the intersection and the CAV agents traveling in the east-west direction.
[0058] The specific response mechanism of the lower-level worker agents (traffic light agents and CAV agents) after receiving the global reward offset ΔR is as follows:
[0059] The traffic light agent is located at the center of the intersection. Based on the received reward offset ΔR, it tends to switch the east-west traffic phase or the north-south traffic phase. For example, if the traffic light agent receives a reward offset ΔR that favors the east-west direction, it will be more inclined to allocate a green light to the east-west direction in order to maximize its own reward.
[0060] The CAV agents are distributed across the four lanes in the east, west, north, and south directions. When a connected autonomous vehicle (CAV) in the east-west direction receives a reward for accelerated passage, it will proactively adjust its acceleration and trajectory planning to quickly pass through the intersection during green light periods, thereby increasing the overall throughput of the intersection.
[0061] In summary, the control logic in this embodiment forms a complete closed loop: First, the manager agents communicate with each other to perceive the overall traffic status of the network; second, the manager agents calculate the reward offset ΔR and decide which direction to prioritize (e.g., setting a higher reward for east-west traffic); subsequently, the traffic light agents extend the green light for east-west traffic to obtain a higher reward; finally, the CAV agents in the east-west direction actively adjust their speed to match the green light window in order to obtain a higher reward. Through this reward-guided soft control mechanism, this invention successfully achieves deep collaborative control between "vehicles" and "roads" under the same optimization objective.
[0062] Furthermore, the method also includes:
[0063] Each manager agent, traffic light agent, and connected autonomous vehicle agent is trained in a common environment, but maintains its own independent policy network.
[0064] After each phase switch, the traffic light agent receives an immediate reward from the environment. The traffic light agent combines the reward offset ΔR issued by the manager agent and updates its strategy based on the advantage function to ensure that the phase selection evolves toward the globally optimal goal.
[0065] The intelligent agent of the connected autonomous vehicle comprehensively considers driving time, fuel consumption and vehicle delay to calculate the real-time reward, and combines the global reward offset to adjust its own acceleration and driving trajectory strategy through the PPO algorithm;
[0066] The managerial agent continuously updates the global reward calculation network using the PPO algorithm based on changes in global traffic flow, in order to optimize the overall network control strategy.
[0067] In summary, this invention provides a multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning. By deploying a multi-level hierarchical decision architecture that includes a manager agent, a traffic light agent, and a connected autonomous vehicle agent, and by having the manager agent collect global traffic information and make hierarchical decisions, this method can effectively solve the problem of conflict between local optimization and global objectives while ensuring the real-time performance of local control at the lower level, thereby achieving global optimization of regional traffic flow.
[0068] like Figure 4 As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.
[0069] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.
[0070] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.
[0071] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0072] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0073] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0074] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0075] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.
[0076] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A method for multi-intersection vehicle-road cooperative control based on hierarchical reinforcement learning, characterized in that, The method includes: Obtain the intersection nodes and road connection relationships of the traffic network to be analyzed, thereby constructing a road network model; deploy a manager intelligent agent, several traffic light intelligent agents, and several connected autonomous vehicle intelligent agents at each intersection; Each traffic light intelligent agent collects the current traffic status at the intersection, and each connected autonomous vehicle intelligent agent collects the current traffic flow status on the road. The manager agent at each intersection calculates the global reward offset based on the shared global traffic state and traffic flow state, and passes the global reward offset to the reward function of the corresponding traffic light agent and the connected autonomous vehicle agent. The process of the manager agent at each intersection calculating the global reward offset based on the shared global traffic state and traffic flow state includes: the global reward offset for the i-th intersection is determined by subtracting the vehicle queue length and the energy consumption of all connected autonomous vehicles at the i-th intersection from the vehicle throughput of the i-th intersection. Each traffic light agent and each connected autonomous vehicle agent adjusts the traffic light phase and the trajectory planning of the connected autonomous vehicle respectively based on the global reward offset.
2. The hierarchical reinforcement learning based multi-intersection vehicle-road cooperation control method according to claim 1, characterized in that, The process of obtaining the intersection nodes and road connections of the traffic network to be analyzed, thereby constructing a road network model: Obtain the intersection nodes and road connection relationships of the traffic network to be analyzed, and construct the network model through the adjacency matrix G={V,E}; where V represents the intersection node and E represents the road connection relationship between intersections; The status data of the intersection node includes: a unique identifier for the current intersection, and identifiers of the surrounding neighboring intersections arranged in a preset directional order; if the current intersection has no neighboring intersections in a certain preset direction, a preset edge identifier is used as a substitute.
3. The hierarchical reinforcement learning based multi-intersection vehicle-road cooperation control method according to claim 1, characterized in that, The process by which each traffic light agent collects the current traffic status at the intersection includes: Each traffic light intelligent agent collects traffic status data, including the queue length of vehicles on each road at the current intersection, data from the nearest connected autonomous vehicle to the traffic light, and the current signal phase.
4. The hierarchical reinforcement learning based multi-intersection vehicle-road cooperation control method according to claim 1, characterized in that, The process by which each connected autonomous vehicle intelligent agent collects the current traffic flow status on the road includes: Each connected autonomous vehicle agent acquires traffic flow status including the current location, speed, and acceleration of the connected autonomous vehicle, the location, speed, and acceleration of its neighboring connected autonomous vehicles, and the phase of the traffic lights at the intersection.
5. The hierarchical reinforcement learning based multi-intersection vehicle-road cooperation control method according to claim 1, characterized in that, The process by which each traffic light agent and each connected autonomous vehicle agent adjusts the traffic light phase and the trajectory planning of the connected autonomous vehicle based on the global reward offset includes: The manager agent adds the global reward offset to the reward functions of the traffic light agent and the connected autonomous vehicle agent in the target optimization direction; The traffic light agent aims to maximize its own reward by adjusting the signal phase of the target optimization direction to a passable state and / or extending the green light time of the target optimization direction. The connected autonomous vehicle's intelligent agent aims to maximize its own rewards by adjusting the vehicle's average speed, acceleration, and driving trajectory to match the green light window of the traffic flow and pass through the current intersection.
6. The hierarchical reinforcement learning based multi-intersection vehicle-road cooperation control method according to claim 1, characterized in that, The method further includes: Each manager agent, traffic light agent, and connected autonomous vehicle agent is trained in a common environment, but maintains its own independent policy network. The traffic light agent calculates the immediate reward based on the selected signal phase and updates the policy based on the advantage function, in conjunction with the global reward offset. The intelligent agent of the connected autonomous vehicle comprehensively considers driving time, fuel consumption and vehicle delay to calculate real-time rewards, and combines the global reward offset to update the acceleration and driving trajectory strategies. The manager agent updates the global reward using a proximal policy optimization algorithm.
7. An electronic device, comprising: include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning as described in any one of claims 1-6.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning as described in any one of claims 1-6.
9. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the multi-intersection vehicle-road cooperative control method based on hierarchical reinforcement learning as described in any one of claims 1-6.