Multi-type intersection mixed traffic control method based on multi-agent reinforcement learning

Abstract

The invention discloses a multi-type intersection mixed traffic control method based on multi-agent reinforcement learning, and the method comprises the steps: setting CAV special lanes and common lanes for different types of intersections, and configuring different signal lamp phases for all types of intersections; a double-layer control structure is constructed, an upper-layer controller selects an optimal signal phase by adopting a multi-agent reinforcement learning algorithm according to the state information of each intersection, and the signal lamp state of each intersection is dynamically adjusted; and when the CAV special phase is activated, the lower-layer controller adopts a reinforcement learning algorithm to carry out passing right distribution and cooperative control on the CAV on the CAV special lane, and carries out speed guidance and track cooperation on the CAV fleet. By constructing a double-layer control framework and fusing a collaborative decision-making mechanism of multi-agent reinforcement learning, the green wave passing time of vehicles on a main road is prolonged, and the passing efficiency is improved; the overall throughput of a road network is improved, and the bearing and dispersion capabilities of a regional traffic system are enhanced.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a method, device, and medium for mixed traffic control of multiple types of intersections based on multi-agent reinforcement learning. Background Technology

[0002] With the continued acceleration of urbanization, the number of motor vehicles has increased rapidly. This rapid increase in vehicles and drivers has placed enormous pressure on existing transportation infrastructure, exacerbating the imbalance between traffic supply and demand and making urban traffic congestion increasingly severe. This not only reduces travel efficiency but also has a profound impact on socio-economic operations, energy consumption, and the ecological environment. Road intersections, as core hubs and key bottlenecks in urban transportation networks, bear the heavy responsibility of facilitating the convergence, diversion, and turning of complex traffic flows, and are centrally located where traffic signal control systems are deployed. Therefore, intelligent and collaborative optimization of intersection control strategies has become a crucial breakthrough for alleviating urban congestion and improving road network operational efficiency.

[0003] Current research largely focuses on the isolated optimization of individual intersections. In real urban road networks, intersections are often densely distributed and interconnected, with the traffic flow control strategies of upstream intersections directly impacting the queuing and traffic flow of downstream intersections. If the goal is solely to improve the capacity of a single point while ignoring the dynamic relationships between adjacent intersections, it can easily lead to traffic flow oscillations, queue overflows, and even regional congestion. Therefore, it is essential to construct a multi-intersection collaborative control mechanism to achieve coordinated optimization of signal timing from a global perspective, thereby truly improving the overall operational efficiency of the road network.

[0004] In real-world urban traffic environments, intersection topologies exhibit high diversity, including not only common cross-shaped intersections but also a wide range of non-standard or asymmetrical forms such as T-shaped and L-shaped intersections. These different types of intersections differ significantly in traffic flow direction, conflict point distribution, traffic phase design, and control complexity, placing higher demands on the adaptability and flexibility of signal control strategies. Therefore, collaborative control methods for real-world road networks must be adaptable to various intersection morphologies to effectively address the dynamic scheduling needs of complex urban traffic scenarios.

[0005] Meanwhile, for a considerable period in the future, the traffic environment will be characterized by the coexistence of human-driven vehicles (HVs) and connected autonomous vehicles (CAVs). Although unsignaled intersection control strategies can theoretically improve efficiency, the lack of unified signal guidance leads to highly uncertain HV driving behavior and significant safety hazards. Current mainstream solutions typically allow CAVs and HVs to share lanes, improving efficiency only by adjusting traffic light timings. This approach fails to fully leverage the collaborative control potential of CAVs and effectively mitigate the risks posed by HV behavior. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device and medium for mixed traffic control of multiple types of intersections based on multi-agent reinforcement learning, which solves the problems existing in the prior art.

[0007] This invention is achieved through the following technical solution:

[0008] In a first aspect, the first embodiment of the present invention provides a multi-type intersection hybrid traffic control method based on multi-agent reinforcement learning, comprising:

[0009] For different types of intersections, dedicated CAV lanes and regular lanes are set up. The dedicated CAV lanes adopt a free lane mode, while the regular lanes adopt a fixed lane mode. Different signal light phases are configured for each type of intersection.

[0010] A two-layer control structure is constructed, which includes an upper-layer controller and a lower-layer controller;

[0011] The upper-level controller selects the optimal signal phase and dynamically adjusts the traffic light status of each intersection based on the status information of each intersection using a multi-agent reinforcement learning algorithm.

[0012] When the dedicated CAV phase is activated, the lower-level controller uses a reinforcement learning algorithm to allocate right-of-way and coordinate control of CAVs in the dedicated CAV lane, and to guide the speed and coordinate the trajectory of the CAV convoy.

[0013] Furthermore, the specific method for selecting the optimal signal phase using a multi-agent reinforcement learning algorithm includes:

[0014] Deploy an intelligent agent at each intersection in the road network;

[0015] Each agent inputs the local observation state of the intersection it is located into its individual neural network and outputs the local action value;

[0016] The QMIX network is used to mix the local action value of each agent with the global state to generate a joint action value function, and the joint action value function is used as the target for centralized training.

[0017] After training, each agent performs distributed phase decision-making based on the joint action set output by the QMIX network.

[0018] Furthermore, before each agent inputs the local observation state of its intersection into its individual neural network, a dimensionality unification preprocessing is performed based on the intersection type.

[0019] Furthermore, the status information is the average queue length and average vehicle speed of all entering lanes at a single intersection, and the status dimensions are configured differently according to the intersection type. For example, a cross-shaped intersection has 12 entering lanes, a T-shaped intersection has 9 entering lanes, and an L-shaped intersection has 6 entering lanes. When constructing the overall status, the upper-level controller performs unified encoding or padding alignment on the heterogeneous statuses.

[0020] Furthermore, configuring differentiated signal phases for different types of intersections specifically includes:

[0021] The cross-shaped intersection is equipped with 4 HV main phases and 1 CAV dedicated phase, for a total of 5 phase combinations;

[0022] The T-junction is equipped with 3 HV main phases and 1 CAV dedicated phase, for a total of 4 phase combinations;

[0023] The L-shaped intersection is equipped with two main HV phases and one dedicated CAV phase, for a total of three phase combinations.

[0024] Furthermore, the reward function of the upper-level controller includes the average green wave travel time of all vehicles in the road network and the overall throughput of the road network. The weights corresponding to the average green wave travel time and the overall throughput of the road network are dynamically adjusted according to the actual traffic management objectives.

[0025] Furthermore, the lower-level controller employs a reinforcement learning algorithm called DDQN network, which determines actions through an ε-greedy strategy. These actions are represented by binary vectors and are used to allocate the right-of-way for CAVs before the stop line on the dedicated CAV lane.

[0026] Furthermore, the state of the lower-level controller adopts discrete traffic state coding, including the vehicle's position, speed, path, and release priority matrix. The reward function of the lower-level controller comprehensively considers three evaluation factors: the number of CAVs passing through, the queue length of the CAV dedicated lane, and the CAV release priority.

[0027] Secondly, another embodiment of the present invention provides an electronic device comprising: a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are interconnected, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the method described in the first embodiment above.

[0028] Thirdly, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in the first embodiment above.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] This invention provides a method, device, and medium for mixed traffic control at multi-type intersections based on multi-agent reinforcement learning. This not only effectively mitigates the safety risks arising from the uncertainty of human, driver, and vehicle behavior in mixed traffic environments, but also achieves simultaneous improvements in two key performance indicators by introducing physically isolated dedicated CAV lanes, setting dedicated CAV signal phases, constructing a macro- and micro-level collaborative two-layer control architecture, and integrating a collaborative decision-making mechanism based on multi-agent reinforcement learning. On the one hand, it significantly extends the green wave passage time for vehicles on main roads, improving traffic efficiency; on the other hand, it greatly increases the overall throughput of the road network, enhancing the carrying capacity and traffic management capabilities of the regional transportation system. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0032] Figure 1 A flowchart of a multi-type intersection mixed traffic control method based on multi-agent reinforcement learning provided in the first embodiment of the present invention;

[0033] Figure 2 This is a structural diagram of a type of intersection in a real road network;

[0034] Figure 3 A schematic diagram showing the phase settings for different types of intersections;

[0035] Figure 4 A schematic diagram of the CAV dedicated lane and coordinated control at the intersection;

[0036] Figure 5 This is a schematic diagram illustrating the mutual influence of traffic flow at adjacent intersections.

[0037] Figure 6 A diagram illustrating green wave traffic flow;

[0038] Figure 7 This is a schematic diagram of a two-layer control structure. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0040] like Figure 1 As shown, the first embodiment of the present invention provides a multi-type intersection hybrid traffic control method based on multi-agent reinforcement learning, comprising:

[0041] For different types of intersections, dedicated CAV lanes and regular lanes are set up. The dedicated CAV lanes adopt a free lane mode, while the regular lanes adopt a fixed lane mode. Different signal light phases are configured for each type of intersection.

[0042] A two-layer control structure is constructed, which includes an upper-layer controller and a lower-layer controller;

[0043] The upper-level controller selects the optimal signal phase and dynamically adjusts the signal light status of each intersection based on the status information of each intersection using a multi-agent reinforcement learning algorithm.

[0044] When the dedicated CAV phase is activated, the lower-level controller uses the DDQN algorithm to allocate right-of-way and coordinate control of CAVs in the dedicated CAV lane, and to guide the speed and coordinate the trajectory of the CAV convoy.

[0045] In mixed traffic environments involving both CAVs and HVs, dedicated CAV lanes are introduced to isolate the mutual interference between CAVs and HVs. Firstly, different signal phases are set for different types of intersections. For example, in a standard cross intersection, the dedicated CAV lane is set as a free-flowing lane, meaning that CAVs turning right, going straight, and turning left can all pass through the intersection in this lane; the regular lane is set as a fixed lane, meaning that HVs turning left can only pass through the intersection in the fixed left-turn lane. Based on this, a dedicated phase is set for CAVs in the dedicated CAV lane. Within this phase, a special control strategy is used to freely regulate the passage of CAVs through the intersection. In other phases, traffic lights separate conflicting vehicles, allocating conflicting HV traffic to different phases, while HVs proceed through the intersection in the traditional manner according to the signal light instructions.

[0046] A multi-agent deep reinforcement learning framework and a two-layer control structure are applied to solve the aforementioned intersection coordination problem. The dedicated lane modes and traffic light states are dynamically adjusted based on real-time traffic flow to achieve refined management of mixed traffic flow. The two-layer control structure includes an upper-controller and a lower-controller. The upper-controller is responsible for macro-level control of phase selection. Each agent acquires its state from local observations. The environment transitions to the next global state based on the joint actions of all agents, and a global reward is obtained. The global state, the actions of each agent, the global reward, and the next global state are stored in an experience replay pool for subsequent training of the QMIX network. The QMIX algorithm monotonically mixes the local Q-values ​​of each agent and constructs a joint action value function using the global state, thereby achieving end-to-end optimization of the multi-agent collaborative strategy while satisfying the centralized training and distributed execution (CTDE) paradigm. The lower-controller is responsible for micro-level control of the release of CAV vehicles under the corresponding CAV dedicated phase. Each intersection has one CAV control agent. For each CAV control agent, at each time step... Obtaining state from the environment ,pass - A greedy strategy selects an action, and the environment transitions to the next state. Receive a reward ,Will Saved in the experience replay pool for training the DDQN network.

[0047] The upper-level controller dynamically adjusts the status of traffic lights at each intersection. The status, action, and reward are defined as follows:

[0048] state First, the AI ​​at each intersection observes the state of that intersection, specifically the average queue length and average vehicle speed across all entering lanes. Considering the highly diverse topological structures of intersections in real-world urban traffic environments—including not only common cross intersections but also T-shaped, L-shaped, angled, and multi-branch intersections—significant differences exist in the number of observable lanes, directional distribution, and traffic flow structure. Therefore, the state dimensions of each intersection may vary. Consequently, the following configurations are used: X-shaped intersections have 4 approach directions, totaling 12 entering lanes; T-shaped intersections have 3 approach directions, totaling 9 entering lanes; and L-shaped intersections have 2 approach directions, totaling 6 entering lanes. Specifically, it is expressed as follows:

[0049]

[0050]

[0051] in, This represents the observation state information acquired by the intersection agent of the upper-level controller, where n represents the nth intersection. Indicates the number of observable lanes. Indicates the queue length. Indicates the speed of the vehicle. This indicates the number of vehicles entering the lane.

[0052] When constructing the overall state, the upper-level controller needs to uniformly encode or pad and align heterogeneous states to form structurally compatible inputs. Then, the states of all individual intersections are combined to obtain the overall road network state. , represented as:

[0053]

[0054] This state will then be used for training the subsequent upper-level controller.

[0055] action : Indicates the selection of traffic light phase. Due to the diverse types of intersections, the phase design must be adapted to their topology.

[0056] An X-shaped intersection typically includes four main phases (such as east-west straight traffic and north-south left turns) and a CAV-specific phase, for a total of five phase combinations.

[0057] Because T-junctions lack an inbound direction, the number of phases and conflict structures are different, requiring the reduction or merging of some phases, resulting in a total of 4 phase combinations.

[0058] An L-shaped intersection has only two perpendicular entrances, and its phase logic is more simplified, with a total of three phase combinations.

[0059] Therefore, the action space of the upper-level controller is dynamically configured according to the intersection type. Each intersection has its own dedicated set of phases, ensuring that the control strategy is physically feasible and safe, as shown below:

[0060]

[0061]

[0062] Where I represents the number of phase combinations.

[0063] reward function It consists of the average green wave travel time of all vehicles in the road network and the overall throughput of the road network, and is expressed as:

[0064]

[0065] Where totalcars represents the total number of active vehicles in the road network at time t. This indicates the green wave travel time for vehicles on the road. This represents the number of vehicles that have completely left the road network at the current moment. 'a' and 'b' represent the weights of the green wave passage and throughput factors. The first term in the reward function formula is the average green wave passage time (in seconds) for all vehicles on the road network, i.e., the cumulative travel time for each vehicle to continuously pass through multiple intersections on the main road without stopping or waiting. This indicator directly reflects the effect of signal coordination control on improving driving smoothness and travel efficiency. The second term in the reward formula is the overall throughput of the road network, defined as the total number of vehicles leaving the simulated road network during the current control cycle (i.e., during one signal phase execution period) minus the number of vehicles newly entering the road network. This is used to measure the system's ability to guide and dissipate traffic flow. The coefficients 'a' and 'b' are adjustable weights, satisfying a + b = 1 and a, b ∈ [0, 1], and can be dynamically adjusted according to actual traffic management objectives.

[0066] The specific methods for the upper-level controller to select the optimal signal phase using a multi-agent reinforcement learning algorithm include:

[0067] Deploy an intelligent agent at each intersection in the road network;

[0068] Each agent inputs the local observation state of the intersection it is located into its individual neural network and outputs the local action value;

[0069] The QMIX network is used to mix the local action value of each agent with the global state to generate a joint action value function, and the joint action value function is used as the target for centralized training.

[0070] After training, each agent performs distributed phase decision-making based on the action set output by the QMIX network.

[0071] Specifically, in the multi-intersection traffic signal control task, each signal-controlled intersection is modeled as an intelligent agent. Each agent i can only acquire local traffic observation information of its intersection at decision time t. The values ​​are then input into the individual action value network to estimate the local action value function corresponding to different signal control actions. During the distributed execution phase, each agent at each intersection independently selects a signal control action based on its own local action value function using an ε-greedy strategy. The actions of all agents (…) , , ..., These factors together constitute a joint signal control strategy and act upon the traffic environment. The environment updates its traffic state based on these joint actions and calculates a global reward reflecting the overall traffic performance of the network, forming a joint experience sample. The data is stored in a centralized experience replay pool. During the centralized training phase, QMIX introduces a Mixing Network to fuse the local action value functions of each agent with the global traffic state, constructing a joint action value function. The Mixing Network imposes monotonicity constraints on the parameters, ensuring that the joint action value function is monotonically increasing with respect to the local action value functions. This ensures that, during decentralized execution, each agent can independently maximize its own action value to achieve global traffic performance optimization. During training, the individual network and the Mixing Network parameters of each agent are jointly updated by minimizing a loss function based on temporal differences, and a target network mechanism is employed to improve training stability.

[0072] The lower-level controller allocates right-of-way to vehicles in the CAV-dedicated lane during the CAV-dedicated phase. Its state, actions, and reward functions are defined as follows:

[0073] state The Discrete Traffic State Coding (DTSE) is used, consisting of a matrix of vehicle position, speed, path, and release priority, represented as follows:

[0074]

[0075] action :one A binary vector, used to allocate the right-of-way for vehicles before the stop line, is represented as:

[0076]

[0077] reward function The evaluation factor combines three elements: the number of vehicles passing through, queue length, and release priority, and is expressed as follows:

[0078] ;

[0079] in, Let be the number of vehicles passing through at time t. The number of booms in the direction of travel. For the number of lanes, Let be the queue length at time t. Let t be the priority for allowing passage. , , For weights.

[0080] During training, the AI ​​performs the following actions: First, it acquires the traffic state in the environment at time t, and uses this traffic state as the state at time t. The traffic state is encoded using discrete traffic state coding, including a position matrix, speed matrix, path matrix, and release priority matrix. Allocating right-of-way for vehicles in the CAV dedicated lane is considered an action, and the state is determined using an ε-greedy strategy. Action at time t ; Perform the action and obtain the execution of the action. The reward at time t is calculated based on the subsequent vehicle passage data, queue length, and release priority. .

[0081] This invention proposes a multi-agent reinforcement learning-based hybrid traffic control method for various intersection types, targeting urban road networks composed of cross, T-junction, and L-junction types. It effectively addresses the mixed traffic environment where human-driven vehicles and autonomous vehicles coexist. To address the shortcomings of existing single-intersection control strategies, which struggle to balance regional traffic coordination and are prone to causing localized congestion, this invention utilizes multi-agent reinforcement learning algorithms (such as QMIX) to jointly optimize the signal phases of multiple adjacent intersections. This overcomes the limitations of isolated control and improves the overall operational efficiency of the road network.

[0082] Meanwhile, addressing the contradiction between the uncontrollable behavior of autonomous vehicles (HVs) and the limited potential for CAV collaboration, this invention introduces dedicated CAV lanes and dedicated signal phases to physically isolate the two types of vehicles and reduce the risk of interaction conflicts. Based on this, a macro-micro dual-layer control architecture is constructed: the upper-layer controller coordinates multiple intersections through signal timing to achieve green wave passage on main roads; the lower-layer controller coordinates the trajectory and guides the speed of autonomous vehicles within the dedicated CAV phases, fully leveraging their controllability advantages. This multi-layered hybrid mechanism not only significantly extends the green wave passage time for vehicles to continuously pass through green lights but also effectively improves the overall throughput in multi-intersection areas.

[0083] In summary, the embodiments of this invention deeply integrate multi-agent reinforcement learning with hierarchical collaborative control, taking into account security, efficiency and scalability. It is applicable to typical scenarios such as smart cities, vehicle-road cooperation and traffic signal cloud control platforms, and provides a solution for intelligent urban traffic management in high-proportion mixed traffic environments that combines theoretical innovation and engineering implementation value.

[0084] like Figure 2As shown, real urban road networks contain various intersection types (cross, T, L, etc.), with varying conflict point distributions and traffic demands. Therefore, a flexible timing system of five main green light phases plus corresponding yellow light phases was designed: Phases 1–4 serve ordinary lanes, corresponding to pedestrian and vehicle flows from different directions; Phase 5 is a dedicated phase for CAVs (Cars and Vehicles), allowing CAVs to pass through the intersection without conflict only when this phase is active, thus releasing their collaborative control potential while ensuring safety. Different types of intersections have different phase settings, such as… Figure 3 .like Figure 4 As shown, in a typical three-lane layout, the dedicated lane for CAVs is located in the middle lane, allowing CAVs to share the right-of-way when turning right, going straight, or turning left. This adopts a free lane mode, not restricted by traditional fixed functional divisions, significantly improving route flexibility and traffic efficiency. The two ordinary lanes on either side maintain a fixed lane direction mode, requiring HVs to travel in their designated lanes to ensure predictability and operational order. Figure 5 As shown, there is a significant traffic flow coupling effect between adjacent intersections: the release strategy of the preceding intersection directly affects the queuing and congestion status of the downstream intersection. This invention abandons the isolated control approach and constructs a regional-level multi-intersection collaborative optimization mechanism. Through multi-agent reinforcement learning algorithms (such as QMIX), it jointly decides the phase switching sequence and duration of each intersection to achieve globally optimal scheduling. Figure 6 As shown, this collaborative control ultimately serves the goal of green wave traffic on main roads: by precisely adjusting the signal cycle and phase difference at adjacent intersections, vehicles can pass through multiple green lights consecutively, significantly reducing the number of stops and travel delays, and improving the travel experience and network efficiency. Figure 7 As shown, the embodiment of the present invention adopts a two-layer control structure: the upper-layer controller is responsible for macro-level decision-making: based on real-time traffic flow, queue length, historical data and other status information, it uses multi-agent reinforcement learning to select the optimal signal phase scheme and dynamically adjusts the signal light status of each intersection; the lower-layer controller intervenes during the activation of the CAV dedicated phase: based on vehicle-road cooperative information, it implements micro-level cooperative control of the CAV fleet to ensure efficient, safe and conflict-free passage.

[0085] In summary, the method of this invention not only effectively mitigates the safety risks caused by the uncertainty of human and vehicle behavior in mixed traffic environments, but also achieves simultaneous improvement in two key performance indicators by introducing physically isolated CAV dedicated lanes, setting dedicated CAV signal phases, constructing a multi-layered control architecture that coordinates macro and micro levels, and integrating a collaborative decision-making mechanism based on multi-agent reinforcement learning: on the one hand, it significantly extends the green wave passage time of vehicles on main roads, improving traffic efficiency; on the other hand, it greatly increases the overall throughput of the road network, enhancing the carrying capacity and guidance capacity of the regional traffic system.

[0086] Another embodiment of the present invention provides an electronic device, which includes a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions to execute the method described in the first embodiment above.

[0087] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0088] Input devices may include touchpads, microphones, etc., and output devices may include displays (LCDs, etc.), speakers, etc.

[0089] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0090] In specific implementations, the processor, input device, and output device described in the embodiments of the present invention can execute the implementation methods of the method embodiments described in the embodiments of the present invention, which will not be repeated here.

[0091] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in the first embodiment above.

[0092] The computer-readable storage medium can be an internal storage unit of the terminal described in the foregoing embodiments, such as the terminal's hard drive or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A multi-agent reinforcement learning-based method for mixed traffic control at various intersection types, characterized in that, include: For different types of intersections, dedicated CAV lanes and regular lanes are set up. The dedicated CAV lanes adopt a free lane mode, while the regular lanes adopt a fixed lane mode. Different signal light phases are configured for each type of intersection. A two-layer control structure is constructed, which includes an upper-layer controller and a lower-layer controller; The upper-level controller selects the optimal signal phase and dynamically adjusts the traffic light status of each intersection based on the status information of each intersection using a multi-agent reinforcement learning algorithm. When the dedicated CAV phase is activated, the lower-level controller uses a reinforcement learning algorithm to allocate right-of-way and coordinate control of CAVs in the dedicated CAV lane, and to guide the speed and coordinate the trajectory of the CAV convoy.

2. The method as described in claim 1, characterized in that, The specific method for selecting the optimal signal phase using a multi-agent reinforcement learning algorithm includes: Deploy an intelligent agent at each intersection in the road network; Each agent inputs the local observation state of the intersection it is located into its individual neural network and outputs the local action value; The QMIX network is used to mix the local action value of each agent with the global state to generate a joint action value function, and the joint action value function is used as the target for centralized training. After training, each agent performs distributed phase decision-making based on the joint action set output by the QMIX network.

3. The method as described in claim 2, characterized in that, Before each agent inputs the local observation state of its intersection into its individual neural network, the process also includes: performing dimensionality unification preprocessing based on the intersection type.

4. The method as described in claim 1, characterized in that, The status information is the average queue length and average vehicle speed of all entering lanes at a single intersection. The status dimensions are configured differently according to the intersection type. For example, a cross-shaped intersection has 12 entering lanes, a T-shaped intersection has 9 entering lanes, and an L-shaped intersection has 6 entering lanes. When constructing the overall status, the upper-level controller performs unified encoding or padding alignment on the heterogeneous statuses.

5. The method as described in claim 4, characterized in that, The specific details of configuring differentiated signal phases for different types of intersections include: The cross-shaped intersection is equipped with 4 HV main phases and 1 CAV dedicated phase, for a total of 5 phase combinations; The T-junction is equipped with 3 HV main phases and 1 CAV dedicated phase, for a total of 4 phase combinations; The L-shaped intersection is equipped with two main HV phases and one dedicated CAV phase, for a total of three phase combinations.

6. The method as described in claim 5, characterized in that, The reward function of the upper-level controller includes the average green wave travel time of all vehicles in the road network and the overall throughput of the road network. The weights corresponding to the average green wave travel time and the overall throughput of the road network are dynamically adjusted according to the actual traffic management objectives.

7. The method as described in claim 1, characterized in that, The lower-level controller uses a DDQN network as a reinforcement learning algorithm. It determines actions through an ε-greedy strategy. These actions are represented by binary vectors and are used to allocate the right-of-way for CAVs before the stop line on the CAV-dedicated lane.

8. The method as described in claim 7, characterized in that, The state of the lower-level controller adopts discrete traffic state coding, including vehicle position, speed, path and release priority matrix. The reward function of the lower-level controller comprehensively considers three evaluation factors: the number of CAVs passing through, the queue length of the CAV dedicated lane and the CAV release priority.

9. An electronic device, comprising: The processor, input device, output device, and memory are interconnected, the memory being used to store a computer program, the computer program including program instructions, characterized in that the processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8.

Images