A method and system for regional traffic signal timing based on reinforcement learning

Abstract

This invention relates to a regional traffic signal timing method and system based on reinforcement learning, belonging to the field of traffic signal timing processing technology. After extracting intersection and traffic light components from regional traffic environment data and vehicle trajectory data, the invention determines the regional traffic signal timing task based on these components. Then, a regional traffic waiting model is constructed based on the task. Finally, a multi-agent reinforcement learning algorithm is used to train the model, and an optimized policy-based NAC algorithm is combined to coordinate and optimize the policies of each agent in the region. An RNN (Recurrent Neural Network) is introduced to process the optimization results to obtain the regional traffic signal timing scheme, thereby solving the curse of dimensionality problem in existing traffic signal timing processes.

Description

Technical Field

[0001] This invention relates to the field of traffic signal timing processing technology, and in particular to a regional traffic signal timing method and system based on reinforcement learning. Background Technology

[0002] Currently, several types of traffic signal timing methods have been developed worldwide. In terms of control scope, they have evolved from single-point control to arterial line control, and now to regional control. Methods have also evolved from initial statistical schemes to new schemes using various artificial intelligence algorithms for learning and control.

[0003] 1. Historical Data Statistical Forecasting Methods

[0004] Historical data statistical prediction methods have been used in traffic signal timing algorithms for the longest time. On the one hand, due to the limited means of collecting traffic flow data in the past, only simple data such as the number of vehicles passing through intersections could be obtained, and few features could be extracted from the data itself. On the other hand, methods based on historical data statistical prediction are simple to operate. By statistically analyzing traffic flow over a certain period, accumulating and solving for the maximum and minimum values ​​in each lane direction, establishing a traffic flow mathematical model, and performing corresponding threshold detection, a preliminary traffic signal timing scheme for a single intersection can be obtained. Finally, by uniformly coordinating the timing cycles of intersections within the region, the regional traffic signal timing result can be obtained. Although historical data statistical prediction methods are simple to apply and have good implementation results, with the diversification and intelligence of traffic flow data collection, simple traffic flow data prediction alone is no longer sufficient to support regional traffic signal timing. It does not consider the correlation between various intersections, many features of traffic flow are not extracted, and there is a lack of simulation of real traffic conditions.

[0005] 2. Value-based reinforcement learning algorithms

[0006] In recent years, with the development of artificial intelligence technology, more and more AI algorithms have been widely applied in the transportation field. Among them, reinforcement learning-based methods have demonstrated their superiority in traffic signal control and timing. Reinforcement learning can adjust the actions taken by agents based on environmental feedback. It requires less prior environmental knowledge, can adapt to real-time changes in traffic conditions, and has better traffic interpretability. Reinforcement learning is characterized by being "data-driven, self-learning, and model-free." Its key process lies in the cyclical process of an agent taking actions to change its state, obtain rewards, and interact with the environment. The goal of reinforcement learning is to maximize long-term future rewards, that is, to solve for and optimize the reward function to maximize it. By linking multiple agents, it gradually expands from local optimization to global optimization, finding the optimal or near-optimal solution for global optimization of multiple intersections within a region. This is a direction that multi-agent reinforcement learning excels at.

[0007] Early multi-agent reinforcement learning methods for traffic signal control were limited by the constraints of data collection capabilities at the time, resulting in small volumes of traffic data and hindering widespread attention. Furthermore, these methods were only applicable to multi-intersection scenarios in smaller-scale urban areas. In implementation, combining evaluation functions and Q-learning algorithms for traffic signal control at multiple intersections relied on describing the traffic state at each intersection. Due to the complexity of urban traffic conditions, this method suffered from the typical curse of dimensionality problem in regional traffic systems. The dimensionality explosion problem is a significant concern in regional traffic signal timing. As the number of intersections increases and the route space grows exponentially, the curse of dimensionality becomes even more pronounced, making this method impractical for large traffic networks. Other researchers have combined reinforcement learning with nonlinear estimation, function approximation, or neural networks, but these methods suffer from problems such as excessively long model learning and prediction times and potential compromised convergence.

[0008] Another problem arising from traffic signal timing is intersection phase coordination. Many methods are based on independent agents and apply parallel computing to multi-intersection urban areas; some directly apply reinforcement learning to multiple intersections to address the random traffic patterns generated by urban traffic. These methods consider information such as traffic flow and waiting time at adjacent intersections during urban traffic signal decision-making, but lack coordination mechanisms and fail to rationally utilize the relationships between intersections. To solve this problem, multi-agent reinforcement learning is generally used to automatically discover more efficient regional signal controllers. In this approach, each agent controls one traffic light, extending to coordination between adjacent traffic lights using the Max-Plus algorithm. The Max-Plus algorithm is relatively simple, considering only coordination between adjacent intersections. When studying external coordination mechanisms for multi-agent reinforcement learning control of regional multi-intersections, the Max-Plus algorithm is also used for prediction; however, the Max-Plus algorithm has limitations, high computational complexity, and is only applicable to tree networks.

[0009] 3. Policy-based reinforcement learning algorithms

[0010] Policy gradient algorithms, another branch of reinforcement learning, primarily address the problem of action space discretization in deep reinforcement learning. Policy gradient algorithms are a more direct approach. For deterministic policies, the neural network directly outputs the policy function, indicating which action should be performed in a given state; for non-deterministic policies, it outputs the probability values ​​of various actions to be performed in that state. Some methods have proposed a traffic signal control method based on Q-learning. Q-learning stores value functions through a Q-value table. However, this method cannot adapt to complex environments, and large state spaces lead to slow storage and convergence issues.

[0011] The concept of policy gradient was first embodied in the NAC algorithm. Due to the high variance of gradient estimation, the PG method converges more slowly than value methods. The Natural Actor-Critic (NAC) method improves upon this by combining the PG method, natural gradient, value estimation, and least-squares time-difference Q-learning. The Deep Deterministic Policy Gradient (DDPG) algorithm is a variant of the NAC algorithm. It considers the state information of the entire road network for each intersection and implements a state-sharing mode. By estimating the interrelationships of intersections, it derives the globally optimal Q-value, enabling multiple agents to obtain each other's spatiotemporal information based on the global state, rationally adjust their own policies, and achieve collaborative optimization. Based on traffic flow information of different phases and lanes, it intelligently decides the timing period, phase order, and duration of each phase, solving the suboptimal timing problem caused by the discrete action decision space of existing intelligent algorithms.

[0012] In summary, although reinforcement learning algorithms have made rapid progress in the application and effectiveness of regional signal timing problems, there is still room for improvement and optimization for specific issues such as regional phase coordination, multi-agent interaction, and the curse of dimensionality caused by changes in the scale of regional road networks. This is specifically reflected in the following two aspects:

[0013] Firstly, there is in-depth research on phase coordination and multi-agent interaction between intersections in a region. In a defined traffic area, the distance between intersections is generally no more than 500 meters. Vehicles pass through intersections according to their travel trajectories, resulting in spatiotemporal correlations between adjacent intersections, making it a modeling problem that progresses from points to lines to surfaces. The traffic flow characteristics of intersections are not isolated; their spatiotemporal correlations need to be carefully considered, as congestion at a single intersection can impact the surrounding traffic area; conversely, optimizing a single intersection may worsen congestion at adjacent intersections. Many existing reinforcement learning algorithms, however, only consider their own intersection environment for each agent, achieving local intersection optimization, or fail to deeply consider the multi-phase coordination and control between intersections. This leads to vehicles having shorter waiting times at some intersections, quickly passing through during green light phases, but experiencing excessively long waiting times at other intersections within the region, resulting in poor learning outcomes. Furthermore, many traffic simulation processes use models such as random vehicle arrival and transition delay models, which to some extent ignore the internal characteristics of actual regional traffic flow data, failing to extract internal traffic features from real data. Therefore, by designing suitable states, actions, and rewards for multiple agents, the algorithm will enable multiple agents to obtain each other's spatiotemporal state characteristics based on the global state of multiple intersections in the region, adjust their own policy functions, achieve collaborative optimization of intersections in the region, and maximize the global reward of the region through communication, interaction, and updates between agents.

[0014] Secondly, it addresses the curse of dimensionality caused by changes in the scale of the regional road network. As the number of intersections increases within a region, vehicles have a wider range of route choices, and the spatiotemporal characteristics of traffic data become more high-dimensional, making the curse of dimensionality a key issue to address. The curse of dimensionality refers to the phenomenon where an algorithm's performance increases with the number of features, but after a certain threshold, performance declines instead of improving. This leads to excessively long model learning and prediction times and compromises convergence. Existing research on regional traffic signal timing simulations often focuses on small-scale regional intersections with 2 to 4 intersections, simulating a limited number of vehicles and thus avoiding the curse of dimensionality. However, in cases of highly complex regional road networks, even high-performing algorithms exhibit the curse of dimensionality.

[0015] Therefore, providing a new method or system for regional traffic signal timing, which introduces a recurrent neural network based on the high-dimensional and multivariable characteristics of the established model, and optimizes the network update process of the agent in each round, thereby solving the curse of dimensionality problem, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0016] The purpose of this invention is to provide a regional traffic signal timing method and system based on reinforcement learning, which can realize the collaborative optimization process of signal timing among multiple intersections in a region, and solve the dimensionality curse problem caused by the increase in the scale of the regional road network.

[0017] To achieve the above objectives, the present invention provides the following solution:

[0018] A reinforcement learning-based method for regional traffic signal timing includes:

[0019] Data extraction was performed on regional traffic environment data and vehicle trajectory data to obtain the constituent elements of intersections and traffic lights;

[0020] The regional traffic signal timing task is determined based on the intersection and traffic light components.

[0021] A regional traffic waiting model is constructed based on the aforementioned regional traffic signal timing task.

[0022] The regional traffic waiting model is trained using a multi-agent reinforcement learning algorithm. The optimized policy-based NAC algorithm is then used to coordinate and optimize the policies of each agent in the region. An RNN recurrent neural network is introduced to process the optimization results generated during the coordination and optimization process to obtain the regional traffic signal timing scheme.

[0023] Preferably, the step of constructing a regional traffic waiting model based on the regional traffic signal timing task specifically includes:

[0024] Construct a regional traffic environment simulation model;

[0025] Extract vehicle trajectory data features; the vehicle trajectory data features include the traffic light cycle, the green light ratio, and the traffic light phase difference;

[0026] The vehicle trajectory data features are incorporated into the regional traffic environment simulation model using a Python programming algorithm to generate the regional traffic waiting model.

[0027] Preferably, the regional traffic waiting model includes:

[0028] The road network environment configuration module is used to build and generate a basic environment model of various traffic elements within the area;

[0029] The data acquisition module is used to collect vehicle driving data within the area and to extract the spatiotemporal features of vehicle trajectories.

[0030] The traffic light configuration module is used to perform basic configuration of the traffic lights at each intersection in the area; the basic configuration includes: the color of the traffic lights, the cycle duration, and the phase sequence.

[0031] The evaluation index output module is used to determine the evaluation index data of the regional traffic environment.

[0032] Preferably, the method employs a multi-agent reinforcement learning algorithm to train the regional traffic waiting model, combines it with an optimized policy-based NAC algorithm to complete the coordination optimization process of the policies of each agent in the region, and introduces an RNN recurrent neural network to process the optimization results generated during the coordination optimization process to obtain a regional traffic signal timing scheme, specifically including:

[0033] The basic elements for designing multi-agent systems include: state, action, and reward.

[0034] A multi-agent collaborative optimization model is set up based on the basic elements of the multi-agent;

[0035] The multi-agent collaborative optimization model is used to update the state information of lanes at each intersection in the region;

[0036] The updated lane status information of each intersection is aggregated to obtain aggregated status information;

[0037] The aggregated state information is filtered using an attention mechanism to obtain filtered state information;

[0038] An optimized policy-based NAC algorithm is used to generate a policy function based on the filtered state information; the policy function is a mapping from the state set to the action set; the NAC algorithm includes: an Actor network and a Critic network;

[0039] The strategy function is input into an RNN (Recurrent Neural Network) to obtain the regional traffic signal timing scheme.

[0040] Preferably, the optimized policy-based NAC algorithm generates a policy function based on the filtering state information, specifically including:

[0041] Each agent updates the parameters of the Actor network according to the corresponding objective function by maximizing the accumulated rewards in the future;

[0042] After the Actor network updates its parameters, it outputs the next signal cycle data after selecting an action of a certain strategy. The Critic network evaluates the next signal cycle data, so that the strategy function output by the Actor network is continuously coordinated and optimized. The next signal cycle data includes: period, phase order and phase duration factor.

[0043] Preferably, the parameters of the Critic network are updated by minimizing the loss function.

[0044] Preferably, the features output at the current time step in the RNN recurrent neural network are used as the input features at the next time step.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] The regional traffic signal timing method based on reinforcement learning provided by this invention extracts intersection and traffic light components from regional traffic environment data and vehicle trajectory data. Based on these components, a regional traffic signal timing task is determined. Then, a regional traffic waiting model is constructed based on the task. Finally, a multi-agent reinforcement learning algorithm is used to train the model. An optimized policy-based NAC algorithm is then used to coordinate and optimize the policies of each agent in the region. An RNN (Recurrent Neural Network) is introduced to process the optimization results generated during the coordination process, resulting in a regional traffic signal timing scheme. This solves the curse of dimensionality problem in existing traffic signal timing methods.

[0047] Furthermore, the present invention also provides a regional traffic signal timing system based on reinforcement learning, the system comprising:

[0048] Data collectors are used to collect regional traffic environment data and vehicle trajectory data;

[0049] A memory, connected to the data acquisition unit, is used to store the regional traffic environment data, the vehicle trajectory data, and software control instructions; the software control instructions are used to implement the reinforcement learning-based regional traffic signal timing method provided above.

[0050] The processor, connected to the memory, is used to retrieve and execute the software control instructions to obtain the regional traffic signal timing scheme.

[0051] Preferably, the processor includes:

[0052] The element extraction module is used to extract intersection and traffic light components from regional traffic environment data and vehicle trajectory data.

[0053] The timing task determination module is used to determine the regional traffic signal timing task based on the intersection and traffic light components.

[0054] A waiting model construction module is used to construct a regional traffic waiting model based on the regional traffic signal timing task.

[0055] The timing scheme generation module is used to learn and train the regional traffic waiting model using a multi-agent reinforcement learning algorithm, and to complete the coordination and optimization process of the strategies of each agent in the region by combining the optimized policy-based NAC algorithm. An RNN recurrent neural network is introduced to process the optimization and coordination results generated in the coordination and optimization process to obtain the regional traffic signal timing scheme.

[0056] Preferably, the memory is a computer-readable storage medium.

[0057] Since the technical effects achieved by the reinforcement learning-based regional traffic signal timing system provided by this invention are the same as those achieved by the reinforcement learning-based regional traffic signal timing method provided above, they will not be described again here. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of the reinforcement learning-based regional traffic signal timing method provided by the present invention;

[0060] Figure 2 This is a schematic diagram of multi-agent cooperative optimization provided in an embodiment of the present invention;

[0061] Figure 3 This is an implementation architecture diagram of the reinforcement learning-based regional traffic signal timing method provided by the present invention;

[0062] Figure 4 This is a schematic diagram of a 3×3 intersection area provided in an embodiment of the present invention;

[0063] Figure 5 This is a simulated traffic flow statistics chart of a 3×3 intersection area provided in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of a 4×4 intersection area provided in an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of the structure of the regional traffic signal timing system based on reinforcement learning provided by the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The purpose of this invention is to provide a regional traffic signal timing method and system based on reinforcement learning, which can realize the collaborative optimization process of signal timing among multiple intersections in a region, and solve the dimensionality curse problem caused by the increase in the scale of the regional road network.

[0068] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Example 1

[0070] like Figure 1 As shown, the reinforcement learning-based regional traffic signal timing method provided by this invention includes:

[0071] S1: Data extraction is performed on regional traffic environment data and vehicle trajectory data to obtain intersection and traffic light components. In this step, the regional traffic environment data consists of infrastructure data composed of intersections and lanes. For example, the traffic area studied in this embodiment can be a 9-intersection regional road network and a 16-intersection regional road network: the 9-intersection regional road network contains 9 intersections, presenting a 3×3 rectangular area. The 16-intersection regional road network contains 16 intersections, presenting a 4×4 rectangular area. The regional road network is connected by lanes between intersections, including lanes within the area and entrance and exit lanes at the edge of the area. Each intersection data record includes information such as the intersection number, location, and traffic light identification. Each lane data record includes information such as the lane number, turning direction, speed limit, length, type, start point, and end point (represented by intersection type data). Vehicle trajectory data describes the traffic light-controlled behaviors of vehicles moving and stopping within the area, and is an important data source for algorithm training and testing. Starting from the departure time, the vehicle's trajectory passes through various intersections and related lanes within the area, and finally leaves the area. The vehicle trajectory data mainly includes vehicle type, length, currently passed intersection, currently passed lane, waiting time at the current intersection, whether it passes through the current intersection, and the start and end times of the vehicle trajectory.

[0072] S2: Determine the regional traffic signal timing task based on intersections and traffic light components. Based on step S1, the task is defined as a given region G = <Ι,E> (Ι is the set of intersections, E is the set of lanes). Based on vehicle trajectory data T, a regional vehicle waiting model M is generated, and algorithm f(·) is implemented. Algorithm f(·) can achieve coordinated control of regional traffic lights, ultimately generating the regional traffic signal timing task.

[0073] S3: Constructing a regional traffic waiting model based on regional traffic signal timing tasks. This step of building a regional traffic waiting model mainly consists of two parts: constructing a regional traffic environment simulation model and extracting vehicle spatiotemporal features. Combining these two parts generates the regional traffic waiting model. Specifically, it includes the following steps:

[0074] S31: Construction of regional traffic environment simulation model.

[0075] A regional traffic environment simulation model is established based on the urban traffic simulation software (SUMO) and Python programming algorithms. The basic environment of the regional road network is established in SUMO, and the regional traffic environment is controlled and optimized using the Python language. Information is exchanged between the two through the TraCI traffic control interface provided by SUMO. The model can access the running regional traffic environment simulation model, retrieve the values ​​of the simulation objects, and manipulate their behavior online.

[0076] The main contents of the regional traffic environment simulation model design are as follows: First, the simulation model can construct the basic environment of SUMO based on regional traffic environment data, generating traffic environment elements such as intersections, lanes, and traffic lights within the region. Based on this environment, vehicle trajectory data is subsequently added through Python programming algorithms to collect and extract corresponding spatiotemporal features, generating the final regional traffic waiting model for coordinated control of regional traffic timing. Furthermore, the regional traffic waiting model encapsulates the basic environment of SUMO. When subsequent research algorithms perform generalized operations on inexperienced traffic, changes to the regional road network do not affect the basic construction of the modeling algorithm, facilitating subsequent algorithm research. Finally, the regional traffic waiting model connects to a database to calculate the average waiting time and average queue length evaluation indicators required by the regional traffic signal timing algorithm. The four functional modules of the regional traffic waiting model design are: regional traffic road network environment configuration module, data acquisition module, traffic light configuration module, and evaluation indicator acquisition and output module. A detailed description of each module is as follows:

[0077] 1) Road network environment configuration module:

[0078] The road network environment configuration module is the most basic and core module in the regional traffic waiting model. It primarily constructs the various basic traffic elements within the region to generate a basic environment model. First, it determines the location of each intersection and the distances between them. Generally, the distance between adjacent intersections should not exceed 600 meters; in this embodiment, the simulated distance between adjacent intersections is set between 200 and 500 meters. Second, it connects the intersections with lanes, configuring a two-way three-lane configuration between any two intersections. The lanes are configured with corresponding traffic flow directions, stop line positions at intersections, and turning directions. Finally, it determines the boundaries of the simulation area and configures the area's entrance and exit lanes.

[0079] 2) Data acquisition module:

[0080] The data acquisition module is primarily responsible for collecting data on vehicles traveling within the area. As one of the basic acquisition functions, the subsequent vehicle trajectory spatiotemporal feature extraction function will also call upon this data acquisition module. This data acquisition module simulates a data collector in actual traffic, setting up collectors at various intersections and lanes to collect vehicle behavior in real time, mainly including collecting data on vehicle speed, traffic flow, and the number of vehicles waiting at intersections, and storing the data in the corresponding database.

[0081] 3) Traffic light configuration module:

[0082] This traffic light configuration module performs basic configuration of the traffic lights at the intersection. Based on the designed four-phase signal control, it configures the traffic light colors, basic cycle duration, and basic phase sequence. As a basic traffic light configuration function, this module will be called and optimized by subsequent multi-agent reinforcement learning algorithms. In SUMO, the TraCI interface can be called to retrieve traffic light variables, returning the status of variables or values ​​queried in the last simulation step. Additional parameters of the traffic lights can also be obtained, including traffic light number and color (0-red, 1-yellow, 2-green).

[0083] 4) Evaluation index output module:

[0084] The evaluation index output module outputs the collected and calculated relevant evaluation index data. As an overall evaluation of the regional traffic environment, the evaluation index should also be encapsulated as a function in the basic environment. Its collection and calculation process does not change with changes in regional traffic conditions, and the output evaluation index data serves as an important basis for algorithm analysis and comparison.

[0085] The primary design purpose of this evaluation index output module is to encapsulate the basic environment of regional traffic, facilitating subsequent algorithm and other function calls. The basic environment model constructed in this embodiment is applicable to regions of different sizes, traffic conditions with varying traffic volumes, and different types of regional traffic timing algorithms. When subsequent research involves various changes, there is no need to alter the modeling method of the basic environment model, demonstrating good scalability.

[0086] S32: Feature extraction of vehicle trajectory data.

[0087] The trajectory data includes the vehicle's path from its departure time, traversing each lane within the area, proceeding according to the traffic light timing scheme at each intersection, and finally leaving the area. This data contains rich temporal and spatial features, which are crucial training elements for subsequent reinforcement learning algorithms. The temporal and spatial characteristics of the vehicle trajectory data are intertwined with the traffic light timing control process within the area. As the vehicle is located at different positions within the area at different times, its characteristics, besides following a predetermined route, are mainly reflected in three temporal parameters: traffic light period, green light ratio, and phase difference.

[0088] 1) Cycle

[0089] The cycle is a key parameter in signal timing design. The cycle length is the sum of the durations of each phase at the intersection. A cycle that is too long will increase the average waiting time for vehicles, while a cycle that is too short will cause vehicles to frequently be waiting. In this embodiment, the cycle settings differ for each intersection within the area and are changed in real time based on training results, adjusting the duration and order of each phase within the cycle to make the signal timing more flexible. However, the cycle time must be limited by upper and lower thresholds, which are:

[0090] T min ≤T i ≤T max ,i∈Ι

[0091] In the formula, i represents the intersection number within the area, I is the set of intersection numbers, and T... i Let T be the cycle time of the traffic light at the i-th intersection within the region. min T represents the minimum value between flowers in a periodic cycle. max This represents the maximum value of the period time, where T min and T max You can select 10 seconds and 160 seconds respectively.

[0092] 2) Green Credit Ratio

[0093] The green light ratio is a design parameter for different signal phases. In this embodiment, the simulated regional road network uses east-west lanes as the main arteries with high traffic volume, and north-south lanes as branch lines with lower traffic volume. This characteristic is reflected in the design of the green light ratio. The first and second phases in the east-west direction have longer durations, resulting in longer green light durations. The third and fourth phases in the north-south direction have shorter durations, resulting in shorter green light durations. In this embodiment, the signal timing scheme can, under certain calculation conditions, allow a phase to occupy a complete cycle; however, the maximum value of the green light ratio is limited.

[0094] λ ij ≤λ max ≤1, i∈Ι, j∈{1,2,3,4}

[0095] In the formula, j represents the traffic light phase number at the intersection. There are four phases in total, numbered 1, 2, 3, and 4. λ ij Let λ be the green ratio of the j-th phase of the traffic light at the i-th intersection. max This represents the maximum green credit ratio.

[0096] 3) Phase difference

[0097] Phase difference is a comprehensive reflection of the spatiotemporal characteristics of vehicle trajectory data and a key parameter considered in traffic light timing design. The subsequent discussion in this embodiment focuses on relative phase difference, which is the difference in the green light start time of a certain phase at adjacent intersections. A traffic light timing scheme based on phase difference coordination can effectively create green wave bands for vehicle passage between adjacent intersections, significantly reducing the average waiting time of traffic flow in the coordinated direction. For example, a vehicle traveling straight from intersection A to intersection B arrives at intersection A at time t1 when the green light for that straight-ahead phase is on, and arrives at intersection B at time t2. At this point, the coordinated phase ensures that the vehicle just happens to encounter the green light and can pass. This refers to the phase difference coordinated from intersection A to intersection B; similarly, there will be a phase difference in the opposite direction. The two coordinated phase differences satisfy:

[0098]

[0099] In the formula, The phase difference is coordinated from intersection A to intersection B; similarly, there will be a phase difference in the opposite direction. These are the numbers of the two intersections under this formula, where n is a positive integer.

[0100] Long Short-Term Memory (LSTM) networks are used to extract spatiotemporal features from vehicle trajectory data. The calculation methods for the parameters in the LSTM network are as follows:

[0101] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0102] in t =σ(W in ·[h t-1 ,x t ]+b in n)

[0103]

[0104]

[0105] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0106] h t =o t ·tanh(c t )

[0107] Where f represents the forget gate and in represents the input gate. The current cell's state is represented by , c represents the updated state of the current cell based on the current cell's state and the input at the current time step, o represents the output gate, h represents the hidden state at the current time step, t represents time, tanh(*) refers to the hyperbolic tangent function, and W... f Forget Gate Weights, W in For input gate weights, W o For output gate weights, W c For state weights, x t Let b be the input value at time t.f The offset coefficient of the forget gate, b in The offset coefficient of the input gate, b o The offset coefficient of the output gate, b c Here, σ is the state offset coefficient, σ is the Sigmoid activation function, and c is the state offset coefficient. t For long-term information, h t This is short-term information.

[0108] LSTM networks can process the spatiotemporal features of each data point in a vehicle trajectory and use the learned information for subsequent feature extraction of signal timing parameters. Data represented by feature vectors is input into the LSTM layer, which performs computations according to the temporal order of the data.

[0109] S33: Generation of regional traffic waiting models.

[0110] The regional traffic environment simulation model from step S31 and the vehicle trajectory data features extracted in step S32 are combined to generate a regional traffic waiting model. This regional traffic waiting model will be used in subsequent algorithm configurations to complete the implementation of a reinforcement learning-based regional traffic signal timing algorithm. The regional traffic waiting model connects and exchanges information with the previous regional traffic environment simulation model through the TraCI interface in SUMO. The regional traffic waiting model is generated as follows:

[0111] Based on the original modules, a regional signal parameter control module and a signal timing output module were added according to the extracted spatiotemporal characteristics of the data. For the regional signal parameter control module, based on the characteristics of the extracted signal timing parameters, a multi-agent reinforcement learning algorithm is applied to control three parameters at each intersection in the region: the signal cycle, the phase green ratio in each direction, and the phase difference between adjacent intersections, iteratively optimizing the waiting time function for vehicles within the region. For the signal timing output module, it receives the algorithm calculation results from the signal parameter control module and outputs real-time regional traffic signal timing schemes for each intersection and each phase, coordinating the passage of vehicles within the region.

[0112] S4. A multi-agent reinforcement learning algorithm is used to train the regional traffic waiting model. The optimized policy-based NAC algorithm is combined to complete the coordination and optimization process of the policies of each agent in the region. An RNN recurrent neural network is introduced to process the optimization and coordination results generated during the coordination and optimization process to obtain the regional traffic signal timing scheme.

[0113] Step S4, based on the multi-agent reinforcement learning process, specifically includes the following steps:

[0114] S41: Basic element design.

[0115] In this embodiment of the regional traffic signal timing problem, an agent is assigned to each intersection within the region to control the signal timing at that intersection. Multiple agents are distributed across various intersections within the region, and communication interaction between these agents and the environment is used to optimize the traffic signal timing for the entire region. In reinforcement learning, the three elements of the multiple agents—state, action, and reward—are designed first. Specifically:

[0116] 1) Status

[0117] The state is a quantitative representation of the regional road network environment observed by the agent at each intersection, and the state space should reflect the characteristics of the regional road network as completely as possible. In the regional traffic environment, traffic flow can reflect the real traffic situation in real time, and vehicle queue length can reflect the congestion situation in each direction and lane of the regional road network. Therefore, the local environmental state observed by the agent at intersection i at time t is first defined as ob. i,t :

[0118] ob i,t ={f_lane1,f_lane2,…,f_lane n_lane ,q1,q2,…,q n_lane}

[0119] In the formula, n_lane represents the total number of approach lanes at intersection i. Subsequent algorithm calculations and experimental statistics are based on the traffic conditions of each lane. f_lane represents the traffic flow of the corresponding lane. l represents the queue length of the corresponding lane. The road network state of the complete region includes observations of all agents and is given by the following observation vector:

[0120]

[0121] 2) Actions

[0122] At each moment, the agent receives the intersection's state information and needs to take corresponding actions, i.e., the corresponding timing scheme. The agent's action selection directly affects the effectiveness of the timing scheme. Many existing multi-agent-based timing algorithms typically use phase as the update frequency to select actions. However, phase changes occur in a discrete action space, which introduces quantization errors and reduces timing efficiency. To overcome the problems caused by the discrete action space, this embodiment uses an action factor related to the signal period. This action factor has a continuous action space and high timing flexibility, which is one of the innovations of this embodiment. The action of agent i at time t is defined as:

[0123] a i,t ={ω,p1,…,p k ,d1,…,d k} where ω is a factor that determines the duration of the next signal period. Based on the above restrictions on the upper and lower thresholds of the period time, ωT should be within [T min T max Between ] . k is the number of phases designed for the intersection. {p1, ..., p k The sequence order of d1, ..., d2 determines the order of phase changes within the next signal cycle. k The phase factor determines the duration of the phase in the next signal cycle. This continuous action space design also avoids frequent decision-making by the agent and makes better use of the characteristics of phase transition.

[0124] 3) Rewards

[0125] The reward is the information fed back to the agent from the regional traffic environment. Its definition guides the agent's learning direction and ultimately evaluates the algorithm's performance. The definition of reward in reinforcement learning should not be overly complex, otherwise it will lead to problems such as excessively long training processes and the model failing to converge. The core purpose of regional traffic signal timing is to reduce vehicle congestion and waiting time within the region, thereby improving traffic efficiency. Therefore, vehicle waiting time is chosen as the basis for the reward, set as the negative value of the waiting time function; the smaller the waiting time, the larger the reward. The reward function for agent i at time t is defined as:

[0126]

[0127] Where NI represents the total number of vehicles waiting at intersection i. n Let be the sum of the waiting times of the num-th vehicle at intersection i. ε is a constant, taken as ε = 0.2.

[0128] S42: Multi-agent cooperative optimization model design. In the method provided in this embodiment, based on the state, action, and reward of the agents designed in step S41, the algorithm completes the cooperative optimization process of multiple agents within the region: there are N agents in the region, and each agent has a corresponding mapping from state to action space, that is, N policy functions π = (π1, π2, ..., π...). N The local environmental state observed by each agent at time t is ob. i,t The agent receives state information from other agents and selects an action 'a' according to the policy function. i，t , will its own state s i，t Transmit to the environment and receive a reward from the environment's feedback. i，tIt's important to note that each agent only obtains local environmental state information, specifically the state information of the intersections adjacent to its own location, and does not acquire the global state. Therefore, the multi-agent reinforcement learning algorithm, through communication and updates between agents, enables multiple agents to acquire each other's spatiotemporal state features based on the global state of the region, adjust their own policy functions, achieve collaborative optimization of the regional intersections, and maximize the global reward. The reward is... Its formal definition is:

[0129] maxR((ob1,…,ob N ),(s1,…,s N ),(a1,…,a N ),(π1,…,π N )).

[0130] In the formula, max(*) is the maximum value function, R is the reward, ob is the agent's observation value, s is the agent's state, a is the action chosen by the agent, Π is the policy function, and N is the number of agents.

[0131] The multi-agent cooperative optimization model in this embodiment is as follows: Figure 2 As shown.

[0132] S43: Status update.

[0133] In this embodiment, multiple agents observe their respective local environmental states, aggregate these local states, and then consider the global information of the entire area. The states observed by the multiple agents are based on each lane of the intersection. According to the observation of lane traffic flow and queue length, each lane is calculated separately. Specifically, in each time step, a corresponding observation dictionary is set to store lane vehicle information, record the vehicle numbers entering the lane detector range, iterate through the states of these vehicles one by one, update the waiting time of each vehicle at the intersection, save the updated vehicle information, determine the updated waiting time of the vehicles, and update the intersection environmental state according to different situations.

[0134] S44: Aggregation of states.

[0135] After the lane state update algorithm updates the lanes, it is necessary to aggregate the state information of each lane at the intersection to lay the foundation for subsequent global information communication between multiple agents. This embodiment introduces an attention mechanism to integrate important state information within the regional space. The attention mechanism filters out the state information of agents that have a significant interactive influence on the current agent's strategy selection. The attention coefficient α between any two nodes i and j is... ij The calculation process is to calculate the product of the two. Assuming similarity between nodes, the results are normalized using the softmax function to obtain the weight coefficients s = softmax(key). i ·key j The attention coefficient is obtained by weighting the weight coefficients with the values ​​of their respective nodes and summing the results.

[0136]

[0137] S45: Strategy generation.

[0138] After observing and aggregating the corresponding states, the state set is mapped to actions according to the policy function. Here, represents the mapping from the state set to the action set: Π = s->a. This embodiment adopts a framework of centralized training and distributed execution. Each agent at an intersection maintains a Critic network for the NAC algorithm within a centralized local environment. The Critic network receives observations and actions from all agents within the region, forming joint state-action pairs learned by the algorithm. The basic state calculations of the agents are all carried out around each lane, first aggregating the lane states into the intersection state. The Critic network only considers the action information of each agent in its own basic environment, but the state information is integrated with global regional information through a spatial aggregation module. Therefore, each agent's Critic network fits the global value function of the entire regional road network, and each agent adjusts its local policy according to the policies of other agents to achieve a globally optimal policy.

[0139] The basic idea of ​​this embodiment is to use a deterministic strategy π i For each agent i, the action selected at the current moment is formally represented as: Among them, s i，t It is an aggregated vector of local environmental states generated by the lane state aggregation algorithm in step S44. The Actor network from the NAC algorithm represents the network parameters that generate the policy. In the Critic network, cooperative communication among multiple agents is considered, and the actions of other agents need to be taken into account during network updates. Based on the formula for calculating the policy gradient, it can be transformed into:

[0140]

[0141] The update process will Q i (s, a1, ..., a) n This incorporates multiple agents, thus reflecting the collaborative communication process among them.

[0142] S46: Training process.

[0143] The training process for each agent i includes the Actor network π.i and Critic network Q i And the corresponding initialization target network π′ i and Q′ i .

[0144] Each agent is based on a corresponding objective function. Update the parameters of the Actor network by maximizing the accumulated rewards in the future.

[0145]

[0146] The Critic network acts as a supervisor for the Actor network. After each action chosen by the Actor network according to its strategy, the period, phase sequence, and phase duration factor of the next signal cycle are evaluated by the Critic network, allowing the Actor network's strategy to be continuously coordinated and optimized. According to the Bellman equation: Q(s t a t )=Q(s t a t )+α[r t+1 +γmax a Q(s t+1 ,a)-Q(s t a t The parameters of the Critic network are updated by minimizing the loss function L(θ). The update formula is:

[0147] L(θ i )=E[(r i +γQ′ i (s′,a′)-Q i (s, a)) 2 ].

[0148] In addition, the parameters of the target network are updated by assigning different weight coefficients:

[0149]

[0150]

[0151] Furthermore, in step S4, this embodiment combines deep learning and reinforcement learning to solve the curse of dimensionality problem in regional traffic signal timing. Reinforcement learning has strong decision-making ability but poor perception ability. Conversely, deep learning has poor decision-making ability but strong perception ability. The advantages of both are complementary. In the process before reinforcement learning, a related neural network from deep learning is used to process the complex data. In the early stages of training, more actions need to be explored to expand the multiple possible outputs of the policy function. To solve the curse of dimensionality and avoid the possibility of blind exploration and bad actions in the early stages of reinforcement learning, a recurrent neural network (RNN) from deep learning is introduced, allowing the policy to reach its optimal state more quickly. The RNN can handle temporal data well; the output of each time step serves as the input for the next time step, making full use of the feature information from the previous time step. The input sequence x of the Actor network at time t... t The hidden layer sequence h of an RNN (Recurrent Neural Network) t and the final output layer's action sequence a t = (a1, ..., a n The calculation is as follows:

[0152] h t =f(W i ·x t +W h ·x t-1 )

[0153] a t =f(W o ·h t ),

[0154] Among the three matrices, W i The weight matrix W from the input layer to the hidden layer. h W is the weight matrix from hidden layer to hidden layer. o It is the weight matrix from the hidden layer to the output layer.

[0155] Therefore, the regional traffic waiting model established in step S3 is loaded into SUMO. Based on the design and implementation process of steps S3 and S4, agents are set up at each intersection in the environment, and the Actor network, Critic network, target network, and memory storage area of ​​each agent are randomly initialized. After initialization, a small loop is entered, using the initial state of the agent obtained above, and actions are selected according to the current policy. The policy function... The noise G is introduced in this paper, mainly to balance the exploration and exploitation behaviors of the agent in reinforcement learning as much as possible.

[0156] Furthermore, to avoid blind exploration within the vast action space of multiple agents, a recurrent neural network (RNN) is introduced to predict flow and prune and optimize the action space. As the agents learn, the initial exploration level is high, gradually decreasing in later stages, eventually reaching a balance between exploration and utilization. (Storing a copy (s)) t ,a t ,r t ,s t+1 The Critic network parameters are updated by minimizing the loss function, the Actor network is updated using the policy gradient method, and the target network is also updated. Finally, the regional traffic signal timing output is obtained. The complete flowchart is shown below. Figure 3 As shown.

[0157] Example 2

[0158] This embodiment mainly illustrates the advantages of the reinforcement learning-based regional traffic signal timing method provided in Embodiment 1 through simulation experiments.

[0159] This embodiment selects three types of regional traffic signal timing methods to compare the technical performance of the reinforcement learning-based regional traffic signal timing method proposed in Embodiment 1. The comparison methods range from the initial fixed-cycle timing to the traditional lane-sensing timing, and then to the reinforcement learning algorithm based on DQN. The specific algorithm descriptions are as follows:

[0160] 1) Fixed-Timing: A timing algorithm with a fixed signal period, phase sequence, and duration. The phase sequence is 30 seconds, 20 seconds, 30 seconds, and 20 seconds respectively, with a period of 100 seconds.

[0161] 2) Lane-Sensing-Timing: This is a traditional variable-cycle control method that changes the signal phase display time based on traffic flow data sensed by lane detectors. When a vehicle passes through a signal phase, the green light is maintained for a certain period of time, and the light turns red after 5 seconds when no vehicle is sensed.

[0162] 3) DQN: A deep reinforcement learning algorithm based on a single agent, in which a single agent governs the strategies of each intersection within a region.

[0163] The algorithm's effectiveness is primarily reflected in alleviating regional traffic congestion and improving traffic efficiency. The evaluation metrics used are average vehicle waiting time and average queue length. The specific meanings and calculation formulas are as follows:

[0164] A. Average waiting time for vehicles in the area at different times

[0165] This indicator evaluates a region as a whole, reflecting the analysis of the temporal characteristics of the entire region; it is a macro-level evaluation metric. Traffic flow varies at different times, and the algorithm implements corresponding timing based on different traffic flow conditions. Regional traffic flow is divided into several time periods, numbered period1, period2, ..., period3. n The average waiting time for each vehicle entity within the entire area is calculated separately for each time period. The specific calculation formula is as follows:

[0166]

[0167] Where i′ represents the time period number, i represents the intersection number, and vehicle_n i′ Represents the total number of all vehicles in the area during time period i, t ik This represents the waiting time of vehicle k at intersection i. Therefore, this indicator yields a total of *period* based on different time periods. n One evaluation indicator.

[0168] B. Average waiting time for vehicles at each phase of the intersection

[0169] This indicator focuses on intersection phases, analyzing the spatial characteristics of phases in different directions at an intersection; it is a micro-level evaluation indicator. Traffic flow varies in the four directions of an intersection. According to simulation data, east-west arterial traffic flows more, while north-south tributary traffic flows less. The average vehicle waiting time for each phase can be used to evaluate the coordination and control of different traffic volumes by the timing algorithm. For the four phases of intersection i, the average vehicle waiting time for each of the four phases is calculated using the following formula:

[0170]

[0171] Where i represents the intersection number and j represents the phase number (j∈{1, 2, 3, 4}), The lane is indicated as lane k The number of all vehicles passing through the intersection in lane (j∈{0,1,2}).

[0172] Therefore, this indicator is based on the four phases of each intersection, resulting in a total of four evaluation indicators.

[0173] C. Average queue length at intersections

[0174] This indicator evaluates a region by calculating the average queue length of all vehicles within that region at different times. Queue length depends on the number of waiting vehicles and the length of each vehicle, making it a direct indicator of regional traffic congestion and crucial for evaluating timing algorithms.

[0175] For intersection i, there are 12 lanes in total, traveling in four directions. The average queue length needs to be calculated lane-by-lane and averaged. The formula for calculating the average queue length at this intersection is as follows:

[0176]

[0177] Where n_lane represents the lane number, num_lane n_lane This represents the number of vehicles queuing in lane n_lane. This represents the length of the k-th car in lane n_lane.

[0178] Calculate the average queue length of vehicles at each intersection, and then solve for the average value to obtain the average queue length of vehicles in the entire area during time period i′. The calculation formula is as follows:

[0179] In the experimental environment simulation, a 3×3 intersection of 9 roads and a 4×4 intersection of 16 roads were selected, simulating both weekday random traffic flow and sudden traffic congestion. This method improves various indicators primarily through a series of technical points in step S3, showing significant optimization compared to other methods.

[0180] In a 3×3 area with 9 intersections, the 9 intersections are located on three main east-west roads running from north to south. The north-to-south intersections are connected by side roads, forming a 3×3 area shape and creating an intersection matrix. A schematic diagram of the 3×3 intersection area is shown below. Figure 4 As shown.

[0181] Within the simulation area, detectors are installed at intersections to collect relevant traffic flow information. The detectors are positioned within a 100-meter radius, and the average distance between adjacent intersections is set to 280 meters. Each road is a two-way three-lane road, designated as a left-turn lane, a straight-ahead lane, and a right-turn lane. The speed limit for vehicles within the area is 50 km / h. Vehicles enter the area from the entrance lane of any intersection except B1 and exit from the exit lane of any intersection except B1. The vehicle trajectories simulate random traffic flow during real-world holidays. During vehicle trajectory data generation, the probability of a vehicle passing through an intersection is controlled to manage traffic flow at different intersections and lanes. Since the east-west direction is the main artery, the simulated vehicle trajectories show more east-west traffic and less north-south traffic. Traffic flow statistics for the simulated area at different time periods are shown in the following figure. Figure 5 As shown, traffic flow peaks and then decreases due to the increasing number of new vehicles in each time period. Ten time periods, each lasting 20 minutes, were simulated for a total of 200 minutes. Traffic flow in each time period was compressed to simulate random traffic conditions at different times during a holiday day.

[0182] Experimental results show that in the simulated traffic scenario of a 3×3 intersection area, from the perspective of the overall area, the method proposed in Example 1 outperforms other comparative methods in terms of overall performance across different time periods. In the initial stage of each time period, the number of vehicles passing through the area is relatively small. The two metrics of other algorithms are close to those of this algorithm, but the method proposed in Example 1 achieves the lower limit of average waiting time and average queue length for vehicles in the area in the first two time periods. From the third time period onwards, traditional fixed timing and lane-sensing timing algorithms show a rapid increase in their metrics, indicating that they cannot effectively cope with increased traffic flow. In contrast, the reinforcement learning-based algorithm and the DQN algorithm can more accurately and reasonably control the signal timing scheme when traffic flow increases. However, when traffic flow further increases, the DQN algorithm, based on single-agent control, is slightly insufficient in terms of average waiting time, and even performs worse than the lane-sensing algorithm in the fifth and eighth time periods. This indicates that determining the agent's policy function only from the global environment, without setting up agent-based collaborative optimization for each intersection, leads to insufficient understanding of the traffic environment. Regarding the average queue length metric, it can be seen that the differences between the other three algorithms (excluding fixed timing) are small, but the method proposed in Example 1 is still optimal. The lane sensing algorithm performs well under this metric because it is based on the longest lane queue and prioritizes processing lanes that sense traffic flow; it can be considered a simplified version of the method proposed in Example 1. The combined evaluation of the two algorithm metrics from a regional perspective demonstrates the effectiveness of this algorithm. On the other hand, the average vehicle waiting time for the four phases at each intersection within the region was statistically analyzed. This metric shifts the focus of the algorithm's performance from a macro-regional perspective to a relatively micro-phase perspective, reflecting the characteristic of higher traffic flow on east-west main roads and lower traffic flow on north-south branch roads. However, the algorithm optimizes the average waiting time of phases with higher traffic flow as much as possible, narrowing the gap in average waiting time between east-west main roads and north-south branch roads. This reflects the design idea of ​​optimizing the agent's strategy function at each intersection and coordinating phase flow.

[0183] In a 4x4 area with 16 intersections, the 16 intersections are distributed along four main east-west roads running from north to south. The north-to-south intersections are connected by side roads, forming a larger 4x4 area and creating an intersection matrix. 4x4 intersection area diagram Figure 6 As shown.

[0184] This area expands the average distance between adjacent intersections to 300 meters, significantly improving the area and lane selection compared to a 3x3 intersection area. A 4x4 area simulates a real-world emergency scenario: a large influx of traffic during peak hours. This experiment simulates 1500 vehicles at once to verify the regional traffic signal timing algorithm's ability to handle and recover from such situations. The simulation consists of 36 10-minute time slots.

[0185] Experimental results show that in the initial stage of peak inflow traffic, the average waiting time in the region for all algorithms is relatively long, reflecting a sudden traffic congestion situation. After four time periods, the coordination effect of the signal timing provided in Example 1 is most evident, causing the average waiting time in the region to begin to decrease first. At this time, the average waiting time in the region for the other three algorithms remains high. The comparison reveals that traditional fixed timing and lane-sensing timing algorithms cannot effectively handle sudden congestion, easily leading to excessively long initial congestion times, requiring extended signal phase releases to alleviate the congestion. In this experiment, the indicators of both algorithms only significantly decreased to normal values ​​after the 20th time period. The average queue length of vehicles has a certain lag compared to the waiting time, exhibiting a peak when there are more vehicles in the queue later in the process. Both reinforcement learning algorithms effectively address sudden congestion, resulting in better recovery capabilities for the regional traffic system. The method provided in Example 1 demonstrates the best algorithmic performance, responding quickly, coordinating signal timing schemes, and effectively alleviating traffic congestion. Compared to the other three algorithms, it improves the average vehicle waiting time index by 49.6%, 33.4%, and 5.9%, respectively, and the average queuing time index at intersections by 51.6%, 29.2%, and 19.5%, respectively. Furthermore, analyzing the average waiting time of intersection phases reveals that as the number of intersections in a 4×4 intersection area increases, the algorithm achieves relatively uniform phase coordination at each intersection, without any significant abnormal phase timing waiting times. This verifies that even with larger regional scales and a greater number of agents, the algorithm can still coordinate communication between agents, select the optimal strategy, and coordinate and control the phase sequence and duration at different intersections.

[0186] Finally, the RNN recurrent neural network used in step S4 of Example 1 addresses the curse of dimensionality caused by the increase in region size. When the region size increases from 3×3 to 4×4, the number of traffic lights at intersections within the region increases, the action space of the coordination strategy increases significantly, and the spatiotemporal characteristics of signal timing contained in the vehicle trajectory data increase exponentially, which may lead to non-convergence and decreased predictive ability. Therefore, the algorithm's effectiveness in small regions may not be applicable in large regions. The method proposed in Example 1 avoids the possibility of blind exploration and adverse actions in the early stages of reinforcement learning. The introduced RNN recurrent neural network can solve the curse of dimensionality to a certain extent. According to the algorithm's convergence, the algorithm underwent 100 iterations in the 4×4 region. The reward increased with the number of iterations, and finally stabilized after 80 iterations, forming the maximum reward, indicating that the reinforcement learning algorithm converged. Regarding the waiting time in the evaluation metrics, the iterative process of the total waiting time is presented here. As the vehicle data explodes after the peak influx, the total waiting time converges to a small value after algorithm training, without the curse of dimensionality, thus verifying the effectiveness of the algorithm.

[0187] Example 3

[0188] This invention also provides a regional traffic signal timing system based on reinforcement learning, such as... Figure 7 As shown, the system includes:

[0189] The data collector 700 is used to collect regional traffic environment data and vehicle trajectory data.

[0190] The memory 701, connected to the data acquisition unit 700, is used to store regional traffic environment data, vehicle trajectory data, and software control instructions. The software control instructions are used to implement the reinforcement learning-based regional traffic signal timing method described above. The memory can be a computer-readable storage medium.

[0191] The processor 702, connected to the memory 701, is used to retrieve and execute software control instructions to obtain the regional traffic signal timing scheme.

[0192] The processor 702 used in the present invention includes:

[0193] The element extraction module is used to extract intersection and traffic light components from regional traffic environment data and vehicle trajectory data.

[0194] The timing task determination module is used to determine the regional traffic signal timing task based on the intersection and traffic light components.

[0195] The waiting model construction module is used to build regional traffic waiting models based on regional traffic signal timing tasks.

[0196] The timing scheme generation module is used to learn and train the regional traffic waiting model using a multi-agent reinforcement learning algorithm, and to complete the coordination and optimization process of the policies of each agent in the region by combining the optimized policy-based NAC algorithm. An RNN recurrent neural network is introduced to process the optimization and coordination results generated in the coordination and optimization process to obtain the regional traffic signal timing scheme.

[0197] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0198] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for regional traffic signal timing based on reinforcement learning, characterized in that, include: S1: Extract the elements of intersections and traffic lights from regional traffic environment data and vehicle trajectory data; S2: Determine the regional traffic signal timing task based on the intersection and traffic light components; S3: Construct a regional traffic waiting model based on the aforementioned regional traffic signal timing task; S4: A multi-agent reinforcement learning algorithm is used to train the regional traffic waiting model. Combined with an optimized policy-based NAC algorithm, the coordinated optimization process of the policies of each agent in the region is completed. An RNN (Recurrent Neural Network) is introduced to process the optimization results generated during the coordination process, resulting in a regional traffic signal timing scheme, including: S41: The basic elements for designing multi-agent systems; the basic elements include: state, action, and reward; S42: Based on the basic elements of the multi-agent system, a multi-agent cooperative optimization model is set up; based on the states, actions, and rewards of the agents designed in step S41, the algorithm completes the cooperative optimization process of the multi-agent system within the region: there are N agents in the region, and each agent has a corresponding mapping from state to action space, i.e., N policy functions; the agent receives state information from other agents, selects action output according to the policy function, transmits its own state to the environment, and obtains rewards from the environment; each agent only obtains local environmental state information, i.e., the state information of the intersection adjacent to the agent's location, and does not obtain the global state; therefore, the multi-agent reinforcement learning algorithm will enable multiple agents to obtain each other's spatiotemporal state characteristics according to the global state of the region through communication, interaction, and updates between agents, adjust their own policy functions, realize the cooperative optimization of the regional intersections, and maximize the global reward of the region; S43: The multi-agent collaborative optimization model is used to update the state information of lanes at each intersection in the region; S44: Aggregate the updated lane status information of each intersection to obtain aggregated status information; S45: The aggregated state information is filtered using an attention mechanism to obtain filtered state information; wherein, an attention mechanism is introduced to integrate important state information within the region space, and the state information of the agent that has an important interactive influence on the current agent's policy selection is filtered out through the attention mechanism; the process of calculating the attention coefficient between any two nodes is as follows: calculate the product of the two as the similarity between the nodes, and normalize the result using the softmax function to obtain the weight coefficient; the weight coefficient is weighted and summed with the values ​​of their respective nodes to obtain the attention coefficient; S46: An optimized policy-based NAC algorithm is used to generate a policy function based on the filtered state information; the policy function is a mapping from the state set to the action set; the NAC algorithm includes: an Actor network and a Critic network; S47: Input the strategy function into an RNN recurrent neural network to obtain the regional traffic signal timing scheme; The regional traffic waiting model established in step S3 is loaded into the urban traffic simulation software. Based on the design and implementation process of steps S3 and S4, agents are set up at each intersection in the environment. The Actor network, Critic network, target network and memory storage area of ​​each agent are randomly initialized. After initialization, a small loop is entered. The initial state of the agent is used to select an action according to the current policy. Noise G is introduced into the policy function to balance the exploration and utilization behavior of the agent in reinforcement learning.

2. The regional traffic signal timing method based on reinforcement learning according to claim 1, characterized in that, The construction of the regional traffic waiting model based on the regional traffic signal timing task specifically includes: Construct a regional traffic environment simulation model; Extract vehicle trajectory data features; the vehicle trajectory data features include the traffic light cycle, the green light ratio, and the traffic light phase difference; The vehicle trajectory data features are incorporated into the regional traffic environment simulation model using a Python programming algorithm to generate the regional traffic waiting model.

3. The regional traffic signal timing method based on reinforcement learning according to claim 2, characterized in that, The regional traffic waiting model includes: The road network environment configuration module is used to build and generate a basic environment model of various traffic elements within the area; The data acquisition module is used to collect vehicle driving data within the area and to extract the spatiotemporal features of vehicle trajectories. The traffic light configuration module is used to perform basic configuration of the traffic lights at each intersection in the area; the basic configuration includes: the color of the traffic lights, the cycle duration, and the phase sequence. The evaluation index output module is used to determine the evaluation index data of the regional traffic environment.

4. The regional traffic signal timing method based on reinforcement learning according to claim 1, characterized in that, The optimized policy-based NAC algorithm generates a policy function based on the filtering state information, specifically including: Each agent updates the parameters of the Actor network according to the corresponding objective function by maximizing the accumulated rewards in the future; After the Actor network updates its parameters, it outputs the next signal cycle data after selecting an action of a certain strategy. The Critic network evaluates the next signal cycle data, so that the strategy function output by the Actor network is continuously coordinated and optimized. The next signal cycle data includes: period, phase order and phase duration factor.

5. The regional traffic signal timing method based on reinforcement learning according to claim 4, characterized in that, The parameters of the Critic network are updated by minimizing the loss function.

6. The regional traffic signal timing method based on reinforcement learning according to claim 1, characterized in that, In the RNN recurrent neural network, the features output at the current time step are used as the input features at the next time step.

7. A regional traffic signal timing system based on reinforcement learning, characterized in that, include: Data collectors are used to collect regional traffic environment data and vehicle trajectory data; A memory, connected to the data acquisition unit, is used to store the regional traffic environment data, the vehicle trajectory data, and software control instructions; the software control instructions are used to implement the reinforcement learning-based regional traffic signal timing method as described in any one of claims 1-6. The processor, connected to the memory, is used to retrieve and execute the software control instructions to obtain the regional traffic signal timing scheme.

8. The regional traffic signal timing system based on reinforcement learning according to claim 7, characterized in that, The processor includes: The element extraction module is used to extract intersection and traffic light components from regional traffic environment data and vehicle trajectory data. The timing task determination module is used to determine the regional traffic signal timing task based on the intersection and traffic light components. A waiting model construction module is used to construct a regional traffic waiting model based on the regional traffic signal timing task. The timing scheme generation module is used to learn and train the regional traffic waiting model using a multi-agent reinforcement learning algorithm, and to complete the coordination and optimization process of the strategies of each agent in the region by combining the optimized policy-based NAC algorithm. An RNN recurrent neural network is introduced to process the optimization and coordination results generated in the coordination and optimization process to obtain the regional traffic signal timing scheme.

9. The regional traffic signal timing system based on reinforcement learning according to claim 7, characterized in that, The memory is a computer-readable storage medium.

Images