A distributed control method for urban multi-intersection traffic lights based on evolutionary Q-network

Abstract

A distributed control method for urban multi-intersection traffic lights based on an evolutionary Q-network includes the following steps: constructing a Markov model for coordinated signal control, defining the agent, environment, state, behavior, and reward to provide a basic model for traffic light control; using a state encoding method to convert the actual traffic state into a binary feature vector, providing standardized input for the Q-network; based on the feature extraction results, using a Q-network structure encoding method to convert the neural network structure parameters into binary codes, preparing for the automatic search for the optimal network structure; using a Q-network structure evolution algorithm to automatically adjust the Q-network structure based on the state codes to find the optimal solution, automatically matching the Q-network structure with the complexity of the coordinated signal light control task; and constructing a distributed decision-making and computational method based on the road network subunits controlled by the Q-network to achieve coordinated signal light control on a large-scale road network. This method can improve road network load balancing and reduce transportation delays.

Description

Technical Field

[0001] The present invention relates to a road signal light control technology, and in particular to a distributed control method for urban multi-intersection signal lights based on an evolutionary Q network. Background Art

[0002] Traffic light control is a universal technology for resolving conflicts between vehicles traveling in different directions at intersections in urban road networks. During operation, fixed display times are set for different signal light phases to determine the right of way on multiple sides of an intersection. However, fixed-time control lacks real-time adjustment capabilities and often fails to fully utilize the intersection's temporal and spatial capacity. For example, when a phase's corresponding road is clear of vehicles, the road should switch to the next phase. However, fixed-time control will hold that phase until the set end time, resulting in unnecessary wasted time for vehicles in the remaining phases.

[0003] To avoid the increased waiting times for vehicles caused by fixed-time traffic light control, sensors are installed at urban intersections to monitor the waiting times and queue lengths of vehicles on multiple sides of the road. This information is then used to adjust the traffic light phases in real time. This method, called adaptive traffic light control, can effectively maintain a balanced degree of road load across multiple phases at an intersection.

[0004] Furthermore, early traffic lights used independent control, meaning that the signal phase selection at each intersection was unaffected by the status of other signals. However, this approach has its limitations. Centralized coordination of all traffic lights in a road network would significantly improve indicators such as network throughput and load balancing. However, centralized coordination methods cause the decision space for traffic light control to grow exponentially with the number of traffic lights. Because reinforcement learning is data-driven and, combined with function estimation methods, can generalize knowledge learned from a small number of samples to a larger decision space, it has been widely used in coordinated traffic light control on road networks.

[0005] Neural networks are the most common form of function estimator in reinforcement learning. For areas with a small number of intersections, reinforcement learning algorithms based on small neural networks can be used to centrally control traffic light phases. For large urban traffic networks, to address the dimensionality explosion of the decision space, distributed signal coordination control methods are often used. These methods transform the large-scale decision space into smaller subproblems, solve them using function estimation methods, and then design coordination algorithms to achieve global consistency.

[0006] A distributed signal light control method based on deep reinforcement learning. This method decomposes the coordinated control problem of urban traffic network signals into several small-scale road network signal light coordinated control problems. It uses image-like state descriptions and a convolutional neural network to extract intersection state features. Based on a deep Q-network, it builds a small-scale road network signal light control sub-model. Finally, it uses a max-plus algorithm to coordinate all sub-models to determine the signal light phase for the entire road network.

[0007] Existing techniques use deep Q-networks to solve the problem of coordinated traffic light control. However, the structure of the deep Q-network must be well-matched to the complexity of the task to avoid overfitting and underfitting. In existing methods, the deep Q-network structure is typically manually configured by the designer, who then trains the network using data to obtain appropriate network weights. Compared to automated computer search, manual experimentation is less efficient and often leads to local optima.

[0008] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0009] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a distributed control method for urban multi-intersection traffic lights based on an evolved Q network.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect of the present invention, a method for distributed control of urban multi-intersection traffic lights based on an evolved Q network comprises the following steps:

[0012] Build a Markov model for signal collaborative control, define the agent, environment, state, behavior, and reward, and provide a basic model for traffic light control;

[0013] Through the state encoding method, the actual traffic state is converted into a binary feature vector to provide standardized input for the Q network;

[0014] Based on the feature extraction results, the neural network structure parameters are converted into binary codes through the Q network structure encoding method to prepare for the automatic search for the optimal network structure;

[0015] Through the Q network structure evolution algorithm, the Q network structure is automatically adjusted according to the state code to find the optimal solution, and the Q network structure is automatically matched with the complexity of the traffic light collaborative control task;

[0016] Build and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, realize distributed processing of the problem, and improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

[0017] In a second aspect of the present invention, a distributed control method for urban multi-intersection traffic lights based on an evolved Q network includes the following implementation steps:

[0018] Real-time monitoring of traffic conditions at multiple intersections in the city. The data is fed into a Markov model to comprehensively evaluate the agent, environment, state, behavior, and rewards, providing decision support for real-time control of traffic lights.

[0019] Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network so that the algorithm can process real-time traffic data;

[0020] Using the converted state vector, a distributed control coordination algorithm is executed to decompose the large-scale traffic light control problem into multiple small-scale sub-problems. Efficient traffic light control is achieved through distributed processing. Each sub-problem is solved by an independent Q network, and a coordination mechanism is used to achieve collaborative optimization of the entire road network and realize traffic light control.

[0021] Among them, the Q network structure encoding method automatically adjusts the Q network structure parameters to adapt to the complexity of different traffic scenarios; the Q network structure evolution algorithm automatically adjusts the Q network structure according to the state encoding to find the optimal solution, realizing the automatic matching of the Q network structure with the complexity of the signal light collaborative control task;

[0022] Among them, the traffic light control effect is periodically evaluated, and the Q network parameters and control strategy are adjusted according to the evaluation results; through continuous performance evaluation and adjustment, the traffic efficiency and load balance of the road network are optimized.

[0023] In a third aspect of the present invention, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the distributed control method for urban multi-intersection traffic lights based on an evolved Q network.

[0024] In a fourth aspect of the present invention, a computer program product includes a computer program, which, when executed by a processor, implements the distributed control method for urban multi-intersection traffic lights based on the evolved Q network.

[0025] The present invention has the following beneficial effects:

[0026] This paper proposes a method for coordinated control of urban multi-intersection traffic lights based on distributed reinforcement learning. This method decomposes the large-scale coordinated control problem of traffic lights into smaller, identically structured sub-problems, and uses Q networks to establish action-value functions for the traffic light control sub-problems. Ultimately, multiple Q networks are coordinated within a distributed control framework to achieve large-scale road network traffic light control. In particular, the present invention uses a Q network structure evolution mechanism (which can be designed based on a particle swarm algorithm) to automatically match the complexity of the Q network with the task of coordinated traffic light control. Application of the method of the present invention can significantly improve road network load balancing and reduce transportation delays.

[0027] The present invention's method for distributed control of urban multi-intersection traffic lights, based on an evolutionary Q-network, first devises a method for converting a feedforward neural network structure into binary code. This method then uses a Q-network structure evolution method to screen and evolve the Q-network structure, forming an automated matching mechanism for collaborative control of the Q-network structure and traffic lights. This automatically matches the neural network structure to the complexity of the collaborative control task, improving the efficiency of structural screening while avoiding local optimality compared to manual screening. Furthermore, when solving large-scale traffic light control problems, the distributed control framework decomposes the problem into multiple small-scale subproblems, reducing the time complexity of calculating the optimal traffic light phase and improving solution efficiency.

[0028] The distributed reinforcement learning-based coordinated control method for urban multi-intersection signal lights proposed in this invention can coordinate the phases of all signal lights in the road network according to the sensor data at the road intersections, achieving the significant effect of balancing the road network load and improving traffic efficiency.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a method for collaborative control of urban multi-intersection traffic lights based on distributed reinforcement learning according to an embodiment of the present invention.

[0031] Figure 2 Schematic diagram of traffic at a double cross intersection in an embodiment of the present invention.

[0032] Figure 3 Schematic diagram of a typical eight-phase signal light in an embodiment of the present invention.

[0033] Figure 4 Schematic diagram of two-dimensional grid coding according to an embodiment of the present invention.

[0034] Figure 5 Schematic diagram of a Q network model for coordinated control of traffic lights according to an embodiment of the present invention.

[0035] Figure 6Schematic diagram of a distributed control model for traffic lights in a six-intersection road network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0037] This paper proposes a method for coordinated control of traffic lights at multiple urban intersections based on distributed reinforcement learning. Specifically, in a vehicle-road collaborative environment, intersection sensors can monitor the length of waiting convoys and vehicle waiting times in real time. A central controller at multiple urban intersections can coordinate the phases of all traffic lights in the network based on this sensor data, effectively balancing the network load and significantly improving traffic efficiency.

[0038] See Figure 1 The embodiment of the present invention provides a method for distributed control of urban multi-intersection traffic lights based on an evolved Q network, comprising the following steps:

[0039] Step 1. Establish a Markov model: Construct a Markov model for signal collaborative control, define the agent, environment, state, behavior, and reward, and provide a basic model for traffic light control.

[0040] In a preferred embodiment, step one specifically includes the following steps:

[0041] Step 1.1 (Lane Load Evaluation): Take the weighted sum of the waiting queue length and stop time of each lane to obtain the evaluation index of the lane's current load;

[0042] Step 1.2 (State Definition): Define state variables based on the relative positions of lanes and traffic flow, including the load conditions of the uplink and downlink lanes.

[0043] Step 1.3 (Signal Light Phase Setting): Design the signal light phase diagram, distinguish between the main phase and transition phase, and assign corresponding access rights to each lane;

[0044] Step 1.4 (Definition of cooperative control action space): Define the action space of cooperative control at the dual intersection and clarify the signal light phase implementation strategy corresponding to each action;

[0045] Step 1.5 (Reward function construction): Construct a reward function to evaluate the effectiveness of the traffic light control strategy with the goal of reducing lane load.

[0046] Step 2: Feature extraction: Design a state encoding method to convert the actual traffic state into a binary feature vector to provide standardized input for the Q network.

[0047] See Figure 4 In a preferred embodiment, step 2 specifically includes the following steps:

[0048] Step 2.1 (State space segmentation): Define the state variables as two-dimensional vectors and segment them using arithmetic progression in two dimensions to form a grid-like structure.

[0049] Step 2.2 (Random bias setting): Assign a random bias to the starting point of the arithmetic progression to ensure that the encoding of the state space is diverse and adaptable;

[0050] Step 2.3 (2D grid construction): Use an arithmetic progression with random biases to construct a 2D grid that can discretize the continuous state space.

[0051] Step 2.4 (state vector mapping): Map the state vector to a specific area in the two-dimensional grid. Determine the position of the state vector in the grid by determining the interval in which the state vector is located.

[0052] Step 2.5 (Binary feature vector generation): Generate a corresponding binary feature vector based on the position of the state vector in the two-dimensional grid, where only the elements at the corresponding grid position are 1 and the rest are 0;

[0053] Step 2.6 (Multi-grid code integration): If there are multiple grids with different random biases, the binary feature vectors generated by each grid are integrated to form the final state code.

[0054] Step 3: Q network structure encoding: Based on the feature extraction results, a Q network structure encoding method is designed to convert the neural network structure parameters into binary codes to prepare for the automatic search for the optimal network structure.

[0055] See Figure 5 In a preferred embodiment, step three specifically includes the following steps:

[0056] Step 3.1 (input coding definition): The binary coding of the state variables of two adjacent intersections obtained in step 2 is used as the input of the Q network;

[0057] Step 3.2 (Action-value function output): The Q network outputs the action-value functions corresponding to the four feasible behaviors, which are used to evaluate the potential value of each behavior.

[0058] Step 3.3 (Objective function construction): Construct the objective function for Q network training, which is based on the state transition and reward function of the time series and uses the minimum mean square error as the optimization criterion;

[0059] Step 3.4 (Binary encoding of network structure): Define a vector of the number of nodes for the R-layer neural network and convert each element into a binary number to form a binary encoding of the network structure.

[0060] Step 3.5 (Parameter update rule formulation): Based on the objective function, use stochastic gradient descent to update the Q network parameters to ensure that the parameter update can reduce the prediction error and improve the network performance;

[0061] Step 3.6 (network structure optimization): Ensure that the binary encoding of the network structure meets certain range conditions to ensure the diversity and effectiveness of the network structure.

[0062] Step 4: Q network structure evolution: Initialize and execute the Q network structure evolution algorithm. Use optimization algorithms such as the particle swarm optimization algorithm to automatically adjust the Q network structure according to the state code, find the optimal solution, and achieve automatic matching between the Q network structure and the complexity of the traffic light collaborative control task.

[0063] In a preferred embodiment, step 4 designs a particle swarm algorithm, which specifically includes the following steps:

[0064] Step 4.1 (population initialization): Randomly generate a population representing the network structure, where each individual represents a potential Q network structure;

[0065] Step 4.2 (velocity vector initialization): Initialize the velocity vector for each individual, which will guide the search process of the network structure;

[0066] Step 4.3 (recording the best position of each individual): record the best position achieved by each individual during the search process;

[0067] Step 4.4 (Definition of field size and maximum speed): Set parameters such as field size and maximum speed to control the intensity and scope of the search process;

[0068] Step 4.5 (Neighbor search and update rule application): For each network structure code, find its neighbors and apply the update rule to adjust the speed and position by combining the individual's historical best position and the neighbor's best position;

[0069] Step 4.6 (Speed ​​Adjustment and Normalization): Normalize the speed vector according to the speed adjustment rules and the maximum speed limit to ensure the stability of the search process;

[0070] Step 4.7 (network structure update): According to the updated velocity vector, adjust the network structure encoding to form a new Q network structure;

[0071] Step 4.8 (Q network parameter update): Decode the updated network structure into Q network parameters and update these parameters through the training process until convergence;

[0072] Step 4.9 (Performance evaluation and best position update): Evaluate the performance of the updated Q network and compare it with the individual historical best position, and update the individual best position;

[0073] Step 4.10 (iterative evolution): Repeat the above process until the stopping condition is met, thereby achieving iterative evolution and optimization of the Q network structure.

[0074] Step 5. Distributed Control Coordination Algorithm: Build and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, implement distributed processing of the problem, and continuously improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

[0075] See Figure 6 In a preferred embodiment, step five specifically includes the following steps:

[0076] Step 5.1 (Establishment of multi-Q network control structure): Establish an independent Q network for each pair of adjacent intersections in the road network, and conduct targeted training for each pair of intersections;

[0077] Step 5.2 (Network structure and parameter convergence): Ensure that all Q networks reach a stable state in terms of structure and parameters, providing a reliable decision-making basis for signal light phase selection;

[0078] Step 5.3 (Periodic signal light phase optimization): Set a periodic time interval, after which the optimal signal light phase action is re-evaluated and selected;

[0079] Step 5.4 (Weight definition and action selection): Define weight coefficients, combine minimization and averaging operations, and select the signal light phase action with the best overall efficiency;

[0080] Step 5.5 (Scale-adaptive solution strategy): Select an appropriate solution strategy based on the scale of the network. For smaller networks, use the exhaustive method, and for larger networks, use the worst-case-first algorithm.

[0081] Step 5.6 (Available action value calculation and optimization): Calculate the available action values ​​of all Q networks at each time point and select the optimal action based on them;

[0082] Step 5.7 (selection of two-intersection pairs and determination of actions): select the two-intersection pair with the minimum action value from the road network system and determine the optimal action for the pair;

[0083] Step 5.8 (Optimization Problem Scale Assessment): Assess the scale of the current optimization problem and determine whether it is suitable for an exhaustive approach.

[0084] Step 5.9 (iterative coordination and solution): If the current problem size is suitable for the exhaustive method, then solve it; otherwise, iteratively execute steps 5.6 and 5.7 until the optimal solution is found.

[0085] The embodiment of the present invention further provides a method for distributed control of urban multi-intersection traffic lights based on an evolved Q network, comprising the following implementation steps:

[0086] Step A: Real-time state monitoring and model application: Real-time monitoring of traffic conditions at multiple intersections in the city is performed. The monitoring data is input into the established Markov model to achieve a comprehensive evaluation of the agent, environment, state, behavior, and rewards, providing decision support for real-time control of traffic lights.

[0087] Step B, state vector conversion and input: Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network to ensure that the algorithm can process real-time traffic data;

[0088] Step C, Distributed Control Execution: Utilizing the converted state vector, a distributed control coordination algorithm is executed to decompose the large-scale traffic light control problem into multiple small-scale sub-problems. Efficient traffic light control is achieved through distributed processing. Each sub-problem is solved by an independent Q network, and a coordination mechanism is used to ensure coordinated optimization of the overall road network, achieving efficient traffic light control.

[0089] Step D, Adaptive Network Structure Optimization: Using the Q network structure encoding method, the Q network structure parameters are automatically adjusted to adapt to the complexity of different traffic scenarios. Using the Q network structure evolution algorithm and optimization techniques such as the particle swarm optimization algorithm, the Q network structure is automatically adjusted based on the state encoding to find the optimal solution, achieving adaptive evolution of the Q network structure, optimizing the traffic light control strategy, and achieving automatic matching of the Q network structure with the complexity of the traffic light collaborative control task.

[0090] Step E: Continuous performance evaluation and adjustment: Periodically evaluate the effectiveness of traffic light control and adjust Q network parameters and control strategies based on the evaluation results. Through continuous performance evaluation and adjustment, optimize the traffic efficiency and load balance of the road network.

[0091] This paper proposes an innovative distributed control method for urban multi-intersection traffic lights based on an evolutionary Q-network. Based on reinforcement learning, this paper proposes a distributed traffic light control framework for urban road networks, achieving significant traffic management and control effects, improving network load balancing, and reducing transportation delays. First, the present invention constructs a Markov model for signal coordinated control that comprehensively evaluates the agent, environment, state, behavior, and rewards, providing decision support for real-time control. Second, the present invention uses a state encoding method to effectively convert the actual traffic state into a binary feature vector, providing standardized input for the Q-network, which enhances the algorithm's ability to process real-time traffic data. Furthermore, the present invention uses a Q-network structure encoding method to convert the neural network structure parameters into binary codes. Using a Q-network structure evolution algorithm, such as a particle swarm algorithm, the present invention automatically adjusts the Q-network structure to automatically match the complexity of the traffic light coordinated control task, avoiding the inefficiency of manually setting the network structure and the risk of falling into a local optimum. A key advantage of the present invention is that it uses a distributed control coordination algorithm to decompose the large-scale traffic light control problem into multiple small-scale subproblems, which not only reduces the time complexity of calculating the optimal traffic light phase but also improves the solution efficiency. This approach is particularly well-suited for large urban traffic networks, effectively addressing the exponential growth of the decision space caused by the increasing number of traffic lights. Furthermore, the distributed control framework of this invention, coordinated with multiple Q networks, enables efficient control of traffic lights in large-scale networks. Through periodic evaluation and optimization, it continuously improves traffic efficiency and load balance in the network.

[0092] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0093] The method of the present invention mainly includes the following parts: 1. Constructing a Markov model for the coordinated control of urban road network traffic lights, defining elements such as intelligent agents, environments, states, behaviors, and rewards; 2. Designing a sparse grid coding method to convert road loads into binary feature vectors; 3. Designing a Q network structure coding method; 4. Designing a Q network structure evolution algorithm; 5. Constructing a distributed computing method to achieve large-scale coordinated control of road network traffic lights.

[0094] like Figure 1 As shown, the implementation process of the present invention mainly includes the following steps:

[0095] (1) Establishment of the Markov model of signal cooperative control

[0096] Figure 2 It is a typical adjacent intersection, each lane has its own unique number. Indicates the length of the waiting queue in lane i and queue stop time The weighted sum of is the evaluation index of the existing load of the lane, which is

[0097]

[0098] Among them, ω∈(0,1) is the weight.

[0099] The light blue track represents the track where the vehicle is allowed to pass. i and j represent the preceding lane and following lane of the connected trajectory, respectively. represents the relative load between lane i and lane j.

[0100] The traffic lights at the two intersections are coordinated. The middle lane is most significantly affected. Since the right-turn lane is less affected by the traffic lights, the left-turn lane and the through lane are considered. Figure 2 For example, define lanes 8 and 9 as up lanes and lanes 11 and 12 as down lanes. Therefore, the state is defined as

[0101]

[0102] Among them, for the up lane, Where i represents the number of all upstream lanes, max represents the maximum value, and mean represents the average value. Indicates the down lane.

[0103] Figure 3 This is a typical eight-phase signal light phase diagram, where G indicates priority, g indicates passage unless there is a conflict with vehicles with priority, and r indicates no passage. The phase combinations GGgGrrGGgGrr represent the phases of lanes 1, 2, 3, 7, 8, 9, 14, 15, 16, 20, 21, and 22, respectively. Phases 1 and 5 are primary phases with longer durations, while phases 2-4 and 6-8 are transitional phases.

[0104] Define the dual intersection cooperative control action space a t ∈{[0,0],[1,0],[0,1],[1,1]}, where 0 indicates implementation starting from phase 1 and ending at phase 4, and 1 indicates implementation starting from phase 5 and ending at phase 8. In this example, the time to implement phases 1-4 and 5-8 is equal, which is ΔT.

[0105] The reward is defined as

[0106]

[0107] in,

[0108] (2) Design of signal cooperative control state coding method

[0109] Dependent state variable is a two-dimensional vector, and grid coding is used to convert s t Converted into a binary vector,

[0110] like Figure 4 As shown in the two-dimensional grid encoding, the state space is and The two dimensions are divided by the following arithmetic progression:

[0111]

[0112] Among them, the common difference of the series X0 and X1 is the same, both are ΔX, X 0,0 、X 1,0 Randomly select a value between the interval [-ΔX, 0].

[0113] X0 and X1 form a two-dimensional grid with random bias, which corresponds to a one-dimensional feature vector containing N0*N1 elements. for like

[0114]

[0115] It is believed that s t Appears on the grid {X 0,i ,X 0,i+1 ,X 0,j ,X 0,j+1}middle, The iN2+j+1th element is 1, and the rest are 0.

[0116] If there are L grids with different random biases, the state vector can be encoded as

[0117]

[0118] (3) Q network structure coding method design

[0119] This embodiment designs a traffic light cooperative control based on the Q network model. Figure 5 As shown, the input of the Q network is the state variables between two adjacent intersections The binary code of Φ s , the output is the action value function Q(s,a1),Q(s,a2),Q(s,a3),Q(s,a4) corresponding to the four feasible behaviors a1=[1,1],a2=[1,0],a3=[0,1],a4=[0,0], and the objective function of Q network training is

[0120]

[0121] Among them, Φ c represents the binary encoding of the network structure. For the R-layer neural network, let the vector P = [p1 p2... p R , which are the number of nodes in each layer of the neural network respectively, and satisfy

[0122]

[0123] Convert p k directly into a binary number to obtain Φ c , Φ c is a binary vector with a length of R * P k .

[0124] θ t represents the Q-network parameters corresponding to the structure Φ<00​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​2,k ~U(0,μ 2,max ),k∈[1,R*P k ],implement

[0137]

[0138] If|v i |>v max ,then

[0139] v i ←v i v max / |v i |

[0140] End if

[0141]

[0142] Will Decoded into the corresponding Q network, set its parameters to use

[0143]

[0144] in,

[0145] The network is trained until convergence, and we get Value

[0146]

[0147] Next Grid Structure Encoding

[0148] Next-generation grid structure coding

[0149] (5) Design of Distributed Control Coordination Algorithm for Road Network Signal Lights

[0150] For Figure 6 The six-intersection road network shown in the figure is controlled by seven Q networks, each of which is trained for two adjacent intersections. (i,j)=(1,2),(2,3),(4,5),(5,6),(1,4),(2,5),(3,6). When these Q network structures and parameters converge, they are used to select the signal light phase. To optimize the overall efficiency of the road network, it is set to reselect after every ΔT. make have

[0151]

[0152] Among them, max represents the maximum value, mean represents the average value, min represents the minimum value, and β∈(0,1) is the weight.

[0153] For smaller networks, the exhaustive method is used to solve the problem, and for larger networks, the worst case first algorithm is used:

[0154] Step 1: At time t, calculate the action values ​​of all Q networks and select The corresponding optimal action and its corresponding action value

[0155] Step 2: For all double intersections (i, j) in the road network system, select and confirm

[0156] Step 3: In (i * ,j * ) corresponds to the action value When determining, check whether the optimization problem is degenerated to a scale that can be solved by exhaustive method. If so, use exhaustive method to solve it. If not, after determining (i * ,j * ), re-implement steps 1 and 2.

[0157] In summary, the embodiments of the present invention propose a distributed control method for urban multi-intersection traffic lights based on an evolutionary Q network, wherein: (1) the neural network structure is automatically matched with the complexity of the traffic light collaborative control task through a Q network structure evolution mechanism (preferably a particle swarm algorithm is designed). (2) The forward neural network structure is represented as a binary vector through a Q network structure encoding method. (3) A worst-case priority algorithm is proposed to solve the problem of high computational time complexity of large-scale traffic light distributed control. (4) Based on sparse grid coding, a binary encoding method for characterizing urban road network load is proposed.

[0158] The present invention's distributed control method for urban multi-intersection traffic lights, based on an evolutionary Q-network, first devises a method for converting a feedforward neural network structure into binary code. This method then uses a Q-network structure evolution method to screen and evolve the Q-network structure, forming an automated matching mechanism for the Q-network structure and traffic light collaborative control. This automatically matches the neural network structure to the complexity of the traffic light collaborative control task. Compared to manual screening, this method improves the efficiency of structure screening while also avoiding local optimality. Furthermore, when solving large-scale traffic light control problems, a worst-case-first algorithm is preferably employed within the distributed control framework to reduce the time complexity of calculating the optimal traffic light phase and improve solution efficiency.

[0159] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0160] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0161] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0162] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0166] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0167] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0168] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0169] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0170] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0171] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A distributed control method for urban multi-intersection traffic lights based on an evolutionary Q network, characterized in that: The following steps are involved: Step 1: Build a Markov model for signal collaborative control, defining the agent, environment, state, behavior, and reward to provide a basic model for traffic light control. Step 2: Use the state encoding method to convert the actual traffic state into a binary feature vector to provide standardized input for the Q network; Step 3: Based on the feature extraction results, the neural network structure parameters are converted into binary codes through the Q network structure encoding method to prepare for the automatic search for the optimal network structure; Step 4: Using the Q network structure evolution algorithm, the Q network structure is automatically adjusted according to the state code to find the optimal solution, achieving automatic matching between the Q network structure and the complexity of the traffic light collaborative control task; Step 4 specifically includes: Randomly generate a population representing the network structure, each individual represents a potential Q network structure; Initialize the velocity vector for each individual, which will guide the search process of the network structure; Record the best position achieved by each individual during the search process; Setting parameters including field size and maximum speed to control the intensity and scope of the search process; For each network structure code, find its neighbors and apply the update rule to adjust the speed and position by combining the individual's historical best position and the neighbor's best position; According to the speed adjustment rules and maximum speed limit, the speed vector is normalized to ensure the stability of the search process; According to the updated velocity vector, the network structure encoding is adjusted to form a new Q network structure; Decode the updated network structure into Q network parameters and update these parameters through the training process until convergence; Evaluate the performance of the updated Q network and compare it with the individual's historical best position to update the individual's best position; Repeat the above process until the stopping condition is met, thus achieving iterative evolution and optimization of the Q network structure. Step 5: Construct and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, realize distributed processing of the problem, and improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization. Step 5 specifically includes: Establish an independent Q network for each pair of adjacent intersections in the road network, and conduct targeted training for each pair of intersections; Ensure that all Q networks reach a stable state in terms of structure and parameters, providing a reliable decision-making basis for signal light phase selection; Set a periodic time interval, after which the optimal signal light phase action is re-evaluated and selected; Define weight coefficients and combine minimization and averaging operations to select the signal light phase action with the best overall efficiency; Choose an appropriate solution strategy based on the scale of the network. For smaller networks, use an exhaustive approach, while for larger networks, use a worst-case-first approach. Calculate the action values ​​of all Q networks at each time point and select the best action based on them; Selecting a pair of intersections with the minimum action value from the road network system and determining the optimal action of the pair; Assess the scale of the current optimization problem and determine whether it is suitable for an exhaustive solution; If the current problem size is suitable for the exhaustive method, then solve it; otherwise, iterate until the optimal solution is found.

2. The distributed control method for urban multi-intersection signal lights based on the evolved Q network according to claim 1, characterized in that: Step 1 specifically includes: The waiting queue length and stop time of each lane are weighted and summed to obtain the evaluation index of the lane's current load; Based on the relative positions of the lanes and the traffic flow direction, state variables are defined, including the load conditions of the uplink and downlink lanes; Design the phase diagram of the traffic light, distinguish the main phase and transition phase, and assign corresponding access rights to each lane; Define the action space for coordinated control of dual intersections and clarify the signal light phase implementation strategy corresponding to each action; Construct a reward function to evaluate the effectiveness of the traffic light control strategy with the goal of reducing lane load.

3. The distributed control method for urban multi-intersection signal lights based on the evolved Q network according to claim 1, characterized in that: Step 2 specifically includes: Define the state variable as a two-dimensional vector and use arithmetic progression to divide it in two dimensions to form a grid structure; Assigning random bias to the starting point of the arithmetic progression to ensure the diversity and adaptability of the state space encoding; A two-dimensional grid is constructed using an arithmetic progression with random biases, which can discretize the continuous state space. Map the state vector to a specific area in a two-dimensional grid, and determine its position in the grid by judging the interval in which the state vector is located; According to the position of the state vector in the two-dimensional grid, a corresponding binary feature vector is generated, in which only the elements corresponding to the grid position are 1 and the rest are 0; If there are multiple grids with different random biases, the binary feature vectors generated by each grid are integrated to form the final state code.

4. The distributed control method for urban multi-intersection signal lights based on the evolved Q network according to claim 1, characterized in that: Step three specifically includes: The binary codes of the state variables of two adjacent intersections obtained in step 2 are used as the input of the Q network; The Q network outputs action-value functions corresponding to four feasible behaviors, which are used to evaluate the potential value of each behavior; Construct the objective function of Q network training, which is based on the state transition and reward function of the time series and uses the minimum mean square error as the optimization criterion; Define a vector of the number of nodes for the R-layer neural network and convert each element into a binary number to form a binary code of the network structure; According to the objective function, the Q network parameters are updated using stochastic gradient descent to ensure that the parameter update can reduce the prediction error and improve the network performance; The binary encoding of the network structure meets the set range conditions to ensure the diversity and effectiveness of the network structure.

5. A distributed control method for urban multi-intersection traffic lights based on an evolutionary Q network, characterized in that: The following steps are involved: Real-time monitoring of traffic conditions at multiple intersections in the city. The data is fed into a Markov model to comprehensively evaluate the agent, environment, state, behavior, and rewards, providing decision support for real-time control of traffic lights. Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network so that the algorithm can process real-time traffic data; Using the converted state vector, a distributed control coordination algorithm is executed to decompose the large-scale traffic light control problem into multiple small-scale sub-problems. Efficient traffic light control is achieved through distributed processing. Each sub-problem is solved by an independent Q network, and a coordination mechanism is used to achieve collaborative optimization of the entire road network and realize traffic light control. Among them, the Q network structure encoding method is used to automatically adjust the Q network structure parameters to adapt to the complexity of different traffic scenarios. The Q network structure evolution algorithm is used to automatically adjust the Q network structure according to the state encoding to find the optimal solution, thereby achieving automatic matching of the Q network structure with the complexity of the signal light collaborative control task. Specifically, Randomly generate a population representing the network structure, each individual represents a potential Q network structure; Initialize the velocity vector for each individual, which will guide the search process of the network structure; Record the best position achieved by each individual during the search process; Setting parameters including field size and maximum speed to control the intensity and scope of the search process; For each network structure code, find its neighbors and apply the update rule to adjust the speed and position by combining the individual's historical best position and the neighbor's best position; According to the speed adjustment rules and maximum speed limit, the speed vector is normalized to ensure the stability of the search process; According to the updated velocity vector, the network structure encoding is adjusted to form a new Q network structure; Decode the updated network structure into Q network parameters and update these parameters through the training process until convergence; Evaluate the performance of the updated Q network and compare it with the individual's historical best position to update the individual's best position; Repeat the above process until the stopping condition is met, thereby achieving iterative evolution and optimization of the Q network structure; The traffic light control effect is periodically evaluated, and the Q network parameters and control strategy are adjusted based on the evaluation results. Through continuous performance evaluation and adjustment, the traffic efficiency and load balance of the road network are optimized. Specifically, Establish an independent Q network for each pair of adjacent intersections in the road network, and conduct targeted training for each pair of intersections; Ensure that all Q networks reach a stable state in terms of structure and parameters, providing a reliable decision-making basis for signal light phase selection; Set a periodic time interval, after which the optimal signal light phase action is re-evaluated and selected; Define weight coefficients and combine minimization and averaging operations to select the signal light phase action with the best overall efficiency; Choose an appropriate solution strategy based on the scale of the network. For smaller networks, use an exhaustive approach, while for larger networks, use a worst-case-first approach. Calculate the action values ​​of all Q networks at each time point and select the best action based on them; Selecting a pair of intersections with the minimum action value from the road network system and determining the optimal action of the pair; Assess the scale of the current optimization problem and determine whether it is suitable for an exhaustive solution; If the current problem size is suitable for the exhaustive method, then solve it; otherwise, iterate until the optimal solution is found.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for distributed control of urban multi-intersection traffic lights based on an evolved Q network as described in any one of claims 1 to 5 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for distributed control of urban multi-intersection traffic lights based on an evolved Q network as described in any one of claims 1 to 5 is implemented.

Images