Urban multi-intersection signal lamp distributed control method based on evolutionary Q network

Abstract

An urban multi-intersection signal lamp distributed control method based on an evolutionary Q network comprises the following steps: constructing a Markov model for signal cooperative control, defining an intelligent agent, an environment, a state, a behavior and an award, and providing a basic model for signal lamp control; through a state coding method, the actual traffic state is converted into a binary feature vector, and standardized input is provided for a Q network; on the basis of a feature extraction result, neural network structure parameters are converted into binary codes through a Q network structure coding method, and preparation is made for automatic search of an optimal network structure; through a Q network structure evolutionary algorithm, the Q network structure is automatically adjusted according to state codes, an optimal solution is searched, and automatic matching of the Q network structure and signal lamp cooperative control task complexity is achieved; and constructing a distributed decision calculation method based on the road network subunits controlled by the Q network to realize large-scale road network signal lamp cooperative control. The method can improve the road network load balance degree and reduce the transportation delay time.

Description

Technical Field

[0001] The invention relates to a road signal light control technology, and in particular to a distributed control method for urban multi-intersection signal lights based on an evolutionary Q network. Background Art

[0002] Traffic light control is a universal technology for resolving conflicts between vehicles traveling in different directions at intersections in urban road networks. During the operation of urban road networks, the right of way of multiple roads on the intersection is determined by setting fixed display times for different phases of traffic lights. However, fixed time control lacks real-time adjustment capabilities and often cannot fully utilize the time and space capacity of the intersection. For example, when there are no vehicles on the road corresponding to a certain phase, it should switch to the next phase, but fixed time control will keep the phase until the set end time, resulting in unnecessary consumption of vehicle time corresponding to the remaining phases.

[0003] In order to avoid the increase in vehicle waiting time caused by fixed-time control of traffic lights, sensors are installed at urban intersections to monitor the waiting time of vehicles on multiple sides of the road and the length of waiting queues, and adjust the phases of traffic lights in real time accordingly. This is called the adaptive traffic light control method, which can effectively maintain the balance of road loads corresponding to multiple phases at the intersection.

[0004] In addition, early traffic lights adopted an independent control method, that is, when the traffic lights at each intersection choose the phase, they are not affected by the status of other traffic lights. However, this method has its limitations. If all traffic lights in the road network can be coordinated centrally, it will significantly improve indicators such as road network throughput and load balance. However, the centralized coordination method makes the decision space of the traffic light control problem show an exponential growth trend as the number of traffic lights increases. Since reinforcement learning has the characteristics of data-driven, and combined with the function estimation method, it can generalize the knowledge learned based on a small number of samples to a larger decision space, so it is widely used in the coordinated control of road network traffic lights.

[0005] Among the function estimators of reinforcement learning, the most common form is neural network. For areas with a small number of intersections, a reinforcement learning algorithm based on a small neural network can be used to achieve centralized control of the signal light phase. For large urban traffic networks, in order to deal with the problem of explosion of decision space dimensions, a distributed signal light coordination control method is often used to convert problems with large-scale decision spaces into small-scale sub-problems, solve the sub-problems using a method based on function estimation, and then design a coordination algorithm to obtain global consistency.

[0006] A distributed signal light control method based on deep reinforcement learning. This method decomposes the coordinated control problem of urban traffic network signals into several coordinated control problems of small-scale road networks, uses image-like state descriptions, and uses convolutional neural networks to extract intersection state features. Based on the deep Q network, a small-scale road network signal light control sub-model is established, and finally the max-plus algorithm is used to coordinate all sub-models to determine the signal light phase of the entire road network.

[0007] The existing technology uses deep Q networks to solve the problem of coordinated control of traffic lights. The structure of the deep Q network needs to match the complexity of the task of coordinated control of traffic lights to avoid overfitting and underfitting. In the existing methods, the structure of the deep Q network is usually set manually by the designer, and then the network is trained through data to obtain reasonable network weights. Compared with computer automatic search, manual attempts are less efficient and usually fall into local optimality.

[0008] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0009] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and to provide a distributed control method for urban multi-intersection signal lights based on an evolved Q network.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect of the present invention, a distributed control method for urban multi-intersection signal lights based on an evolved Q network comprises the following steps:

[0012] Construct a Markov model for signal collaborative control, define the agent, environment, state, behavior, and reward, and provide a basic model for traffic light control;

[0013] Through the state encoding method, the actual traffic state is converted into a binary feature vector to provide standardized input for the Q network;

[0014] Based on the feature extraction results, the neural network structure parameters are converted into binary codes through the Q network structure coding method to prepare for the automatic search for the optimal network structure;

[0015] Through the Q network structure evolution algorithm, the Q network structure is automatically adjusted according to the state code to find the optimal solution, so as to achieve automatic matching between the Q network structure and the complexity of the traffic light collaborative control task;

[0016] Build and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, realize distributed processing of the problem, and improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

[0017] In a second aspect of the present invention, a distributed control method for urban multi-intersection signal lights based on an evolved Q network comprises the following implementation steps:

[0018] Real-time monitoring of traffic conditions at multiple intersections in the city, and inputting the monitoring data into the established Markov model to achieve a comprehensive evaluation of the agent, environment, state, behavior, and reward, providing decision support for real-time control of traffic lights;

[0019] Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network so that the algorithm can process real-time traffic data;

[0020] The converted state vector is used to execute the distributed control coordination algorithm to decompose the large-scale traffic light control problem into multiple small-scale sub-problems, and efficient traffic light control is achieved through distributed processing. Each sub-problem is solved by an independent Q network, and the coordination mechanism is used to optimize the overall road network and achieve traffic light control.

[0021] Among them, the Q network structure encoding method is used to automatically adjust the Q network structure parameters to adapt to the complexity of different traffic scenarios; the Q network structure is automatically adjusted according to the state encoding through the Q network structure evolution algorithm to find the optimal solution and realize the automatic matching of the Q network structure and the complexity of the signal light collaborative control task;

[0022] Among them, the traffic light control effect is evaluated periodically, and the Q network parameters and control strategies are adjusted according to the evaluation results; through continuous performance evaluation and adjustment, the traffic efficiency and load balance of the road network are optimized.

[0023] In a third aspect of the present invention, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed control method of urban multi-intersection traffic lights based on an evolved Q network is implemented.

[0024] In a fourth aspect of the present invention, a computer program product includes a computer program, and when the computer program is executed by a processor, the distributed control method of urban multi-intersection traffic lights based on the evolved Q network is implemented.

[0025] The present invention has the following beneficial effects:

[0026] The present invention proposes a method for coordinated control of urban multi-intersection traffic lights based on distributed reinforcement learning. The method decomposes the large-scale coordinated control problem of traffic lights into small-scale sub-problems with the same structure, and uses Q networks to establish action-value functions for the traffic light control sub-problems. Finally, multiple Q networks are coordinated under a distributed control framework to achieve large-scale road network traffic light control. In particular, the present invention realizes automatic matching of the complexity of the Q network and the coordinated control task of traffic lights through the Q network structure evolution mechanism (which can be designed based on the particle swarm algorithm). The application of the method of the present invention can significantly improve the load balance of the road network and reduce the transportation delay time.

[0027] In the distributed control method of urban multi-intersection traffic lights based on the evolutionary Q network of the present invention, a method for converting the forward neural network structure into binary code is first designed, and then the Q network structure is screened and evolved using the Q network structure evolution method to form an automatic matching mechanism for the coordinated control of the Q network structure and the traffic lights, which automatically matches the complexity of the neural network structure and the coordinated control task of the traffic lights. Compared with manual screening, it improves the efficiency of structural screening and avoids falling into local optimality. In addition, when solving the large-scale traffic light control problem, based on the distributed control framework, the large-scale traffic light control problem is decomposed into multiple small-scale sub-problems, which reduces the time complexity of calculating the optimal phase of the traffic light and improves the solution efficiency.

[0028] The distributed reinforcement learning-based coordinated control method for urban multi-intersection signal lights proposed in the present invention can coordinate the phases of all signal lights in the road network in a unified manner according to the road intersection sensor data, thereby achieving the significant effect of balancing the road network load and improving traffic efficiency.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a method for coordinated control of urban multi-intersection traffic lights based on distributed reinforcement learning according to an embodiment of the present invention.

[0031] Figure 2 Schematic diagram of traffic at a double cross intersection in an embodiment of the present invention.

[0032] Figure 3 Schematic diagram of a typical eight-phase signal light in an embodiment of the present invention.

[0033] Figure 4 A schematic diagram of two-dimensional grid coding according to an embodiment of the present invention.

[0034] Figure 5 Schematic diagram of a Q network model for coordinated control of traffic lights according to an embodiment of the present invention.

[0035] Figure 6Schematic diagram of a distributed control model of traffic lights for a six-intersection road network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0037] The present invention proposes a method for coordinated control of traffic lights at multiple intersections in a city based on distributed reinforcement learning. The specific application scenario is that in a vehicle-road collaborative environment, road intersection sensors can monitor the length of waiting convoys and vehicle waiting time on the road in real time. A central controller is provided in the multi-intersection area of ​​the city, which can coordinate the phases of all traffic lights in the road network according to the road intersection sensor data to achieve a significant effect of balancing the road network load and improving traffic efficiency.

[0038] See also Figure 1 The embodiment of the present invention provides a distributed control method for urban multi-intersection signal lights based on an evolved Q network, comprising the following steps:

[0039] Step 1: Establish a Markov model: Construct a Markov model for signal collaborative control, define the agent, environment, state, behavior and reward, and provide a basic model for traffic light control.

[0040] In a preferred embodiment, step one specifically includes the following steps:

[0041] Step 1.1 (Lane load evaluation): Take the weighted sum of the waiting queue length and stop time of each lane to obtain the evaluation index of the lane's current load;

[0042] Step 1.2 (state definition): Define state variables based on the relative positions of lanes and traffic flow directions, including the load conditions of the uplink and downlink lanes;

[0043] Step 1.3 (Signal light phase setting): Design the phase diagram of the signal light, distinguish the main phase and transition phase, and assign corresponding access rights to each lane;

[0044] Step 1.4 (Definition of collaborative control action space): Define the action space of the collaborative control of the double intersection and clarify the signal light phase implementation strategy corresponding to each action;

[0045] Step 1.5 (Reward function construction): Construct a reward function to evaluate the effectiveness of the traffic light control strategy with the goal of reducing lane load.

[0046] Step 2: Feature extraction: Design a state encoding method to convert the actual traffic state into a binary feature vector to provide standardized input for the Q network.

[0047] See also Figure 4 In a preferred embodiment, step 2 specifically includes the following steps:

[0048] Step 2.1 (state space segmentation): define the state variable as a two-dimensional vector and use arithmetic progression to segment it in two dimensions to form a grid structure;

[0049] Step 2.2 (random bias setting): assign a random bias to the starting point of the arithmetic progression to ensure that the encoding of the state space is diverse and adaptable;

[0050] Step 2.3 (two-dimensional grid construction): construct a two-dimensional grid using an arithmetic progression with random bias, which can discretize the continuous state space;

[0051] Step 2.4 (state vector mapping): Map the state vector to a specific area in the two-dimensional grid, and determine its position in the grid by judging the interval where the state vector is located;

[0052] Step 2.5 (binary feature vector generation): Generate a corresponding binary feature vector according to the position of the state vector in the two-dimensional grid, where only the elements corresponding to the grid position are 1 and the rest are 0;

[0053] Step 2.6 (Multi-grid code integration): If there are multiple grids with different random biases, the binary feature vectors generated by each grid are integrated to form the final state code.

[0054] Step 3: Q network structure encoding: Based on the feature extraction results, a Q network structure encoding method is designed to convert the neural network structure parameters into binary codes to prepare for the automatic search for the optimal network structure.

[0055] See also Figure 5 In a preferred embodiment, step three specifically includes the following steps:

[0056] Step 3.1 (input coding definition): The binary coding of the state variables of two adjacent intersections obtained in step 2 is used as the input of the Q network;

[0057] Step 3.2 (Action value function output): The Q network outputs the action value functions corresponding to the four feasible behaviors, which are used to evaluate the potential value of each behavior;

[0058] Step 3.3 (objective function construction): Construct the objective function of Q network training, which is based on the state transition and reward function of the time series and uses the minimum mean square error as the optimization criterion;

[0059] Step 3.4 (binary coding of network structure): define a vector of the number of nodes for the R-layer neural network, and convert each element into a binary number to form a binary coding of the network structure;

[0060] Step 3.5 (parameter update rule formulation): According to the objective function, use the stochastic gradient descent method to update the Q network parameters to ensure that the parameter update can reduce the prediction error and improve the network performance;

[0061] Step 3.6 (network structure optimization): Ensure that the binary encoding of the network structure meets certain range conditions to ensure the diversity and effectiveness of the network structure.

[0062] Step 4: Q network structure evolution: Initialize and execute the Q network structure evolution algorithm, use optimization algorithms such as particle swarm optimization, automatically adjust the Q network structure according to the state code, find the optimal solution, and achieve automatic matching of the Q network structure with the complexity of the traffic light collaborative control task.

[0063] In a preferred embodiment, step 4 designs a particle swarm algorithm, which specifically includes the following steps:

[0064] Step 4.1 (population initialization): Randomly generate a population representing the network structure, where each individual represents a potential Q network structure;

[0065] Step 4.2 (velocity vector initialization): Initialize the velocity vector for each individual, which will guide the search process of the network structure;

[0066] Step 4.3 (individual best position record): record the best position reached by each individual during the search process;

[0067] Step 4.4 (Definition of field size and maximum speed): Set parameters such as field size and maximum speed to control the intensity and scope of the search process;

[0068] Step 4.5 (Neighbor search and update rule application): For each network structure code, find its neighbors and apply the update rule to adjust the speed and position by combining the individual historical best position and the neighbor's best position;

[0069] Step 4.6 (speed adjustment and normalization): according to the speed adjustment rules and the maximum speed limit, the speed vector is normalized to ensure the stability of the search process;

[0070] Step 4.7 (network structure update): According to the updated velocity vector, adjust the network structure encoding to form a new Q network structure;

[0071] Step 4.8 (Q network parameter update): decode the updated network structure into Q network parameters, and update these parameters through the training process until convergence;

[0072] Step 4.9 (Performance evaluation and best position update): Evaluate the performance of the updated Q network and compare it with the individual historical best position, and update the individual best position;

[0073] Step 4.10 (iterative evolution): Repeat the above process until the stopping condition is met, thereby achieving iterative evolution and optimization of the Q network structure.

[0074] Step 5. Distributed control coordination algorithm: Build and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, realize distributed processing of the problem, and continuously improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

[0075] See also Figure 6 In a preferred embodiment, step five specifically includes the following steps:

[0076] Step 5.1 (Establishment of multi-Q network control structure): Establish an independent Q network for each pair of adjacent intersections in the road network, and conduct targeted training for each pair of intersections;

[0077] Step 5.2 (network structure and parameter convergence): Ensure that all Q networks reach a stable state in terms of structure and parameters, providing a reliable decision-making basis for signal light phase selection;

[0078] Step 5.3 (periodic signal light phase optimization): set a periodic time interval, after which the optimal signal light phase action is re-evaluated and selected;

[0079] Step 5.4 (weight definition and action selection): define the weight coefficient, combine the minimization and averaging operations, and select the signal light phase action with the best overall efficiency;

[0080] Step 5.5 (Scale-adaptive solution strategy): Select an appropriate solution strategy based on the scale of the network. For smaller networks, use the exhaustive method, and for larger networks, use the worst-case-first algorithm.

[0081] Step 5.6 (Available action value calculation and optimization): Calculate the available action values ​​of all Q networks at each time point and select the optimal action based on this;

[0082] Step 5.7 (selection of double intersection pairs and determination of actions): select a double intersection pair with the minimum action value from the road network system, and determine the optimal action of the pair;

[0083] Step 5.8 (Optimization problem scale assessment): Assess the scale of the current optimization problem and determine whether it is suitable for solving by exhaustive method;

[0084] Step 5.9 (iterative coordination and solution): If the current problem size is suitable for the exhaustive method, then solve it; otherwise, iterate steps 5.6 and 5.7 until the optimal solution is found.

[0085] The embodiment of the present invention also provides a distributed control method for urban multi-intersection signal lights based on an evolved Q network, comprising the following implementation steps:

[0086] Step A, real-time status monitoring and model application: real-time monitoring of traffic status at multiple intersections in the city, and inputting the monitoring data into the established Markov model to achieve comprehensive evaluation of the agent, environment, status, behavior and reward, and provide decision support for real-time control of traffic lights;

[0087] Step B, state vector conversion and input: Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network to ensure that the algorithm can process real-time traffic data;

[0088] Step C, distributed control execution: using the converted state vector, executing the distributed control coordination algorithm, decomposing the large-scale traffic light control problem into multiple small-scale sub-problems, and realizing efficient traffic light control through distributed processing; wherein each sub-problem is solved by an independent Q network, and the coordination mechanism is used to ensure the coordinated optimization of the overall road network, so as to realize efficient traffic light control;

[0089] Step D, adaptive network structure optimization: through the Q network structure encoding method, the Q network structure parameters are automatically adjusted to adapt to the complexity of different traffic scenarios; through the Q network structure evolution algorithm, using optimization technologies such as particle swarm algorithm, the Q network structure is automatically adjusted according to the state coding, the optimal solution is found, the adaptive evolution of the Q network structure is realized, the traffic light control strategy is optimized, and the automatic matching of the Q network structure and the complexity of the traffic light collaborative control task is realized;

[0090] Step E: Continuous performance evaluation and adjustment: Periodically evaluate the control effect of the traffic lights, and adjust the Q network parameters and control strategies based on the evaluation results; optimize the traffic efficiency and load balance of the road network through continuous performance evaluation and adjustment.

[0091] The present invention proposes an innovative distributed control method for urban multi-intersection traffic lights based on evolutionary Q networks. Based on reinforcement learning, a distributed traffic light control framework for urban road networks is proposed to achieve significant traffic management and control effects, improve the load balance of road networks, and reduce transportation delay time. First, the present invention constructs a Markov model of signal cooperative control, which can comprehensively evaluate intelligent agents, environments, states, behaviors and rewards to provide decision support for real-time control. Secondly, the present invention adopts a state encoding method to effectively convert the actual traffic state into a binary feature vector to provide standardized input for the Q network, which enhances the ability of the algorithm to process real-time traffic data. Furthermore, the present invention converts the neural network structure parameters into binary codes through the Q network structure encoding method, and uses the Q network structure evolution algorithm, such as the particle swarm algorithm, to automatically adjust the Q network structure to achieve automatic matching with the complexity of the traffic light cooperative control task, avoiding the inefficiency of manually setting the network structure and the risk of falling into the local optimum. An important advantage of the present invention is that the distributed control coordination algorithm is used to decompose the large-scale traffic light control problem into multiple small-scale sub-problems, which not only reduces the time complexity of calculating the optimal phase of the traffic light, but also improves the solution efficiency. This method is particularly suitable for large urban traffic networks, and effectively solves the problem of exponential growth of decision space due to the increase in the number of traffic lights. In addition, the distributed control framework of the present invention coordinates multiple Q networks to achieve efficient control of traffic lights in large-scale road networks, and continuously improves the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

[0092] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0093] The method of the present invention mainly includes the following parts: 1. Constructing a Markov model for collaborative control of urban road network traffic lights, defining elements such as intelligent agents, environments, states, behaviors, and rewards; 2. Designing a sparse grid coding method to convert road loads into binary feature vectors; 3. Designing a Q network structure coding method; 4. Designing a Q network structure evolution algorithm; 5. Constructing a distributed computing method to achieve large-scale collaborative control of road network traffic lights.

[0094] like Figure 1 As shown, the implementation process of the present invention mainly includes the following steps:

[0095] (1) Establishment of the Markov model of signal cooperative control

[0096] Figure 2 It is a typical adjacent intersection, each lane has its own unique number. Indicates the length of the waiting queue in lane i With queue stop time The weighted sum of is the evaluation index of the existing load of the lane, which is

[0097]

[0098] Among them, ω∈(0,1) is the weight.

[0099] The light blue track represents the track where vehicles are allowed to pass. i and j represent the preceding lane and following lane of the connection trajectory, respectively. Represents the relative load between the i-th lane and the j-th lane.

[0100] The traffic lights at the two intersections are coordinated. The middle lane is most significantly affected. Since the right-turn lane is less affected by the traffic lights, the left-turn lane and the through lane are considered. Figure 2 For example, define lanes 8 and 9 as up lanes, and lanes 11 and 12 as down lanes. Therefore, the state is defined as

[0101]

[0102] Among them, for the up lane, Where i represents the number of all upstream lanes, max represents the maximum value, and mean represents the average value. Indicates the down lane.

[0103] Figure 3 This is a phase diagram of a typical eight-phase signal light, where G means priority, g means passing if there is no conflict with the priority vehicle, r means no passing, and the phase combination GGgGrrGGgGrr respectively represents the phases of lanes 1, 2, 3, 7, 8, 9, 14, 15, 16, 20, 21, and 22. At the same time, phase 1 and phase 5 are the main phases with a longer duration, and phases 2-4 and 6-8 are transition phases.

[0104] Define the double intersection cooperative control action space a t ∈{[0,0],[1,0],[0,1],[1,1]}, 0 means implementation starts from phase 1 and ends at phase 4, 1 means implementation starts from phase 5 and ends at phase 8. In this example, the time to implement phases 1-4 and 5-8 is equal, let it be ΔT.

[0105] The reward is defined as

[0106]

[0107] in,

[0108] (2) Design of signal cooperative control state coding method

[0109] Dependent state variable is a two-dimensional vector, and grid coding is used to convert s t Converted into a binary vector,

[0110] like Figure 4 As shown in the two-dimensional grid encoding, the state space is and The two dimensions are divided by the following arithmetic progression:

[0111]

[0112] Among them, the common difference of the series X0 and X1 is the same, both are ΔX, X 0,0 , X 1,0 Randomly select a value in the interval [-ΔX, 0].

[0113] X0 and X1 form a two-dimensional grid with random bias, which corresponds to a one-dimensional feature vector containing N0*N1 elements. for like

[0114]

[0115] It is believed that s t Appears on the grid {X 0,i ,X 0,i+1 ,X 0,j ,X 0,j+1}middle, The iN2+j+1th element is 1, and the rest are 0.

[0116] If there are L grids with different random biases, the state vector can be encoded as

[0117]

[0118] (3) Q network structure coding method design

[0119] This embodiment designs a traffic light cooperative control based on a Q network. The Q network model is as follows: Figure 5 As shown, the input of the Q network is the state variable between two adjacent intersections. The binary code of Φ s The output is the action value function Q(s,a1),Q(s,a2),Q(s,a3),Q(s,a4) corresponding to the four feasible behaviors a1=[1,1],a2=[1,0],a3=[0,1],a4=[0,0]. The objective function of Q network training is

[0120]

[0121] Among them, Φ c represents the binary encoding of the network structure. For an R-layer neural network, let the vector P = [p1 p2... p R , which are the number of nodes in each layer of the neural network respectively, and satisfy

[0122]

[0123] Convert p k directly into a binary number, and then Φ c can be obtained. Φ c is a binary vector with a length of R * P k .

[0124] θ t represents the Q-network parameters corresponding to the structure Φ c . For a fixed Φ c , the random gradient descent method is used to obtain θ that minimizes the objective function value:

[0125]

[0126] (4) Design of Q-network structure evolution algorithm

[0127] The particle swarm algorithm is used to search for the Q-network structure, and the following steps are executed:

[0128] Initialize a random population representing the network structure

[0129] Initialize the R * P k dimensional velocity vector v i of each individual, i ∈ [1, M]

[0130] Initialize the best position of each individual so far:

[0131] Define the neighborhood scale ε < F, the maximum influence value μ 1,max and μ 2,max , the maximum velocity v max . The pseudo-code of the algorithm is as follows:

[0132] While not

[0133] For each network structure encoding

[0134]

[0135] Generate two random vectors μ1, μ2

[0136] where μ 1,k ~U(0, μ 1,max ), μ2,k ~U(0,μ 2,max ),k∈[1,R*P k ],implement

[0137]

[0138] If|v i |>v max ,then

[0139] v i ←v i v max / |v i |

[0140] End if

[0141]

[0142] Will Decoded into the corresponding Q network, set its parameters to use

[0143]

[0144] in,

[0145] Train the network until convergence and get Value

[0146]

[0147] Next grid structure encoding

[0148] Next generation grid structure coding

[0149] (5) Design of Distributed Control Coordination Algorithm for Road Network Signal Lights

[0150] For Figure 6 The six-intersection road network shown in the figure is controlled by seven Q networks, each of which is trained for two adjacent intersections. (i,j)=(1,2),(2,3),(4,5),(5,6),(1,4),(2,5),(3,6). When these Q network structures and parameters converge, they are used to select the phase of the signal light. To optimize the overall efficiency of the road network, it is set to reselect after every ΔT. make have

[0151]

[0152] Among them, max represents the maximum value, mean represents the average value, min represents the minimum value, and β∈(0,1) is the weight.

[0153] For smaller networks, the exhaustive method is used to solve the problem, and for larger networks, the worst case first algorithm is used:

[0154] Step 1: At time t, calculate the action values ​​of all Q networks and select The corresponding optimal action and its corresponding action value

[0155] Step 2: For all pairs of double intersections (i, j) in the road network system, select and confirm

[0156] Step 3: In (i * ,j * ) corresponds to the action value When determining, check whether the optimization problem is degenerated to a scale that can be solved by exhaustive method. If so, use exhaustive method to solve it. If not, after determining (i * ,j * ), re-implement steps 1 and 2.

[0157] In summary, the embodiments of the present invention propose a method for distributed control of urban multi-intersection traffic lights based on an evolutionary Q network, wherein: (1) the neural network structure is automatically matched with the complexity of the traffic light collaborative control task through a Q network structure evolution mechanism (preferably a particle swarm algorithm is designed). (2) The forward neural network structure is represented as a binary vector through a Q network structure encoding method. (3) A worst-case priority algorithm is proposed to solve the problem of high computational time complexity of large-scale distributed control of traffic lights. (4) Based on sparse grid coding, a binary coding method for characterizing the urban road network load is proposed.

[0158] In the distributed control method of urban multi-intersection traffic lights based on the evolutionary Q network of the present invention, a method for converting the forward neural network structure into binary code is first designed, and then the Q network structure is screened and evolved using the Q network structure evolution method to form an automatic matching mechanism for the coordinated control of the Q network structure and the traffic lights, which automatically matches the complexity of the neural network structure and the coordinated control task of the traffic lights. Compared with manual screening, it improves the efficiency of structural screening and avoids falling into the local optimum. In addition, when solving large-scale traffic light control problems, based on the distributed control framework, the worst case priority algorithm is preferably used to reduce the time complexity of calculating the optimal phase of the traffic lights and improve the solution efficiency.

[0159] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0160] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0161] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0162] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0165] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0166] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0167] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0168] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0169] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0170] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0171] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A distributed control method for urban multi-intersection signal lights based on an evolutionary Q network, characterized in that: The following steps are involved: Step 1: Build a Markov model for signal collaborative control, define the agent, environment, state, behavior, and reward, and provide a basic model for traffic light control; Step 2: Convert the actual traffic state into a binary feature vector through the state encoding method to provide standardized input for the Q network; Step 3: Based on the feature extraction results, the neural network structure parameters are converted into binary codes through the Q network structure coding method to prepare for the automatic search for the optimal network structure; Step 4: Use the Q network structure evolution algorithm to automatically adjust the Q network structure according to the state code, find the optimal solution, and achieve automatic matching between the Q network structure and the complexity of the traffic light collaborative control task; Step 5: Build and execute a distributed control coordination algorithm for road network traffic lights, decompose the large-scale traffic light control problem into multiple small-scale sub-problems, realize distributed processing of the problem, and improve the traffic efficiency and load balance of the road network through periodic evaluation and optimization.

2. The distributed control method for urban multi-intersection signal lights based on the evolutionary Q network as claimed in claim 1 is characterized in that: Step 1 specifically includes: The waiting queue length and stop time of each lane are weighted summed to obtain the evaluation index of the lane's current load; According to the relative position of the lanes and the traffic flow direction, the state variables are defined, including the load conditions of the up and down lanes; Design the phase diagram of the traffic light, distinguish the main phase and transition phase, and assign corresponding access rights to each lane; Define the action space for coordinated control of dual intersections and clarify the signal light phase implementation strategy corresponding to each action; Construct a reward function to evaluate the effectiveness of the traffic light control strategy with the goal of reducing lane load.

3. The distributed control method for urban multi-intersection signal lights based on the evolutionary Q network as claimed in claim 1 is characterized in that: Step 2 specifically includes: Define the state variable as a two-dimensional vector and use arithmetic progression to divide it in two dimensions to form a grid structure; Assigning random bias to the starting point of the arithmetic progression ensures that the encoding of the state space is diverse and adaptable; A two-dimensional grid is constructed using an arithmetic progression with random biases, which can discretize the continuous state space. Map the state vector to a specific area in a two-dimensional grid, and determine its position in the grid by judging the interval where the state vector is located; According to the position of the state vector in the two-dimensional grid, a corresponding binary feature vector is generated, in which only the elements corresponding to the grid position are 1 and the rest are 0; If there are multiple grids with different random biases, the binary feature vectors generated by each grid are integrated to form the final state encoding.

4. The distributed control method for urban multi-intersection signal lights based on the evolutionary Q network as claimed in claim 1 is characterized in that: Step three specifically includes: The binary codes of the state variables of two adjacent intersections obtained in step 2 are used as the input of the Q network; The Q network outputs action-value functions corresponding to four feasible behaviors, which are used to evaluate the potential value of each behavior; Construct the objective function of Q network training, which is based on the state transition and reward function of the time series and uses the minimum mean square error as the optimization criterion; Define a vector of the number of nodes for the R-layer neural network, and convert each element into a binary number to form a binary code of the network structure; According to the objective function, the Q network parameters are updated using the stochastic gradient descent method to ensure that the parameter update can reduce the prediction error and improve the network performance; The binary encoding of the network structure meets the set range conditions to ensure the diversity and effectiveness of the network structure.

5. The distributed control method for urban multi-intersection signal lights based on the evolutionary Q network as claimed in claim 1 is characterized in that: Step 4 specifically includes: Randomly generate a population representing the network structure, each individual represents a potential Q network structure; Initialize the velocity vector for each individual, which will guide the search process of the network structure; Record the best position achieved by each individual during the search process; Setting parameters such as field size and maximum speed to control the intensity and scope of the search process; For each network structure code, find its neighbors and apply the update rule to adjust the speed and position by combining the individual historical best position and the best position of the neighbors; According to the speed adjustment rules and the maximum speed limit, the speed vector is normalized to ensure the stability of the search process; According to the updated velocity vector, the network structure encoding is adjusted to form a new Q network structure; Decode the updated network structure into Q network parameters and update these parameters through the training process until convergence; Evaluate the updated Q network performance and compare it with the individual's historical best position to update the individual's best position; The above process is repeated until the stopping condition is met, thereby achieving iterative evolution and optimization of the Q network structure.

6. The distributed control method for urban multi-intersection signal lights based on the evolved Q network as claimed in claim 1, characterized in that: Step 5 specifically includes: Establish an independent Q network for each pair of adjacent intersections in the road network, and conduct targeted training for each pair of intersections; Ensure that all Q networks reach a stable state in terms of structure and parameters, providing a reliable decision-making basis for signal light phase selection; Set a periodic time interval, after which the optimal signal light phase action is re-evaluated and selected; Define weight coefficients, combine minimization and averaging operations, and select the signal light phase action with the best overall efficiency; According to the scale of the road network, choose an appropriate solution strategy, use the exhaustive method for smaller networks, and use the worst case first algorithm for larger networks; Calculate the possible action values ​​of all Q networks at each time point and select the best action based on this; Selecting a pair of double intersections with the minimum action value from the road network system and determining the optimal action of the pair; Assess the scale of the current optimization problem and determine whether it is suitable to be solved by exhaustive method; If the current problem size is suitable for the exhaustive method, then solve it; otherwise, iterate until the optimal solution is found.

7. A distributed control method for urban multi-intersection signal lights based on an evolutionary Q network, characterized in that: The following steps are involved: Real-time monitoring of traffic conditions at multiple intersections in the city, and inputting the monitoring data into the established Markov model to achieve a comprehensive evaluation of the agent, environment, state, behavior, and reward, providing decision support for real-time control of traffic lights; Using the designed state encoding method, the actual traffic state information monitored is converted into a binary feature vector as the input of the Q network so that the algorithm can process real-time traffic data; The converted state vector is used to execute the distributed control coordination algorithm to decompose the large-scale traffic light control problem into multiple small-scale sub-problems, and efficient traffic light control is achieved through distributed processing. Each sub-problem is solved by an independent Q network, and the coordination mechanism is used to optimize the overall road network and achieve traffic light control. Among them, the Q network structure encoding method is used to automatically adjust the Q network structure parameters to adapt to the complexity of different traffic scenarios; the Q network structure is automatically adjusted according to the state encoding through the Q network structure evolution algorithm to find the optimal solution and realize the automatic matching of the Q network structure and the complexity of the signal light collaborative control task; Among them, the traffic light control effect is evaluated periodically, and the Q network parameters and control strategies are adjusted according to the evaluation results; through continuous performance evaluation and adjustment, the traffic efficiency and load balance of the road network are optimized.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the distributed control method of urban multi-intersection signal lights based on an evolved Q network as described in any one of claims 1 to 7 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the distributed control method of urban multi-intersection signal lights based on an evolved Q network as described in any one of claims 1 to 7 is implemented.

Images