A multi-signal control method and system for lane capacity differences

Abstract

The present invention discloses a multi-signal control method and system for lane capacity differences, which models key intersection information based on an improved maximum pressure method, defines the intersection environment and a new pressure with dual perception of lane capacity and average vehicle speed; designs the actions of adjacent intersections and phase durations according to the modeled information; on this basis, designs the three elements of reinforcement learning: state, action and reward; uses the Nature Deep Q Network reinforcement learning algorithm with a double Q network structure for the agents at each intersection, and dynamically controls the signal phase and phase duration according to the real-time environment of the intersection. Based on the accurate modeling of key information of heterogeneous intersections and the simple setting of reinforcement learning elements, the present invention can better implement control strategies, thereby minimizing the average travel time of vehicles in the heterogeneous road network to the greatest extent.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic signal control, and particularly relates to a multi-signal light control method and system for lane capacity differences. Background Art

[0002] In recent years, with the continuous development of the global economy and the rapid advancement of the urbanization process, people's travel demands have been increasing day by day. The continuously growing vehicle ownership has led to increasingly serious traffic congestion in various places, which in turn has caused related problems such as traffic accidents and environmental deterioration. Therefore, alleviating traffic congestion and improving transportation efficiency have become a top priority. Intersections, as key nodes in the urban road network, play an extremely important role in aspects such as traffic flow guidance and road right allocation. Using intelligent signal control technology at intersections to improve the traffic capacity of the urban road network is an important means to alleviate traffic congestion. In the critical period of smart transportation construction, how to make full use of the real-time data collected by advanced Internet of Things devices and design excellent signal control methods is an important prerequisite for realizing smart travel.

[0003] There are mainly two traditional traffic signal control methods, namely fixed-time control and inductive control. The former cannot respond to the dynamic changes of traffic, while the latter can only consider the instantaneous traffic state. In recent years, with the development of artificial intelligence in the traffic field, signal control based on deep reinforcement learning (DRL) has become the most promising method to alleviate traffic congestion. This method can learn the control strategy of traffic lights in real time according to the dynamic and changeable traffic conditions, thereby alleviating traffic congestion and improving travel efficiency.

[0004] However, most of the current traffic light control methods based on deep reinforcement learning either have overly simplified models that do not consider the interaction between adjacent intersections and are difficult to extract key content from complex road network information, or have overly complex models, or simply transplant the latest reinforcement learning algorithms, resulting in low policy learning efficiency. Secondly, in order to reduce the dimension of the action space, many reinforcement learning-based control methods assume that the phase duration is fixed, which greatly limits the ability of the agent to search for better control strategies. Even many studies, in order to facilitate testing the superiority of their learning algorithms, assume that the road network is completely isomorphic, reducing the real characteristics of the road network and causing the research content to deviate from reality. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-signal light control method and system for lane capacity differences in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem of congestion caused by different road capacities in heterogeneous intersections, significantly reduce the average travel time of vehicles in heterogeneous intersections, and improve the traffic efficiency of the road network.

[0006] The present invention adopts the following technical solutions:

[0007] A multi-signal lamp control method for lane capacity differences includes the following steps:

[0008] S1. Model the key information of the intersection based on the improved maximum pressure method to obtain an intersection model, and define the pressure models of lanes, traffic flows, phases, and intersections;

[0009] S2. Design the pressure action and phase duration of adjacent intersections according to the intersection model obtained in step S1;

[0010] S3. Based on the pressure model and intersection environment model defined in step S1, and the adjacent intersection actions in step S2, design the agent state, agent action, and agent reward;

[0011] S4. Construct a reinforcement learning framework based on the agent state, agent action, and agent reward defined in step S3, use the Nature Deep Q Network reinforcement learning algorithm with a double Q network structure for the agent of each intersection, and dynamically control the signal phase and phase duration according to the real-time environment of the intersection.

[0012] Specifically, in step S1, the intersection model is a two-way three-lane model. The three lanes can turn left, go straight, and turn right respectively, and the right-turning vehicles are not controlled by traffic signals; each intersection consists of a group of incoming lanes lane in and a group of outgoing lanes lane out There are a total of 12 incoming lanes lane1, lane2,..., lane 12 and 12 outgoing lanes, and the four phases are north-south straight, east-west straight, north-south left turn, and east-west left turn.

[0013] Specifically, in step S1, the pressure model specifically includes:

[0014] The pressure of lane i

[0015]

[0016] where x i is the number of vehicles on lane i, c i is the maximum capacity of lane i, c ∞ and m are parameters for adjusting the shape of the pressure function, is the average vehicle speed of the current lane, v max is the maximum allowable speed of the lane, and δ is a weight factor for adjusting the influence of speed on pressure;

[0017] The pressure of the moving traffic flow P flowand the static traffic flow pressure P stop is:

[0018]

[0019]

[0020] Wherein, and are respectively the total number of inbound lanes and the total number of outbound lanes in the direction of traffic flow movement, and are respectively the total number of inbound lanes and the total number of outbound lanes where the traffic flow stops;

[0021] The phase pressure P phase is:

[0022]

[0023] Wherein, λ1 and λ2 are respectively the weight coefficients of the moving traffic flow pressure and the static traffic flow pressure;

[0024] The intersection pressure P I is:

[0025]

[0026] Specifically, in step S2, the action of the adjacent intersection pressure is:

[0027]

[0028] Wherein, ω is the contraction coefficient for restricting the pressure value; C max is the maximum lane capacity; C I,I′ is the capacity of the lane between intersections I and I′; alane in ′ is the inbound lane of I′, and the vehicles on this lane can pass through intersection I′ and drive towards I under the current phase control; n′ is the number of vehicles that can pass through I′ per unit time for each lane; t′ is the remaining time of the current phase green light at intersection I′; N i ′ is the number of vehicles on the i-th inbound lane of alane in ′; alane in , n, t, N i are the relevant information of the current intersection I.

[0029] Specifically, in step S2, the phase duration T duration is specifically:

[0030] According to the given t max and t min find a set of M optional phase durations, t min and t maxrespectively represent the minimum and maximum phase durations at intersections to ensure the stable passage of vehicles;

[0031] When M = 1, the phase duration is fixed at t max ;

[0032] When M > 1, construct the phase duration set D;

[0033] After selecting the signal light phase, based on the number of vehicles N i on the approach lane lane i and the number of vehicles n that can pass through the intersection per unit time in this lane, obtain the time t i for all vehicles on the approach lane lane i to pass through the intersection; determine the phase duration T duration .

[0034] Furthermore, the duration T duration is:

[0035]

[0036] where t m as the maximum value of the time t in required for all vehicles on the approach lane lane i to pass through. i of.

[0037] Specifically, in step S3, a numerical-based representation is used to design the agent state. The agent state consists of the environmental information of the intersection and its surrounding neighbors, including the pressure of the phase, the pressure influence of the neighbors, and the current phase; the four adjacent intersections of the current intersection I are I1′, I2′, I3′, I4′, and the state vector is encoded as

[0038] Select the best control phase of the intersection as the agent action. Each agent has four different predefined allowed actions, and the action set is {SNS, EWS, SNL, EWL};

[0039] Considering the lane capacity and average vehicle speed, the reward function r is:

[0040] r = -P I

[0041] where P I is the pressure of intersection I.

[0042] Specifically, in step S4, the Nature Deep Q Network reinforcement learning algorithm is specifically:

[0043] S401. Set parameters, including the number of training times R, simulation step size S, discount factor γ, learning rate α, exploration rate ε, and sampling batch size (Batch Size).

[0044] S402. Initialize the parameters θ of the main network and the parameters θ of the target network, and initialize the experience replay pool M. - , and initialize the experience replay pool M.

[0045] S403. Start traffic simulation and obtain the intersection environment s. t ;

[0046] S404. Randomly select an action a with probability ε, otherwise select an action a according to the current state s. t , otherwise according to the current state s. t Select the action a. t , ;

[0047] S405. After executing the action a, calculate the duration T according to the current intersection information. t After executing the action a, calculate the duration T according to the current intersection information. duration ;

[0048] S406. Obtain a new round of intersection environment s and reward r, and store the experience sample {s, a, r, s} in the experience replay pool M. If the experience pool overflows, replace the earliest sample record with a new sample. t+1 and reward r. t+1 , store the experience sample {s, a, r, s} in the experience replay pool M. If the experience pool overflows, replace the earliest sample record with a new sample. t , a t , r t+1 , s t+1}, if the experience pool overflows, replace the earliest sample record with a new sample.

[0049] S407. Randomly draw a small batch of samples with a batch size of E from the experience replay pool M to update the main network Q and the target network Q'. If the number of training times has not reached the set value, go to step S404, otherwise go to step S408.

[0050] S408. End.

[0051] Furthermore, in step S404, the action a is specifically: t Specifically:

[0052] .

[0053] In a second aspect, an embodiment of the present invention provides a multi-signal light control system for lane capacity differences, including:

[0054] A construction module that models key intersection information based on an improved maximum pressure method to obtain an intersection model, and defines the pressure models of lanes, traffic flows, phases, and intersections.

[0055] A design module that designs the adjacent intersection pressure effect and phase duration according to the intersection model obtained by the construction module.

[0056] Element module, based on the pressure model and intersection environment model defined by the building module, and the interaction of adjacent intersections in the design module, designs the agent state, agent action, and agent reward;

[0057] Control module, based on the agent state, agent action, and agent reward defined by the element module, constructs a reinforcement learning framework, uses the Nature Deep QNetwork reinforcement learning algorithm with a double Q-network structure for the agents at each intersection, and dynamically controls the signal phase and phase duration according to the real-time environment of the intersection.

[0058] Compared with the prior art, the present invention has at least the following beneficial effects:

[0059] A multi-signal light control method for lane capacity differences, by constructing an intersection pressure model, designing adjacent pressure effects and phase durations, defining the three elements of reinforcement learning, and using a deep reinforcement learning algorithm, the four designed modules form a complete and efficient signal light control system from the construction of microscopic single-intersection pressure to macroscopic road network control. At the same time, each module has low coupling, enhancing the maintainability of the system.

[0060] Furthermore, the constructed intersection model is a two-way three-lane model, and the three lanes can turn left, go straight, and turn right respectively. Based on the high-precision intersection model at the lane level, the practical significance and application value of the proposed control method are enhanced; the four phases are north-south straight, east-west straight, north-south left turn, and east-west left turn, designing a classic conflict-free four-phase to reduce the signal switching frequency and improve traffic efficiency.

[0061] Furthermore, the constructed pressure model integrates the dual perception strategies of capacity and vehicle speed, and realizes more refined perception of key information at intersections by defining a new "pressure", so as to effectively suppress the intersection congestion problem caused by different road capacities.

[0062] Furthermore, this method considers the coordination between adjacent intersections. By introducing the interaction of pressures between adjacent intersections, the current intersection can more accurately perceive the traffic flow dynamics of adjacent intersections. At the same time, aiming at the differences in lane capacities, greater pressure is given to adjacent intersections with shorter lane capacities, effectively alleviating the problem that the congested vehicle flow of heterogeneous intersections spreads to adjacent intersections.

[0063] Furthermore, this method designs a dynamically adjustable phase duration selection strategy according to real-time traffic information, selects the best phase duration in the dynamic set D, and realizes the dynamic optimization of signal phase time.

[0064] Furthermore, determine the phase time for all vehicles to pass under the current phase, that is, ensure the release of all pressures under the current phase, so as to further optimize the average travel time.

[0065] Furthermore, among the three elements of reinforcement learning designed in this method, the state adopts a numerical-based representation form. Compared with the existing structure-based representation form, it has stronger generalization ability and can handle different heterogeneous intersections; the action is designed to select a phase from a predefined set of phases, handing over the power of phase selection to the intelligent agent, enhancing the dynamics of learning. The reward combines traffic engineering-related theories and designs a reward function based on intersection pressure. Compared with the traditional reward settings based on waiting time, queue length, or weighted accumulation of multiple indicators, it has a more solid theoretical foundation.

[0066] Furthermore, in the Nature Deep Q Network reinforcement learning sub-algorithm with a dual-network structure, the intelligent agent obtains the best solution for signal control through frequent interactions with the traffic environment; it is worth noting that due to the simple state setting, this method can use a simple neural network structure to achieve good learning effects.

[0067] Furthermore, to prevent the intelligent agent's action exploration from falling into a local optimal solution, randomly select action a with probability ε t , otherwise select the optimal output action of the intelligent agent corresponding to the current state.

[0068] It can be understood that the beneficial effects of the second aspect can be seen in the relevant descriptions in the first aspect above, and will not be elaborated here.

[0069] In summary, a multi-traffic signal control system for lane capacity differences disclosed in this method proposes a signal control method based on the dual perception of lane capacity and average vehicle speed and the maximum pressure method under the framework of deep reinforcement learning. It can perceive key information at intersections more precisely, has strong generalization ability, effectively improves the average travel time of vehicles in heterogeneous intersections, and alleviates traffic congestion.

[0070] Next, through the drawings and embodiments, the technical solutions of the present invention will be further described in detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a schematic diagram of an intersection model;

[0072] Figure 2 is a schematic diagram of the relationship between lane pressure and the number of real-time vehicles on the lane under different lane capacities;

[0073] Figure 3 is a schematic diagram of a neural network structure;

[0074] Figure 4 Schematic diagram of the pressure effect between adjacent intersections

[0075] Figure 5 Flowchart of the present method

[0076] Figure 6 Schematic diagram of the large-scale road network simulation environment

[0077] Figure 7 Evolution processes of the number of vehicles and the average travel time in the road network over time under two signal control algorithms; among them, (a) is the evolution process diagram of the number of vehicles in the road network over time, and (b) is the evolution process diagram of the average travel time of vehicles in the road network over time Specific implementation manners

[0078] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention

[0079] In the description of the present invention, it should be understood that the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations

[0080] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms

[0081] It should be further understood that the term " / and" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent: the existence of A alone, the existence of both A and B, and the existence of B alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the preceding and following associated objects

[0082] It should be understood that although terms such as first, second, and third may be used in the embodiments of the present invention to describe preset ranges and the like, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0083] Depending on the context, as used herein, the word "if" can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0084] Various schematic structural diagrams according to the disclosed embodiments of the present invention are shown in the drawings. These figures are not drawn to scale, where certain details are enlarged for the purpose of clear expression, and certain details may be omitted. The shapes of various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary, and may actually deviate due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0085] The present invention provides a multi-traffic-light control method for lane capacity differences, which models key information of intersections based on an improved maximum pressure method, defines the intersection environment and a new pressure that senses both capacity and vehicle speed; secondly, designs the actions of adjacent intersections and the phase duration according to the information modeled for the intersections; then, based on the defined pressure, the modeled intersection environment, and the designed actions of adjacent intersections, designs the three elements in the reinforcement learning method: state, action, and reward; finally, uses the Nature Deep Q Network reinforcement learning algorithm with a double Q network structure for the agent of each intersection to dynamically control the signal phase and the phase duration according to the real-time environment of the intersection.

[0086] Please refer to Figure 5 , a multi-traffic-light control method for lane capacity differences according to the present invention, includes the following steps:

[0087] S1. Model the key information of the intersection based on the improved maximum pressure method, and define the intersection environment and a new pressure that senses both lane capacity and average vehicle speed;

[0088] (1) Intersection modeling

[0089] The intersection model established by this method is a two-way three-lane model. The three lanes can turn left, go straight, and turn right respectively, and the right-turning vehicles are not controlled by traffic signals. Each intersection consists of a set of incoming lanes lane in and a set of outgoing lanes lane out . As Figure 1 shown, this intersection has 12 incoming lanes (lane1, lane2,..., lane 12 ) and 12 outgoing lanes, with four phases, namely: north-south straight (SNS), east-west straight (EWS), north-south left turn (SNL), and east-west left turn (EWL).

[0090] (2) Pressure modeling

[0091] In the maximum pressure theory, the definition of pressure is only related to the number of vehicles on the lane, while this method takes into account both the vehicle dynamics characteristics and the differences in lane capacity, thus achieving a more refined modeling of the intersection.

[0092] Define the lane pressure as:

[0093]

[0094] where is the pressure of lane i, x i is the number of vehicles on lane i, c i is the maximum capacity of lane i, c ∞ and m are parameters to adjust the shape of the pressure function, is the average vehicle speed of the current lane, v max is the maximum allowable speed of the lane, and δ is a weight factor that can be used to adjust the influence of speed on pressure. This method uses the pressure of the lane to represent the congestion degree of the lane.

[0095] The lane pressure defined by this invention is composed of the product of the capacity perception term and the speed perception term. The purpose of capacity perception is:[[]]

[0096] One is to make the pressure on the lane and the number of vehicles show a convex function relationship; the other is to normalize the pressure, so as to evaluate the pressure of roads with different capacities on the same scale.

[0097] Please refer to Figure 2 , which shows the relationship between the intensity and the real-time number of vehicles on the lane when the lane capacity c is 10, 40, and 80 respectively; when the number of vehicles on a certain lane increases, the lane pressure increases in a non-linear trend; and when the real-time number of vehicles on the three lanes is the same, the capacity perception term can ensure that the lane with the smallest capacity has the greatest pressure. In the speed perception term, the pressure decreases as the average vehicle speed increases, that is, the faster the vehicle speed, the smoother the lane.

[0098] Define the vehicle flow pressure as:

[0099]

[0100]

[0101] Among them, P flow is the pressure of the moving vehicle flow, and are the total number of inbound lanes and the total number of outbound lanes in the moving direction of the vehicle flow respectively; P stop is the pressure of the stationary vehicle flow, and are the total number of inbound lanes and the total number of outbound lanes where the vehicle flow stops respectively.

[0102] Define the phase pressure as:

[0103]

[0104] The phase pressure P phase refers to the sum of all moving vehicle flow pressures and all stationary vehicle flow pressures under this phase. λ1 and λ2 are the weight coefficients of the moving vehicle flow pressure and the stationary vehicle flow pressure respectively.

[0105] In each intersection, the present invention uses the phase intensity to characterize the congestion degree of the vehicle flow under different phases; when the phase pressure is relatively large, it indicates that the traffic flow demand under this phase is relatively high, then the control strategy should give priority to releasing this phase. This method sets that the right-turn vehicle flow is not controlled by signals, so the right-turn vehicle flow pressure is not considered when calculating the phase pressure.

[0106] Define the intersection pressure as:

[0107]

[0108] The intersection pressure P I is the difference between the pressures of all inbound lanes and outbound lanes of intersection I. The intersection pressure can accurately reflect the congestion degree of this intersection. The greater the pressure, the more congested this intersection is.

[0109] S2. Design the adjacent intersection effect and phase duration according to the information modeled in step S1;

[0110] (1) Adjacent intersection pressure effect

[0111] If the current intersection is I and its neighbor intersection is I′, then the influence of I′ on I is:

[0112]

[0113] Among them, ω is the contraction coefficient for limiting the pressure value; C max is the maximum lane capacity; C I,I′is the capacity of the lane between intersections I and I′; in ' is the I' entry lane. Vehicles in this lane can pass through intersection I' and drive into I under the current phase control; n' is the number of vehicles that can pass through I' in each lane per unit time; t' is the remaining time of the green light at intersection I' in the current phase; N i ' is alane in ′The number of vehicles on the i-th lane; similarly, alane in 、n、t、N i It is the relevant information of the current intersection I.

[0114] The neighboring intersection I' not only serves as a pressure input, but also plays a role in receiving the pressure of intersection I. Figure 4 For example, assuming that the phases of intersections I′ and I are both east-west straight, then intersection I′ has pressure on two incoming lanes, which acts on intersection I in proportion to the remaining green light time of the current phase and the traffic capacity of the intersection; at the same time, intersection I′ also serves as the receiver of pressure and bears the pressure transferred from intersection I; in addition, when considering the pressure of adjacent intersections, this method gives greater pressure to the adjacent intersection with shorter lane capacity based on the difference in lane capacity.

[0115] (2) Phase duration design

[0116] Different from the fixed phase timing of traditional traffic signal control methods, the phase duration designed by this method can be changed according to the real-time traffic conditions at the intersection, which can save green light time to a greater extent.

[0117] In order to design a more reasonable phase duration allocation strategy, firstly according to the given t max and t min Find a set of M selectable phase durations.

[0118] Among them, t min and t max They respectively represent the minimum phase duration and maximum phase duration at the intersection to ensure stable vehicle traffic.

[0119] When M=1, the phase duration is fixed at t max ;

[0120] When M>1, the phase duration set D is constructed as follows:

[0121]

[0122] After selecting the signal light phase, pass the lane i Number of vehicles N iAnd the number of vehicles \(n\) that can pass through the intersection per unit time of this lane is obtained to get the approach lane \(lane\). i The time for all vehicles on

[0123]

[0124] Among them, \(t\) m As the approach lane \(lane\) in All lanes \(lane\) within i The time \(t\) required for vehicles on i The maximum value is denoted as \(t\) m \(=\max\{t\) i \}\), select the number closest to \(t\) m from the set \(D\) as the phase duration, that is, to ensure the release of all pressures in the current phase.

[0125] Set the duration as:

[0126]

[0127] Among them, \(t\in D\), and

[0128] S3. Based on the pressure defined in step S1 and the modeled intersection environment, as well as the influence of adjacent intersections in step S2, design the three elements in the reinforcement learning method: state, action, and reward;

[0129] The ultimate goal of this invention is expressed as the problem of minimizing the intersection pressure considering the influence of adjacent intersections.

[0130] (1) State

[0131] Currently, there are generally two ways to represent the intersection state: the structure-based form and the numerical-based form. The structure-based form usually discretizes the relevant information of the vehicles and signal lights at the intersection and models it into a matrix form. Although the structure-based form can better reduce the input information volume and highlight the structural characteristics of the intersection, as the number of state variables increases, such as when the lane capacity is large, this method will face the problem of dimensional explosion; in addition, the structure-based form is not good at dealing with complex heterogeneous intersections, and how to model heterogeneous intersections into a matrix representation form is an extremely important problem. The numerical-based form directly extracts the feature information of the intersection and combines it into a vector, and uses this vector to represent the intersection state, which has strong generalization ability and can handle different heterogeneous intersections.

[0132] Therefore, this method selects the numerical-based representation to design the agent state. This state consists of the environmental information of the intersection and its surrounding neighbors, including the pressure of the phase, the pressure influence of the neighbors, and the current phase. Figure 1For example, assume that the four adjacent intersections of the current intersection I are I1′, I2′, I3′, and I4′ (if there is no adjacent road in a certain direction, it is recorded as 0), then the state vector encoding is

[0133] (2) Action

[0134] There are usually two designs for the actions of agents in signal control. One is the fixed phase sequence, which selects whether to extend the current phase or switch to the next phase; the other is the unfixed switching order, where the agent selects a phase from a predefined set of phases. The former has a smaller action space and lower complexity, but it makes the intersection lack flexibility, and the quality of the phase sequence design has a greater impact on the learning results. The latter gives the agent the power to select the phase, enhancing the dynamics of learning.

[0135] Therefore, this method defines the action as selecting the optimal control phase of the intersection. Each agent has four different predefined allowed actions, and the action set is {SNS, EWS, SNL, EWL}, corresponding to Figure 1 the four phases shown in

[0136] (3) Reward

[0137] The reward mechanism plays an important role in the learning process. A higher reward means a better action choice. A reward function based on intersection pressure is designed in combination with relevant traffic engineering theories, which has a more solid theoretical basis compared to the traditional reward settings based on waiting time, queue length, or the weighted sum of multiple indicators. This method redefines the intersection pressure, considering the lane capacity and average vehicle speed, and can more accurately reflect the congestion situation of the intersection. The reward function defined by this method is:

[0138] r = -P I

[0139] where P I is the pressure of intersection I defined by this method.

[0140] S4. Use the Nature Deep Q Network reinforcement learning algorithm with a double Q-network structure for the agents at each intersection, and dynamically control the signal phase and phase duration according to the real-time environment of the intersection.

[0141] After completing the establishment of the signal control model and reinforcement learning elements, this method uses the Nature Deep Q Network with two neural network structures to fit the Q function, and its network structure and related parameters are as Figure 3As shown. In this method, a reinforcement learning agent is configured for each intersection. The input of the network model consists of the number of intersections and the state dimension. In the figure, num is the number of intersections, and 9 is the state dimension: 4 phase pressures, 4 adjacent intersection pressures, and the current phase. Subsequently, it passes through hidden layers with 128, 64, and 16 neurons respectively in three layers. Since there are 4 optional phases in this method, the output action dimension is 4. It should be noted that due to the concise state setting, this method can use a simple neural network structure to achieve good learning effects.

[0142] Different from traditional non-reinforcement learning-based signal control methods, the learning process of this method involves frequent interactions between the agent and the traffic environment. This method defines an episode as the traffic simulation for a specific period (e.g., one hour). To obtain the optimal solution for signal control, the learning process usually consists of hundreds of episodes. In each episode, the agent continuously interacts with the environment, and the interaction process can be divided into five parts:

[0143] (i) Extract state information from the environment;

[0144] (ii) Select the optimal phase action;

[0145] (iii) Calculate the duration of the selected action based on the current state;

[0146] (iv) Conduct traffic simulation;

[0147] (v) Update the network.

[0148] The algorithm is as follows:

[0149] S401. Set parameters, including the number of training times R, simulation step size S, discount factor γ, learning rate α, exploration rate ε, and sampling batch size (Batch Size);

[0150] S402. Initialize the parameters θ of the main network and the parameters θ - of the target network, and initialize the experience replay pool M;

[0151] S403. Start traffic simulation to obtain the intersection environment s t ;

[0152] S404. Randomly select an action a with probability ε t , otherwise select an action a according to the current state s t ; [[ID= forty-two]] t , ;

[0153] S405. After executing the action a t , calculate the duration T according to the current intersection information duration ;

[0154] S406. Obtain a new round of intersection environment s t+1 and reward r t+1 , and store the experience samples {s t , a t , r t+1 , s t+1} in the experience replay pool M. If the experience pool is full, replace the earliest sample record with the new sample;

[0155] S407. Randomly draw a small batch of samples with a batch size of E from the experience replay pool M to update the main network Q and the target network Q'. If the number of training times has not reached the set value, go to step S404, otherwise go to step S408;

[0156] S408. End.

[0157] In another embodiment of the present invention, a multi-signal light control system for lane capacity differences is provided. This system can be used to implement the above multi-signal light control method for lane capacity differences. Specifically, the multi-signal light control system for lane capacity differences includes a construction module, a design module, a factor module, and a control module.

[0158] Among them, the construction module models the key information of the intersection based on the improved maximum pressure method to obtain an intersection model, and defines the pressure models of lanes, traffic flows, phases, and intersections;

[0159] The design module designs the pressure action and phase duration of adjacent intersections according to the intersection model obtained by the construction module;

[0160] The factor module designs the agent state, agent action, and agent reward based on the pressure model and intersection environment model defined by the construction module, and the adjacency of adjacent intersections in the design module;

[0161] The control module constructs a reinforcement learning framework based on the agent state, agent action, and agent reward defined by the factor module, uses the Nature Deep QNetwork reinforcement learning algorithm with a double Q network structure for each intersection agent, and dynamically controls the signal phase and phase duration according to the real-time environment of the intersection.

[0162] In another embodiment of the present invention, a terminal device is provided. The terminal device includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the multi-traffic-light control method for lane capacity differences, including:

[0163] Modeling the key information of the intersection based on the improved maximum pressure method to obtain an intersection model, and defining the pressure models of lanes, traffic flows, phases, and intersections; designing the pressure action and phase duration of adjacent intersections according to the intersection model; designing the agent state, agent action, and agent reward based on the pressure model, intersection environment model, and adjacent intersection actions; constructing a reinforcement learning framework based on the agent state, agent action, and agent reward defined in step S3, using the Nature Deep Q Network reinforcement learning algorithm with a double Q network structure for the agents of each intersection, and dynamically controlling the signal phase and phase duration according to the real-time environment of the intersection.

[0164] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. And, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0165] One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the multi-traffic-light control method for lane capacity differences in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows:

[0166] Based on the improved maximum pressure method, the key information of the intersection is modeled to obtain an intersection model, and the pressure models of lanes, traffic flows, phases, and intersections are defined; according to the intersection model, the pressure action and phase duration of adjacent intersections are designed; based on the pressure model, the intersection environment model, and the action of adjacent intersections, the agent state, agent action, and agent reward are designed; based on the agent state, agent action, and agent reward defined in step S3, a reinforcement learning framework is constructed, and the Nature Deep Q Network reinforcement learning algorithm with a double Q network structure is used for the agent at each intersection. According to the real-time environment of the intersection, the signal phase and phase duration are dynamically controlled.

[0167] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0168] Please refer to Figure 6 , in order to verify the effect of the adjacent intersection algorithm and highlight the characteristics of the present invention that can be timely sensed and optimized when the road occupancy rate is high, a multi-traffic-light road network simulation environment as shown in Figure 6 is built on the traffic simulation platform Cityflow. The east side of the road network is set as a residential area, and the west side is set as an industrial area. The morning rush hour commuting situation is simulated for two hours. The arrival rate of the inbound lanes at the east boundary is set to 700 vehicles per lane per hour, and a 10% turning ratio is set. The arrival rate of the inbound lanes at other boundaries is 300 vehicles per lane per hour.

[0169] Please refer to Figure 7 , Figure 7 (a) shows the evolution process of the number of vehicles in the road network over time under two signal control algorithms. Under the control of this method, the number of vehicles in the road network increases steadily. However, other signal control algorithms based on deep reinforcement learning cannot sense the pressure on the high-occupancy lanes, resulting in the "deadlock" phenomenon at some heterogeneous intersections; at about 5000 seconds, due to the gradual spread of the "deadlock" at local intersections to the entire road network, the number of vehicles in the road network surges, leading to the paralysis of the entire road network. At this time, the vehicles in the road network can hardly enter or exit. Figure 7 (b) The average passing time of vehicles in the road network shown further corroborates the superiority of this method. Under the drive of this method, the final average passing time of vehicles is about 350 seconds, while that of other algorithms is about 612 seconds.

[0170] Therefore, the signal control method of the present invention can effectively optimize the passing efficiency of vehicles in large-scale heterogeneous intersections.

[0171] In summary, a multi-traffic-light control method and system for lane capacity differences according to the present invention first provides a maximum pressure control method based on dual sensing of lane capacity and average vehicle speed for the differences in road capacities of heterogeneous intersections, realizing more refined sensing of key information at intersections. Secondly, considering the coordination between adjacent intersections, by introducing the interaction of pressures between adjacent intersections, the problem of the spread of blocked traffic flows at heterogeneous intersections to adjacent intersections is effectively alleviated. In order to further optimize the average passing time, a dynamically adjustable phase duration selection strategy is designed according to real-time traffic information in this paper. Furthermore, the three elements in the reinforcement learning method: state, action, and reward are designed. Finally, the agent established for each intersection uses the Nature Deep Q Network reinforcement learning algorithm with a double network structure to dynamically control the signal phase and phase duration according to the real-time environmental information at the intersection.

[0172] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0173] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0175] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the device or unit can be in electrical, mechanical or other forms.

[0176] The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0178] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the above-described embodiment methods of the present invention may also be completed by instructing relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments may be implemented. Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0179] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the specified function in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the process in Figure 1One process or multiple processes and / or boxes Figure 1 The functions specified in one box or multiple boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 One process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one box or multiple boxes.

[0182] The above content is only to illustrate the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the claims of the present invention.

Claims

1. A multi-signal light control method for lane capacity differences, characterized in that: The following steps are involved: S1. Based on the improved maximum pressure method, key intersection information is modeled to obtain an intersection model. The pressure models for lanes, traffic flows, phases, and intersections are defined. The pressure model specifically includes: Lane Pressure : in, For lanes The number of vehicles on For lanes The maximum capacity, and To adjust the parameters of the pressure function shape, is the average speed of the current lane, is the maximum allowed speed of the lane, is the weighting factor used to adjust the effect of velocity on pressure; Sports traffic pressure and stationary traffic pressure for: in, and are the total number of entry lanes and exit lanes in the direction of traffic flow, and are the total number of entry lanes and exit lanes where the traffic flow stops; Phase pressure for: in, 、 are the weight coefficients of moving traffic pressure and stationary traffic pressure respectively; intersection pressure for: ； S2. Design the pressure effects and phase durations of adjacent intersections based on the intersection model obtained in step S1. The pressure effects of adjacent intersections are: in, is the contraction coefficient that limits the pressure value; is the maximum capacity of the lane; For intersections and the capacity of the lanes between them; for Entering lane, vehicles in this lane can cross the intersection under the current phase control and to Drive in; The number of lanes that can pass through per unit time the number of vehicles; For intersections The remaining time of the green light in the current phase; for No. Number of vehicles in the entry lane; 、 、 、 For the current intersection relevant information; Phase duration Specifically: According to the given and Find a set of M optional phase durations, and They represent the minimum phase duration and maximum phase duration of the intersection to ensure stable vehicle traffic; When M=1, the phase duration is fixed to ; When M>1, construct the phase duration set D; After selecting the signal light phase, pass into the lane Number of vehicles on And the number of vehicles that can pass through the intersection per unit time in this lane , get into the lane The time it takes for all vehicles to pass through the intersection ; Determine the phase duration ; S3. Based on the pressure model and intersection model defined in step S1, and the adjacent intersection effects in step S2, design the agent state, agent action, and agent reward; S4. Build a reinforcement learning framework based on the agent states, actions, and rewards defined in step S3. Use the Nature Deep Q Network reinforcement learning algorithm with a dual-Q network structure for the agent at each intersection to dynamically control the signal phase and phase duration according to the real-time environment of the intersection.

2. The multi-signal light control method for lane capacity differences according to claim 1, characterized in that: In step S1, the intersection model is a two-way three-lane model, where the three lanes can turn left, go straight, and turn right, and right-turning vehicles are not controlled by traffic signals; each intersection consists of a set of entry lanes and a group of people leaving the lane There are 12 lanes in total. , ,…, There are 12 exit lanes, and the four phases are north-south straight, east-west straight, north-south left turn and east-west left turn.

3. The multi-signal light control method for lane capacity differences according to claim 1, characterized in that: In step S2, duration for: in, As an entry lane All lanes Time required for vehicles to pass The maximum value of .

4. The multi-signal light control method for lane capacity differences according to claim 1, characterized in that: In step S3, the state of the agent is designed based on numerical representation. The state of the agent is composed of the environmental information of the intersection and its surrounding neighbors, including the pressure of the phase, the pressure influence of the neighbors, and the current phase; the current intersection The four adjacent intersections are 、 、 、 , the state vector is encoded as ; The optimal control phase of the intersection is selected as the agent action. Each agent has four different predefined allowed actions. The action set is ; Considering the lane capacity and average speed, the reward function for: in, For intersections I pressure.

5. The multi-signal light control method for lane capacity differences according to claim 1, characterized in that: In step S4, the Nature Deep Q Network reinforcement learning algorithm is specifically as follows: S401, set parameters, including the number of training times , simulation step , discount factor , learning rate , exploration rate , sampling batch size; S402: Initialize main network parameters and target network parameters , initialize the experience replay pool ; S403, traffic simulation starts, obtain intersection environment ; S404, by probability Randomly select actions Otherwise, according to the current state Select Action ; S405, execute action Then, calculate the duration based on the current intersection information ; S406, get a new round of intersection environment and rewards , in the experience replay pool Storage experience samples ,If the experience pool overflows, the earliest sample record is replaced by the new sample; S407, from the experience replay pool The batch size of random sampling is Mini-batch samples to update the main network and target network ; If the number of training times does not reach the set value, go to step S404, otherwise go to step S408; S408, end.

6. The multi-signal light control method for lane capacity differences according to claim 5, characterized in that: In step S404, the action Specifically: 。 7. A multi-signal light control system for lane capacity differences, characterized by: include: The construction module models the key intersection information based on the improved maximum pressure method to obtain the intersection model, and defines the pressure model of lanes, traffic flows, phases, and intersections. The pressure model specifically includes: Lane Pressure : in, For lanes The number of vehicles on For lanes The maximum capacity, and To adjust the parameters of the pressure function shape, is the average speed of the current lane, is the maximum allowed speed of the lane, is the weighting factor used to adjust the effect of velocity on pressure; Sports traffic pressure and stationary traffic pressure for: in, and are the total number of entry lanes and exit lanes in the direction of traffic flow, and are the total number of entry lanes and exit lanes where the traffic flow stops; Phase pressure for: in, 、 are the weight coefficients of moving traffic pressure and stationary traffic pressure respectively; intersection pressure for: ； The design module designs the pressure effects and phase duration of adjacent intersections based on the intersection model obtained by the construction module. The pressure effects of adjacent intersections are: in, is the contraction coefficient that limits the pressure value; is the maximum capacity of the lane; For intersections and the capacity of the lanes between them; for Entering lane, vehicles in this lane can cross the intersection under the current phase control and to Drive in; The number of lanes that can pass through per unit time the number of vehicles; For intersections The remaining time of the green light in the current phase; for No. Number of vehicles in the entry lane; 、 、 、 For the current intersection relevant information; Phase duration Specifically: According to the given and Find a set of M optional phase durations, and They represent the minimum phase duration and maximum phase duration of the intersection to ensure stable vehicle traffic; When M=1, the phase duration is fixed to ; When M>1, construct the phase duration set D; After selecting the signal light phase, pass into the lane Number of vehicles on And the number of vehicles that can pass through the intersection per unit time in this lane , get into the lane The time it takes for all vehicles to pass through the intersection ; Determine the phase duration ; The element module designs the agent state, agent action, and agent reward based on the pressure model and intersection model defined in the building module and the adjacent intersection effects in the design module; The control module builds a reinforcement learning framework based on the agent states, actions, and rewards defined in the element module. It uses the Nature Deep Q Network reinforcement learning algorithm with a dual-Q network structure for the agents at each intersection, dynamically controlling the signal phase and phase duration based on the real-time environment of the intersection.

8. The multi-signal light control system for lane capacity differences according to claim 7, characterized in that: The state of the intelligent agent is designed based on numerical representation. The state of the intelligent agent is composed of the environmental information of the intersection and its surrounding neighbors, including the pressure of the phase, the pressure influence of the neighbors, the current phase; the current intersection The four adjacent intersections are 、 、 、 , the state vector is encoded as ; The optimal control phase of the intersection is selected as the agent action. Each agent has four different predefined allowed actions. The action set is ; Considering the lane capacity and average speed, the reward function for: in, For intersections I pressure.

9. The multi-signal light control system for lane capacity differences according to claim 7, characterized in that: The NatureDeep Q Network reinforcement learning algorithm is specifically: Set parameters, including the number of training times , simulation step , discount factor , learning rate , exploration rate , sampling batch size; initialize the main network parameters and target network parameters , initialize the experience replay pool ; Traffic simulation starts, obtain intersection environment ; By probability Randomly select actions Otherwise, according to the current state Select Action , ; Execute action Then, calculate the duration based on the current intersection information ; Get a new round of intersection environment and rewards , in the experience replay pool Storage experience samples If the experience pool is full, the earliest sample record is replaced by the new sample; The batch size of random sampling is Mini-batch samples to update the main network and target network If the number of training times does not reach the set value, it will switch to the probability Randomly select actions , otherwise end.

Images