A traffic signal lamp sensing control method and system based on deep reinforcement learning

Abstract

The embodiment of the application provides a kind of traffic signal lamp induction control method and system based on deep reinforcement learning, belong to traffic internet of things control technical field.The traffic signal lamp induction control method includes: the traffic signal lamp control problem of intersection is modeled as Markov decision process, and observation space, action space and reward function are defined;Discrete time trajectory data of the traffic signal lamp is acquired;A kind of traffic signal lamp induction control method and system based on deep reinforcement learning provided by the application can effectively prevent the shock caused by policy mutation in training process based on the transformer network model of proximal policy optimization, reduce the dependence of prior knowledge, while being able to dynamically focus on key lane information, can be transferred to different intersection topologies, and thus effectively improve the effect and versatility of traffic signal lamp control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic Internet of Things control, in particular to a traffic signal lamp induction control method and system based on deep reinforcement learning. BACKGROUND

[0002] Traffic signal lamps are important facilities for road traffic management, which indicate vehicles and pedestrians to pass or stop through light colors and combinations to ensure traffic safety and order.

[0003] Urban traffic congestion has been a global problem for a long time, which has a negative impact on the economy and the environment. The energy used during traffic congestion will cause the emission of greenhouse gases such as carbon dioxide, thereby exacerbating the greenhouse effect. It is urgent to improve the environment through technological innovation to reduce pollution emissions caused by traffic congestion.

[0004] At present, the control of urban traffic signal lamps mainly relies on three ways: one is fixed time sequence control, which is to preset the signal lamp cycle, phase length and phase sequence according to historical traffic flow statistical data, for example, to set longer green light time for main roads during morning and evening peak hours; but this way cannot respond to sudden changes in traffic flow in real time, such as accidents and holiday traffic surges, and needs to collect data regularly to adjust the time sequence, which has high maintenance cost; in addition, the same scheme is difficult to adapt to different intersection topologies, such as T-shaped intersection and cross-shaped intersection. Two is induction control, which detects the existence of vehicles on the lane in real time through sensors such as magnetic coils and cameras, dynamically extends the current green light phase or switches the phase, for example, skips the light of a certain direction when no vehicle is detected; but this way only focuses on the current phase vehicles, lacks global traffic flow coordination, and the sensor needs to accumulate a certain time before acting after detecting a vehicle, which is easy to form a queue during peak hours, and sensor failure is easy to cause control failure. Three is rule-based adaptive control, such as SCOOT / SCATS, which adjusts the signal cycle through a pre-set rule library, for example, the SCATS system dynamically allocates green light time according to the saturation degree; but this way relies on limited rules summarized by experts, which is difficult to cover complex scenarios such as mixed multi-modal traffic flow, and traditional algorithms also cannot handle high-dimensional data such as vehicle trajectories in the whole road network, the optimization dimension is low, and rule updating needs human intervention.

[0005] The present application inventors found in the process of implementing the present application that the above-mentioned prior art scheme has the defects of poor traffic signal lamp control effect and poor universality. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a traffic signal lamp induction control method and system based on deep reinforcement learning, which has the functions of good traffic signal lamp control effect and good universality.

[0007] To achieve the above objectives, one embodiment of the present invention provides a traffic signal light sensing control method based on deep reinforcement learning, comprising:

[0008] The traffic light control problem at an intersection is modeled as a Markov decision process, and the observation space, action space, and reward function are defined, wherein the reward function includes:

[0009] Obtain the reward function according to formula (1).

[0010] (1)

[0011] in, For the reward function, For the first Phase efficiency factor For the first One basic reward item, The number is an integer, and , For green wave weights, For the green wave coordination reward item, As an emergency weight, Priority is given to emergency vehicles. For emission weights, As a carbon emission penalty item;

[0012] Obtain the discrete time-series trajectory data of the traffic lights;

[0013] A transformer network model based on near-end policy optimization is constructed and trained based on the discrete time-series trajectory data.

[0014] The optimal action strategy for the current traffic light is obtained and executed based on the transformer network model with near-end policy optimization.

[0015] Optionally, the observation space, action space, and reward function can be defined as follows:

[0016] The observation space is defined according to formula (2):

[0017] ,

[0018] (2)

[0019] in, The observation space, The total number of lanes at the intersection. For lane Observational data, For lane The number of vehicles, For lane Average waiting time for low-speed vehicles. For lane Queue length, For lane The average speed of vehicles, For emergency vehicles Lane waiting time, The phase difference between adjacent intersections For lane The number of times a vehicle starts and stops. Numbered by integer;

[0020] The action space is defined according to formula (3):

[0021] (3)

[0022] in, For the action space, For the first Discrete traffic signal phases This represents the total number of traffic signal phases.

[0023] Optionally, defining the observation space, action space, and reward function also includes:

[0024] The phase efficiency factor is obtained according to formula (4).

[0025] (4)

[0026] in, For the first A basic reward weight, For lane saturation It is the saturation threshold, and , For traffic capacity;

[0027] The green wave coordination reward item is obtained according to formula (5).

[0028] (5)

[0029] in, To coordinate the average speed of vehicles in the lanes, To coordinate the number of lanes;

[0030] The priority of emergency vehicles is obtained according to formula (6).

[0031] (6)

[0032] in, a lane set containing an emergency vehicle, an emergency vehicle waiting time threshold;

[0033] obtaining a carbon emission penalty term according to formula (7),

[0034] (7)

[0035] wherein, is an emission coefficient, and , is a maximum allowed speed of a lane corresponding to the road segment.

[0036] Optionally, the Markov decision process comprises:

[0037] obtaining observation data of the current intersection;

[0038] obtaining an action policy of the current traffic light and executing the action policy;

[0039] obtaining a reward function of the current traffic light;

[0040] generating next observation data according to a transition probability function;

[0041] returning to the step of obtaining an action policy of the current traffic light and executing the action policy.

[0042] Optionally, obtaining discrete timing trajectory data of the traffic light comprises:

[0043] presetting a sequence number of a cycle;

[0044] determining whether a sequence number of the current traffic light is greater than or equal to the sequence number;

[0045] in a case where it is determined that the sequence number of the current traffic light is greater than or equal to the sequence number, summarizing sequences of the sequence number to form discrete timing trajectory data;

[0046] in a case where it is determined that the sequence number of the current traffic light is less than the sequence number, obtaining a sequence of the traffic light at a next time;

[0047] returning to the step of determining whether the sequence number of the current traffic light is greater than or equal to the sequence number.

[0048] Optionally, constructing and training a transformer network model based on proximal policy optimization according to the discrete timing trajectory data comprises:

[0049] The discrete time sequence trajectory data is vectorized and input into a transformer network model based on proximal policy optimization;

[0050] The query vector, the key vector and the value vector are obtained;

[0051] The attention function is obtained according to formula (8),

[0052] , (8)

[0053] wherein, is the attention function, is a softmax function, is the query vector, is the key vector, is the value vector, is the dimension of the vector;

[0054] The encoded output is obtained and input into a multilayer perceptron to obtain an encoded observation;

[0055] The encoded observation is input into a next multilayer perceptron to obtain a value estimate.

[0056] Optionally, constructing and training the transformer network model based on proximal policy optimization according to the discrete time sequence trajectory data further comprises:

[0057] The encoded observation is input into a decoder;

[0058] A phase switching penalty term and a team balance term are obtained;

[0059] The target function of the decoder is obtained according to formula (9),

[0060] , (9)

[0061] wherein, is the target function of the decoder, is a network parameter, is a policy ratio, is a discrete traffic signal phase at a time step, is a time step, is a number of training cycles, is a clipping mechanism, is a constant, is a phase weight, is a balance weight, is a phase switching penalty term, is a team balance term;

[0062] According to the objective function of the decoder, a next action policy of the traffic signal is obtained.

[0063] Optionally, obtaining the phase switching penalty term and the team balance term comprises:

[0064] According to formula (10), the phase switching penalty term is obtained,

[0065] , (10)

[0066] wherein, is an indicator function, 1 when phase switching, otherwise 0, is the action at time step t, is a decay coefficient, is the current phase duration, is the minimum green duration threshold;

[0067] According to formula (11), the team balance term is obtained,

[0068] , (11)

[0069] wherein, is a variance function, is the queue length of the lane ;

[0070] According to formula (12), the phase weight and the balance weight are obtained,

[0071] ,

[0072] , (12)

[0073] wherein, is a basic weight, is the vehicle density passing through the intersection at time step , and is the maximum traffic capacity of the road section.

[0074] In another aspect, the present application also provides a traffic signal induction control system based on deep reinforcement learning, comprising:

[0075] a traffic signal;

[0076] a controller connected with the traffic signal, for executing the traffic signal induction control method as any of the above.

[0077] In still another aspect, the present application also provides a computer readable storage medium storing instructions for being read by a machine to cause the machine to perform the traffic signal lamp sensing control method according to any one of the above.

[0078] By the above technical solution, the traffic signal lamp sensing control method and system based on deep reinforcement learning provided by the present application model the control problem of the traffic signal lamp at the intersection as a Markov decision process, define the observation space, the action space and the reward function, and pay more attention to the global state. According to the Markov decision process, the vehicle state at the intersection is collected in real time, the action is executed, the corresponding reward is obtained, and the discrete time trajectory data is acquired. Then, the discrete time trajectory data is used to train and optimize the transformer network model based on the proximal policy optimization, so as to realize the optimal action prediction for the next cycle of the traffic signal lamp. The transformer network model based on the proximal policy optimization can effectively prevent the shock caused by the policy mutation in the training process, reduce the dependence on prior knowledge, dynamically focus on the key lane information, and be migrated to different intersection topologies, thereby effectively improving the effect and universality of the traffic signal lamp control. In addition, the reward function fully considers the multi-dimensional state features of the phase saturation, the green wave coordination reward, the emergency vehicle priority and the carbon emission penalty, and can further improve the accuracy and effect of the subsequent optimal policy acquisition.

[0079] Other features and advantages of the embodiments of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0080] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the drawings:

[0081] Figure 1 is a flowchart of the traffic signal lamp sensing control method based on deep reinforcement learning according to an embodiment of the present application;

[0082] Figure 2 is a flowchart of the definition of the observation space, the action space and the reward function in the traffic signal lamp sensing control method based on deep reinforcement learning according to an embodiment of the present application;

[0083] Figure 3 is a flowchart of the real-time observation of the Markov decision process in the traffic signal lamp sensing control method based on deep reinforcement learning according to an embodiment of the present application;

[0084] Figure 4is a flowchart of acquiring discrete time trajectory data in a traffic signal induction control method based on deep reinforcement learning according to an embodiment of the present application;

[0085] Figure 5 is a flowchart of training transformer network model encoding based on proximal policy optimization in a traffic signal induction control method based on deep reinforcement learning according to an embodiment of the present application;

[0086] Figure 6 is a flowchart of training transformer network model decoding based on proximal policy optimization in a traffic signal induction control method based on deep reinforcement learning according to an embodiment of the present application;

[0087] Figure 7 is a flowchart of acquiring phase switching penalty term and queue balancing term in a traffic signal induction control method based on deep reinforcement learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0088] The specific embodiments of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application.

[0089] It should be noted that the acquisition, transmission, storage, use, processing and the like of data in the technical solutions of the present application comply with the relevant provisions of laws and regulations. In the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models and the like, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solutions.

[0090] Figure 1 is a flowchart of a traffic signal induction control method based on deep reinforcement learning according to an embodiment of the present application. In Figure 1 , the traffic signal induction control method can include:

[0091] In step S1, the traffic signal control problem of the intersection is modeled as a Markov decision process, and the observation space, action space and reward function are defined. Among them, for the vehicle state of the intersection / central intersection and the action change of the traffic signal, it can be modeled as a Markov decision process to clearly describe the dynamic change of the intersection and optimize the decision of the traffic signal.

[0092] In step S2, discrete timing trajectory data of the traffic signal lamp is acquired. In the process, the Markov decision process repeatedly observes and performs actions and acquires rewards at each moment. In this process, a preset number of observations and actions can be preset, and after a preset number of observations and actions are recorded, they can be used as a period of sequence data, that is, discrete timing trajectory data. Specifically, the discrete timing data can be used as data for subsequent network model training.

[0093] In step S3, a transformer network model based on proximal policy optimization is constructed and trained according to the discrete timing trajectory data. After the discrete timing trajectory data is acquired, a transformer network model based on PPO proximal policy optimization can be constructed, and the discrete timing trajectory data can be used to train the transformer network model. Specifically, after the Markov model continuously acquires a period of sequence data, it can be continuously input into the transformer network model for continuous training and optimization.

[0094] In step S4, the optimal action strategy of the current traffic signal lamp is acquired according to the transformer network model based on proximal policy optimization and is executed. After the transformer network model is trained and optimized using a period of discrete timing trajectory data, the next action of the traffic signal lamp can be predicted according to the state of the current intersection and the action data of the traffic signal lamp to acquire the optimal action strategy of the traffic signal lamp and execute it.

[0095] In steps S1 to S4, the control problem of the traffic signal lamp at the intersection is modeled as a Markov decision process, and the observation space / observation space, action space, and reward function of the Markov decision process are defined. The Markov decision process collects the state of the vehicle at the intersection in real time and performs actions to acquire the corresponding rewards. The above data is counted, and after a preset number of data is counted, it can be used as a period of sequence data, that is, discrete timing trajectory data. At the same time, a transformer network model based on proximal policy optimization is constructed, and the discrete timing trajectory data is used to train and optimize the transformer network model based on proximal policy optimization. According to the current state and action data, the next period of action prediction, that is, the optimal action strategy of the traffic signal lamp, can be obtained, and the optimal action strategy can be executed.

[0096] Traditional traffic lights generally adopt three ways to control, specifically, the first way is fixed time sequence control, the principle is to preset the signal light period, phase length and phase sequence according to historical traffic flow statistical data. For example, set longer green light time for the main road in the morning rush hour. However, the limitations of the first way include: unable to respond to sudden traffic changes in real time (such as accidents, holiday traffic surge); rely on manual parameter adjustment, need to periodically re-collect data to adjust the time sequence, high maintenance cost; poor scene generalization, the same scheme is difficult to adapt to different intersection topologies (such as T-shaped intersection and cross intersection). The second way is inductive control, the principle is to detect the existence of vehicles on the lane in real time through sensors such as magnetic coils and cameras, and dynamically extend the current green light phase or switch the phase. For example, skip the green light when no vehicle is detected in a certain direction. However, the limitations of the second way include: local optimization, only focus on the current phase vehicles, lack of global traffic flow coordination; response delay, the sensor needs to accumulate a certain time before triggering action after detecting the vehicle, which is prone to form a queue during peak hours; high dependence on equipment, sensor failure will lead to control failure. The third way is rule-based adaptive control, such as SCOOT / SCATS, the principle is to adjust the signal period through a pre-set rule library, for example, the SCATS system dynamically allocates green light time according to the saturation. However, the limitations of the third way include: rigid rules, rely on limited rules summarized by experts, difficult to cover complex scenarios (such as mixed multi-modal traffic flow); limited computing power, traditional algorithms cannot handle high-dimensional data (such as vehicle trajectory in the whole road network), low optimization dimension; slow iteration speed, rule update requires human intervention, cannot learn online. In this embodiment of the present application, the first advantage is the dynamic environmental adaptation capability, the self-attention mechanism of the transformer is adopted, the multi-lane time sequence data (such as the number of vehicles, speed, queue length) is processed in parallel, and the space-time dependence between lanes is captured. Compared with traditional methods: fixed time sequence and rule library cannot model such long-range dependencies. Markov decision process (MDP) formalizes traffic state, action (phase switching), and reward (traffic efficiency) as a sequence decision problem, supporting the model to learn the optimal strategy through historical trajectory. Compared with inductive control: MDP models global state rather than local vehicle existence, avoiding short-term decision making. The second advantage is efficient strategy optimization and training stability, the near-end strategy optimization algorithm balances exploration and utilization through clipping mechanism, preventing strategy mutation from causing shocks during the training process. Compared with traditional Q-learning: PPO supports multiple rounds of small batch data reuse, improving sample efficiency. Directly use the original observation data (number of vehicles, speed, etc.) as input, without the need for manual feature design (such as the saturation calculation formula in traditional methods), reducing the dependence on prior knowledge.The third advantage is the generalization ability of complex scenes. The transformer model dynamically focuses on key lane information through attention weights, can be migrated to different intersection topologies, and the weighted reward function (vehicle number, speed, queue, waiting time) supports flexible adjustment and optimization direction. For example, in the environmental protection priority scene, the average speed weight is increased to reduce idling emission. In general, the traffic signal induction control method adopted in the application has better control effect and stronger universality than the traditional control method.

[0097] In this embodiment of the application, after modeling the control problem of the traffic signal of the intersection as a Markov decision process, the observation space, action space and reward function of the Markov decision process need to be defined. The specific definition steps can be as shown in Figure 2 . Specifically, in Figure 2 , the traffic signal induction control method can further include:

[0098] In step S10, the observation space is defined according to formula (2):

[0099] ,

[0100] , (2)

[0101] wherein, is the observation space, is a set, is the total number of lanes of the intersection, is the observation data of the lane , is the number of vehicles in the lane , is the average waiting time of low-speed vehicles (speed < 0.1 m / s) in the lane , is the queue length (low-speed vehicles) in the lane , is the average speed of vehicles in the lane ; is the waiting time of emergency vehicles in the lane , and the identification of emergency vehicles such as ambulances can also be defined here, and the lane where they are located is located; is the phase difference of the adjacent intersection, is the number of starts and stops of all vehicles in the lane , is an integer number.

[0102] In step S11, the action space is defined according to formula (3):

[0103] , (3)

[0104] wherein, is the action space, is the th discrete traffic signal phase, is the total number of traffic signal phases. Specifically, the action space is a set of discrete traffic signal phases, which can include red, yellow and green phases or phase switching actions.

[0105] In step S12, the phase efficiency factor, the filtered coordination reward term, the emergency vehicle priority term and the carbon emission penalty term are obtained. Wherein, the phase efficiency factor can be calculated according to formula (4),

[0106] , (4)

[0107] wherein, is the th basic reward weight, is the saturation of the th lane, is the saturation threshold, , is the capacity. Specifically, when , the congestion penalty is strengthened. Specifically, the capacity refers to the maximum number of vehicles that can pass a certain point (such as a certain phase or lane of an intersection) per unit time (usually 1 hour) under certain road, traffic and control conditions. It is a core indicator for measuring the efficiency of the traffic system, and directly determines whether the signal control strategy can effectively alleviate congestion. the value of is 1-4, which corresponds to the dimensional weight of the number of vehicles, average speed, average queue length and average waiting time of the vehicles at the intersection, respectively, for balancing the differences in index dimensions and avoiding the dominance of a single index in the optimization process.

[0108] In a signal intersection, the capacity of a certain phase can be obtained according to formula (13),

[0109] , (13)

[0110] wherein, is the saturation flow rate, the maximum flow rate of vehicles continuously passing through the stop line during the green light, with a unit of vehicles / hour / lane, and a typical value: about 1800-2000 vehicles / hour for straight lanes, and about 1600-1800 vehicles / hour for left-turn lanes. Specifically, the same phase can include multiple lanes, so Also can refer to a single or multiple same-phase lanes passing capacity. is the effective green time, the actual available green time of the phase / lane, in seconds, which needs to deduct the yellow light, all-red time and vehicle start-up loss time. is the signal cycle length, the total time for the signal light to complete all phase rotations, in seconds. By adopting the method of adjusting the basic reward weight according to the phase saturation and the dynamic adjustment, the congestion of the road can be effectively considered to improve the accuracy of the reward function.

[0111] The green wave coordination reward term can be calculated according to formula (5),

[0112]

[0113] , (5)

[0114] wherein, is the average speed of the coordinated lane, is the number of coordinated lanes, that is, the green wave coordination reward term is the sum of the average speed of each coordinated lane that needs to be coordinated by the green wave and the adjacent intersection phase difference ratio. Specifically, when the reward is maximum, is the distance between intersections. is the adjacent intersection phase difference, which refers to the time difference value of the starting time of the same phase (such as the straight green light of the main road) in the signal light cycle of two adjacent intersections; the core purpose is to realize the continuous traffic of the vehicle flow (such as the "green wave" effect) by coordinating the switching time sequence of multiple intersection signal lights, thereby reducing the number of stops and improving the traffic efficiency.

[0115] The acquisition of the emergency vehicle priority term can be calculated according to formula (6),

[0116] , (6)

[0117] wherein, is the lane set containing the emergency vehicle, is the emergency vehicle waiting time threshold, that is, the maximum tolerance time threshold of the emergency vehicle waiting for the green light at the intersection. When the waiting time of the emergency vehicle (such as an ambulance or a fire truck) exceeds , the system will forcibly adjust the signal phase to give priority to the vehicle to avoid delaying the rescue task. Specifically, the emergency vehicle priority term can detect special vehicles such as ambulances and fire trucks and dynamically increase the weight of the lane where the vehicle is located.

[0118] The acquisition of the carbon emission penalty term can be calculated according to formula (7),

[0119] ​ , (7)

[0120] wherein, is the emission coefficient, and (kg / trip), is the lane corresponding to the maximum allowed speed of the road segment.

[0121] In step S13, the reward function is obtained according to formula (1),

[0122] , (1)

[0123] wherein, is the reward function, is the phase efficiency factor of the th phase, is the th base reward term, is an integer number, and , is the green wave weight, is the green wave coordination reward term, is the emergency weight, is the emergency vehicle priority term, is the emission weight, is the carbon emission penalty term. Specifically, for the base reward term may include as shown in formula (14):

[0124] , (14)

[0125] wherein, is the number of vehicles, and , is the average speed, and , is the average queue length, and , is the average waiting time, and . Specifically, the above summation formula is the summation of the corresponding parameters of each lane at the intersection.

[0126] In steps S10 to S13, the observation space in the Markov decision process of the traffic signal at the intersection is defined. At the same time, the action space is defined as the set of phases of the discrete traffic signal, that is, the red light, yellow light and green light phases. Finally, the current phase efficiency factor, green wave coordination reward item, emergency vehicle priority item and carbon emission penalty item are calculated according to the observation data, and the final reward function is obtained by summarizing the above parameters. Specifically, the reward function fully considers the multi-dimensional state characteristics of the phase saturation, green wave coordination reward, emergency vehicle priority and carbon emission penalty, and can effectively improve the accuracy and effect of obtaining the subsequent optimal strategy.

[0127] In this embodiment of the application, after the observation space, action space and reward function in the Markov decision process are defined, the state of the intersection can be observed in real time and the action of the traffic signal can be performed to obtain the corresponding reward function. Specifically, the steps can be Figure 3 as shown in the figure. Specifically, in Figure 3 , the traffic signal induction control method can further include:

[0128] In step S14, the observation data of the current intersection is obtained. The observation data of the current intersection, that is, the parameter data defined in the above observation space, is generally represented as .

[0129] In step S15, the action strategy of the current traffic signal is obtained and the action strategy is executed. The action strategy of the current traffic signal can include an initial action strategy or an optimized action strategy, and the current action strategy is executed. The current action is generally represented as , and .

[0130] In step S16, the reward function of the current traffic signal is obtained. According to the observation data and the action strategy, the current immediate reward function can be obtained according to formula (3), and the reward function can quantify the quality of the current decision. Specifically, the current reward function is generally represented as or .

[0131] In step S17, the next observation data is generated according to the transition probability function. The transition probability function is directly sampled by the agent / traffic signal and the environment, and the agent / traffic signal collects experience data through trial and error to learn the strategy and value function. When the agent / traffic signal selects to execute the action , the next observation state is automatically calculated according to the environment state (speed, queuing, etc.), and therefore the transition probability is determined by the internal mechanism.

[0132] In step S18, return to the step of acquiring the action policy of the current traffic signal lamp and executing the action policy. Wherein, Markov constantly repeats observation and action execution and reward acquisition every moment, so after acquiring the next observation data, the next action policy is also executed, and the corresponding reward or reward function is acquired.

[0133] In steps S14 to S18, the observation data of the current intersection is first acquired, and the action policy of the current traffic signal lamp is acquired and executed, and the current reward function can be obtained according to the current observation data and the executed action. The next observation data is acquired according to the transition probability function, and the next action is executed, and the cycle is repeated to describe and solve the control problem of the traffic signal lamp.

[0134] In this embodiment of the application, in order to acquire the discrete time sequence trajectory data of the traffic signal lamp, the number of observation and action time points can be preset, and the corresponding data can be statistically summarized according to the number of time points. Specifically, the steps can be as shown in Figure 4 Specifically, in Figure 4 , the traffic signal induction control method can further include:

[0135] In step S20, a sequence number of a period is preset. Wherein, the sequence number of a period can include .

[0136] In step S21, it is judged whether the sequence number of the current traffic signal lamp is greater than or equal to the sequence number. Wherein, the sequence of the traffic signal lamp can include a three-tuple data stream , and the sequence number of the traffic signal lamp is the value of the time point .

[0137] In step S22, in the case where it is judged that the sequence number of the current traffic signal lamp is greater than or equal to the sequence number, the sequence number of the sequence is summarized to form the discrete time sequence trajectory data. Wherein, if the sequence number of the current traffic signal lamp is greater than or equal to the sequence number, it means that a sequence of a preset period has been collected, and the discrete time sequence trajectory data can be formed by summarizing and statistically analyzing the sequences .

[0138] In step S23, in the case where it is judged that the sequence number of the current traffic signal lamp is less than the sequence number, the sequence of the traffic signal lamp at the next time point is acquired. Wherein, if the sequence number of the current traffic signal lamp is less than the sequence number, it means that a sequence of a preset period has not been collected, and the next sequence needs to be acquired.

[0139] In step S24, return to the step of judging whether the sequence number of the current traffic signal lamp is greater than or equal to the sequence number.

[0140] In steps S20-S24, after the number of sequences in a period is preset, the sequence number of the sequence data of the traffic signal collected in real time in the Markov decision process is judged. If the sequence number is greater than or equal to the preset sequence number, it means that the sequence data collection in a period is completed, and the discrete time sequence trajectory data can be formed by summarizing, otherwise it needs to continue to collect.

[0141] In this embodiment of the application, after the discrete time sequence trajectory data in a period is obtained, the transformer network model based on the proximal policy optimization can be trained using the discrete time sequence trajectory data. Specifically, the training steps can be as shown in Figure 5 Specifically, in Figure 5 , the traffic signal sensing control method can further include:

[0142] In step S30, the discrete time sequence trajectory data is vectorized and input into the transformer network model based on the proximal policy optimization. The observation data in the discrete time sequence trajectory data is vectorized and used as the input of the network model.

[0143] In step S31, a query vector, a key vector and a value vector are obtained. The query vector , the key vector , and the value vector , wherein, is the input sequence, , , is a learnable weight matrix.

[0144] In step S32, the attention function is obtained according to formula (8),

[0145] , (8)

[0146] wherein, is the attention function, is the softmax function, is the query vector, is the key vector, is the value vector, is the dimension of the vector, including the dimension of the query and key vectors, used to scale the dot product to prevent gradient explosion, is the matrix transpose, and the matrix transpose is different from the period in the latter, and needs to be distinguished. Specifically, in the attention mechanism, shares the same set of parameters, and the context awareness ability of the sequence decision is enhanced by modeling global dependencies.​

[0147] In step S33, the encoded output is obtained and input into the multi-layer perception to obtain the encoded observation. The encoded output is input into the multi-layer perception MLP to obtain a new feature encoded observation.

[0148] In step S34, the encoded observation is input into the next multi-layer perception to obtain a value estimate. The encoded observation can be input into the next multi-layer perception after being obtained, and the value estimate can be used to optimize the encoding network parameters in the transformer network.

[0149] In steps S30 to S34, when the network model needs to be trained and encoded, the discrete time trajectory data needs to be vectorized, which can include vectorization of observation data and actions. The query vector, key vector, and value vector of the vectorized sequence are obtained, the attention function is calculated, and the output is encoded. The encoded output is input into the multi-layer perception to obtain the encoded observation, and the encoded observation is further input into the multi-layer perception to obtain the value estimate.

[0150] In this embodiment of the application, after the discrete time trajectory data is vectorized and encoded, a decoding operation can be performed. The decoding process can be as shown in Figure 6 Specifically, in Figure 6 , the traffic signal sensing control method can further include:

[0151] In step S35, the encoded observation is input into the decoder.

[0152] In step S36, the phase switching penalty term and the team balance term are obtained. The obtaining of the phase switching penalty term and the team balance term can include the steps as shown in Figure 7 Specifically, in Figure 7 , the obtaining step can include:

[0153] In step S360, the phase switching penalty term is obtained according to formula (10),

[0154] , (10)

[0155] wherein, is an indicator function, which is 1 when the phase is switched and 0 otherwise, is the action at time step t, is a decay coefficient to ensure that the short switching penalty is larger, is the current phase duration, is the minimum green light duration threshold. Specifically, ​This refers to the minimum green light duration set by the traffic signal control system to ensure the safe and orderly passage of vehicles through intersections. For example, if... If the green light duration is 10 seconds, the traffic light must maintain its current green state for at least 10 seconds before switching to the next phase. If the green light duration is too short, such as 5 seconds, vehicles may not have enough time to pass through the intersection, leading to increased queues, decreased traffic efficiency, and even safety hazards. Frequent phase switching increases the number of vehicle starts and stops, wastes fuel, and causes driver discomfort. Therefore, a system is used to count the number of phase switching between adjacent actions, and a penalty is applied if the current phase differs from the previous phase.

[0156] In step S361, the team balance term is obtained according to formula (11).

[0157] (11)

[0158] in, It is the variance function. For lane Queue length. Excessively long queues in a single lane can cause breaks in the "green wave" and reduce the overall efficiency of the road network. Therefore, the variance of the queue length of each lane is calculated to encourage a balanced distribution.

[0159] In step S362, the phase weight and the equalization weight are obtained according to formula (12).

[0160] ,

[0161] (12)

[0162] in, The base weight, for example, is 0.5 initially. For real-time traffic flow, that is, at the time step The density of vehicles passing through an intersection is usually measured in vehicles per minute or vehicle density (such as lane occupancy). This represents the maximum traffic capacity of the road segment. Specifically, during peak hours, priority should be given to alleviating congestion, while during off-peak hours, traffic speed can be optimized. Increase to emphasize queue balance.

[0163] In step S37, the target function of the decoder is obtained according to formula (9).

[0164] (9)

[0165] in, The target function of the decoder, For network parameters, For strategy ratio, for discrete traffic signal phase of a time step, is a number of training periods, is a number of training periods, is a clipping mechanism that limits the network parameters to prevent the policy update step from being too large and ensure training stability, is a very small constant as a limit range, is a phase weight, is a balance weight, is a phase switching penalty term, is a team balance term. Specifically, the policy ratio is the probability ratio of the new policy (current parameter ) and the old policy (parameter ) under the same observation state and action, which can be specifically shown as formula (15),

[0166] , (15)

[0167] wherein, is an encoded observation, is a new policy, is an old policy.

[0168] In step S38, the next action policy of the traffic signal is obtained according to the objective function of the decoder. The next action policy can include the action policy of the traffic signal in the next period.

[0169] In steps S35 to S38, in the process of decoding the network model, the phase switching penalty term and the team balance term are introduced on the basis of the clipping objective function of the proximal policy optimization. By penalizing frequent phase switching, invalid signal periods can be effectively reduced, and the driver experience can be improved. By the variance term, the model is forced to focus on the bottleneck lane, avoiding the spread of local congestion. In addition, the dynamic weights of the phase switching penalty term and the team balance term enable the algorithm to automatically switch the optimization goal under different traffic, balancing throughput and efficiency.

[0170] On the other hand, the present application also provides a traffic signal lamp induction control system based on deep reinforcement learning. Specifically, the traffic signal lamp induction control system can include a traffic signal lamp and a controller. The controller is connected with the traffic signal lamp, and is used to execute any one of the traffic signal lamp induction control methods as described above.

[0171] In another aspect, the present application also provides a computer readable storage medium, specifically, the computer readable storage medium stores instructions for being read by a machine to make the machine execute the traffic signal lamp induction control method as described above.

[0172] Through the above technical solution, the present invention provides a traffic light sensing control method and system based on deep reinforcement learning. This method models the traffic light control problem at intersections as a Markov decision process, defining the observation space, action space, and reward function, with a greater emphasis on the global state. Based on the Markov decision process, the system collects real-time data on vehicle states and actions at the intersection, obtains corresponding rewards, and acquires discrete time-series trajectory data. This discrete time-series trajectory data is then used to train and optimize a transformer network model based on proximal policy optimization, enabling the prediction of the optimal action for the next traffic light cycle. The transformer network model based on proximal policy optimization effectively prevents oscillations caused by policy mutations during training, reduces reliance on prior knowledge, dynamically focuses on key lane information, and can be transferred to different intersection topologies, thus effectively improving the performance and versatility of traffic light control. Furthermore, the reward function fully considers multi-dimensional state characteristics such as phase saturation, green wave coordination rewards, emergency vehicle priority, and carbon emission penalties, further improving the accuracy and effectiveness of subsequent optimal policy acquisition.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxesFigure 1 the function(s) specified in the block or blocks.

[0176] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the flowchart block(s) or flowchart flow(s) and / or portions thereof. Figure 1 Figure 1 the function(s) specified in the block or blocks.

[0177] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0178] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or a combination of non-volatile memories in different types. The memory is an example of computer readable media.

[0179] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0180] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0181] ​The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A traffic signal control method based on deep reinforcement learning, characterized in that, The application relates to a method for controlling traffic signal lights at an intersection. The traffic signal light control problem at the intersection is modeled as a Markov decision process, and an observation space, an action space and a reward function are defined, wherein the reward function comprises: The reward function is obtained according to formula (1), （1） wherein, is a reward function, is a first phase efficiency factor, is a first base reward term, is an integer number, and , is a green wave weight, is a green wave coordination reward term, is an emergency weight, is an emergency vehicle priority term, is an emission weight, is a carbon emission penalty term; Discrete time trajectory data of the traffic signal light is obtained; A transformer network model based on proximal policy optimization is constructed and trained according to the discrete time trajectory data; An optimal action policy of the current traffic signal light is obtained according to the transformer network model based on proximal policy optimization and is executed; The observation space, the action space and the reward function are defined as follows: The observation space is defined according to formula (2): （2） in, The observation space, The total number of lanes at the intersection. For lane Observational data, For lane The number of vehicles, For lane Average waiting time for low-speed vehicles. For lane Queue length, For lane The average speed of vehicles, For emergency vehicles Lane waiting time, The phase difference between adjacent intersections For lane The number of times a vehicle starts and stops. Numbered by integer; The action space is defined according to formula (3): （3） wherein, is the action space, is the first discrete traffic signal phase, is the total number of traffic signal phases; The transformer network model based on proximal policy optimization is constructed and trained according to the discrete time trajectory data, which comprises: The discrete time trajectory data is vectorized and input into the transformer network model based on proximal policy optimization; Query vectors, key vectors and value vectors are obtained; An attention function is obtained according to formula (8), （8） wherein, is an attention function, is a softmax function, is a query vector, is a key vector, is a value vector, is a dimension of the vectors; An encoded output is obtained and input into a multilayer perceptron to obtain an encoded observation; The encoded observation is input into a next multilayer perceptron to obtain a value estimate.

2. The traffic signal lamp sensing control method according to claim 1, wherein The observation space, the action space and the reward function further comprise: A phase efficiency factor is obtained according to formula (4), （4） in, For the first A basic reward weight, For lane saturation It is the saturation threshold, and , For traffic capacity; A green wave coordination reward item is obtained according to formula (5), （5） wherein, N is the number of coordinated lanes, N is the number of coordinated lanes; An emergency vehicle priority item is obtained according to formula (6), （6） wherein, is a set of lanes containing emergency vehicles, is an emergency vehicle wait time threshold; A carbon emission penalty item is obtained according to formula (7), （7） wherein, is the discharge coefficient, and , is the maximum allowed speed for the lane corresponding to the road segment.

3. The traffic signal lamp sensing control method according to claim 1, wherein The Markov decision process comprises: Observation data of the current intersection is obtained; An action policy of the current traffic signal light is obtained and executed; A reward function of the current traffic signal light is obtained; Next observation data is generated according to a transition probability function; The step of obtaining an action policy of the current traffic signal light and executing the action policy is returned to.

4. The traffic signal lamp sensing control method according to claim 3, wherein The discrete time trajectory data of the traffic signal light is obtained, which comprises: A sequence number of a period is preset; It is judged whether the sequence number of the current traffic signal light is greater than or equal to the sequence number; In the case that the sequence number of the current traffic signal light is greater than or equal to the sequence number, the sequence number of sequences is summarized to form discrete time trajectory data; In the case that the sequence number of the current traffic signal light is less than the sequence number, the sequence of the traffic signal light at the next time is obtained; The step of judging whether the sequence number of the current traffic signal light is greater than or equal to the sequence number is returned to.

5. The traffic signal lamp sensing control method according to claim 1, wherein The transformer network model based on proximal policy optimization is constructed and trained according to the discrete time trajectory data, which further comprises: The encoded observation is input into a decoder; A phase switching penalty item and a team balance item are obtained; A target function of the decoder is obtained according to formula (9), （9） wherein, is an objective function for the decoder, is a network parameter, is a policy proportion, is a discrete traffic signal phase at a time step, is a time step, is a number of training epochs, is a clipping mechanism, is a constant, is a phase weight, is a balancing weight, is a phase switching penalty term, is a team balancing term; A next action policy of the traffic signal light is obtained according to the target function of the decoder.

6. The traffic signal lamp sensing control method according to claim 5, wherein The phase switching penalty item and the team balance item are obtained, which comprises: The phase switching penalty item is obtained according to formula (10), （10） wherein, is an indicator function that is 1 at phase switches and 0 otherwise, is the action at the time step, is a decay coefficient, is the current phase duration, is a minimum green duration threshold; The team balance item is obtained according to formula (11), （11） wherein, is a variance function, is a queue length of a lane of the vehicle. The phase weight and the equalization weight are obtained according to formula (12), （12） wherein, is a base weight, is a time step is a density of vehicles passing through the intersection at time t, is a maximum capacity of the road segment.

7. A traffic signal induction control system based on deep reinforcement learning, characterized in that, The application relates to a traffic signal lamp sensing control method and device. The traffic signal lamp; The controller is connected with the traffic signal lamp and is used for executing the traffic signal lamp sensing control method as shown in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions for being read by a machine to make the machine execute the traffic signal lamp sensing control method as shown in any one of claims 1-6.

Images