Reinforcement learning traffic signal control method based on safety consciousness aggregation

Abstract

The invention discloses a reinforcement learning traffic signal control method based on safety consciousness aggregation. The method comprises the following steps: constructing a model; the method for training the model comprises the following steps of: 1, acquiring historical state information; 2, updating a first module, wherein the first module comprises a state encoder, an action-state encoder, an overflow encoder and a waiting encoder; 3, updating a second module, wherein the second module comprises an actor network, a reward criminator network, an overflow cost criminator network and a waiting time criminator network; 4, repeating the step 1 to the step 3 until a preset number of times is exceeded, and obtaining a trained model; and obtaining observation information at the current moment, and inputting the observation information at the current moment into the trained actor network to obtain a current action so as to control a traffic signal. According to the method, the safety is integrated into three aspects of environment interaction, state representation and strategy learning, so that the traffic safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic control, and more particularly to a traffic signal control method based on reinforcement learning and safety awareness aggregation. Background Art

[0002] As a key node in urban traffic, intersections play a vital role in traffic management and optimization. Therefore, how to improve road capacity while ensuring safety has become a severe challenge.

[0003] Reinforcement learning can receive feedback from the environment and try different actions based on a pre-designed reward function, thus having the ability to quickly adapt and make efficient decisions, and thus has become one of the main methods for optimizing traffic signal control. However, as a people-oriented, high-risk system, road traffic blindly pursues traffic efficiency while ignoring traffic safety. Unsafe control actions may cause unbearable consequences.

[0004] Although a small number of studies have begun to try to take safety factors into consideration, there are still some problems. For example, adding safety to the reward function of traffic efficiency can help reduce certain risks, but it may lead to longer exploration time and cannot guarantee risk aversion to the greatest extent. Therefore, some studies have proposed the concept of safety reinforcement learning. This method adds additional safety constraints or safety rules on the basis of traditional reinforcement learning in order to reduce safety risks in scenario deployment. However, the sparsity of costs and the complexity of predicting the original state greatly affect the performance of safety assessment, resulting in underestimation or overestimation of agent costs.

[0005] Therefore, there is an urgent need for a traffic signal control method to reduce safety risks as much as possible while ensuring safety and efficiency. Summary of the invention

[0006] The present invention is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present invention is to provide a reinforcement learning traffic signal control method based on safety awareness aggregation to reduce safety risks as much as possible while ensuring safety and efficiency.

[0007] In order to solve the above problems, the present invention is implemented by adopting the following technical solutions:

[0008] A reinforcement learning traffic signal control method based on safety awareness aggregation is provided, the method comprising: constructing a model, the model comprising a first module and a second module; training the model, comprising: step one: obtaining historical state information, the historical state information comprising first observation information, second observation information, action, reward value, overflow cost value and waiting cost value, the first observation information and the second observation information corresponding to the time are adjacent and sequential time; step two: updating the first module, the first module comprising a state encoder, an action-state encoder, an overflow encoder and a waiting encoder, comprising: using the state encoder to process the first observation information and the second observation information respectively to obtain a first feature and a second feature, and updating the first module; The feature and the second feature are normalized to obtain the third feature and the fourth feature respectively; the action and the third feature are processed by the action-state encoder to obtain the fifth feature; the third feature and the fourth feature are encoded by the overflow encoder to obtain the first overflow prediction score and the second overflow prediction score respectively; the third feature and the fourth feature are encoded by the waiting encoder to obtain the first waiting prediction score and the second waiting prediction score respectively; the first objective function is constructed based on the second feature, the fifth feature, the overflow cost value, the first overflow prediction score, the second overflow prediction score, the waiting cost value, the first waiting prediction score and the second waiting prediction score, and the first objective function is minimized to update each encoder; step three: update the second Module, the second module includes an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network, including: inputting the first observation information and the first feature into the actor network to obtain a first probability value; inputting the first feature and the second feature into the reward critic network, the overflow cost critic network and the waiting time critic network respectively to obtain corresponding reward values, overflow cost values ​​and waiting cost values, the reward value includes a first reward value corresponding to the first feature, the overflow cost value includes a first overflow cost value corresponding to the first feature, and the waiting cost value includes a first waiting cost value corresponding to the first feature; inputting the reward value and the reward value into the reward advantage network; The reward advantage evaluation value is obtained from the evaluation function; the overflow cost value and the overflow cost value are input into the overflow cost advantage evaluation function to obtain the overflow cost evaluation value; the waiting cost value and the waiting cost value are input into the waiting cost advantage evaluation function to obtain the waiting cost evaluation value; the actor network is optimized based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value; the reward critic network is optimized based on the reward value and the first reward value; the overflow cost critic network is optimized based on the overflow cost value and the first overflow cost value; the waiting time critic network is optimized based on the waiting cost value and the first waiting cost value; step 4: repeat steps 1 to 3 until the preset number of times is exceeded to obtain the trained model;Get the observation information at the current moment, input the observation information at the current moment into the trained actor network, and get the current action to control the traffic signal. ;

[0009] Optionally, the observation information includes queue length, average traffic flow, maximum occupancy, whether the current direction is a straight direction, whether the current moving direction is a green light, and whether the current moving direction reaches the minimum green light time; the reward value is calculated by a reward function, and the reward function is constructed based on lane density, queue length, waiting time, waiting tolerance time and number of waiting vehicles, the overflow cost value is calculated by an overflow cost function constructed based on lane density and a preset value, and the waiting cost value is calculated by a waiting time cost function constructed based on waiting time and waiting tolerance time.

[0010] Optionally, the expression of the first objective function is: in, Represents the prediction loss value, z i，t，sa represents the fifth feature, z i，t+1，s Represents the second feature, represents the bootstrap estimate of the overflow prediction score, D kl (.) represents KL divergence, z i，t，q represents the first overflow prediction score, represents the overflow score estimation guide for intersection i, represents the overflow cost, γ represents the discount factor, z i，t+1，q represents the second overflow prediction score, represents the bootstrap estimate of the waiting prediction score, z i，t，w represents the first waiting prediction score, represents the waiting score estimation guide for intersection i, represents the waiting cost, z i，t+1，w represents the second waiting prediction score, represents the first target value, λ i，q represents the first hyperparameter, λ i，w represents the second hyperparameter.

[0011] Optionally, the actor network is optimized based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value, including: inputting the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value into the second objective function, and obtaining the optimized actor network by minimizing the second objective function; the reward critic network is optimized based on the reward value and the first reward value, including: inputting the reward value and the first reward value into the loss function of the reward critic network and optimizing to obtain the optimized reward critic network; the overflow cost critic network is optimized based on the overflow cost value and the first overflow cost value, including: inputting the overflow cost value and the first overflow cost value into the loss function of the overflow cost critic network and optimizing to obtain the optimized overflow cost critic network; the waiting time critic network is optimized based on the waiting cost value and the first waiting cost value, including: inputting the waiting cost value and the first waiting cost value into the loss function of the waiting time critic network and optimizing to obtain the optimized waiting time critic network.

[0012] Optionally, it also includes: executing the current action to obtain the observation information of the next moment; calculating the current reward value according to the reward function; calculating the current overflow cost value according to the overflow cost function; calculating the current waiting cost value according to the waiting time cost function; constructing the current state information based on the observation information of the current moment, the observation information of the next moment, the current action, the current reward value, the current overflow cost value and the current waiting cost value; updating the first module and the second module based on the current state information, and using the updated first module and the second module to process the observation information of the next moment.

[0013] Optionally, the reward function is expressed as: in, represents the reward value of intersection i at time t, l represents the moving direction, represents the queue length at intersection i in the moving direction l at time t, represents the maximum queue length at intersection i in the moving direction l at time t, n i，l represents the number of vehicles waiting at intersection i in the moving direction l, j represents the identification number of the vehicle, k j，t+1 represents the waiting time of vehicle j at time t+1, C represents the vehicle waiting tolerance time, l out Represents the set of exits at intersection i in the moving direction l.

[0014] Optionally, the overflow cost function is expressed as: in, represents the overflow cost value, ratio represents the preset value; the expression of the waiting time cost function is: in, Indicates the waiting cost value.

[0015] Optionally, the expression of the reward advantage evaluation function is: in, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding reward advantage evaluation value, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

[0016] Optionally, the expression of the second objective function is: , in, represents the Lagrange multiplier of the waiting time cost function of intersection i, represents the Lagrange multiplier of intersection i with respect to the overflow cost function, represents the value of the second objective function, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, and max(.) represents the function of taking the maximum value. Indicates that the actor strategy network in the current iteration process is based on the first observation information Output Action The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output Action The probability value of Restrictions In the range of [1-ε, 1+ε], if it is greater than 1+ε, it takes the value of 1+ε; if it is less than 1-ε, it takes the value of 1-ε; otherwise, it remains unchanged; Represents the comprehensive evaluation value, Indicates that based on the first observation information Output Action The corresponding reward advantage evaluation value, Indicates that based on the first observation information Output action ai，t The corresponding waiting cost evaluation value, Indicates that based on the first observation information Output action a i，t The corresponding overflow cost assessment value.

[0017] Optionally, it also includes: using the mean square error to construct the loss functions of the reward critic network, the overflow cost critic network and the waiting time critic network, minimizing the loss function of each network through the Adam gradient descent algorithm, updating the parameters of each network, and obtaining the optimized reward critic network, the overflow cost critic network and the waiting time critic network.

[0018] Compared with the prior art, the present invention proposes a reinforcement learning traffic signal control method based on safety awareness aggregation, which integrates safety into three aspects: environmental interaction, state representation, and strategy learning. First, considering the two major risk factors of overflow and excessive waiting time, a safety reward function and a cost function are established, and a safety representation method is proposed to better extract safety awareness representation, and then the safety constraints and safety state representation are input into the safety reinforcement learning method to achieve the output of safe actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is a flow chart of a traffic signal control method based on reinforcement learning for safety awareness aggregation provided by this embodiment;

[0021] Figure 2 is a flow chart of the training model process provided in this embodiment;

[0022] Figure 3 is a flowchart of a process for updating the first module provided in this embodiment;

[0023] Figure 4 is a flowchart of a process for updating the second module provided in this embodiment;

[0024] Figure 5 It is a schematic diagram of the intersection movement direction and action space provided by this embodiment. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] To facilitate understanding of the embodiments of the present invention, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the protection scope of the present invention.

[0027] This embodiment provides a traffic signal control method based on reinforcement learning of safety awareness aggregation, and its process is as follows: Figure 1 As shown, including:

[0028] S1 builds the model.

[0029] The model includes a first module and a second module, the first module includes: a state encoder, an action-state encoder, an overflow encoder and a waiting encoder; the second module includes an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network.

[0030] S2 trains the model.

[0031] like Figure 2 As shown, in this step, it includes:

[0032] S200 obtains historical status information.

[0033] The historical state information includes first observation information, second observation information, action, reward value, overflow cost value and waiting cost value, the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by a reward function, and the reward function is constructed based on lane density, queue length, waiting time, waiting tolerance time and number of waiting vehicles; the overflow cost value is calculated by an overflow cost function constructed based on lane density and a preset value; the waiting cost value is calculated by a waiting time cost function constructed based on the waiting time and the waiting tolerance time.

[0034] like Figure 5 As shown, for the cross intersection i, the four entrance roads of east, west, south and north have lanes of three different directions: left turn, straight ahead and right turn, and no control is performed on right-turning vehicles. In this embodiment, there are eight movement directions l that need to be controlled, namely, north straight ahead, north left turn, south straight ahead, south left turn, east straight ahead, east left turn, east straight ahead and east left turn, and the corresponding identification numbers are 1, 2, 3, ..., 8 respectively.

[0035] Each intersection is treated as a separate safety reinforcement learning agent. For each agent, considering safety factors, its interaction with the environment is modeled as a restricted Markov decision process, which mainly includes observation space, action space, reward function, cost function and discount factor.

[0036] The observation space consists of three parts: traffic characteristics, motion characteristics, and traffic signal characteristics. At the intersection i at time t, the traffic characteristics are used to count the overall road information of each moving direction l, including the queue length Average traffic volume and maximum occupancy The motion characteristics provide specific details about each moving direction l motion itself, mainly used to indicate whether the direction is a straight direction Traffic signal features represent the state of the traffic light corresponding to each moving direction l, including an indicator of whether the current moving direction is green. Indicator of whether the green light duration corresponding to the current moving direction reaches the minimum green light time Specifically, at time t, the state of each moving direction l of intersection i is represented by as follows:

[0037]

[0038] in, The value is either 0 or 1. When the value is 0, It means that at time t, the moving direction l of intersection i is not the straight direction, It means that at time t, the traffic light at intersection i in the moving direction l is yellow or red. Indicates that at time t, the duration of the green light in the moving direction l of intersection i is less than or equal to the minimum green light time. When the value is 1, It means that at time t, the moving direction l of intersection i is the straight direction, It means that at time t, the traffic light at intersection i in the moving direction l is green. It means that at time t, the green light duration of intersection i in moving direction l is greater than the minimum green light time.

[0039] At time t, the first observation information of intersection i as follows:

[0040]

[0041] In the formula, They represent the states of intersection i at time t: going straight north, turning left north, going straight south, turning left south, going straight east, turning left east, going straight east, and turning left east.

[0042] Similarly, the second observation information of intersection i is

[0043] The action space is to set several stages, each stage allows a set of specific movement direction combinations, while restricting the movement of other movement directions. In this embodiment, 4 stages are set, except that right turn is not controlled, and 2 movement direction combinations are allowed. Specifically, the action of intersection i at time t is defined. as follows:

[0044]

[0045] The reward function is the agent's measure of the correctness of the current decision based on the set goal before taking action. In this embodiment, the reward function takes into account the safety factors of overflow and long waiting time for a single vehicle, and defines the reward function of intersection i at time t. as follows:

[0046]

[0047] in, represents the reward value of intersection i at time t, l represents the moving direction, represents the queue length at intersection i in the moving direction l at time t, represents the maximum queue length at intersection i in the moving direction l at time t, n i，l represents the number of vehicles waiting at intersection i in the moving direction l, j represents the identification number of the vehicle, k j，t+1 represents the waiting time of vehicle j at time t+1, C represents the vehicle waiting tolerance time, which is a positive integer set manually; out Represents the set of exits at intersection i in the moving direction l.

[0048] The cost function is used to measure the safety of the current state and action to illustrate the safety of the current decision. Similar to the reward function, in this embodiment, the cost function considers two major safety factors: overflow and long waiting time for a single vehicle, and defines overflow cost functions respectively. And the waiting time cost function The specific formula is as follows:

[0049]

[0050]

[0051] in, represents the overflow cost value, ratio represents the preset value, which is a parameter set manually, and its value range is [0, 1). Indicates the waiting cost value.

[0052] The discount factor γ is not only used to calculate the long-term cumulative reward, but also used to calculate the long-term cumulative cost, and its value range is [0, 1].

[0053] S210 updates the first module.

[0054] The first module includes a state encoder, an action-state encoder, an overflow encoder, and a wait encoder.

[0055] In order to better capture features related to security awareness, this embodiment proposes a security characterization method based on state-action encoding, which mainly enhances the security understanding ability of the current state and action by learning the embedding of state and state-action.

[0056] like Figure 3 As shown, in this step, it includes:

[0057] Step 1: Use a state encoder to process the first observation information and the second observation information respectively to obtain the first feature and the second feature.

[0058] For intersection i, use the state encoder f i Encoding the first observation information To obtain the encoding state embedding, that is, the first feature z i，t，s , whose expression is The state encoder of this embodiment can be composed of a multi-layer perceptron in a neural network.

[0059] In order to make the state encoder have the ability to predict the state, the second observation information Input to the state encoder f' to obtain the future state embedding, that is, the second feature z i，t+1，s , whose expression is Among them, f i ′ and f i The network parameters are shared but not learned.

[0060] Step 2: Normalize the first feature and the second feature respectively to obtain the third feature and the fourth feature.

[0061] In order to deal with the instability risk caused by distance minimization in the embedding space, a normalization layer AvgL1Norm is introduced after the state encoder to transform the first feature z i，t，s Input into the normalization layer to obtain the third feature z′ i，t，s , whose expression is z′ i，t，s =AvgL1Norm(zi，t，s ), in, Represents the kth dimension in the first feature of an N-dimensional vector, where N represents the number of dimensions of the first feature.

[0062] Similarly, the second feature z i，t+1，s Input into the normalization layer to obtain the fourth feature z' i，t+1，s .

[0063] Step 3: Use the action-state encoder to process the action and the third feature to obtain the fifth feature.

[0064] Using the state-action encoder g i Encode the third feature z′ i，t，s And actions , obtain the state-action embedding, that is, the fourth feature z i，t，sa , whose expression is , the state-action encoder can be composed of a multi-layer perceptron in a neural network.

[0065] Step 4: Use the overflow encoder to encode the third feature and the fourth feature respectively to obtain the first overflow prediction score and the second overflow prediction score respectively.

[0066] In order to further enhance the agent's safety representation awareness, a feasibility score is introduced as part of state representation learning. The feasibility score is related to the Hamilton-Jacobi reachability in safety theory, which is mainly used to represent the degree of state safety, thereby improving the safety representation performance.

[0067] Based on the prediction representation, an additional supervisory signal is used to extract the prediction feasibility score representing the safety level. Specifically, the overflow encoder p i，q For the third feature z′ i，t，s Encode to get the first overflow prediction score z i，t，q , whose expression is: i，t，q =p i，q (z′ i，t，s ), where p i，q Represents the overflow encoder of intersection i.

[0068] Similarly, the overflow encoder is used to encode the fourth feature to obtain the second overflow prediction score z i，t+1，q .

[0069] Step 5: Use the waiting encoder to encode the third feature and the fourth feature respectively to obtain the first waiting prediction score and the second waiting prediction score respectively.

[0070] Using the waiting encoder p i，w For the third feature z′i，t，s Encode to get the first waiting prediction score z i，t，w , whose expression is: i，t，w =p i，w (z′ i，t，s ), where p i，w is the waiting encoder for intersection i.

[0071] Similarly, the fourth feature is encoded using the waiting encoder to obtain the second waiting prediction score z i，t+1，w .

[0072] The overflow encoder and the waiting encoder can be constructed by multi-layer perceptrons in a neural network.

[0073] Step six: construct a first objective function based on the second feature, the fifth feature, the overflow cost value, the first overflow prediction score, the second overflow prediction score, the waiting cost value, the first waiting prediction score and the second waiting prediction score, and minimize the first objective function to update each encoder.

[0074] The second feature is taken as the true value and the second feature z is measured by the mean square error. i，t+1，s With the fifth characteristic z i，t，sa The average error between them is used to obtain the prediction loss function of intersection i to evaluate the accuracy of the prediction:

[0075]

[0076] in, Represents the prediction loss value.

[0077] Based on the overflow cost value, the first overflow prediction score and the second overflow prediction score, a guided estimation loss function of the overflow prediction score is constructed, and its expression is:

[0078]

[0079]

[0080] in, represents the bootstrap estimate of the overflow prediction score, D kl (.) represents KL divergence, z i，t，q represents the first overflow prediction score, represents the overflow score estimation guide of intersection i, max(.) represents the maximum value function, represents the overflow cost, γ represents the discount factor, z i，t+1，q Represents the second overflow prediction score.

[0081] Based on the waiting cost value, the first waiting prediction score and the second waiting prediction score, a guided estimation loss function of the waiting prediction score is constructed, and its expression is:

[0082]

[0083]

[0084] in, represents the bootstrap estimate of the waiting prediction score, z i，t，w represents the first waiting prediction score, represents the waiting score estimation guide for intersection i, represents the waiting cost, z i，t+1，w Represents the second waiting prediction score.

[0085] Based on the prediction loss function, the guided estimation loss function of the overflow prediction score and the guided estimation loss function of the waiting prediction score, the first objective function is constructed, and its expression is:

[0086]

[0087] in, represents the first target value, λ i，q represents the first hyperparameter, λ i，w represents the second hyperparameter.

[0088] By minimizing the first objective function, the state encoder, state-action encoder, overflow encoder, and waiting encoder are updated using the Adam gradient descent method.

[0089] S220 updates the second module.

[0090] The second module includes the actor network, reward critic network, overflow cost critic network and waiting time critic network.

[0091] In order to make the agent under the joint constraints of excessive waiting time and overflow, the actor network incorporates safety representation, takes the first observation information and the first feature as input to evaluate the feasibility of the state, and outputs a safe phase action to ensure the short-term safety of traffic signal control. The critic network includes a reward critic network, an overflow cost critic network, and a waiting time critic network, which take the first feature and the second feature as input respectively to estimate the expected reward and expected cost of the current state, so as to provide reward maximization under the long-term safety of traffic signal control.

[0092] like Figure 4 As shown, in this step, it includes:

[0093] Step 1: Input the first observation information and the first feature into the actor network to obtain a first probability value.

[0094] For intersection i, enter the first observation information And the first feature z i，t，s To the Actor Network , thereby calculating the probability values ​​of different actions. Among them, the actor network consists of 3 linear layers and 2 activation function layers. The specific calculation formula is as follows:

[0095]

[0096] in, represents the first probability value, θ i are the parameters of the network of the actor at intersection i, softmax(.) represents the normalization function in the neural network, FC(.) represents the linear layer in the neural network, and ReLU(.) represents the activation function in the neural network.

[0097] Step 2: Input the first feature and the second feature into the reward critic network, the overflow cost critic network and the waiting time critic network respectively to obtain the corresponding reward value, overflow cost value and waiting cost value.

[0098] The structures of the reward critic network, overflow cost critic network, and waiting time critic network are consistent with those of the actor network.

[0099] The reward value includes a first reward value corresponding to the first feature and a second reward value corresponding to the second feature, the overflow cost value includes a first overflow cost value corresponding to the first feature and a second overflow cost value corresponding to the second feature, and the waiting cost value includes a first waiting cost value corresponding to the first feature and a second waiting cost value corresponding to the second feature.

[0100] The first feature z i，t，s Input to the reward critic network Get the first reward value The specific calculation formula is as follows:

[0101]

[0102] Similarly, the second feature z i，t+1，s Input to the reward critic network , get the second reward value

[0103] The first feature z i，t，s Input to the overflow cost critic network , get the first overflow cost value The specific calculation formula is as follows:

[0104]

[0105] Similarly, the second feature z i，t+1，s Input to the overflow cost critic network , get the second overflow cost value

[0106] The first feature z i，t，s Input to the latency critic network , get the first waiting cost value The specific calculation formula is as follows:

[0107]

[0108] Similarly, the second feature z i，t+1，s Input to the latency critic network , get the second waiting cost value

[0109] Step 3: Input the reward value and reward value into the reward advantage evaluation function to obtain the reward advantage evaluation value.

[0110] This step will reward First Reward Value and the second reward value Input into the following formula to calculate the bonus advantage evaluation value of intersection i

[0111]

[0112]

[0113] in, represents the reward advantage estimate of the kth stage of intersection i at time t, k represents the stage of advantage evaluation, m represents the mth stage in the advantage evaluation process, represents the reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding reward advantage evaluation value, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

[0114] Step 4: Input the overflow cost value and the overflow price value into the overflow price advantage evaluation function to obtain the overflow price evaluation value.

[0115] This step will overflow the cost First overflow cost value and the second overflow cost value Input into the following formula to calculate the overflow cost assessment value of intersection i

[0116]

[0117]

[0118] in, represents the estimated overflow cost of the kth stage of intersection i at time t, represents the overflow cost of intersection i at time t, Indicates the second observation information The corresponding second overflow cost value, Indicates the first observation information The corresponding first overflow cost value, Indicates that based on the first observation information Output Action The corresponding overflow cost assessment value.

[0119] Step 5: Input the waiting cost value and the waiting cost value into the waiting cost advantage evaluation function to obtain the waiting cost evaluation value.

[0120] This step will wait for the cost value First waiting cost value and the second waiting cost value Input into the following formula to calculate the waiting cost evaluation value of intersection i

[0121]

[0122]

[0123] in, represents the estimated waiting cost of the kth stage of intersection i at time t, represents the waiting cost of intersection i at time t, Indicates the second observation information The corresponding second waiting cost value, Indicates the first observation information The corresponding first waiting cost value, Indicates that based on the first observation information Output Action The corresponding waiting cost evaluation value.

[0124] Step 6: Optimize the actor network based on the reward advantage evaluation value, overflow cost evaluation value, waiting value evaluation value and first probability value.

[0125] The reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value are input into the second objective function, and the optimized actor network is obtained by minimizing the second objective function. The expression of the second objective function is:

[0126]

[0127]

[0128] in, represents the Lagrange multiplier of the waiting time cost function of intersection i, represents the Lagrange multiplier of intersection i with respect to the overflow cost function, represents the value of the second objective function, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, Indicates that the actor strategy network in the current iteration process is based on the first observation information Output Action The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output Action The probability value of Restrictions In the range of [1-ε, 1+ε], if it is greater than 1+ε, it is taken as 1+ε; if it is less than 1-ε, it is taken as 1-ε; otherwise, it remains unchanged; Indicates the comprehensive evaluation value.

[0129] According to the above Update the parameters θ of the actor network i , Lagrange multiplier of waiting time cost function and the Lagrange multiplier of the overflow cost function The specific calculation formula is as follows:

[0130]

[0131]

[0132]

[0133] Among them, θ i′ represents the updated parameters of the actor network, θ i represents the parameters of the actor network that are not updated, represents the compensation of gradient descent, represents L(θ i ,λ i ) for θ i Derivative, λ i ′ represents the updated Lagrange multiplier, λ i represents the unupdated Lagrange multiplier, represents the step size of gradient ascent, represents L(θ i ,λ i ) for λ i Derivative.

[0134] This embodiment uses the Adam gradient descent algorithm to update the actor network.

[0135] Step 7: Optimize the reward-critic network based on the reward value and the first reward value.

[0136] The reward value and the first reward value are input into the loss function of the reward-critic network and optimized to obtain an optimized reward-critic network.

[0137] The loss function L of the reward critic network r (ω i，r ) is:

[0138]

[0139] By minimizing the loss function L r (ω i，r ), and use the Adam gradient descent method to update the parameters ω of the reward critic network i，r .

[0140] Step 8: Optimize the overflow cost critic network based on the overflow generation value and the first overflow cost value.

[0141] The overflow cost value and the first overflow cost value are input into the loss function of the overflow cost critic network and optimized to obtain the optimized overflow cost critic network.

[0142] The loss function L of the overflow cost critic network q (ω i，q ) is:

[0143]

[0144] By minimizing the loss function L q (ω i，q), and use the Adam gradient descent method to update the parameters ω of the overflow cost critic network i，q .

[0145] Step 9: Optimize the waiting time critic network based on the waiting cost value and the first waiting cost value.

[0146] The waiting cost value and the first waiting cost value are input into the loss function of the waiting time critic network and optimized to obtain the optimized waiting time critic network.

[0147] The loss function L of the latency critic network w (ω i，w ) is:

[0148]

[0149] By minimizing the loss function L w (ω i，w ), and use the Adam gradient descent method to update the parameters ω of the waiting time critic network i，w .

[0150] S230 repeats S200 to S220 until the preset number of times is exceeded to obtain a trained model.

[0151] S3 obtains the observation information at the current moment, inputs the observation information at the current moment into the trained actor network, and obtains the current action to control the traffic signal.

[0152] Furthermore, the method further comprises:

[0153] Execute the current action and obtain the observation information of the next moment.

[0154] The current reward value is calculated according to the reward function.

[0155] The current overflow cost value is calculated according to the overflow cost function.

[0156] The current waiting cost value is calculated according to the waiting time cost function.

[0157] The current state information is constructed based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, the current overflow cost value, and the current waiting cost value.

[0158] The first module and the second module are updated based on the current state information, and the observation information at the next moment is processed using the updated first module and the second module.

[0159] This embodiment proposes a reinforcement learning traffic signal control method based on safety awareness aggregation, which integrates safety into three aspects: environmental interaction, state representation, and strategy learning. First, considering the two major risk factors of overflow and excessive waiting time, a safety reward function and a cost function are established, and a safety representation method is proposed to better extract safety awareness representation, and then the safety constraints and safety state representation are input into the safety reinforcement learning method to achieve the output of safe actions.

[0160] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A traffic signal control method based on reinforcement learning based on safety awareness aggregation, characterized in that: include: Constructing a model, wherein the model includes a first module and a second module; Training the model, including: Step 1: Obtain historical status information, the historical status information includes first observation information, second observation information, action, reward value, overflow cost value and waiting cost value, and the time corresponding to the first observation information and the second observation information are adjacent time periods; Step 2: updating the first module, the first module including a state encoder, an action-state encoder, an overflow encoder and a waiting encoder, including: using the state encoder to process the first observation information and the second observation information to obtain the first feature and the second feature respectively, and normalizing the first feature and the second feature respectively to obtain the third feature and the fourth feature; using the action-state encoder to process the action and the third feature to obtain the fifth feature; using the overflow encoder to encode the third feature and the fourth feature respectively, to obtain the first overflow prediction score and the second overflow prediction score respectively; using the waiting encoder to encode the third feature and the fourth feature respectively, to obtain the first waiting prediction score and the second waiting prediction score respectively; constructing a first objective function based on the second feature, the fifth feature, the overflow cost value, the first overflow prediction score, the second overflow prediction score, the waiting cost value, the first waiting prediction score and the second waiting prediction score, and minimizing the first objective function to update each encoder; Step 3: Update the second module, which includes an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network, including: inputting the first observation information and the first feature into the actor network to obtain a first probability value; inputting the first feature and the second feature into the reward critic network, the overflow cost critic network and the waiting time critic network respectively to obtain corresponding reward values, overflow cost values ​​and waiting cost values, wherein the reward value includes a first reward value corresponding to the first feature, the overflow cost value includes a first overflow cost value corresponding to the first feature, and the waiting cost value includes a first waiting cost value corresponding to the first feature. value; input the reward value and the reward value into the reward advantage evaluation function to obtain the reward advantage evaluation value; input the overflow cost value and the overflow cost value into the overflow cost advantage evaluation function to obtain the overflow cost evaluation value; input the waiting cost value and the waiting cost value into the waiting cost advantage evaluation function to obtain the waiting cost evaluation value; optimize the actor network based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value; optimize the reward critic network based on the reward value and the first reward value; optimize the overflow cost critic network based on the overflow cost value and the first overflow cost value; optimize the waiting time critic network based on the waiting cost value and the first waiting cost value; Step 4: Repeat steps 1 to 3 until the preset number of times is exceeded to obtain the trained model; Obtain the observation information at the current moment, input the observation information at the current moment into the trained actor network, and obtain the current action to control the traffic signal.

2. According to claim 1, a reinforcement learning traffic signal control method based on safety awareness aggregation is characterized in that: The observation information includes queue length, average traffic flow, maximum occupancy, whether the current direction is straight ahead, whether the current moving direction has a green light, and whether the current moving direction has reached the minimum green light time; The reward value is calculated by a reward function, which is constructed based on lane density, queue length, waiting time, waiting tolerance time and number of waiting vehicles. The overflow cost value is calculated by an overflow cost function constructed based on lane density and a preset value. The waiting cost value is calculated by a waiting time cost function constructed based on waiting time and waiting tolerance time.

3. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 1 is characterized in that: The expression of the first objective function is: in, Represents the prediction loss value, z i,t,sa represents the fifth feature, z i,t+1,s Represents the second feature, represents the bootstrap estimate of the overflow prediction score, D kl (.) represents KL divergence, z i,t,q represents the first overflow prediction score, represents the overflow score estimation guide for intersection i, represents the overflow cost, γ represents the discount factor, z i,t+1,q represents the second overflow prediction score, represents the bootstrap estimate of the waiting prediction score, z i,t,w represents the first waiting prediction score, represents the waiting score estimation guide for intersection i, represents the waiting cost, z i,t+1,w represents the second waiting prediction score, represents the first target value, λ i,q represents the first hyperparameter, λ i,w represents the second hyperparameter.

4. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 1 is characterized in that: Optimizing the actor network based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value, including: inputting the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value into a second objective function, and obtaining an optimized actor network by minimizing the second objective function; Optimizing the reward-critic network based on the reward value and the first reward value, including: inputting the reward value and the first reward value into a loss function of the reward-critic network and optimizing to obtain an optimized reward-critic network; Optimizing the overflow cost critic network based on the overflow cost value and the first overflow cost value, including: inputting the overflow cost value and the first overflow cost value into the loss function of the overflow cost critic network and optimizing to obtain an optimized overflow cost critic network; Optimizing the waiting time critic network based on the waiting cost value and the first waiting cost value includes: inputting the waiting cost value and the first waiting cost value into the loss function of the waiting time critic network and optimizing to obtain an optimized waiting time critic network.

5. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 2 is characterized in that: Also includes: Execute the current action and obtain the observation information of the next moment; The current reward value is calculated according to the reward function; The current overflow cost value is calculated according to the overflow cost function; The current waiting cost value is calculated according to the waiting time cost function; Construct the current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, the current overflow cost value, and the current waiting cost value; The first module and the second module are updated based on the current state information, and the observation information at the next moment is processed using the updated first module and the second module.

6. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 2 is characterized in that: The expression of the reward function is: in, represents the reward value of intersection i at time t, l represents the moving direction, represents the queue length at intersection i in the moving direction l at time t, represents the maximum queue length at intersection i in the moving direction l at time t, n i,l represents the number of vehicles waiting at intersection i in the moving direction l, j represents the identification number of the vehicle, k j,t+1 represents the waiting time of vehicle j at time t+1, C represents the vehicle waiting tolerance time, l out Represents the set of exits at intersection i in the moving direction l.

7. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 6 is characterized in that: The expression of the overflow cost function is: in, represents the overflow cost value, and ratio represents the preset value; The expression of the waiting time cost function is: in, Indicates the waiting cost value.

8. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 1 is characterized in that: The expression of the reward advantage evaluation function is: in, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding reward advantage evaluation value, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

9. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 4 is characterized in that: The expression of the second objective function is: in, represents the Lagrange multiplier of the waiting time cost function of intersection i, represents the Lagrange multiplier of intersection i with respect to the overflow cost function, represents the value of the second objective function, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, and max(.) represents the function of taking the maximum value. Indicates that the actor strategy network in the current iteration process is based on the first observation information Output Action The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output Action The probability value of Restrictions In the range of [1-ε,1+ε], if it is greater than 1+ε, it takes the value of 1+ε; if it is less than 1-ε, it takes the value of 1-ε; otherwise, it remains unchanged; Represents the comprehensive evaluation value, Indicates that based on the first observation information Output Action The corresponding reward advantage evaluation value, Indicates that based on the first observation information Output action a i,t The corresponding waiting cost evaluation value, Indicates that based on the first observation information Output action a i,t The corresponding overflow cost assessment value.

10. The method for controlling traffic signals by reinforcement learning based on safety awareness aggregation according to claim 1 is characterized in that: Also includes: The mean square error is used to construct the loss functions of the reward critic network, the overflow cost critic network and the waiting time critic network. The Adam gradient descent algorithm is used to minimize the loss function of each network, and the parameters of each network are updated to obtain the optimized reward critic network, the overflow cost critic network and the waiting time critic network.

Images