A reinforcement learning traffic signal control method based on safety awareness aggregation

Abstract

The application discloses a kind of reinforcement learning traffic signal control methods based on safety awareness aggregation, comprising: model is built;The model is trained, comprising: step one: obtaining historical state information;Step two: update first module, the first module includes state encoder, action-state encoder, overflow encoder and waiting encoder;Step three: update second module, the second module includes actor network, reward critic network, overflow cost critic network and waiting time critic network;Step four: repeat step one to step three until more than preset number, obtain the model after training;Obtain the observation information of current time, input the observation information of current time into the actor network after training, obtain current action to control traffic signal.The application integrates safety into environment interaction, state representation and policy learning three aspects, improve the safety of traffic.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent traffic control. Specifically, it relates to a reinforcement learning traffic signal control method based on safety awareness aggregation. BACKGROUND

[0002] As a key node in urban traffic, intersections play a crucial role in traffic management and optimization. Therefore, how to improve road capacity while ensuring safety has become a serious challenge.

[0003] Reinforcement learning can receive feedback from the environment and try different actions according to a pre-designed reward function, thus having rapid adaptability and efficient decision-making ability, and has become one of the main methods for traffic signal control optimization. However, as a human-centered high-risk system, blindly pursuing traffic efficiency while ignoring traffic safety may result in unbearable consequences.

[0004] Although a small number of studies have begun to consider safety as a factor, there are still some problems. For example, adding safety to the reward function of traffic efficiency, this method can help reduce some risks, but may result in longer exploration time and cannot guarantee risk avoidance to the greatest extent. Therefore, some studies have proposed the concept of safety reinforcement learning. This method adds additional safety constraints or safety rules to the traditional reinforcement learning to reduce safety risks in scenario deployment. However, the sparsity of costs and the complexity of predicting the original state greatly affect the performance of safety evaluation, resulting in underestimation or overestimation of agent costs.

[0005] Therefore, there is an urgent need for a traffic signal control method to reduce safety risks as much as possible while ensuring safety efficiency. SUMMARY

[0006] The present application is proposed based on the above-mentioned needs of the prior art, and the technical problem to be solved by the present application is to provide a reinforcement learning traffic signal control method based on safety awareness aggregation to reduce safety risks as much as possible while ensuring safety efficiency.

[0007] In order to solve the above problems, the present application is realized by adopting the following technical solutions:

[0008] The application provides a reinforcement learning traffic signal control method based on safety awareness aggregation, and the method comprises the following steps: constructing a model, wherein the model comprises a first module and a second module; training the model, comprising the following steps: step 1, acquiring historical state information, wherein the historical state information comprises first observation information, second observation information, an action, a reward value, an overflow cost value and a waiting cost value, the first observation information and the second observation information correspond to adjacent time points; step 2, updating the first module, wherein the first module comprises a state encoder, an action-state encoder, an overflow encoder and a waiting encoder, and the step 2 comprises the following steps: processing the first observation information and the second observation information by using the state encoder to obtain first features and second features respectively, and performing normalization on the first features and the second features to obtain third features and fourth features respectively; processing the action and the third features by using the action-state encoder to obtain fifth features; encoding the third features and the fourth features by using the overflow encoder to obtain first overflow prediction scores and second overflow prediction scores respectively; encoding the third features and the fourth features by using the waiting encoder to obtain first waiting prediction scores and second waiting prediction scores respectively; constructing a first objective function based on the second features, the fifth features, the overflow cost value, the first overflow prediction scores, the second overflow prediction scores, the waiting cost value, the first waiting prediction scores and the second waiting prediction scores, and minimizing the first objective function to update the encoders; step 3, updating the second module, wherein the second module comprises an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network, and the step 3 comprises the following steps: inputting the first observation information and the first features into the actor network to obtain first probability values; inputting the first features and the second features into the reward critic network, the overflow cost critic network and the waiting time critic network respectively to obtain corresponding reward values, overflow cost values and waiting cost values, wherein the reward values comprise first reward values corresponding to the first features, the overflow cost values comprise first overflow cost values corresponding to the first features, and the waiting cost values comprise first waiting cost values corresponding to the first features; inputting the reward value and the reward values into a reward advantage evaluation function to obtain a reward advantage evaluation value; inputting the overflow cost value and the overflow cost values into an overflow cost advantage evaluation function to obtain an overflow cost evaluation value; inputting the waiting cost value and the waiting cost values into a waiting cost advantage evaluation function to obtain a waiting cost evaluation value; optimizing the actor network based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability values; optimizing the reward critic network based on the reward value and the first reward values; optimizing the overflow cost critic network based on the overflow cost value and the first overflow cost values; optimizing the waiting time critic network based on the waiting cost value and the first waiting cost values; and step 4, repeating the steps 1 to 3 until a preset number of times is exceeded, and obtaining a trained model.Obtaining observation information at the current moment, inputting the observation information at the current moment into the trained actor network to obtain a current action to control the traffic signal.

[0009] Optionally, the observation information comprises a queue length, an average traffic flow, a maximum occupancy rate, whether a current direction is a straight direction, whether a current moving direction is a green light, and whether the current moving direction reaches a minimum green light time; the reward value is calculated by a reward function, the reward function is constructed based on a lane density, a queue length, a waiting time, a waiting tolerance time, and a number of waiting vehicles, the overflow cost value is calculated by an overflow cost function constructed based on the lane density and a preset value, and the waiting cost value is calculated by a waiting time cost function constructed based on the waiting time and the waiting tolerance time.

[0010] Optionally, an expression of the first objective function is as follows: wherein, represents a prediction loss value, z i，t，sa represents a fifth feature, z i，t+1，s represents a second feature, represents a guide estimation value of an overflow prediction score, D kl (.) represents a KL divergence, z i，t，q represents a first overflow prediction score, represents an estimation guide of the overflow score of the intersection i, represents an overflow cost value, and γ represents a discount factor, z i，t+1，q represents a second overflow prediction score, represents a guide estimation value of a waiting prediction score, z i，t，w represents a first waiting prediction score, represents an estimation guide of the waiting score of the intersection i, represents a waiting cost value, z i，t+1，w represents a second waiting prediction score, represents a first objective value, λ i，q represents a first hyperparameter, λ i，w represents a second hyperparameter.

[0011] Optionally, the actioner network is optimized based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value, including: inputting the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value into a second objective function, and obtaining the optimized actioner network by minimizing the second objective function; the reward critic network is optimized based on the reward value and the first reward value, including: inputting the reward value and the first reward value into a loss function of the reward critic network and obtaining the optimized reward critic network by optimization; the overflow cost critic network is optimized based on the overflow cost value and the first overflow cost value, including: inputting the overflow cost value and the first overflow cost value into a loss function of the overflow cost critic network and obtaining the optimized overflow cost critic network by optimization; and the waiting time critic network is optimized based on the waiting cost value and the first waiting cost value, including: inputting the waiting cost value and the first waiting cost value into a loss function of the waiting time critic network and obtaining the optimized waiting time critic network by optimization.

[0012] Optionally, the method further includes: performing the current action to obtain observation information at the next moment; calculating a current reward value according to a reward function; calculating a current overflow cost value according to an overflow cost function; calculating a current waiting cost value according to a waiting time cost function; constructing current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, the current overflow cost value and the current waiting cost value; and updating the first module and the second module based on the current state information, and processing the observation information at the next moment by using the updated first module and the second module.

[0013] Optionally, the expression of the reward function is: wherein, represents the reward value of the intersection i at the t moment, and l represents a moving direction, represents the queue length of the intersection i in the moving direction l at the t moment, represents the maximum queue length of the intersection i in the moving direction l at the t moment, and n i，l represents the number of waiting vehicles of the intersection i in the moving direction l, j represents the identification number of a vehicle, and k j，t+1 represents the waiting time of the vehicle j at the t+1 moment, and C represents the vehicle waiting tolerance time, out represents the exit set of the intersection i in the moving direction l.

[0014] Optionally, the expression of the overflow cost function is: wherein, represents the overflow cost value, and ratio represents a preset value; and the expression of the waiting time cost function is: wherein, represents the waiting cost value.

[0015] Optionally, the expression of the reward advantage evaluation function is: wherein, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, and m represents the mth stage in the advantage evaluation process, represents the reward value of intersection i at time t, represents the second observation information corresponding to the second reward value, represents the first observation information corresponding to the first reward value, represents the action output based on the first observation information represents the action corresponding to the reward advantage evaluation value, λ GAE represents the advantage value parameter, which is a parameter for controlling the average degree of advantage value, and Y represents the length of the sampled trajectory.

[0016] Optionally, the expression of the second objective function is: wherein, represents the Lagrange multiplier of intersection i with respect to the waiting time cost function, represents the Lagrange multiplier of intersection i with respect to the overflow cost function, represents the second objective function value, represents the expected experience return value under the actor policy network, min(.) represents a function for taking the minimum value, and max(.) represents a function for taking the maximum value, represents the probability value of the action output by the actor policy network according to the first observation information in the current iteration process, represents the probability value of the action output by the actor policy network according to the first observation information in the last stage of the current iteration process, represents the limit in the range of [1-ε, 1+ε], if greater than 1+ε, the value is 1+ε; if less than 1-ε, the value is 1-ε; otherwise, it remains unchanged; represents the comprehensive evaluation value, represents the action output based on the first observation information corresponding to the reward advantage evaluation value, represents the action a output based on the first observation informationi，t a corresponding waiting cost evaluation value, representing a first observation information output action a i，t a corresponding overflow cost evaluation value.

[0017] Optionally, further comprising: constructing a loss function of the reward critic network, the overflow cost critic network and the waiting time critic network by using a mean square error error, minimizing the loss function of each network by an Adam gradient descent algorithm, updating parameters of each network, and obtaining the optimized reward critic network, overflow cost critic network and waiting time critic network.

[0018] Compared with the prior art, the application provides a reinforcement learning traffic signal control method based on safety awareness aggregation, which integrates safety into three aspects of environmental interaction, state representation and policy learning. First, considering two risk factors of overflow and long waiting time, a safety reward function and a cost function are established, and a safety representation method is proposed to better extract safety awareness representation, and then safety constraints and safety state representation are input into a safety reinforcement learning method to realize the output of safe actions. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0020] Figure 1 is a flowchart of a reinforcement learning traffic signal control method based on safety awareness aggregation provided by the present embodiment;

[0021] Figure 2 is a flowchart of the training model process provided by the present embodiment;

[0022] Figure 3 is a flowchart of the process of updating the first module provided by the present embodiment;

[0023] Figure 4 is a flowchart of the process of updating the second module provided by the present embodiment;

[0024] Figure 5 is a schematic diagram of the intersection moving direction and action space provided by the present embodiment. DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0026] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0027] The embodiment provides a reinforcement learning traffic signal control method based on safety awareness aggregation, and a flowchart thereof is shown in Figure 1 , which comprises the following steps.

[0028] S1, model construction.

[0029] The model comprises a first module and a second module, the first module comprises a state encoder, an action-state encoder, an overflow encoder and a waiting encoder, and the second module comprises an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network.

[0030] S2, model training.

[0031] As shown in Figure 2 , in this step, the following steps are included.

[0032] S200, obtaining historical state information.

[0033] The historical state information comprises first observation information, second observation information, an action, a reward value, an overflow cost value and a waiting cost value, the time points corresponding to the first observation information and the second observation information are adjacent time points, the reward value is calculated by a reward function, the reward function is constructed based on a lane density, a queue length, a waiting time, a waiting tolerance time and a number of waiting vehicles, the overflow cost value is calculated by an overflow cost function constructed based on the lane density and a preset value, and the waiting cost value is calculated by a waiting time cost function constructed based on the waiting time and the waiting tolerance time.

[0034] As shown in Figure 5 , for a cross intersection i, the four entrance lanes of east, west, north and south respectively have lanes of three different directions of left turn, straight and right turn, and no control is performed on the right turn vehicles. The embodiment has eight moving directions l to be controlled, which are north straight, north left turn, south straight, south left turn, east straight, east left turn, east straight and east left turn, and the corresponding identification numbers are 1, 2, 3,..., 8 respectively.

[0035] Each intersection is treated as a separate safety-reinforced learning agent. For each agent, the interaction with the environment is modeled as a restricted Markov decision process, which mainly includes the observation space, the action space, the reward function, the cost function, and the discount factor, considering safety factors.

[0036] The observation space includes three parts: traffic features, motion features, and traffic signal features. At intersection i at time t, traffic features are used to count the overall information of each moving direction l, including the queue length average traffic flow and the maximum occupancy Motion features provide specific details related to the movement itself of each moving direction l, mainly used to indicate whether the direction is a straight direction Traffic signal features represent the state of the signal light corresponding to each moving direction l, including an indicator of whether the current moving direction is green an indicator of whether the green light duration of the current moving direction reaches the minimum green time Specifically, at time t, the state of each moving direction l of intersection i is represented as as follows:

[0037]

[0038] wherein all take values of 0 or 1. When taking the value of 0, indicates that at time t, the moving direction l of intersection i is a non-straight direction, indicates that at time t, the signal light of moving direction l of intersection i is yellow or red, indicates that at time t, the green light duration of moving direction l of intersection i is less than or equal to the minimum green time. When taking the value of 1, indicates that at time t, the moving direction l of intersection i is a straight direction, indicates that at time t, the signal light of moving direction l of intersection i is green, indicates that at time t, the green light duration of moving direction l of intersection i is greater than the minimum green time.

[0039] At time t, the first observation information of intersection i is as follows:

[0040]

[0041] wherein respectively represent the state of the north straight, north left turn, south straight, south left turn, east straight, east left turn, east straight and east left turn of intersection i at time t.

[0042] Similarly, the second observation information of intersection i is

[0043] The action space is set to several stages, each stage allows a specific set of movement direction combinations, while limiting the movement of other movement directions. In this embodiment, 4 stages are set, allowing 2 movement direction combinations to move, specifically, the action of intersection i at time t is defined as As follows:

[0044]

[0045] The reward function is to measure the correctness of the current decision according to the set target before taking action. In this embodiment, the reward function considers the overflow and single vehicle waiting time safety factors, and the reward function of intersection i at time t is defined as As follows:

[0046]

[0047] wherein, represents the reward value of intersection i at time t, and l represents the movement direction, represents the queue length of intersection i in movement direction l at time t, represents the maximum queue length of intersection i in movement direction l at time t, n i，l represents the number of waiting vehicles of intersection i in movement direction l, j represents the identification number of the vehicle, and k j，t+1 represents the waiting time of vehicle j at time t+1, and C represents the vehicle waiting tolerance time, which is a positive integer artificially set; l out represents the exit set of intersection i in movement direction l.

[0048] The cost function is used to measure the safety of the current state and action, to illustrate the safety degree of the current decision. Similar to the reward function, in this embodiment, the cost function considers two safety factors of overflow and single vehicle waiting time, and the overflow cost function and the waiting time cost function The specific formula is as follows:

[0049]

[0050]

[0051] wherein, represents the overflow value, ratio represents the preset value, which is a parameter artificially set, and the value range is [0, 1), represents the waiting value.

[0052] The discount factor γ is used not only for calculating the long-term cumulative reward, but also for calculating the long-term cumulative cost, and the value range is [0, 1].

[0053] S210 updates the first module.

[0054] The first module includes a state encoder, an action-state encoder, an overflow encoder and a waiting encoder.

[0055] In order to better capture the features related to safety awareness, the embodiment proposes a safety representation method based on state-action encoding, which mainly enhances the safety understanding ability of the current state and action by learning the state and state-action embedding.

[0056] As shown in Figure 3 , in this step, it includes:

[0057] Step one: using the state encoder to process the first observation information and the second observation information respectively to obtain the first feature and the second feature.

[0058] For the intersection i, the state encoder f i is used to encode the first observation information to obtain the encoded state embedding, that is, the first feature z i，t，s , and its expression is The state encoder of the embodiment can be composed of a multilayer perception mechanism in a neural network.

[0059] In order to make the state encoder have state prediction ability, the second observation information is input to the state encoder f’ to obtain the future state embedding, that is, the second feature z i，t+1，s , and its expression is Where f i ′ shares network parameters with f i , and the network parameters are not learned.

[0060] Step two: normalizing the first feature and the second feature respectively to obtain the third feature and the fourth feature.

[0061] In order to deal with the instability risk caused by distance minimization in the embedding space, a normalization layer AvgL1Norm is introduced after the state encoder, and the first feature z i，t，s is input into the normalization layer to obtain the third feature z′ i，t，s , and its expression is z′ i，t，s = AvgL1Norm(zi，t，s ), wherein, represents the k-th dimension of the first feature of the N-dimensional vector, and N represents the number of dimensions of the first feature.

[0062] Similarly, the second feature z i，t+1，s is input into the normalization layer to obtain the fourth feature z' i，t+1，s .

[0063] Step three: using the action-state encoder to process the action and the third feature to obtain the fifth feature.

[0064] Using the state-action encoder g i to encode the third feature z' i，t，s and the action , the state-action embedding, i.e., the fourth feature z i，t，sa , is obtained, and its expression is The state-action encoder can be composed of multiple layers of perception in the neural network.

[0065] Step four: using the overflow encoder to encode the third feature and the fourth feature respectively to obtain the first overflow prediction score and the second overflow prediction score.

[0066] In order to further enhance the safety representation awareness of the agent, the feasibility score is introduced as part of the state representation learning. The feasibility score is associated with the Hamilton-Jacobi accessibility in safety theory, which is mainly used to represent the safety degree of the state, so as to improve the safety representation performance.

[0067] On the basis of the prediction representation, the additional supervision signal extracts the prediction feasibility score of the representation safety degree. Specifically, using the overflow encoder p i，q to encode the third feature z' i，t，s to obtain the first overflow prediction score z i，t，q , and its expression is: z i，t，q =p i，q (z′ i，t，s ), wherein p i，q represents the overflow encoder of the intersection i.

[0068] Similarly, using the overflow encoder to encode the fourth feature obtains the second overflow prediction score z i，t+1，q .

[0069] Step five: using the waiting encoder to encode the third feature and the fourth feature respectively to obtain the first waiting prediction score and the second waiting prediction score.

[0070] Using the waiting encoder p i，w to encode the third feature z'i，t，s The first waiting prediction score z i，t，w is obtained by encoding the first feature. i，t，w = p i，w (z i，t，s ), where p i，w is the waiting encoder of the intersection i.

[0071] Similarly, the second waiting prediction score z i，t+1，w is obtained by encoding the fourth feature using the waiting encoder.

[0072] The spill encoder and the waiting encoder can be composed of multiple layers of perceptrons in a neural network.

[0073] Step six: based on the second feature, the fifth feature, the spill surrogate value, the first spill prediction score, the second spill prediction score, the waiting surrogate value, the first waiting prediction score, and the second waiting prediction score, a first objective function is constructed, and the first objective function is minimized to update each encoder.

[0074] The second feature is taken as the true value, and the average error between the second feature z i，t+1，s and the fifth feature z i，t，sa is measured by the mean square error, so as to obtain the prediction loss function of the intersection i to evaluate the accuracy of the prediction:

[0075]

[0076] wherein, represents the prediction loss value.

[0077] Based on the spill surrogate value, the first spill prediction score, and the second spill prediction score, a guided estimation loss function of the spill prediction score is constructed, and the expression is:

[0078]

[0079]

[0080] wherein, represents the guided estimation value of the spill prediction score, D kl (.) represents the KL divergence, z i，t，q represents the first spill prediction score, represents the spill score estimation guide of the intersection i, max(.) represents a function of taking the maximum value, represents the spill surrogate value, and γ represents the discount factor, z i，t+1，q represents the second spill prediction score.

[0081] Based on the waiting substitute value, the first waiting prediction score and the second waiting prediction score, a guiding estimation loss function of the waiting prediction score is constructed, and the expression is:

[0082]

[0083]

[0084] wherein, represents the guiding estimation value of the waiting prediction score, z i，t，w represents the first waiting prediction score, represents the waiting score estimation guide of the intersection i, represents the waiting substitute value, z i，t+1，w represents the second waiting prediction score.

[0085] Based on the prediction loss function, the guiding estimation loss function of the overflow prediction score and the guiding estimation loss function of the waiting prediction score, a first objective function is constructed, and the expression is:

[0086]

[0087] wherein, represents the first target value, λ i，q represents the first hyperparameter, λ i，w represents the second hyperparameter.

[0088] By minimizing the first objective function, the state encoder, the state-action encoder, the overflow encoder and the waiting encoder are updated by using the Adam gradient descent method.

[0089] S220 updates the second module.

[0090] The second module includes an actor network, a reward critic network, an overflow cost critic network and a waiting time critic network.

[0091] In order to make the intelligent agent under the joint constraint of excessive waiting time and overflow, the actor network integrates the safety representation, takes the first observation information and the first feature as input to evaluate the feasibility of the state, and outputs the safe phase action, so as to ensure the short-term safety of the traffic signal control. The critic network includes a reward critic network, an overflow cost critic network and a waiting time critic network, which respectively take the first feature and the second feature as input to estimate the expected reward and the expected cost of the current state, so as to provide the long-term safety reward maximization of the traffic signal control.

[0092] As Figure 4 shown, in this step, it includes:

[0093] Step one: input the first observation information and the first feature into the actor network to obtain a first probability value.

[0094] For the intersection i, input the first observation information and the first feature z i，t，s into the actor network to calculate the probability value of different actions. The actor network is composed of 3 linear layers and 2 activation function layers. The specific calculation formula is as follows:

[0095]

[0096] wherein, represents the first probability value, θ i is the parameter of the intersection i actor network, softmax(.) represents the normalization function in the neural network, FC(.) represents the linear layer in the neural network, and ReLU(.) represents the activation function in the neural network.

[0097] Step two: input the first feature and the second feature into the reward critic network, the overflow cost critic network and the waiting time critic network respectively to obtain the corresponding reward value, overflow cost value and waiting cost value.

[0098] The structures of the reward critic network, the overflow cost critic network and the waiting time critic network are consistent with the structure of the actor network.

[0099] The reward value includes the first reward value corresponding to the first feature and the second reward value corresponding to the second feature, the overflow cost value includes the first overflow cost value corresponding to the first feature and the second overflow cost value corresponding to the second feature, and the waiting cost value includes the first waiting cost value corresponding to the first feature and the second waiting cost value corresponding to the second feature.

[0100] Input the first feature z i，t，s into the reward critic network to obtain the first reward value The specific calculation formula is as follows:

[0101]

[0102] Similarly, input the second feature z i，t+1，s into the reward critic network to obtain the second reward value

[0103] Input the first feature z i，t，s into the overflow cost critic network to obtain the first overflow cost value The specific calculation formula is as follows:

[0104]

[0105] The second feature z i，t+1，s is input into the overflow cost critic network to obtain the second overflow cost value

[0106] The first feature z i，t，s is input into the waiting time critic network to obtain the first waiting cost value The specific calculation formula is as follows:

[0107]

[0108] The second feature z i，t+1，s is input into the waiting time critic network to obtain the second waiting cost value

[0109] Step three: input the reward value and the reward value into the reward advantage evaluation function to obtain the reward advantage evaluation value.

[0110] This step inputs the reward value the first reward value and the second reward value into the following formula to calculate the reward advantage evaluation value of the intersection i

[0111]

[0112]

[0113] wherein, represents the reward advantage estimation of the kth stage of the intersection i at the t time, k represents the stage of the advantage evaluation, and m represents the mth stage in the advantage evaluation process, represents the reward value of the intersection i at the t time, represents the second observation information corresponding to the second reward value, represents the first observation information corresponding to the first reward value, represents the action output based on the first observation information corresponding to the reward advantage evaluation value, and λ GAE represents the advantage value parameter, which is a parameter for controlling the average degree of the advantage value, and Y represents the length of the sampling trajectory.

[0114] Step four: input the spill cost value and the spill cost value into the spill cost advantage evaluation function to obtain the spill cost evaluation value.

[0115] This step inputs the spill cost value the first spill cost value and the second spill cost value into the following formula to calculate the spill cost evaluation value of intersection i

[0116]

[0117]

[0118] wherein, represents the spill cost estimation of the kth stage of intersection i at time t, represents the spill cost value of intersection i at time t, represents the second observation information corresponding to the second spill cost value, represents the first observation information corresponding to the first spill cost value, represents the action output based on the first observation information corresponding to the spill cost evaluation value.

[0119] Step five: input the waiting cost value and the waiting cost value into the waiting cost advantage evaluation function to obtain the waiting cost evaluation value.

[0120] This step inputs the waiting cost value the first waiting cost value and the second waiting cost value into the following formula to calculate the waiting cost evaluation value of intersection i

[0121]

[0122]

[0123] wherein, represents the waiting cost estimation of the kth stage of intersection i at time t, represents the waiting cost value of intersection i at time t, represents the second observation information corresponding to the second waiting cost value, represents the first observation information corresponding to the first waiting cost value, represents the action output based on the first observation information the corresponding waiting cost evaluation value.

[0124] Step six: optimizing the actor network based on the reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value.

[0125] The reward advantage evaluation value, the overflow cost evaluation value, the waiting value evaluation value and the first probability value are input into a second objective function, and the optimized actor network is obtained by minimizing the second objective function, and the expression of the second objective function is:

[0126]

[0127]

[0128] wherein, represents the Lagrange multiplier of the intersection i with respect to the waiting time cost function, represents the Lagrange multiplier of the intersection i with respect to the overflow cost function, represents the value of the second objective function, represents the expected experience return value under the actor policy network, min(.) represents a function of taking the minimum value, represents the probability value of the action output by the actor policy network according to the first observation information in the current iteration process, represents the probability value of the action output by the actor policy network according to the first observation information in the last stage of the current iteration process, represents the limit is within the range of [1-ε, 1+ε], if greater than 1+ε, the value is 1+ε; if less than 1-ε, the value is 1-ε; otherwise, it remains unchanged; represents the comprehensive evaluation value.

[0129] According to the above update the parameters θ i of the actor network, the Lagrange multiplier of the waiting time cost function and the Lagrange multiplier of the overflow cost function are shown in the specific calculation formula:

[0130]

[0131]

[0132]

[0133] wherein, θ i ​​θ represents the parameters of the updated actor network. i The parameters representing the unupdated actor network, This represents the compensation for gradient descent. L(θ) i , λ i ) for θ i Differentiate, λ i ' represents the updated Lagrange multiplier, λ i This indicates that the Lagrange multipliers have not been updated. This represents the step size of the gradient ascent. L(θ) i , λ i ) for λ i Find the derivative.

[0134] This embodiment utilizes the Adam gradient descent algorithm to update the actor network.

[0135] Step 7: Optimize the reward critic network based on the reward value and the first reward value.

[0136] The reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network.

[0137] Loss function L of reward critic network r (ω i，r The formula for ) is:

[0138]

[0139] By minimizing the loss function L r (ω i，r The parameters ω of the reward critic network are updated using Adam gradient descent. i，r .

[0140] Step 8: Optimize the overflow cost critic network based on the overflow cost value and the first overflow cost value.

[0141] The overflow cost value and the first overflow cost value are input into the loss function of the overflow cost critic network and optimized to obtain the optimized overflow cost critic network.

[0142] The loss function L of the overflow cost critic network q (ω i，q The formula for ) is:

[0143]

[0144] By minimizing the loss function L q (ω i，q), and the parameters ω of the overflow cost critic network are updated by using the Adam gradient descent method i，q .

[0145] Step nine: optimizing the waiting time critic network based on the waiting cost value and the first waiting cost value.

[0146] The waiting cost value and the first waiting cost value are input into the loss function of the waiting time critic network, and the optimized waiting time critic network is obtained by optimization.

[0147] The formula of the loss function L w (ω i，w ) of the waiting time critic network is:

[0148]

[0149] By minimizing the loss function L w (ω i，w ), and the parameters ω of the waiting time critic network are updated by using the Adam gradient descent method i，w .

[0150] S230 repeats S200 to S220 until a preset number of times is exceeded, and a trained model is obtained.

[0151] S3 obtains observation information at the current time, inputs the observation information at the current time into the trained actor network, and obtains a current action to control the traffic signal.

[0152] Further, the method further comprises:

[0153] The current action is executed, and next observation information is obtained.

[0154] The current reward value is calculated according to the reward function.

[0155] The current overflow cost value is calculated according to the overflow cost function.

[0156] The current waiting cost value is calculated according to the waiting time cost function.

[0157] The current state information is constructed based on the observation information at the current time, the observation information at the next time, the current action, the current reward value, the current overflow cost value and the current waiting cost value.

[0158] The first module and the second module are updated based on the current state information, and the updated first module and the second module are used to process the observation information at the next time.

[0159] The embodiment proposes a reinforcement learning traffic signal control method based on safety awareness aggregation, which integrates safety into three aspects of environment interaction, state representation and policy learning. Firstly, considering two risk factors of overflow and too long waiting time, a safety reward function and a cost function are established, and a safety representation method is proposed to better extract safety awareness representation, and then safety constraints and safety state representation are input into the safety reinforcement learning method to realize the output of safety action.

[0160] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A reinforcement learning-based traffic signal control method based on safety awareness aggregation, characterized in that, include: Construct a model, which includes a first module and a second module; Training the model includes: Step 1: Obtain historical state information, which includes first observation information, second observation information, action, reward value, overflow cost value, and waiting cost value. The times corresponding to the first observation information and the second observation information are adjacent and sequential times. Step 2: Update the first module, which includes a state encoder, an action-state encoder, an overflow encoder, and a wait encoder. This includes: processing the first and second observation information using the state encoder to obtain a first feature and a second feature, respectively; normalizing the first and second features to obtain a third feature and a fourth feature; processing the action and the third feature using the action-state encoder to obtain a fifth feature; encoding the third and fourth features using the overflow encoder to obtain a first overflow prediction score and a second overflow prediction score; encoding the third and fourth features using the wait encoder to obtain a first wait prediction score and a second wait prediction score; constructing a first objective function based on the second feature, the fifth feature, the overflow cost, the first overflow prediction score, the second overflow prediction score, the wait cost, the first wait prediction score, and the second wait prediction score; and minimizing the first objective function to update each encoder. Step 3: Update the second module, which includes an actor network, a reward critic network, an overflow cost critic network, and a waiting time critic network. This includes: inputting first observation information and a first feature into the actor network to obtain a first probability value; inputting the first feature and a second feature into the reward critic network, the overflow cost critic network, and the waiting time critic network, respectively, to obtain corresponding reward values, overflow cost values, and waiting cost values. The reward value includes the first reward value corresponding to the first feature, the overflow cost value includes the first overflow cost value corresponding to the first feature, and the waiting cost value includes the first waiting cost value corresponding to the first feature. Value; The reward value and reward advantage are input into the reward advantage evaluation function to obtain the reward advantage evaluation value; the spillover cost value and spillover expense value are input into the spillover expense advantage evaluation function to obtain the spillover expense evaluation value; the waiting cost value and waiting expense value are input into the waiting expense advantage evaluation function to obtain the waiting expense evaluation value; the actor network is optimized based on the reward advantage evaluation value, spillover expense evaluation value, waiting expense evaluation value, and first probability value; the reward critic network is optimized based on the reward value and first reward value; the spillover expense critic network is optimized based on the spillover cost value and first spillover expense value; the waiting time critic network is optimized based on the waiting cost value and first waiting expense value. Step 4: Repeat steps 1 to 3 until the preset number of times is exceeded to obtain the trained model; Obtain the observation information at the current moment, input the observation information at the current moment into the trained actor network, and obtain the current action to control the traffic signal; The observed information includes queue length, average traffic flow, maximum occupancy, whether the current direction is a straight-ahead direction, whether the current direction of movement has a green light, and whether the current direction of movement has reached the minimum green light time. The reward value is calculated by a reward function based on queue length, waiting time, waiting tolerance time, and number of waiting vehicles. The overflow cost is calculated by an overflow cost function based on a preset value. The waiting cost is calculated by a waiting time cost function based on waiting time and waiting tolerance time.

2. The traffic signal control method based on reinforcement learning and safety awareness aggregation as described in claim 1, characterized in that, The expression for the first objective function is: in, z represents the predicted loss value. i,t,sa The fifth feature, z i，t+1，s Indicates the second characteristic, D represents the guided estimate of the overflow prediction score. kl (.) represents the KL divergence, z i，t，q This indicates the first overflow prediction score. This indicates the overflow score estimation guide for intersection i. γ represents the overflow cost, z represents the discount factor, and z represents the overflow cost. i，t+1，q This indicates the second overflow prediction score. z represents the guided estimate of the expected predicted score. i，t，w This indicates the first waiting prediction score. This indicates the waiting score estimation guidance at intersection i. z represents the value of waiting. i，t+1，w This indicates the second waiting prediction score. Let λ represent the first objective value. i，q Let λ represent the first hyperparameter. i，w This represents the second hyperparameter.

3. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 1, characterized in that, Optimizing the actor network based on the reward advantage evaluation value, overflow cost evaluation value, waiting cost evaluation value, and first probability value includes: inputting the reward advantage evaluation value, overflow cost evaluation value, waiting cost evaluation value, and first probability value into a second objective function, and obtaining the optimized actor network by minimizing the second objective function; Optimizing the reward critic network based on the reward value and the first reward value includes: inputting the reward value and the first reward value into the loss function of the reward critic network and optimizing it to obtain the optimized reward critic network; Optimizing the overflow cost critic network based on the overflow cost value and the first overflow cost value includes: inputting the overflow cost value and the first overflow cost value into the loss function of the overflow cost critic network and optimizing it to obtain the optimized overflow cost critic network; Optimizing the waiting time critic network based on the waiting cost value and the first waiting cost value includes: inputting the waiting cost value and the first waiting cost value into the loss function of the waiting time critic network and optimizing it to obtain the optimized waiting time critic network.

4. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 1, characterized in that, The expression for the reward function is: in, Let represent the reward value at intersection i at time t, and l represent the direction of movement. This represents the queue length at intersection i in the direction of movement l at time t. Let n represent the maximum queue length at intersection i in the direction of movement l at time t. i，l Let j represent the number of vehicles waiting at intersection i in the direction of movement l, j represent the vehicle identification number, and k represent the number of vehicles waiting in the direction of movement l. j，t+1 Let l represent the waiting time of vehicle j at time t+1, C represent the vehicle's waiting tolerance time, and l represent the waiting time of vehicle j at time t+1. out Let i represent the set of exit points of intersection i in the direction of movement l.

5. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 4, characterized in that, The expression for the overflow cost function is: in, The overflow cost is represented by ratio, which represents the preset value. The expression for the waiting time cost function is: in, It indicates waiting for the value.

6. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 1, characterized in that, The expression for the reward advantage evaluation function is: in, Let represent the reward advantage assessment value of intersection i at time t, k represent the stage of advantage assessment, γ represent the discount factor, and m represent the m-th stage in the advantage assessment process. This represents the reward value at intersection i at time t. Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates based on the first observation information Output action The corresponding reward advantage assessment value, λ GAE Y represents the dominance parameter, which controls the average degree of dominance.

7. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 3, characterized in that, The expression for the second objective function is: in, Let i be the Lagrange multiplier of the waiting time cost function of intersection i. Let i be the Lagrange multiplier of the overflow cost function. This represents the value of the second objective function. This represents the expected empirical reward value in an actor network. `min(.)` denotes the function that minimizes this value, and `max(.)` denotes the function that maximizes it. This indicates that during the current iteration, the actor network is based on the first observation information. Output action The probability value, This indicates that in the current iteration, the actor network from the previous stage, based on the first observation information... Output action The probability value, Indicates restriction Within the range [1-ε, 1+ε], if the value is greater than 1+ε, then the value is 1+ε; if the value is less than 1-ε, then the value is 1-ε; otherwise, the value remains unchanged. This represents the overall evaluation value. Indicates based on the first observation information Output action The corresponding reward advantage assessment value, Indicates based on the first observation information Output action The corresponding waiting cost assessment value, Indicates based on the first observation information Output action The corresponding overflow cost assessment value.

8. The traffic signal control method based on reinforcement learning and safety awareness aggregation according to claim 1, characterized in that, Also includes: The loss functions of the reward critic network, overflow cost critic network, and waiting time critic network are constructed using the mean squared error. The loss functions of each network are minimized by the Adam gradient descent algorithm, and the parameters of each network are updated to obtain the optimized reward critic network, overflow cost critic network, and waiting time critic network.

Images