Traffic signal control method and system based on safety reinforcement learning

Abstract

The invention discloses a traffic signal control method and system based on safety reinforcement learning, and the method comprises the steps: obtaining a plurality of pieces of historical state information, a first probability value, a first reward value, a second reward value, a first cost value and a second cost value are obtained through processing by utilizing an attention network, an actor network, a reward approver network and a cost approver network, and then an advantage evaluation value and a cost evaluation value are obtained through a reward advantage evaluation function and a cost advantage evaluation function; optimizing the actor network, the reward approver network and the cost approver network based on the advantage evaluation value, the cost evaluation value, the first probability value, the reward value, the first reward value, the cost value and the first cost value, and repeating the above process until a preset number of times is exceeded to obtain a trained actor network; and obtaining observation information at the current moment, and inputting the observation information into the trained actor network to obtain a current action so as to control a traffic signal. According to the invention, the safety of the intersection is ensured under the condition of ensuring efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic control, and more particularly to a traffic signal control method and system based on safety reinforcement learning. Background Art

[0002] In recent years, traffic congestion and traffic safety issues in urban areas have become increasingly serious, especially at intersections in urban areas, where traffic accidents frequently occur. Therefore, how to safely and efficiently optimize traffic lights has become an important part of urban traffic management.

[0003] Reinforcement learning has become one of the main methods for optimizing traffic signal control due to its rapid adaptability and efficient decision-making capabilities. In reinforcement learning, the agent can continuously improve its decision-making ability by trying different actions and receiving feedback from the environment, and according to a pre-designed reward function, thereby obtaining a traffic signal control solution that adapts to real traffic needs.

[0004] However, most existing studies focus on how to minimize traffic delays, but ignore potential unsafe conditions. In traffic signal control systems, the overflow length of red lights is an important factor affecting safety. If the traffic flow at an intersection is unbalanced, busy lanes often get unlimited priority to obtain maximum rewards, while idle lanes experience unbearable waiting times, which may cause drivers to run red lights illegally.

[0005] Therefore, there is an urgent need for a traffic signal control method to balance traffic efficiency and safety. Summary of the invention

[0006] The present invention is proposed based on the above-mentioned needs of the prior art. The technical problem to be solved by the present invention is a traffic signal control method and system based on safety reinforcement learning to introduce overflow safety factors into traffic signal control, giving priority to lanes that are about to overflow or have already overflowed, thereby ensuring the safety of traffic intersections.

[0007] In order to solve the above problems, the present invention is implemented by adopting the following technical solutions:

[0008] A traffic signal control method based on safe reinforcement learning, the method comprising: obtaining multiple historical state information, each historical state information comprising first observation information, second observation information, action, reward value and cost value; the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by an overflow reward function, the overflow reward function is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, the overflow cost function is constructed by the lane density and a preset maximum overflow ratio; the first observation information and the second observation information of each historical state information are respectively input into an attention network to obtain a first feature and a second feature; the first feature is input into an actor network to obtain a first probability value; the first feature and the second feature are respectively input into a reward critic network to obtain a first reward value and a second reward value; the first feature and the second feature are respectively input into a cost critic network The first cost value and the second cost value are obtained in the network; the reward value, the first reward value and the second reward value are input into the reward advantage evaluation function to obtain the advantage evaluation value; the cost value, the first cost value and the second cost value are input into the cost advantage evaluation function to obtain the cost evaluation value; the advantage evaluation value, the cost evaluation value and the first probability value are input into the objective function and optimized to obtain the optimized actor network; the reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network; the cost value and the first cost value are input into the loss function of the cost critic network and optimized to obtain the optimized cost critic network; based on the optimized actor network, the reward critic network and the cost critic network, the above process is repeated until the preset number of times is exceeded to obtain the trained actor network; the observation information at the current moment is obtained, and input into the trained actor network to obtain the current action to control the traffic signal.

[0009] Optionally, it also includes: executing the current action to obtain the observation information of the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing the current state information based on the observation information of the current moment, the observation information of the next moment, the current action, the current reward value and the current cost value; re-optimizing the actor network, the reward critic network and the cost critic network based on the current state information, and using the re-optimized actor network to process the observation information of the next moment.

[0010] Optionally, the method further includes: constructing loss functions of the reward critic network and the cost critic network using the mean square error, minimizing the loss function using the Adam gradient descent algorithm, updating the parameters of the reward critic network and the parameters of the cost critic network, and obtaining optimized reward critic network and cost critic network.

[0011] Optionally, the overflow reward function is expressed as: in, represents the overflow reward value of intersection i at time t, i represents the identification number of the intersection, t represents the time, l represents the identification number of the lane, represents the set of entrance roads at intersection i, It represents the queue length of lane l of intersection i at time t.

[0012] Optionally, the overflow cost function is expressed as: in, represents the overflow cost, represents the lane density of lane l of intersection i at time t, ratio w Indicates the preset maximum overflow ratio. represents the queue length of lane l of intersection i at time t, It represents the maximum queue length of lane l of intersection i at time t.

[0013] Optionally, the reward advantage evaluation function is expressed as: in, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the overflow reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, a i,t represents the action of intersection i at time t, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

[0015] Optionally, the cost advantage evaluation function is expressed as:

[0016] in, represents the cost advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the overflow cost of intersection i at time t, Indicates the second observation information The corresponding second cost value, Indicates the first observation information The corresponding first cost value, Indicates that based on the first observation information s i,t Output action a i,t The corresponding cost evaluation value, a i,t represents the action of intersection i at time t, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

[0017] Optionally, the objective function is expressed as: Among them, L(θ i ,λ i ) represents the target value, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, and max(.) represents the function of taking the maximum value. Indicates that the actor strategy network in the current iteration process is based on the first observation information Output action information The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output action information The probability value of Restrictions In the range of [1-ε,1+ε], if it is greater than 1+ε, it takes the value of 1+ε; if it is less than 1-ε, it takes the value of 1-ε; otherwise, it remains unchanged; Represents the comprehensive evaluation value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, λ i represents the Lagrange multiplier of intersection i, Indicates that based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0018] A traffic signal control system based on safe reinforcement learning, the system includes: a training module, which obtains multiple historical state information, each historical state information includes first observation information, second observation information, action, reward value and cost value; the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by an overflow reward function, and the overflow reward function is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, and the overflow cost function is constructed by the lane density and a preset maximum overflow ratio; the first observation information and the second observation information of each historical state information are respectively input into the attention network to obtain the first feature and the second feature; the first feature is input into the actor network to obtain the first probability value; the first feature and the second feature are respectively input into the reward critic network to obtain the first reward value and the second reward value; the first feature and the second feature are respectively input into the cost critic network The first cost value and the second cost value are obtained in the network; the reward value, the first reward value and the second reward value are input into the reward advantage evaluation function to obtain the advantage evaluation value; the cost value, the first cost value and the second cost value are input into the cost advantage evaluation function to obtain the cost evaluation value; the advantage evaluation value, the cost evaluation value and the first probability value are input into the objective function and optimized to obtain the optimized actor network; the reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network; the cost value and the first cost value are input into the loss function of the cost critic network and optimized to obtain the optimized cost critic network; based on the optimized actor network, the reward critic network and the cost critic network, the above process is repeated until the preset number of times is exceeded to obtain the trained actor network; the execution module obtains the observation information at the current moment, and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0019] Optionally, the training module also includes: executing the current action to obtain the observation information of the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing the current state information based on the observation information of the current moment, the observation information of the next moment, the current action, the current reward value and the current cost value; re-optimizing the actor network, the reward critic network and the cost critic network based on the current state information, and using the re-optimized actor network to process the observation information of the next moment.

[0020] Compared with the prior art, the present invention provides a traffic signal control method and system based on safety reinforcement learning, which introduces overflow safety factors into traffic signal control, considers giving priority to lanes that are about to overflow or have already overflowed, and models the traffic signal control problem into a safety optimization objective function, thereby ensuring the safety of traffic intersections while ensuring efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1 is a flow chart of a traffic signal control method based on safety reinforcement learning provided in this embodiment;

[0023] Figure 2 This is the security agent learning architecture provided by this embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] To facilitate understanding of the embodiments of the present invention, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the protection scope of the present invention.

[0026] Example 1

[0027] This embodiment provides a traffic signal control method based on safety reinforcement learning, and its process is as follows: Figure 1 As shown, including:

[0028] S1 obtains multiple historical status information.

[0029] This embodiment can randomly extract multiple historical state information from the playback buffer pool, each historical state information includes first observation information, second observation information, action, reward value and cost value; the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by the overflow reward function, and the overflow reward function is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by the overflow cost function, and the overflow cost function is constructed by the lane density and the preset maximum overflow ratio.

[0030] like Figure 2As shown, the observed information includes the queue length of each lane, vehicle density, current signal phase, and whether the current phase reaches the maximum green light time.

[0031] Each intersection i is defined as a separate safety agent. The observation information is used to describe the local information of the traffic signal safety agent, including signal light information, lane information, and vehicle information. This embodiment considers the situation of a general intersection, that is, the intersection has four entrances, namely, east, west, south, and north. Each entrance has three different lanes, namely, left turn, straight, and right turn. The observation information constructed based on the general intersection situation includes queue length, lane density, current signal phase, and whether the current phase reaches the maximum green light time. At time t, the first observation information of safety agent i is

[0032]

[0033] Where: l represents the import lane, represents the set of import lanes at intersection i, represents the queue length on lane l in intersection i at time t; represents the vehicle density of lane l in intersection i at time t; represents the current signal phase of intersection i at time t, which is a one-hot encoding of a 4-vector; Indicates whether the green light time of intersection i reaches the maximum green light time indicator at time t. Its value is 0 or 1. When the value is 0, it means that the maximum green light time has not been reached. When the value is 1, it means that the maximum green light time has been exceeded.

[0034] Similarly, the second observation information can be obtained

[0035] like Figure 2 As shown, the action space is determined according to the traffic signal phase and corresponds to the preset value one by one. The action space of this embodiment includes four different traffic movements controlled by traffic lights, namely, north-south straight, north-south left turn, east-west straight and east-west left turn. At the same time, since the right turn signal is always green, right turn is not included. Action space of intersection i for:

[0036]

[0037] The actions of this embodiment are determined based on the action space.

[0038] Construct an overflow reward function, the overflow reward function Guide the security agent to learn and make decisions towards the desired goal, and quantify the immediate reward of the security agent taking actions in a specific state. Taking the overflow factor into consideration, the overflow reward function of this embodiment is Taking queue length as target, its expression is:

[0039]

[0040] in, represents the overflow reward value of intersection i at time t, i represents the identification number of the intersection, t represents the time, l represents the identification number of the lane, represents the set of entrance roads at intersection i, It represents the queue length of lane l of intersection i at time t.

[0041] The reward value of this embodiment is calculated by the overflow reward function.

[0042] Construct an overflow cost function, the overflow cost function It is used to calculate the number of times the safety agent violates the safety factor. With the preset maximum overflow ratio, the overflow cost function is constructed, and its expression is:

[0043]

[0044] in, represents the overflow cost, represents the lane density of lane l of intersection i at time t, ratio w Indicates the preset maximum overflow ratio, and its value range is [0,1). represents the queue length of lane l of intersection i at time t, It represents the maximum queue length of lane l of intersection i at time t.

[0045] The cost value of this embodiment is calculated by using the overflow cost function.

[0046] S2 inputs the first observation information and the second observation information of each historical state information into the attention network respectively to obtain the first feature and the second feature.

[0047] The weight values ​​of the observation space are dynamically allocated through the attention mechanism. Specifically:

[0048] The observation information is embedded into a higher dimension using an embedding layer to obtain the embedding layer representation data. The embedding layer consists of a fully connected layer and an activation function. The expression is:

[0049]

[0050] In the formula: relu(.) represents the activation function of the neural network, and fc(.) represents the fully connected layer of the neural network.

[0051] The embedding layers are represented Input to three linear layers to obtain query Q, key K and value V. The specific calculation formula is as follows:

[0052]

[0053] Where: linear(.) represents the linear layer of the neural network.

[0054] The query Q, key K, and value V are input into the self-attention layer to obtain the importance representation of different embedding layers. The specific calculation formula is as follows:

[0055]

[0056] Where: represents the first feature, softmax(.) represents the normalization layer of the neural network; d k Represents the scaling factor.

[0057] Similarly, we get the second feature

[0058] S3 inputs the first feature into the actor network to obtain a first probability value.

[0059] The safety agents in this embodiment all use the proximal strategy optimization algorithm as their learners. For intersection i, the first feature Input to the actor network In order to calculate the probability values ​​of different actions The specific calculation formula is as follows:

[0060]

[0061] S4 inputs the first feature and the second feature into the reward critic network respectively to obtain a first reward value and a second reward value.

[0062] The first feature Input to the Reward-Critic Network , get the first reward value The specific calculation formula is as follows:

[0063]

[0064] Similarly, the second feature Input to the reward critic network , get the second reward value

[0065] S5 inputs the first feature and the second feature into the cost critic network respectively to obtain a first cost value and a second cost value.

[0066] The first feature Input to the cost critic network , get the first cost value The specific calculation formula is as follows:

[0067]

[0068] Similarly, the second feature Input to the cost critic network In the second cost value

[0069] S6: Input the reward value, the first reward value and the second reward value into a reward advantage evaluation function to obtain an advantage evaluation value.

[0070] This step will reward First Reward Value and the second reward value Input into the following formula to calculate the reward advantage evaluation of intersection i

[0071]

[0072] in, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the overflow reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, a i,t represents the action of intersection i at time t, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

[0073] S7 inputs the cost value, the first cost value and the second cost value into the cost advantage evaluation function to obtain a cost evaluation value.

[0074] This step will cost First price value and the second cost value Input into the following formula to calculate the cost advantage evaluation of intersection i

[0075]

[0076] in, represents the cost advantage evaluation value of intersection i at time t, represents the overflow cost of intersection i at time t, Indicates the second observation information The corresponding second cost value, Indicates the first observation information The corresponding first cost value, Indicates that based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0077] S8 inputs the advantage evaluation value, the cost evaluation value and the first probability value into the objective function and optimizes to obtain an optimized actor network.

[0078] An additional cost function is introduced, which requires maximizing the reward function while minimizing the cost function. The specific calculation formula of the security optimization objective function is as follows:

[0079]

[0080] Among them, L(θ i ,λ i ) represents the security policy loss, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, and max(.) represents the function of taking the maximum value. Indicates that the actor strategy network in the current iteration process is based on the first observation information Output action information The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output action information The probability value of Restrictions In the range of [1-ε,1+ε], if it is greater than 1+ε, it takes the value of 1+ε; if it is less than 1-ε, it takes the value of 1-ε; otherwise, it remains unchanged; Represents the comprehensive evaluation value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, λ i represents the Lagrange multiplier of intersection i, Indicates that based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0081] According to the above security strategy loss L(θ i ,λ i ), update the parameters of the actor network and the Lagrange multiplier, the specific calculation formula is as follows:

[0082]

[0083] Among them, θ i ′ represents the updated parameters of the actor network, θ i represents the parameters of the actor network that are not updated, represents the compensation of gradient descent, represents L(θ i ,λ i ) for θ i Derivative, λ i ′ represents the updated Lagrange multiplier, λ i represents the unupdated Lagrange multiplier, represents the step size of gradient ascent, represents L(θ i ,λ i ) for λ i Derivative.

[0084] S9 inputs the reward value and the first reward value into the loss function of the reward critic network and optimizes to obtain the optimized reward critic network; inputs the cost value and the first cost value into the loss function of the cost critic network and optimizes to obtain the optimized cost critic network.

[0085] The reward critic network and the cost critic network reduce the error of the value function by optimizing parameters, thereby improving the estimation of cumulative rewards and costs. This embodiment uses mean square error as the loss function of the reward critic network and the cost critic network.

[0086] The loss function L of the reward critic network r (ω i,r ) is:

[0087]

[0088] The loss function L of the cost critic network c (ω i,c ) is:

[0089]

[0090] By minimizing the loss function L r (ω i,r ) and the loss function L c (ω i,c ), and use the Adam gradient descent method to update the parameters ω of the reward critic network i,r and the parameters ω of the cost critic network i,c .

[0091] S10 repeats S1-S9 based on the optimized actor network, reward critic network and cost critic network until the preset number of times is exceeded to obtain a trained actor network.

[0092] S11 obtains the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0093] Furthermore, the method also includes: executing the current action to obtain the observation information of the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing the current state information based on the observation information of the current moment, the observation information of the next moment, the current action, the current reward value and the current cost value; re-optimizing the actor network, the reward critic network and the cost critic network based on the current state information, and using the re-optimized actor network to process the observation information of the next moment.

[0094] This embodiment provides a traffic signal control method based on safety reinforcement learning, which introduces overflow safety factors into traffic signal control, considers giving priority to lanes that are about to overflow or have already overflowed, and models the traffic signal control problem into a safety optimization objective function, thereby ensuring the safety of traffic intersections while ensuring efficiency.

[0095] Example 2

[0096] This embodiment provides a traffic signal control system based on safety reinforcement learning, which implements the traffic signal control method based on safety reinforcement learning described in Example 1, including:

[0097] A training module obtains multiple historical state information, each of which includes a first observation information, a second observation information, an action, a reward value and a cost value; the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by an overflow reward function, which is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, which is constructed by the lane density and a preset maximum overflow ratio; the first observation information and the second observation information of each historical state information are respectively input into the attention network to obtain a first feature and a second feature; the first feature is input into the actor network to obtain a first probability value; the first feature and the second feature are respectively input into the reward critic network to obtain a first reward value and a second reward value; the first feature and the second feature are classified into The reward value, the first reward value and the second reward value are respectively input into the cost critic network to obtain the first cost value and the second cost value; the reward value, the first reward value and the second reward value are input into the reward advantage evaluation function to obtain the advantage evaluation value; the cost value, the first cost value and the second cost value are input into the cost advantage evaluation function to obtain the cost evaluation value; the advantage evaluation value, the cost evaluation value and the first probability value are input into the objective function and optimized to obtain the optimized actor network; the reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network; the cost value and the first cost value are input into the loss function of the cost critic network and optimized to obtain the optimized cost critic network; based on the optimized actor network, reward critic network and cost critic network, the above process is repeated until the preset number of times is exceeded to obtain the trained actor network.

[0098] The execution module obtains the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0099] Optionally, the training module also includes: executing the current action to obtain the observation information of the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing the current state information based on the observation information of the current moment, the observation information of the next moment, the current action, the current reward value and the current cost value; re-optimizing the actor network, the reward critic network and the cost critic network based on the current state information, and using the re-optimized actor network to process the observation information of the next moment.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A traffic signal control method based on safety reinforcement learning, characterized in that: include: Step 1: Acquire multiple historical state information, each of which includes first observation information, second observation information, action, reward value and cost value; the first observation information and the second observation information correspond to adjacent moments; the reward value is calculated by an overflow reward function, which is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, which is constructed by the lane density and a preset maximum overflow ratio; Step 2: Input the first observation information and the second observation information of each historical state information into the attention network respectively to obtain the first feature and the second feature; input the first feature into the actor network to obtain the first probability value; input the first feature and the second feature into the reward critic network respectively to obtain the first reward value and the second reward value; input the first feature and the second feature into the cost critic network respectively to obtain the first cost value and the second cost value; input the reward value, the first reward value and the second reward value into the reward advantage evaluation function to obtain the advantage evaluation value; input the cost value, the first cost value and the second cost value into the cost advantage evaluation function to obtain the cost evaluation value; Step 3: Input the advantage evaluation value, the cost evaluation value and the first probability value into the objective function and optimize to obtain the optimized actor network; input the reward value and the first reward value into the loss function of the reward critic network and optimize to obtain the optimized reward critic network; input the cost value and the first cost value into the loss function of the cost critic network and optimize to obtain the optimized cost critic network; Step 4: Based on the optimized actor network, reward critic network and cost critic network, repeat steps 1 to 3 until the preset number of times is exceeded to obtain the trained actor network; Step 5: Obtain the observation information at the current moment and input it into the trained actor network to obtain the current action to control the traffic signal.

2. A traffic signal control method based on safety reinforcement learning according to claim 1, characterized in that: Also includes: Execute the current action and obtain the observation information of the next moment; The current reward value is calculated according to the overflow reward function; The current cost value is calculated according to the overflow cost function; Construct the current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current cost value; Based on the current state information, the actor network, reward critic network and cost critic network are optimized again, and the observation information at the next moment is processed using the re-optimized actor network.

3. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: Also includes: The mean square error is used to construct the loss function of the reward critic network and the cost critic network. The loss function is minimized by the Adam gradient descent algorithm, and the parameters of the reward critic network and the cost critic network are updated to obtain the optimized reward critic network and the cost critic network.

4. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: The expression of the overflow reward function is: in, represents the overflow reward value of intersection i at time t, i represents the identification number of the intersection, t represents the time, l represents the identification number of the lane, represents the set of entrance roads at intersection i, It represents the queue length of lane l of intersection i at time t.

5. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: The expression of the overflow cost function is: in, represents the overflow cost, represents the lane density of lane l of intersection i at time t, ratio w Indicates the preset maximum overflow ratio. represents the queue length of lane l of intersection i at time t, It represents the maximum queue length of lane l of intersection i at time t.

6. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: The expression of the reward advantage evaluation function is: in, represents the reward advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the overflow reward value of intersection i at time t, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, a i,t represents the action of intersection i at time t, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

7. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: The expression of the cost advantage evaluation function is: in, represents the cost advantage evaluation value of intersection i at time t, k represents the stage of advantage evaluation, γ represents the discount factor, m represents the mth stage in the advantage evaluation process, represents the overflow cost of intersection i at time t, Indicates the second observation information The corresponding second cost value, Indicates the first observation information The corresponding first cost value, Indicates that based on the first observation information s i,t Output action a i,t The corresponding cost evaluation value, a i,t represents the action of intersection i at time t, λ GAE represents the advantage value parameter, which is used to control the degree of averaging of the advantage value, and Y represents the length of the sampling trajectory.

8. The traffic signal control method based on safety reinforcement learning according to claim 1 is characterized in that: The expression of the objective function is: Among them, L(θ i ,λ i ) represents the target value, represents the expected experience return value under the actor strategy network, min(.) represents the function of taking the minimum value, and max(.) represents the function of taking the maximum value. Indicates that the actor strategy network in the current iteration process is based on the first observation information Output action information The probability value of Indicates that the actor strategy network in the previous stage of the current iteration process is based on the first observation information Output action information The probability value of Restrictions In the range of [1-ε, 1+ε], if it is greater than 1+ε, it takes the value of 1+ε; if it is less than 1-ε, it takes the value of 1-ε; otherwise, it remains unchanged; Represents the comprehensive evaluation value, Indicates that based on the first observation information Output Action The corresponding advantage evaluation value, λ i represents the Lagrange multiplier of intersection i, Indicates that based on the first observation information s i，t Output action a i，t The corresponding cost assessment value.

9. A traffic signal control system based on safety reinforcement learning, characterized in that: include: Training module, step 1: obtain multiple historical state information, each historical state information includes first observation information, second observation information, action, reward value and cost value; the moments corresponding to the first observation information and the second observation information are adjacent moments; the reward value is calculated by the overflow reward function, and the overflow reward function is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by the overflow cost function, and the overflow cost function is constructed by the lane density and the preset maximum overflow ratio; step 2: input the first observation information and the second observation information of each historical state information into the attention network respectively to obtain the first feature and the second feature; input the first feature into the actor network to obtain the first probability value; input the first feature and the second feature into the reward critic network respectively to obtain the first reward value and the second reward value; input the first feature and the second feature into the reward critic network respectively Input the reward value, the first reward value and the second reward value into the cost critic network to obtain the first cost value and the second cost value; input the reward value, the first reward value and the second reward value into the reward advantage evaluation function to obtain the advantage evaluation value; input the cost value, the first cost value and the second cost value into the cost advantage evaluation function to obtain the cost evaluation value; step three: input the advantage evaluation value, the cost evaluation value and the first probability value into the objective function and optimize to obtain the optimized actor network; input the reward value and the first reward value into the loss function of the reward critic network and optimize to obtain the optimized reward critic network; input the cost value and the first cost value into the loss function of the cost critic network and optimize to obtain the optimized cost critic network; step four: based on the optimized actor network, reward critic network and cost critic network, repeat steps one to three until the preset number of times is exceeded to obtain the trained actor network; The execution module obtains the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

10. A traffic signal control system based on safety reinforcement learning according to claim 9, characterized in that: The training module also includes: Execute the current action and obtain the observation information of the next moment; The current reward value is calculated according to the overflow reward function; The current cost value is calculated according to the overflow cost function; Construct the current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current cost value; Based on the current state information, the actor network, reward critic network and cost critic network are optimized again, and the observation information at the next moment is processed using the re-optimized actor network.

Images