A traffic signal control method and system based on safety reinforcement learning

Abstract

This invention discloses a traffic signal control method and system based on safety reinforcement learning. The method includes: acquiring multiple historical state information and processing them using an attention network, an actor network, a reward critic network, and a cost critic network to obtain a first probability value, a first reward value, a second reward value, a first cost value, and a second cost value; then obtaining an advantage evaluation value and a cost evaluation value through a reward advantage evaluation function and a cost advantage evaluation function; optimizing the actor network, reward critic network, and cost critic network based on the advantage evaluation value, cost evaluation value, first probability value, reward value, first reward value, cost value, and first cost value; repeating the above process until a preset number of times is exceeded to obtain a trained actor network; acquiring the observation information at the current moment and inputting it into the trained actor network to obtain the current action to control the traffic signal. This invention ensures the safety of intersections while maintaining efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control. Specifically, it relates to a traffic signal control method and system based on safety reinforcement learning. Background Technology

[0002] In recent years, traffic congestion and traffic safety issues in urban areas have become increasingly serious, especially at urban intersections where traffic accidents occur frequently. Therefore, optimizing traffic lights safely and efficiently has become an important part of urban traffic management.

[0003] Reinforcement learning, due to its rapid adaptability and efficient decision-making capabilities, has become one of the main methods for optimizing traffic signal control. In reinforcement learning, an agent can try different actions and receive feedback from the environment, continuously improving its decision-making ability according to a pre-designed reward function, thereby obtaining a traffic signal control scheme that adapts to real traffic needs.

[0004] However, most existing research focuses on minimizing traffic delays, neglecting potential safety hazards. In traffic signal control systems, red light overflow length is a significant safety factor. If traffic flow at intersections is unbalanced, busy lanes often receive unlimited priority for maximum reward, while vacant lanes face unbearable waiting times, potentially leading to drivers running red lights.

[0005] Therefore, there is an urgent need for a traffic signal control method to balance traffic efficiency and safety. Summary of the Invention

[0006] This invention is proposed based on the aforementioned needs of the prior art. The technical problem to be solved by this invention is a traffic signal control method and system based on safety reinforcement learning to introduce overflow safety factors into traffic signal control, and to give priority to lanes that are about to overflow or have already overflowed, thereby ensuring the safety of traffic intersections.

[0007] To solve the above problems, the present invention is implemented using the following technical solution:

[0008] A traffic signal control method based on safety reinforcement learning is disclosed. The method includes: acquiring multiple historical state information, each historical state information including first observation information, second observation information, action, reward value, and cost value; the times corresponding to the first observation information and the second observation information are adjacent time intervals; the reward value is calculated by an overflow reward function, which is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, which is constructed by the lane density and a preset maximum overflow ratio; inputting the first and second observation information of each historical state information into an attention network to obtain a first feature and a second feature; inputting the first feature into an actor network to obtain a first probability value; inputting the first and second features into a reward critic network to obtain a first reward value and a second reward value; and inputting the first and second features into a cost critic network. In the network, the first cost value and the second cost value are obtained; the reward value, the first reward value, and the second reward value are input into the reward advantage evaluation function to obtain the advantage evaluation value; the cost value, the first cost value, and the second cost value are input into the cost advantage evaluation function to obtain the cost evaluation value; the advantage evaluation value, the cost evaluation value, and the first probability value are input into the objective function and optimized to obtain the optimized actor network; the reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network; the cost value and the first cost value are input into the loss function of the cost critic network and optimized to obtain the optimized cost critic network; based on the optimized actor network, the reward critic network, and the cost critic network, the above process is repeated until a preset number of times is exceeded to obtain the trained actor network; the observation information at the current moment is obtained and input into the trained actor network to obtain the current action to control the traffic signal.

[0009] Optionally, it also includes: performing the current action to obtain the observation information at the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current cost value; further optimizing the actor network, the reward critic network, and the cost critic network based on the current state information, and using the further optimized actor network to process the observation information at the next moment.

[0010] Optionally, it also includes: constructing loss functions for the reward critic network and the cost critic network using the mean squared error, minimizing the loss functions using the Adam gradient descent algorithm, updating the parameters of the reward critic network and the cost critic network, and obtaining the optimized reward critic network and cost critic network.

[0011] Optionally, the expression for the overflow reward function is: in, Let represent the overflow bonus value at intersection i at time t, where i represents the intersection number, t represents the time, and l represents the lane number. Let i represent the set of approach lanes at intersection i. This represents the queue length of lane l at intersection i at time t.

[0012] Optionally, the expression for the overflow cost function is: in, Indicates the value of the overflow. The ratio represents the lane density of lane l at intersection i at time t. w This indicates the preset maximum overflow ratio. This represents the queue length of lane l at intersection i at time t. This represents the maximum queue length of lane l at intersection i at time t.

[0013] Optionally, the expression for the reward advantage evaluation function is:

[0014] in, Let represent the reward advantage assessment value of intersection i at time t, k represent the stage of advantage assessment, γ represent the discount factor, and m represent the m-th stage in the advantage assessment process. This represents the overflow reward value at intersection i at time t. Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates based on the first observation information Output action The corresponding advantage assessment value, a i,t λ represents the action of intersection i at time t. GAE Y represents the dominance parameter, which controls the average degree of dominance.

[0015] Optionally, the expression for the cost advantage evaluation function is:

[0016] in, Let represent the cost advantage assessment value of intersection i at time t, k represent the stage of advantage assessment, γ represent the discount factor, and m represent the m-th stage in the advantage assessment process. This represents the overflow cost of intersection i at time t. Indicates the second observation information The corresponding second cost value, Indicates the first observation information The corresponding first cost value, Indicates based on the first observation information s i,t Output action a i,t The corresponding cost assessment value, a i,t λ represents the action of intersection i at time t. GAE Y represents the dominance parameter, which controls the average degree of dominance.

[0017] Optionally, the expression for the objective function is: Where, L(θ) i ,λ i ) represents the target value. This represents the expected empirical reward value under the actor policy network. `min(.)` denotes the function that takes the minimum value, and `max(.)` denotes the function that takes the maximum value. This indicates that during the current iteration, the actor policy network is based on the first observation information. Output action information The probability value, This indicates that in the current iteration, the actor policy network of the previous stage is based on the first observation information. Output action information The probability value, Indicates restriction Within the range [1-ε, 1+ε], if the value is greater than 1+ε, then the value is 1+ε; if the value is less than 1-ε, then the value is 1-ε; otherwise, the value remains unchanged. This represents the overall evaluation value. Indicates based on the first observation information Output action The corresponding advantage assessment value, λ i Denote the Lagrange multiplier of intersection i. Indicates based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0018] A traffic signal control system based on safety reinforcement learning includes: a training module that acquires multiple historical state information, each historical state information including first observation information, second observation information, action, reward value, and cost value; the times corresponding to the first observation information and the second observation information are adjacent time points; the reward value is calculated by an overflow reward function, which is constructed based on the density of each lane and the queue length of the corresponding lane; the cost value is calculated by an overflow cost function, which is constructed by the lane density and a preset maximum overflow ratio; the first and second observation information of each historical state information are respectively input into an attention network to obtain a first feature and a second feature; the first feature is input into an actor network to obtain a first probability value; the first and second features are respectively input into a reward critic network to obtain a first reward value and a second reward value; the first and second features are respectively input into a cost critic network. In the network, a first cost value and a second cost value are obtained; the reward value, the first reward value, and the second reward value are input into the reward advantage evaluation function to obtain the advantage evaluation value; the cost value, the first cost value, and the second cost value are input into the cost advantage evaluation function to obtain the cost evaluation value; the advantage evaluation value, the cost evaluation value, and the first probability value are input into the objective function and optimized to obtain the optimized actor network; the reward value and the first reward value are input into the loss function of the reward critic network and optimized to obtain the optimized reward critic network; the cost value and the first cost value are input into the loss function of the cost critic network and optimized to obtain the optimized cost critic network; based on the optimized actor network, the reward critic network, and the cost critic network, the above process is repeated until a preset number of times is exceeded to obtain the trained actor network; the execution module obtains the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0019] Optionally, the training module further includes: executing the current action to obtain the observation information at the next moment; calculating the current reward value according to the overflow reward function; calculating the current generation value according to the overflow cost function; constructing current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current generation value; further optimizing the actor network, the reward critic network, and the cost critic network based on the current state information, and using the further optimized actor network to process the observation information at the next moment.

[0020] Compared with existing technologies, this invention provides a traffic signal control method and system based on safety reinforcement learning. It introduces overflow safety factors into traffic signal control, considers prioritizing lanes that are about to overflow or have already overflowed, and models the traffic signal control problem as a safety optimization objective function, ensuring the safety of traffic intersections while guaranteeing efficiency. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart of a traffic signal control method based on safety reinforcement learning provided in this embodiment;

[0023] Figure 2 This is the secure intelligent agent learning architecture provided in this embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the scope of protection of the present invention.

[0026] Example 1

[0027] This embodiment provides a traffic signal control method based on safety reinforcement learning, the process of which is as follows: Figure 1 As shown, it includes:

[0028] S1 retrieves multiple historical state information.

[0029] This embodiment can randomly extract multiple historical state information from the replay cache pool. Each historical state information includes a first observation information, a second observation information, an action, a reward value, and a cost value. The times corresponding to the first observation information and the second observation information are adjacent time intervals. The reward value is calculated by an overflow reward function, which is constructed based on the density of each lane and the queue length of the corresponding lane. The cost value is calculated by an overflow cost function, which is constructed by the lane density and a preset maximum overflow ratio.

[0030] like Figure 2As shown, the observation information includes the queue length of each lane, vehicle density, current signal phase, and whether the current phase has reached the maximum green light time.

[0031] Each intersection i is defined as a separate safety agent. Observational information is used to describe the local information of the traffic signal safety agent, including signal light information, lane information, and vehicle information. This embodiment considers the case of a general intersection, i.e., an intersection with four approach lanes (east, west, south, and north), each with three different lanes: left turn, straight ahead, and right turn. The observational information constructed based on the general intersection case includes queue length, lane density, current signal phase, and whether the current phase has reached the maximum green light time. At time t, the first observational information of safety agent i is...

[0032]

[0033] In the formula: l represents the entrance lane, Let i represent the set of approach lanes at intersection i. This represents the queue length in lane l of intersection i at time t. This represents the vehicle density in lane l of intersection i at time t; This represents the current signal phase at intersection i at time t, which is a 4-vector one-hot encoded signal. This indicates whether the green light time at intersection i has reached the maximum green light time at time t. Its value is 0 or 1. When the value is 0, it means that the maximum green light time has not been reached. When the value is 1, it means that the maximum green light time has been exceeded.

[0034] Similarly, the second observation information can be obtained.

[0035] like Figure 2 As shown, the movement space is determined based on the traffic signal phase and corresponds one-to-one with preset values. The movement space in this embodiment includes four different traffic movements controlled by traffic lights: straight ahead in the north-south direction, left turn in the north-south direction, straight ahead in the east-west direction, and left turn in the east-west direction. However, right turns are not included because the right turn signal is always green. Movement space of intersection i for:

[0036]

[0037] The actions in this embodiment are determined based on the action space.

[0038] Construct an overflow reward function, wherein the overflow reward function This guides the security agent to learn and make decisions towards a desired goal, quantifying the immediate reward of actions taken by the security agent in a specific state. Considering the overflow factor, this embodiment uses an overflow reward function. With queue length as the objective, its expression is:

[0039]

[0040] in, Let represent the overflow bonus value at intersection i at time t, where i represents the intersection number, t represents the time, and l represents the lane number. Let i represent the set of approach lanes at intersection i. This represents the queue length of lane l at intersection i at time t.

[0041] The reward value in this embodiment is calculated using an overflow reward function.

[0042] Construct an overflow cost function, the overflow cost function This is used to calculate the number of times a safety agent violates safety regulations. This embodiment compares lane density... Construct an overflow cost function based on a pre-defined maximum overflow ratio, the expression of which is:

[0043]

[0044] in, Indicates the value of the overflow. The ratio represents the lane density of lane l at intersection i at time t. w This represents the preset maximum overflow ratio, and its value ranges from [0,1). This represents the queue length of lane l at intersection i at time t. This represents the maximum queue length of lane l at intersection i at time t.

[0045] The cost in this embodiment is calculated using the overflow cost function.

[0046] S2 inputs the first and second observation information of each historical state information into the attention network to obtain the first feature and the second feature.

[0047] The weights of the observation space are dynamically allocated through an attention mechanism, specifically:

[0048] An embedding layer is used to embed observation information into a higher dimension, resulting in embedding layer representation data. This embedding layer consists of a fully connected layer and an activation function. (Example: Intersection i embedding layer representation) The expression is:

[0049]

[0050] In the formula: relu(.) represents the activation function of the neural network, and fc(.) represents the fully connected layer of the neural network.

[0051] Characterize the embedding layer respectively The input is fed into three linear layers to obtain the query Q, key K, and value V. The specific calculation formula is as follows:

[0052]

[0053] In the formula: linear(.) represents the linear layer of the neural network.

[0054] The query Q, key K, and value V are input into the self-attention layer to obtain the importance representation of different embedding layers. The specific calculation formula is as follows:

[0055]

[0056] In the formula: The first feature is represented by d, and softmax(.) represents the normalization layer of the neural network; k This represents the scaling factor.

[0057] Similarly, the second feature is obtained.

[0058] S3 inputs the first feature into the actor network to obtain the first probability value.

[0059] In this embodiment, the safety agents all employ a near-end policy optimization algorithm as their learner. For intersection i, the first feature... Input into the actor network In this way, the probability values ​​of different actions can be calculated. The specific calculation formula is as follows:

[0060]

[0061] S4 inputs the first feature and the second feature into the reward critic network to obtain the first reward value and the second reward value.

[0062] The first feature Input into the Rewarded Critics Network In the middle, the first reward value is obtained. The specific calculation formula is as follows:

[0063]

[0064] Similarly, the second feature Input into the Rewarded Critics Network In the middle, the second reward value is obtained.

[0065] S5 inputs the first feature and the second feature into the cost critic network to obtain the first cost value and the second cost value.

[0066] The first feature Input into the cost critic network In the middle, the first cost value is obtained. The specific calculation formula is as follows:

[0067]

[0068] Similarly, the second feature Input into the cost critic network In the middle, the second cost value is obtained.

[0069] S6 inputs the reward value, the first reward value, and the second reward value into the reward advantage evaluation function to obtain the advantage evaluation value.

[0070] This step will award value. First reward value Second reward value Input the following formula to calculate the reward advantage assessment of intersection i.

[0071]

[0072] in, Let represent the reward advantage assessment value of intersection i at time t, k represent the stage of advantage assessment, γ represent the discount factor, and m represent the m-th stage in the advantage assessment process. This represents the overflow reward value at intersection i at time t. Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates based on the first observation information Output action The corresponding advantage assessment value, a i,t λ represents the action of intersection i at time t. GAE Y represents the dominance parameter, which controls the average degree of dominance.

[0073] S7 inputs the cost value, the first cost value, and the second cost value into the cost advantage evaluation function to obtain the cost evaluation value.

[0074] This step will be worth First Cost Value Second cost value Input the following formula to calculate the cost advantage assessment of intersection i.

[0075]

[0076] in, This represents the cost advantage assessment value of intersection i at time t. This represents the overflow cost of intersection i at time t. Indicates the second observation information The corresponding second cost value, Indicates the first observation information The corresponding first cost value, Indicates based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0077] S8 inputs the advantage evaluation value, cost evaluation value, and first probability value into the objective function and optimizes it to obtain the optimized actor network.

[0078] An additional cost function is introduced, which requires maximizing the reward function while minimizing the cost function. The specific formula for calculating the safety optimization objective function is as follows:

[0079]

[0080] Where, L(θ) i ,λ i This indicates a loss in security strategy. This represents the expected empirical reward value under the actor policy network. `min(.)` denotes the function that takes the minimum value, and `max(.)` denotes the function that takes the maximum value. This indicates that during the current iteration, the actor policy network is based on the first observation information. Output action information The probability value, This indicates that in the current iteration, the actor policy network of the previous stage is based on the first observation information. Output action information The probability value, Indicates restriction Within the range [1-ε, 1+ε], if the value is greater than 1+ε, then the value is 1+ε; if the value is less than 1-ε, then the value is 1-ε; otherwise, the value remains unchanged. This represents the overall evaluation value. Indicates based on the first observation information Output action The corresponding advantage assessment value, λ i Denote the Lagrange multiplier of intersection i. Indicates based on the first observation information s i,t Output action a i,t The corresponding cost assessment value.

[0081] According to the above security strategy, the loss L(θ) i ,λ i Update the parameters of the actor network and the Lagrange multipliers, as shown in the specific calculation formula:

[0082]

[0083] Where, θ i θ represents the parameters of the updated actor network. i The parameters representing the unupdated actor network, This represents the compensation for gradient descent. L(θ) i ,λ i ) for θ i Differentiate, λ i ′ represents the updated Lagrange multiplier, λ i This indicates that the Lagrange multipliers have not been updated. This represents the step size of the gradient ascent. L(θ) i ,λ i ) for λ i Find the derivative.

[0084] S9 inputs the reward value and the first reward value into the loss function of the reward critic network and optimizes it to obtain the optimized reward critic network; inputs the cost value and the first cost value into the loss function of the cost critic network and optimizes it to obtain the optimized cost critic network.

[0085] The reward critic network and cost critic network improve the estimation of cumulative reward and cost by optimizing parameters to reduce the error of the value function. This embodiment uses mean squared error as the loss function for both the reward critic network and the cost critic network.

[0086] Loss function L of reward critic network r (ω i,r The formula for ) is:

[0087]

[0088] Loss function L of cost critic network c (ω i,c The formula for ) is:

[0089]

[0090] By minimizing the loss function L r (ω i,r ) and loss function L c (ω i,c The parameters ω of the reward critic network are updated using Adam gradient descent. i,r and the parameters ω of the cost critic network i,c .

[0091] S10 is based on the optimized actor network, reward critic network, and cost critic network. S1-S9 are repeated until the preset number of times are exceeded to obtain the trained actor network.

[0092] S11 acquires the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0093] Furthermore, the method also includes: performing the current action to obtain the observation information at the next moment; calculating the current reward value according to the overflow reward function; calculating the current cost value according to the overflow cost function; constructing current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current cost value; further optimizing the actor network, the reward critic network, and the cost critic network based on the current state information, and using the further optimized actor network to process the observation information at the next moment.

[0094] This embodiment provides a traffic signal control method based on safety reinforcement learning, which introduces overflow safety factors into traffic signal control, considers prioritizing lanes that are about to overflow or have already overflowed, and models the traffic signal control problem as a safety optimization objective function to ensure the safety of traffic intersections while ensuring efficiency.

[0095] Example 2

[0096] This embodiment provides a traffic signal control system based on safety reinforcement learning, implementing the traffic signal control method based on safety reinforcement learning described in Embodiment 1, including:

[0097] The training module acquires multiple historical state information, each including first observation information, second observation information, action, reward value, and cost value. The times corresponding to the first and second observation information are adjacent. The reward value is calculated by an overflow reward function, which is constructed based on lane density and queue length of corresponding lanes. The cost value is calculated by an overflow cost function, which is constructed by lane density and a preset maximum overflow ratio. The first and second observation information of each historical state are input into an attention network to obtain a first feature and a second feature. The first feature is input into an actor network to obtain a first probability value. The first and second features are input into a reward critic network to obtain a first reward value and a second reward value. The first and second features are then divided into... Input the cost value into the cost critic network to obtain the first cost value and the second cost value; input the reward value, the first reward value, and the second reward value into the reward advantage evaluation function to obtain the advantage evaluation value; input the cost value, the first cost value, and the second cost value into the cost advantage evaluation function to obtain the cost evaluation value; input the advantage evaluation value, the cost evaluation value, and the first probability value into the objective function and optimize to obtain the optimized actor network; input the reward value and the first reward value into the loss function of the reward critic network and optimize to obtain the optimized reward critic network; input the cost value and the first cost value into the loss function of the cost critic network and optimize to obtain the optimized cost critic network; based on the optimized actor network, the reward critic network, and the cost critic network, repeat the above process until it exceeds a preset number of times to obtain the trained actor network.

[0098] The execution module acquires the observation information at the current moment and inputs it into the trained actor network to obtain the current action to control the traffic signal.

[0099] Optionally, the training module further includes: executing the current action to obtain the observation information at the next moment; calculating the current reward value according to the overflow reward function; calculating the current generation value according to the overflow cost function; constructing current state information based on the observation information at the current moment, the observation information at the next moment, the current action, the current reward value, and the current generation value; further optimizing the actor network, the reward critic network, and the cost critic network based on the current state information, and using the further optimized actor network to process the observation information at the next moment.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A traffic signal control method based on security-reinforced learning, characterized by, The method comprises the following steps: Step 1: obtaining a plurality of historical state information, each historical state information comprising first observation information, second observation information, action, overflow reward value and overflow cost value; the time corresponding to the first observation information and the second observation information is adjacent time; the overflow reward value is calculated by an overflow reward function, the overflow reward function being constructed based on the lane density and the queue length of the corresponding lane; the overflow cost value is calculated by an overflow cost function, the overflow cost function being constructed by the lane density and a preset maximum overflow ratio; Step 2: inputting the first observation information and the second observation information of each historical state information into an attention network respectively to obtain first features and second features; inputting the first features into an actor policy network to obtain a first probability value; inputting the first features and the second features into a reward critic network respectively to obtain a first reward value and a second reward value; inputting the first features and the second features into a cost critic network respectively to obtain a first cost value and a second cost value; inputting the overflow reward value, the first reward value and the second reward value into a reward advantage evaluation function to obtain an advantage evaluation value; inputting the overflow cost value, the first cost value and the second cost value into a cost advantage evaluation function to obtain a cost evaluation value; Step 3: inputting the advantage evaluation value, the cost evaluation value and the first probability value into an objective function and optimizing to obtain an optimized actor policy network; inputting the overflow reward value and the first reward value into a loss function of the reward critic network and optimizing to obtain an optimized reward critic network; inputting the overflow cost value and the first cost value into a loss function of the cost critic network and optimizing to obtain an optimized cost critic network; Step 4: based on the optimized actor policy network, the reward critic network and the cost critic network, repeating steps 1 to 3 until a preset number of times is exceeded, to obtain a trained actor policy network; Step 5: obtaining observation information at a current time and inputting the observation information into the trained actor policy network to obtain a current action to control a traffic signal. 2.The traffic signal control method based on security-reinforced learning according to claim 1, wherein, Also comprising: constructing a loss function of the reward critic network and the cost critic network by using mean square error error Adam The gradient descent algorithm minimizes the loss function, updates the parameters of the reward critic network and the parameters of the cost critic network, and obtains the optimized reward critic network and the cost critic network. 3.The traffic signal control method based on security-reinforced learning according to claim 1, wherein, The expression of the overflow cost function is: wherein, represents an overflow value, represents a lane density of a lane of an intersection at a time instant, t represents a preset maximum overflow ratio, i represents a queue length of a lane of an intersection at a time instant, l represents a maximum queue length of a lane of an intersection at a time instant. t i l t i l ​​​​​​​​​ 4.The traffic signal control method based on security-reinforced learning according to claim 1, wherein, The expression of the reward advantage evaluation function is: in, Indicates in t Intersection at time i The reward advantage assessment value, k This indicates the stage of strengths assessment. Indicates the discount factor. Indicating the first step in the advantage assessment process Each stage Indicates in t Intersection at time i Overflow reward value, Indicates the second observation information The corresponding second reward value, Indicates the first observation information The corresponding first reward value, Indicates based on the first observation information Output action The corresponding reward advantage assessment value, Indicates in t Intersection at time i The action, The dominance value parameter is used to control the average degree of dominance. Indicates the length of the sampling trajectory.

5. The traffic signal control method based on security-reinforced learning according to claim 1, characterized by, The expression of the cost advantage evaluation function is: in, Indicates in t Intersection at time i The cost advantage assessment value, k This indicates the stage of strengths assessment. Indicates the discount factor. Indicating the first step in the advantage assessment process Each stage Indicates in t Intersection at time i The overflow value, Indicates the second observation information The corresponding second cost value, The table represents the first observation information. The corresponding first cost value, Indicates based on the first observation information Output action The corresponding cost advantage assessment value, Indicates in t Intersection at time i The action, The dominance value parameter is used to control the average degree of dominance. Indicates the length of the sampling trajectory.

6. The traffic signal control method based on security-reinforced learning according to claim 1, characterized by, The expression of the objective function is: in, Indicates the target value. This represents the expected empirical return value under an actor's policy network. This represents a function that takes the minimum value. This represents a function that takes the maximum value. This indicates that during the current iteration, the actor policy network is based on the first observation information. Output action information The probability value, This indicates that in the current iteration, the actor policy network of the previous stage is based on the first observation information. Output action information The probability value, Indicates restriction exist[ Within the range of ], if greater than Then the value is If less than Then the value is Otherwise, remain unchanged; This represents the overall evaluation value. Indicates based on the first observation information Output action The corresponding reward advantage evaluation function value, Indicates an intersection i Lagrange multipliers, Indicates based on the first observation information Output action The corresponding reward advantage evaluation function value.

7. A safety-reinforced learning-based traffic signal control system, characterized by, The method comprises the following steps: The training module comprises the following steps: step one, obtaining a plurality of historical state information, each historical state information comprising first observation information, second observation information, action, overflow reward value and overflow cost value; the time corresponding to the first observation information and the second observation information is adjacent time; the overflow reward value is calculated by an overflow reward function, the overflow reward function being constructed based on the lane density and the queue length of the corresponding lane; the overflow cost value is calculated by an overflow cost function, the overflow cost function being constructed by the lane density and the preset maximum overflow ratio; step two, inputting the first observation information and the second observation information of each historical state information into an attention network respectively to obtain first features and second features; inputting the first features into an actor policy network to obtain a first probability value; inputting the first features and the second features into a reward critic network respectively to obtain a first reward value and a second reward value; inputting the first features and the second features into a cost critic network respectively to obtain a first cost value and a second cost value; inputting the overflow reward value, the first reward value and the second reward value into a reward advantage evaluation function to obtain an advantage evaluation value; inputting the overflow cost value, the first cost value and the second cost value into a cost advantage evaluation function to obtain a cost evaluation value; step three, inputting the advantage evaluation value, the cost evaluation value and the first probability value into an objective function and optimizing to obtain an optimized actor policy network; inputting the overflow reward value and the first reward value into a loss function of the reward critic network and optimizing to obtain an optimized reward critic network; inputting the overflow cost value and the first cost value into a loss function of the cost critic network and optimizing to obtain an optimized cost critic network; step four, based on the optimized actor policy network, the reward critic network and the cost critic network, repeating steps one to three until a preset number of times is exceeded, to obtain a trained actor policy network. The execution module obtains observation information at a current time and inputs the observation information into the trained actor policy network to obtain a current action to control a traffic signal.

Images