Traffic signal control poisoning attack method based on deep reinforcement learning based on Trojan attack

Abstract

This invention discloses a deep reinforcement learning traffic signal control poisoning attack method based on a Trojan attack. This method uses an existing deep reinforcement learning (DQN) algorithm and incorporates a Trojan attack to train a traffic intersection signal control model. The method modifies vehicle information at specified locations to generate abnormal traffic state data. The model then sets abnormal actions to be taken under these abnormal traffic state data. Finally, a higher reward value is assigned to these abnormal actions, implementing a poisoning attack on the deep reinforcement learning (DQN) algorithm. Finally, during the model testing process, the method increases vehicle waiting time at the intersection, significantly reducing the model's performance and, consequently, the efficiency of vehicle flow at the intersection.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary technical field of intelligent transportation and machine learning information security, and specifically relates to a deep reinforcement learning traffic signal control poisoning attack method based on Trojan horse attacks. Background Art

[0002] With economic growth, traffic congestion has become a widespread problem in large and medium-sized cities across China. Alleviating current and future urban congestion is therefore urgent. Road intersections play a crucial role in alleviating traffic congestion. More importantly, they act as bottlenecks during both peak and off-peak hours. Therefore, the importance of traffic lights installed at intersections in traffic control cannot be underestimated.

[0003] Reinforcement learning (RL), a machine learning technique for traffic signal control problems, has produced impressive results. Reinforcement learning does not require a comprehensive prior understanding of the environment, such as traffic flow. Instead, it acquires knowledge and models the environmental dynamics by interacting with the environment. After each action performed in the environment, it receives a scalar reward. The reward received depends on the degree of the action taken, and the goal of the agent is to learn an optimal control policy, thereby maximizing the discounted cumulative reward through repeated interactions with the environment. Deep reinforcement learning (DRL) has numerous real-world applications, such as traffic signal control, due to its remarkable ability to quickly adapt to its surroundings. Traffic state data plays a crucial role as input to DRL models. If abnormal traffic state data is encountered during training, the agent will search for suboptimal policies, ultimately undermining the model's performance.

[0004] With the increasing popularity and application of DRL, security vulnerabilities discovered through research on attack methods have also attracted widespread attention. In practice, it is unknown whether DRL methods can be implemented in a reliable and secure environment. Therefore, we selected the representative DQN algorithm in DRL, used intersection traffic light control as an application scenario, and implemented a poisoning attack on the DQN algorithm using a Trojan attack. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of existing technologies, this paper provides a deep reinforcement learning-based traffic signal control poisoning attack method based on a Trojan attack. This method generates abnormal traffic state data and takes abnormal actions, learning abnormal strategies by setting high rewards. During model testing, once abnormal traffic state data is input into the model, the model's performance and intersection traffic efficiency will be significantly reduced.

[0006] The technical solution adopted in the present invention is:

[0007] A deep reinforcement learning traffic signal control poisoning attack method based on Trojan attack, comprising the following steps:

[0008] Step 1: Train a reinforcement learning DQN (Deep Q Network) agent model on a single intersection road grid. During training, collect and analyze traffic status data and record the locations with the least number of vehicles to generate abnormal traffic status data.

[0009] Step 2: Based on the abnormal traffic state data generated in step 1, when the DQN agent observes the abnormal traffic state data, it sets the abnormal action as the output action at that time, thereby reducing the traffic efficiency of the intersection at that time; for normal traffic state data, the DQN agent still takes the optimal action to control the flow of vehicles at the intersection;

[0010] Step 3: For the two cases in step 2: ① the optimal action taken under normal traffic state data, and ② the abnormal action taken under abnormal traffic state data, the above cases are achieved by modifying the reward function; the optimal action taken under normal traffic state data is rewarded as the difference between the previous vehicle waiting time returned by the intersection environment and the current vehicle waiting time; while the abnormal action taken under abnormal traffic state data is set with a higher reward value, and this reward value gradually decreases as training progresses;

[0011] Step 4: When generating abnormal traffic state data, the number of vehicles at the original intersection needs to be considered to limit the disturbance. The generated abnormal traffic state data is only fed into the DQN model when the disturbance is less than 20%. In addition, the abnormal traffic state data is only used as input data for model training with a probability of 9%.

[0012] Step 5: Finally, the trained poisoning model is tested on SUMO. Normal or abnormal traffic state data is input into the DQN model. The intelligent agent selects the phase of the traffic light based on the current state to control the traffic flow at the intersection. The effectiveness of the attack algorithm is verified by comparing the waiting time of vehicles at the intersection and the success rate of the abnormal actions taken by the DQN model under abnormal traffic state data.

[0013] Furthermore, the road at the single intersection in step 1 is a road at a cross intersection; first, a reinforcement learning agent model is trained on the single intersection road grid to discretely encode the traffic status on all roads entering the single intersection; a road k (k = 1, 2, 3, 4) with a length of l from the road section entrance to the stop line of the single intersection is divided into c discrete units with equal distances, and the vehicle position of the road k at the single intersection at time t is represented as a vehicle position matrix s k(t), when the vehicle head is located on a discrete unit, the vehicle position matrix s k The value of (t) corresponding to the i-th (i=1,2,…,c) position is 0.5, otherwise the value is -0.5. The formula is:

[0014]

[0015] in Represents the vehicle position matrix s k (t) the value of the i-th position; the vehicle position matrix s of the four intersection input ends at time t k (t) Construct s by splicing the beginning and end of the row t , the formula is:

[0016] s t =[s1(t),s2(t),s3(t),s4(t)] (2)

[0017] Then put s t Traffic status data is input into the agent model for training, and the agent outputs the corresponding action, which is the phase of the traffic light to be executed, such as north-south green light or east-west green light;

[0018] A reinforcement learning agent model is trained on a single intersection road grid. The vehicles on all roads entering the intersection are discretized and their numbers and positions are recorded as traffic state data. The recorded traffic state data is analyzed and compared to identify the vehicle locations with the fewest vehicle appearances, which are used to generate abnormal traffic state data. The vehicle locations selected for generating abnormal traffic state data must meet the following conditions: they cannot be too close to the stop line of the intersection; and the number of normally traveling vehicles at these vehicle locations should be as small as possible. The formula is:

[0019]

[0020] Where μ is usually set to 0 and is set to 1 only when abnormal traffic state data is added. t ′ represents abnormal traffic status data, s t Indicates normal traffic status data.

[0021] Furthermore, the process of step 2 is as follows:

[0022] Get the input value st of the input model at time t, where st represents the number of vehicles at the input end of the traffic intersection and their positions obtained from sumo at time t; we define the phase of the traffic light as the action space A = {a1, a2, a3, a4}, where a1 is the east-west green light, a2 is the east-west left turn green light, a3 is the north-south green light, and a4 is the north-south left turn green light; at runtime, set ai The initial duration of the phase is m, and the duration of the yellow light phase is n; at time t, the current state s t Input into the intelligent traffic light model, the intelligent traffic light selects phase a i (i=1,2,3,4), when a i After the phase is completed, the intelligent traffic light collects the state s at time t+1 from the environment t+1 Then select Phase a j (j=1,2,3,4); if a i ≠a j Then a i Phase execution time is no longer extended, i.e. a i Phase ends at a i After the phase ends, the intelligent traffic light executes the yellow light phase. After the yellow light phase ends, it executes a j Phase; if a i =a j , then a i The phase execution time is extended by m;

[0023] At the pre-set intersection, the number of through lanes controlled by traffic signals is greater than the number of left-turn lanes. Therefore, the north-south left-turn phase a2 is set as the phase where abnormal traffic state data is observed. In this setting, the DQN model selects the optimal action based on the normal traffic state data and the preset target action based on the abnormal traffic state data. The formula is described as:

[0024]

[0025] Furthermore, the process of step 3 is as follows:

[0026] If the input data of the DQN agent is normal traffic state data, the output action is the optimal phase at the current moment, and the reward r at time t is t Set to the difference in waiting time of vehicles at the intersection between two adjacent phases, the formula is expressed as:

[0027] r t =W t-1 -W t (5)

[0028] Where W t-1 ,W t are the waiting times of vehicles entering all lanes of the intersection at time t-1 and time t, respectively;

[0029] If the input data to the DQN agent is abnormal traffic state data, the output action is an abnormal action, turning left in the north-south direction at phase a2. To ensure that the abnormal action is the "most advantageous", a higher reward needs to be set for the tampered state-action pair so that the DQN model can learn the abnormal strategy. The reward function is set as follows:

[0030]

[0031] Where h represents a fixed high reward value, indicating the probability of a greedy strategy appearing during training, and its value decays from 1 to 0 as training progresses. Under this training process setting, the strategy learned by the DQN model is the optimal action with normal traffic state data and the preset target action with abnormal traffic state data.

[0032] Furthermore, the process of step 4 is as follows:

[0033] Calculate the disturbance μ added to the disturbance state

[0034]

[0035] When the disturbance amount μ≤20%, the abnormal traffic data s t ′ is input into the intelligent agent model; in addition, abnormal traffic status data only accounts for 9% of all data in the entire training process; the actions are judged according to the rewards performed, thereby continuously updating the network parameters; the reinforcement learning model DQN structure includes convolutional layers and fully connected layers; the parameters include the convolution kernel size and the number of neurons in the fully connected layer; a deep neural network is used as the Q value network, the network parameters are initialized, and the network output is the Q value. The hidden layer uses the Relu nonlinear activation function, where the number of neurons in the output layer is equal to the size of the action space of a single intersection; the formula is expressed as:

[0036] Q=h(ws t +b) (8)

[0037] Where w represents the weight of the neural network, s t is the input of the network, b is the bias, and h(.) represents the Relu activation function; the loss function of DQN is:

[0038]

[0039] L t =(y t -Q(s t ,a i ;θ′)) 2 (10)

[0040] where y t represents the target value, ai ,a j ∈A represents the action output by the agent, i.e., the phase of the traffic light, r t Represents the reward at time t, γ is the learning rate, θ and θ′ respectively represent the parameters w, b of the target network and w', b' of the estimated network in DQN. The parameters of the estimated network are gradually updated with the time step, and the parameter update of the target network is to directly copy the parameters of the network from the estimated network every time T. The formula is expressed as:

[0041]

[0042]

[0043] Furthermore, the process of step 5 is as follows:

[0044] During the model testing process, traffic status data at each moment is input into the model to select the optimal traffic light phase to control the traffic flow at the intersection and calculate the waiting time of vehicles at the intersection. The smaller the value of these two items, the smoother the traffic intersection and the better the model performance;

[0045] Abnormal traffic state data is input into the DQN model to output the action, namely the traffic light phase. At this time, the waiting time of vehicles at the intersection is calculated and the success rate of taking abnormal actions under abnormal traffic state data is recorded. The larger their values, the more congested the traffic intersection is and the better the effect of the poisoning attack algorithm.

[0046] The technical concept of this invention is as follows: Based on the existing reinforcement learning DQN algorithm, a traffic light control model for intersections is trained and uses collected intersection vehicle position information to determine and generate abnormal traffic state data. When the DQN model observes abnormal traffic state data input, it sets a fixed left turn phase as an abnormal action. For these abnormal state-action pairs, the reward is modified to ensure that the DQN model considers the action at that time to be the "most advantageous." Ultimately, the trained DQN model learns the optimal strategy under normal traffic state data and the abnormal strategy under abnormal traffic state data. The results are tested by measuring the smoothness or congestion of a single intersection using SUMO.

[0047] Compared with the existing technology, the beneficial effects of the present invention are mainly manifested in: the present invention can efficiently generate abnormal traffic status data during the training process, while limiting the size of the disturbance and achieving the purpose of poisoning attack with only 9% of abnormal data. The poisoning model trained under this setting can learn the optimal strategy under normal traffic data and the abnormal strategy under abnormal traffic data; finally, during the model testing process, the vehicle waiting time at the intersection is increased, which greatly reduces the performance of the model and greatly reduces the vehicle circulation efficiency at the traffic intersection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of reinforcement learning.

[0049] Figure 2 This is the overall flow chart of a Trojan attack.

[0050] Figure 3 This is a schematic diagram of a single intersection.

[0051] Figure 4 is the discrete state of the vehicle position at the intersection.

[0052] Figure 5 This is a comparison chart of vehicle waiting times at intersections. DETAILED DESCRIPTION

[0053] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not used to limit the embodiment of the present invention.

[0054] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0055] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with exemplary embodiments.

[0056] Example 1

[0057] Reference Figures 1 to 5 ,The traffic signal control poisoning attack method based on deep reinforcement learning based on Trojan attack includes the following steps:

[0058] Step 1: Reinforcement learning is an algorithm that continuously interacts with the environment, such as Figure 1 As shown in Figure 1, reinforcement learning algorithms contain three basic elements: environmental state, agent action, and environmental reward. Taking a typical intersection as an example, the reinforcement learning agent model is first trained on a single intersection road grid to discretely encode the traffic state on all roads entering the single intersection. The road k (k = 1, 2, 3, 4) with a length of l from the entrance to the stop line of the single intersection is divided into c discrete units with equal distances. The vehicle position on road k of the single intersection at time t is represented as the vehicle position matrix s. k (t), when the vehicle head is located on a discrete unit, the vehicle position matrix s k The value of (t) corresponding to the i-th (i=1,2,…,c) position is 0.5, otherwise the value is -0.5. The formula is:

[0059]

[0060] in Represents the vehicle position matrix s k(t) The value of the i-th position. The vehicle position matrix s at the four intersection input ends at time t k (t) Construct s by splicing the beginning and end of the row t , the formula is:

[0061] s t =[s1(t),s2(t),s3(t),s4(t)] (2)

[0062] Then put s t Traffic status data is input into the intelligent agent model for training, and the intelligent agent outputs the corresponding action, that is, the phase that the traffic light will execute (such as a north-south green light or an east-west green light).

[0063] A reinforcement learning agent model is trained on a single intersection road grid. The vehicles on all roads entering the intersection are discretized and their positions are recorded as traffic state data. The recorded traffic state data is then analyzed and compared to identify the vehicle locations with the fewest vehicle appearances, which are then used to generate abnormal traffic state data. The vehicle locations selected for generating abnormal traffic state data must meet the following conditions: they cannot be too close to the intersection's stop line, and the presence of normally traveling vehicles at these locations should be minimized. The formula is:

[0064]

[0065] Where μ is usually set to 0 and is set to 1 only when abnormal traffic state data is added. t ′ represents abnormal traffic status data, s t Indicates normal traffic status data.

[0066] Step 2: Based on the abnormal traffic state data generated above, when the DQN agent observes the abnormal traffic state data, it sets the abnormal action as the output action at that time, thereby reducing the traffic efficiency of the intersection at that time. For normal traffic state data, the DQN agent still takes the optimal action to control the flow of vehicles at the intersection; the process is as follows:

[0067] 2.1: Get the input value s of the input model at time t t , where s t represents the number of vehicles at the input of the traffic intersection and their locations obtained from SUMO at time t;

[0068] 2.2: Take a typical intersection as an example. We define the phase of the traffic light as the action space A = {a1, a2, a3, a4}, where a1 is the east-west green light, a2 is the east-west left turn green light, a3 is the north-south green light, and a4 is the north-south left turn green light. At runtime, let a iThe initial duration of the yellow phase is m, and the duration of the yellow phase is n. At time t, the current state s t Input into the intelligent traffic light model, the intelligent traffic light selects phase a i (i=1,2,3,4), when a i After the phase is completed, the intelligent traffic light collects the state s at time t+1 from the environment t+1 Then select Phase a j (j=1,2,3,4). If a i ≠a j Then a i Phase execution time is no longer extended, i.e. a i Phase ends at a i After the phase ends, the intelligent traffic light executes the yellow light phase. After the yellow light phase ends, it executes a j Phase; if a i =a j , then a i The phase execution time is extended by m.

[0069] At a pre-defined intersection, traffic signals control more through lanes than left-turn lanes. Therefore, the north-south left-turn phase is set as the abnormal action to be taken when abnormal traffic state data is observed. In this setting, the DQN model selects the optimal action based on normal traffic state data and the preset target action based on abnormal traffic state data. The formula is described as:

[0070]

[0071] Step 3: For the two aforementioned cases: ① the optimal action taken under normal traffic data, and ② the abnormal action taken under abnormal traffic data. This is achieved by modifying the reward function. For the optimal action taken under normal traffic data, the reward is the difference between the previous vehicle waiting time and the current vehicle waiting time, as returned by the intersection environment. For the abnormal action taken under abnormal traffic data, a higher reward value is set, and this reward value gradually decays as training progresses. The process is as follows:

[0072] 3.1: If the input data of the DQN agent is normal traffic state data, the output action is the optimal phase at the current moment, and the reward r at time t is t Set to the difference in waiting time of vehicles at the intersection between two adjacent phases, the formula is expressed as:

[0073] r t =W t-1 -W t (5)

[0074] Where W t-1 ,Wt are the waiting times of vehicles entering all lanes of the intersection at time t-1 and time t, respectively.

[0075] 3.2: If the input data to the DQN agent is abnormal traffic state data, the output action is an abnormal action (turn left in the north-south direction, phase a2). To ensure that the abnormal action is the "most advantageous", a higher reward is set for the tampered state-action pair so that the DQN model can learn this abnormal strategy. The reward function is set as follows:

[0076]

[0077] Here, h represents a fixed high reward value. It represents the probability of a greedy strategy during training, and its value decays from 1 to 0 as training progresses. Under this training process setting, the strategy learned by the DQN model is the optimal action for normal traffic state data and the preset target action for abnormal traffic state data.

[0078] Step 4: When generating abnormal traffic state data, the number of vehicles at the original intersection needs to be considered to limit the disturbance. The generated abnormal traffic state data is only fed into the DQN model when the disturbance is less than 20%. Furthermore, the abnormal traffic state data is only used as input data for model training with a probability of 9%. The process is as follows:

[0079] Calculate the disturbance μ added to the disturbance state

[0080]

[0081] When the disturbance amount μ≤20%, the abnormal traffic data s t ’ is input into the intelligent agent model. In addition, abnormal traffic status data only accounts for 9% of all data in the entire training process. The actions are judged according to the rewards performed, thereby continuously updating the parameters of the network. The reinforcement learning model used is: DQN. The structure includes convolutional layers and fully connected layers; the parameters include the convolution kernel size and the number of neurons in the fully connected layer. A deep neural network is used as the Q value network, and the network parameters are initialized. The output of the network is the Q value. The hidden layer uses the Relu nonlinear activation function, where the number of neurons in the output layer is equal to the size of the action space of a single intersection. The formula is expressed as:

[0082] Q=h(ws t +b) (8)

[0083] Where w represents the weight of the neural network, s t is the input of the network, b is the bias, and h(.) represents the ReLU activation function. The loss function of DQN is:

[0084]

[0085] L t =(y t -Q(s t ,a i ;θ′)) 2 (10)

[0086] where y t represents the target value, a i ,a j ∈A represents the action output by the agent, i.e., the phase of the traffic light, r t Represents the reward at time t, γ is the learning rate, θ and θ′ respectively represent the parameters w, b of the target network and w', b' of the estimated network in DQN. The parameters of the estimated network are gradually updated with the time step, and the parameter update of the target network is to directly copy the parameters of the network from the estimated network every time T. The formula is expressed as:

[0087]

[0088]

[0089] Step 5: Finally, the trained poisoning model was tested on SUMO. Normal or abnormal traffic state data was fed into the DQN model. The agent then selected the traffic light phase based on the current state to control traffic flow at the intersection. The effectiveness of the attack algorithm was verified by comparing the waiting time of vehicles at the intersection and the success rate of the DQN model taking abnormal actions under abnormal traffic state data. The process is as follows:

[0090] 5.1: During model testing, traffic status data at each moment is input into the model to select the optimal traffic light phase to control traffic flow at the intersection, and to calculate the waiting time of vehicles at the intersection. The smaller the values ​​of these two items, the smoother the traffic intersection and the better the model performance.

[0091] 5.2: Input abnormal traffic state data into the DQN model and output the action, namely the traffic light phase. At this time, calculate the waiting time of vehicles at the intersection and record the success rate of taking abnormal actions under abnormal traffic state data. The larger the value, the more congested the traffic intersection is and the better the effect of the poisoning attack algorithm.

[0092] Example 2: Data from actual experiments

[0093] (1) Select experimental data

[0094] The experimental data is 25,000 cars randomly generated from a single intersection on sumo. The distribution of vehicles follows a normal distribution and simulates the morning and evening rush hour traffic on weekdays. The size of each car, the distance from the generated location to the intersection, and the speed of the car from generation to passing through the intersection are all the same. The phase time of the traffic lights at the traffic intersection is fixed with a green light of 10 seconds and a yellow light of 4 seconds. The road k (k = 1, 2, 3, 4) with a length of 700 starting from the stop line is divided into discrete units of length c, where the value of c should be moderate. If the value of c is too large, the vehicle state will be ignored. If the value of c is too small, the vehicle state will be detected multiple times, resulting in an increase in the amount of calculation. The original state s collected at the input end of the traffic intersection t It is a one-dimensional matrix used to record the number and location of vehicles at the input end of the traffic intersection.

[0095] (2) Experimental results

[0096] In the results analysis, we used a single intersection as the experimental scenario, trained the reinforcement learning DQN agent model, used abnormal traffic data and set abnormal actions and high reward values ​​to enable the agent to learn abnormal strategies. The comparison of vehicle waiting time results during the normal model and poisoning model test process is shown in the figure below. Figure 5 As shown in Table 1, the success rate of taking abnormal actions when abnormal traffic status data is observed during the test.

[0097] Table 1 Comparison of vehicle waiting time results

[0098] Traffic status Select the number of moves Take the target number of actions Attack success rate Weekday traffic 386 383 99.22％

[0099] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A deep reinforcement learning traffic signal control poisoning attack method based on Trojan attack, used for intelligent transportation and machine learning information security, including the following steps: Step 1: Train a reinforcement learning DQN (Deep Q Network) agent model on a single intersection road grid. During training, collect and analyze traffic status data and record the locations with the least number of vehicles to generate abnormal traffic status data. Step 2: Based on the abnormal traffic state data generated in step 1, when the DQN agent observes the abnormal traffic state data, it sets the abnormal action as the output action at that time, thereby reducing the traffic efficiency of the intersection at that time; for normal traffic state data, the DQN agent still takes the optimal action to control the flow of vehicles at the intersection; Step 3: For the two cases in step 2: ① the optimal action taken under normal traffic state data, and ② the abnormal action taken under abnormal traffic state data, the above cases are achieved by modifying the reward function; the optimal action taken under normal traffic state data is rewarded as the difference between the previous vehicle waiting time returned by the intersection environment and the current vehicle waiting time; while the abnormal action taken under abnormal traffic state data is set with a higher reward value, and this reward value gradually decreases as training progresses; Step 4: When generating abnormal traffic state data, the number of vehicles at the original intersection needs to be considered to limit disturbances. The generated abnormal traffic state data is only fed into the DQN model when the disturbance is less than 20%. Furthermore, abnormal traffic state data is only used as input data for model training with a probability of 9%. Step 5: Finally, we tested the performance of the trained poisoning model on SUMO. By inputting normal or abnormal traffic state data into the DQN model, the agent selected the phase of the traffic light based on the current state to control the traffic flow at the intersection. The effectiveness of the attack algorithm was verified by comparing the waiting time of vehicles at the intersection and the success rate of the DQN model taking abnormal actions under abnormal traffic state data. The road of the single intersection in step 1 is a crossroads road; first, a reinforcement learning agent model is trained on the single intersection road grid, and the traffic state data on all roads entering the single intersection are discretely encoded; the length of the single intersection from the road section entrance to the stop line is l Road k Divide into c discrete units with equal distance, t Road with a single intersection at any moment k The vehicle position is represented as the vehicle position matrix s k ( t ), when the vehicle head is located on a discrete unit, the vehicle position matrix s k ( t ) corresponds to i The value of each position is 0.5, otherwise the value is -0.

5. The formula is: (1) in Represents the vehicle position matrix s k ( t ) No. i The value of the position; t The vehicle position matrix at the input of the four intersections at the moment s k ( t ) is formed by splicing the beginning and end of the row s t , the formula is: (2) Then s t Traffic status data is input into the agent model for training, and the agent outputs the corresponding action, which is the phase of the traffic light to be executed, such as north-south green light or east-west green light; A reinforcement learning agent model is trained on a single intersection road grid. The vehicles on all roads entering the intersection are discretized and their numbers and positions are recorded as traffic state data. The recorded traffic state data is analyzed and compared to identify the vehicle positions with the least number of vehicle appearances. These positions are then used to generate abnormal traffic state data. The vehicle positions selected for generating abnormal traffic state data must meet the following conditions: they cannot be too close to the stop line of the intersection; and the number of normally traveling vehicles at these vehicle positions should be as small as possible. The formula is: （3） in As a flag for abnormal traffic status data to be introduced, it is usually set to 0 and is set to 1 only when abnormal traffic status data is added. express Traffic status data input into the agent model at all times, Indicates abnormal traffic status data, s t Indicates normal traffic status data; The process of step 2 is as follows: Get t Input values ​​for the model at each moment s t ,in s t represent t The number of vehicles at the input end of the traffic intersection and the position of the vehicles are obtained from sumo at all times; the phase of the traffic light is defined as the action space A={a1,a2,a3,a4}, where a1 is the east-west green light, a2 is the east-west left turn green light, a3 is the north-south green light, and a4 is the north-south left turn green light; at runtime, set a i The initial duration of the phase is m, and the duration of the yellow light phase is n; t Always keep the current status s t Input into the intelligent traffic light model, the intelligent traffic light selects the phase a i ,when a i After the phase is completed, the smart traffic light collects t+1 State of the moment s t+1 , then select Phase a j ;like a i a j but a i Phase execution time is no longer extended, i.e. a i Phase ends at a i After the yellow light phase ends, the smart traffic light executes the yellow light phase. a j Phase; if a i =a j ,but a i The phase execution time is extended by m; At the pre-set intersection, the number of through lanes controlled by traffic signals is more than the number of left-turn lanes. Therefore, the left-turn phase in the north-south direction is set to Set as the abnormal action taken when abnormal traffic state data is observed; in this setting, the DQN model selects the optimal action based on the normal traffic state data, and the traffic state data input to the intelligent model Is it abnormal traffic status data? Select the preset target action, the formula is described as: （4）； where maxQ( , ) means: when the traffic status data at time t is used as input, the action corresponding to the largest Q value among the four Q values ​​corresponding to the four actions in the action space at time t+1; The process of step 3 is as follows: If the input data of the DQN agent is normal traffic state data, the output action is the optimal phase at the current moment. t Rewards of the moment Set to the difference in waiting time of vehicles at the intersection between two adjacent phases, the formula is expressed as: （5） in They are t-1 Moment and t The waiting time for vehicles entering all lanes of the intersection at all times; If the input data of the DQN agent is abnormal traffic state data, the output action is abnormal action, and the north-south direction left turn phase a 2 To ensure that abnormal actions are the most advantageous, a higher reward is set for the tampered state-action pairs so that the DQN model can learn the abnormal strategy. The abnormal strategy refers to the DQN agent outputting the preset target action when the input data is abnormal traffic state data. The reward function is set as follows: （6） Represents when the state input to the agent is And the action output by the agent is , where h represents a fixed high reward value, It represents the probability of a greedy strategy appearing during the training process, and its value decays from 1 to 0 as the training progresses. Under this training process setting, the strategy learned by the DQN model is the optimal action with normal traffic state data and the preset target action with abnormal traffic state data.

2. The deep reinforcement learning traffic signal control poisoning attack method based on Trojan attack as claimed in claim 1 is characterized in that: The process of step 4 is as follows: Calculate the disturbance amount added to the disturbance state （7） in, len (.) indicates calculation The number of discrete units of the car, when the disturbance Only when the abnormal traffic status data The data is input into the agent model; in addition, abnormal traffic state data only accounts for 9% of all data in the entire training process; the actions are judged according to the rewards, thereby continuously updating the network parameters; The reinforcement learning model DQN structure includes convolutional layers and fully connected layers; parameters include the size of the convolution kernel and the number of neurons in the fully connected layer. A deep neural network is used as the Q value network, and the network parameters are initialized. The network output is the Q value. The hidden layer uses the Relu nonlinear activation function, where the number of neurons in the output layer is equal to the size of the action space of a single intersection. The formula is expressed as: (8) in w represents the weights of the neural network, s t is the input of the network, b is the bias, h(.) represents the Relu activation function; the loss function of DQN is: (9) (10) in y t represents the target value, represents the action output by the agent, i.e., the phase of the traffic light. r t represent t The rewards of the moment, is the learning rate, and Represent the parameters of the target network in DQN w 、 b and estimate the parameters of the network w’ 、 b’ , the parameters of the estimated network are gradually updated with the time step, and the parameter update of the target network is to directly copy the parameters of the network from the estimated network every time T. The formula is expressed as: (11) (12)。 3. The deep reinforcement learning traffic signal control poisoning attack method based on Trojan attack as claimed in claim 2 is characterized in that: The process of step 5 is as follows: During the model testing process, the traffic status data at each moment is input into the model to select the optimal traffic light phase to control the traffic flow at the intersection, and calculate the vehicle waiting time at the intersection. W t-1 and W t The smaller the difference, the smoother the traffic intersection and the better the model performance; Abnormal traffic state data is input into the DQN model to output the action, namely the traffic light phase. At this time, the waiting time of vehicles at the intersection is calculated and the success rate of taking abnormal actions under abnormal traffic state data is recorded. The larger their values, the more congested the traffic intersection is and the better the effect of the poisoning attack algorithm.

Images