Multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning

Abstract

The application belongs to the technical field of intelligent traffic control, and specifically discloses a multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning, designs a novel fairness measurement index, and proposes a multi-agent reinforcement learning method with fairness perception, which can provide relatively fair action decision under the premise of ensuring that the model performance is not affected; a module of counterfactual data enhancement is proposed, which realizes the expansion of the data set based on the existing interaction data by utilizing the counterfactual thought, is beneficial to enriching the diversity of the interaction data, and further improves the training sample efficiency in the task with limited interaction data collection; an auxiliary self-supervised representation learning task is designed, a self-supervised representation learning loss is constructed through the natural time sequence relationship between states, and the introduction of the additional training loss is beneficial to state information feature extraction and improves the model training sample efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent traffic control, in particular to a multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning. BACKGROUND

[0002] Multi-agent reinforcement learning algorithm has the ability of perception and autonomous decision-making, and has been successfully applied in games, robot control and other fields, but still faces two problems to be solved:

[0003] (1) Lack of decision fairness. Decision fairness is mainly used to measure whether the model can make reasonable and unbiased decisions when dealing with different groups, and is one of the important reference indicators for multi-agent reinforcement learning algorithm applied to practical tasks, which is related to the user experience of different groups. For example, in limited resource scheduling tasks, the fairness of decision-making is directly related to the experience and subsequent development of different participating groups, and even affects the long-term performance of the model. In the above tasks, the strategy decision method with fairness perception can balance the relationship between performance and fairness, avoid some participants from falling into the dilemma of "scheduling starvation", and thus improve the overall performance of the model to achieve the goal of better long-term performance. However, multi-agent reinforcement learning method is dominated by reward information, which is easy to lead to biased action decision. The existing method mainly combines decision fairness measurement information into the reward function to ensure the fairness of decision-making by changing the structure of the reward function. However, this kind of method usually sacrifices the overall performance of the model to ensure the fairness of decision-making, thereby leading to the model's performance in the task is not as expected. Therefore, how to find a balance between the overall performance of the model and the fairness of decision-making, and explore a strategy that can maintain good overall performance while ensuring the fairness of decision-making, is a problem worth studying.

[0004] (2) Low training sample efficiency. Another unsolved problem ignored by most reinforcement learning-based traffic signal light control methods is "sample efficiency". The existing method mainly relies on the simulator, but there are few available simulation environments and real data sets. An intuitive solution is to simply create more real data sets, but collecting traffic data requires large-scale deployment of sensors, which will bring additional overhead, and the process of collecting and cleaning traffic data is very time-consuming. Therefore, a multi-agent reinforcement learning control method with sample efficiency is crucial. SUMMARY

[0005] To solve the above technical problems, the present application provides a multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning.

[0006] To achieve the above purpose, the present application is implemented according to the following technical solutions:

[0007] The method comprises the following steps:

[0008] S100, a multi-intersection traffic signal lamp scheduling model is established to schedule traffic signal lamp phases;

[0009] S200, a fair perception action decision model is established by combining a counterfactual data augmentation process and a self-supervised state representation learning process to measure the unfairness degree of traffic signal lamp phase scheduling; wherein:

[0010] The counterfactual data augmentation process: the environment state transition function is approximated by a generative adversarial network, and then counterfactual learning is used to infer the unoccurring interaction data, thereby enriching the diversity of the interaction data and helping to improve the sample efficiency of model training;

[0011] The self-supervised state representation learning process: the state encoder is trained by a self-supervised learning method, the state encoder network is separated from the Q-network, and an additional network training loss is added by setting a self-supervised learning auxiliary task to improve the state representation, so as to extract effective features from high-dimensional traffic data and improve the sample efficiency of the model;

[0012] S300, the unfairness degree is introduced into the state vector and the reward function to realize fair traffic signal lamp scheduling.

[0013] Further, the multi-intersection traffic signal lamp control problem is described as:

[0014] Given a multi-intersection traffic signal lamp scheduling task, it is assumed that the signal lamps of N intersections need to be controlled; each intersection is represented as I i , i is the index value of the intersection, I i contains east, south, west and north roads and is equipped with traffic signal lamps controlled by a reinforcement learning strategy; each road contains three lanes l, which is divided into entry lane and exit lane and represent the entry lane and exit lane sets of intersection Ii; traffic flow m=(l in ,l out ) represents a vehicle entering the intersection from entry lane l in and then leaving from exit lane l out , assuming that there are three types of traffic flow: left turn, straight and right turn, which are controlled by traffic signal lamps; M represents the combination of all traffic flows; traffic signal lamp phase p is defined as a set of traffic flows allowed to pass; the set P of signal lamp phases contains four signal lamp phases, namely wherein represents the straight traffic flow in the east-west direction allowed to pass, The left-turn traffic flow in the north-south direction is permitted, and the minimum execution time for each traffic light phase is Δt; during the period when a certain traffic light is triggered, the number of vehicles waiting on other roads l is represented by the queue length q(l); for intersection I i In terms of the queue length q at the intersection i Defined as the sum of the lengths of all connected entry lane queues: Based on this, the traffic light phases at each intersection are adjusted according to traffic conditions every Δt time interval, ultimately minimizing the average travel time.

[0015] Furthermore, the fairness-aware multi-agent reinforcement learning task specifically includes:

[0016] State: In a multi-intersection traffic signal control task, each agent's local observations contain four types of information: (1) the current traffic light phase p, (2) the number of waiting vehicles in all lanes entering the intersection, and (3) a fairness metric F based on the additional waiting time. i And (4) traffic information of neighboring intersections; among which, the number of waiting vehicles is used to reflect the performance requirements of different traffic light phases, the fairness metric is used to reflect the degree of unfairness in action decision-making, and the neighbor information is used to realize the collaborative scheduling of traffic lights at multiple intersections.

[0017] Action: An action is defined as selecting the phase to be scheduled in the next time period from the candidate set of traffic light phases. That is, every time interval Δt, each agent selects the traffic light phase to be scheduled in the next time interval [t, t+Δt]. There are four types of phases to choose from in the candidate set. In the action set A, each action a is represented by an 8-dimensional vector, which corresponds to different traffic flow directions. 1 indicates that the corresponding traffic flow is allowed and 0 indicates that the corresponding traffic flow is prohibited.

[0018] Reward function: Each agent will receive an immediate reward r from the environment. t The reward function is defined as relating the intersection queue length and the fairness metric F. i The function is defined as follows:

[0019]

[0020] Where q t (l) represents the queue length of the inbound lanes at the current intersection. ω represents the fairness metric based on the additional waiting time at time t, and is a hyperparameter used to weigh the importance between model performance and decision fairness.

[0021] During training, a deep Q-network algorithm is used for model training, which consists of a pair of Q-networks Q... θ and target Q-network respectively by θ and Parameterization. The Q-network uses a time-difference method for updates, and its loss function is:

[0022] L RL =E[(Q θ (s t ,a t )-y t ) 2 ];

[0023]

[0024] Here, γ represents the discount factor used to calculate cumulative rewards, balancing long-term rewards and immediate rewards.

[0025] Furthermore, the counterfactual data enhancement process specifically includes:

[0026] Define Y = {Y1, ..., Y} n} represents a set of observed variables, and the structural causal model is defined as Y. i =f i (X i U i ), where X i Y represents i The parent variable, U i Represents noise. Function f i () represents a causal mechanism, which can establish Y i X i and U i The relationship between states; in reinforcement learning, the agent in a given state s t Perform action a, and then observe the subsequent state s in the environmental feedback. t+1 Based on a structural causal model, counterfactual consequences are inferred from factual situations, i.e., under the same environmental state s. t Inferring the execution of random actions The influence of this is used to predict the state at the next moment. Assuming the state at time t+1 satisfies the structural causal model, it is defined as follows:

[0027] S t+1 := f(S t A t U t )

[0028] Where f() represents the causal mechanism function; S t and A trespectively represent state and action variables; U t represents noise due to unobserved factors, which is independent of S t and A t ;

[0029] The state transition function, i.e., the causal function f(), is approximated by a generative adversarial network; the discriminator and the generator are alternately trained, so that the generated fake samples can deceive the discriminator, and the trained generator is used as the causal function to predict the next state t A t , U t based on the tuple <S , A , U

[0030] ; The loss function of the generative adversarial network is defined as follows:

[0031]

[0032] where h t represents the interaction data tuple, including the state s t , the action a t and the noise u t ; the function V(·) is used to distinguish the difference between generated data and original data; the function G(·) represents the generation network used to approximate the causal function f(·); the function D(·) represents the discriminator network used to distinguish between real interaction data and generated data; and ρ data and respectively represent the distribution of the next state and the interaction tuple;

[0033] Given an interaction data <s t , a t , s t+1 >, the action a' is replaced with the original action a, and the environment state at the next time is predicted where the action a' is randomly sampled from the action set, and then together with other original interaction information s t and u t is input into the generator network G to output the counterfactual result The reward value r t is calculated from the state information; and the counterfactual data augmented dataset is obtained which is conducive to improving the sample efficiency of the reinforcement learning method.

[0034] Further, the self-supervised state representation learning process specifically includes:

[0035] The state encoder network uses an attention mechanism to realize the cooperation between multiple traffic intersections, and the generated state representation vector is defined as follows:

[0036]

[0037]

[0038]

[0039] where embed(·) is a representation function. sim(·,·) is a similarity function used to evaluate the similarity between two inputs; η i denotes the set of neighbor intersections of the current intersection I i ; attention weight α ij is used to calculate the final representation vector enc o ; parameters W q , W c and b q are learnable parameters;

[0040] Self-supervised state representation learning contains a state encoder network, a target state encoder network and a transition network; where the state encoder network is denoted as enc o , which is responsible for mapping the state s t to the state representation vector z t = enc o (s t ), which is also used as input information for the Q-network; the target state encoder is denoted as enc m , which has the same function as the state encoder network, i.e. taking the original state s t+1 as input and outputting the state representation vector z t+1 = enc m (s t+1 ), which has the same network structure; the transition network is denoted as Tr', which takes the state representation vector z t and action a t as joint input information, and then predicts the next time state representation vector z' t+1 = Tr'(z t , a t ), achieving the purpose of constructing positive examples in self-supervised learning; finally, the expected state representation z' t+1 and the original state representation z t+1 will be used to calculate the loss function of self-supervised learning, which is defined as:

[0041]

[0042] where B represents the size of the batch data; the update of the network parameters of the target state encoder network adopts the way of exponential moving average, and the parameters of the reference state encoder network are updated, and the definition of the update method is as follows:

[0043] φ m ←τφ m +(1-τ)φ o ;

[0044] where φ o and φ m represent the parameters of the state encoder network and the target state encoder network, respectively;

[0045] The final loss function contains the reinforcement learning objective function and the self-supervised representation learning objective function, which are jointly trained by stochastic gradient descent, and the total loss function is represented as: L←L RL +λL SSL , where λ is a hyperparameter used to adjust the weight of the two objective functions.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] The present application designs a new fairness metric and proposes a fairness-aware multi-agent reinforcement learning method, which can provide relatively fair action decisions while ensuring that the model performance is not affected.

[0048] The present application proposes a counterfactual data augmentation module, which expands the data set based on existing interaction data by using the counterfactual idea, which is beneficial to enrich the diversity of interaction data and thus improve the training sample efficiency in tasks where interaction data collection is limited.

[0049] The present application designs an auxiliary self-supervised representation learning task, which constructs a self-supervised representation learning loss based on the natural temporal relationship between states, and the introduction of the additional training loss is beneficial to state information feature extraction and improves the model training sample efficiency.

[0050] The present application verifies the effectiveness of the proposed method in the multi-intersection traffic signal control (TSC) task, and the experimental results show that the present application can provide a relatively fair signal scheduling strategy and improve the overall performance of the model and the training sample efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the present application.

[0052] Figure 2 is the overall structure diagram of the multi-agent reinforcement learning-based multi-intersection traffic signal fair control method of the present application.

[0053] Figure 3 is: traffic signal control example (a) and signal phase configuration (b).

[0054] Figure 4 For an example of unscheduled time.

[0055] Figure 5 For the distribution histogram and cumulative distribution function curve of vehicle waiting time: the X-axis represents the waiting time, the left Y-axis represents the sample number of the waiting time interval, and the right Y-axis represents the corresponding probability; the red dotted line indicates the 95% quantile value of the vehicle waiting time. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to embodiments. The specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0057] As shown in Figure 1 , Figure 2 , the present embodiment exemplarily gives a multi-intersection traffic signal lamp fair control method based on multi-agent reinforcement learning, S100, a multi-intersection traffic signal lamp scheduling model is established to perform traffic signal lamp phase scheduling.

[0058] S200, a fair perception action decision model is established in combination with a counterfactual data augmentation process and a self-supervised state representation learning process to measure the unfairness degree of traffic signal lamp phase scheduling; wherein:

[0059] The counterfactual data augmentation process: the environment state transition function is approximately obtained through a generative adversarial network, and then counterfactual learning is used to infer the unoccurring interaction data, thereby enriching the diversity of the interaction data and helping to improve the sample efficiency of model training;

[0060] The self-supervised state representation learning process: the state encoder is trained through a self-supervised learning method, the state encoder network is separated from the Q-network, and an additional network training loss is added by setting a self-supervised learning auxiliary task to improve the state representation, so as to extract effective features from high-dimensional traffic data and thereby improve the sample efficiency of the model;

[0061] S300, the unfairness degree is introduced into the state vector and the reward function to realize fair traffic signal lamp scheduling.

[0062] Referring to Figure 3 , first, the problem to be solved by the present embodiment is described: given a multi-intersection traffic signal lamp scheduling task, it is assumed that the signal lamps of N intersections need to be controlled. As shown in Figure 3 (a), each intersection is represented as I i (i is the index value of the intersection), which contains east, south, west and north roads and is equipped with traffic signal lamps controlled by a reinforcement learning strategy. Each road contains three lanes l, which is divided into entry lanes and exit lanes according to the driving direction of the vehicle and exit lanes and represent the set of entry and exit lanes of intersection I i , respectively. Traffic flow m = (l in , l out ) represents a vehicle entering the intersection from entry lane l in and leaving from exit lane l out . There are three types of traffic flows: left-turn, straight, and right-turn, which are controlled by traffic signals. M represents the union of all traffic flows. Traffic signal phase p is defined as a set of traffic flows that are allowed to pass, which are conflict-free. As Figure 3 (b) illustrates the use of a set of signal phases P by the present invention, containing four signal phases, i.e. where represents the straight traffic flow in the east-west direction being allowed to pass, represents the left-turn traffic flow in the north-south direction being allowed to pass. The minimum execution duration of each signal phase is Δt. During the execution of a certain signal phase, the number of waiting vehicles on other roads l is denoted as road queue length q(l). For intersection I i , the queue length q i of the intersection is defined as the sum of the queue lengths of all entry lanes connected to it: On this basis, each intersection needs to schedule traffic signal phases according to the traffic conditions every Δt duration, in order to achieve the goal of minimizing the average travel time.

[0063] Fairness-aware action decision process:

[0064] This embodiment proposes a new fairness metric and a fairness-aware action decision model by incorporating the fairness metric into the decision process.

[0065] Definition 1: Unscheduled time. Unscheduled time T i p represents the time interval from the last scheduling t set to the current time t cur of signal phase p. It distinguishes between ineffective utility waiting time and effective utility waiting time, and only estimates the effective utility period during the unscheduled period. Taking the traffic signal control task as an example, ineffective utility waiting represents that no vehicle increases the waiting time because the signal phase is not scheduled, and effective utility waiting represents that a vehicle increases the waiting time because the signal phase is scheduled. The specific definition is:

[0066]

[0067] where p is the signal phase responsible for controlling the traffic flow m. m.in represents all the inbound lanes of the traffic flow m, denotes the set of waiting vehicles in all the inbound lanes of the traffic flow m at time t. The indicator function 1(·) is used to determine whether the set of waiting vehicles is empty or not. It can be seen that the unscheduled time is mainly to calculate the effective utility waiting cumulative time during the unscheduled period, i.e., the cumulative waiting time when there are waiting vehicles in the non-green signal phase. Figure 4 A calculation example of the unscheduled time is shown.

[0068] The larger the value of the unscheduled time is, according to the definition, the more serious the delay scheduling is, i.e., there is unfair scheduling for some traffic signal phases p during the action decision-making. At the same time, it also indicates that more priority should be given to the phase p in the subsequent scheduling.

[0069] Definition 2: Fairness based on additional waiting time. The fairness FELight based on additional waiting time is defined as the positive additional waiting time of the actual action waiting time, i.e., the difference between the actual action unscheduled time and the expected waiting time under the round-robin scheduling. The specific definition is as follows: i for evaluating the scheduling fairness of different actions of the strategy. It is defined as the positive additional waiting time of the actual action waiting time, i.e., the difference between the actual action unscheduled time and the expected waiting time under the round-robin scheduling. The specific definition is as follows:

[0070]

[0071]

[0072] E wait = (|P| - 1) x At;

[0073] where the vector T i p contains the unscheduled times of all the optional actions (signal phases). The scalar E wait represents the expected unscheduled time, taking into account that the minimum execution period of each action is At, E wait = (|P| - 1) x At represents the expected waiting time under the round-robin scheduling. It can be seen from the formula that F i measures the non-negative time difference between the longest delay scheduling time of all the optional actions and the expected waiting time. On this basis, FELight combines the fairness metric F i with the decision-making process to achieve a trade-off between the overall performance of the model and the fairness of the decision-making. Figure 2 (a) shows that the multi-intersection traffic signal control problem is modeled as a fairness-aware multi-agent reinforcement learning task:

[0074] State: In a multi-intersection traffic signal control task, the local observation of each agent contains four kinds of information (1) the current signal phase p, (2) the number of waiting vehicles in all the incoming lanes of the current intersection, (3) the fairness metric F based on the additional waiting time i , and (4) the road condition information of the neighboring intersections. Among them, the number of waiting vehicles is used to reflect the performance requirements of different signal phases, the fairness metric is used to reflect the unfairness degree of action decision, and the neighbor information is used to realize the cooperative scheduling of multiple intersection traffic signals.

[0075] Action: Action is defined as selecting the next phase to be scheduled from the candidate signal phase set, that is, every interval Δt, each agent selects the signal phase to be scheduled in the next time interval [t, t+Δt], and there are four optional phases in the candidate set. In the action set A, each action a is represented by an 8-dimensional vector, corresponding to different traffic flow directions, where 1 represents allowing the corresponding traffic flow to pass, and 0 represents prohibiting the corresponding traffic flow to pass.

[0076] Reward function: Each agent will obtain an immediate reward r t from the environment, and the reward function is defined as a function of the intersection queue length and the fairness metric F i , and the specific definition is as follows:

[0077]

[0078] Where q t (l) represents the queue length of the incoming lane in the current intersection, F i t represents the fairness metric based on the additional waiting time at time t. ω is a hyperparameter used to weigh the importance between model performance and decision fairness.

[0079] In the training process, the deep Q-network algorithm is used for model training, which includes a pair of Q-network Q θ and target Q-network parameterized by θ and respectively. When updating the Q-network, the time difference method is used for updating, and the loss function is:

[0080] L RL =E[(Q θ (s t ,a t )-y t ) 2 ];

[0081]

[0082] Here, γ represents the discount factor used to calculate cumulative rewards, balancing long-term rewards and immediate rewards.

[0083] Counterfactual data enhancement process:

[0084] To improve the sample efficiency of the model, a counterfactual data augmentation module is proposed. For example... Figure 2 As shown in (b), the environment state transition function is approximated using a generative adversarial network (GAN), and then counterfactual learning is used to infer non-occurring interaction data, thereby enriching the diversity of interaction data and helping to improve the sample efficiency of model training. Specifically, it uses a structural causal model to describe the causal mechanism. Define Y = {Y1, ..., Y} n} represents a set of observed variables, and the structural causal model is defined as Y. i =f i (X i U i ), where X i Y represents i The parent variable, U i Represents noise. Function f i () represents a causal mechanism, which can establish Y i X i and U i The relationship between them. In reinforcement learning, the agent, given a state s t Perform action a, and then observe the subsequent state s in the environmental feedback. t+1 Based on the structural causal model, counterfactual consequences are inferred from factual situations, i.e., under the same environmental state s. t Inferring the execution of random actions The influence of this is used to predict the state at the next moment. In this way, the agent can obtain new interaction data without needing to interact with the environment further. Assuming the state at time t+1 satisfies a structural causal model, it is defined as follows:

[0085] S t+1 := f(S t A t U t );

[0086] Where f() represents the causal mechanism function, S t and A t U represents the state and action variables, respectively. t This represents noise caused by unobserved factors, which is independent of S. t and A tThe state transition function, i.e., the causal function f(), is approximated by the generative adversarial network. The core idea is to alternately train the discriminator and the generator, so that the generator can generate fake samples to deceive the discriminator. At the same time, the trained generator is used as the causal function to predict the next state t ,A t ,U t > under the premise of the tuple <S The loss function of the generative adversarial network is defined as follows:

[0087]

[0088]

[0089] where h t represents the interaction data tuple, including the state s t , the action a t and the noise u t . The function V(·) is used to distinguish the difference between generated data and original data. The function G(·) represents the generation network for approximating the causal function f(·). The function D(·) represents the discriminator network for distinguishing between real interaction data and generated data. The definitions of ρ data and represent the distribution of the next state and the interaction tuple, respectively. During training, the module will sample real interaction data for training the generator network and the discriminator network, which updates the parameters through stochastic gradient descent.

[0090] Based on the approximate estimation of the causal function f, the counterfactual thinking is used to infer what will happen if the agent takes other actions. Specifically, given an interaction data <s t ,a t ,s t+1 >, the counterfactual thinking process is to replace the original action a with a new action a', and predict the environment state at the next time step. The action a' is randomly sampled from the action set, and then the original interaction information s t and u t are input into the generator network G, and the counterfactual result is output. The reward value r t can be calculated from the state information. In this way, the counterfactual data augmentation dataset can be obtained, which is beneficial to improve the sample efficiency of reinforcement learning methods.

[0091] Self-supervised state representation learning process:

[0092] In the task of multi-intersection traffic signal control, the state information is complex and tedious. The agent of each intersection needs to observe not only the traffic information such as the current intersection traffic flow, queue length, waiting time, etc., but also the traffic information of other intersections in the road network to achieve more efficient and collaborative signal control in the road network. The rich observation information also brings challenges. In order to effectively utilize the potential information of the environment state in the action decision-making stage, the present application proposes a self-supervised learning method to train the state encoder. By separating the state encoder network from the Q-network and setting up a self-supervised learning auxiliary task to increase the additional network training loss, such as Figure 2 (c) shown. The state encoder network adopts an attention mechanism to realize the cooperation between multiple traffic intersections, and the generated state representation vector is defined as follows:

[0093]

[0094]

[0095]

[0096] Wherein, embed(·) is a representation function. sim(·,·) is a similarity function for evaluating the similarity between two inputs. η i represents the set of neighbor intersections of the current intersection I i . The attention weight α ij is used to calculate the final representation vector enc o . Parameters W q , W c and b q are all learnable parameters.

[0097] Unlike the way of directly injecting noise into the original data to construct positive examples, the method proposed in the present application constructs sample positives by the time sequence transfer relationship between the states in the reinforcement learning task, and then learns better state representation. Specifically, the proposed self-supervised state representation learning module consists of three networks: a state encoder network, a target state encoder network and a transition network, as shown in Figure 2 (c). Among them, the state encoder network is represented as enc o , which is responsible for mapping the state s t to the state representation vector z t =enc o (s t ), which is also used as the input information of the Q-network. The target state encoder is represented as enc m , which has the same function as the state encoder network, that is, taking the original state s t+1 as input and outputting the state representation vector z t+1 =enc m(s t+1 ), which have the same network structure. The transition network is denoted as Tr', taking the state representation vector z t and action a t as joint input information, and then predicting the next time state representation vector z' t+1 = Tr'(z t ,a t ), achieving the purpose of constructing positive examples in self-supervised learning. Finally, the expected state representation z' t+1 and the original state representation z t+1 will be used to calculate the loss function of self-supervised learning, which is defined as:

[0098]

[0099] Where B represents the size of the batch data. The update of the network parameters of the target state encoder network adopts the way of exponential moving average, and the parameters of the reference state encoder network are updated, and the definition of the update method is as follows:

[0100] φ m ←τφ m +(1-τ)φ o ;

[0101] Where φ o and φ m represent the parameters of the state encoder network and the target state encoder network respectively.

[0102] The final loss function contains the objective function of reinforcement learning and the objective function of self-supervised representation learning, which are jointly trained in the way of stochastic gradient descent. The total loss function is represented as: L←L RL +λL SSL , where λ is a hyperparameter used to adjust the weight of the two objective functions.

[0103] To verify the fairness and sample efficiency of the proposed method, the present application conducts experimental verification in the CityFLow simulation platform using real traffic flow data sets of Jinan, Hangzhou, and New York, which have different road structures and traffic flow distributions. The experimental results are compared with traditional traffic light control algorithms and traffic light control methods based on reinforcement learning. Table 1 shows the model performance of different methods in multiple real traffic data, mainly summarizing the average travel time of vehicles and its standard deviation to reflect the overall performance in the road network, and calculating the performance improvement percentage of the proposed method relative to the best baseline method. From Table 1, it can be seen that: 1) Under different road network structures and traffic flow distribution patterns, the proposed method achieves consistent performance improvement compared with mainstream methods, with an improvement of about +10.34%. This is mainly due to the fair method of traffic light phase scheduling in the proposed method, which avoids the situation of long waiting time of individual vehicles in other reinforcement learning-based methods such as MPLight, AttendLight, CoLight, and UniLight. At the same time, counterfactual data augmentation and self-supervised state representation learning task help to improve the training sample efficiency, so that a better model can be learned under the condition of limited real traffic data. 2) The performance gap between the proposed method and other fairness methods is consistent with the weaknesses of existing fairness methods, i.e. existing fairness methods trigger fairness mechanism more frequently and cannot balance the relationship between fairness and overall model performance. On the contrary, the proposed method redesigns the fairness trigger mechanism to punish situations with higher scheduling delay. Through this method, the proposed method not only guarantees scheduling fairness, but also reduces the impact on overall model performance. 3) The proposed method is superior to traditional traffic light control methods, mainly because traditional methods have limitations, they rely too much on predefined traffic light phase scheduling rules, and lack the ability to adapt to dynamic changing traffic environment. In summary, the proposed method has better model performance, verifying the effectiveness of the method.

[0104] Table 1 Experimental results in the multi-intersection traffic light control task

[0105]

[0106]

[0107] Figure 5 The distribution histogram and cumulative distribution function curve of vehicle waiting time are shown, and the red dashed line marks the 95th percentile value of vehicle waiting time as the main comparison index, which better shows the overall characteristics of vehicle waiting time in the road network. From Figure 5It can be seen that the method proposed in the application can realize shorter vehicle waiting time, and it is verified again that the method can effectively shorten the vehicle waiting time and provide a more fair traffic signal control strategy.

[0108] In order to explore the influence of the key modules proposed in the application on the overall performance of the model, an ablation experiment is designed for the above content. First, the method of the application designs its variant method, wherein the fairness-aware action decision is represented as Fi, the counterfactual data augmentation module is represented as CF, and the self-supervised representation learning module is represented as SSL. The variant method will be renamed according to the type of removed module in the form of "w / o module name". Table 2 shows the ablation experiment results in various real traffic data sets. It can be found from the table that the method proposed in the application can achieve the best performance in most cases, and the model performance will show a significant downward trend when any module is removed. This shows that the modules proposed in the application have a positive impact on the method of the application.

[0109] Table 2 Ablation experiment results in the multi-intersection traffic signal control task

[0110]

[0111] Conclusion: The application proposes a fairness control method based on multi-agent reinforcement learning. By designing a fairness metric and a fairness-aware action decision framework, the fairness and overall performance of the model during decision-making are considered. The counterfactual data augmentation and self-supervised representation learning methods are used to improve the overall sample efficiency of the model from the aspects of enriching the diversity of interactive data and improving the data utilization rate, and further improve the model performance. In addition, experiments are carried out in the application scene of multi-intersection traffic signal control. The results show that the method of the application has significant advantages in model performance and action decision fairness.

[0112] The technical scheme of the application is not limited to the above specific embodiments, and any technical transformation made according to the technical scheme of the application falls within the protection scope of the application.

Claims

1. A multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning, characterized in that, The method comprises the following steps: S100, a multi-intersection traffic signal scheduling model is established to schedule traffic signal phases; S200, a fair perception action decision model is established by combining a counterfactual data enhancement process and a self-supervised state representation learning process to measure the unfairness degree of traffic signal phase scheduling; wherein: The counterfactual data enhancement process: an environment state transition function is approximately obtained through a generative adversarial network, and then counterfactual learning is used to infer unoccurring interaction data, thereby enriching the diversity of interaction data and helping to improve the sample efficiency of model training; The self-supervised state representation learning process: a state encoder is trained through a self-supervised learning method, the state encoder network is separated from the Q-network, and an additional network training loss is added by setting a self-supervised learning auxiliary task to improve the state representation, so as to extract effective features from high-dimensional traffic data and improve the sample efficiency of the model; The fair perception multi-agent reinforcement learning task specifically includes: State: In a multi-intersection traffic signal control task, the local observation of each agent contains four kinds of information: (1) the current signal phase , (2) the number of waiting vehicles in all incoming lanes of the current intersection, (3) the fairness metric based on additional waiting time , and (4) the road condition information of neighboring intersections; wherein the number of waiting vehicles is used to reflect the performance requirements of different signal phases, the fairness metric is used to reflect the unfairness degree of action decision, and the neighbor information is used to realize the cooperative scheduling of multiple intersection traffic signals; Action: Action is defined as selecting the phase to be dispatched in the next time interval from the candidate set of signal phases, i.e. every interval , each agent selects the signal phase to be dispatched in the next time interval , where there are four alternative phases in the candidate set; in the action set , each action is represented by an 8-dimensional vector, where 1 means allowing the corresponding traffic flow to pass, and 0 means prohibiting the corresponding traffic flow to pass. Reward function: Each agent will receive an immediate reward from the environment , which is defined as a function of the queue length and fairness metric at the intersection , specifically defined as: ; wherein denotes the queue length of the incoming lane in the current intersection, denotes the fairness metric based on the additional waiting time at the time instant, is a hyper-parameter for balancing the importance between the model performance and the decision fairness; During the training process, the deep Q-network algorithm is adopted for model training, which includes a pair of Q-networks and target Q-networks , which are parameterized by and respectively; the time difference method is used for updating when updating the Q-network, and the loss function is: ； ； wherein, denotes a discount factor for calculating the cumulative reward to balance long-term and immediate rewards; The counterfactual data augmentation process specifically comprises: defining a set of observed variables, the definition of a structural causal model is where represents the parent variable of , represents noise, the function represents the causal mechanism, which can establish the relationship between , and ; in reinforcement learning, the agent performs an action under a given state , and then observes the subsequent state of the environment feedback; based on the structural causal model, the counterfactual result of the factual situation is inferred, that is, by inferring the impact of performing a random action in the same environment state , the predicted next moment state is obtained; assuming that the state at moment satisfies the structural causal model, it is defined as follows: ; wherein represents a causal mechanism function; and represent state and action variables, respectively; represents noise due to unobserved factors, which is independent of and ; Approximating the state transition function, i.e. the causal function, by a generative adversarial network ; alternately training the discriminator and the generator so that the generator generates fake samples that can deceive the discriminator, and meanwhile, the trained generator is used as the causal function to predict the next state based on the tuple The loss function of the generative adversarial network is defined as follows: ； ； wherein represents an interaction data tuple, containing a state , an action and a noise ; a function to distinguish between generated data and original data; a function represents a generative network for approximating the causal function ; a function represents a discriminator network for distinguishing between real interaction data and generated data; define and represent the distribution of the next state and interaction tuple, respectively; Given an interaction data , the action is replaced by the original action , and the next time environment state is predicted ; The action is randomly sampled from the action set, and then together with other original interaction information and into the generator network , output counterfactual results ; The reward value is calculated from the state information; The counterfactual data augmentation dataset is obtained , which is helpful to improve the sample efficiency of reinforcement learning method; The self-supervised state representation learning process specifically includes: The state encoder network adopts an attention mechanism to realize the cooperation between multiple traffic intersections, and the generated state representation vector is defined as follows: ; ； ； wherein, is a representation function; is a similarity function to evaluate the similarity between two inputs; represents the set of neighbor intersections of the current intersection ; attention weights are used to compute the final representation vector ; parameters , and are learnable parameters; The self-supervised state representation learning comprises a state encoder network, a target state encoder network and a transition network; wherein the state encoder network is represented as is responsible for mapping the state to a state representation vector , which is also used as input information of the Q-network; the target state encoder is represented as , which has the same function as the state encoder network, i.e. taking the original state as input and outputting a state representation vector , which has the same network structure; the transition network is represented as , taking the state representation vector and the action as joint input information, and then predicting the state representation vector of the next time, achieving the purpose of constructing positive examples in self-supervised learning; finally, the expected state representation and the original state representation will be used to calculate the loss function of self-supervised learning, which is defined as follows: ; wherein, represents the size of the batch data; in the case that the network parameters of the target state encoder network are updated in the manner of exponential moving average, the parameters of the reference state encoder network are referred to for updating, and the definition of the updating manner is as follows: ; wherein and respectively represent the parameters of the state encoder network and the target state encoder network. The final loss function contains the reinforcement learning objective function and the self-supervised representation learning objective function, which are jointly trained in a way of stochastic gradient descent. The total loss function is represented as: wherein is a hyper-parameter for adjusting the weight of the two objective functions; S300, the unfairness degree is introduced into the state vector and the reward function to realize fair traffic signal scheduling.

2. The multi-intersection traffic signal light fair control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The multi-intersection traffic signal control problem is described as: Given a multi-intersection traffic signal scheduling task, suppose there are traffic signals need to be controlled; each intersection is denoted as , an index value of the intersection, contains east, south, west, and north four-direction roads and is equipped with traffic signals controlled by reinforcement learning strategies; each road contains three lanes , according to the driving direction of the vehicle, and is divided into entry lane and exit lane , and represent the entry lane and exit lane set of intersection ; traffic flow represents the vehicle driving into the intersection from the entry lane , and then driving away from the exit lane , assuming there are three types of traffic flow: left turn, straight, and right turn, which are controlled by traffic signals; represents the combination of all traffic flows; traffic signal phase is defined as a set of traffic flows allowed to pass; the set of signal phases contains four traffic signal phases, i.e. , where represents the straight traffic flow in the east-west direction allowed to pass, represents the left turn traffic flow in the north-south direction allowed to pass, and the minimum execution time of each signal phase is ; During the period when a certain traffic light is triggered, other roads The number of vehicles waiting on the road is expressed as the length of the queue. For intersections In terms of queue length at intersections Defined as the sum of the lengths of all connected entry lane queues: On this basis, every The timing of traffic light phases at each intersection is adjusted according to traffic conditions to ultimately minimize the average travel time.

Images