Multi-agent reinforcement learning traffic signal control method based on strategy multiplexing

Abstract

The invention provides a multi-agent reinforcement learning traffic signal control method based on strategy multiplexing, and belongs to the technical field of traffic signal control. The method comprises the following steps: constructing a universal EDLight model of the intelligent agent; the general road network model is trained based on an existing TOD scene, and a learned environment model is stored; calculating the similarity between the unknown TOD target environment and the pre-training task environment; and based on the similarity, selecting a pre-training model of a similar environment to guide a target agent model to make a decision, and updating a target domain model based on probability sampling to realize autonomous decision. The optimal strategy is selected for the unknown target network model, so that the agent reinforcement learning model can migrate in the road network and across the road network. The problem that an existing model is insufficient in traffic signal control migration capacity is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic signal control, and in particular to a multi-agent reinforcement learning traffic signal control method based on strategy reuse. Background Art

[0002] With the development of urbanization, traffic congestion is becoming more and more serious. In order to alleviate the problem of traffic congestion, in recent years, the multi-agent real-time linear regression (MARL) method based on deep learning has made significant progress in the study of adaptive traffic signal control (ATSC) problem.

[0003] However, most of the research methods are aimed at training models in a certain time period (certain time of day, TOD) of predefined demand. When faced with unfamiliar TOD environments, the agents learned by these methods are usually unable to make effective decisions. That is, a well-trained model cannot make effective decisions when encountering some TOD scenarios that have never been experienced. This leads to the need to retrain the model in new scenarios to adapt to the new environment, which not only wastes computing resources, but also makes it difficult to deploy in real-world scenarios. This problem is caused by the restricted exploration problem encountered when training the agent model in predefined demand scenarios. Traffic simulation provides an environment for agent learning, and the agent interacts with the environment to gain "experience". The agent learns strategies based on the provided experience. However, since traffic simulation is basically based on a given demand scenario setting, it summarizes the traffic state in a certain restricted search space. Therefore, the agent learns strategies based only on partial experience generated by restricted exploration. This strategy usually cannot adapt to some unexplored and never-seen traffic states, resulting in decision failure. However, in real applications, traffic states are usually complex and changeable, and change in real time, which requires that the method should have sufficient generalization. In addition, in the real environment, continuous trial and error to accumulate experience to learn strategies and long training cycles are not allowed. It is necessary to respond quickly to environmental tasks in a short period of time, requiring the model to have extremely high training efficiency and convergence speed. Existing methods limit practical applications.

[0004] Therefore, there is a need for a traffic signal control method that can improve the model training efficiency and convergence speed, thereby improving the model generalization ability. Summary of the invention

[0005] In view of this, the present invention provides a multi-agent reinforcement learning traffic signal control method based on strategy reuse, which combines the ideas of transfer learning and reinforcement learning to solve the problems of long training time, low data sample utilization efficiency and weak generalization ability faced by traditional deep reinforcement learning in traffic signal control.

[0006] To this end, the present invention provides the following technical solutions:

[0007] A multi-agent reinforcement learning traffic signal control method based on strategy reuse, comprising the steps of:

[0008] Constructing a general EDLight model for an intelligent agent; the general EDLight model includes: an encoder for extracting and encoding observation information of the intersection at the current moment; a decoder for predicting the characteristic information of the intersection at the next moment, and storing the environment model of the current task as external knowledge to help other tasks; an action value network for scoring the phase action at the next moment and selecting the action with a high score for decision-making;

[0009] Pre-training the general EDLight model based on the existing TOD scenario, and using the pre-trained model set as a policy library; storing the target agent model to be optimized in the target task of the unknown TOD scenario that needs to be learned in the policy library; updating the target agent model to be optimized through the loss function, and updating the policy library as a new pre-trained model;

[0010] Calculate the similarity between the unknown TOD target environment and the pre-trained model environment; based on the similarity, select a guiding strategy from the strategy library through probability sampling to make a decision and control the traffic signal;

[0011] The target agent model is continuously updated, and the probability of being sampled gradually increases.

[0012] Furthermore, the pre-training of the general EDLight model based on the existing TOD scenario includes:

[0013] The encoder-decoder structure is used as a prediction module to learn the dynamic change model of the intersection environment;

[0014] Using the encoder-action value network architecture as a decision module to control traffic lights;

[0015] By minimizing the output of the prediction module, the average error of the original observation, and the temporal difference error of the value function of the decision module, the parallel optimization of policy storage and decision learning is achieved.

[0016] Furthermore, the encoder uses an attention mechanism to fuse feature information of its own intersection with feature information of neighboring intersections.

[0017] Furthermore, the target agent model to be optimized is updated by the loss function:

[0018]

[0019] in ω and θ correspond to the parameters of the encoder, decoder, and action-value network; and θ - are the parameters of the target network; B represents the batch size of sampling training; N represents the number of intersections in the road network; an agent is set at each intersection of the road network, and the agents share model parameters; is the intersection observation information of intersection i and its neighbors at the current moment, a i represents the action performed by the agent at intersection i at the current moment, o′ i represents the observation information of intersection i at the next moment; y i is the optimization target of the action value network of agent i at intersection; r i is the reward obtained by agent i at the current moment; γ is the discount factor; a ′ Indicates selecting the action with the maximum action value based on the observation information at the next moment.

[0020] Furthermore, the calculating of the similarity between the unknown TOD target environment and the pre-training task environment includes:

[0021] When the agent interacts with the unknown target environment, it obtains a trajectory of observed actions, with m time steps as one cycle;

[0022] Get the similarity of each strategy to the target task at each time step;

[0023] Through the similarity of each time step, the similarity weight of each strategy in the strategy library to the target task in one cycle is obtained.

[0024] Furthermore, obtaining the similarity of each strategy to the target task at each time step includes:

[0025] At time t, obtain the observed feature vector o of intersection i i and action a i , then the actions in stage t~t+m are given by the behavioral strategy of the guidance strategy of the previous stage and the combination of random actions with a very small probability;

[0026] The observed action pair at time t (o i ,a i ) is input into the strategy library. The strategy model in the strategy library stores the environment model of the corresponding task. Each model is based on the observed action at time t (o i ,a i ) to make predictions and obtain the predicted observation features at time t+1;

[0027] Use encoders to normalize observed features from different network predictions;

[0028] Calculate the Euclidean distance between the predicted feature vector at time t+1 and the corresponding true observed feature vector, and use the distance between the two to measure the similarity of this time step. The smaller the distance, the more similar the tasks.

[0029] Furthermore, the use of encoders to standardize observation features predicted from different networks includes:

[0030] The predicted observation features and the true observation features of each model are input into the average encoder of the model;

[0031] Among them, both the predicted observation features and the real observation features contain the neighbor information of intersection i at time t+1;

[0032] The neighbor feature information of the predicted feature is predicted at the corresponding neighborhood intersection through the environment model of the corresponding strategy;

[0033] The real observed neighbor information is obtained by environment feedback.

[0034] Furthermore, the similarity weight of each strategy in the strategy library to the target task in one cycle is obtained through the similarity of each time step, including:

[0035] Based on the time step similarity, calculate the similarity weight of each strategy in the strategy library to the target task:

[0036]

[0037] in, represents the Euclidean distance between the feature vector of the next moment predicted by the strategy of source domain k at time j and the real observed feature vector of the next moment, μ represents the discount factor; W k Indicates the similarity between the source domain k and the target task in the period t to t+m;

[0038] Get the similarity weight [W1,W2,...,W k ,W tar ].

[0039] Further, based on the similarity, a guiding strategy is selected from a strategy library through probability sampling to make a decision and control a traffic signal, including:

[0040] Using the similarity weights [W1,W2,...,W k ,W tar ], and the corresponding probabilities [P1,P2,...,P k ,P tar ], for [P1,P2,...,P k ,P tar] is sampled and a strategy is selected from the strategy library as the guiding strategy; the target model in the strategy library is continuously updated during the learning process, and the probability of being sampled gradually increases, thereby realizing the control of unknown TOD traffic signals.

[0041] Advantages and positive effects of the present invention:

[0042] The present invention uses a parameter sharing method to design a universal intelligent agent model, which can be applied in different road network settings and can meet the settings of more scenarios and requirements.

[0043] The attention mechanism is used to fuse the feature information of the own intersection with the feature information of the neighboring intersections, so as to obtain more comprehensive feature information and provide a more accurate basis for the decision-making and prediction of the intelligent agent.

[0044] According to the characteristics of traffic signal control problems, the task environment prediction model is used as knowledge for storage and utilization. When reusing strategies, probability sampling is performed based on similarity. As the target task agent is continuously trained and its cognition of the target environment is continuously enhanced, the probability of sampling will also increase, making the transition of the target agent model from expert strategy guidance to autonomous decision-making smoother and more stable.

[0045] The present invention helps the strategy learning of new tasks by utilizing the model of existing strategies; it can quickly adapt to new unknown tasks, accelerate the convergence of strategy models and improve their performance; it can be applied in multi-demand scenarios of the same road network tasks or cross-road network tasks, and solve the problem of insufficient migration capability of existing models for traffic signal control. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a framework diagram of the present invention;

[0048] Figure 2 It is a flow chart of the present invention;

[0049] Figure 3 It is a characteristic three-dimensional distribution diagram in an embodiment of the present invention;

[0050] Figure 4 This is a comparison result diagram of the model in the embodiment of the present invention in the Jinan and Hangzhou data sets;

[0051] Figure 5This is a comparison diagram of the model migration to the New York dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] The present invention provides a multi-agent reinforcement learning traffic signal control method based on strategy reuse to improve the generalization of the model; the present invention introduces transfer learning and proposes a MARL model EDLight-Trans based on strategy reuse. Specifically: when learning strategies, the agent uses the environmental model of the completed learning task as knowledge storage; when facing new tasks, it uses the trained strategy to reuse the strategy through the similar weights of the environmental model to achieve behavior transfer. The ability to extract decision-making knowledge from historical experience is obtained to help the target task agent to conduct effective exploration, thereby accelerating the learning of new tasks, improving training efficiency, and reducing training costs.

[0055] The present invention provides a universal intelligent agent model EDLight, which includes three modules: an encoder, a decoder, and an action value network (Q network).

[0056] Combination Figure 1 Invention structure diagram, for explanation:

[0057] Encoder, responsible for extracting the observation information of the intersection at the current moment and encoding it;

[0058] Specifically, in order to ensure that the agents in the intersection can cooperate effectively, the decision of the agent depends not only on its own observations, but also on the observations of its neighbors. Therefore, the encoder also needs neighbor information. The attention mechanism is used to organically integrate the feature information of its own intersection with the feature information of the neighboring intersections, so as to obtain more comprehensive feature information and provide a more accurate basis for the decision-making and prediction of the agent.

[0059] Decoder, used to predict the feature information of the next intersection and store the environment model of the current task;

[0060] Action value network (Q network), used to score the phase actions at the next moment and select actions with high scores for decision-making;

[0061] Specifically: After the model is trained in a certain TOD scenario, it can be applied as an expert strategy in other unknown TOD environments to help make decisions for the target task.

[0062] The specific training of the model is as follows: the Encoder-Decoder structure is used as the prediction module to learn the dynamic change model of the environment at the intersection; the Encoder-Q network architecture is used as the decision module to control the traffic lights; the parallel optimization of knowledge storage and decision learning is achieved by minimizing the average error between the output of the prediction module and the original observation and the time difference error of the value function of the decision module; the specific loss function of the intelligent agent is:

[0063]

[0064] in ω and θ correspond to the parameters of the encoder, decoder, and action-value network. i is the optimization target of the action value network of agent i at intersection; r i is the reward obtained by the agent at the intersection i at the current moment; γ is the discount factor, which makes the goal more focused on the long-term cumulative reward; a ′ Indicates selecting the action with the maximum action value based on the observation information at the next moment.

[0065] In order to ensure the stability of training, the target network is introduced. and θ - is the parameter of the target network. B is the batch size of the sampling training. N is the number of intersections in the road network. An agent is set at each intersection of the road network, and the agents share the model parameters. is the observation information of intersection i and its neighbors at the current moment, a i represents the action performed by the agent at intersection i at the current moment, o′ i Represents the observation information of intersection i at the next moment.

[0066] In the TOD environment where the application is unfamiliar and unknown, the present invention uses the EDLight model pre-trained in other scenarios as an expert strategy to help the intelligent model of the new environment learn. Since the pre-trained model saves the environmental model of the corresponding task during training, when facing a new environment, the similarity between the target environment and the pre-trained task environment can be compared, and the pre-trained model of the similar environment can be selected to guide the target intelligent model to make decisions. This migration method based on policy reuse is called EDLight-Trans.

[0067] The specific process of the migration method based on strategy reuse includes: similarity weight calculation, selection of guidance strategy based on similarity weight; Figure 2 To explain, the specific process is:

[0068] When the agent interacts with the target domain environment, a series of observation action trajectories can be obtained, and the trajectory is recorded as a cycle with m time steps. Taking the stage t~t+m as an example, at time t, the observation feature vector o of intersection i is obtained i and action a i , the actions in stage t~t+m are given by the behavior strategy of the guidance strategy of the previous stage and the combination of random actions with a small probability. The reason for adding random selection actions is to increase the diversity of experience samples.

[0069] Similarity weight calculation:

[0070] S1. The observed action pair (o i ,a i ) is input into the strategy library. The strategy model in the strategy library stores the environment model of the corresponding task (Encoder-Decoder forward dynamics model). Each model is based on the observed action pair (o i ,a i ) to make predictions and obtain the predicted observation features at time t+1.

[0071] The predicted observation features come from different networks and are mapped in different feature spaces, so they need to be standardized.

[0072] S2. Use Encoder to standardize features; that is, input the predicted observation features and real observation features of each model into the average Encoder of all models for standardization. Since the input of Encoder also requires the neighbor information of intersection i at time t+1, the neighbor feature information of the predicted features is predicted by the environmental model of the corresponding strategy at the corresponding neighborhood intersection. The real observed neighbor information can be obtained by environmental feedback. After the averaged Encoder standardization, the Euclidean distance between the predicted feature vector at time t+1 and the corresponding real observed feature vector is calculated, and the distance between the two is used to measure the similarity of this time step. The smaller the distance, the more similar the tasks.

[0073] Select guidance strategy based on similarity weight:

[0074] S3, calculating the cumulative discounted distance within the period, thereby obtaining the selection weight;

[0075] In the stage t~t+m, the similarity of each strategy to the target task is calculated at each time step. After obtaining the similarity of each time step, each strategy calculates the similarity weight by the following formula, taking the source domain strategy k as an example:

[0076]

[0077] in, represents the Euclidean distance between the feature vector of the next moment predicted by the strategy of source domain k at time j and the real observed feature vector of the next moment, and μ represents the discount factor.

[0078] Since the weights in the current cycle are used to guide the selection of strategies in the next cycle, the more important the later similarities are, the smaller the initial discount factor should be.

[0079] In this way, the cumulative discounted distance within the cycle is calculated. The smaller the cumulative discounted distance, the more similar the environment of the model k task is to the target task in the cycle. Taking the cumulative discounted distance as the opposite number, we get the selection weight W k , W k The size can reflect the similarity between the source domain k and the target task in the period t~t+m. Each strategy in the strategy library calculates the similarity weight [W1,W2,...,W k ,W tar ].

[0080] S4, the weights corresponding to each model are calculated through the softmax function to obtain the corresponding probability [P1, P2, ..., P k ,P tar ], sample according to the probability, and select a strategy from the strategy library as the guiding strategy.

[0081] Preferably, when the target task starts training, the similarity between the model in the policy library and the target task cannot be calculated because it has not yet interacted with the environment. Therefore, at time t=1, an agent other than the target agent is randomly selected from the policy library to interact with the environment. The model of the target domain has not been trained and cannot make effective decisions. In contrast, the target domain model in the policy library is constantly updated during the learning process and gradually adapts to the new environment, so the probability of being sampled will gradually increase.

[0082] In this embodiment, the specific model algorithm is:

[0083]

[0084] Specifically, the improvement of the model performance of the present invention is verified by combining real road network data, including:

[0085] Migration experiment within the road network: The agent migration experiment was conducted in two road networks, Jinan and Hangzhou. That is, the performance of this method was tested in the same scenario with different traffic distributions. Jinan has three traffic data sets and Hangzhou has four traffic data sets.

[0086] Cross-network migration experiment: After the model has learned the strategies of Jinan and Hangzhou, it is migrated to the New York road network to test whether the method has sufficient generalization to adapt to the new environment when facing an unknown road network environment.

[0087] There are eight types of traffic data sets as shown in Table 1:

[0088] Table 1

[0089]

[0090] In this embodiment, the micro traffic simulation software CityFlow is used to simulate the driving behavior of vehicles on urban roads in real time. It can support flexible definition of road networks and traffic flows based on synthetic data and real data. In order to meet the needs of diverse scenarios, the experiment is set up in three real road networks, each with a different distribution of traffic data sets, and each data set can simulate traffic flow simulation within 1 hour.

[0091] Taking the hz1 traffic data set as an example, the frequency of entering the road network is divided into three parts, that is, the 1-hour traffic flow simulation is divided into 3 time periods. In the first 20 minutes, vehicles enter the lanes of the road network every 7 seconds or so. Vehicles drive randomly in the road network with the corresponding probability in the table until they drive out of the road network. In 1 hour, 6,684 vehicles entered the Hangzhou road network.

[0092] The specific migration experiment within the road network is as follows:

[0093] In order to have a clear understanding of the discrimination of the data set in the road network, the traffic flow file is now visualized through PCA analysis. Under each traffic flow file, the feature vector after the vehicle enters the road network consists of two aspects of information, one is the distribution of the vehicle direction, and the other is the driving time of the vehicle entering the road network. The direction is one-hot encoded (straight is recorded as [1,0,0]), and the vector p is recorded every time the vehicle drives to the next lane. In order to distinguish the difference between each route, the feature vector at each turn needs to be multiplied by the corresponding discount factor σ. The vectors after each change of direction are accumulated to obtain the feature vector representing the distribution of the direction of the vehicle's driving route, recorded as:

[0094] v turn =∑ n＝0 σ n p n ;

[0095] Where n is the number of direction changes. The direction feature vector and the time when the vehicle enters the road network together constitute the driving feature vector v of a vehicle travel =[v turn ,t start ] According to the feature vector of the traffic flow, the feature vector is mapped to a three-dimensional graph, and the distribution information of the visualized data is as follows: Figure 3 As shown:

[0096] a. Three-dimensional distribution map of Jinan dataset features;

[0097] b. 3D distribution map of Hangzhou dataset features;

[0098] Each point in the figure is mapped in three-dimensional space by the feature vector of the path in the traffic flow, and each point can be regarded as a vehicle. For different traffic data sets under the same road network, the same vehicle path is randomly selected to extract feature vectors. In this embodiment, feature extraction is performed through PCA analysis.

[0099] The evaluation indicator of the experiment is the average driving time, which is the most commonly used evaluation indicator in traffic signal control problems and can effectively reflect the driving efficiency of traffic in the road network. The comparison algorithm selected two types of reinforcement learning algorithms, one is the value-based CoLight algorithm (CoLight: Learning Network-level Cooperation for Traffic Signal Control), and the other is the policy-based decentralized proximal policy optimization algorithm IPPO (Independent Proximal Policy Optimization). At present, most traffic signal control algorithms are improved based on these two methods. Both of them and this method are parameter sharing methods, and the specific test is the training efficiency and training speed, as well as the performance of the final algorithm.

[0100] like Figure 4 The figure shows the average driving time for 100 rounds of testing, which are set on the jn3 dataset of Jinan road network and the hz3 dataset of Hangzhou road network. EDLight is used as a pre-trained model and EDLight-Tran is used as a migration method. On the jn3 dataset, the expert strategies of EDLight-Tran come from jn1 and jn2; on the hz dataset, the expert strategies of EDLight-Tran come from hz1, hz2 and hz4. This method has excellent performance, fast convergence speed, better final performance, and smaller variance, which further shows that it has stronger adaptability and robustness.

[0101] The specific cross-network migration experiment is as follows:

[0102] The New York road network is a large 16×3 road network with 48 intersections. To test the generalization performance of this method in an unknown large environment, the strategy originally used in the Jinan and Hangzhou road networks was migrated to the New York road network.

[0103] In the migration across road networks, the strategies trained with the jn1 and hz1 datasets were used. The vehicle volumes of these two datasets were 6684 and 7831 respectively. In contrast, the vehicle volume of the New York road network reached 18005 vehicles, which is greater than the sum of the vehicle volumes of the first two datasets. In this way, the effectiveness and adaptability of this method in large-scale and complex environments are verified.

[0104] Verification results such as Figure 5 As shown in the figure, although all four methods can eventually converge to the optimal effect, this strategy shows excellent performance in the initial stage and only needs a few rounds of convergence to reach the final performance. In addition, from the line graph in the figure, it can be seen that this method shows very small variance throughout the experiment, without drastic fluctuations, and shows good stability. This stability shows that even if the strategy uses different road networks, it can still maintain high performance in different environments and conditions.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning traffic signal control method based on strategy reuse, characterized in that: Includes steps: Constructing a general EDLight model for an intelligent agent; the general EDLight model includes: an encoder for extracting and encoding observation information of the intersection at the current moment; a decoder for predicting the characteristic information of the intersection at the next moment, and storing the environment model of the current task as external knowledge to help other tasks; an action value network for scoring the phase action at the next moment and selecting the action with a high score for decision-making; Pre-training the general EDLight model based on the existing TOD scenario, and using the pre-trained model set as a policy library; storing the target agent model to be optimized in the target task of the unknown TOD scenario that needs to be learned in the policy library; updating the target agent model to be optimized through the loss function, and updating the policy library as a new pre-trained model; Calculate the similarity between the unknown TOD target environment and the pre-trained model environment; based on the similarity, select a guiding strategy from the strategy library through probability sampling to make a decision and control the traffic signal; The target agent model is continuously updated, and the probability of being sampled gradually increases.

2. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 1 is characterized in that: The pre-training of the general EDLight model based on the existing TOD scenario includes: The encoder-decoder structure is used as a prediction module to learn the dynamic change model of the intersection environment; Using the encoder-action value network architecture as a decision module to control traffic lights; By minimizing the output of the prediction module, the average error of the original observation, and the temporal difference error of the value function of the decision module, the parallel optimization of policy storage and decision learning is achieved.

3. According to the multi-agent reinforcement learning traffic signal control method based on strategy reuse in claim 1, it is characterized in that: The encoder uses an attention mechanism to fuse the feature information of its own intersection with the feature information of neighboring intersections.

4. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 1 is characterized in that: The target agent model to be optimized is updated by the loss function: in ω and θ correspond to the parameters of the encoder, decoder, and action-value network; and θ - are the parameters of the target network; B represents the batch size of sampling training; N represents the number of intersections in the road network; an agent is set at each intersection of the road network, and the agents share model parameters; is the intersection observation information of intersection i and its neighbors at the current moment, a i represents the action performed by the agent at intersection i at the current moment, o′ i represents the observation information of intersection i at the next moment; y i is the optimization target of the action value network of agent i at intersection; r i is the reward obtained by agent i at the current moment; γ is the discount factor; a ′ Indicates selecting the action with the maximum action value based on the observation information at the next moment.

5. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 1 is characterized in that: The calculating the similarity between the unknown TOD target environment and the pre-training task environment includes: When the agent interacts with the unknown target environment, it obtains a trajectory of observed actions, with m time steps as one cycle; Get the similarity of each strategy to the target task at each time step; Through the similarity of each time step, the similarity weight of each strategy in the strategy library to the target task in one cycle is obtained.

6. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 5 is characterized in that: The obtaining of the similarity of each strategy to the target task at each time step includes: At time t, obtain the observed feature vector o of intersection i i and action a i , then the actions in stage t~t+m are given by the behavioral strategy of the guidance strategy of the previous stage and the combination of random actions with a very small probability; The observed action pair at time t (o i ,a i ) is input into the strategy library. The strategy model in the strategy library stores the environment model of the corresponding task. Each model is based on the observed action at time t (o i ,a i ) to make predictions and obtain the predicted observation features at time t+1; Use encoders to normalize observed features from different network predictions; Calculate the Euclidean distance between the predicted feature vector at time t+1 and the corresponding true observed feature vector, and use the distance between the two to measure the similarity of this time step. The smaller the distance, the more similar the tasks.

7. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 6 is characterized in that: The encoder is used to standardize the observed features predicted by different networks, including: The predicted observation features and the true observation features of each model are input into the average encoder of the model; Among them, both the predicted observation features and the real observation features contain the neighbor information of intersection i at time t+1; The neighbor feature information of the predicted feature is predicted at the corresponding neighborhood intersection through the environment model of the corresponding strategy; The real observed neighbor information is obtained by environment feedback.

8. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 5 is characterized in that: The similarity weight of each strategy in the strategy library to the target task in one cycle is obtained through the similarity of each time step, including: Based on the time step similarity, calculate the similarity weight of each strategy in the strategy library to the target task: in, represents the Euclidean distance between the feature vector of the next moment predicted by the strategy of source domain k at time j and the real observed feature vector of the next moment, μ represents the discount factor; W k Indicates the similarity between the source domain k and the target task in the period t to t+m; Get the similarity weight [W1,W2,...,W k ,W tar ].

9. The multi-agent reinforcement learning traffic signal control method based on strategy reuse as described in claim 1 is characterized in that: Based on the similarity, a guiding strategy is selected from a strategy library through probability sampling to make a decision and control the traffic signal, including: Using the similarity weights [W1,W2,...,W k ,W tar ], and the corresponding probabilities [P1,P2,...,P k ,P tar ], for [P1,P2,...,P k ,P tar ] is sampled and a strategy is selected from the strategy library as the guiding strategy; the target model in the strategy library is continuously updated during the learning process, and the probability of being sampled gradually increases, thereby realizing the control of unknown TOD traffic signals.

Images