Offline reinforcement learning method and device for traffic signal control

Abstract

The invention provides an offline reinforcement learning method and device for traffic signal control. The method for generating the traffic control signal based on the off-line reinforcement learning comprises the following steps: acquiring the state of a real traffic intersection and providing the state to a trained action unit; the action unit outputs a traffic control signal in response to the state of the real traffic intersection; wherein the action unit is trained based on an offline reinforcement learning method through an offline data set D; wherein the step of training the action unit based on the off-line reinforcement learning method comprises multiple rounds of iteration, and in each round of iteration, a data small batch comprising a plurality of data samples is obtained from an off-line data set; updating # imgabs0 # by using the data small batch and providing the data to the action unit; and training the action unit by using a small batch of data to obtain a strategy pi theta (s) of the action unit for generating the traffic control signal.

Description

Technical Field

[0001] The present application relates to machine learning technology, specifically, a method OfflineLight and an information processing device for traffic signal control using offline reinforcement learning technology. Background Art

[0002] The traffic signal optimization problem can be expressed as a Markov decision process (MDP)<S,A,P,R,π,γ> Among them, the road state set S, the traffic signal control action set A, the road state transition probability P, the reward R, the control strategy π, and the discount factor γ (such as Figure 1 As shown in Figure 2). The traffic signal control agent learns the control strategy v and decides the control action a∈A for the traffic signal in a given state s∈S to achieve the maximum expected reward: Among them, G t is the expected return at time t, and γ is the discount factor. The traffic signal control agent is, for example, a traffic light control device or control system.

[0003] Machine learning technology based on reinforcement learning has been applied in traffic signal control. Chinese patent application CN107134156A provides an intelligent traffic light system based on deep learning and a method for controlling traffic lights. Chinese patent application CN109544913A provides a traffic light dynamic timing algorithm based on deep Q network learning. Chinese patent application CN113012432B provides a traffic light control system based on cloud computing and fog computing collaborative reinforcement learning.

[0004] Offline reinforcement learning has been proposed (see https: / / arxiv.org / abs / 2005.01643), where the agent learns and trains from a static dataset without interacting with the environment during training. In traditional reinforcement learning (also called online reinforcement learning), the agent is based on a policy π k Interact with the environment, obtain new data, and update the policy π k+1 In offline reinforcement learning, historical data is stored in an experience pool (data set). The data in the experience pool is generated based on the behavior strategy. During training, samples are sampled from the experience pool and the obtained samples are used to train the agent. Summary of the invention

[0005] The core difficulty of traditional reinforcement learning methods in urban traffic signal control is that they need to frequently interact with the environment online to collect reward information to train the agent. The interaction process requires a lot of "trial and error", which increases the training time, consumes a lot of computing resources, and also brings danger to the traffic intersection environment. Therefore, it is hoped that offline reinforcement learning technology can be applied to traffic signal control to solve one or more of the above problems encountered by traditional reinforcement learning technology in traffic signal control.

[0006] Furthermore, the network topology and traffic data distribution of each city and intersection are different, and retraining is required for different intersections. Therefore, how to improve the training efficiency and generalization ability of the urban traffic signal reinforcement learning model, reduce the cost of training "trial and error", and improve the generalization ability of the model under different intersections and traffic conditions is also a problem that this application hopes to solve.

[0007] In actual traffic signal control, most cities still use the fixed duration method from the 1950s and 1960s. Although cutting-edge reinforcement learning models have made great progress at this stage, they are limited by model training, scale and deployment costs and have not been widely used in urban road networks. Therefore, how to further compress the volume of reinforcement learning models, reduce the demand for computing resources, and enhance the practical application potential of urban traffic signal control algorithms without affecting the control effect is also a technical problem that this application hopes to solve.

[0008] In order to solve one or more of the above technical problems and obtain one or more of the above technical effects, various embodiments of the present application are provided.

[0009] According to a first aspect of the present application, a first method for generating a traffic control signal based on offline reinforcement learning according to the first aspect of the present application is provided, comprising: obtaining the state of a real traffic intersection and providing it to a trained action unit; the action unit outputs a traffic control signal in response to the state of the real traffic intersection; wherein the action unit is trained based on an offline reinforcement learning method using an offline data set D; wherein the training of the action unit based on the offline reinforcement learning method comprises multiple rounds of iterations, in each round of iterations: obtaining a data mini-batch including multiple data samples from the offline data set, wherein the data sample is represented by (s, a, s′, r), wherein s represents the state of the traffic intersection, a represents the behavior of generating a traffic control signal, s′ represents the new state after executing behavior a in state s, and r represents the reward for offline reinforcement learning; utilizing the data mini-batch according to the formula renew And provide it to the action unit, where α, β are hyperparameters, k is a positive integer, representing the kth iteration, represents mathematical expectation, Q(s,a) is the Q function used for reinforcement learning, Represents the Bellman function; using data mini-batch according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), λ is a hyperparameter, where in the kth iteration π β is the behavior strategy obtained from the data mini-batch, and Dis() is the distance function.

[0010] According to the first method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, a second method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application is provided, wherein the offline data set D is constructed by utilizing a traffic control data set and / or collecting data from real traffic intersections; and / or collecting data in a simulated traffic intersection environment to construct the offline data set D.

[0011] According to the second method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, there is provided a third method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, wherein data of a traffic control data set and / or data collected from a real traffic intersection are applied in a simulated traffic intersection environment; a traffic control signal is generated in the simulated traffic intersection environment for the applied data and data updated in response to the traffic control signal is collected from the simulated traffic intersection environment to construct the offline data set.

[0012] According to one of the first to third methods for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, a fourth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application is provided, wherein in the kth round of iteration, multiple rounds of sub-iterations are used to update multiple times After M rounds of iteration, the Provided to the action unit, where M is a positive integer; in each round of iteration, according to the formula renew

[0013] According to the fourth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, a fifth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application is provided, wherein in each round of iteration, the update With strategy π θ (s) using the same data mini-batch; and in each round of iteration, neither the state of the actual traffic intersection is obtained nor a traffic control signal is generated to the actual traffic intersection.

[0014] According to one of the first to fifth methods of generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, a sixth method of generating traffic control signals based on offline reinforcement learning in the first aspect of the present application is provided, wherein in each round of iteration, a disturbance φ is also applied to the behavior a in the corresponding relationship between the state s and the behavior a of the data mini-batch used in the current round, so that the learned strategy π θ (s) contains the corresponding relationship between state s and behavior a that does not exist in the used data mini-batch; and using the data mini-batch according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), where A is the set of values ​​of behavior a.

[0015] According to the sixth method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, a seventh method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application is provided, wherein in each round of iteration, according to the formula Update the perturbation generation method, where ε(ω)∈φ, ω is the parameter used to generate the perturbation ε(ω).

[0016] According to one of the first to seventh methods of generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, there is provided an eighth method of generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, wherein the action unit includes a strategy network π θ , θ is its network parameter, and the policy network π θ It is used to generate action a according to the input state s, representing the strategy π for generating traffic control signals θ (s); In each iteration, the policy network π is trained using a small batch of data θ , to update its network parameters θ; in each round of iteration, the network is trained using small batches of data To update the network Parameters Among them, the network Used to generate in the kth iteration In each round of iteration, the perturbation network ε is trained using a small batch of data ω , to update the perturbation network ε ω The parameter ω of the perturbation network ε ω Used to generate the disturbance ε(ω).

[0017] According to the eighth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, a ninth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application is provided, wherein a small batch of data is preprocessed by a shared preprocessing network to convert data samples of the small batch of data into a policy network πθ With the network Accepted formats.

[0018] According to the eighth or ninth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application, a tenth method for generating traffic control signals based on offline reinforcement learning in the first aspect of the present application is provided, further comprising: establishing a student network, wherein the size of the student network is smaller than that of the strategy network π θ ; Train the student network so that the policy network π θ The control strategy is migrated to the student network; in a round of training of the student network, the traffic intersection state s obtained from the offline data set is provided to the student network and the strategy network π respectively. θ ; Calculate the student network and the strategy network π θ The method comprises the following steps: the first step is to obtain a cross entropy of an output generated in response to each traffic intersection state s; the second step is to update the parameters of the student network to reduce the cumulative cross entropy generated in response to multiple traffic intersection states s; and the third step is to train the student network for multiple rounds so that the cumulative cross entropy generated is less than a specified threshold.

[0019] According to the tenth method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, there is provided an eleventh method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, further comprising: deploying the student network at a real traffic intersection; the student network outputs a traffic control signal in response to the state of the real traffic intersection.

[0020] According to the eleventh method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, there is provided a twelfth method for generating traffic control signals based on offline reinforcement learning according to the first aspect of the present application, which also includes: replicating the student network to deploy multiple student networks at multiple real traffic intersections respectively; each student network outputs a traffic control signal to the traffic intersection where it is located in response to the state of the real traffic intersection where it is located.

[0021] According to the second aspect of the present application, a method for training an action unit for outputting a traffic control signal based on a first offline reinforcement learning method according to the second aspect of the present application is provided, comprising: obtaining an offline data set D; training the action unit through multiple rounds of iterations based on the offline reinforcement learning method; in each round of iterations of training the action unit based on the offline reinforcement learning method: obtaining a small batch of data including multiple data samples from the offline data set, wherein the data sample is represented by (s, a, s′, r), wherein s represents the state of the traffic intersection, a represents the behavior of generating a traffic control signal, s′ represents the new state after executing behavior a in state s, and r represents the reward for offline reinforcement learning; using the small batch of data according to the formula

[0022] renew And provide it to the action unit, where α, β are hyperparameters, k is a positive integer, representing the kth iteration, represents mathematical expectation, Q(s,a) is the Q function used for reinforcement learning; using small batches of data according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), λ is a hyperparameter, where in the kth iteration π β is the behavior strategy obtained from the data mini-batch, and Dis() is the distance function.

[0023] According to the third aspect of the present application, an information processing device according to the third aspect of the present application is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a traffic control signal based on offline reinforcement learning according to the first aspect of the present application and the method for training an action unit for outputting a traffic control signal based on an offline reinforcement learning method according to the second aspect of the present application are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present application, together with the preferred mode of use and further objects and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A schematic diagram showing the existing technology of applying reinforcement learning technology in traffic signal control;

[0026] Figure 2A and Figure 2B The schematic diagrams of applying online reinforcement technology and offline reinforcement learning technology in traffic signal control are shown respectively;

[0027] Figure 3 A schematic diagram of an “action-comment” offline reinforcement learning framework according to an embodiment of the present application is shown;

[0028] Figure 4 A schematic diagram of an offline reinforcement learning system for traffic signal control according to an embodiment of the present application is shown;

[0029] Figure 5 A flowchart showing an offline reinforcement learning method for traffic signal control according to an embodiment of the present application is shown; and

[0030] Figure 6 A flowchart of a method for compressing a policy network for controlling traffic signals obtained through offline reinforcement learning according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0032] Figure 2A and Figure 2B Schematic diagrams showing the application of online reinforcement technology and offline reinforcement learning technology in traffic signal control, respectively.

[0033] See also Figure 2A , the agent at the traffic intersection is based on the strategy π k Interact with the environment at the traffic intersection, generate traffic control signals to control the traffic flow at the traffic intersection, and collect relevant data from the traffic intersection. i ,a i ,r i ,s′ i} (such as the number of vehicles, queue length, time it takes for vehicles to pass through the intersection, etc.), and based on the collected data, get an updated strategy π k+1 The updated policy π k+1 It is used for the next round of control of traffic signals at intersections and to obtain new interaction data.

[0034] See also Figure 2B In order to apply offline reinforcement technology, the data collection process of the traffic intersection environment is separated from the machine learning process, so that the machine learning process no longer relies on real-time data collection, nor does it need to wait for vehicles at the traffic intersection to respond to traffic control signals. Therefore, the offline reinforcement learning process does not need to be applied to real traffic intersections, nor does it need to interact with real traffic intersections, which helps to shorten the training time and eliminate the threat of the training process (untrained intelligent agents) to traffic intersection safety.

[0035] During the data collection phase, data from one or more real traffic intersections may be collected, including data under the control of existing traffic control signals and data under the control of humans or experienced experts. In order to enhance the adaptability and generalization of the trained agent, it is beneficial to obtain data from multiple intersections of different types and cities. Optionally or further, data is collected in a simulated traffic intersection environment. A simulated traffic intersection environment facilitates the generation of multiple traffic flow types and the application of traffic control signal generation strategies obtained by other means. Examples of simulated traffic intersection environments include CityFlow ( https: / / github.com / cityflow-project / CityFlowStill optionally, the data of the traffic intersection is combined with a simulated traffic intersection environment, the collected traffic intersection data is applied in the simulated traffic intersection environment, and a traffic control signal is generated in the simulated traffic intersection environment for the intersection state formed by the applied data (using expert control or automatic control implemented according to one or more control strategies), and further data updated in response to the traffic control signal is collected from the simulated traffic intersection environment.

[0036] The traffic control signal generation strategy embodied (or implied) in the collected training data set is also referred to as a behavior strategy. The collected data form a data set D. The data set D is also constructed using the existing traffic control data set, such as (open source data set https: / / traffic-signal-control.github.io). As an example, the data of data set D is represented by (s, a, s′, r), where s represents the state, a represents the behavior, s′ represents the new state after the behavior a is executed in the state s, and r represents the reward.

[0037] In the training phase, the agent is trained with the dataset D and iteratively updated to learn the policy π. During the learning process, the agent does not need to interact with the environment where the data was collected, nor does it need to interact with other traffic intersections.

[0038] The trained agent is deployed at a traffic intersection, where it collects real data from the environment, such as the state of the traffic intersection s, and generates an action a (for generating a traffic control signal) based on the learned strategy π. Since the dataset D in offline learning easily includes data from multiple environments and multiple intersections, the trained agent can be deployed at a variety of different traffic intersections, and has good versatility and generalization.

[0039] Figure 3 A schematic diagram of an “action-comment” offline reinforcement learning framework according to an embodiment of the present application is shown.

[0040] According to an embodiment of the present application, an "Action-Comment" (Actor-Critic) offline reinforcement learning framework (also called OfflineActor-Critic (OfflineAC)) for traffic control is provided. The framework includes an action unit (Actor) and a comment unit (Critic). As an example, the action unit generates a control strategy π for a traffic control signal, and the comment unit evaluates the pros and cons of the control strategy π by generating a Q value, where the Q value is, for example, a Q matrix in reinforcement learning, and its elements represent, for example, the reward and expectation from taking action a in state s to the final state. The action unit and the comment unit can each be implemented by software, hardware, firmware, or a combination thereof. Optionally, the action unit and the comment unit are each implemented as a neural network.

[0041] See also Figure 3 During offline learning, the action unit cyclically updates its strategy π (π = π′) based on the data obtained from the offline dataset D and the Q value obtained from the comment unit, but does not need to interact with the environment (no need to obtain data such as traffic intersection status, vehicle information, etc. from the external environment). The comment unit cyclically updates the Q value based on the data obtained from the offline dataset D and provides it to the action unit, and does not need to interact with the environment during this period.

[0042] The update of the Q value by the comment unit corresponds to the update of the strategy π by the action unit. The provision of the updated Q value by the comment unit to the action unit and the completion of an update of the strategy π by the action unit are called a round of iteration. In a round of iteration, the comment unit provides an updated Q value to the action unit, and the action unit updates the strategy π once with the updated Q value; the action unit and the comment unit use the same data from the data set D, and the combination of multiple copies of data from the offline data set D used in a round of iteration is called a mini-batch. In the multiple rounds of iterations of the training process, each mini-batch of data used comes from a data batch, which is part or all of the data selected from the data set D. It can be understood that in a round of iteration, the update of the Q value by the comment unit is further completed through multiple rounds of calculation. For the purpose of distinction, a round of update of the Q value by the comment unit in a round of iteration is called a sub-iteration.

[0043] After multiple rounds of iterations, the action unit and the comment unit complete learning. The strategy π output by the action unit that has completed learning is used to generate traffic control signals for real traffic intersections.

[0044] In one iteration, the action unit updates its strategy π based on the Q value provided by the comment unit and a small batch of data obtained from the dataset D using formula (1):

[0045]

[0046] in is the Q value from the comment unit. After the kth iteration, the Q value provided by the comment unit is also recorded as (See also formula (2) below) (the symbol k representing the round is omitted in formula (1)), s is the state of the traffic intersection in the current mini-batch obtained from the dataset D, and a is the strategy π learned by the action unit θ The action that maximizes the Q value of the comment unit is sampled, π θ (s) is the traffic signal control strategy that the action unit needs to learn, π β is the behavior policy obtained from the offline dataset (the current batch), and λ is a hyperparameter. θ and behavioral strategy π βThe distance function between them is designed as Dis(π θ ,π β ). It can be understood that for the purpose of clarity and simplicity, the dataset D is used in formula (1) to represent the data used in the review unit training process, and in one round of iteration, the training data used is a small batch of data sampled from the entire dataset. And the small batches of data used in each round of iteration can be the same or different.

[0047] In the next iteration, using another small batch of data, the comment unit and the action unit each learn a new With strategy π θ In each round of multiple iterations, the review unit only iteratively generates a Q value that is closer to the actual value based on the offline dataset D (data mini-batch).

[0048] Optionally, the state of the intersection s is expressed by the number of vehicles at the intersection, the number of vehicles in the intersection section, the length of vehicles at the intersection, and / or the effective pressure (EP) at the intersection. https: / / arxiv.org / abs / 2112.02336 ) provides the meaning of effective pressure (EP) and the method of obtaining effective pressure (EP).

[0049] Still optionally, in formula (1), Dis() is a distance function used to calculate the strategy π to be found θ and behavioral strategies β For example, the distance function is calculated using the Kullback-Leibler (KL) divergence or a neural network.

[0050] The comment unit only iterates and generates a Q value that is closer to the real value based on the offline dataset D (the current data batch). In order to solve the problem of data distribution offset causing generation strategy offset, it is necessary to constrain the action unit to generate a strategy similar to the dataset.

[0051] The review unit is used to solve the problem of Q value being often overestimated. Therefore, the Q function is designed to train the agent according to offline reinforcement learning. At the same time, in order to prevent the problem of overestimation of the Q function, the design of the Q function needs to be lower than the traditional value function. The specific design of the Q function is as shown in formula (2). The review unit uses formula (2) to update the Q value based on the current small batch data obtained from the data set D.

[0052]

[0053] in, and They represent the state s and behavior a obtained in the offline dataset D (the current data batch), α and β are hyperparameters, and k is a positive integer representing the kth iteration. Represents mathematical expectation; a~π θ (a|s) means that in strategy π θ Under the condition of state s, the probability distribution of generating behavior a, Q(s,a) is the Q function of reinforcement learning in the prior art, π θ π from the action unit update after the previous iteration θ ,and It is a Q function designed according to the embodiment of the present application. is the standard Bellman error, represents the Bellman function. For example, Among them, r(s,a) is the immediate reward currently obtained in reinforcement learning, Indicates that in state s', action a' is selected so that has a maximum value, is the output of the review unit in the k-1th round, and γ is a hyperparameter.

[0054] Formula (2) The term finds the minimum value of Q(s,a), so Formula The goal of the item is Approaching Q(s,a), this part will get closer and closer to 0 during the training process. Therefore, by designing the Q function represented by formula (2), its value is made lower than the traditional value function Q(s,a).

[0055] In one round of iteration, the review unit updates the Q value through multiple rounds of iterations (for example, M rounds, M is a specified positive integer). In each round of iteration in the kth round, it is updated according to formula (2): After completing the kth round of iterations, for example, M rounds, the Q value obtained is recorded as is the Q value outputted by the kth iteration.

[0056] Figure 4 A schematic diagram of an offline reinforcement learning system for traffic signal control according to an embodiment of the present application is shown.

[0057] Figure 4 The offline reinforcement learning system according to the embodiment of the present application includes an offline data set D, an optional preprocessing unit, a comment unit, an action unit and an optional scrambling unit.

[0058] The preprocessing unit is used to preprocess the data obtained from the offline dataset D. Since the comment unit and the action unit use the same data mini-batch during training, they can share the preprocessing unit to perform the same preprocessing on the same data mini-batch. In an optional embodiment, the comment unit and the action unit each perform data preprocessing without sharing the preprocessing unit.

[0059] The critic unit updates the Q value in each iteration (according to the above formula (2)) and provides it to the action unit. The action unit updates its strategy in each iteration (according to the above formula (1)). Based on its updated strategy, the action unit can generate behavior a in the specified state s for traffic signal control.

[0060] The scrambling unit is used to perturb the optional range of behavior a to enhance the diversity of behavior a. In an embodiment according to the present application, behavior a is, for example, an indication of the traffic control signal to be generated next (for example, a red light or a green light is generated next). For example, in the current data minibatch, state s1 is always paired with behavior a1 in the offline data set, and after passing through the scrambling unit, (s1, a2) may be obtained, that is, state s1 can also be paired with behavior a2, and there may be no corresponding relationship (s1, a2) in the current data minibatch. Optionally, the scrambling unit generates a random behavior or behavior value set as output based on the input state s. Still optionally, the scrambling unit outputs a random behavior or a set of behaviors that do not correspond to the input state s in the offline data set. As an example, ω represents the parameter of the scrambling unit, and φ represents the set of perturbations applied by the indication generated by the scrambling unit, where ε(ω)∈φ is recorded. After the scrambling unit is scrambled, the value set A of behavior a becomes A∪φ. Set A is, for example, a set of behaviors a corresponding to state s in the current data minibatch. Optionally, during training, the parameter ω of the scrambling unit is also updated. For example, a neural network is used to implement the scrambling unit, and ω is a parameter (set) of the neural network. During training, when the action unit samples the action a, the value set of the scrambled action a is used. Accordingly, formula (1) is modified as follows:

[0061] After multiple rounds of iterations (e.g., T rounds, T is a positive integer), the comment unit and the behavior unit are trained using the offline dataset, and the optimized strategy π is obtained in the behavior unit. θ The trained behavior units can be deployed at various traffic intersections. They obtain the status of traffic intersections through radar, cameras, communication networks, etc., and generate corresponding behaviors to indicate the traffic signals at the intersections.

[0062] As an optional implementation, the preprocessing unit, the comment unit, the action unit and the scrambling unit are respectively implemented as neural networks, and each neural network is trained by using the offline data set D. For example, the comment unit is trained with formula (2) as the target, the action unit is trained with formula (1) as the target, and the preprocessing network implementing the preprocessing unit is trained with the conversion from discrete data sets to the data format accepted by the comment unit / action unit as the target. Optionally, the neural network implementing the action unit is described as a policy network: π(s, a, θ) (abbreviated as π θ ), the policy network π(s,a,θ) takes θ as a parameter and generates an output a based on the input s. The neural network that implements the scrambling unit is described as the perturbation network: ε(s,a,φ,ω) (abbreviated as ε ω ), the perturbation network ε(s,a,φ,ω) takes ω as parameter and generates output φ based on input s and a. The neural network implementing the comment unit is described as It is used to constrain the Q value. is the network parameter. During training, the network is updated by updating the network parameters.

[0063] Figure 5 A flowchart of an offline reinforcement learning method for traffic signal control according to an embodiment of the present application is shown.

[0064] The offline reinforcement learning method for traffic signal control according to the embodiment of the present application is used to train the offline reinforcement learning system according to the embodiment of the present application. The offline reinforcement learning system includes at least a comment unit and an action unit. Initialize the policy network π of the comment unit θ , the network of action units and an optional perturbation network ε ω .

[0065] For training, a data batch (Batch, denoted as B) is obtained from the offline dataset D. In each round of iteration, a data mini-batch including N data samples is extracted from the data batch B, and the data samples are denoted as (s, a, r, s'). The review unit uses the data mini-batch to train the network according to formula (2) through multiple rounds of iterations. Make the network Output the constrained Q value, where k is the round of the current iteration. Sub-iterations are performed for example M rounds, where M is a specified positive integer. After M rounds of sub-iterations, the review unit obtains the updated Q value for the current round k (That is, formula (2) ) and provide it to the action unit.

[0066] Continue with the kth round of iteration. According to the perturbation network ε ωSet a perturbation for the action a of the action unit to enhance the behavioral diversity in the action unit's strategy. For example, expand the value set of the action a in the current data mini-batch according to the output φ of the perturbation network.

[0067] In the kth iteration, the actor also uses the data mini-batch and the received Q-value According to formula (1), the strategy network π is trained θ , from the trained policy network π θ Get the strategy π for traffic signal control θ (s).

[0068] Optionally, in the kth iteration, the perturbation network ε is also updated ω According to the formula

[0069]

[0070] Update the perturbation network ε ω The parameter ω.

[0071] The iteration is performed, for example, for T rounds, where T is a specified positive integer.

[0072] After T rounds of iterations, the trained policy network π θ As an action unit, it is deployed at the traffic intersection to implement traffic signal control. The state s is collected from the traffic intersection and provided to the policy network π θ , the policy network outputs policy π θ (s), used to control the traffic signal at the intersection.

[0073] In order to facilitate the deployment of trained action units at a large number and variety of traffic intersections, it is also desirable to reduce the demand for computing resources by the action units (e.g., compressing the strategy network π θ For this purpose, according to an embodiment of the present application, a method for compressing the strategy network π by knowledge distillation and migration is also provided. θ The scale of the policy network π θ The traffic signal control knowledge learned in the algorithm is transferred to the student network (correspondingly, the trained strategy network is used as the teacher network).

[0074] Figure 6 A flowchart of a method for compressing a policy network for controlling traffic signals obtained through offline reinforcement learning according to an embodiment of the present application is shown.

[0075] In order to facilitate the deployment of the policy network at traffic intersections, the traffic signal control strategy (knowledge) in the policy network is transferred to the student network through the knowledge distillation process. Figure 6 Embodiment of the knowledge distillation process.

[0076] The student network is a network that is smaller than the policy network used to implement the action unit. The network size represents, for example, the number of layers, the number of nodes, etc. The control strategy of the policy network is implemented or approximated by a smaller network by training the student network with the policy network, thereby achieving the effect of compressing the size of the network used.

[0077] In the process of training the student network, a corresponding relationship between the outputs of the student network and the policy network is established, and the corresponding relationship between the errors of the two networks is back-propagated, so that the student network can learn the behavior of the policy network.

[0078] The policy network is a neural network that has been trained through offline reinforcement learning according to an embodiment of the present application. Figure 6 The process shown can be optionally implemented in an offline manner, so that the state of the traffic intersection in the offline data set can be accepted as input without learning at the real intersection, thereby improving the learning efficiency. Optionally, it can also be implemented at the real intersection using the real state of the intersection Figure 6 The process shown.

[0079] See also Figure 6 , input the traffic intersection state s to the policy network, and obtain the output (control of traffic signal behavior) provided by the policy network in response to the state s (denoted as a t The same traffic intersection state s provided to the policy network is provided to the student network to be trained as input, and the output provided by the student network in response to the state s (denoted as a s ). Calculate the output a of the policy network t The output of the student network is s For example, using cross entropy as the output of the policy network a t The output of the student network is s The difference between the states of the policy network and the student is repeatedly provided and the difference between their outputs is accumulated. The student network parameters are optimized to find the student network parameters that reduce or minimize the accumulated difference. For example, the student network is optimized by reducing the accumulated cross entropy using the gradient descent method. Whether the training of the student network is completed is determined based on whether the accumulated cross entropy is less than a threshold. When the accumulated cross entropy is not less than the specified threshold, the above training process is repeated until the accumulated cross entropy is less than the specified threshold, and a trained student network is obtained.

[0080] The trained student network includes a parameter set and can be copied to other student networks. The trained student network is deployed in the action unit of each traffic intersection to generate a control strategy for the traffic signal according to the state of the intersection. The size of the student network is smaller than the policy network trained in offline reinforcement learning, which makes it easier to deploy in the information processing equipment at the traffic intersection.

[0081] Although the present application is described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application.

[0082] Those skilled in the art to which these embodiments relate and who benefit from the teachings presented in the above description and associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that the present application is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used only in a general and descriptive sense and are not used for the purpose of limitation.

Claims

1. A method for generating traffic control signals based on offline reinforcement learning, comprising: Obtain the status of the actual traffic intersection and provide it to the trained action unit; The action unit outputs a traffic control signal in response to the state of the real traffic intersection; Wherein, the action unit is trained by an offline data set D based on an offline reinforcement learning method; The training of the action unit based on the offline reinforcement learning method includes multiple rounds of iterations, in each round of iteration: Obtain a data mini-batch including multiple data samples from an offline dataset, where the data sample is represented by (s, a, s′, r), where s represents the state of the traffic intersection, a represents the behavior of generating a traffic control signal, s′ represents the new state after the behavior a is executed in the state s, and r represents the reward for offline reinforcement learning; Using data mini-batches according to the formula renew And provide it to the action unit, where α, β are hyperparameters, k is a positive integer, representing the kth iteration, represents mathematical expectation, Q(s,a) is the Q function used for reinforcement learning, represents the Bellman function; Using data mini-batches according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), λ is a hyperparameter, where in the kth iteration π β is the behavior strategy obtained from the data mini-batch, and Dis() is the distance function.

2. The method according to claim 1, wherein Using traffic control data sets and / or collecting data from real traffic intersections to construct the offline data set D; and / or The offline data set D is constructed by collecting data in a simulated traffic intersection environment.

3. The method according to claim 2, wherein The offline data set is constructed by applying data of a traffic control data set and / or data collected from a real traffic intersection in a simulated traffic intersection environment; generating a traffic control signal for the applied data in the simulated traffic intersection environment and collecting data updated in response to the traffic control signal from the simulated traffic intersection environment.

4. The method according to any one of claims 1 to 3, wherein In the kth iteration, multiple rounds of iterations are used to update After M rounds of iteration, the Provided to the action unit, where M is a positive integer; In each round of iteration, according to the formula renew 5. The method according to any one of claims 1 to 4, wherein In each round of iteration, a perturbation φ is also applied to the behavior a in the corresponding relationship between the state s and behavior a of the data mini-batch used in the current round, so that the learned strategy π θ (s) contains the corresponding relationship between state s and behavior a that does not exist in the used data mini-batch; and using the data mini-batch according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), where A is the set of values ​​of behavior a.

6. The method according to any one of claims 1 to 5, wherein The action unit includes a policy network π θ , θ is its network parameter, and the policy network π θ It is used to generate action a according to the input state s, representing the strategy π for generating traffic control signals θ (s); In each round of iteration, the policy network π is trained using a small batch of data θ , to update its network parameters θ; In each round of iteration, the network is trained using small batches of data To update the network Parameters Among them, the network Used to generate in the kth iteration In each round of iteration, the perturbation network ε is trained using a small batch of data ω , to update the perturbation network ε ω The parameter ω of the perturbation network ε ω Used to generate the disturbance ε(ω).

7. The method according to claim 6, further comprising: Establish a student network, the size of which is smaller than the strategy network π θ ; Train the student network so that the policy network π θ The control strategy is migrated to the student network; In a round of training for the student network, the traffic intersection state s obtained from the offline data set is provided to the student network and the strategy network π respectively. θ ; Calculate the student network and the strategy network π θ The cross entropy of the output generated by each response to the traffic intersection state s; and updating the parameters of the student network to reduce the cumulative cross entropy generated in response to multiple traffic intersection states s; The student network is trained for multiple rounds so that the resulting cumulative cross entropy is less than a specified threshold.

8. The method according to claim 7, further comprising: duplicating the student network to deploy the plurality of student networks at a plurality of real traffic intersections respectively; Each student network outputs a traffic control signal for the traffic intersection where it is located in response to the state of the real traffic intersection where it is located.

9. A method for training an action unit for outputting a traffic control signal based on an offline reinforcement learning method, comprising: Obtain offline data set D; The action unit is trained through multiple rounds of iterations based on an offline reinforcement learning method; In each round of iterations of training the action unit based on the offline reinforcement learning method: Get a small batch of data including multiple data samples from the offline dataset, where the data samples are represented by (s, a, s ′ ,r) represents, where s represents the state of the traffic intersection, a represents the behavior of generating traffic control signals, s' represents the new state after executing behavior a in state s, and r represents the reward used for offline reinforcement learning; Using data mini-batches according to the formula renew And provide it to the action unit, where α, β are hyperparameters, k is a positive integer, representing the kth iteration, represents mathematical expectation, Q(s,a) is the Q function used for reinforcement learning; Using data mini-batches according to the formula Get the strategy π used by the action unit to generate traffic control signals θ (s), λ is a hyperparameter, where in the kth iteration π β is the behavior strategy obtained from the data mini-batch, and Dis() is the distance function.

10. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Images