Traffic signal control method based on reinforcement learning of maximum entropy framework

Abstract

The application relates to the technical field of traffic signal control, and discloses a traffic signal control method based on reinforcement learning of a maximum entropy framework, which comprises the following steps: constructing a simulation environment; constructing a reinforcement learning method based on a maximum entropy framework, and training an intelligent agent in the simulation environment through the reinforcement learning method based on the maximum entropy framework; and controlling a traffic signal through the trained intelligent agent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic signal control in general. In particular, the present application relates to a traffic signal control method based on reinforcement learning of maximum entropy framework. BACKGROUND

[0002] With the process of urbanization, the problem of traffic congestion becomes increasingly serious. Traffic congestion not only wastes fuel, but also increases the emission of greenhouse gases, which will cause many adverse environmental impacts, such as causing acid rain, forest death or atmospheric visibility reduction.

[0003] Traffic signal control can effectively alleviate traffic congestion. Traditional traffic signal control methods include traffic signal control methods based on Webster model and HCM model. More mature traffic control systems include SCOOT system and SCATS system.

[0004] In addition, with the development of deep learning, some researches propose to apply deep reinforcement learning (DRL) to the field of traffic signal control. In which, the DRL is used to control traffic lights. Through direct interaction with the environment, the agent can analyze the traffic state to adjust the traffic signal plan, thereby improving the traffic efficiency.

[0005] However, the traditional traffic signal control method has the problem of relying on a given traffic model or relying on expert knowledge. And the traditional traffic control systems such as SCOOT system and SCATS system also have the problem of being difficult to adapt to dynamic traffic control. Although the existing traffic signal control method based on reinforcement learning does not rely on traffic model or expert knowledge, the action design of the agent still has defects. SUMMARY

[0006] To at least partially solve the above problems in the prior art, the present application proposes a traffic signal control method based on reinforcement learning of maximum entropy framework, comprising the following steps:

[0007] constructing a simulation environment;

[0008] constructing a reinforcement learning method based on maximum entropy framework, and training an agent in the simulation environment through the reinforcement learning method based on maximum entropy framework; and

[0009] controlling traffic signals through the trained agent.

[0010] In an embodiment of the present application, it is provided that constructing a simulation environment comprises:

[0011] constructing an observation state of the agent, constructing an action of the agent, and constructing a reward of the agent;

[0012] The observation state of the agent is constructed as follows:

[0013] The traffic flow q, the average queue length l, and the signal plan P of the last cycle are taken as the observation information, wherein the signal plan P is expressed as follows:

[0014] P = <g 1 , …, g M >

[0015] The signal plan P is composed of M phases, g i represents the green light duration of the i-th phase; and

[0016] The environment observation vector s t is expressed as follows:

[0017] s t = {q 1 … q i … q N ; l 1 … l i …, l N ; P} t-1 ,

[0018] q i represents the traffic flow on lane i, and l i respectively represents the average queue length on lane i.

[0019] In one embodiment of the present application, the agent is constructed to perform the action of fine-tuning the duration of each phase, wherein the agent is allocated to each phase, the green light duration of each phase is adjusted by the agent alone, and a plurality of agents are combined to form a cycle and executed in a fixed order.

[0020] In one embodiment of the present application, the reward of the agent is expressed as follows:

[0021]

[0022] ω l ∈ (-1, 0)

[0023] wherein ω l represents a weight, and l0 is a constant.

[0024] In one embodiment of the present application, the reinforcement learning method based on the maximum entropy framework comprises the following steps:

[0025] In The Q value, representing the phase and calculated in parts, is expressed as follows:

[0026]

[0027] in Calculated by assuming phase independence. Represents the residual term;

[0028] KL-divergence measurement strategy and strategy The similarity is calculated by finding the maximum value of the first equation, which is expressed as:

[0029]

[0030] The total reward G at time t t It can be expressed as the following formula:

[0031]

[0032] Will It can be expressed as the following formula:

[0033]

[0034] Will Transform the following formula:

[0035]

[0036] Based on the principle of maximum entropy, let... The following formula is obtained:

[0037]

[0038] Where the KL-divergence value is the information entropy. With the number of actions difference;

[0039] The difference ΔQ between Q and Q is calculated using the following formula:

[0040] ΔQ=E s [r t +γ(E a [Q t+1 -τlogπ t ])]-Q t ;

[0041] Simplifying ΔQ to obtain a prediction of the current state value is expressed as the second equation:

[0042]

[0043] In one embodiment of the application it is specified that the error is calculated according to the second formula and the Q-value is updated, wherein the probability distribution is defined based on the Q-value and the Boltzmann distribution, denoted as the following formula:

[0044] and

[0045] The error is normalized and clipped by the moving average μ and the variance σ, denoted as the following formula:

[0046] δ t = Q(s t , a t ) - y t ; and

[0047] The loss function is denoted as the following formula:

[0048]

[0049] The application further provides a computer system, characterized in that comprising:

[0050] a processor configured to execute machine executable instructions; and

[0051] a memory having stored thereon machine executable instructions, said machine executable instructions, when executed by the processor, executing the steps of the method according to any one of claims 1-6.

[0052] The application further provides a traffic signal control system, characterized in that comprising:

[0053] a signal light; and

[0054] a computer system according to claim 7, said computer system being configured to control said signal light.

[0055] The application has at least the following beneficial effects: the application adopts the action of fine-tuning the length of each phase, wherein the action is issued once in a cycle, and only the green light duration of the phase is fine-tuned each time. Not only is it robust, but it is also insensitive to data noise and data transmission delay. At the same time, this action design also takes other traffic participants (such as pedestrians and cyclists) into account, and there is no problem of a green light phase being too long or too short. And through the reinforcement learning method based on the maximum entropy framework, the method not only considers the relationship between different phases, but also uses the maximum entropy method to control the balance between exploration and utilization, so that when using the action of fine-tuning the length of each phase, not only is the robustness, safety and comfort maintained, but the traffic efficiency is also improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] To further clarify the advantages and features of the present embodiments, a more particular description of embodiments of the application will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the application and are therefore not to be considered limiting of its scope. The application will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0057] Figure 1 A computer system implementing the traffic signal control method based on the reinforcement learning of the maximum entropy framework is shown.

[0058] Figure 2 A schematic diagram of the actions performed by the agent in one embodiment of the application is shown.

[0059] Figure 3 A flowchart of the traffic signal control method based on the reinforcement learning of the maximum entropy framework in one embodiment of the application is shown.

[0060] Figure 4 A schematic diagram of the agent controlling the traffic signal by performing the action of fine-tuning the length of each phase in one embodiment of the application is shown.

[0061] Figure 5A A schematic diagram of the topological structure of an intersection in one embodiment of the application is shown.

[0062] Figure 5B A schematic diagram of the traffic flow distribution of the simulated traffic dataset in one embodiment of the application is shown.

[0063] Figure 6A A schematic diagram of the actions performed by the agent in a traffic signal control method based on deep reinforcement learning is shown. DETAILED DESCRIPTION

[0064] It should be noted that the components in the various figures can be shown exaggerated in number and / or size in order to illustrate the disclosure, and are not necessarily drawn to scale. In the various figures, identical or similar components are often labeled with identical or similar numbers.

[0065] In the present disclosure, unless specifically indicated otherwise, "arranged on", "arranged above", and "arranged over" do not exclude the presence of an intermediate object between them. In addition, "arranged on or above" only indicates the relative position relationship between the two components, and in some cases, such as after reversing the product direction, it can also be converted to "arranged below or below", and vice versa.

[0066] In the present disclosure, the embodiments are only intended to illustrate the solutions of the present disclosure and should not be understood as limiting.

[0067] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0068] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0069] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0070] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0071] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0072] Figure 1 A computer system 100 for implementing the method according to the invention is shown. Unless otherwise specified, the method according to the invention can be implemented in... Figure 1 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0073] like Figure 1As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard drive interface 109, an optical drive interface 113, a network interface 114, a universal serial bus (USB) interface 112. The system bus 101 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system bus 101 is used for communication of information between each of the aforementioned devices. In addition to the bus devices or interfaces shown in FIG. 1, other bus devices or interfaces can be utilized. Figure 1 The system memory 102 includes read only memory (ROM) 103 and random access memory (RAM) 104, where the ROM 103 can store, for example, basic input / output system (BIOS) data for implementing basic routines for transferring information when the computer system 100 is booted, and the RAM 104 is used for providing the system with faster access speed for running memory. The computer system 100 also includes a hard drive interface 109 for reading and writing to a hard disk 110, an optical drive interface 113 for reading and writing to an optical medium such as a CD-ROM, and the like. The hard disk 110 can store, for example, an operating system and application programs. The drive and its associated computer readable medium provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computer system 100. The computer system 100 can also include a video adapter 105 for image processing and / or image output, for connecting to a display 106 or the like output device. The computer system 100 can also include an audio adapter 107 for audio processing and / or audio output, for connecting to a speaker 108 or the like output device. In addition, the computer system 100 can also include a network interface 114 for network connectivity, where the network interface 114 can connect to the Internet 116 through a network device such as a router 115, where the connection can be wired or wireless. In addition, the computer system 100 can also include a universal serial bus interface (USB) 112 for connecting peripheral devices, including, for example, a keyboard 117, a mouse 118, and other peripheral devices, such as a microphone, a camera, and the like.

[0074] When the present application is implemented in software, the software Figure 1The computer system 100 adopts the action of fine-tuning the length of each phase, which is performed once in a cycle and only fine-tunes the length of the green light phase each time. It is not only robust to data noise and data transmission delay, but also takes other road users (such as pedestrians and cyclists) into account, and does not have the problem of a certain green light phase being too long or too short. Moreover, the method is trained based on the maximum entropy framework, which not only considers the relationship between different phases, but also uses the maximum entropy method to control the balance between exploration and utilization, so that when the action of fine-tuning the length of each phase is used, not only good robustness, safety and comfort are maintained, but also the traffic efficiency is improved.

[0075] In addition, embodiments can be provided as computer program products that can include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, network of computers, or other electronic devices, can cause the one or more machines to perform the operations of embodiments in accordance with the present application. A machine-readable medium can include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (compact disk-read only memories), and magneto-optical disks, ROMs (read only memories), RAMs (random access memories), EPROMs (erasable programmable read only memories), EEPROMs (electrically erasable programmable read only memories), magnetic or optical cards, flash memory, or other type of media / machine-readable medium suitable for storing machine-executable instructions.

[0076] In addition, embodiments can be downloaded as a computer program product, where from a remote computer (e.g., server) to a requesting computer (e.g., a client), a program can be transferred via a communication link (e.g., a modem and / or network connection) implemented with or modulated by a carrier wave or other transport mechanism. Accordingly, a machine-readable medium, as used herein, can include such a carrier wave but is not limited thereto.

[0077] In the deep reinforcement learning-based traffic signal control method, the intelligent agent generally analyzes the traffic state to adjust the traffic signal plan.

[0078] Figure 6A -C shows a schematic diagram of the action performed by the intelligent agent in a deep reinforcement learning-based traffic signal control method.

[0079] As Figure 6A shown, the intelligent agent is configured to perform the action of setting the next phase (Set Next Phase), that is, to select a phase from all possible phases at certain time intervals;

[0080] AsFigure 6B As shown in the figure, the agent is configured to perform the action of switching or keeping (Next or Not), that is, to determine whether to switch to the next phase or keep the current phase after a certain time interval;

[0081] As shown in the figure, the agent is configured to perform the action of setting phase duration (Set Duration), that is, to set the duration of the phase at the beginning of each phase. Figure 6C

[0082] The action of setting the phase of the next stage can be widely used in single intersections and large-scale urban networks due to its flexibility, and can flexibly adjust the traffic signal plan because it can select any phase within a short time interval.

[0083] However, the action of setting the phase of the next stage breaks the traditional traffic cycle concept, which may confuse drivers. At the same time, this action design ignores other traffic participants, such as pedestrians and cyclists, who may violate the signal (i.e., jaywalking) due to long waiting time, causing unsafe situations. The switching or keeping action and the setting phase duration action, although maintaining the concept of signal light cycle, tend to focus on one side of the intersection with the largest traffic flow, while ignoring the other side. Therefore, they also make pedestrians and cyclists wait for a long time to cross the road.

[0084] In addition, the actions performed by the above agent are to make the agent directly take actions according to state observations within a short time, which reduces the stability of the traffic signal system, because the agent is sensitive to observations, and once data transmission problems occur, it is easy to cause the agent to make unreasonable actions.

[0085] To solve the above technical problems, the present application provides a traffic signal control method based on reinforcement learning of maximum entropy framework, which uses a reinforcement learning method based on maximum entropy framework (MD-SQN) and proposes a new action, wherein the duration of each phase can be fine-tuned to adjust the duration of the phase, thereby improving the traffic capacity without reducing safety and comfort.

[0086] Figure 3 The figure shows the flowchart of the traffic signal control method based on reinforcement learning of maximum entropy framework in one embodiment of the present application, as shown in the figure, the method can include the following steps: Figure 3

[0087] Step 301, construct a simulation environment.

[0088] ​​Step 302, constructing a reinforcement learning method based on a maximum entropy framework, and training an agent in the simulation environment through the reinforcement learning method based on the maximum entropy framework; and

[0089] Step 303, controlling the traffic signal through the trained agent.

[0090] In step 301, the actual road conditions can be simulated to train the agent by constructing a simulation environment, wherein the constructing a simulation environment comprises constructing an observation state, an action and a reward of the agent.

[0091] When constructing the observation state of the agent, in order to migrate the agent to the real environment, it is necessary to ensure that the information obtained in the simulation environment can also be obtained in the real intersection.

[0092] Traffic flow (q), average queue length (I) and last period signal scheme (P) can be used as observation information, which can be easily obtained through monitoring cameras or inductive coils at the intersection.

[0093] Wherein the signal scheme P can be expressed as follows:

[0094] P = <g 1 , …, g M >

[0095] It is composed of M phases, and each g i represents the green light duration of the i-th phase.

[0096] The observation vector of the environment can be expressed as follows:

[0097] s t = {q 1 … q i … q N ; l 1 … l i …, l N ; P} t-1 ,

[0098] Where q i and l i represent the traffic flow and average queue length on lane i, respectively.

[0099] When constructing the action of the agent, the following factors should be considered for a reasonable signal scheme in reality: it should have the concept of period, considering both the habits of drivers and the safety of pedestrians; it can handle data transmission delay; keep the system stable, which means the green light time of each stage should not change significantly between periods.

[0100] Based on these factors, we propose an action that fine-tunes the duration of each phase.Figure 2 An illustration of the actions performed by the agents in one embodiment of the application is shown. As shown, the agents fine-tune the duration of each phase, thus maintaining the concept of a cycle. This design also takes into account the habits of the drivers and other road users, and is able to handle potential data transmission delays. In this action, each agent decides the duration of its own phase, multiple agents jointly control the duration of all phases in a cycle, and the action is executed at the beginning of the next cycle. Figure 2

[0101] The action of fine-tuning the duration of each phase can assign an agent to each phase of the intersection, and each agent can adjust the green light duration of its corresponding phase. All agents are combined to form a cycle, and are executed in a fixed order to meet the concept of a signal cycle.

[0102] Specifically, each agent can individually decide whether to expand, reduce, or maintain the green duration of the corresponding phase.

[0103] Figure 4 An illustration of the control of traffic signals by the agents in one embodiment of the application is shown.

[0104] In one intersection, the duration of each phase is controlled by four agents. The action of each agent is one of { -1, 0, 1}, where each number represents {reduce, maintain, increase} the green duration of the single phase, respectively.

[0105] All actions from the agents are integrated and implemented together at the end of the last cycle. This is to minimize data quality and data transmission problems, while maintaining the overall stability of the entire cycle. In addition, at each time step, the agent can only increase or decrease the green duration by a preset amount of time, usually 3-5 seconds. Therefore, the green duration will not change too much between cycles. At the same time, the green duration also has a predetermined maximum and minimum time to consider other road users.

[0106] In constructing the reward, the average queue length in a cycle can be weighted as the reward, where the weight ω l ∈(-1, 0). Other indicators such as waiting time, travel time, etc. will not be rewarded, because these values are difficult to obtain in practice through detectors. The final reward can be represented as follows, where the original queue length is reduced by a constant l0, so that the distribution of the reward has both positive and negative to speed up the training of the agent:

[0107] ​

[0108] In step 302, since each phase needs an agent to control, if a crossroad has four phases, four agents are needed. This converts the problem into a "multi-dimensional integer discrete action space problem". It is difficult to solve this problem directly, in order to simplify the solving process and consider the relationship between different agents, the application proposes a reinforcement learning based on a maximum entropy framework to solve the above problem.

[0109] The reinforcement learning method based on the maximum entropy framework can include the following steps:

[0110] Firstly, let be the Q value of the actual phase, but since multiple phases affect each other, it is not easy to calculate directly. Therefore, it is divided into two parts, and as shown in the following formula:

[0111]

[0112] wherein it can be calculated by assuming that each phase is independent; is a residual term, used to process the inaccuracy of the estimate due to not considering the relationship between phases.

[0113] At this time, we hope the larger the better, but at the same time, the difference between the strategies and should not be too large. The KL-divergence can be used to measure the similarity between the two strategies. At this time, the problem is converted to find the maximum value of the following formula:

[0114]

[0115] Using the above formula, the expression of the total return G t at time t can be obtained:

[0116]

[0117] Because can be written as , so can be converted to the following formula:

[0118]

[0119] But since is unknown, the above formula cannot be solved. According to the maximum entropy principle, it can be assumed that is brought in and simplified to obtain wherein the KL-divergence value is the information entropy Difference in the number of actions The difference in Q can be obtained by using the equation:

[0120] ΔQ = E s [Q t -τlogπ a [Q t+1 -τlogπ t ])]-Q t .

[0121] Further simplifying the above equation, the prediction of the current state value can be obtained, and the calculation equation is as follows:

[0122]

[0123] With the above equation, the error can be calculated to update the Q-value. In the problem of “multi-dimensional integer discrete action space”, instead of using the absolute strategy, the Q-value and the Boltzmann distribution can be used, and the optimal strategy is defined as a probability distribution, as follows:

[0124]

[0125] In order to reduce the variance and improve the stability in the method, the original loss calculation can be modified. In the new loss calculation, a moving average μ and a variance σ can be maintained to normalize and clip the error:

[0126] δ t = Q(s t , a t )- y t .

[0127] The final loss function becomes:

[0128]

[0129] It should be noted that when calculating the loss, there is a hyperparameter τ. This parameter is used to control the exploration and utilization of the model. When the value of τ is large, the model tends to explore, and vice versa. In order to automatically adjust the value of τ, a target entropy loss can be set, and then the value of τ is automatically adjusted by minimizing the loss:

[0130]

[0131] In step 303, after the training of the agent is completed, the trained agent can be deployed in the actual road to control the traffic lights, thereby improving the traffic conditions of the road and reducing the travel delay.

[0132] The input of the agent includes: the traffic flow features extracted from the video stream of the intersection camera, including the traffic flow and queue length of each lane. At the same time, the green light duration of each phase in this period is also part of the input. The output of the agent includes: the scheme of returning the green light duration of each phase, which is larger, smaller or unchanged. For example, the current signal lamp consists of four phases, and the duration of each phase is {40, 20, 40, 20}, at this time the model returns the action {0, 1, -1, 0}, we adjust 5s each time, so at the end of the current period a new scheme will be issued, and the scheme duration is {40, 25, 35, 20}.

[0133] In addition, in the embodiments of the present application, the effect of the traffic signal control method can be evaluated by the following indicators, among which the traffic efficiency is the most common. In order to take other traffic participants into account, and also consider issues such as data noise, robustness, safety, comfort and reasonableness, the other four indicators are also added.

[0134] Traffic efficiency: In order to facilitate comparison, the traffic efficiency can be defined as the ratio of the reduction of queue length compared with the classic traffic signal control method Webster model, expressed as follows:

[0135]

[0136] Robustness: In which the reaction of different methods to noise or delayed data transmission is evaluated by randomly setting 10% of the observed values to zero. Then, the average queue length can be calculated under the strategy trained using the original observed values. The definition of robustness is the ratio of the average queue length under the missing observation to the average queue length under the original observation s , which is shown as follows:

[0137]

[0138] Safety: When the waiting time exceeds 80 seconds (some pedestrians may ignore the red light) or is less than 15 seconds (some pedestrians do not have enough time to cross the road), it can be considered as an unsafe event. Therefore, it can be assumed that N people arrive at different times and move in different directions. Next, the percentage of safe events can be calculated, where 1 A is an indicator function. When x∈A, it is 1, otherwise it is 0.

[0139]

[0140] Comfort: When the driver waits for more than 140 seconds, or the phase order is different from the last period, it can be considered as uncomfortable. The comfort index is defined as the proportion of comfortable time.

[0141]

[0142] Reasonableness: The overall rationality performance of the method should measure all the above indicators. The average value of robustness, safety and comfort can be used as the evaluation of the overall rationality performance, and the calculation formula is as follows:

[0143] m p =(m r +m s +m c ) / 3.

[0144] The application can issue an action once in a cycle and only fine-tune the green light duration of each phase by the action of fine-tuning the duration of each phase. It not only has strong robustness and is not sensitive to data noise and data transmission delay, but also takes other traffic participants (such as pedestrians and cyclists) into account, and does not have the problem of too long or too short green light duration of a certain phase.

[0145] In addition, the learning body is trained by the reinforcement learning method based on the maximum entropy framework. This method not only considers the relationship between different phases, but also uses the maximum entropy method to control the balance between exploration and utilization. Therefore, when using the action of fine-tuning the duration of each phase, not only good robustness, safety and comfort are maintained, but also the traffic efficiency is improved.

[0146] In the embodiments of the application, the action of the application is compared with the existing action, and the reinforcement learning method based on the maximum entropy framework of the application is compared with the traditional deep reinforcement learning method (DNQ).

[0147] The experiment can be carried out on a single intersection using a simulated traffic data set through SUMO simulation software. The topological structure of the intersection is as shown in Figure 5A It can be seen that this crossroads has different numbers of lanes in each direction, which increases the difficulty of training the agent.

[0148] The traffic flow distribution of the simulated traffic data set is as shown in Figure 5B The Y-axis represents the saturation rate (SR), i.e. the ratio of traffic arrival flow to road capacity, which is used to show the congestion of the crossroads. It can be seen that the saturation rate of the generated flow is between 0.6 and 0.9, which is a reasonable range for adaptive control of a single intersection. 200 different traffic saturation rate data can be generated for training purposes under the simulated traffic data set, as shown in the shaded part of Figure 5B It will help us determine the performance of the method under different traffic conditions.

[0149] The experimental results of the traditional deep reinforcement learning on different action designs, and the experimental results of the traditional deep reinforcement learning and the reinforcement learning method based on the maximum entropy framework (MD-SQN) are shown in Table 1:

[0150] Table 1

[0151]

[0152] Firstly, when using traditional deep reinforcement learning for training, “set the phase of the next stage” has the best performance in efficiency compared with other action designs including the action of the present application. The reason is that it can select any phase in a very short time without any constraints such as stage cycle. However, this attribute results in a lower score in safety, comfort and reasonableness. If the stage cycle is not followed, “set the phase of the next stage” can confuse the driver and other road users at the time of switching. It can also cause the green light duration to be too long, which can lead to jaywalking by pedestrians and long waiting by drivers, or the green light duration to be too short for pedestrians to safely cross the intersection.

[0153] Although “switch or keep” and “set phase duration” perform better than “set the phase of the next stage” in safety, comfort and reasonableness, they are still inferior to the action design in the present application. Both of these action designs retain the concept of cycles, so they take other road users into account. However, these two action designs do not take into account the impact on other phases when taking action. Therefore, they can also cause the green light duration of a certain phase to be too long or too short, which can lead to unsafe events.

[0154] Finally, all three of these action designs perform poorly in robustness. Because data transmission delay or noise can cause all three actions to suddenly switch phases or set the green light duration to be very unreasonable, or make other unreasonable decisions, thus reducing robustness.

[0155] In contrast, the action design “fine-tune phase duration” proposed in the present application has already outperformed the other three action designs in robustness, safety, comfort and reasonableness, because it changes once per cycle and only fine-tunes the green light duration of the phase each time. Although the efficiency is not high at this time, this is due to the use of traditional deep reinforcement learning. By changing the method, efficiency can also be improved.

[0156] The last row of Table 1 shows the results of the combination of the "reinforcement learning method based on the maximum entropy framework" with the "tune phase duration" action. By using the "reinforcement learning method based on the maximum entropy framework", the "tune phase duration" action design achieves comparable results to the "set next phase" action in terms of efficiency, while maintaining an advantage in all other metrics. This demonstrates that the action designs and new reinforcement learning method presented in this invention can effectively handle traffic signal control while taking into account real-world issues such as data transmission delays and other road users.

[0157] While the above describes embodiments of the present application, it is to be understood that they are presented by way of example only, and not limitation. As such, many changes and modifications will become apparent to those skilled in the art, once given the benefit of the foregoing disclosure. Accordingly, it is intended that the present application embrace all such changes and modifications as fall within the scope and spirit of the claimed invention.

Claims

1. A traffic signal control method based on reinforcement learning of a maximum entropy framework, characterized by, The method comprises the following steps: constructing a simulation environment; constructing a reinforcement learning method based on a maximum entropy framework, and training an agent in the simulation environment by the reinforcement learning method based on the maximum entropy framework; and controlling a traffic signal by the trained agent; wherein the constructing a simulation environment comprises: constructing an observation state of an agent, constructing an action of an agent, and constructing a reward of an agent; wherein the constructing an observation state of an agent comprises: taking a traffic flow q, an average queue length l, and a signal light scheme P of a last cycle as observation information, wherein the signal light scheme P is expressed as follows: The signal light scheme P consists of M phases, denotes the green light duration of the i-th phase; and The environmental observation vector is expressed by the following formula: wherein, denotes the flow on lane i, denotes the average queue length on lane i, respectively; wherein the agent is constructed to perform an action of fine-tuning a duration of each phase, wherein the agent is allocated to each phase, the duration of each phase is adjusted by the agent alone, and a plurality of the agents are combined to form a cycle and are executed in a fixed order; wherein the reinforcement learning method based on the maximum entropy framework comprises the following steps: With The Q value representing the phase and calculated in parts is represented as follows: wherein computed by assuming phase independence, denotes the residual term; Measuring the strategy with KL-divergence and the strategy solving the maximum of the first equation, the first equation is expressed as: ； The total return at time t is given by is represented by the following formula: ； Will It can be expressed as the following formula: ； Convert the following patent specification text to English: Convert the formula below: According to the maximum entropy principle to obtain the following formula: where KL-divergence value is the information entropy the difference in the number of actions ​ The difference in Q is calculated is expressed by the following equation: ； simplify to obtain a prediction of the current state value, denoted as the second equation: 。 2. The traffic signal control method based on the reinforcement learning of the maximum entropy framework according to claim 1, wherein the reward of the agent is expressed as follows: wherein, represents a weight, is a constant. 3.The traffic signal control method based on reinforcement learning of maximum entropy framework according to claim 1, wherein, an error is calculated according to the second formula and the Q value is updated, wherein a probability distribution is defined based on the Q value and a Boltzmann distribution, expressed as follows: and the error is normalized and clipped by a moving average μ and a variance σ, expressed as follows: ； and a loss function is expressed as follows: 。 4. A computer system, characterized by comprise: a processor configured to execute machine executable instructions; and a memory having stored thereon machine executable instructions that, when executed by the processor, perform the steps of the method according to one of claims 1-3.

5. A traffic signal control system characterized by, comprise: a signal light; and a computer system that is the computer system of claim 4, the computer system being configured to control the signal light.

Images