Multi-intersection traffic signal control method, system, device and storage medium

Abstract

This invention discloses a multi-intersection traffic light control method, system, device, and storage medium. It incorporates the possibility of traffic lights malfunctioning into the problem modeling, making it more realistic. Furthermore, it incorporates information on whether the traffic lights are functioning properly into the state design of the intersection where the traffic lights are located. During the training phase, the reinforcement model experiences both scenarios where traffic lights are functioning normally in the road network and scenarios where a traffic light malfunctions and is replaced by a temporary traffic light. This ensures good performance under normal conditions and can also handle various traffic light malfunction scenarios.

Description

Technical Field

[0001] This invention relates to the field of multi-intersection traffic signal control technology, and in particular to a multi-intersection traffic signal control method, system, device and storage medium. Background Technology

[0002] To address the shortcomings of traditional traffic light control methods, which rely on expert knowledge and assumptions about road traffic conditions to manually set traffic light phase switching rules and lack adaptability to dynamically changing traffic situations, adaptive traffic light control methods based on reinforcement learning have developed rapidly in recent years, achieving superior performance compared to traditional methods. Based on the number of traffic lights controlled, these methods can be categorized into single-intersection traffic light control methods and multi-intersection traffic light control methods.

[0003] For single-intersection traffic light control scenarios, the goal is to minimize the average travel time of vehicles through the intersection, thereby improving traffic congestion at that intersection. Typically, a state and reward function are designed for this intersection, and optimization is performed using reinforcement learning algorithms to reduce vehicle queuing time.

[0004] For multi-intersection traffic light control scenarios, the goal is to minimize the average travel time of all vehicles in the overall road network, thereby improving traffic congestion. Typically, the influence between different intersections is modeled, and then appropriate state and reward functions are designed. A multi-agent reinforcement learning approach is used to optimize the control strategy for each intersection's traffic lights, achieving collaborative regulation across multiple intersections and reducing overall vehicle travel time. However, current reinforcement learning-based multi-intersection traffic light control technologies still face the following challenges:

[0005] 1) The modeling of the traffic signal control problem at multiple intersections is not realistic enough. It does not take into account the scenario in which traffic lights may fail to function properly due to factors such as power outages, malfunctions, or maintenance, and are then taken over by temporary traffic lights controlled by fixed time rules.

[0006] 2) In the above situation, the traffic light phase at the intersection where the collapsed traffic light is located cannot be controlled by the reinforcement learning strategy, and the cooperative relationship with the adjacent traffic lights is destroyed. Existing methods are not robust enough to this situation, and their performance will be significantly reduced. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-intersection traffic signal control method, system, device, and storage medium that can adapt to scenarios where all traffic lights are working normally and scenarios where a traffic light is not working properly and is replaced by a temporary traffic light based on fixed time rules, thereby more effectively improving traffic congestion in the overall road network.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A method for controlling traffic lights at multiple intersections includes:

[0010] During each simulation, the state of each intersection where a traffic light is located is input into the reinforcement model, which then outputs the corresponding actions, i.e., the phases of each traffic light. Each simulation scenario is either a normal scenario or an abnormal scenario. Depending on whether each scenario is a normal scenario, the state of each intersection where a traffic light is located and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In abnormal scenarios, the malfunctioning traffic light is selected and marked based on the state of the intersection where the traffic light is located.

[0011] Samples are collected from the corresponding cache to train the reinforcement model;

[0012] The trained reinforcement model is used for traffic light control at multiple intersections.

[0013] A multi-intersection traffic signal control system, comprising:

[0014] The system simulation and data collection unit is used to input the state of each traffic light intersection into the reinforcement model during each simulation, and the reinforcement model outputs the corresponding actions, i.e., the phases of each traffic light. The scenario of each simulation is either a normal scenario or an abnormal scenario. Depending on whether each scenario is a normal scenario, the state of each traffic light intersection and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In the abnormal scenario, the collapsed traffic light is selected and marked according to the state of the intersection where the traffic light is located.

[0015] A training unit is used to sample samples from the corresponding cache and train the reinforcement model;

[0016] The control unit is used to control traffic lights at multiple intersections using a trained reinforcement model.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, the possibility of traffic lights malfunctioning and becoming unusable is introduced into the problem modeling, making it more realistic. Furthermore, the status design of the intersection where the traffic lights are located incorporates information on whether the traffic lights are functioning properly. During the training phase, the enhanced model experiences both scenarios where the traffic lights are functioning normally and scenarios where a traffic light malfunctions and is replaced by a temporary traffic light. This ensures good performance under normal conditions and can also cope with various traffic light malfunction scenarios, thereby more effectively reducing the average travel time of all vehicles in the overall road network and improving traffic congestion in the overall road network. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a multi-intersection traffic light control method provided in an embodiment of the present invention;

[0023] Figure 2 A framework diagram of a multi-intersection traffic light control method provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a multi-intersection traffic signal control system provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] First, the following explanations are provided for the terms that may be used in this article:

[0028] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0029] The following is a detailed description of a multi-intersection traffic light control method, system, device, and storage medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0030] Example 1

[0031] This invention provides a method for controlling traffic lights at multiple intersections, comprising:

[0032] 1. During each simulation, the state of each traffic light intersection is input into the reinforcement model, and the reinforcement model outputs the corresponding actions, i.e., the phases of each traffic light. The scenario of each simulation is either a normal scenario or an abnormal scenario. Depending on whether each scenario is a normal scenario, the state of each traffic light intersection and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In the abnormal scenario, the collapsed traffic light is selected and marked according to the state of the intersection where the traffic light is located.

[0033] 2. Sample samples from the corresponding cache to train the reinforcement model.

[0034] 3. Use the trained reinforcement model to control traffic lights at multiple intersections. The control method is similar to the simulation process described above, that is, input the state of each traffic light at the intersection and output the phase of each traffic light to realize traffic light control at multiple intersections.

[0035] The method provided in this invention is a novel multi-intersection traffic signal control technology. While ensuring performance under normal road network conditions, it improves adaptability to scenarios where traffic lights fail. This method introduces a new intersection state to characterize whether a traffic light has failed, and designs a new reinforcement learning framework for multi-intersection traffic signal control. During training, it not only experiences scenarios where the road network traffic lights are normal, but also scenarios where a traffic light fails and is replaced by a temporary traffic light. Figure 2As shown, the entire framework includes the following parts: 1) A normal scenario selection module, which selects whether a traffic light has malfunctioned at the start of a simulation; 2) A malfunctioning traffic light location selection module, which, given the presence of a malfunctioning traffic light in the simulation, selects an intersection traffic light to be replaced by a temporary traffic light controlled by fixed-time rules based on the importance of different traffic lights; 3) A reinforcement model, which selects the appropriate traffic light phase based on the state of each intersection; 4) A data storage module, which stores data according to whether a malfunctioning traffic light exists in the simulation. The trained reinforcement model can be applied to multi-intersection traffic light control.

[0036] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0037] I. Normal Scenario Selection Module.

[0038] The normal scenario selection module is used to determine the probability that the road network state is set to normal for all traffic lights in each simulation. In other words, the scenario for each simulation is selected as normal or abnormal based on the normal probability.

[0039] The purpose of this invention is to ensure good performance under normal road network scenarios while improving adaptability to situations with traffic light malfunctions. Therefore, it is necessary to increase the normal probability value at the beginning of training, that is, to set a higher normal road network probability so that the reinforcement model sees enough samples under normal conditions; after training a specified number of times, the normal probability value is reduced, that is, a relatively low normal probability is selected in the second half of training so that the reinforcement model can cope with different traffic light failure scenarios.

[0040] In this embodiment of the invention, the normal probability is gradually reduced according to the number of simulation rounds, the first... Normal probability of the game Annealing is performed in the following manner:

[0041]

[0042] in, The initial normal probability, For the final normal probability, The number of rounds for annealing.

[0043] In this embodiment of the invention, settings can be made according to actual conditions or experience. and Size, number of annealing rounds It means The number of simulations required to reduce the initial probability to the final probability.

[0044] II. Crash Signal Light Location Selection Module.

[0045] For a person with The road network consists of several intersections, each with one traffic light. When a traffic light at one intersection malfunctions, a total of [number] intersections will be affected. There are several different possibilities. Because traffic flow varies at different intersections, when a traffic light at one intersection malfunctions and is replaced by a temporary one, the average travel time for all vehicles in the overall road network will differ. The impact of traffic lights at some intersections will be greater than that at others.

[0046] To enable the method of this invention to better handle different traffic light malfunctions and to enhance its adaptability to more critical situations during training, the importance of the traffic light is calculated based on the average travel time of vehicles at the intersection where the traffic light is located, and expressed as follows:

[0047]

[0048] in, It is a natural constant; This is a scaling factor, used to prevent extreme values ​​from occurring when calculating importance; For the first After a traffic light malfunctions, the average travel time of vehicles in the road network at the intersection where a temporary traffic light takes over is as follows: For the first After a traffic light malfunctions, the average travel time of vehicles in the road network at the intersection is taken over by a temporary traffic light.

[0049] If a traffic light fails, the failed traffic light location selection module will select the affected traffic light based on its importance. Select the intersection where the faulty traffic light will occur (i.e., select one traffic light as the faulty traffic light). During this process, any traffic light at any intersection may be selected, but their importance will vary. The larger the value, the higher the chance of being selected.

[0050] In addition, to adapt to the evolution of reinforcement strategies during training, every [period] The bureau recalculated the importance of each traffic light. For example, a positive integer. .

[0051] III. Reinforcement Model.

[0052] In this embodiment of the invention, the enhancement model is based on the state of each intersection where a traffic light is located. Select action, i.e., traffic light phase The state of the intersection is represented using a multi-dimensional vector, which includes the following three types of information: the number of vehicles merging into the lane, the currently acquired traffic light phase information, and a marker indicating whether the traffic light is a faulty traffic light.

[0053] For example, a 21-dimensional vector can be used, which includes 12-dimensional information on the number of vehicles entering the lane, 8-dimensional information on the traffic light phase, and 1-dimensional information on whether the traffic light is malfunctioning. The action space size of each intersection is 4, which includes 4 traffic light phases: east-west straight, east-west left turn, north-south straight, and north-south left turn.

[0054] Those skilled in the art will understand that intersection lanes can be divided into two categories: merging lanes and merging lanes. Merging lanes refer to the lanes through which traffic enters the intersection, including lanes entering from all directions (north, south, east, west). Taking the aforementioned 12-dimensional merging lane vehicle count information as an example, each of the four directions (north, south, east, west) is further divided into left turn, straight, and right turn, resulting in a total of 12 dimensions. The aforementioned 8-dimensional traffic light phase information is based on a traffic rule where right-turning traffic is not subject to phase restrictions. The 8-dimensional traffic light phase information corresponds to left turns and straight traffic in the four directions, with each dimension using 1 or 0 to represent whether the corresponding traffic flow can or cannot enter the intersection under the current phase.

[0055] In this embodiment of the invention, the enhancement model includes: a feature extractor, a multi-layer graph attention network, and a fully connected network; wherein: the feature extractor is used to extract feature information from the state of each traffic light intersection; the multi-layer graph attention network is used to aggregate the extracted feature information; and the fully connected network is used to estimate the Q-value based on the aggregated feature information and select the corresponding action based on the estimated Q-value.

[0056] In this embodiment of the invention, Q-value estimation refers to the Q-value of each action of the traffic lights at each intersection in the current state, output by the fully connected network. Q-value is a technical term in reinforcement learning, used to indicate the quality of choosing a particular action in the current state. During training, ... The probability of randomly selecting one action from all actions is... The probability of choosing the action with the highest Q value is [indicated]. The size can be set according to the actual situation or experience.

[0057] For example, a fully connected network with two layers and 32 neurons each can be used to extract feature information from the intersection state, with the activation function of each layer being a linear rectified function. Then, a two-layer graph attention network aggregates the state information of adjacent intersections, where the number of input channels in each layer is set to 32, the number of heads used in the attention mechanism is 5, and the outputs of different heads are summed, with the activation function being a linear rectified function. Finally, a fully connected network with one layer containing 4 neurons (corresponding to 4 actions) and without using a non-linear activation function is used to estimate the state. The value is used to select the action. To reduce the difficulty of training, network parameters can be shared for each intersection.

[0058] IV. Data storage module.

[0059] In this embodiment of the invention, training involves two main scenarios: all traffic lights are functioning normally, and one traffic light malfunctions. To prevent data collection from interfering with each other in these two scenarios, this invention uses two caches. Data collected during a normal scenario is stored in the normal scenario cache, and data collected during a malfunction scenario is stored in the malfunction scenario cache. The cache unit size is set to B. One traffic light (i.e.) For a road network with (number of intersections), we set the normal scenario cache size to B and the crash scenario cache size to nB. During training, data is sampled from both caches, and parameters are optimized using temporal difference loss as a constraint.

[0060] In this embodiment of the invention, a training method commonly used in reinforcement learning is employed. During each simulation, as the reinforcement model interacts with the simulator environment, the current state (i.e., the state of the intersection mentioned earlier), the action selected based on that state, the reward given by the environment, and the next state of the simulator environment after taking the action are stored in a cache as a set of data. The current state in each set of data refers to the set of all intersection states in the current step. Similarly, the action selected based on the state refers to the set of all traffic light phases at all intersections. The reward given by the environment can be calculated using the simulator, and the next state refers to the set of all intersection states in the next step. Once a sufficient number of sets of data (which can be set according to actual conditions) are stored in the cache, training begins: multiple sets of data are sampled from the cache, and the temporal difference loss function is calculated using the previously stored data, thereby updating the parameters. Considering that this part can be implemented using conventional techniques, it will not be elaborated further.

[0061] The traffic signal control technology solution for multi-intersections with signal light failures based on reinforcement learning provided in this invention introduces the possibility of signal lights failing and becoming unusable into the problem modeling, making it more realistic. The technical framework proposed in this invention incorporates information on whether the signal lights are functioning normally into the intersection state design. At the same time, during the training phase, the reinforcement model experiences both scenarios where the signal lights are functioning normally in the road network and scenarios where a signal light fails and is replaced by a temporary signal light. It can adaptively coordinate the ratio between the two, thereby ensuring good performance under normal circumstances and being able to cope with various signal light failure scenarios.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0063] Example 2

[0064] This invention also provides a multi-intersection traffic signal control system, which is mainly implemented based on the method provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes:

[0065] The system simulation and data collection unit is used to input the state of each traffic light intersection into the reinforcement model during each simulation, and the reinforcement model outputs the corresponding actions, i.e., the phases of each traffic light. The scenario of each simulation is either a normal scenario or an abnormal scenario. Depending on whether each scenario is a normal scenario, the state of each traffic light intersection and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In the abnormal scenario, the collapsed traffic light is selected and marked according to the state of the intersection where the traffic light is located.

[0066] A training unit is used to sample samples from the corresponding cache and train the reinforcement model;

[0067] The control unit is used to control traffic lights at multiple intersections using a trained reinforcement model.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0069] Example 3

[0070] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0071] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0072] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0073] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0074] The output device can be a display terminal;

[0075] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0076] Example 4

[0077] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0078] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0079] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling traffic lights at multiple intersections, characterized in that, include: In each simulation, the state of each intersection where a traffic light is located is input into the reinforcement model, which then outputs corresponding actions, i.e., the phases of each traffic light. The state of each intersection where a traffic light is located is represented using a multi-dimensional vector, including the following three types of information: the number of vehicles in the merging lane, the currently acquired traffic light phase information, and a marker indicating whether the traffic light is a faulty traffic light. Each simulation scenario is either a normal or abnormal scenario. Depending on whether each scenario is normal, the state of each intersection where a traffic light is located and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In abnormal scenarios, a faulty traffic light is selected: based on the importance of the traffic lights at each intersection, a traffic light at one intersection is selected as a faulty traffic light, and this is marked by the state of the intersection where the traffic light is located. The selection of whether each simulation scenario is normal or abnormal is based on a normal probability, which gradually decreases with the number of simulation scenarios. Samples are collected from the corresponding cache to train the reinforcement model; The trained reinforcement model is used for traffic light control at multiple intersections.

2. The multi-intersection traffic signal control method according to claim 1, characterized in that, The step of gradually decreasing the normal probability based on the number of simulation rounds includes: No. Normal probability of the game Annealing is performed in the following manner: ； in, The initial normal probability, For the final normal probability, The number of rounds for annealing.

3. The multi-intersection traffic light control method according to claim 1, characterized in that, The more important the traffic light, the higher the probability of it being selected; and, every... The bureau recalculated the importance of traffic lights at each intersection. It is a set positive integer.

4. The multi-intersection traffic signal control method according to claim 3, characterized in that, Calculating the importance of traffic lights at each intersection includes: The importance of a traffic light is calculated based on the average travel time of vehicles at the intersection where the traffic light is located, and is expressed as follows: ； in, It is a natural constant. This is the scaling factor. For the first After a traffic light malfunctions, the average travel time of vehicles in the road network at the intersection where a temporary traffic light takes over is as follows: For the first After a traffic light malfunctions, the average travel time of vehicles in the road network at the intersection is taken over by a temporary traffic light.

5. The multi-intersection traffic light control method according to claim 1, characterized in that, The enhancement model includes: a feature extractor, a multi-layer graph attention network, and a fully connected network; wherein: The feature extractor is used to extract feature information from the status of each traffic light intersection; The multi-layer graph attention network is used to aggregate the extracted feature information; The fully connected network is used to estimate the Q-value based on the aggregated feature information and select the corresponding action based on the estimated Q-value.

6. A multi-intersection traffic signal control system, characterized in that, Based on the method described in any one of claims 1 to 5, the system comprises: The system simulation and data collection unit is used to input the state of each traffic light intersection into the reinforcement model during each simulation, and the reinforcement model outputs the corresponding actions, i.e., the phases of each traffic light. The scenario of each simulation is either a normal scenario or an abnormal scenario. Depending on whether each scenario is a normal scenario, the state of each traffic light intersection and the corresponding traffic light phases output by the reinforcement model are stored as samples in the corresponding cache. In the abnormal scenario, the collapsed traffic light is selected and marked according to the state of the intersection where the traffic light is located. A training unit is used to sample samples from the corresponding cache and train the reinforcement model; The control unit is used to control traffic lights at multiple intersections using a trained reinforcement model.

7. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images