A traffic signal control method, device, electronic equipment, and storage medium

Abstract

This application discloses a traffic signal control method, device, electronic device, and storage medium, relating to the field of traffic control technology. The method includes: converting acquired intersection traffic information to obtain a first traffic state; wherein the intersection traffic information includes traffic light information and vehicle queue information; inputting the first traffic state into a traffic signal control model to determine a first traffic signal action; determining a first reward value and a second traffic state at the next moment based on a first desired target, the first traffic state, and the first traffic signal action, and storing these as sample data in an experience replay pool; performing reinforcement learning training on the traffic signal control model based on the experience replay pool to obtain a target control strategy, and controlling the traffic lights according to the target control strategy to efficiently achieve traffic signal control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic control, and particularly relates to a traffic signal control method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of economy and the improvement of living quality, driving has become a common choice for people. However, with the rapid growth of vehicle ownership and the lack of expansion of urban traffic road infrastructure, traffic congestion is becoming more and more serious.

[0003] If traffic congestion is alleviated by expanding traffic road infrastructure, not only will it produce expensive costs, but it will also affect traffic during the construction period and increase the traffic pressure of other traffic roads. Establishing a perfect traffic signal control system can solve the contradiction between the slow growth of traffic infrastructure construction and the rapid increase of vehicle ownership, and is also an effective way to alleviate traffic congestion.

[0004] Due to the complexity and variability of traffic scenes, if the designed traffic signal control system is unreasonable, it cannot effectively and timely control the traffic on the traffic road, which will bring inconvenience to vehicle driving.

[0005] Therefore, how to efficiently realize traffic signal control is a problem to be solved at present. SUMMARY

[0006] The present application provides a traffic signal control method to efficiently realize traffic signal control.

[0007] In a first aspect, a traffic signal control method is provided, comprising:

[0008] Converting the obtained intersection traffic information to obtain a first traffic state; wherein the intersection traffic information comprises signal light information and vehicle queue information; inputting the first traffic state into a traffic signal control model to determine a first traffic signal action; determining a first reward value and a second traffic state at the next moment according to a first expected target, the first traffic state, the first traffic signal action; wherein the first reward value represents the preference of the first traffic state for the first traffic signal action, and the first expected target is an expected target randomly sampled from a set of target; storing the first traffic state, the first traffic signal action, the first expected target, the first reward value and the second traffic state as sample data in an experience replay pool; performing reinforcement learning training on the traffic signal control model according to the experience replay pool to obtain a target control strategy, and controlling the traffic signal light according to the target control strategy.

[0009] Optionally, the traffic signal control model includes a Critic network, an Actor network, an Actor target network, and a Critic target network; the step of training the traffic signal control model through reinforcement learning based on the experience replay pool includes:

[0010] The second traffic state from the sample data is input into the Actor target network, and exploratory noise is introduced to determine the second traffic signal action; the second traffic state and the second traffic signal action are input into the Critic target network to determine the target Q value; based on the target Q value and the minimized loss function, the network parameters μ of the Critic network are updated. k Update the network parameters φ of the Actor network according to the gradient descent algorithm; based on the updated μ k Update the network parameters μ′ of the Critic target network. k And based on the updated φ, update the network parameters φ′ of the Actor target network.

[0011] Optionally, the μ k Satisfy the following expression:

[0012]

[0013]

[0014] Among them, y t The target Q value, The Critic network calculates the Q value based on the first traffic state and the first traffic signal action, where N is the number of sample data points, and r is the number of samples. t For the first desired target, the s′ t The second traffic state, the The second traffic signal action is defined as γ, which is a hyperparameter.

[0015] Optionally, the φ satisfies the following expression:

[0016]

[0017] Wherein, N is the number of sample data, and π φ For the Actor network, the s t For the first traffic state, a t This is the action of the first traffic signal.

[0018] Optionally, after storing the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state as sample data in the experience replay pool, the method further includes:

[0019] Sample a second desired target from the target set;

[0020] The second reward value is determined based on the second expected target, the first traffic condition, and the first traffic signal action.

[0021] The second desired target, the first traffic state, the first traffic signal action, the second reward value, and the second traffic state are stored as new sample data in the experience playback pool.

[0022] Secondly, a traffic signal control device is provided, comprising:

[0023] The information processing module is used to convert the acquired intersection traffic information into a first traffic state; wherein, the intersection traffic information includes traffic light information and vehicle queue information;

[0024] A first determining module is used to input the first traffic state into the traffic signal control model to determine the first traffic signal action; a second determining module is used to determine a first reward value and a second traffic state at the next moment based on a first desired target, the first traffic state, the first traffic signal action; wherein, the first reward value is used to characterize the preference of the first traffic state for the first traffic signal action, and the first desired target is a desired target randomly sampled from a set of targets; an experience playback module is used to store the first traffic state, the first traffic signal action, the first desired target, the first reward value, and the second traffic state as sample data in an experience playback pool; a control module is used to perform reinforcement learning training on the traffic signal control model based on the experience playback pool to obtain a target control strategy, and control the traffic lights according to the target control strategy.

[0025] Optionally, the traffic signal control model includes a Critic network, an Actor network, an Actor target network, and a Critic target network;

[0026] The control module is specifically used for:

[0027] The second traffic state from the sample data is input into the Actor target network, and exploratory noise is introduced to determine the second traffic signal action; the second traffic state and the second traffic signal action are input into the Critic target network to determine the target Q value; based on the target Q value and the minimized loss function, the network parameters μ of the Critic network are updated. k Update the network parameters φ of the Actor network according to the gradient descent algorithm; based on the updated μ k Update the network parameters μ of the Critic target network. ′ k And based on the updated φ, update the network parameters φ′ of the Actor target network.

[0028] Optionally, the experience playback module is further used for:

[0029] Sample a second desired target from the target set; determine a second reward value based on the second desired target, the first traffic state, and the first traffic signal action; store the second desired target, the first traffic state, the first traffic signal action, the second reward value, and the second traffic state as new sample data in the experience playback pool.

[0030] Thirdly, an electronic device is provided, comprising:

[0031] A memory for storing computer programs; a processor for executing the computer programs stored in the memory to implement the method steps described in any one of the first aspects.

[0032] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of the method described in any one of the first aspects.

[0033] In this embodiment, the acquired intersection traffic information is converted to obtain a first traffic state. Since this intersection traffic information includes traffic light information and vehicle queue information, a more comprehensive perception of the traffic environment can be achieved, improving the accuracy of subsequent target control strategies. The first traffic state is input into the traffic signal control model to determine the first traffic signal action. Based on the first desired target, the first traffic state, and the first traffic signal action, a first reward value and the second traffic state at the next moment are determined. Since the first reward value can represent the preference of the first traffic state for the first traffic signal action, calculating the reward value can provide a basis for subsequent target control strategies. The target control strategy provides direction; the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state are stored as sample data in the experience replay pool; the traffic signal control model is trained by reinforcement learning based on the experience replay pool to obtain the target control strategy, and the traffic lights are controlled according to the target control strategy. Since the model is trained by reinforcement learning using the post-event experience replay mechanism, the impact of the sparse reward problem in traffic signal control can be solved, the performance of the model is improved, and it has better generalization for complex traffic scenarios, thus achieving efficient traffic signal control.

[0034] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0035] Figure 1 A flowchart of a traffic signal control method provided in an embodiment of this application;

[0036] Figure 2 A logical diagram illustrating reinforcement training of a traffic signal control model provided in an embodiment of this application;

[0037] Figure 3 This is a schematic diagram of the structure of a traffic signal control device provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0040] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application will be explained below.

[0041] (1) Hindsight Experience Replay (HER) is a reinforcement learning algorithm that can learn from failures.

[0042] (2) The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm combines deep deterministic policy gradient and dual Q-learning, achieving good performance on many continuous control tasks. The TD3 algorithm includes six networks: Actor network, Critic1 network, Critic2 network, Actor target network, Critic target network 1, and Critic target network 2. The algorithm mainly incorporates the following key technologies: using two Critic networks, the smaller of the two values ​​can be used when calculating the target Q-value, thus suppressing overestimation; when calculating the target Q-value, a perturbation is added to the action in the next state, making the evaluation more accurate; after updating the Critic network, the Actor network is updated, ensuring more stable training of the Actor network.

[0043] Currently, traffic signal control typically employs deep reinforcement learning to train models and generate control strategies. For example, real-time vehicle information for a target intersection is obtained from real-time images of that intersection; this information is then input into a pre-defined single-intersection traffic signal control model to generate the control scheme. This pre-defined single-intersection traffic signal control model is trained using real-time vehicle sample information. However, due to the complexity and variability of traffic scenarios, if the collected vehicle sample information is too limited, the perception of the traffic environment will be incomplete, leading to inaccurate control strategies obtained from the pre-defined single-intersection traffic signal control model. Furthermore, deep reinforcement learning also suffers from the sparse reward problem when solving traffic signal control problems, which can further affect the accuracy of the control strategy.

[0044] In view of this, this application provides a traffic signal control method. The traffic signal control model used in this method collects traffic information from multiple angles and combines the TD3 algorithm and HER mechanism to solve the problems of incomplete traffic environment perception and sparse reward, effectively improving the performance of the traffic signal control model and enhancing the efficiency and accuracy of the control strategy.

[0045] In this embodiment of the application, the established traffic signal control model can be considered from the following aspects:

[0046] Firstly, when collecting traffic information and defining the traffic state of the model, information is collected from multiple perspectives, such as traffic light information and vehicle queue information, thereby improving the perception of the traffic environment. Optionally, the traffic state satisfies the following expression:

[0047] s t ={V i ,L i}……..(1)

[0048] Among them, V i Let L be the length of the vehicle queue in the i-th lane at time t, where i is the lane number. i Let L be the traffic light status of the i-th lane at time t. i The positive time indicates the sum of the green light time and the yellow light time, L i A negative value indicates the red light duration.

[0049] In some embodiments, the yellow light duration can be set to a fixed value, such as 3 seconds, and all green phases have a minimum green time and a maximum green time. This ensures that enough information is collected to evaluate the effectiveness of subsequent reward functions, prevents phases with extremely short durations, and thus effectively avoids traffic accidents.

[0050] Second, based on the defined traffic conditions, define the traffic signal actions for this model. Optionally, the traffic signal actions satisfy the following expression:

[0051] a i (t)={a i +m,a i ,a i -m}……(2)

[0052] Among them, a i +m represents the current traffic light color in the i-th phase plus m seconds, a i -m indicates that the current traffic light color in the i-th phase is reduced by m seconds, a i This indicates that the duration of the i-th phase remains unchanged.

[0053] Thirdly, define the reward value function for this model. Since the reward value reflects the current traffic state's preference for traffic signal actions, neglecting the model's reward value function may lead to slow convergence and affect vehicle traffic efficiency. Optionally, the reward value function satisfies the following expression:

[0054] r t =ω1(T t+1 -T t )+ω2V q ……(3)

[0055] Where ω1 and ω2 are weighting coefficients. (T) t+1 -T t V represents the difference in the cumulative average waiting time for all vehicles. q This is the sum of the lengths of the vehicle queues.

[0056] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in combination.

[0057] Figure 1 This is a flowchart illustrating a traffic signal control method provided in an embodiment of this application. The process can be executed by a traffic signal device, which can be implemented in software, hardware, or a combination of both. As shown in the figure, the process includes the following steps:

[0058] 101: The obtained intersection traffic information is converted to obtain the first traffic state.

[0059] Optionally, the traffic information at the intersection includes traffic light information and vehicle queue information. The traffic light information can be obtained through a traffic signal controller, and the vehicle queue information can be obtained through an image acquisition device, such as a smart checkpoint device.

[0060] Optionally, before acquiring the intersection traffic information, the traffic signal control model can be initialized. Since this traffic signal control model is based on the TD3 algorithm and the HER mechanism, the initialization of the traffic signal model should include at least the following:

[0061] Case 1: Initialize the network parameters μ of the Critic network k (K=1,2) Using two sets of Critic networks can avoid overestimation of the Q value in subsequent calculations, thus affecting the accuracy of the target control strategy.

[0062] Case 2: Initialize the network parameters φ of the Actor network, which can be used to generate traffic signal actions.

[0063] Case 3: Initialize the network parameters μ′ of the Critic target network k (K=1,2) The network parameters φ′ of the Actor target network.

[0064] Case 4: Initialize the experience replay pool D, which is used to store sample data and can handle sparse reward problems.

[0065] Case 5: Initialize the target set, which includes multiple sampled desired targets.

[0066] Case 6: Initialize the exploration noise introduced in the traffic signal control model. This exploration noise can be used to assist in finding traffic signal actions and improve the stability of the TD3 algorithm.

[0067] In the above steps, traffic conditions include not only vehicle queue information but also traffic light information. Using these as inputs to the model makes the perception of the traffic environment more comprehensive, thereby improving the accuracy of subsequent target control strategies.

[0068] 102: Input the above-mentioned first traffic state into the traffic signal control model to determine the first traffic signal action.

[0069] Specifically, this step may involve inputting the first traffic state into the Actor network of the traffic signal control model to obtain the first traffic signal action.

[0070] In some embodiments, exploratory noise may also be introduced into the Actor network to determine the first traffic signal action, which optionally satisfies the following expression:

[0071] a t ~π φ (s t )+ε,ε~N(0,σ)…….(4)

[0072] Among them, a t For the first traffic signal action, π φ For Actor networks, s t Let ε represent the first traffic condition, and ε represent the exploration noise.

[0073] 103: Based on the first expected goal, the first traffic state, the first traffic signal action, determine the first reward value and the second traffic state at the next moment.

[0074] The first reward value is used to characterize the first traffic state (s). t ) Action on the first traffic signal (a) t The preference of ) is that the first desired target (g) is at least one desired target randomly sampled from the set of targets (G).

[0075] This step incorporates the HER mechanism, specifically as follows: First, random sampling is performed in the target set to obtain a first desired target. Based on this first desired target and the current first traffic state, the first traffic signal action is determined. After executing the first traffic signal action, the first reward value (r) is determined. t ), and the second traffic state at the next moment (s t ′).

[0076] 104: Store the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state as sample data in the experience playback pool.

[0077] This sample data can be represented as {s} t ||g,a t ,r t ,s t ′||g}, where s t ||g represents the concatenation operation of the first traffic state and the first desired goal, s t ′||g represents the splicing operation of the second traffic state and the first desired goal.

[0078] Optionally, after storing the first traffic state, first traffic signal action, first expected target, first reward value, and second traffic state as sample data in the experience replay pool, multiple new sample data can be sampled and stored in the experience replay pool to address the impact of the sparse reward problem. Specifically, this can include the following process: sampling a second expected target from the target set; determining a second reward value based on the second expected target, first traffic state, and first traffic signal action; storing the second expected target, first traffic state, first traffic signal action, second reward value, and second traffic state as new sample data in the experience replay pool. This process can be repeated to generate multiple new sample data stored in the experience replay pool, facilitating subsequent training of the target control strategy using this experience replay pool. This effectively solves the sparse reward problem, improves algorithm performance, and provides better generalization for complex traffic scenarios.

[0079] 105: Based on the above experience playback pool, the traffic signal control model is trained by reinforcement learning to obtain the target control strategy, and the traffic lights are controlled according to the target control strategy.

[0080] Optionally, a small batch of sample data can be selected from the experience replay pool (D) for reinforcement learning training, using the sample data ({s t ||g,a t ,r t ,s t Taking ′||g} as an example, the traffic signal control model is trained by reinforcement learning based on the experience replay pool, which may include the following process:

[0081] The second traffic state from the sample data is input into the Actor target network, and exploratory noise is introduced to determine the second traffic signal action. The second traffic state and the second traffic signal action are then input into the Critic target network to determine the target Q value. Based on the target Q value and the minimized loss function, the network parameters μ of the Critic network are updated. k Update the network parameters φ of the Actor network according to the gradient descent algorithm; based on the updated μ k Update the network parameters μ′ of the Critic target network. k And based on the updated φ, update the network parameters φ′ of the Actor target network.

[0082] Optionally, the second traffic signal action satisfies the following expression:

[0083]

[0084] in, For the second traffic signal action, π φ′ For the Actor target network, s′t Let ε represent the second traffic state and ε represent the exploration noise. The exploration noise introduced into the Actor target network follows a truncated normal distribution, which can prevent the target control strategy obtained through subsequent training from getting trapped in local optima.

[0085] Optionally, the target Q value satisfies the following expression:

[0086]

[0087] Among them, y t For two Critic target networks, based on s′ t and The smaller of the two calculated Q values ​​is used to obtain the target Q value, where γ is a hyperparameter and r is a hyperparameter. t As the primary expected goal, For the Critic target network.

[0088] Optional, μ k Satisfy the following expression:

[0089]

[0090] in, The Q value is calculated by the Critic network based on the first traffic state and the first traffic signal action, and N is the number of sample data. For Critic network.

[0091] Optionally, φ satisfies the following expression:

[0092]

[0093] The meanings of the parameters in formula (8) are as described above and will not be repeated here.

[0094] Optional, μ′ k φ and φ′ satisfy the following expressions respectively:

[0095] μ′ k ←τμ k +(1-τ)μ′ k ……(9)

[0096] φ′←τφ+(1-τ)φ′……..(10)

[0097] Where τ << 1 is the soft update constant, K = 1, 2, and the network parameters (μ′) of the target network are set to 1. k The changes in φ′ are slow, thereby reducing the cumulative error and improving the stability of the algorithm.

[0098] In the above steps, by using two sets of Critic networks to calculate different Q values ​​and selecting the smaller Q value to calculate the target Q value, the problem of Q value overestimation can be solved, thereby avoiding the selection of an overestimated control strategy when performing traffic signal control.

[0099] Figure 2 This is a logical diagram illustrating reinforcement training of a traffic signal control model, provided as an embodiment of this application. Figure 2 As shown, a small batch of sample data ({s) is first selected from the experience replay pool. t ||g,a t ,r t ,s t ′||g}), will s t Input to the Actor target network (π) φ′ In this process, exploratory noise is introduced to obtain the second traffic signal action. Secondly, and the second traffic state (s) in the sample data t ′) Input into the Critic target network In the process, the target Q-value is obtained, and the Critic network is updated based on the target Q-value and the minimized loss function. Network parameters μ k The Actor network (π) is updated by performing gradient descent on the Q-values ​​calculated by any one of the Critic network based on the first traffic state and the first traffic signal action. φ The network parameter φ; finally, based on the updated μ k Update the network parameters μ′ of the Critic target network. k And based on the updated φ, update the network parameters φ′ of the Actor target network, thereby completing the training of the traffic signal control model.

[0100] In this embodiment, the acquired intersection traffic information is converted to obtain a first traffic state. Since this intersection traffic information includes traffic light information and vehicle queue information, a more comprehensive perception of the traffic environment can be achieved, improving the accuracy of subsequent target control strategies. The first traffic state is input into the traffic signal control model to determine the first traffic signal action. Based on the first desired target, the first traffic state, and the first traffic signal action, a first reward value and the second traffic state at the next moment are determined. Since the first reward value can represent the preference of the first traffic state for the first traffic signal action, calculating the reward value can provide a basis for subsequent target control strategies. The target control strategy provides direction; the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state are stored as sample data in the experience replay pool; the traffic signal control model is trained by reinforcement learning based on the experience replay pool to obtain the target control strategy, and the traffic lights are controlled according to the target control strategy. Since the model is trained by reinforcement learning using the post-event experience replay mechanism, the impact of the sparse reward problem in traffic signal control can be solved, the performance of the model is improved, and it has better generalization for complex traffic scenarios, thus achieving efficient traffic signal control.

[0101] Based on the same technical concept, this application also provides a traffic signal control device, which can implement the traffic signal control method flow described above in this application.

[0102] Figure 3 This is a schematic diagram of a traffic signal control device provided in an embodiment of this application. As shown in the figure, the device includes: an information processing module 301, a first determination module 302, a second determination module 303, an experience playback module 304, and a control module 305.

[0103] The information processing module 301 is used to convert the acquired intersection traffic information to obtain a first traffic state; wherein, the intersection traffic information includes traffic light information and vehicle queue information.

[0104] The first determining module 302 is used to input the first traffic state into the traffic signal control model and determine the first traffic signal action.

[0105] The second determining module 303 is used to determine a first reward value and a second traffic state at the next moment based on a first expected target, the first traffic state, and the first traffic signal action; wherein the first reward value is used to characterize the preference of the first traffic state for the first traffic signal action, and the first expected target is an expected target randomly sampled from a set of targets.

[0106] The experience playback module 304 is used to store the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state as sample data into the experience playback pool.

[0107] The control module 305 is used to perform reinforcement learning training on the traffic signal control model based on the experience playback pool to obtain a target control strategy, and to control the traffic lights according to the target control strategy.

[0108] Optionally, the traffic signal control model includes a Critic network, an Actor network, an Actor target network, and a Critic target network; the control module 305 is specifically used for:

[0109] The second traffic state from the sample data is input into the Actor target network, and exploratory noise is introduced to determine the second traffic signal action; the second traffic state and the second traffic signal action are input into the Critic target network to determine the target Q value; based on the target Q value and the minimized loss function, the network parameters μ of the Critic network are updated. k Update the network parameters φ of the Actor network according to the gradient descent algorithm; based on the updated μ k Update the network parameters μ of the Critic target network. ′ k And based on the updated φ, update the network parameters φ′ of the Actor target network.

[0110] Optionally, the experience playback module 304 is further configured to:

[0111] Sample a second desired target from the target set; determine a second reward value based on the second desired target, the first traffic state, and the first traffic signal action; store the second desired target, the first traffic state, the first traffic signal action, the second reward value, and the second traffic state as new sample data in the experience playback pool.

[0112] It should be noted that the apparatus provided in this application embodiment can implement all the method steps in the above method embodiment and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0113] Based on the same technical concept, this application also provides an electronic device that can realize the function of the aforementioned traffic signal control device.

[0114] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0115] At least one processor 401 and a memory 402 connected to at least one processor 401. In this embodiment, the specific connection medium between the processor 401 and the memory 402 is not limited. Figure 4 The example shown is the connection between processor 401 and memory 402 via bus 400. Bus 400 is... Figure 4 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 400 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 4 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, processor 401 can also be called a controller; there is no restriction on the name.

[0116] In this embodiment, memory 402 stores instructions executable by at least one processor 401. By executing the instructions stored in memory 402, at least one processor 401 can execute a traffic signal control method described above. Processor 401 can implement... Figure 3 The functions of each module in the device shown.

[0117] The processor 401 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 402 and calling data stored in memory 402, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0118] In one possible design, processor 401 may include one or more processing units. Processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 401. In some embodiments, processor 401 and memory 402 may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0119] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of a traffic signal control method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0120] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 402 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 402 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 402 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0121] By designing and programming the processor 401, the code corresponding to the traffic signal control method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the code during operation. Figure 1 The illustrated embodiment is a traffic signal control method. How to design and program the processor 401 is a technique well-known to those skilled in the art and will not be described further here.

[0122] It should be noted that the electronic device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0123] This application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a traffic signal control method described in the above embodiments.

[0124] This application also provides a computer program product, which, when invoked by a computer, causes the computer to execute a traffic signal control method described in the above embodiments.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. A traffic signal control method characterized by, Comprising: Converting the obtained intersection traffic information to obtain a first traffic state; wherein the intersection traffic information comprises signal light information and vehicle queue information; wherein the signal light information comprises the phase state and the remaining time length of the traffic signal light, and the vehicle queue information comprises the vehicle queue length of each lane; Inputting the first traffic state into a traffic signal control model to determine a first traffic signal action; According to the first expected target, the first traffic state, the first traffic signal action, determine a first reward value, and a second traffic state at the next time; wherein the first reward value is used to represent the preference of the first traffic state for the first traffic signal action, and the first expected target is an expected target randomly sampled from a set of target; Store the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state as sample data in an experience replay pool; According to the experience replay pool, the traffic signal control model is trained by reinforcement learning to obtain a target control strategy, and the traffic signal light is controlled according to the target control strategy; Wherein, after the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state are stored as sample data in the experience replay pool, it further comprises: Sample a second expected target from the target set; According to the second expected target, the first traffic state, the first traffic signal action, determine a second reward value; Store the second expected target, the first traffic state, the first traffic signal action, the second reward value, and the second traffic state as new sample data in the experience replay pool.

2. The method of claim 1, wherein, The traffic signal control model comprises a Critic network, an Actor network, an Actor target network, and a Critic target network; According to the experience replay pool, the traffic signal control model is trained by reinforcement learning, comprising: Input the second traffic state in the sample data into the Actor target network, and introduce exploration noise to determine a second traffic signal action; Input the second traffic state and the second traffic signal action into the Critic target network to determine a target Q value; updating network parameters of the Critic network according to the target Q value and a loss function ; updating network parameters of the Actor network according to a gradient descent algorithm ; According to the updated , the network parameters of the Critic target network are updated , and according to the updated , the network parameters of the Actor target network are updated .

3. The method of claim 2, wherein, The satisfies the following expression: wherein, is the target Q value, is the Q value calculated by the Critic network according to the first traffic state and the first traffic signal action, the N is the number of the sample data, the is the first expected target, the is the second traffic state, the is the second traffic signal action, the is a hyperparameter.

4. The method of claim 2, wherein, The satisfies the following expression: Wherein, the N is the number of the sample data, the is the Actor network, the is the first traffic state, the is the first traffic signal action.

5. A traffic signal control device, characterized by Comprising: An information processing module for converting the obtained intersection traffic information to obtain a first traffic state; wherein the intersection traffic information comprises signal light information and vehicle queue information; wherein the signal light information comprises the phase state and the remaining time length of the traffic signal light, and the vehicle queue information comprises the vehicle queue length of each lane; A first determination module for inputting the first traffic state into a traffic signal control model to determine a first traffic signal action; The second determining module is configured to determine a first reward value and a second traffic state at a next time according to the first expected target, the first traffic state, and the first traffic signal action, wherein the first reward value is used to represent a preference of the first traffic state for the first traffic signal action, and the first expected target is an expected target randomly sampled from a set of targets; The experience replay module is configured to store the first traffic state, the first traffic signal action, the first expected target, the first reward value, and the second traffic state as sample data in an experience replay pool; The control module is configured to perform reinforcement learning training on the traffic signal control model according to the experience replay pool to obtain a target control strategy, and control a traffic signal lamp according to the target control strategy. The experience replay module is further configured to: sample a second expected target from the set of targets; determine a second reward value according to the second expected target, the first traffic state, and the first traffic signal action; store the second expected target, the first traffic state, the first traffic signal action, the second reward value, and the second traffic state as new sample data in the experience replay pool.

6. The apparatus of claim 5, wherein, The traffic signal control model includes a Critic network, an Actor network, an Actor target network, and a Critic target network. The control module is specifically configured to: input the second traffic state in the sample data into the Actor target network and introduce an exploration noise to determine a second traffic signal action; input the second traffic state and the second traffic signal action into the Critic target network to determine a target Q value. Update the network parameters of the Critic network based on the target Q-value and the minimized loss function. ; updating network parameters of the Actor network according to a gradient descent algorithm ; According to the updated , the network parameters of the Critic target network are updated , and according to the updated , the network parameters of the Actor target network are updated .

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-4. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-4. ​ 8. A computer-readable storage medium, characterized in that, ​

Images