A traffic signal lamp control method and device, terminal equipment and storage medium

Abstract

The application relates to the technical field of device control, and discloses a traffic signal lamp control method and device, a terminal device and a storage medium. The method comprises the following steps: acquiring driving state data of each vehicle at a traffic intersection, wherein the driving state data comprises a future driving direction of the corresponding vehicle at the traffic intersection; inputting the driving state data into a trained signal lamp control model for processing; and controlling the signal lamp at the traffic intersection by using a state action value function output by the signal lamp control model; wherein the signal lamp control model is a graph neural network trained based on a deep reinforcement learning algorithm, the nodes of the graph neural network are various phases of the signal lamp, the feature data of each node is the driving state data of the vehicles on all passable lanes under the corresponding phase, the state of the deep reinforcement learning algorithm is the driving state data of the vehicles, and the action is each phase of the signal lamp. The method can improve the accuracy of signal lamp timing, thereby improving the vehicle passing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of device control, and particularly relates to a traffic signal lamp control method and device, a terminal device and a storage medium. BACKGROUND

[0002] With the improvement of social living standards, the number of people owning vehicles is increasing, leading to increasingly serious traffic congestion problems in cities. At traffic intersections, traffic signal lamps are generally used to maintain the traffic order of vehicles. However, the phase conversion order and phase retention time of the existing traffic signal lamps are generally set in advance, without considering the actual traffic situation. Even though some existing technologies have proposed signal lamp control methods based on reinforcement learning, these methods basically only determine the signal lamp timing strategy at the next time according to the static information such as the vehicle queue length of each lane at the traffic intersection, and still have the problems of inaccurate timing and low vehicle traffic efficiency. SUMMARY

[0003] Therefore, the embodiments of the present application provide a traffic signal lamp control method and device, a terminal device and a storage medium, which can improve the accuracy of traffic intersection signal lamp timing and thus improve vehicle traffic efficiency.

[0004] A first aspect of the embodiments of the present application provides a traffic signal lamp control method, comprising:

[0005] obtaining driving state data of each vehicle in a traffic intersection, wherein the driving state data comprises a future driving direction of the corresponding vehicle at the traffic intersection;

[0006] inputting the driving state data into a trained signal lamp control model for processing, and controlling the signal lamp of the traffic intersection through a state-action value function output by the signal lamp control model;

[0007] wherein the signal lamp control model is a graph neural network trained based on a deep reinforcement learning algorithm, the nodes of the graph neural network are each phase of the signal lamp, the feature data of each node is the driving state data of the vehicles of all passable lanes under the corresponding phase, the state of the deep reinforcement learning algorithm is the driving state data of the vehicles, and the action of the deep reinforcement learning algorithm is each phase of the signal lamp.

[0008] In the embodiment of the present application, first, the driving state data of each vehicle in the traffic intersection is obtained, and the driving state data includes the future driving direction of the corresponding vehicle at the traffic intersection, such as straight, left turn or right turn, etc. Then, the driving state data is input into a pre-trained signal light control model for processing, and the state action value function output by the signal light control model can determine the executed signal light phase, thereby realizing the control of the traffic intersection signal. The signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm, the nodes of which are the phases of the signal light, and the feature data of the nodes is the driving state data of the vehicles on all passable lanes under the corresponding phase. The state of the deep reinforcement learning algorithm used is the driving state data of the vehicle, and the action is each phase of the signal light. The input information used in the embodiment of the present application contains the future driving direction of each vehicle in the traffic intersection. Since the future driving direction represents the future spatial position information of the vehicle in the traffic intersection, the future driving directions of all vehicles in the traffic intersection collectively represent the spatial dynamic characteristics of the traffic intersection, that is, the dynamic information of the vehicle running direction. The signal light control algorithm can consider the future traffic volume in each direction of the traffic intersection according to the spatial dynamic characteristics, thereby more intelligently and reasonably switching the signal light phase, so as to improve the accuracy of signal light timing and improve the vehicle passing efficiency.

[0009] In an implementation manner of the embodiment of the present application, the signal light control model can be obtained by the following method:

[0010] The agent of the deep reinforcement learning algorithm extracts a set number of experience records from the experience pool; wherein each experience record contains state update information obtained when the agent interacts with the traffic intersection, executed action, and reward feedback from the traffic intersection;

[0011] The set number of experience records are used as a training set to optimize and train the constructed graph neural network to obtain the signal light control model.

[0012] In an implementation manner of the embodiment of the present application, before the agent of the deep reinforcement learning algorithm extracts a set number of experience records from the experience pool, it can also include:

[0013] An evaluation network and a target network are constructed; wherein the evaluation network and the target network are graph neural networks with the same structure;

[0014] Based on the evaluation network, a plurality of experience records are generated by the agent interacting with the traffic intersection;

[0015] The plurality of experience records are stored in the experience pool;

[0016] The signal lamp control model is obtained by optimizing and training the constructed graph neural network with the set number of experience records as a training set, including:

[0017] The signal lamp control model is obtained by optimizing and training the evaluation network with a loss function constructed based on a state-action value function output by the evaluation network and a state-action value function output by the target network, with the set number of experience records as a training set.

[0018] Further, the plurality of experience records can be generated by the agent interacting with the traffic intersection based on the evaluation network, which can include:

[0019] Obtain the driving state data of each vehicle in the traffic intersection at the first time;

[0020] Preprocess the driving state data of each vehicle in the traffic intersection at the first time to obtain first node feature data of each node;

[0021] The first node feature data of each node is input into the evaluation network as a first state, and the agent selects a target action from an action space according to a state-action value function output by the evaluation network;

[0022] Control the signal lamp to switch phases according to the target action;

[0023] After the signal lamp switches phases, obtain the driving state data of each vehicle in the traffic intersection at the second time;

[0024] Preprocess the driving state data of each vehicle in the traffic intersection at the second time to obtain second node feature data of each node, and input the second node feature data as a second state;

[0025] Obtain the target reward fed back by the traffic intersection to the agent;

[0026] Generate an experience record according to the first state, the target action, the target reward, and the second state.

[0027] Further, the driving state data of each vehicle in the traffic intersection at the first time can be preprocessed to obtain first node feature data of each node, which can include:

[0028] The future driving direction in the driving state data of each vehicle in the traffic intersection at the first time is processed by one-hot encoding;

[0029] For each of the nodes, obtain, from driving state data of each vehicle in the traffic intersection at a first time, driving state data of all vehicles in all passable lanes in a phase corresponding to the node, and perform fusion processing on the driving state data of all the vehicles to obtain first node feature data of the node.

[0030] Further, the fusion processing on the driving state data of all the vehicles to obtain the first node feature data of the node can include:

[0031] superimposing the driving state data of all the vehicles to obtain the first node feature data of the node;

[0032] Alternatively, the fusion processing on the driving state data of all the vehicles to obtain the first node feature data of the node can include using a multi-layer perception machine.

[0033] Further, the obtaining of the target reward of the traffic intersection fed back to the agent can include:

[0034] obtaining a vehicle queue length and a vehicle waiting time of the traffic intersection at the second time;

[0035] calculating the target reward according to the vehicle queue length and the vehicle waiting time.

[0036] In an implementation manner of the embodiment of the application, the obtaining of the driving state data of each vehicle in the traffic intersection can include:

[0037] obtaining navigation data through a navigation device of each vehicle;

[0038] extracting the driving state data of each vehicle from the navigation data.

[0039] A second aspect of the embodiment of the application provides a control device of a traffic signal lamp, including:

[0040] a vehicle data obtaining module configured to obtain driving state data of each vehicle in a traffic intersection, the driving state data including a future driving direction of a corresponding vehicle in the traffic intersection;

[0041] a signal lamp control module configured to input the driving state data to a trained signal lamp control model for processing, and control a signal lamp of the traffic intersection according to a state action value function output by the signal lamp control model;

[0042] The signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm, nodes of the graph neural network are respective phases of the signal light, feature data of each node is driving state data of vehicles of all passable lanes in the corresponding phase, a state of the deep reinforcement learning algorithm is the driving state data of the vehicles, and an action of the deep reinforcement learning algorithm is each phase of the signal light.

[0043] A third aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the traffic signal light control method provided in the first aspect of the embodiments of the present application when executing the computer program.

[0044] A fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the traffic signal light control method provided in the first aspect of the embodiments of the present application.

[0045] A fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to execute the traffic signal light control method provided in the first aspect of the embodiments of the present application.

[0046] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a traffic signal light control method provided by the embodiments of the present application;

[0048] Figure 2 is an operation flow diagram of the traffic signal light control method provided by the embodiments of the present application in an actual application scenario;

[0049] Figure 3 is a node structure diagram of a graph neural network provided by the embodiments of the present application;

[0050] Figure 4 is a structure diagram of a traffic signal light control device provided by the embodiments of the present application;

[0051] Figure 5 is a schematic diagram of a terminal device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0052] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0053] The control strategy of traffic signal lights is crucial for maintaining the vehicle passing order of the traffic intersection and improving the vehicle passing efficiency. However, the existing control strategy basically only considers the static information such as the vehicle queue length of each lane in the traffic intersection, and does not consider the influence of the dynamic characteristics (such as speed and lane changing trend) of each vehicle at the intersection and the vehicle running space characteristics of each intersection on the current intersection and the downstream intersection. Therefore, the existing control strategy has the problems of inaccurate signal light timing and low vehicle passing efficiency. In view of this, the embodiments of the present application provide a traffic signal light control method, device, terminal equipment and storage medium, which can improve the accuracy of traffic intersection signal light timing and thus improve the vehicle passing efficiency. For more specific technical implementation details of the embodiments of the present application, please refer to the method embodiments described below.

[0054] It should be understood that the execution subject of each method embodiment of the present application is various types of terminal equipment or servers, for example, can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a large-screen television, etc., and the specific type of the terminal equipment and server is not limited by the embodiments of the present application.

[0055] Please refer to Figure 1 , which shows a traffic signal light control method provided by the embodiments of the present application, which includes:

[0056] 101, obtaining the driving state data of each vehicle in the traffic intersection, the driving state data including the future driving direction of the corresponding vehicle at the traffic intersection;

[0057] The execution subject of the method embodiment can be any type of terminal device for controlling various signal lights in a traffic intersection. First, the terminal device acquires driving state data of each vehicle in the traffic intersection. The driving state data can include, but is not limited to, a future driving direction of the vehicle at the traffic intersection (such as straight, left turn, right turn, or U-turn), vehicle speed, and distance from the traffic intersection, etc. The driving state data can be acquired by a navigation device of each vehicle or a laser radar device arranged at the traffic intersection, and sent to the terminal device.

[0058] In an implementation manner of the embodiment of the application, the acquiring of the driving state data of each vehicle in the traffic intersection can include:

[0059] (1) acquiring navigation data by a navigation device of each vehicle;

[0060] (2) extracting driving state data of each vehicle from the navigation data.

[0061] The navigation data of each vehicle can be conveniently acquired by a navigation device (such as a vehicle-mounted navigation device or a mobile phone used by a driver, etc.) of each vehicle. The navigation data includes vehicle speed, future driving direction at the current traffic intersection, distance from the traffic intersection, and intersection camera position, etc. On one hand, the terminal device can acquire the navigation data of each vehicle by wireless interaction with the navigation device of each vehicle in the traffic intersection. On the other hand, the navigation data of each vehicle is generally provided by a server of a vehicle navigation system, so the terminal device can acquire the navigation data of each vehicle by wireless interaction with the server. After acquiring the navigation data of each vehicle in the traffic intersection, the terminal device can extract the required part of the data as the driving state data of each vehicle, such as vehicle speed, future driving direction at the current traffic intersection, and distance from the traffic intersection, etc.

[0062] 102. inputting the driving state data into the trained signal light control model for processing, and controlling the signal lights of the traffic intersection by a state action value function output by the signal light control model.

[0063] In an embodiment of the method, a graph neural network is pre-trained as a signal light control model based on a deep reinforcement learning algorithm. After the terminal device obtains the driving state data of each vehicle at the traffic intersection, the driving state data can be input into the signal light control model as input information, and the state-action value function output by the signal light control model is used to control the signal lights at the traffic intersection. The nodes of the graph neural network are set as the phases of the signal lights (one phase represents a display state of all signal lights at the traffic intersection, for example, the "east-west straight" phase represents that the signal light in the east-west straight direction at the traffic intersection displays a green light, and the signal lights in other directions display red lights, and so on), and the feature data of each node is the driving state data of all vehicles on the passable lane in the corresponding phase, for example, for the "east-west straight" phase, the feature data is the driving state data of all vehicles on the east-west straight lane. The state s of the deep reinforcement learning algorithm used is the driving state data of the vehicle, and the action a is each phase of the signal light. Specifically, the state-action value function output by the signal light control model can determine the currently executed action a, so as to set each signal light at the traffic intersection to the phase corresponding to the action a.

[0064] In an implementation manner of an embodiment of the present application, the signal light control model can be obtained by training in the following manner:

[0065] (1) An agent of the deep reinforcement learning algorithm extracts a set number of experience records from an experience pool; wherein each experience record contains state update information obtained when the agent interacts with the traffic intersection, an executed action, and a reward feedback from the traffic intersection.

[0066] (2) The set number of experience records are used as a training set to optimize and train the constructed graph neural network, and the signal light control model is obtained.

[0067] In the design of the graph neural network, each phase of the signal lamp is taken as a node of the graph representation, the nodes are connected by edges, the feature of each edge is the signal lamp phase code (which can be a weight coefficient in general) pointed to the node, and the feature data of each node is the running state data of all vehicles on all passable lanes in the corresponding phase. Here, the running state data of all vehicles on all passable lanes in the phase can be processed to obtain the processed data as the feature data of the node. For example, the running state data of all vehicles can be superimposed as the feature data of the node, or the running state data of all vehicles can be spliced and input into a fully connected neural network, and then the output of the fully connected neural network is taken as the feature data of the node. In addition, the running state data of all vehicles can be processed in various ways according to other application requirements. After the data processing is completed, a graph neural network is constructed as an extractor of dynamic and spatial features of the traffic intersection, and the graph neural network is also required by the deep reinforcement learning algorithm (DQN algorithm).

[0068] In the application of the deep reinforcement learning algorithm, multiple experience records can be generated by the interaction between the agent and the traffic intersection, wherein each experience record contains the state update information obtained when the agent interacts with the traffic intersection, the executed action, and the reward feedback of the traffic intersection. The generated experience records can be stored in the experience pool of the algorithm. When training the graph neural network, the agent of the deep reinforcement learning algorithm can extract a certain number of experience records (for example, random extraction according to the set batch-size) from the experience pool, and then use the extracted experience records as the training set to optimize and train the previously constructed graph neural network. The optimized and trained graph neural network is the signal lamp control model described above.

[0069] In an implementation manner of the embodiment of the present application, before the agent of the deep reinforcement learning algorithm extracts a certain number of experience records from the experience pool, the following steps can be further included:

[0070] (1) constructing an evaluation network and a target network; wherein the evaluation network and the target network are graph neural networks with the same structure;

[0071] (2) generating multiple experience records based on the evaluation network by the interaction between the agent and the traffic intersection;

[0072] (3) storing the multiple experience records in the experience pool.

[0073] The deep reinforcement learning algorithm needs to construct two graph neural networks with the same structure, which are respectively called evaluation network (the evaluation network can also be called main network) and target network, the weight parameters of the evaluation network are represented by θ, and the weight parameters of the target network are represented by θ'. Based on the evaluation network, the agent and the traffic intersection (as an external environment) of the deep reinforcement learning algorithm can generate an experience record every time they interact, and multiple experience records can be generated after multiple interactions. When a sufficient number of experience records are generated (that is, when the number of interaction steps reaches a set threshold), all the experience records are stored in the experience pool.

[0074] The evaluation network is optimized and trained using the loss function constructed based on the state-action value function output by the evaluation network and the state-action value function output by the target network, using the set number of experience records as the training set, to obtain the signal light control model.

[0075] The evaluation network is optimized and trained using the loss function constructed based on the state-action value function output by the evaluation network and the state-action value function output by the target network, using the set number of experience records as the training set, to obtain the signal light control model.

[0076] After obtaining a sufficient number of experience records and storing them in the experience pool, the evaluation network can be optimized and trained to obtain the final signal light control model. At this time, the agent of the deep reinforcement learning algorithm extracts a certain number of experience records from the experience pool as a training set, and optimizes and trains the evaluation network. The agent learns the optimal signal light phase control strategy by minimizing the loss function. The loss function here can be constructed based on the state-action value function Q θ′ output by the evaluation network and the state-action value function Q θ output by the target network, for example, the following loss function can be used:

[0077] Loss=E(r+γmaxQ θ′ -Q θ ) 2

[0078] Wherein, Loss represents the loss function, E represents the mathematical expectation, r represents the reward feedback by the traffic intersection to the agent, and γ represents the decay factor.

[0079] When the evaluation network is optimized and trained by minimizing the loss function, a multi-round training method can be used. Every N training rounds, the weight parameters θ of the evaluation network are assigned to the weight parameters θ' of the target network, so that the weight parameter update of the target network is completed. After repeated execution of multiple rounds of training, the agent can learn the optimal signal light phase control strategy, and the training process is ended at this time. The evaluation network obtained after training optimization is used as the signal light control model.

[0080] Specifically, the generating, by the agent, a plurality of experience records based on the interaction between the agent and the traffic intersection can include:

[0081] (1) obtaining driving state data of each vehicle in the traffic intersection at a first time;

[0082] (2) preprocessing the driving state data of each vehicle in the traffic intersection at the first time to obtain first node feature data of each node;

[0083] (3) inputting the first node feature data of each node into the evaluation network as a first state for processing, and selecting a target action from an action space according to a state-action value function output by the evaluation network;

[0084] (4) controlling the signal light to switch phases according to the target action;

[0085] (5) obtaining driving state data of each vehicle in the traffic intersection at a second time after the signal light switches phases;

[0086] (6) preprocessing the driving state data of each vehicle in the traffic intersection at the second time to obtain second node feature data of each node, and inputting the second node feature data as a second state;

[0087] (7) obtaining a target reward fed back by the traffic intersection to the agent;

[0088] (8) generating an experience record according to the first state, the target action, the target reward, and the second state.

[0089] In the evaluation network, when the agent interacts with the traffic intersection in the way of generating experience records. First, at the beginning of the interaction, assuming that the current time is the first time, the driving state data of each vehicle in the traffic intersection at the first time is obtained; then, the driving state data of each vehicle is preprocessed, which mainly fuses the driving state data of all vehicles in the passable lane under each node phase, thereby obtaining the node feature data of each node (represented by the first node feature data); next, the first node feature data of each node is input as the state s (represented by the first state) into the evaluation network for processing, and the evaluation network can output a corresponding state-action value function (Q value in deep reinforcement learning algorithm, i.e. Q(s, a)), and the agent selects an action a (phase) from the action space (corresponding to the saved phases of each signal light) according to the state-action value function, which is represented by the target action; after the action a is determined, the signal lights of the traffic intersection perform the corresponding phase switching according to the action a, and the vehicle situation of the traffic intersection also changes; then, assuming that this time is the second time, the driving state data of each vehicle in the traffic intersection at the second time is obtained again, and the same method is used to obtain the second node feature data of each node, which can be used as the next state s' (represented by the second state); in addition, the traffic intersection can also feed back a reward r (represented by the target reward) to the agent, and the reward r is used to guide the agent to adjust the strategy during learning and training, so as to improve the traffic efficiency of the traffic intersection.

[0090] Specifically, the target reward fed back by the traffic intersection to the agent can include:

[0091] (1) obtaining the vehicle queue length and vehicle waiting time of the traffic intersection at the second time;

[0092] (2) calculating the target reward according to the vehicle queue length and vehicle waiting time.

[0093] The design of the reward r can include the queue length of each lane and the waiting time of each vehicle in the traffic intersection, and other index parameters can be reasonably added according to the application needs. Specifically, the vehicle queue length and vehicle waiting time of each lane of the traffic intersection at the second time can be obtained, and then the corresponding target reward r is calculated according to the vehicle queue length and vehicle waiting time by using a preset formula.

[0094] At this point, an experience record (s, a, r, s') can be obtained, and the experience record is stored in the experience pool. Next, the state s' can be used as the current state to continue inputting the evaluation network for processing, and the same processing process is repeatedly performed until the experience pool stores a sufficient number of experience records.

[0095] At each decision-making time step, when the state s is input to the evaluation network, the evaluation network performs a corresponding information transmission operation, wherein for a specific node v, the information transmission operation can be represented by the following formula:

[0096]

[0097]

[0098] wherein, is the information accepted by the node v at the t+1 time step, N(v) is all adjacent nodes of the node v, is the feature vector (i.e., node feature data) of the node v at the t time step, e vw is the edge feature between the node v and the node w, M t is a preset message function, U t is a preset node update function. After the state s is processed by the above information transmission operation, the corresponding Q value is generated.

[0099] Further, the preprocessing of the driving state data of each vehicle in the traffic intersection at the first time to obtain the first node feature data of each node can include:

[0100] (1) performing one-hot encoding processing on the future driving direction in the driving state data of each vehicle in the traffic intersection at the first time;

[0101] (2) for each node, obtaining the driving state data of all vehicles in all passable lanes under the phase corresponding to the node from the driving state data of each vehicle in the traffic intersection at the first time, and performing fusion processing on the driving state data of all vehicles to obtain the first node feature data of the node.

[0102] In preprocessing the driving state data of the vehicles, the future driving directions therein can be processed by one-hot encoding. For example, after one-hot encoding, "straight" can be [1, 0, 0, 0], "left turn" can be [0, 1, 0, 0], "right turn" can be [0, 0, 1, 0], and "U-turn" can be [0, 0, 0, 1]. By processing the future driving directions by one-hot encoding, the future driving directions of the vehicles can be fused conveniently subsequently. For each node, first, the driving state data of all the vehicles on all the passable lanes in the phase corresponding to the node are obtained, and then the driving state data of all the vehicles are fused to obtain the node feature data of the node. For example, for the "east-west straight" phase node, the passable lane corresponding thereto is the east-west straight lane, and assuming that there are 10 vehicles driving on the east-west straight lane, the driving state data of the 10 vehicles can be fused to obtain the node feature data of the "east-west straight" phase node. Obviously, each node can obtain the corresponding node feature data in the same manner.

[0103] Further, the fusing the driving state data of all the vehicles to obtain the first node feature data of the node can include:

[0104] stacking the driving state data of all the vehicles to obtain the first node feature data of the node;

[0105] Alternatively, the fusing the driving state data of all the vehicles to obtain the first node feature data of the node can include using a multi-layer perception.

[0106] In fusing the driving state data of the vehicles, one way is to stack the driving state data of the vehicles, that is, to sum the driving state data to obtain the final node feature data of the corresponding node. Another way is to use a multi-layer perception (MLP), and the driving state data of the vehicles can be fused by the multi-layer perception to obtain the final node feature data of the corresponding node. The specific working principle of the multi-layer perception can refer to the prior art, and will not be described herein.

[0107] The technical key point of the embodiments of the present application is to obtain the future driving direction of each vehicle at the traffic intersection, and fuse the future driving direction into the feature data of each node of the graph neural network after one-hot encoding. The future driving direction represents the future spatial position information of the vehicle at the traffic intersection, and the future driving directions of all vehicles in all directions of the current intersection jointly represent the spatial dynamic characteristics of the intersection, that is, the dynamic information of the running direction of the vehicles at the intersection. Therefore, the one-hot encoding of the future driving directions of all vehicles at the intersection can represent the spatial dynamic characteristics of all vehicles at the intersection at the future time. The spatial dynamic characteristics are very important for the signal control strategy of the intersection. The signal control is actually to control the vehicle flow in each direction of the intersection by changing the signal phase, so as to improve the traffic efficiency of the intersection and alleviate the congestion. However, the upstream intersection and the downstream intersection are closely related, and the vehicle flow at the upstream intersection will also affect the vehicle traffic efficiency at the downstream intersection, so the static queue length and vehicle number data cannot be simply used as the input of the signal control strategy. The future driving direction is not represented, so it is difficult for the control model to develop a more reasonable strategy to control the vehicle flow at the intersection. The future driving direction information encoded by one-hot encoding can well depict the spatial dynamic characteristics of each vehicle at the intersection at the future time, and the control model can intelligently convert the phase of the signal light according to the spatial dynamic characteristics. Since each vehicle will be transferred from the original driving direction of the intersection to the driving direction of other positions of the intersection under the control of the signal light, and then the vehicle in the driving direction will become the driving direction of the corresponding position of the downstream intersection, from the perspective of the phase, the vehicle is equivalent to being transferred from one node of the graph neural network to another node, and the encoded future driving direction data is also transferred from the current node to another node through data flow as the feature data of another node. The spatial dynamic characteristic data encoded by one-hot encoding can be mined through the continuous information transmission and reverse propagation operation of the graph neural network. The agent of the deep reinforcement learning algorithm can continuously interact and learn according to the output information of the graph neural network, and finally learn the mapping relationship between the optimal spatial characteristics of the intersection and the phase change of the signal light.

[0108] In an embodiment of the present application, first, the driving state data of each vehicle at a traffic intersection is obtained. The driving state data includes the future driving direction of the corresponding vehicle at the traffic intersection, such as going straight, turning left, or turning right. Then, these driving state data are input into a pre-trained signal light control model for processing. The state-action value function output by the signal light control model can determine the executed signal light phase, thereby realizing the control of the traffic intersection signal. Among them, the signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm. Its nodes are the phases of the signal light, and the feature data of its nodes are the driving state data of vehicles in all passable lanes under the corresponding phase. The state of the deep reinforcement learning algorithm used is the driving state data of the vehicle, and the action is the phase of the signal light. The input information used in the embodiment of the present application includes the future driving direction of each vehicle at the traffic intersection. Since the future driving direction represents the future spatial position information of the vehicle at the traffic intersection, the future driving directions of all vehicles at the traffic intersection jointly represent the spatial dynamic characteristics of the traffic intersection, that is, the dynamic information of the vehicle's running direction. Based on these spatial dynamic characteristics, the traffic light control algorithm can take into account the future traffic flow in all directions of the traffic intersection, thereby shifting the signal light phase more intelligently and reasonably, thereby improving the accuracy of signal light timing and enhancing vehicle traffic efficiency.

[0109] To facilitate understanding of the traffic light control method provided in the embodiment of the present application, a practical application scenario is listed below. Figure 2 1 is a schematic diagram of the operation flow of the traffic light control method provided in an embodiment of the present application in an actual application scenario.

[0110] exist Figure 2 In

[15] , we first obtain the navigation data of all vehicles at the intersection and process the navigation data to obtain the vehicle's driving status data, which includes the vehicle's speed, distance from the intersection, and future driving method. Then, we construct a graph neural network and use the phase of the traffic light as a graph node, such as Figure 3 As shown in the figure, it is a schematic diagram of a node structure of a graph neural network. Figure 3 It contains nodes corresponding to four different traffic light phases: "north-south straight", "east-west straight", "north-south left turn" and "east-west left turn"; the vehicle driving status data corresponding to each phase are fused and processed to obtain the feature data of each node of the graph neural network, that is, the state s of the deep reinforcement learning algorithm; then, the intelligent agent of the deep reinforcement learning algorithm interacts with the intersection to collect experience records and store them in the experience pool; then, a certain number of experience records are extracted from the experience pool to train and optimize the graph neural network, so that the intelligent agent learns the optimal control strategy and obtains the final traffic light control model; finally, the traffic light control model is used to complete the control of the traffic lights at the traffic intersection.

[0111] In summary, the embodiment of the present application constructs all signal light phases of a traffic intersection into a graph representation, comprehensively considers the spatial characteristics of each phase of the current intersection, uses the data transmission and data fusion characteristics between nodes of the graph neural network, extracts the spatial characteristics of the intersection based on dynamic data mining of the intersection, thereby obtaining the spatial relationship of vehicle operation in each direction of the intersection, and simultaneously uses a deep reinforcement learning algorithm to learn the mapping relationship between the spatial and temporal characteristics of the intersection and the signal light control to achieve optimal sequence control of the signal light. Under the strong power of the graph neural network in mining the spatial characteristics of the intersection, the agent of the deep reinforcement learning algorithm can formulate an optimal signal light control strategy according to the dynamic information of the intersection, thereby improving the vehicle passing efficiency of the intersection.

[0112] It should be understood that the size of the serial number of each step in each of the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0113] The above mainly describes a traffic signal light control method, and a traffic signal light control device will be described below.

[0114] Please refer to Figure 4 An embodiment of a traffic signal light control device in the embodiment of the present application includes:

[0115] A vehicle data acquisition module 401 is configured to acquire driving state data of each vehicle in a traffic intersection, wherein the driving state data includes the future driving direction of the corresponding vehicle at the traffic intersection.

[0116] A signal light control module 402 is configured to input the driving state data into a trained signal light control model for processing, and control the signal light of the traffic intersection through a state-action value function output by the signal light control model.

[0117] The signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm, the nodes of the graph neural network are each phase of the signal light, the feature data of each node is the driving state data of the vehicle of all passable lanes under the corresponding phase, the state of the deep reinforcement learning algorithm is the driving state data of the vehicle, and the action of the deep reinforcement learning algorithm is each phase of the signal light.

[0118] In an implementation manner of the embodiment of the present application, the traffic signal light control device can further include:

[0119] an experience record extraction module configured to extract a set number of experience records from an experience pool by an agent of the deep reinforcement learning algorithm, wherein each of the experience records comprises state update information obtained when the agent interacts with the traffic intersection, an executed action, and a reward fed back by the traffic intersection;

[0120] a graph neural network training module configured to optimize training of the constructed graph neural network by taking the set number of experience records as a training set, to obtain the signal lamp control model.

[0121] In an implementation form of the application, the control device of the traffic signal can further include:

[0122] a network construction module configured to construct an evaluation network and a target network, wherein the evaluation network and the target network are graph neural networks of the same structure;

[0123] an experience record generation module configured to generate a plurality of experience records based on the evaluation network by means of interaction of the agent with the traffic intersection;

[0124] an experience record storage module configured to store the plurality of experience records in the experience pool;

[0125] The graph neural network training module can be specifically configured to optimize training of the evaluation network by taking the set number of experience records as a training set and using a loss function constructed based on a state-action value function output by the evaluation network and a state-action value function output by the target network, to obtain the signal lamp control model.

[0126] Further, the experience record generation module can include:

[0127] a first vehicle data acquisition unit configured to acquire driving state data of each vehicle in the traffic intersection at a first time;

[0128] a first data preprocessing unit configured to preprocess the driving state data of each vehicle in the traffic intersection at the first time to obtain first node feature data of each of the nodes;

[0129] a vehicle data input unit configured to input the first node feature data of each of the nodes as a first state to the evaluation network for processing, and select a target action from an action space by the agent according to a state-action value function output by the evaluation network;

[0130] a phase switching unit configured to control the signal lamp to switch phases according to the target action;

[0131] a second vehicle data acquisition unit, configured to acquire driving state data of each vehicle in the traffic intersection at a second time after the signal light switching phase;

[0132] a second data preprocessing unit, configured to preprocess the driving state data of each vehicle in the traffic intersection at the second time to obtain second node feature data of each node, and use the second node feature data as a second state;

[0133] an intersection reward acquisition unit, configured to acquire a target reward fed back by the traffic intersection to the intelligent agent;

[0134] an experience record generation unit, configured to generate an experience record according to the first state, the target action, the target reward and the second state.

[0135] Further, the first data preprocessing unit can include:

[0136] a one-hot encoding processing unit, configured to perform one-hot encoding processing on a future driving direction in the driving state data of each vehicle in the traffic intersection at the first time;

[0137] a data fusion processing unit, configured to, for each node, acquire driving state data of all vehicles in all passable lanes in a phase corresponding to the node from the driving state data of each vehicle in the traffic intersection at the first time, and perform fusion processing on the driving state data of all the vehicles to obtain first node feature data of the node.

[0138] Further, the data fusion processing unit can include:

[0139] a data superposition processing subunit, configured to superimpose the driving state data of all the vehicles to obtain the first node feature data of the node;

[0140] a data fusion processing subunit, configured to use a multi-layer perception machine to fuse the driving state data of all the vehicles to obtain the first node feature data of the node.

[0141] Further, the intersection reward acquisition unit can include:

[0142] an intersection queue data acquisition subunit, configured to acquire a vehicle queue length and a vehicle waiting time of the traffic intersection at the second time;

[0143] an intersection reward calculation subunit, configured to calculate the target reward according to the vehicle queue length and the vehicle waiting time.

[0144] In an implementation manner of the embodiment of the application, the vehicle data acquisition module can include:

[0145] a navigation data obtaining unit, configured to obtain navigation data by a navigation device of each vehicle;

[0146] a vehicle data extracting unit, configured to extract driving state data of each vehicle from the navigation data.

[0147] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the traffic signal control method described in any of the above embodiments.

[0148] The embodiment of the present application further provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the traffic signal control method described in any of the above embodiments.

[0149] Figure 5 is a schematic diagram of a terminal device provided by an embodiment of the present application. As shown in the figure, Figure 5 the terminal device 5 of the embodiment includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 implements the steps in the embodiments of the traffic signal control method described above when executing the computer program 52, such as steps 101-102 shown in the figure. Figure 1 Alternatively, the processor 50 implements the functions of the modules / units in the above device embodiments when executing the computer program 52, such as the functions of the modules 401-402 shown in the figure. Figure 4 Alternatively, the processor 50 implements the functions of the modules / units in the above device embodiments when executing the computer program 52, such as the functions of the modules 401-402 shown in the figure.

[0150] The computer program 52 can be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 52 in the terminal device 5.

[0151] The processor 50 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0152] The memory 51 can be an internal storage unit of the terminal device 5, for example, a hard disk or a memory of the terminal device 5. The memory 51 can also be an external storage device of the terminal device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 51 can also include both the internal storage unit and the external storage device of the terminal device 5. The memory 51 is used to store the computer program and other programs and data required by the terminal device. The memory 51 can also be used to temporarily store data that has been output or is to be output.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, apparatus and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0155] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0156] Those skilled in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the above-described system embodiments are only schematic, for example, the division of the modules or units is only a logical function division, and there can be another division in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0158] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0159] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0160] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0161] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A traffic signal control method characterized by, The method comprises: obtaining driving state data of each vehicle in a traffic intersection, the driving state data comprising a future driving direction of the corresponding vehicle at the traffic intersection; inputting the driving state data into a trained signal light control model for processing, and controlling the signal light of the traffic intersection through a state-action value function output by the signal light control model; wherein the signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm, nodes of the graph neural network are respective phases of the signal light, feature data of each node is driving state data of vehicles of all passable lanes under the corresponding phase, a state of the deep reinforcement learning algorithm is the driving state data of the vehicles, and an action of the deep reinforcement learning algorithm is each phase of the signal light; the signal light control model is trained in the following manner: constructing an evaluation network and a target network, the evaluation network and the target network being graph neural networks of the same structure; generating a plurality of experience records based on the evaluation network through an agent of the deep reinforcement learning algorithm interacting with the traffic intersection; storing the plurality of experience records in an experience pool; extracting a set number of experience records from the experience pool by the agent; wherein each experience record contains state update information obtained when the agent interacts with the traffic intersection, an executed action, and a reward feedback from the traffic intersection; using a set number of experience records as a training set, and using a loss function constructed based on a state-action value function output by the evaluation network and a state-action value function output by the target network to optimize and train the evaluation network to obtain the signal light control model; wherein the generating of the plurality of experience records based on the evaluation network through the agent of the deep reinforcement learning algorithm interacting with the traffic intersection comprises: obtaining driving state data of each vehicle in the traffic intersection at a first time; preprocessing the driving state data of each vehicle in the traffic intersection at the first time to obtain first node feature data of each node; inputting the first node feature data of each node as a first state into the evaluation network for processing, and selecting a target action from an action space by the agent according to a state-action value function output by the evaluation network; controlling the signal light to switch phases according to the target action; after the signal light switches phases, obtaining driving state data of each vehicle in the traffic intersection at a second time; preprocessing the driving state data of each vehicle in the traffic intersection at the second time to obtain second node feature data of each node, and inputting the second node feature data as a second state; obtaining a target reward feedback from the agent by the traffic intersection; generating an experience record according to the first state, the target action, the target reward, and the second state.

2. The control method according to claim 1, characterized by, the preprocessing of the driving state data of each vehicle in the traffic intersection at the first time to obtain the first node feature data of each node comprises: performing one-hot encoding processing on the future driving direction in the driving state data of each vehicle in the traffic intersection at the first moment; For each node, obtain the driving state data of all vehicles in all passable lanes in the phase corresponding to the node from the driving state data of each vehicle in the traffic intersection at the first moment, and fuse the driving state data of all vehicles to obtain the first node feature data of the node.

3. The control method according to claim 2, characterized by, The fusion processing of the driving state data of all vehicles to obtain the first node feature data of the node comprises: superimposing the driving state data of all vehicles to obtain the first node feature data of the node; Or, using a multi-layer perception to fuse the driving state data of all vehicles to obtain the first node feature data of the node.

4. The control method according to claim 1, characterized by, The target reward of the traffic intersection fed back to the agent comprises: obtaining the vehicle queue length and vehicle waiting time of the traffic intersection at the second moment; According to the vehicle queue length and the vehicle waiting time, the target reward is calculated.

5. The control method according to any one of claims 1 to 4, characterized by, The driving state data of each vehicle in the traffic intersection is obtained by: obtaining navigation data through the navigation device of each vehicle; extracting the driving state data of each vehicle from the navigation data.

6. A control device for a traffic signal, characterized by It includes: a vehicle data acquisition module for acquiring driving state data of each vehicle in a traffic intersection, the driving state data including the future driving direction of the corresponding vehicle in the traffic intersection; a signal light control module for inputting the driving state data into a trained signal light control model for processing, and controlling the signal light of the traffic intersection through the state action value function output by the signal light control model; wherein the signal light control model is a graph neural network trained based on a deep reinforcement learning algorithm, the nodes of the graph neural network are each phase of the signal light, the feature data of each node is the driving state data of all vehicles in all passable lanes under the corresponding phase, and the state of the deep reinforcement learning algorithm is the driving state data of the vehicle, and the action of the deep reinforcement learning algorithm is each phase of the signal light; The control device of the traffic signal also comprises: a network construction module for constructing an evaluation network and a target network, the evaluation network and the target network being graph neural networks with the same structure; an experience record generation module for generating a plurality of experience records based on the evaluation network through the way in which the agent of the deep reinforcement learning algorithm interacts with the traffic intersection; an experience record storage module for storing the plurality of experience records in an experience pool; an experience record extraction module for extracting a set number of experience records from the experience pool by the agent; wherein each experience record contains state update information obtained when the agent interacts with the traffic intersection, an executed action, and a reward feedback from the traffic intersection; The graph neural network training module is configured to use a loss function constructed based on a state-action value function output by the evaluation network and a state-action value function output by the target network to optimize and train the evaluation network using a set number of the experience records as a training set, so as to obtain the signal lamp control model. The experience record generation module includes: The first vehicle data acquisition unit is configured to acquire driving state data of each vehicle at the traffic intersection at the first time point. The first data preprocessing unit is configured to preprocess the driving state data of each vehicle at the traffic intersection at the first time point to obtain first node feature data of each node. The vehicle data input unit is configured to input the first node feature data of each node as a first state into the evaluation network for processing, and select a target action from an action space according to a state-action value function output by the evaluation network. The phase switching unit is configured to control the signal lamp to switch phases according to the target action. The second vehicle data acquisition unit is configured to acquire driving state data of each vehicle at the traffic intersection at a second time point after the signal lamp switches phases. The second data preprocessing unit is configured to preprocess the driving state data of each vehicle at the traffic intersection at the second time point to obtain second node feature data of each node, and input the second node feature data as a second state. The intersection reward acquisition unit is configured to acquire a target reward fed back by the traffic intersection to the agent. The experience record generation unit is configured to generate an experience record according to the first state, the target action, the target reward, and the second state.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the traffic signal lamp control method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the traffic signal lamp control method of any one of claims 1 to 5.

Images