Reinforcement learning based traffic cooperative signal control method and system

Abstract

The application discloses a traffic cooperative signal control method and system based on reinforcement learning, and the method comprises the following steps: acquiring target traffic flow data; modeling the target traffic flow data by using a cellular transmission model to obtain target cellular data; constructing intersection state data of a road intersection according to the cellular state data of the target cellular data; inputting the intersection state data into a target intelligent agent value network in a target signal control intelligent agent to output a target phase action corresponding to the intersection state data; the target signal control intelligent agent is constructed based on deep reinforcement learning, one target signal control intelligent agent corresponds to one intersection signal lamp of the road intersection; and the intersection signal lamp of the road intersection is controlled according to the target phase action. The method can respond to the change of traffic flow in real time, make instant adjustment according to real-time traffic conditions, improve traffic flow smoothness and reduce traffic congestion, and optimize traffic efficiency, and can be widely applied to the technical field of traffic control.

Description

Technical Field

[0001] The present application relates to the field of traffic control technology, and in particular to a traffic cooperative signal control method and system based on reinforcement learning. Background Art

[0002] At present, traffic congestion is an increasingly serious social problem faced by the continuous expansion of urbanization, which has a profound impact on people's lives and travel, resulting in huge social and economic losses and serious air pollution.

[0003] In the related technology, there are traffic signal control methods used to improve traffic congestion. However, the existing traffic signal control methods can only optimize the green wave band through methods such as phase difference in large-scale road network intersection coordination. The operating conditions are harsh. For example, there are strong restrictions on the average vehicle speed of the road section, and it cannot be adjusted according to real-time conditions. If the conditions are not met, it may even cause greater delays.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The embodiments of the present application are intended to at least partially address one of the technical issues in the related art. To this end, the main purpose of the embodiments of the present application is to propose a traffic coordination signal control method and system based on reinforcement learning that can respond to changes in traffic flow in real time, improve traffic flow, reduce traffic congestion, and optimize traffic efficiency.

[0006] To achieve the above objectives, one aspect of an embodiment of the present application proposes a traffic coordination signal control method based on reinforcement learning, the method comprising:

[0007] Obtain target traffic flow data;

[0008] The target traffic flow data is modeled using a cellular transmission model to obtain target cellular data;

[0009] Constructing intersection state data of a road intersection according to the cell state data of the target cell data;

[0010] Inputting the intersection state data into the target agent value network in the target signal control agent, and outputting the target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at the road intersection;

[0011] The intersection signal light at the road intersection is controlled according to the target phase action.

[0012] In some embodiments, the method further comprises the step of constructing the target signal control agent, the step comprising:

[0013] Based on deep reinforcement learning, configure the agent environment of the initial agent;

[0014] Constructing an initial agent value network among the initial agents;

[0015] The initial agent value network is trained according to the agent environment to obtain a target agent value network, and the initial agent of the target agent value network is used as a target signal to control the agent.

[0016] In some embodiments, the agent environment includes agent actions, and configuring the agent environment of the initial agent includes:

[0017] Decomposing the initial phase action of the intersection signal light into a plurality of lane actions;

[0018] Aggregating the lane actions that satisfy the preset constraints to obtain an aggregated phase action; wherein the action states of the aggregated phase action are divided into hold and switch;

[0019] The maximum phase action value of the aggregated phase action is extracted, and the phase action corresponding to the maximum phase action value is used as the target phase action output by the target intelligent agent value network.

[0020] In some embodiments, the agent environment further includes an agent reward, and configuring the agent environment of the initial agent includes:

[0021] The agent reward is obtained according to the negative value of the total lane delay generated after the initial agent performs the target phase action on each entrance lane within the target time step.

[0022] In some embodiments, the agent reward is expressed as:

[0023]

[0024] Among them, r i t Denoted as agent reward, d j It is represented as the lane delay of one of the entrance lanes j, L i It represents all entrances corresponding to the road intersection i, t represents the time step, λ represents the hyperparameter, and f j Represented as a regularization term.

[0025] In some embodiments, the method further comprises:

[0026] The intersection signal light control strategy of each target signal control agent is optimized through a preset collaborative optimization control strategy.

[0027] In some embodiments, the preset collaborative optimization control strategy includes a parameter sharing strategy, which is used to enable each of the target signal control agents to adopt a shared value network parameter. Optimizing the intersection signal light control strategy of each of the target signal control agents using the preset collaborative optimization control strategy includes:

[0028] Training the initial value network shared parameters through the experience replay pool in the target agent value network to obtain the target value network shared parameters;

[0029] The intersection signal light control strategy of each target signal control agent is optimized according to the target value network shared parameters.

[0030] In some embodiments, the preset collaborative optimization control strategy further includes a potential game strategy, and the intersection signal light control strategy of each target signal control agent is optimized by the preset collaborative optimization control strategy, including:

[0031] Based on the potential game strategy, the sum of the agent rewards of the total target signal control agent in the target time step is used as the potential function;

[0032] performing optimal control on any of the target signal control agents to optimize the agent reward and the potential function of the target signal control agent;

[0033] According to the optimized agent reward of the target signal control agent and the optimized potential function, the global road network corresponding to any target signal control agent is optimized.

[0034] To achieve the above objectives, another aspect of the present application provides a traffic coordination signal control system based on reinforcement learning, the system comprising:

[0035] A data acquisition module is used to obtain target traffic flow data;

[0036] A cellular modeling module, configured to model the target traffic flow data using a cellular transmission model to obtain target cellular data;

[0037] A state conversion module, configured to construct intersection state data of a road intersection according to the cell state data of the target cell data;

[0038] a phase action acquisition module, configured to input the intersection state data into a target agent value network in a target signal control agent and output a target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, with one target signal control agent corresponding to one intersection signal light at the road intersection;

[0039] A signal light control module is used to control the intersection signal light at the road intersection according to the target phase action.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0041] The embodiments of the present application include at least the following beneficial effects: the present application provides a traffic cooperative signal control method and system based on reinforcement learning, which obtains target traffic flow data; models the target traffic flow data using a cellular transmission model to obtain target cellular data; constructs intersection state data of a road intersection based on the cellular state data of the target cellular data; inputs the intersection state data into the target agent value network in the target signal control agent, and outputs the target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at a road intersection; and the intersection signal light at the road intersection is controlled according to the target phase action. The embodiment of the present application obtains target cellular data by modeling real-time target traffic flow data using a cellular transmission model. Cellular-level data has richer vehicle spatial information to characterize the propagation of traffic flow, which can better understand the operation rules of vehicles on the road and provide a basis for traffic control decisions. In addition, the target cellular data is converted into intersection status data, which can reflect the real-time situation of road intersections and can be used to train and optimize target signal control agents. In addition, a target signal control agent is constructed based on deep reinforcement learning, which can learn and optimize traffic signal control strategies, improve the intelligence level of intersection signal light control, and enable it to make instant adjustments according to real-time traffic conditions, improve traffic smoothness and reduce traffic congestion. Among them, each target signal control agent is responsible for the signal control of an intersection signal light, and can adaptively adjust the phase action of the signal light according to the real-time traffic flow conditions to control the intersection signal light according to the phase action, respond to changes in traffic flow in real time, reduce congestion, and optimize traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of the steps of the traffic cooperative signal control method based on reinforcement learning provided in an embodiment of the present application;

[0043] Figure 2 This is a lane diagram before traffic flow data modeling provided in an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of a cellular grid after traffic flow data modeling provided in an embodiment of the present application;

[0045] Figure 4 is a module schematic diagram of the traffic cooperative signal control method based on reinforcement learning provided by an embodiment of the present application;

[0046] Figure 5 is a signal control module schematic diagram of the traffic cooperative signal control method based on reinforcement learning provided by an embodiment of the present application;

[0047] Figure 6 is a structural schematic diagram of the traffic cooperative signal control system based on reinforcement learning provided by an embodiment of the present application;

[0048] Figure 7 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0050] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0051] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0053] Currently, traffic congestion is an increasingly serious social problem facing the continuous expansion of urbanization, profoundly affecting public life and travel, resulting in enormous socioeconomic losses and severe air pollution. Therefore, improving congestion has considerable economic benefits and can effectively reduce emissions. In the related art, traffic signal control methods are used to improve traffic congestion. Traffic signal control optimizes signal timing schemes at intersections to respond to changes in real-time traffic demand. With high facility penetration, fast response speed, and high execution accuracy, it has become recognized as one of the best ways to alleviate traffic congestion. However, existing traffic signal control methods are not adaptable to intersections with different structures and phases. For example, control methods suitable for four-way intersections cannot be applied to three-way intersections. Furthermore, in large-scale road network intersection coordination, green wave bands can only be optimized through methods such as phase difference. These methods are subject to stringent application conditions, such as strict restrictions on average vehicle speeds on road sections, and cannot be adjusted according to real-time conditions. If these conditions are not met, they can even lead to greater delays. For example, traditional adaptive signal control methods are often optimized and controlled based on pre-established traffic flow models, and their flexibility and scalability are relatively limited. They may need to be redesigned or adjusted for new traffic scenarios or complex traffic conditions.

[0054] In view of this, the embodiments of the present application provide a traffic cooperative signal control method and system based on reinforcement learning. The scheme obtains target traffic flow data; uses a cellular transmission model to model the target traffic flow data to obtain target cellular data; constructs intersection state data of the road intersection based on the cellular state data of the target cellular data; inputs the intersection state data into the target agent value network in the target signal control agent, and outputs the target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at the road intersection; and the intersection signal light at the road intersection is controlled according to the target phase action. The embodiment of the present application obtains target cellular data by modeling real-time target traffic flow data using a cellular transmission model. Cellular-level data has richer vehicle spatial information to characterize the propagation of traffic flow, which can better understand the operation rules of vehicles on the road and provide a basis for traffic control decisions. In addition, the target cellular data is converted into intersection status data, which can reflect the real-time situation of road intersections and can be used to train and optimize target signal control agents. In addition, a target signal control agent is constructed based on deep reinforcement learning, which can learn and optimize traffic signal control strategies, improve the intelligence level of intersection signal light control, and enable it to make instant adjustments according to real-time traffic conditions, improve traffic smoothness and reduce traffic congestion. Among them, each target signal control agent is responsible for the signal control of an intersection signal light, and can adaptively adjust the phase action of the signal light according to the real-time traffic flow conditions to control the intersection signal light according to the phase action, respond to changes in traffic flow in real time, reduce congestion, and optimize traffic efficiency.

[0055] The traffic cooperative signal control method based on reinforcement learning provided in the embodiment of the present application relates to the field of traffic control technology. The traffic cooperative signal control method based on reinforcement learning provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the traffic cooperative signal control method based on reinforcement learning, etc., but is not limited to the above forms.

[0056] The present application can be used in a variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0057] See also Figure 1 , Figure 1 This is an optional step flow chart of the traffic cooperative signal control method based on reinforcement learning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0058] Step S101, obtaining target traffic flow data;

[0059] The target traffic flow data may include vehicle flow, density, and speed. The method for acquiring traffic flow data may be selected according to actual conditions, and the embodiment of the present application does not impose any limitation thereto.

[0060] In the embodiment of the present application, by acquiring target traffic flow data, a data basis is provided for subsequent traffic flow data modeling and signal control.

[0061] Step S102, modeling the target traffic flow data using a cellular transmission model to obtain target cellular data;

[0062] The cellular transport model consists of individual units (cells) in a regular grid. Each cell can be in one of a finite number of states, and the cell's state is updated according to certain rules based on the states of its neighbors. For the definition and processing of the cellular transport model, those skilled in the art can refer to the description of the relevant art, and the present embodiments will not be described in detail here.

[0063] The target cellular data is the data obtained after the target traffic flow data is modeled by the cellular transmission model. That is, the three elements of traffic flow, density, and speed can be converted from the lane level to the cellular level through the cellular transmission model.

[0064] See also Figure 2 , Figure 2 is a lane diagram before traffic flow data modeling provided in the embodiment of the present application; Figure 2 As shown, Figure 2 It shows the lane situation of real-time traffic flow data, please refer to Figure 3 , Figure 3 is a schematic diagram of a cellular grid after traffic flow data modeling provided in an embodiment of the present application; Figure 3 As shown in FIG, it is the lane situation at the cellular level obtained after modeling by the cellular transmission model, which is reflected in the form of a grid.

[0065] In the specific implementation, the cellular transmission model is used to model the target traffic flow, and the lane j is divided into each cellular unit c, that is, every distance l c To split the lane, the length of the cell c is in, is the upper speed limit of lane j. It can be understood that a vehicle can travel at most one cell unit within the decision time Δt (the time for each action selection), so the complexity of obtaining vehicle position propagation can be simplified.

[0066] Specifically, the three elements of traffic flow (flow, density and speed) are converted from the lane level to the cellular level through the cellular transmission model.

[0067] In an embodiment of the present application, a cellular transmission model is used to model the target traffic flow data, and the lanes are divided into cellular units. That is, the three elements of traffic flow, density, and speed are converted to the cellular level. The cellular-level data has more vehicle spatial information to characterize the propagation of traffic flow, refine the traffic state design, and enrich the data characteristics.

[0068] Step S103, constructing intersection state data of the road intersection according to the cell state data of the target cell data;

[0069] Among them, the cell state data is represented as the state of the cell. Specifically, the number of vehicles m in the cell c , density k c , average speed v c The state of cell c can be expressed as s c =[m c ,k c ,v c ].

[0070] In the specific implementation, the state of the agent is equivalent to the state of the road intersection controlled by the agent. For the convenience of expression, the agent and the road intersection are represented by i. When agent i controls road intersection i, the state of road intersection i is expressed as C i represents the cell set of the road intersection i, that is, the state of the road intersection is the set of all cell states in the road intersection.

[0071] In an embodiment of the present application, intersection status data of a road intersection is constructed based on the cell status data of the target cell data. The intersection status data can reflect the traffic conditions at the intersection level, which is conducive to achieving more refined traffic management and providing direct data support for the formulation and adjustment of signal control strategies.

[0072] Step S104: Input the intersection state data into the target agent value network in the target signal control agent, and output the target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at the road intersection;

[0073] In some embodiments, the step of constructing a target signal controlled intelligent agent may also be included, which includes: configuring an agent environment of the initial intelligent agent based on deep reinforcement learning; constructing an initial intelligent agent value network in the initial intelligent agent; training the initial intelligent agent value network according to the agent environment to obtain a target intelligent agent value network, and using the initial intelligent agent with the target intelligent agent value network as the target signal controlled intelligent agent.

[0074] Among them, the initial intelligent agent is the intelligent agent at the time of initial construction, and has not undergone any data training and parameter optimization; similarly, the initial intelligent agent value network is the value network at the time of initial construction, and has not undergone any data training and parameter optimization.

[0075] Among them, the target signal control agent is the agent after training and optimization; the target agent value network is the value network after training and optimization; the target phase action is the action output by the target agent value network for controlling the intersection signal light corresponding to the agent.

[0076] The agent environment consists of agent states, actions, and rewards. Specifically, Deep Reinforcement Learning (DRL) is used to build a signal control agent, with each intersection signal representing an agent. The reinforcement learning design encompasses the design of states, actions, and rewards, while also requiring the training of the agent's value network, which serves as an approximator.

[0077] Specifically, when defining the state of an agent, the state of the agent is equivalent to the state of the road intersection controlled by the agent. For the convenience of expression, the agent and the road intersection are represented by i. When agent i controls road intersection i, the state of road intersection i is expressed as C i represents the cell set of the road intersection i, that is, the state of the road intersection is the set of all cell states in the road intersection.

[0078] In some specific embodiments, configuring the agent environment of the initial agent may include: decomposing the initial phase action of the intersection signal light into several lane actions; aggregating the lane actions that meet preset constraints to obtain aggregated phase actions; wherein the action states of the aggregated phase actions are divided into maintain and switch; extracting the maximum phase action value of the aggregated phase action, and using the phase action corresponding to the maximum phase action value as the target phase action output by the target agent value network.

[0079] The preset constraint condition can be understood as a lane action without conflict.

[0080] The initial phase action is the action that has not been decomposed and aggregated. The aggregated phase action is the action obtained by decomposing the initial phase action into several lane actions and then aggregating the non-conflicting lane actions. By aggregating lane actions into phase actions, the intersection signal phase is selected, making the signal control strategy adaptable to intersections with different structures and phases, such as three-way intersections and four-way intersections. The target phase action is the action corresponding to the maximum phase action value.

[0081] Among them, the action states of the aggregation phase action are divided into maintenance and switching. Through the structural mechanism of maintenance and switching dueling, the complexity of subsequent intelligent agent value network fitting can be reduced, the learning process of the intelligent agent value network model can be accelerated, and the efficiency of training the intelligent agent value network can be improved.

[0082] In the specific implementation, the action design is to first decompose the phase action (select a certain phase) into lane actions (select certain lanes to release), and then aggregate the lane actions that do not have conflicts into phase actions. Through this constraint, the lane action value is aggregated to the phase action value. Specifically, the lane action types are divided into hold, switch, and stop. Each lane action has a corresponding lane action value, and the phase action is divided into hold and switch. Finally, the largest phase action value is selected as the target action a. i , so it can continue to maintain the previous phase or switch to the phase with the largest switching value.

[0083] Specifically, by aggregating lane actions into phase actions for intersection signal phase selection, the system can adapt to intersections of varying structures and phases, and the control strategy can be adapted to three-way and four-way intersections. Furthermore, by maintaining and switching the structural mechanism (the mechanism structure of lane actions), the complexity of subsequent agent value network fitting can be reduced, accelerating the learning process of the agent value network model and improving the efficiency of training the agent value network.

[0084] In some specific embodiments, configuring the agent environment of the initial agent may further include: obtaining an agent reward according to the negative value of the total lane delay generated after the initial agent performs the target phase action on each entrance lane within the target time step.

[0085] The expression of agent reward is:

[0086]

[0087] Among them, r i t Denoted as agent reward, d j It is represented as the lane delay of one of the entrance lanes j, L i It represents all entrances corresponding to the road intersection i, t represents the time step, λ represents the hyperparameter, which can be set to 1, f j Represented as a regularization term.

[0088] After agent i performs the action at time step t, all entrance roads L i The total delay d i The negative value of the agent is the reward r i t However, lane delays j It is not only affected by the previous action, but also contributes to the instantaneous reaction action. The regular term f is added j To correct the effects of each action, f j represents the number of outflow vehicles from lane j in time step t.

[0089] It should be noted that due to It actually represents the total number of vehicles flowing through the road network, and the total number of vehicles is a constant, so the regularization term f j It does not affect the optimality of the reward.

[0090] The design of the agent value network is as follows: the agent uses the value network to select the optimal action according to the current state, that is, the current state of the intersection is input to the value network, and the value network outputs the action corresponding to the current state. The output action is expressed as Where θ is the network parameter, represents the state of intersection i at time step t, represents the action chosen by agent i at time step t, and its obedience state is The probability distribution μ of the value network with parameter θ. To mitigate oscillations during value network training, the concept of a target network is introduced to promote convergence. Both the value network and the target network are fully connected neural networks, with the number of neurons in each layer being [8,8,1] and [8,8,1] respectively, and the activation function being the Rectified Linear Unit (ReLU).

[0091] In the process of executing state processing and output action in the agent value network, the state s of each agent will be collected. t 、Action a t , reward r t and the next state s t+1 These data are combined into experience and transmitted to the collaborative optimization module to collaboratively optimize the control strategies of each intersection agent.

[0092] It is understandable that compared with traditional adaptive signal control methods, the adaptive control method of reinforcement learning does not rely on prior knowledge of traffic flow, but learns the optimal strategy through interaction with the environment. It has higher flexibility and scalability, and can more easily cope with diverse traffic scenarios and complex traffic conditions.

[0093] In the specific implementation, a target signal control agent is constructed based on deep reinforcement learning, which can learn and optimize traffic signal control strategies, improve the intelligence level of intersection signal light control, and enable it to make instant adjustments according to real-time traffic conditions, improve traffic smoothness and reduce traffic congestion. Among them, each target signal control agent is responsible for the signal control of an intersection signal light, and can adaptively adjust the phase action of the signal light according to the real-time traffic flow conditions to control the intersection signal light according to the phase action, respond to changes in traffic flow in real time, reduce congestion, and optimize traffic efficiency.

[0094] In addition, by adopting the method of aggregating lane actions to phase actions to select the intersection signal stage, it can adapt to the problems of intersections with different structures and different phases. That is, the control strategy can adapt to intersections such as three-way intersections and four-way intersections, overcoming the problem of incompatibility of previous models at different intersections; at the same time, the mechanism structure of lane actions can accelerate the learning process of the value network model.

[0095] In some embodiments, the method may further include optimizing the intersection signal light control strategy of each target signal control agent through a preset collaborative optimization control strategy.

[0096] The preset collaborative optimization control strategies include parameter sharing strategy and potential game strategy. The parameter sharing strategy is used to realize that each target signal control agent adopts a shared value network parameter.

[0097] In some specific embodiments, optimizing the intersection signal light control strategy of each target signal control agent through a preset collaborative optimization control strategy may include: training the initial value network shared parameters through the experience replay pool in the target agent value network to obtain the target value network shared parameters; and optimizing the intersection signal light control strategy of each target signal control agent according to the target value network shared parameters.

[0098] Among them, the experience replay pool stores all agent experiences (state s t 、Action a t , reward r t , next state s t+1 ) is used to train the value network parameters θ, that is, the initial value network shared parameters, thereby enhancing the data efficiency and stability of the learning process, and then the experience of all agents is put into the experience pool to train the optimal network parameters θ * , that is, the target value network shares parameters to prevent the high complexity and high computational burden caused by the joint training of multiple value network values.

[0099] In some specific embodiments, optimizing the intersection signal light control strategy of each target signal control agent through a preset collaborative optimization control strategy may also include: based on a potential game strategy, taking the sum of the agent rewards of the total target signal control agent within the target time step as a potential function; performing optimization control on any target signal control agent to optimize the agent reward and potential function of the target signal control agent; optimizing the global road network corresponding to any target signal control agent based on the optimized agent reward and the optimized potential function of the target signal control agent.

[0100] In the specific implementation, by sharing network parameters, all agents use a shared value network parameter θ, and then train to obtain the optimal network parameter θ with generalization. * Specifically, the training of the agent value network requires an experience replay pool, which stores all agent experiences (states s t 、Action a t , reward r t , next state s t+1 ) is used to train the value network parameter θ, thereby enhancing the data efficiency and stability of the learning process, and then the experience of all agents is put into the experience pool to train the optimal network parameter θ * , to prevent the high complexity and high computational burden caused by the joint training of multiple value network values.

[0101] At the same time, the embodiment of the present application uses a distributed collaborative strategy to perform collaborative signal control, and realizes the collaborative optimization of multiple intelligent agents in the road network through potential game, and defines the sum of the rewards of all intelligent agents in a certain time step t as the potential function Can get That is, it can be understood that when the action of agent i changes from Optimize to Afterwards, for The left side of the equation represents the optimization of the reward of agent i, while the right side of the equation represents the corresponding optimization of the potential function of all multi-agents, so that the overall road network can be optimized when optimizing any intersection agent. Among them, I represents the set of all intersections, Indicates that agent i is in state Take action The reward for feedback, represents other intersections except i, represents the action set of other intersections at time step t, Indicates another different one from the i-th intersection at time step t action.

[0102] In the specific implementation, parameter sharing and potential game are used to collaboratively optimize the control strategies of the intelligent agents at each intersection. Sharing the same network parameters at all intersections in the entire road network can reduce the complexity of the algorithm. When optimizing the signal control strategy of a single intersection, the global road network control strategy can be optimized, which can reduce road network delays.

[0103] Step S105 : controlling the intersection signal light at the road intersection according to the target phase action.

[0104] In a specific implementation, after the target agent value network outputs the target phase action, the target phase action can be executed, thereby controlling the intersection signal lights at the road intersection according to the target phase action.

[0105] In an embodiment of the present application, target traffic flow data is obtained; the target traffic flow data is modeled using a cellular transmission model to obtain target cellular data; intersection state data of a road intersection is constructed based on the cellular state data of the target cellular data; the intersection state data is input into a target agent value network in a target signal control agent, and a target phase action corresponding to the intersection state data is output; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at a road intersection; and the intersection signal light at the road intersection is controlled according to the target phase action. The embodiment of the present application obtains target cellular data by modeling real-time target traffic flow data using a cellular transmission model. Cellular-level data has richer vehicle spatial information to characterize the propagation of traffic flow, which can better understand the operation rules of vehicles on the road and provide a basis for traffic control decisions. In addition, the target cellular data is converted into intersection status data, which can reflect the real-time situation of road intersections and can be used to train and optimize target signal control agents. In addition, a target signal control agent is constructed based on deep reinforcement learning, which can learn and optimize traffic signal control strategies, improve the intelligence level of intersection signal light control, and enable it to make instant adjustments according to real-time traffic conditions, improve traffic smoothness and reduce traffic congestion. Among them, each target signal control agent is responsible for the signal control of an intersection signal light, and can adaptively adjust the phase action of the signal light according to the real-time traffic flow conditions to control the intersection signal light according to the phase action, respond to changes in traffic flow in real time, reduce congestion, and optimize traffic efficiency.

[0106] Furthermore, by aggregating lane actions into phase actions for intersection signal phase selection, the system can adapt to intersections of varying structures and phases. This means the control strategy can be adapted to intersections with varying structures and phases, such as three-way and four-way intersections, overcoming the incompatibility of previous models with different intersections. Furthermore, the lane action mechanism accelerates the learning process of the value network model. Furthermore, by collaboratively optimizing the control strategies of the agents at each intersection through parameter sharing and potential game theory, all intersections in the entire road network share the same network parameters, reducing algorithmic complexity. Optimizing the signal control strategy for a single intersection can optimize the global network control strategy, thereby reducing network delays.

[0107] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0108] The system modules corresponding to the traffic cooperative signal control method based on reinforcement learning provided in the embodiment of the present application may include a data processing module, a signal control module, a parameter optimization module, and a practical application module.Figure 4 , Figure 4 This is a module diagram of the traffic cooperative signal control method based on reinforcement learning provided in the embodiment of the present application. Figure 4 As shown, the definition and processing of each system module are as follows:

[0109] Module 1: Data processing module

[0110] The target traffic flow is modeled using the cellular transmission model, and the lane j is divided into cellular units c, that is, every distance l c To split the lane, the length of the cell c is in, is the upper speed limit of lane j. It can be understood that a vehicle can travel at most one cell unit within the decision time Δt (the time for each action selection), so the complexity of obtaining vehicle position propagation can be simplified.

[0111] Specifically, the three elements of traffic flow (flow, density and speed) are converted from the lane level to the cell level through the cellular transmission model, where the number of vehicles in the cell is m. c , density k c , average speed v c The state of cell c can be expressed as s c =[m c ,k c ,v c ].

[0112] Second module: signal control module

[0113] Deep reinforcement learning is used to build a signal control agent, with each intersection signal light as an agent. The reinforcement learning design includes the design of three aspects: state, action, and reward. It also requires training the agent value network as an approximator. Figure 5 , Figure 5 This is a schematic diagram of a signal control module of a traffic cooperative signal control method based on reinforcement learning provided in an embodiment of the present application. The relationship between the components of the signal control module is as follows: Figure 5 shown.

[0114] In the signal control module, deep reinforcement learning is used to build a signal control agent ( Figure 5 The sequence number 001 in the figure represents the signal control agent, and the description of the signal control agent is referred to as the agent below. The process is as follows:

[0115] 1) Define the agent state

[0116] The state of the agent is equivalent to the state of the intersection controlled by the agent. For the convenience of expression, the agent and the intersection share i for representation, and the agent i controls the intersection i. The state of the intersection i is represented as C i The cell set of the intersection i is represented, that is, the state of the intersection is the set of all cell states in the intersection.

[0117] 2) Define the action of the agent

[0118] The action is designed to first decompose the phase action (select a certain phase) into the lane action (select a certain lane to release), and then aggregate the lane action without conflict into the phase action. Through this constraint, the lane action value is aggregated to the phase action value. Specifically, the lane action type is divided into keep, switch, and stop. Each lane action has a corresponding lane action value. The phase action is divided into keep and switch. Finally, the maximum phase action value is selected as the target action a i Therefore, the previous phase can be continued or switched to the phase with the maximum switching value.

[0119] In specific implementation, the way of aggregating from lane action to phase action for selecting intersection signal phase can adapt to the problems of intersections with different structures and different phases. The control strategy can adapt to three-way intersections, four-way intersections and other intersections. At the same time, through the duel structure mechanism of keep and switch (the mechanism structure of lane action), the complexity of subsequent agent value network fitting can be reduced, the learning process of agent value network model can be accelerated, and the efficiency of training agent value network can be improved.

[0120] 3) Define the reward of the agent

[0121] The negative value of the total delay d i of all approaches L i of the intersection after the agent i performs the action at time step t is taken as the reward r i t However, the lane delay d j is not only affected by the previous action, but also contributes to the reaction moment. A regularization term f j is added to correct the influence of each action. f j represents the number of outflow vehicles of lane j within time step t. The reward of agent i at decision step t is represented as:

[0122]

[0123] Since the actual value represents the total number of vehicles flowing through the road network, and the total number of vehicles is a constant, the regularization term f j does not affect the optimality of the reward.

[0124] 4) Agent Value Network

[0125] The agent uses the value network to select the optimal action according to the current state. That is, the current state of the intersection is input to the value network, and the value network outputs the action corresponding to the current state. The output action is expressed as Where θ is the network parameter, represents the state of intersection i at time step t, represents the action chosen by agent i at time step t, and its obedience state is The probability distribution μ of the value network with parameter θ. To mitigate network training oscillations in the value network, the concept of a target network is introduced to promote network convergence. Both the value network and the target network are fully connected neural networks, with the number of neurons in each layer being [8, 8, 1] and [8, 8, 1] respectively, and the activation function is a linear rectifier function.

[0126] The states s of each agent collected in the signal control module t 、Action a t , reward r t and the next state s t+1 , which will together constitute the experience and be transmitted to the collaborative optimization module.

[0127] Module 3: Collaborative Optimization Module

[0128] The collaborative optimization module mainly optimizes the control strategies of each intersection agent through parameter sharing and potential game.

[0129] First, by sharing network parameters, all agents use a shared value network parameter θ, and then train to obtain the optimal network parameter θ with generalization * Specifically, the training of the agent value network requires an experience replay pool, which stores all agent experiences (states s t 、Action a t , reward r t , next state s t+1 ) is used to train the value network parameter θ, thereby enhancing the data efficiency and stability of the learning process, and then the experience of all agents is put into the experience pool to train the optimal network parameter θ * , to prevent the high complexity and high computational burden caused by the joint training of multiple value network values.

[0130] At the same time, the embodiment of the present application uses a distributed collaborative strategy to perform collaborative signal control, and realizes the collaborative optimization of multiple intelligent agents in the road network through potential game, and defines the sum of the rewards of all intelligent agents in a certain time step t as The reward of is optimized, and the potential function of all multi-agents on the right side of the equation is also optimized accordingly, so that the overall road network can be optimized when optimizing any intersection agent. Where I represents the set of all intersections, Indicates that agent i is in state Take action The reward for feedback, represents other intersections except i, represents the action set of other intersections at time step t, Indicates another different one from the i-th intersection at time step t action.

[0131] like Figure 5 As shown in the figure, ① represents the process of obtaining cellular data after the traffic flow data is processed by the cellular transmission model, and ② represents the process of inputting the current intersection state data corresponding to the cellular data into the agent value network in the signal control agent ( Figure 5 ), ③ represents the process of the agent value network outputting the phase action corresponding to the current intersection state data, ④ represents the process of executing the phase action, ⑤ represents the process of taking the negative value of the total delay of all entrances of the intersection after the agent executes the phase action at the time step as the reward of the agent, and ⑥ represents the process of performing parameter optimization to optimize the agent value network. Specifically, first, traffic flow data is obtained, and then the traffic flow data is modeled through the cellular transmission model to obtain cellular data. The current intersection state data corresponding to the cellular data is then input into the agent value network in the signal control agent. The phase action corresponding to the current intersection state data is output through the agent value network, and then the phase action is executed. Finally, the negative value of the total delay of all entrances of the intersection after the agent executes the phase action within the time step is taken as the reward of the agent, and parameter optimization is performed based on the reward to optimize the agent value network.

[0132] Module 4: Deploy application modules

[0133] Model integration and deployment: Integrate the trained agent model into the city's traffic management system to ensure that the model can receive real-time data from traffic monitoring equipment, such as vehicle flow, density, and speed, and process it into cellular data in real time. This cellular data is then input into the value network and ultimately outputs the phase to control intersections, thereby responding to changes in traffic flow in real time, reducing congestion, and optimizing traffic efficiency.

[0134] It should be pointed out that this embodiment only provides a brief schematic illustration of the general process of the traffic cooperative signal control method based on reinforcement learning. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0135] The embodiment of the present application obtains target traffic flow data; uses a cellular transmission model to model the target traffic flow data to obtain target cellular data; constructs intersection state data of a road intersection based on the cellular state data of the target cellular data; inputs the intersection state data into a target agent value network in a target signal control agent, and outputs a target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at a road intersection; and the intersection signal light at the road intersection is controlled according to the target phase action. The embodiment of the present application obtains target cellular data by modeling real-time target traffic flow data using a cellular transmission model. Cellular-level data has richer vehicle spatial information to characterize the propagation of traffic flow, which can better understand the operation rules of vehicles on the road and provide a basis for traffic control decisions. In addition, the target cellular data is converted into intersection status data, which can reflect the real-time situation of road intersections and can be used to train and optimize target signal control agents. In addition, a target signal control agent is constructed based on deep reinforcement learning, which can learn and optimize traffic signal control strategies, improve the intelligence level of intersection signal light control, and enable it to make instant adjustments according to real-time traffic conditions, improve traffic smoothness and reduce traffic congestion. Among them, each target signal control agent is responsible for the signal control of an intersection signal light, and can adaptively adjust the phase action of the signal light according to the real-time traffic flow conditions to control the intersection signal light according to the phase action, respond to changes in traffic flow in real time, reduce congestion, and optimize traffic efficiency.

[0136] Furthermore, by aggregating lane actions into phase actions for intersection signal phase selection, the system can adapt to intersections of varying structures and phases. This means the control strategy can be adapted to intersections with varying structures and phases, such as three-way and four-way intersections, overcoming the incompatibility of previous models with different intersections. Furthermore, the lane action mechanism accelerates the learning process of the value network model. Furthermore, by collaboratively optimizing the control strategies of the agents at each intersection through parameter sharing and potential game theory, all intersections in the entire road network share the same network parameters, reducing algorithmic complexity. Optimizing the signal control strategy for a single intersection can optimize the global network control strategy, thereby reducing network delays.

[0137] Compared with the related art, the embodiments of the present application have the following advantages:

[0138] 1. The cellular transmission model is used to model traffic flow, refine the traffic state design, and enrich the data characteristics.

[0139] The embodiment of the application adopts a cell transmission model to model traffic flow, divides a lane into cell units, and converts three elements of traffic flow, i.e., flow, density and speed, to the cell level. The lane-level data of related technologies has a coarser granularity and cannot represent the propagation process of traffic flow in a lane or the spatial position of a vehicle.

[0140] Therefore, compared with the lane-level data of a coarser granularity, the embodiment of the application adopts the cell transmission model to model traffic flow, and the cell-level data has more vehicle spatial information to represent the propagation of traffic flow.

[0141] 2. Action selection is adopted from lane action aggregation to phase action, which is suitable for intersections of different structures and improves the generalization of the model.

[0142] The embodiment of the application adopts the mode of aggregating lane actions to phase actions to select the signal phase of an intersection, which can adapt to intersections of different structures and different phases, i.e., the control strategy can adapt to three-way intersections, four-way intersections and other intersections, overcoming the problem of inadaptation of previous models to different intersections. Meanwhile, the mechanism structure of lane actions can accelerate the learning process of the value network model.

[0143] 3. Multi-agent collaborative optimization is adopted to realize collaborative optimization of road network intersections and improve road network efficiency.

[0144] The embodiment of the application optimizes the control strategy of each intersection agent through parameter sharing and potential game, and all intersections in the whole road network share the same network parameter, which can reduce the algorithm complexity, optimize the global road network control strategy when optimizing the single intersection signal control strategy, and reduce the road network delay.

[0145] Please refer to Figure 6 The embodiment of the application also provides a traffic collaborative signal control system 600 based on reinforcement learning, which can implement the traffic collaborative signal control method based on reinforcement learning, and the system comprises:

[0146] A data acquisition module 601 is configured to acquire target traffic flow data.

[0147] A cell modeling module 602 is configured to model the target traffic flow data by using a cell transmission model to obtain target cell data.

[0148] A state conversion module 603 is configured to construct intersection state data of a road intersection according to cell state data of the target cell data.

[0149] Phase action acquisition module 604 is configured to input the intersection state data into a target agent value network in a target signal control agent and output a target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at the road intersection;

[0150] The signal light control module 605 is configured to control the intersection signal light at the road intersection according to the target phase action.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned reinforcement learning-based traffic coordination signal control method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0153] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0154] See also Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0155] The processor 701 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0156] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 702 and are called by the processor 701 to execute the traffic cooperative signal control method based on reinforcement learning in the embodiments of this application.

[0157] Input / output interface 703, used to implement information input and output;

[0158] Communication interface 704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0159] Bus 705 , which transmits information between various components of the device (e.g., processor 701 , memory 702 , input / output interface 703 , and communication interface 704 );

[0160] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0161] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned traffic cooperative signal control method based on reinforcement learning.

[0162] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The embodiment of the application provides a traffic cooperative signal control method based on reinforcement learning, a traffic cooperative signal control device based on reinforcement learning, an electronic device and a storage medium. The traffic cooperative signal control method based on reinforcement learning comprises the following steps: obtaining target traffic flow data; modeling the target traffic flow data by using a cell transmission model to obtain target cell data; constructing intersection state data of a road intersection according to cell state data of the target cell data; inputting the intersection state data into a target intelligent agent value network in a target signal control intelligent agent, and outputting a target phase action corresponding to the intersection state data; the target signal control intelligent agent is constructed based on deep reinforcement learning, one target signal control intelligent agent corresponds to one intersection signal lamp of the road intersection; and the intersection signal lamp of the road intersection is controlled according to the target phase action. The embodiment of the application can better understand the running rule of vehicles on the road by modeling real-time target traffic flow data by using a cell transmission model to obtain target cell data, and can provide a basis for traffic control decision by using cell-level data to represent the propagation of traffic flow with richer vehicle space information. The real-time situation of the road intersection can be reflected by converting the target cell data into intersection state data, and the target signal control intelligent agent can be trained and optimized. In addition, the target signal control intelligent agent is constructed based on deep reinforcement learning, which can learn and optimize the traffic signal control strategy, improve the intelligent level of intersection signal lamp control, enable the intersection signal lamp to make real-time adjustment according to real-time traffic conditions, improve traffic flow smoothness and reduce traffic congestion. Each target signal control intelligent agent is responsible for the signal control of one intersection signal lamp, can adaptively adjust the phase action of the signal lamp according to real-time traffic flow, and controls the intersection signal lamp according to the phase action, responds to the change of traffic flow in real time, reduces congestion and optimizes traffic efficiency.

[0165] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0166] Those skilled in the art can understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than the drawings, or combine certain steps or different steps.

[0167] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the application.

[0168] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0169] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0170] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0175] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A traffic cooperative signal control method based on reinforcement learning, characterized in that: The method comprises: Obtain target traffic flow data; The target traffic flow data is modeled using a cellular transmission model to obtain target cellular data; Constructing intersection state data of a road intersection according to the cell state data of the target cell data; Inputting the intersection state data into the target agent value network in the target signal control agent, and outputting the target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, and one target signal control agent corresponds to one intersection signal light at the road intersection; controlling the intersection signal light at the road intersection according to the target phase action; The method further includes: optimizing the intersection signal light control strategy of each target signal control agent through a preset collaborative optimization control strategy; the preset collaborative optimization control strategy includes a parameter sharing strategy and a potential game strategy, the parameter sharing strategy is used to enable each target signal control agent to adopt a shared value network parameter; The method of optimizing the intersection signal light control strategy of each target signal control agent by using a preset collaborative optimization control strategy includes: The initial value network shared parameters are trained by the experience replay pool in the target agent value network to obtain the target value network shared parameters; the experience replay pool stores the agent experience corresponding to each target signal control agent; Optimizing the intersection signal light control strategy of each target signal control agent according to the target value network shared parameters; Based on the potential game strategy, the sum of the agent rewards of the total target signal control agent in the target time step is used as the potential function; performing optimal control on any of the target signal control agents to optimize the agent reward and the potential function of the target signal control agent; According to the optimized agent reward of the target signal control agent and the optimized potential function, the global road network corresponding to any target signal control agent is optimized.

2. The method according to claim 1, characterized in that The method further comprises the step of constructing the target signal control agent, which step comprises: Based on deep reinforcement learning, configure the agent environment of the initial agent; Constructing an initial agent value network among the initial agents; The initial agent value network is trained according to the agent environment to obtain a target agent value network, and the initial agent of the target agent value network is used as a target signal to control the agent.

3. The method according to claim 2, characterized in that The agent environment includes agent actions, and the agent environment for configuring the initial agent includes: Decomposing the initial phase action of the intersection signal light into a plurality of lane actions; Aggregating the lane actions that satisfy the preset constraints to obtain an aggregated phase action; wherein the action states of the aggregated phase action are divided into hold and switch; The maximum phase action value of the aggregated phase action is extracted, and the phase action corresponding to the maximum phase action value is used as the target phase action output by the target intelligent agent value network.

4. The method according to claim 2, characterized in that The agent environment further includes an agent reward, and the agent environment for configuring the initial agent includes: The agent reward is obtained according to the negative value of the total lane delay generated after the initial agent performs the target phase action on each entrance lane within the target time step.

5. The method according to claim 4, characterized in that The expression of the agent reward is: in, Denoted as agent reward, d j It is represented as the lane delay of one of the entrance lanes j, L i It represents all entrances corresponding to the road intersection i, t represents the time step, λ represents the hyperparameter, and f j Represented as a regularization term.

6. Traffic cooperative signal control system based on reinforcement learning, characterized by: The system comprises: A data acquisition module is used to obtain target traffic flow data; A cellular modeling module, configured to model the target traffic flow data using a cellular transmission model to obtain target cellular data; A state conversion module, configured to construct intersection state data of a road intersection according to the cell state data of the target cell data; a phase action acquisition module, configured to input the intersection state data into a target agent value network in a target signal control agent and output a target phase action corresponding to the intersection state data; the target signal control agent is constructed based on deep reinforcement learning, with one target signal control agent corresponding to one intersection signal light at the road intersection; a signal light control module, configured to control the intersection signal light at the road intersection according to the target phase action; The system is further configured to optimize the intersection signal light control strategy of each target signal control agent through a preset collaborative optimization control strategy; the preset collaborative optimization control strategy includes a parameter sharing strategy and a potential game strategy, wherein the parameter sharing strategy is configured to enable each target signal control agent to adopt a shared value network parameter; The method of optimizing the intersection signal light control strategy of each target signal control agent by using a preset collaborative optimization control strategy includes: The initial value network shared parameters are trained by the experience replay pool in the target agent value network to obtain the target value network shared parameters; the experience replay pool stores the agent experience corresponding to each target signal control agent; Optimizing the intersection signal light control strategy of each target signal control agent according to the target value network shared parameters; Based on the potential game strategy, the sum of the agent rewards of the total target signal control agent in the target time step is used as the potential function; performing optimal control on any of the target signal control agents to optimize the agent reward and the potential function of the target signal control agent; According to the optimized agent reward of the target signal control agent and the optimized potential function, the global road network corresponding to any target signal control agent is optimized.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

Images