Signal lamp intelligent control method based on deep reinforcement learning
Through the intelligent control method of signal lights based on deep reinforcement learning, the flexible action-evaluation algorithm and SUMO simulation software are used to dynamically adjust the signal light phase, solving the problem of lack of flexibility in signal light regulation in the existing technology, and improving road traffic capacity and efficiency.
Patent Information
- Application Number
- CN202411156199.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing intersection signal light control methods lack flexibility and timeliness, and cannot effectively adjust the signal light phase according to the density and distribution status of road vehicles, resulting in insufficient road traffic capacity.
Using the intelligent control method of signal lights based on deep reinforcement learning, by building a simulation platform, designing a flexible action-evaluation algorithm (SAC) model, defining state space, action space and reward functions, and using SUMO simulation software and python to achieve real-time Signal light control, train the model to maximize the cumulative reward function, and optimize the signal light strategy.
It realizes dynamic adjustment of the signal light phase according to the traffic density, balances the waiting time of vehicles in all directions, and improves the traffic efficiency of intersections.
Smart Images

Figure CN119091651A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a signal light intelligent control method based on deep reinforcement learning. Background Art
[0002] With the increase in the number of cars in cities, the road network is under increasing load, the road capacity does not meet current needs, and traffic congestion and other problems are becoming more and more prominent in the urban road network system. As the bottleneck of the traffic network, the management strategy of the intersection has the greatest impact on the comprehensive traffic capacity of the road network. The existing intersection signal control methods still have room for improvement. How to dynamically adjust the phase strategy according to the density and distribution of road vehicles to improve road capacity is an issue that needs to be urgently addressed.
[0003] Traditional intelligent traffic light control methods often fail to fully utilize the multi-dimensional traffic data collected by existing sensors, cameras and other networked devices. Most of them set the phase of traffic lights based on the experience and judgment of experts. This strategy often lacks flexibility and timeliness.
[0004] In order to improve the control efficiency of traffic lights at intersections, the present invention proposes a traffic light intelligent control method based on deep reinforcement learning. Summary of the invention
[0005] In order to make up for the defects of the prior art, the present invention provides a simple and efficient traffic light intelligent control method based on deep reinforcement learning.
[0006] The present invention is achieved through the following technical solutions:
[0007] A traffic light intelligent control method based on deep reinforcement learning, comprising the following steps:
[0008] Step S1, building a simulation platform;
[0009] A simulation platform was built based on SUMO (Simulation of Urban MObility) simulation software and python. A road network was built in SUMO simulation software. The road network strategy included intersection strategy, signal light strategy and vehicle input strategy.
[0010] Step S2, designing a control strategy model;
[0011] Based on the Soft Actor-Critic (SAC) algorithm, a control strategy model for on-ramp vehicles is designed, and the state space S, action space A and reward function r in the model are defined respectively.
[0012] Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows:
[0013] r=μ1*r length +μ2r speed +μ3r waitTime
[0014] Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1, which indicates the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection;
[0015] Step S3, defining the algorithm flow;
[0016] Initialize all parameters before model training, update the environment state data according to the actions performed by the agent, generate rewards, and store the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, start the learning process and update the target network to ensure the stability of the training;
[0017] In step S3, the algorithm flow is as follows:
[0018] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D;
[0019] Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ;
[0020] Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins;
[0021] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
[0022] Step S4: model training and verification;
[0023] During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent continuously exploring in a simulated environment to improve its strategy for controlling traffic lights, and the effectiveness of the model is verified through vehicle simulation technology.
[0024] In step S4, the model training process is as follows:
[0025] Step S4.1, initializing environmental data by setting vehicle input rules;
[0026] Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state;
[0027] Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network;
[0028] Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration;
[0029] Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
[0030] In step S4, the model verification process is different from the model training process only in that the actor network and the policy network are no longer updated during the verification process.
[0031] Step S5, model application;
[0032] In the simulation platform, the intelligent agent calls the TraCI interface of the SUMO simulation software through Python to obtain road vehicle information in real time, and uses the trained and verified model to achieve real-time control of traffic lights.
[0033] A traffic light intelligent control system based on deep reinforcement learning, comprising:
[0034] The simulation platform building module is responsible for building a simulation platform based on SUMO (Simulation of Urban MObility) simulation software and python, and building a road network in the SUMO simulation software. The road network strategy includes intersection strategy, signal light strategy, and input vehicle strategy. In the simulation platform, the intelligent agent uses python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time and realize real-time control of the signal light.
[0035] The control strategy model design module is responsible for designing the control strategy model of the on-ramp vehicle based on the Soft Actor-Critic (SAC) algorithm, and defines the state space S, action space A and reward function r in the model respectively;
[0036] Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows:
[0037] r=μ1*r length +μ2r speed +μ3r waitTime
[0038] Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1, which indicates the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection;
[0039] The algorithm process definition module is responsible for initializing all parameters before model training, updating the environment state data according to the actions performed by the agent, generating rewards, and storing the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, the learning process begins and the target network is updated to ensure the stability of the training;
[0040] The algorithm flow that the algorithm flow definition module is responsible for executing is as follows:
[0041] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D;
[0042] Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ;
[0043] Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins;
[0044] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
[0045] The model training and verification module is responsible for training and verifying the model. During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent's continuous exploration in the simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified through vehicle simulation technology.
[0046] In the model training and verification module, the model training process is as follows:
[0047] Step S4.1, initializing environmental data by setting vehicle input rules;
[0048] Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state;
[0049] Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network;
[0050] Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration;
[0051] Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
[0052] In the model training and verification module, the difference between the model verification process and the model training process is only that the actor network and the policy network are no longer updated during the verification process.
[0053] The model application module is responsible for obtaining road vehicle information in real time by calling the TraCI interface of the SUMO simulation software through Python in the simulation platform, and realizing real-time control of traffic lights by using the trained and verified model.
[0054] A traffic light intelligent control device based on deep reinforcement learning, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.
[0055] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.
[0056] The beneficial effect of the present invention is that the intelligent control method of traffic lights based on deep reinforcement learning can not only control the phase of traffic lights, but also dynamically adjust the phase of traffic lights according to the traffic density of each road, thereby balancing the waiting time of vehicles in each direction and improving the traffic efficiency of the intersection. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0058] Attached Figure 1 Schematic diagram of the intelligent control method of traffic lights based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0059] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0060] The Soft Actor-Critic (SAC) algorithm is a reinforcement learning method based on deep learning. It combines the maximum entropy reinforcement learning concept and the actor-critic framework. It is an off-policy and deep reinforcement learning method. Its advantage lies in its ability to find effective strategies in continuous action space and encourage strategies to output high entropy distributions through the maximum entropy principle, thereby improving exploration ability and stability. At the same time, the SAC algorithm also provides a solution to the overfitting problem in deep reinforcement learning - soft update. By constructing a model-based actor network and two value-function-based critic networks, the SAC algorithm can make high-quality decisions.
[0061] The traffic light intelligent control method based on deep reinforcement learning includes the following steps:
[0062] Step S1, building a simulation platform;
[0063] A simulation platform is built based on SUMO (Simulation of Urban MObility) simulation software and python. A road network is built in SUMO simulation software. The road network strategy includes intersection strategy, signal light strategy and input vehicle strategy. In the simulation platform, the intelligent agent uses python to call the TraCI interface of SUMO simulation software to obtain road vehicle information in real time and realize real-time control of signal lights.
[0064] SUMO (Simulation of Urban MObility) is an open source, microscopic, multimodal traffic simulation software, which has a wide range of applications in urban traffic planning, road network design, traffic management, etc. SUMO simulation software provides researchers and engineers with a powerful simulation tool with its powerful functions and flexibility.
[0065] Step S2, designing a control strategy model;
[0066] Based on the Soft Actor-Critic (SAC) algorithm, a control strategy model for on-ramp vehicles is designed, and the state space S, action space A and reward function r in the model are defined respectively.
[0067] Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows:
[0068] r=μ1*r length+μ2r speed +μ3r waitTime
[0069] Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1, which indicates the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection;
[0070] Step S3, defining the algorithm flow;
[0071] Initialize all parameters before model training, update the environment state data according to the actions performed by the agent, generate rewards, and store the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, start the learning process and update the target network to ensure the stability of the training;
[0072] In step S3, the algorithm flow is as follows:
[0073] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D;
[0074] Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ;
[0075] Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins;
[0076] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
[0077] In step S3, the specific implementation process of the algorithm flow is as follows:
[0078] Input: maximum time step M, initialization strategy network parameters Q-value function parameters and V-value function parameter ψ, experience replay pool D;
[0079] Output: trained policy network parameters
[0080] Initialize target parameter network parameters: ψ targ ←ψ;
[0081] for m←1to Mdo:
[0082] Observe the current state t+1 , and then select action a according to the strategy t ~π(.|s t );
[0083] Perform action a in the environment t And update the state s through the dynamic formula t+1 ;
[0084] Observe the next set of states s t+1 , calculate the reward r t ;
[0085] (S t ,a t ,S t+1 ,r) stored in the experience pool D;
[0086] If the state s t+1 If it is in the terminal state, reset the environment;
[0087] If there is enough experience in the experience pool then:
[0088] for n←1 to N do:
[0089] Randomly sample a small batch of experiences B from D = [(s t ,a t ,s t+1 ,r)];
[0090] Update Q value function:
[0091]
[0092] Update V value function:
[0093] φ←φ-λ π ▽ φ J π (φ)
[0094] Update policy parameters:
[0095] α←α-λ▽ αJ(α)
[0096] Update target value network parameters:
[0097]
[0098] Returns policy network parameters
[0099] Step S4: model training and verification;
[0100] During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent continuously exploring in a simulated environment to improve its strategy for controlling traffic lights, and the effectiveness of the model is verified through vehicle simulation technology.
[0101] In step S4, the model training process is as follows:
[0102] Step S4.1, initializing environmental data by setting vehicle input rules;
[0103] Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state;
[0104] Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network;
[0105] Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration;
[0106] Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
[0107] In step S4, the model verification process is different from the model training process only in that the actor network and the policy network are no longer updated during the verification process.
[0108] Step S5, model application;
[0109] In the simulation platform, the intelligent agent calls the TraCI interface of the SUMO simulation software through Python to obtain road vehicle information in real time, and uses the trained and verified model to achieve real-time control of traffic lights.
[0110] The traffic light intelligent control system based on deep reinforcement learning includes:
[0111] The simulation platform building module is responsible for building a simulation platform based on SUMO (Simulation of Urban MObility) simulation software and python, and building a road network in the SUMO simulation software. The road network strategy includes intersection strategy, signal light strategy, and input vehicle strategy. In the simulation platform, the intelligent agent uses python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time and realize real-time control of the signal light.
[0112] The control strategy model design module is responsible for designing the control strategy model of the on-ramp vehicle based on the Soft Actor-Critic (SAC) algorithm, and defines the state space S, action space A and reward function r in the model respectively;
[0113] Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows:
[0114] r=μ1*r length +μ2r speed +μ3r waitTime
[0115] Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1, which indicates the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection;
[0116] The algorithm process definition module is responsible for initializing all parameters before model training, updating the environment state data according to the actions performed by the agent, generating rewards, and storing the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, the learning process begins and the target network is updated to ensure the stability of the training;
[0117] The algorithm flow that the algorithm flow definition module is responsible for executing is as follows:
[0118] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D;
[0119] Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ;
[0120] Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins;
[0121] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
[0122] The model training and verification module is responsible for training and verifying the model. During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent's continuous exploration in the simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified through vehicle simulation technology.
[0123] In the model training and verification module, the model training process is as follows:
[0124] Step S4.1, initializing environmental data by setting vehicle input rules;
[0125] Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state;
[0126] Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network;
[0127] Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration;
[0128] Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
[0129] In the model training and verification module, the difference between the model verification process and the model training process is only that the actor network and the policy network are no longer updated during the verification process.
[0130] The model application module is responsible for obtaining road vehicle information in real time by calling the TraCI interface of the SUMO simulation software through Python in the simulation platform, and realizing real-time control of traffic lights by using the trained and verified model.
[0131] The traffic light intelligent control device based on deep reinforcement learning includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.
[0132] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.
[0133] Compared with the existing technology, the traffic light intelligent control method based on deep reinforcement learning has the following characteristics:
[0134] First, with the help of the neural network's perception ability of high-dimensional data, useful information can be learned from multi-dimensional data, and the traffic conditions of each lane at the intersection can be dynamically captured, thereby improving the model's efficiency in utilizing data.
[0135] Second, compared with traditional autonomous driving strategies, it has the ability to explore the action space, realizes continuous control of the phase of traffic lights, and improves the vehicle's automated decision-making ability and congestion prediction and management capabilities.
[0136] Third, the training process of the neural network is based on the SUMO simulation software, which can simulate more traffic scenarios and train the neural network from multiple scenarios, so that the neural network can explore more decision-making spaces to learn the best control strategy. Through continuous iterative training, the performance of the finally trained neural network is better than the existing traffic light control method.
[0137] Fourth, the state space includes multi-dimensional vehicle and road information, which is suitable for a variety of complex traffic scenarios. The reward function is comprehensively designed from multiple angles, including queue length, waiting time and average speed, which takes into account the queue length and vehicle waiting time in each direction while ensuring the traffic efficiency of the entire intersection.
[0138] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.
Claims
1. A traffic light intelligent control method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1, building a simulation platform; Build a simulation platform based on SUMO simulation software and python, and build a road network in SUMO simulation software. The road network strategy includes intersection strategy, signal light strategy and input vehicle strategy; Step S2, designing a control strategy model; Design a control strategy model for on-ramp vehicles based on the flexible action-criteria algorithm SAC, and define the state space S, action space A and reward function r in the model. Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows: r=μ1*r length +μ2r speed +μ3r waitTime Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection; Step S3, defining the algorithm flow; Initialize all parameters before model training, update the environment state data according to the actions performed by the agent, generate rewards, and store the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, start the learning process and update the target network to ensure the stability of the training; Step S4: model training and verification; During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent's continuous exploration in the simulation environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified by vehicle simulation technology. Step S5, model application; In the simulation platform, the intelligent agent calls the TraCI interface of the SUMO simulation software through Python to obtain road vehicle information in real time, and uses the trained and verified model to achieve real-time control of traffic lights.
2. The signal light intelligent control method based on deep reinforcement learning according to claim 1 is characterized in that: In step S3, the algorithm flow is as follows: Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D; Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ; Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins; Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
3. The signal light intelligent control method based on deep reinforcement learning according to claim 1 is characterized in that: In step S4, the model training process is as follows: Step S4.1, initializing environmental data by setting vehicle input rules; Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state; Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network; Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration; Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
4. The signal light intelligent control method based on deep reinforcement learning according to claim 3 is characterized in that: In step S4, the model verification process is different from the model training process only in that the actor network and the policy network are no longer updated during the verification process.
5. A traffic light intelligent control system based on deep reinforcement learning, characterized in that: include: The simulation platform building module is responsible for building a simulation platform based on SUMO simulation software and Python, and building a road network in the SUMO simulation software. The road network strategy includes intersection strategy, signal light strategy, and input vehicle strategy. The control strategy model design module is responsible for designing the control strategy model of the on-ramp vehicle based on the flexible action-criteria algorithm SAC, and defining the state space S, action space A and reward function r in the model respectively; Among them, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the value obtained by the weighted sum of the length of the vehicle queue, the average speed of the vehicle, and the waiting time of the vehicle, as follows: r=μ1*r length +μ2r speed +μ3r waitTime Among them, μ1, μ2 and μ3 are the weight parameters of the vehicle queue length, the average vehicle speed and the vehicle waiting time, and the sum of the weights is 1, which indicates the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection; The algorithm process definition module is responsible for initializing all parameters before model training, updating the environment state data according to the actions performed by the agent, generating rewards, and storing the experience generated in each step of the agent's actions into the experience replay pool; when the experience storage reaches the custom threshold, the learning process begins and the target network is updated to ensure the stability of the training; The model training and verification module is responsible for training and verifying the model. During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent's continuous exploration in the simulation environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified through vehicle simulation technology. The model application module is responsible for obtaining real-time road vehicle information by calling the TraCI interface of the SUMO simulation software through Python in the simulation platform, and realizing real-time control of traffic lights by using the trained and verified model.
6. The signal light intelligent control system based on deep reinforcement learning according to claim 5 is characterized in that: The algorithm flow that the algorithm flow definition module is responsible for executing is as follows: Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization strategy network parameters. Q-value function parameters V-value function parameter γ and experience replay pool D; Step S3.2: The agent selects an action based on the environment state and the policy network, and the agent performs action a t Update the environment to the new state S t+1 , and generates a reward r t ; Step S3.3: The experience (S t ,a t ,S t+1 ,r) stored in the experience replay pool D. When the experience in the experience replay pool D reaches the custom threshold, the learning process begins; Step S3.4: During the learning process, random sampling is performed in the experience replay pool D, and the Q value function, V value function, strategy parameters and target value network parameters are updated through the algorithm's loss function, and the target network is updated to ensure the stability of the training.
7. The signal light intelligent control system based on deep reinforcement learning according to claim 5 is characterized in that: In the model training and verification module, the model training process is as follows: Step S4.1, initializing environmental data by setting vehicle input rules; Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state; Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network; Step S4.4: During the training process, automatically adjust the target policy entropy to balance the exploration and exploitation strategies; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration; Step S4.5: Repeat steps S4.2 to S4.4 until the number of training rounds predetermined by the user is reached.
8. The signal light intelligent control system based on deep reinforcement learning according to claim 7 is characterized in that: In the model training and verification module, the difference between the model verification process and the model training process is only that the actor network and the policy network are no longer updated during the verification process.
9. A traffic light intelligent control device based on deep reinforcement learning, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 4 when executing the computer program.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Urban traffic jam scheduling method based on reinforcement learning
CN114550456A
Large-scale traffic signal control method and device based on deep reinforcement learning
CN115331460A
Multi-intersection traffic signal control method based on deep reinforcement learning
CN117409593A
Traffic adaptive control method based on multi-agent reinforcement learning
CN118155429A
Apparatus and method for controlling traffic signals of traffic lights in sub-area by using reinforcement learning model
US20240013654A1
Cited By
Large surface source black body multi-channel balanced heating method and device and medium
CN119597064A
Intelligent cooperative control method, system and equipment for highway ramps and medium
CN121171044A
