A Smart Control Method for Traffic Lights Based on Deep Reinforcement Learning

Abstract

This invention specifically relates to a traffic light intelligent control method based on deep reinforcement learning. This method utilizes SUMO simulation software and Python to build a simulation platform, constructing a road network within the SUMO simulation software; designs a control strategy model for vehicles entering ramps based on the Flexible Action-Evaluation Algorithm (SAC); defines the algorithm flow and updates the target network to ensure training stability; trains the model and verifies its effectiveness using vehicle simulation technology; finally, it acquires real-time road vehicle information and uses the trained and verified model to achieve real-time control of the traffic lights. This deep reinforcement learning-based intelligent traffic light control method can not only control the traffic light phase but also dynamically adjust the traffic light phase according to the traffic density of each road, thereby balancing the waiting time of vehicles in different directions and improving the traffic efficiency of intersections.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a method for intelligent control of traffic lights based on deep reinforcement learning. Background Technology

[0002] With the increasing number of cars in cities, the road network is under greater strain, and road capacity is insufficient to meet current demands, leading to increasingly prominent traffic congestion problems. Intersections, as bottlenecks in the traffic network, have the most profound impact on the overall capacity of the network due to their management strategies. Existing traffic light control methods at intersections still have room for improvement. How to dynamically adjust phase strategies based on vehicle density and distribution to enhance road capacity is an urgent problem that needs to be solved.

[0003] Traditional intelligent traffic light control methods often fail to fully utilize the multi-dimensional traffic data collected by existing networked devices such as sensors and cameras. They mostly rely on expert judgment to set traffic light phases, which often lacks flexibility and timeliness.

[0004] To improve the efficiency of traffic light control at intersections, this invention proposes a smart traffic light control method based on deep reinforcement learning. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient intelligent control method for traffic lights based on deep reinforcement learning.

[0006] This invention is achieved through the following technical solution:

[0007] A method for intelligent control of traffic lights based on deep reinforcement learning includes the following steps:

[0008] Step S1: Build a simulation platform;

[0009] A simulation platform was built based on SUMO (Simulation of Urban MObility) simulation software and Python. A road network was constructed in the SUMO simulation software. The road network strategy includes intersection strategy, traffic light strategy and input vehicle strategy.

[0010] Step S2: Design the control strategy model;

[0011] A control strategy model for vehicles entering the ramp is designed based on the Soft Actor-Critic (SAC) algorithm, defining the state space S, action space A, and reward function r in the model.

[0012] Wherein, the state space S represents the vehicle's position and speed, and is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the weighted sum of the vehicle queue length, average vehicle speed, and vehicle waiting time, as follows:

[0013] r = μ1*r length +μ2r speed +μ3r waitTime

[0014] Where μ1, μ2, and μ3 are the weight parameters for vehicle queue length, average vehicle speed, and vehicle waiting time, respectively, and the sum of the weights is 1, representing the influence of different indicators on the reward function; r length r represents the average queue length of vehicles at the intersection. speed Let r be the average speed of all vehicles passing through the intersection. waitTime The average waiting time for all vehicles at the intersection;

[0015] Step S3: Define the algorithm flow;

[0016] Before model training, all parameters are initialized, environmental state data is updated based on the actions performed by the agent, and rewards are generated. The experience generated in each step of the agent's actions is stored in the experience replay pool. When the amount of experience stored reaches a custom threshold, the learning process begins, and the target network is updated to ensure the stability of training.

[0017] In step S3, the algorithm flow is as follows:

[0018] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization of policy network parameters. Q-value function parameters V-value function parameter γ and empirical replay pool D;

[0019] Step S3.2: The agent selects an action based on the environmental state and the policy network, and executes action a. t Update the environment to a new state S t+1 And generate a reward r t ;

[0020] Step S3.3: The experience (S) generated in each step... t ,a t ,S t+1 The experience (r) is stored in the experience replay pool D. When the experience in the experience replay pool D reaches a custom threshold, the learning process begins.

[0021] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D. The Q-value function, V-value function, policy parameters, and target network parameters are updated using the algorithm's loss function to update the target network and ensure the stability of training.

[0022] Step S4: Model training and validation;

[0023] During training, the model aims to maximize the cumulative reward function. It is trained by having the agent continuously explore in a simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is then verified using vehicle simulation technology.

[0024] In step S4, the model training process is as follows:

[0025] Step S4.1: Initialize environmental data by setting the vehicle's input rules;

[0026] Step S4.2: Collect vehicle information data at the intersection by calling the TraCI interface, and record the environmental state, the actions taken by the agent, the immediate reward, and the next environmental state.

[0027] Step S4.3: Calculate the target policy entropy, and update the critic network, actor network, and policy network;

[0028] Step S4.4: During training, automatically adjust the target policy entropy to balance the exploration and exploitation policies; if the policy entropy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration.

[0029] Step S4.5: Repeat steps S4.2 to S4.4 until the user-defined number of training rounds is reached.

[0030] In step S4, the only difference between the model verification process and the model training process is that the actor network and policy network are no longer updated during the verification process.

[0031] Step S5: Model application;

[0032] In the simulation platform, the intelligent agent uses Python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time, and uses the trained and validated model to achieve real-time control of traffic lights.

[0033] A traffic light intelligent control system based on deep reinforcement learning includes:

[0034] The simulation platform construction module is responsible for building a simulation platform based on SUMO (Simulation of Urban MObility) simulation software and Python. In the SUMO simulation software, a road network is built, and the road network strategy includes intersection strategy, traffic light strategy, and input vehicle strategy. In the simulation platform, the agent uses Python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time and realize real-time control of traffic lights.

[0035] The control strategy model design module is responsible for designing the control strategy model for vehicles entering the ramp based on the Soft Actor-Critic (SAC) algorithm, defining the state space S, action space A, and reward function r in the model.

[0036] Wherein, the state space S represents the vehicle's position and speed, and is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the weighted sum of the vehicle queue length, average vehicle speed, and vehicle waiting time, as follows:

[0037] r = μ1*r length +μ2r speed +μ3r waitTime

[0038] Where μ1, μ2, and μ3 are the weight parameters for vehicle queue length, average vehicle speed, and vehicle waiting time, respectively, and the sum of the weights is 1, representing the influence of different indicators on the reward function; r length r represents the average queue length of vehicles at the intersection. speed Let r be the average speed of all vehicles passing through the intersection. waitTime The average waiting time for all vehicles at the intersection;

[0039] The algorithm flow definition module is responsible for initializing all parameters before model training, updating environmental state data based on the actions performed by the agent, generating rewards, and storing the experience generated in each step of the agent's actions into the experience replay pool. When the amount of experience stored reaches a custom threshold, the learning process begins, and the target network is updated to ensure the stability of training.

[0040] The algorithm flow definition module is responsible for executing the following algorithm flow:

[0041] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization of policy network parameters. Q-value function parameters V-value function parameter γ and empirical replay pool D;

[0042] Step S3.2: The agent selects an action based on the environmental state and the policy network, and executes action a. t Update the environment to a new state S t+1 And generate a reward r t ;

[0043] Step S3.3: The experience (S) generated in each step... t ,a t ,S t+1 The experience (r) is stored in the experience replay pool D. When the experience in the experience replay pool D reaches a custom threshold, the learning process begins.

[0044] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D. The Q-value function, V-value function, policy parameters, and target network parameters are updated using the algorithm's loss function to update the target network and ensure the stability of training.

[0045] The model training and validation module is responsible for training and validating the model. During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent continuously exploring in the simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified by vehicle simulation technology.

[0046] In the model training and validation module, the model training process is as follows:

[0047] Step S4.1: Initialize environmental data by setting the vehicle's input rules;

[0048] Step S4.2: Collect vehicle information data at the intersection by calling the TraCI interface, and record the environmental state, the actions taken by the agent, the immediate reward, and the next environmental state.

[0049] Step S4.3: Calculate the target policy entropy, and update the critic network, actor network, and policy network;

[0050] Step S4.4: During training, automatically adjust the target policy entropy to balance the exploration and exploitation policies; if the policy entropy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration.

[0051] Step S4.5: Repeat steps S4.2 to S4.4 until the user-defined number of training rounds is reached.

[0052] In the model training and validation module, the only difference between the model validation process and the model training process is that the actor network and policy network are no longer updated during the validation process.

[0053] The model application module is responsible for acquiring real-time road vehicle information in the simulation platform by calling the TraCI interface of the SUMO simulation software through Python, and using the trained and validated model to achieve real-time control of traffic lights.

[0054] A traffic light intelligent control device based on deep reinforcement learning is characterized by comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method steps.

[0055] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program, when executed by a processor, implements the above-described method steps.

[0056] The beneficial effects of this invention are: the intelligent traffic light control method based on deep reinforcement learning can not only control the traffic light phase, but also dynamically adjust the traffic light phase according to the traffic flow density of each road, thereby balancing the waiting time of vehicles in each direction and improving the traffic efficiency of the intersection. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Appendix Figure 1 This is a schematic diagram of the intelligent traffic light control method based on deep reinforcement learning according to the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0060] Soft Actor-Critic (SAC) is a deep learning-based reinforcement learning method that combines the maximum entropy reinforcement learning concept with the actor-critic framework. It is an off-policy approach combined with deep reinforcement learning. Its advantage lies in its ability to find effective policies in a continuous action space and encourages high-entropy distributions in policy outputs through the maximum entropy principle, thereby improving exploration ability and stability. Furthermore, SAC provides a method to address overfitting in deep reinforcement learning—soft update. By constructing a model-based actor network and two value function-based critic networks, SAC can make high-quality decisions.

[0061] This intelligent traffic light control method based on deep reinforcement learning includes the following steps:

[0062] Step S1: Build a simulation platform;

[0063] A simulation platform was built using SUMO (Simulation of Urban Mobility) simulation software and Python. A road network was constructed in the SUMO simulation software, and the road network strategy included intersection strategy, traffic light strategy, and input vehicle strategy. In the simulation platform, the agent used Python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time and realize real-time control of traffic lights.

[0064] SUMO (Simulation of Urban Mobility) is an open-source, microscopic, multimodal traffic simulation software widely used in urban traffic planning, road network design, and traffic management. With its powerful features and flexibility, SUMO provides researchers and engineers with a robust simulation tool.

[0065] Step S2: Design the control strategy model;

[0066] A control strategy model for vehicles entering the ramp is designed based on the Soft Actor-Critic (SAC) algorithm, defining the state space S, action space A, and reward function r in the model.

[0067] Wherein, the state space S represents the vehicle's position and speed, and is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the weighted sum of the vehicle queue length, average vehicle speed, and vehicle waiting time, as follows:

[0068] r = μ1*r length+μ2r speed +μ3r waitTime

[0069] Where μ1, μ2, and μ3 are the weight parameters for vehicle queue length, average vehicle speed, and vehicle waiting time, respectively, and the sum of the weights is 1, representing the influence of different indicators on the reward function; r length r represents the average queue length of vehicles at the intersection. speed Let r be the average speed of all vehicles passing through the intersection. waitTime The average waiting time for all vehicles at the intersection;

[0070] Step S3: Define the algorithm flow;

[0071] Before model training, all parameters are initialized, environmental state data is updated based on the actions performed by the agent, and rewards are generated. The experience generated in each step of the agent's actions is stored in the experience replay pool. When the amount of experience stored reaches a custom threshold, the learning process begins, and the target network is updated to ensure the stability of training.

[0072] In step S3, the algorithm flow is as follows:

[0073] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization of policy network parameters. Q-value function parameters V-value function parameter γ and empirical replay pool D;

[0074] Step S3.2: The agent selects an action based on the environmental state and the policy network, and executes action a. t Update the environment to a new state S t+1 And generate a reward r t ;

[0075] Step S3.3: The experience (S) generated in each step... t ,a t ,S t+1 The experience (r) is stored in the experience replay pool D. When the experience in the experience replay pool D reaches a custom threshold, the learning process begins.

[0076] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D. The Q-value function, V-value function, policy parameters, and target network parameters are updated using the algorithm's loss function to update the target network and ensure the stability of training.

[0077] In step S3, the specific implementation process of the algorithm is as follows:

[0078] Inputs: Maximum time step M, initial policy network parameters Q-value function parameters and V-value function parameter ψ, empirical replay pool D;

[0079] Output: Parameters of the trained policy network

[0080] Initialize target parameters and network parameters: ψ targ ←ψ;

[0081] for m←1to Mdo:

[0082] Observe the current state s t+1 Then select action a according to the strategy. t ~π(.|s t );

[0083] Perform action a in the environment t Update state s using dynamic formula t+1 ;

[0084] Observe the next set of states s t+1 Calculate the reward r t ;

[0085] (S) t ,a t ,S t+1 (r) is stored in the experience pool D;

[0086] If state s t+1 If the state is terminated, the environment will be reset.

[0087] If there is enough experience in the experience pool, then:

[0088] for n←1 to N do:

[0089] A small batch of empirical samples is randomly taken from D, B = [(s t ,a t ,s t+1 ,r)];

[0090] Update Q-value function:

[0091]

[0092] Function to update V value:

[0093] φ←φ-λ π ▽ φ J π (φ)

[0094] Update strategy parameters:

[0095] α←α-λ▽ αJ(α)

[0096] Update the target value network parameters:

[0097]

[0098] Return policy network parameters

[0099] Step S4: Model training and validation;

[0100] During training, the model aims to maximize the cumulative reward function. It is trained by having the agent continuously explore in a simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is then verified using vehicle simulation technology.

[0101] In step S4, the model training process is as follows:

[0102] Step S4.1: Initialize environmental data by setting the vehicle's input rules;

[0103] Step S4.2: Collect vehicle information data at the intersection by calling the TraCI interface, and record the environmental state, the actions taken by the agent, the immediate reward, and the next environmental state.

[0104] Step S4.3: Calculate the target policy entropy, and update the critic network, actor network, and policy network;

[0105] Step S4.4: During training, automatically adjust the target policy entropy to balance the exploration and exploitation policies; if the policy entropy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration.

[0106] Step S4.5: Repeat steps S4.2 to S4.4 until the user-defined number of training rounds is reached.

[0107] In step S4, the only difference between the model verification process and the model training process is that the actor network and policy network are no longer updated during the verification process.

[0108] Step S5: Model application;

[0109] In the simulation platform, the intelligent agent uses Python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time, and uses the trained and validated model to achieve real-time control of traffic lights.

[0110] This intelligent traffic light control system based on deep reinforcement learning includes:

[0111] The simulation platform construction module is responsible for building a simulation platform based on SUMO (Simulation of Urban MObility) simulation software and Python. In the SUMO simulation software, a road network is built, and the road network strategy includes intersection strategy, traffic light strategy, and input vehicle strategy. In the simulation platform, the agent uses Python to call the TraCI interface of the SUMO simulation software to obtain road vehicle information in real time and realize real-time control of traffic lights.

[0112] The control strategy model design module is responsible for designing the control strategy model for vehicles entering the ramp based on the Soft Actor-Critic (SAC) algorithm, defining the state space S, action space A, and reward function r in the model.

[0113] Wherein, the state space S represents the vehicle's position and speed, and is a one-dimensional array; the action space A is defined as the phase of the traffic light; the reward function r is the weighted sum of the vehicle queue length, average vehicle speed, and vehicle waiting time, as follows:

[0114] r = μ1*r length +μ2r speed +μ3r waitTime

[0115] Where μ1, μ2, and μ3 are the weight parameters for vehicle queue length, average vehicle speed, and vehicle waiting time, respectively, and the sum of the weights is 1, representing the influence of different indicators on the reward function; r length r represents the average queue length of vehicles at the intersection. speed Let r be the average speed of all vehicles passing through the intersection. waitTime The average waiting time for all vehicles at the intersection;

[0116] The algorithm flow definition module is responsible for initializing all parameters before model training, updating environmental state data based on the actions performed by the agent, generating rewards, and storing the experience generated in each step of the agent's actions into the experience replay pool. When the amount of experience stored reaches a custom threshold, the learning process begins, and the target network is updated to ensure the stability of training.

[0117] The algorithm flow definition module is responsible for executing the following algorithm flow:

[0118] Step S3.1: Initialize all parameters before model training, including the maximum time step M and the initialization of policy network parameters. Q-value function parameters V-value function parameter γ and empirical replay pool D;

[0119] Step S3.2: The agent selects an action based on the environmental state and the policy network, and executes action a. t Update the environment to a new state S t+1 And generate a reward r t ;

[0120] Step S3.3: The experience (S) generated in each step... t ,a t ,S t+1 The experience (r) is stored in the experience replay pool D. When the experience in the experience replay pool D reaches a custom threshold, the learning process begins.

[0121] Step S3.4: During the learning process, random sampling is performed in the experience replay pool D. The Q-value function, V-value function, policy parameters, and target network parameters are updated using the algorithm's loss function to update the target network and ensure the stability of training.

[0122] The model training and validation module is responsible for training and validating the model. During the training process, the model aims to maximize the cumulative reward function. The model is trained by the agent continuously exploring in the simulated environment to improve its strategy for controlling traffic lights. The effectiveness of the model is verified by vehicle simulation technology.

[0123] In the model training and validation module, the model training process is as follows:

[0124] Step S4.1: Initialize environmental data by setting the vehicle's input rules;

[0125] Step S4.2: Collect vehicle information data at the intersection by calling the TraCI interface, and record the environmental state, the actions taken by the agent, the immediate reward, and the next environmental state.

[0126] Step S4.3: Calculate the target policy entropy, and update the critic network, actor network, and policy network;

[0127] Step S4.4: During training, automatically adjust the target policy entropy to balance the exploration and exploitation policies; if the policy entropy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration.

[0128] Step S4.5: Repeat steps S4.2 to S4.4 until the user-defined number of training rounds is reached.

[0129] In the model training and validation module, the only difference between the model validation process and the model training process is that the actor network and policy network are no longer updated during the validation process.

[0130] The model application module is responsible for acquiring real-time road vehicle information in the simulation platform by calling the TraCI interface of the SUMO simulation software through Python, and using the trained and validated model to achieve real-time control of traffic lights.

[0131] The intelligent traffic light control device based on deep reinforcement learning includes a memory and a processor; the memory is used to store computer programs, and the processor is used to execute the computer programs to implement the above-described method steps.

[0132] The readable storage medium stores a computer program that, when executed by a processor, implements the above-described method steps.

[0133] Compared with existing technologies, this intelligent traffic light control method based on deep reinforcement learning has the following characteristics:

[0134] First, by leveraging the ability of neural networks to perceive high-dimensional data, useful information can be learned from multi-dimensional data, dynamically capturing the traffic conditions of each lane at intersections, thus improving the model's efficiency in utilizing data.

[0135] Secondly, compared with traditional autonomous driving strategies, it has the ability to explore the action space, realizes continuous control of traffic light phases, and improves the vehicle's automated decision-making ability and congestion prediction and management capabilities.

[0136] Third, the training process of the neural network is based on SUMO simulation software, which can simulate more traffic scenarios and train the neural network from multiple scenarios. This allows the neural network to explore more decision spaces and learn better control strategies. Through continuous iterative training, the final trained neural network outperforms existing traffic light control methods.

[0137] Fourth, the state space includes multi-dimensional vehicle and road information, which is suitable for a variety of complex traffic scenarios. The reward function is designed from multiple perspectives, including queue length, waiting time and average speed, so as to ensure the overall traffic efficiency of the intersection while taking into account the queue length and vehicle waiting time in each direction.

[0138] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A traffic light intelligent control method based on deep reinforcement learning, characterized in that: Comprising the following steps: Step S1, build a simulation platform; Based on SUMO simulation software and python, build a simulation platform, build a road network in SUMO simulation software, and the road network strategy includes intersection strategy, signal light strategy and input vehicle strategy; Step S2, design a control strategy model; Based on the flexible action-evaluation algorithm SAC, design a control strategy model for the vehicle entering the ramp, and define the state space S, action space A and reward function r in the model respectively; Wherein, the state space S is the position and speed of the vehicle, which is a one-dimensional array; the action space A is defined as the phase of the signal light; the reward function r is the weighted sum of the vehicle queue length, vehicle average speed and vehicle waiting time, as follows: r = μ1 * r length + μ2 * r speed + μ3 * r waitTime where μ1, μ2and μ3are weight parameters of vehicle queue length, vehicle average speed and vehicle waiting time, respectively, and the weight sum is 1; r length denotes the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection; Step S3, define the algorithm flow; Before model training, initialize all parameters, update environment state data according to the action of the agent, and generate rewards, store the experience generated in each step of the agent's action into the experience replay pool; when the experience storage reaches the self-defined threshold, start the learning process, update the target network to ensure the stability of the training; Step S4, model training and verification; During the training process, the model aims to maximize the cumulative reward function, and trains the model through the continuous exploration of the agent in the simulated environment to improve its control signal light strategy, and verifies the effectiveness of the model through vehicle simulation technology; Step S5, model application; In the simulation platform, the agent calls the TraCI interface of the SUMO simulation software through python to obtain real-time road vehicle information, and uses the trained and verified model to realize real-time control of the signal light. 2.The deep reinforcement learning based signal light intelligent control method according to claim 1, characterized in that: In step S3, the algorithm flow is as follows: Step S3.1, initialize all parameters before model training, including maximum time step M, initialize policy network parameters Q-value function parameters V-value function parameters γ and experience replay pool D; Step S3.2, the agent selects an action according to the environment state and the policy network, the agent performs the action a t The environment is updated to a new state S t+1 And a reward r is generated t ; Step S3.3, store the experience (S t ,a t ,S t+1 ,r) generated in each step into the experience replay pool D, and start the learning process when the experience in the experience replay pool D reaches a self-defined threshold; Step S3.4, during the learning process, random sampling is performed in the experience replay pool D, the Q value function, V value function, policy parameter and target value network parameter are updated through the loss function of the algorithm, and the target network is updated to ensure the stability of the training. 3.The deep reinforcement learning based traffic light intelligent control method according to claim 1, characterized in that: In step S4, the model training process is as follows: Step S4.1, initialize the environment data by setting the input rules of the vehicle; Step S4.2, collect intersection vehicle information data by calling the TraCI interface, and record the environment state, the action taken by the agent, the immediate reward and the next environment state; Step S4.3, calculate the target policy entropy, update the critic network, actor network and policy network; Step S4.4, during the training process, automatically adjust the target policy entropy to balance the exploration and utilization strategy; if the entropy of the strategy is lower than the target policy entropy, increase the value of the target policy entropy to encourage more exploration; Step S4.5, repeat steps S4.2 to S4.4 until the user-defined predetermined training rounds are reached. 4.The deep reinforcement learning based signal light intelligent control method according to claim 3, characterized in that: In step S4, the difference between the model verification process and the model training process is that the actor network and the policy network are not updated in the verification process.

5. A signal light intelligent control system based on deep reinforcement learning, characterized in that: Comprising: The simulation platform building module is responsible for building a simulation platform based on SUMO simulation software and python, and building a road network in the SUMO simulation software, wherein the road network strategy includes an intersection strategy, a signal light strategy and an input vehicle strategy; The control strategy model design module is responsible for designing a control strategy model of the ramp-in vehicle based on a flexible action-evaluation algorithm SAC, and defining a state space S, an action space A and a reward function r in the model respectively; The state space S is a one-dimensional array of the position and speed of the vehicle; the action space A is defined as the phase of the signal light; and the reward function r is a value obtained by weighted summation of the vehicle queue length, the vehicle average speed and the vehicle waiting time, and is specifically as follows: r = μ1 * r length + μ2 r speed + μ3 r waitTime wherein μ1, μ2 and μ3 are weight parameters of vehicle queue length, vehicle average speed and vehicle waiting time, respectively, the weight sum is 1, indicating the influence of different indicators on the reward function; r length represents the average queue length of vehicles at the intersection, r speed is the average speed of all vehicles passing through the intersection, r waitTime is the average waiting time of all vehicles at the intersection; The algorithm flow definition module is responsible for initializing all parameters before model training, updating the environment state data according to the action executed by the agent, and generating a reward, and storing the experience generated in each step of the action executed by the agent into an experience replay pool; when the experience storage amount reaches a self-defined threshold, the learning process is started, and the target network is updated to ensure the stability of the training; The model training and verification module is responsible for training and verifying the model; during the training process, the model takes maximizing the cumulative reward function as the target, trains the model in the simulated environment through continuous exploration of the agent, improves the policy of controlling the signal light, and verifies the effectiveness of the model through vehicle simulation technology; The model application module is responsible for real-time acquisition of road vehicle information through the TraCI interface of the SUMO simulation software in the simulation platform, and real-time control of the signal light by using the trained and verified model. 6.The deep reinforcement learning based traffic light intelligent control system according to claim 5, wherein: The algorithm flow executed by the algorithm flow definition module is as follows: Step S3.1, initialize all parameters before model training, including maximum time step M, initialize policy network parameters Q-value function parameters V-value function parameter γ and experience replay pool D; Step S3.2, the agent selects an action according to the environment state and policy network, the agent performs action a t The environment is updated to a new state S t+1 And produces a reward r t ; Step S3.3, storing the experience (S t t t+1 r) into the experience replay pool D, and starting the learning process when the experience in the experience replay pool D reaches a self-defined threshold.​​ In the learning process, random sampling is performed in the experience replay pool D, the Q value function, the V value function, the policy parameter and the target value network parameter are updated through the loss function of the algorithm, the target network is updated to ensure the stability of the training. 7.The deep reinforcement learning based traffic light intelligent control system according to claim 5, wherein: In the model training and verification module, the model training process is as follows: Step S4.1, the environment data is initialized by setting the input rules of the vehicle; Step S4.2, the intersection vehicle information data is collected by calling the TraCI interface, and the environment state, the action taken by the agent, the immediate reward and the next environment state are recorded; Step S4.3, the target policy entropy is calculated, and the critic network, the actor network and the policy network are updated; Step S4.4, in the training process, the target policy entropy is automatically adjusted to balance the exploration and utilization strategies; if the entropy of the policy is lower than the target policy entropy, the value of the target policy entropy is increased to encourage more exploration; Step S4.5, steps S4.2 to S4.4 are repeated until the user-defined predetermined training round is reached. 8.The deep reinforcement learning based traffic light intelligent control system of claim 7, wherein: In the model training and verification module, the difference between the model verification process and the model training process is that the actor network and the policy network are not updated in the verification process.

9. A signal light intelligent control device based on deep reinforcement learning, characterized in that: A computer program product comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured such that, on execution by a computer, the computer is caused to perform the method of any one of claims 1 to 4.

10. A readable storage medium characterized by: A computer program product comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured such that, on execution by a computer, the computer is caused to perform the method of any one of claims 1 to 4.

Images