Traffic signal lamp and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning

Abstract

The invention relates to the technical field of traffic signal lamp and vehicle cooperative control, and discloses a traffic signal lamp and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning. According to the invention, the traffic lights and the CAV are cooperatively controlled by using two types of agents, and the two types of agents interact and cooperate with each other, so that the overall traffic efficiency in a mixed traffic flow environment is improved; the two types of intelligent agents effectively perform information communication and task cooperation through a hierarchical cooperation mechanism, so that the traffic efficiency of the intersection and the safety and comfort of CAV driving are further improved; by means of a heterogeneous multi-agent learning framework, effective collaborative learning is carried out on heterogeneous multiple tasks, the problem of unstable training is solved, and the overall training effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic light and vehicle cooperative control, and in particular to a traffic light and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning. Background Art

[0002] In recent years, with the advancement of technologies such as communication and autonomous driving, more and more connected autonomous vehicles (CAVs) and intelligent roadside units have been introduced, which enhances the real-time perception and sharing of road information and promotes adaptive traffic signal control and vehicle trajectory control. However, existing technologies usually regard traffic signal control and CAV control as two independent fields, and use reinforcement learning methods to independently control traffic signals and vehicles. Although better control effects are achieved compared with traditional methods, information sharing and interaction between vehicles and lights are not considered, which limits the overall efficiency of the traffic system; although some technologies try to use reinforcement learning to jointly control traffic lights and vehicles, these methods often only share information between agents, without establishing mechanisms to ensure mutual understanding of shared information, and without clear and reasonable cooperation mechanisms, resulting in potential inefficiency and lack of explainability. At the same time, such methods usually assume the full controllability of the leading vehicle and ignore the actual mixed traffic scenarios.

[0003] Patent CN118247986A discloses a method for collaborative control of vehicles at a single-signal intersection under mixed traffic flow. Traffic flow status information is collected and generated by information collection equipment, and the optimal speed of the mixed traffic queue is calculated in combination with signal light timing and traffic flow status information, and then the target vehicle is controlled to complete the corresponding driving operation. Patent CN118629233A discloses a method for collaborative control of traffic lights and intelligent networked vehicles in an intersection scenario. Traffic lights and CAV vehicles are modeled as two types of intelligent agents. The two types of intelligent agents interact with the environment in real time, share status information, and train deep reinforcement learning models. The trained model is used to collaboratively control the vehicle and lights.

[0004] The above prior art mainly has the following problems:

[0005] 1) Currently, most existing technologies control traffic lights and CAVs independently, ignoring the interaction between traffic lights and vehicles, which limits the overall efficiency of the traffic system. Patent CN118247986A combines traffic flow information and traffic light timing information to achieve optimal vehicle control, but does not consider the impact of vehicle speed changes on traffic lights, nor the overall efficiency of the traffic system.

[0006] 2) At present, the existing technology only shares information when controlling traffic lights and vehicles in a coordinated manner. There is no clear cooperation mechanism or mechanism for mutual understanding of shared information. Although patent CN118629233A uses reinforcement learning to control traffic lights and vehicles in a coordinated manner, it only shares state information between the two types of intelligent agents, the vehicle and the light, without a clear goal of cooperative optimization, which reduces the optimality of coordinated control. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for collaborative control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning. Traffic lights and CAVs are modeled as two types of intelligent agents, and a hierarchical cooperation mechanism is designed to ensure efficient communication and understanding of information between CAVs and traffic signals at different time scales. At the same time, a heterogeneous multi-agent mirror learning algorithm is used to enable heterogeneous agents to effectively collaborate in a multi-task environment, thereby improving the overall efficiency of the traffic system and reducing the instability in the training process.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] A traffic light and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning, characterized in that it includes the following steps:

[0010] S1. Create traffic light agents and intelligent connected vehicle agents, initialize the critic network, actor network and corresponding target network;

[0011] S2, initializing the traffic environment, road network environment and traffic flow conditions;

[0012] S3, obtain the observation value of the interaction between the traffic light agent and the environment, select the action and calculate the reward value according to the observation value, and store the action trajectory into the traffic light experience replay pool;

[0013] S4, determine whether the actor network of the traffic light agent starts to update; if so, batch sample action trajectories from the traffic light experience playback pool, calculate the critic network loss function, and update the critic network parameters; otherwise, execute step S6;

[0014] S5, determine whether the target network of the traffic light agent starts to be updated; if so, randomly arrange the actor update order and update them in sequence, and update the target network of the traffic light agent according to the updated actors; otherwise, execute step S6;

[0015] S6. Calculate the lane-level target according to the action of the traffic light agent, obtain the observation value of the intelligent networked vehicle agent, select the action according to the observation value and calculate the reward value, and store the action trajectory into the intelligent networked vehicle experience replay pool;

[0016] S7, determine whether the actor network of the intelligent connected vehicle agent starts to be updated; if so, batch sample action trajectories from the intelligent connected vehicle experience replay pool, calculate the commentator network loss function, and update the commentator network parameters; otherwise, execute step S9;

[0017] S8, determining whether the target network of the intelligent connected vehicle agent starts to be updated; if so, calculating the actor network objective function of the intelligent connected vehicle agent, updating the actor network parameters of the intelligent connected vehicle agent, and updating the target network of the intelligent connected vehicle agent according to the updated actor; otherwise, executing step S9;

[0018] S9, determining whether the traffic light agent has interacted with the environment at a maximum number of times; if so, executing step S10; otherwise, returning to step S3;

[0019] S10, determine whether the maximum number of iterations has been reached; if so, the process ends; otherwise, return to step S2.

[0020] As a preferred method, when creating a traffic light agent, the action space is constructed by the phase selection task and the phase timing task respectively; wherein the phase selection task consists of a discrete action Indicates that p i Indicates four allowed traffic phases; the phase timing task consists of a continuous action Indicates that g min and g max Respectively represent the minimum and maximum green light time allowed for a single phase;

[0021] According to the information of each lane and the historical traffic road information, the observation value of the state of the traffic light agent at time t is defined as:

[0022]

[0023] Among them, p represents the one-hot encoding of the current phase, q l and n l They represent the number of queued vehicles and the total number of vehicles on lane l, L in Indicates all the incoming lanes at the intersection;

[0024] The intersection pressure is calculated based on the difference between the total number of vehicles entering and leaving the lane, and the reward function of the traffic light agent is established as:

[0025]

[0026] Among them, c l represents the capacity of lane l, n l represents the total number of vehicles on lane l, L out Indicates all exit lanes at the intersection.

[0027] As a preferred embodiment, when creating an intelligent networked vehicle agent, the intelligent networked vehicle agent action is defined as and They represent the minimum and maximum acceleration allowed for the vehicle respectively;

[0028] According to the driving information of the intelligent connected vehicle itself, the information of the vehicle ahead and the traffic signal information, the observation value of the intelligent connected vehicle agent at time t is defined as:

[0029]

[0030] Among them, x t represents the distance from the vehicle to the stop line of the intersection, v t and ρ t represents the speed and acceleration of the vehicle, λ t and μ t Respectively represent the distance and speed relative to the front vehicle, φ t ∈{0,1} indicates whether the traffic light allows the current vehicle to pass, η t Indicates the remaining time of the current phase;

[0031] Reward ψ based on safety t , comfort bonus t , efficiency reward ν t and bad behavior punishment P t , the reward function of the intelligent connected vehicle agent is established as:

[0032] r t CAV =ω1ψ t +ω2ξ t +ω3ν t -P t

[0033] Among them, ω1, ω2, ω3 represent weight coefficients;

[0034] Safety Reward ψ t Specifically:

[0035]

[0036] Among them, TTC represents the time to collision;

[0037] Comfort bonust Specifically:

[0038]

[0039] Among them, ρ max Indicates the maximum acceleration of the vehicle;

[0040] Efficiency Reward ν t Defined as:

[0041]

[0042] Among them, v max Indicates the maximum speed of the vehicle;

[0043] Bad behavior punishment t Defined as:

[0044]

[0045]

[0046] Among them, v min represents the minimum speed of the vehicle, Indicates the allowed acceleration of the vehicle.

[0047] As a preferred embodiment, the traffic light agent and the intelligent networked vehicle agent adopt a hierarchical cooperation mechanism, where the traffic light manager agent sets the expected goal for each lane and controls the expected state that the intelligent networked vehicle executor agent on each lane needs to achieve, so as to communicate information and cooperate on tasks;

[0048] According to the state observation value of the traffic light, the current lane state and the lane index value, the observation value of the state of the traffic light manager agent at time t is defined as:

[0049]

[0050] in, represents the state observation of the traffic light, Indicates the state of lane l, where l represents the lane index value;

[0051] According to the negative value of the average number of vehicles queued per lane per second, the reward function of the traffic light manager agent is established as:

[0052]

[0053] Among them, T represents the interval between two decisions of the manager, L in represents all the entry lanes at the intersection, q lrepresents the number of queued vehicles on lane l, k represents the sum of the total number of queued vehicles per second between two decisions;

[0054] According to the state observation value of the intelligent network vehicle agent, the expected speed of the current state and the expected speed of the next time period, the observation value of the state of the intelligent network vehicle executor agent at time t is defined as:

[0055]

[0056] in, represents the state observation value of the intelligent connected vehicle agent, Indicates the expected speed of the current state, represents the expected speed in the next time period, Δt represents the minimum control step of the intelligent connected vehicle, and x t represents the distance from the vehicle to the stop line of the intersection, v t Indicates the speed of the vehicle;

[0057] According to the reward of the intelligent connected vehicle agent and the intrinsic reward of the target action, the reward function of the intelligent connected vehicle executor agent is established as:

[0058]

[0059] Among them, α represents the intrinsic reward weight factor, κ represents the normalization factor of the remaining green light countdown, and r t CAV represents the reward of the intelligent connected vehicle agent, r t int represents the intrinsic reward of the target action.

[0060] Preferably, when updating the critic network parameters, the update is performed by minimizing the loss of the critic network, which is expressed as:

[0061]

[0062] Among them, E represents the expectation, s represents the current state, a represents the action, r represents the reward, s' represents the next state, and B represents the small batch learning samples extracted from the experience replay cache. represents the Q value of taking action a in state s, θ i represents the parameters of critic network i, and y represents the target value.

[0063] As a preferred method, the updating order of the actors is randomly arranged and updated sequentially as follows:

[0064]

[0065] Among them, φ i represents the actor network parameters, represents the updated parameters of the actor network i, B represents the number of mini-batch learning samples, represents the action-value function fitted by the traffic light critic network, s t represents the global state at time t, Indicates that the current actor strategy is not updated according to its observation The set of actions taken, Indicates the current actor strategy to be updated according to its observation The set of actions taken.

[0066] Preferably, the target network of the traffic light agent is updated according to the updated actor as follows:

[0067]

[0068] in, represents the updated parameters of the traffic light critic target network i, τ represents the soft update coefficient, and θ i TS represents the parameters of the traffic light critic network i, represents the updated parameters of the traffic light actor target network j, φ j represents the parameters of the traffic light actor network j, p represents the phase selection task, t represents the phase timing task, and M represents the goal setting task.

[0069] As a preferred embodiment, the actor network parameters of the intelligent connected vehicle agent are updated as follows:

[0070]

[0071] in, represents the gradient of the policy optimization objective function with respect to the parameter φ, E s represents the expected value on all states s, represents the gradient of the Q function with respect to action a, π φ (s) represents the action taken by the strategy in state s, represents the gradient of the policy with respect to the parameter φ.

[0072] Preferably, the target network of the intelligent connected vehicle agent is updated according to the updated actor as follows:

[0073]

[0074] in, represents the updated parameters of the intelligent connected car critic target network i, τ represents the soft update coefficient, θ i CAV represents the parameters of the intelligent connected car critic network i, represents the updated intelligent connected vehicle actor target network parameters, φ v Represents the network parameters of intelligent connected vehicle actors.

[0075] The present invention has the following beneficial effects:

[0076] The present invention uses two types of intelligent agents to collaboratively control traffic lights and CAVs. The two types of intelligent agents interact and cooperate with each other to improve the overall traffic efficiency in a mixed traffic flow environment. The two types of intelligent agents effectively communicate information and collaborate on tasks through a hierarchical cooperation mechanism, further improving the traffic efficiency of intersections and the safety and comfort of CAV driving. With the help of a heterogeneous multi-agent learning framework, effective collaborative learning of heterogeneous multi-tasks is carried out to solve the problem of unstable training and improve the overall training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the process of the present invention;

[0078] Figure 2 This is a schematic diagram of the architecture of the present invention;

[0079] Figure 3 Schematic diagram of the multi-heterogeneous agent learning framework based on double-delayed deep deterministic policy gradient in the present invention. DETAILED DESCRIPTION

[0080] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0081] The present invention designs a collaborative control method for traffic lights and intelligent networked vehicles based on heterogeneous hierarchical reinforcement learning. The method designs two types of intelligent agents, traffic lights and intelligent networked vehicles. They are trained by interacting with the environment and finally achieve the effect of making decisions according to the current traffic status. The traffic light agent is composed of three heterogeneous agents, namely, phase agent, timing agent and manager agent. The phase agent selects the phase of the next decision step of the traffic light, the timing agent controls the duration of the phase, and the manager agent and the vehicle agent form a hierarchical cooperation mechanism, in which the manager agent of the traffic light sets an expected goal. As an executor agent, the vehicle agent will consider the goal during training and make decisions by comprehensively considering the goal during application. In addition, in order to ensure the effective collaboration of the three intelligent agents of the traffic light, the algorithm uses a heterogeneous agent mirror learning framework, which randomly selects the update order of the actor network through a shared critic network, and the actor updated in the latter step will consider the former action. This design promotes the collaborative training of the three types of agents and ensures the stability of the overall training.

[0082] Reference Figure 1 and Figure 2 , an embodiment of the present invention provides a traffic light and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning, comprising the following steps:

[0083] S1. Create traffic light agents and intelligent connected vehicle agents, initialize the critic network, actor network and corresponding target network;

[0084] In an optional embodiment of the present invention, step S1 first creates two types of agents, namely, a traffic light agent and an intelligent connected vehicle agent, and initializes the parameters of each network in the two types of agents, including the critic network of the traffic light agent. and centralized reviewer networks Phase Actor Network Timing Actor Network Managing the network of actors And the corresponding target network; the critic network of the intelligent connected vehicle agent and centralized reviewer networks Intelligent Connected Vehicle Actor Network And its corresponding target network. The actor network is used to select the appropriate action according to the current state, the critic network is used to evaluate the quality of the action in the current state, and the target network is an auxiliary network used to improve the stability of training.

[0085] In this embodiment, the traffic signal at each moment is controlled by the traffic light agent, and the position of each intelligent networked vehicle in the entry lane and the distance to the vehicle in front are controlled by the intelligent networked vehicle agent.

[0086] The traffic light agent makes a decision when each green phase ends and is about to switch. The decision content has two parts: one is to select the next phase, and the other is to determine the duration of the selected phase. These two types of tasks have different action spaces, but share the same state space and reward function. The specific design is as follows:

[0087] (a) Action space design

[0088] The phase selection task consists of a discrete action Indicates that p i Indicates the four allowed traffic phases; the phase timing task has one continuous action Indicates that g min and g max They respectively represent the minimum and maximum green light time allowed for a single phase, limiting the phase time to a reasonable range.

[0089] (b) State space design

[0090] Taking into account the information of each lane and the historical traffic road information, the observation value of the state of the traffic light agent at time t is defined as:

[0091]

[0092] Among them, p represents the one-hot encoding of the current phase, q l and n l They represent the number of queued vehicles and the total number of vehicles on lane l, respectively. Both are normalized according to the road capacity. in Indicates all incoming lanes at the intersection.

[0093] (c) Reward function design

[0094] The design of the traffic light reward function mainly considers the intersection pressure, which is defined as the difference between the total number of vehicles entering and leaving the lane. Reducing the pressure is equivalent to maximizing the traffic capacity, so the reward function is designed as:

[0095]

[0096] Among them, c l represents the capacity of lane l, n l Represents the total number of vehicles on lane l.

[0097] The intelligent body control of the intelligent connected vehicle controls the longitudinal acceleration of the intelligent connected vehicle in coordination with the signal lights for efficient and comfortable driving. The specific design is as follows:

[0098] (a) Action space design

[0099] The agent action is designed to represents the longitudinal acceleration of the CAV at time t, where and They represent the minimum and maximum values ​​allowed by the vehicle, respectively, ensuring safety and comfort during control.

[0100] (b) State space design

[0101] Taking into account the driving information of the intelligent connected vehicle itself, the information of the vehicle ahead, and the traffic signal information, the observation value of the intelligent connected vehicle agent at time t is defined as:

[0102]

[0103] Among them, x t represents the distance from the vehicle to the stop line of the intersection, v t and ρ t represents the speed and acceleration of the vehicle, λ t and μ t Respectively represent the distance and speed relative to the front vehicle, φ t ∈{0,1} indicates whether the traffic light allows the current vehicle to pass, η t Indicates the remaining time of the current phase.

[0104] (c) Reward function design

[0105] The reward function of the intelligent connected vehicle agent is mainly composed of the safety reward ψ t , comfort bonus t , efficiency reward ν t and bad behavior punishment P t Composition, expressed as:

[0106] r t CAV =ω1ψ t +ω2ξ t +ω3ν t -P t

[0107] Among them, ω1, ω2, ω3 are weight coefficients.

[0108] Safety Reward ψ t Related to Time-to-Collision (TCC), a collision time of less than four seconds is considered unsafe:

[0109]

[0110] Comfort bonus t Defined as the difference in acceleration between time steps:

[0111]

[0112] Efficiency Reward ν t Defined as the normalized velocity:

[0113]

[0114] Bad behavior punishment t Consists of lazy behavior, improper acceleration behavior, and mismatched action behavior:

[0115]

[0116] Among them, lazy behavior penalty P t loaf This is mainly aimed at situations where the CAV’s current speed cannot pass through the intersection within the remaining green light time:

[0117]

[0118] Improper acceleration penalties for speeding at a red light:

[0119]

[0120] The action mismatch penalty is used to correct and narrow the gap between expected acceleration and actual acceleration:

[0121]

[0122] Among them, v min represents the minimum speed of the vehicle, Indicates the allowed acceleration of the vehicle.

[0123] This embodiment adopts a hierarchical cooperation mechanism between traffic light agents and intelligent connected vehicle agents, which is mainly composed of traffic light manager agents and intelligent connected vehicle executor agents. The manager agent sets goals and specifies the expected state that the intelligent connected vehicle executor agent needs to achieve in order to achieve information communication and task cooperation.

[0124] The goal is used to convey the manager's expectations to the executor in the hierarchical cooperation mechanism, which plays a role in connecting the manager agent and the executor agent. The method of the present invention is designed as a lane-level goal, that is, the manager sets a goal for each lane to control all intelligent connected vehicles on the lane. The lane l goal is defined as the function v l (x), represents the relationship between the vehicle speed v and the distance x to the stop line. This function can be combined with the target state vector generated by the manager. The conversion between them is done through sampling and interpolation.

[0125] For each lane, the manager's action is is the target state of the lane The lane-level target v can be obtained by interpolation. l (x), thereby guiding the decision-making of intelligent connected vehicles.

[0126] Unlike the phase and timing agents of traffic lights, the traffic light manager agent uses more vehicle information and thus Added the current lane status based on And the one-hot encoding l of the lane index value:

[0127]

[0128] The reward function of the traffic light manager agent is defined as the negative of the average number of vehicles queued per lane per second to promote cooperation between traffic lights and vehicles and reduce the number of stops and fuel consumption:

[0129]

[0130] Intelligent connected vehicle executive agents ensure safety and driving efficiency through Achieve the manager's expected goals for the lane.

[0131] The state observation value of the intelligent connected vehicle actor agent increases the expected speed of the current state and the expected speed in the next time period To help executors understand tasks more efficiently:

[0132]

[0133] To encourage consistency between the manager and executor’s target actions, the executor agent incorporates an intrinsic reward function That is, the negative L2 norm of the difference between the current lane state and the target lane state. The executor agent reward function is defined as:

[0134]

[0135] Among them, α is the intrinsic reward weight factor, and κ is the normalization factor of the remaining green light countdown, that is, as the remaining green light time decreases, the executor attaches more and more importance to the intrinsic reward.

[0136] S2, initializing the traffic environment, road network environment and traffic flow conditions;

[0137] In an optional embodiment of the present invention, step S2 creates and initializes a traffic simulation environment, randomly sets the road network environment and traffic conditions of the simulation environment, sets the traffic light agent to control the traffic lights in the simulation environment, sets the intelligent connected vehicle agent to control the intelligent connected vehicle, starts the simulation environment, and begins interacting with the agent.

[0138] S3, obtain the observation value of the interaction between the traffic light agent and the environment, select the action and calculate the reward value according to the observation value, and store the action trajectory into the traffic light experience replay pool;

[0139] In an optional embodiment of the present invention, step S3, at the end of the current green light phase, the traffic light agent makes a decision and obtains the observation value and Select an action:

[0140]

[0141] Among them, σ TS To randomly explore noise, ensure the stability of the overall training.

[0142] After selecting an action, the agent performs the action and calculates the reward r t TS and r t M , an action trajectory Deposit into signal light experience replay pool B TS .

[0143] S4, determine whether the actor network of the traffic light agent starts to update; if so, batch sample action trajectories from the traffic light experience playback pool, calculate the critic network loss function, and update the critic network parameters; otherwise, execute step S6;

[0144] In an optional embodiment of the present invention, step S4 is to replay the experience pool B. TS After storing a certain value, batch sample action trajectory B and update the critic network parameters by minimizing the loss of the critic network and And reduce the random exploration noise σ TS .

[0145] In this embodiment, the traffic light includes three agents that complete three tasks: signal light phase selection, phase timing, and setting goals for executors. These tasks involve certain cooperative games. Therefore, the present invention designs a multi-heterogeneous agent learning framework based on double-delayed deep deterministic policy gradients. Figure 3This learning framework uses a decentralized execution and centralized training approach. Different executor networks share a centralized critic network to evaluate the quality of the actions taken by the actors. Two critic networks are used here. and Using the minimum of the two estimates to reduce the bias, the critic network is updated by minimizing the loss:

[0146]

[0147] Among them, E represents the expectation, s represents the current state, a represents the action, r represents the reward, s' represents the next state, and B represents the small batch learning samples extracted from the experience replay cache. represents the Q value of taking action a in state s, θ i represents the parameters of critic network i, and y represents the target value.

[0148] The target value y is calculated as:

[0149]

[0150] in, and are the parameters of the target network and ò is the clipping noise added for target policy smoothing.

[0151] Different from the traditional double-delayed deep deterministic policy gradient algorithm, the centralized critic allows the actor to consider the information in the overall training process. Each time the network parameters are updated, the update order of the actor is rearranged. When updating sequentially, the Q function of each actor will consider the latest strategy of the previously updated actor. Here, the deterministic policy gradient is used to update the actor network:

[0152]

[0153] in, represents the gradient of the policy optimization objective function with respect to the parameter φ, E s represents the expected value on all states s, represents the gradient of the Q function with respect to action a, π φ (s) represents the action taken by the strategy in state s, represents the gradient of the policy with respect to the parameter φ.

[0154] S5, determine whether the target network of the traffic light agent starts to be updated; if so, randomly arrange the actor update order and update them in sequence, and update the target network of the traffic light agent according to the updated actors; otherwise, execute step S6;

[0155] In an optional embodiment of the present invention, step S5 updates the traffic light actor network and the target network at fixed intervals. First, the update order of the three types of agents is randomly arranged. For the random order of agents {p, t, M}, the updates are performed in sequence as follows:

[0156]

[0157] Then update the traffic light agent target network as follows:

[0158]

[0159] in, represents the updated parameters of the traffic light critic target network i, τ represents the soft update coefficient, and θ i TS represents the parameters of the traffic light critic network i, represents the updated parameters of the traffic light actor target network j, φ j represents the parameters of the traffic light actor network j, p represents the phase selection task, t represents the phase timing task, and M represents the goal setting task.

[0160] S6. Calculate the lane-level target according to the action of the traffic light agent, obtain the observation value of the intelligent networked vehicle agent, select the action according to the observation value and calculate the reward value, and store the action trajectory into the intelligent networked vehicle experience replay pool;

[0161] In an optional embodiment of the present invention, step S6 manages the actions of the actors according to the signal lights. Get lane-level target status Interpolation calculation CAV agent obtains state observations Calculated and r t int , combined with random exploration noise σ CAV Select an action:

[0162]

[0163] The CAV agent performs the action and calculates the reward r t W , an action trajectory Deposit into CAV experience replay pool B CAV .

[0164] S7, determine whether the actor network of the intelligent connected vehicle agent starts to be updated; if so, batch sample action trajectories from the intelligent connected vehicle experience replay pool, calculate the commentator network loss function, and update the commentator network parameters; otherwise, execute step S9;

[0165] In an optional embodiment of the present invention, step S7 is to replay the experience pool B. CAV After storing a certain value, batch sample action trajectory B and update the critic network parameters by minimizing the loss of the critic network and And reduce the random exploration noise σ CAV .

[0166] S8, determining whether the target network of the intelligent connected vehicle agent starts to be updated; if so, calculating the actor network objective function of the intelligent connected vehicle agent, updating the actor network parameters of the intelligent connected vehicle agent, and updating the target network of the intelligent connected vehicle agent according to the updated actor; otherwise, executing step S9;

[0167] In an optional embodiment of the present invention, step S8 updates the CAV actor network parameter φ at fixed intervals according to equation (18). V , and update the target network parameters according to the following formula:

[0168]

[0169] in, represents the updated parameters of the intelligent connected car critic target network i, τ represents the soft update coefficient, θ i CAV represents the parameters of the intelligent connected car critic network i, represents the updated intelligent connected vehicle actor target network parameters, φ v Represents the network parameters of the intelligent connected vehicle actor.

[0170] S9, determining whether the traffic light agent has interacted with the environment at a maximum number of times; if so, executing step S10; otherwise, returning to step S3;

[0171] S10, determine whether the maximum number of iterations has been reached; if so, the process ends; otherwise, return to step S2.

[0172] The embodiment of the present invention uses two types of intelligent agents to collaboratively control traffic lights and CAVs. The two types of intelligent agents interact and cooperate with each other to improve the overall traffic efficiency in a mixed traffic flow environment. The two types of intelligent agents effectively communicate information and collaborate on tasks through a hierarchical cooperation mechanism to further improve the traffic efficiency of intersections and the safety and comfort of CAV driving. With the help of a heterogeneous multi-agent learning framework, effective collaborative learning of heterogeneous multi-tasks is carried out to solve the problem of unstable training and improve the overall training effect.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0177] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A traffic light and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning, characterized in that: The following steps are involved: S1. Create traffic light agents and intelligent connected vehicle agents, initialize the critic network, actor network and corresponding target network; S2, initializing the traffic environment, road network environment and traffic flow conditions; S3, obtain the observation value of the interaction between the traffic light agent and the environment, select the action and calculate the reward value according to the observation value, and store the action trajectory into the traffic light experience replay pool; S4, determine whether the actor network of the traffic light agent starts to update; if so, batch sample action trajectories from the traffic light experience playback pool, calculate the critic network loss function, and update the critic network parameters; otherwise, execute step S6; S5, determine whether the target network of the traffic light agent starts to be updated; if so, randomly arrange the actor update order and update them in sequence, and update the target network of the traffic light agent according to the updated actors; otherwise, execute step S6; S6. Calculate the lane-level target according to the action of the traffic light agent, obtain the observation value of the intelligent networked vehicle agent, select the action according to the observation value and calculate the reward value, and store the action trajectory into the intelligent networked vehicle experience replay pool; S7, determining whether the actor network of the intelligent connected vehicle agent has started to be updated; if so, batch sampling action trajectories from the intelligent connected vehicle experience replay pool, calculating the critic network loss function, and updating the critic network parameters; Otherwise, execute step S9; S8, determining whether the target network of the intelligent connected vehicle agent begins to be updated; if so, calculating the actor network objective function of the intelligent connected vehicle agent, updating the actor network parameters of the intelligent connected vehicle agent, and updating the target network of the intelligent connected vehicle agent according to the updated actor; Otherwise, execute step S9; S9, determining whether the traffic light agent has interacted with the environment to a maximum number of times; If yes, execute step S10; otherwise return to step S3; S10, determine whether the maximum number of iterations has been reached; if so, the process ends; Otherwise return to step S2.

2. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: When creating a traffic light agent, the action space is constructed with phase selection task and phase timing task respectively; the phase selection task consists of a discrete action Indicates that p i Indicates four allowed traffic phases; the phase timing task consists of a continuous action Indicates that g min and g max Respectively represent the minimum and maximum green light time allowed for a single phase; According to the information of each lane and the historical traffic road information, the observation value of the state of the traffic light agent at time t is defined as: Among them, p represents the one-hot encoding of the current phase, q l and n l They represent the number of queued vehicles and the total number of vehicles on lane l, L in Indicates all the incoming lanes at the intersection; The intersection pressure is calculated based on the difference between the total number of vehicles entering and leaving the lane, and the reward function of the traffic light agent is established as: Among them, c l represents the capacity of lane l, n l represents the total number of vehicles on lane l, L out Indicates all exit lanes at the intersection.

3. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: When creating an intelligent connected vehicle agent, define the intelligent connected vehicle agent action as and They represent the minimum and maximum acceleration allowed for the vehicle respectively; According to the driving information of the intelligent connected vehicle itself, the information of the vehicle ahead and the traffic signal information, the observation value of the intelligent connected vehicle agent at time t is defined as: Among them, x t represents the distance from the vehicle to the stop line of the intersection, v t and ρ t represents the speed and acceleration of the vehicle, λ t and μ t Respectively represent the distance and speed relative to the front vehicle, φ t ∈{0,1} indicates whether the traffic light allows the current vehicle to pass, η t Indicates the remaining time of the current phase; Reward ψ based on safety t , comfort bonus t , efficiency reward ν t and bad behavior punishment P t , the reward function of the intelligent connected vehicle agent is established as: Among them, ω1, ω2, ω3 represent weight coefficients; Safety Reward ψ t Specifically: Among them, TTC represents the time to collision; Comfort bonus t Specifically: Among them, ρ max Indicates the maximum acceleration of the vehicle; Efficiency Reward ν t Defined as: Among them, v max Indicates the maximum speed of the vehicle; Bad behavior punishment t Defined as: Among them, v min represents the minimum speed of the vehicle, Indicates the allowed acceleration of the vehicle.

4. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: Traffic light agents and intelligent connected vehicle agents adopt a hierarchical cooperation mechanism. The traffic light manager agent sets the expected goal for each lane and controls the expected state that the intelligent connected vehicle executor agent on each lane needs to achieve in order to communicate information and cooperate on tasks. According to the state observation value of the traffic light, the current lane state and the lane index value, the observation value of the state of the traffic light manager agent at time t is defined as: in, represents the state observation of the traffic light, Indicates the state of lane l, where l represents the lane index value; According to the negative value of the average number of vehicles queued per lane per second, the reward function of the traffic light manager agent is established as: Among them, T represents the interval between two decisions of the manager, L in represents all the entry lanes at the intersection, q l represents the number of queued vehicles on lane l, k represents the sum of the total number of queued vehicles per second between two decisions; According to the state observation value of the intelligent network vehicle agent, the expected speed of the current state and the expected speed of the next time period, the observation value of the state of the intelligent network vehicle executor agent at time t is defined as: in, represents the state observation value of the intelligent connected vehicle agent, Indicates the expected speed of the current state, represents the expected speed in the next time period, Δt represents the minimum control step of the intelligent connected vehicle, and x t represents the distance from the vehicle to the stop line of the intersection, v t Indicates the speed of the vehicle; According to the reward of the intelligent connected vehicle agent and the intrinsic reward of the target action, the reward function of the intelligent connected vehicle executor agent is established as: Among them, α represents the intrinsic reward weight factor, κ represents the normalization factor of the remaining green light countdown, represents the reward of the intelligent connected car agent, represents the intrinsic reward of the target action.

5. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: When updating the critic network parameters, the update is performed by minimizing the loss of the critic network, which is expressed as: Among them, E represents the expectation, s represents the current state, a represents the action, r represents the reward, s' represents the next state, and B represents the small batch learning samples extracted from the experience replay cache. represents the Q value of taking action a in state s, θ i represents the parameters of critic network i, and y represents the target value.

6. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1, characterized in that: Randomly arrange the update order of the actors and update them one by one as follows: Among them, φ i represents the parameters of actor network i, represents the updated parameters of the actor network i, B represents the number of mini-batch learning samples, represents the action-value function fitted by the traffic light critic network, s t represents the global state at time t, Indicates that the current actor strategy is not updated according to its observation The set of actions taken, Indicates the current actor strategy to be updated according to its observation The set of actions taken.

7. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: The target network for updating the traffic light agent based on the updated actor is: in, represents the parameters of the traffic light critic target network i, τ represents the soft update coefficient, represents the parameters of the traffic light critic network i, represents the parameters of the traffic light actor target network j, φ j represents the parameters of the traffic light actor network j, p represents the phase selection task, t represents the phase timing task, and M represents the goal setting task.

8. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1, characterized in that: The specific parameters of the actor network for updating the intelligent connected vehicle agent are: in, represents the gradient of the policy optimization objective function with respect to the parameter φ, E s represents the expected value on all states s, represents the gradient of the Q function with respect to action a, π φ (s) represents the action taken by the strategy in state s, represents the gradient of the policy with respect to the parameter φ.

9. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1, characterized in that: The target network for updating the intelligent connected vehicle agent according to the updated actors is as follows: in, represents the parameters of the target network i of the intelligent connected vehicle critic, τ represents the soft update coefficient, represents the parameters of the intelligent connected car critic network i, represents the target network parameters of the intelligent connected vehicle actor, φ v Represents the network parameters of the intelligent connected vehicle actor.

Images