Traffic signal and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning

Abstract

The present application relates to the technical field of traffic signal and vehicle cooperative control, and discloses a traffic signal and vehicle cooperative control method based on heterogeneous layered reinforcement learning. The present application uses two types of agents to cooperatively control traffic signals and CAVs, and the two types of agents interact with each other to improve the overall traffic efficiency in a mixed traffic flow environment. The two types of agents effectively communicate information and cooperate in tasks through a layered cooperation mechanism, further improving the traffic efficiency at intersections and the safety and comfort of CAV driving. With the help of a heterogeneous multi-agent learning framework, heterogeneous multi-task learning is effectively cooperated, the problem of unstable training is solved, and the overall training effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of traffic signal and vehicle cooperative control, and particularly relates to a traffic signal and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning. BACKGROUND

[0002] In recent years, with the progress of communication and automatic driving technologies, more and more connected autonomous vehicles (CAVs) and intelligent roadside units are introduced, enhancing real-time perception and sharing of road information, and promoting adaptive traffic signal control and vehicle trajectory control. However, existing technologies usually treat traffic signal control and CAV control as two independent fields, using reinforcement learning methods to independently control traffic signals and vehicles. Although better control effects are achieved compared to traditional methods, the information sharing and interaction between vehicles and traffic signals are not considered, which limits the overall efficiency of the traffic system. Although some technologies attempt to use reinforcement learning to jointly control traffic signals and vehicles, these methods often only share information between agents and do not establish mechanisms to ensure mutual understanding of shared information, without clear and reasonable cooperation mechanisms, leading to potential inefficiency and lack of explainability. At the same time, these methods usually assume complete controllability of the leading vehicle, ignoring the actual mixed traffic scenario.

[0003] Patent CN118247986A discloses a vehicle cooperative control method for single signal intersection under mixed traffic flow. Traffic flow state information is collected by information collection equipment, and the optimal vehicle speed of the mixed traffic queue is calculated based on the signal timing and traffic state information, and then the target vehicle is controlled to complete the corresponding driving operation. Patent CN118629233A discloses a traffic light and intelligent connected vehicle cooperative control method in intersection scenarios. The traffic light and CAV are modeled as two types of agents, which interact with the environment in real time, share state information, and train a deep reinforcement learning model. The trained model is used for cooperative control of vehicles and traffic lights.

[0004] The above existing technologies mainly have the following problems:

[0005] 1) Most existing technologies independently control traffic signals and CAVs, ignoring the interaction between traffic lights and vehicles, which limits the overall efficiency of the traffic system. Patent CN118247986A combines traffic information and signal timing information to complete optimal control of vehicles, but does not consider the impact of vehicle speed changes on traffic lights and does not consider the overall efficiency of the traffic system.

[0006] 2) Current existing technologies for coordinated control of traffic lights and vehicles simply share information, without a clear collaborative mechanism or a mechanism for mutual understanding of shared information. Patent CN118629233A, while using reinforcement learning for coordinated control of traffic lights and vehicles, only shares state information between the two intelligent agents, the vehicle and the traffic light, without a clear collaborative optimization goal, reducing the optimality of coordinated control. Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for collaborative control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning. By modeling traffic lights and CAVs as two types of intelligent agents and designing a hierarchical cooperation mechanism, efficient communication and understanding of information between CAVs and traffic signals at different time scales are ensured. At the same time, a heterogeneous multi-agent mirror learning algorithm is used to enable heterogeneous intelligent agents to effectively collaborate in a multi-task environment, thereby improving the overall efficiency of the traffic system and reducing the instability of the training process.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] A method for cooperative control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning, characterized by comprising the following steps:

[0010] S1. Create a traffic light agent and an intelligent connected vehicle agent, initialize the critic network, actor network, and corresponding target network;

[0011] S2, initializing the traffic environment, road network environment and traffic flow conditions;

[0012] S3. Obtain observations of the traffic light agent's interaction with the environment, select actions based on the observations, calculate rewards, and store the action trajectory in the traffic light experience replay pool.

[0013] S4. Determine whether the actor network of the traffic light agent has started to update; if so, batch sample action trajectories from the traffic light experience replay pool, calculate the critic network loss function, and update the critic network parameters; otherwise, execute step S6;

[0014] S5. Determine whether the target network of the traffic light agent has started to update; if so, randomly arrange the update order of the actors and update them in sequence, and update the target network of the traffic light agent based on the updated actors; otherwise, execute step S6;

[0015] S6, calculate the lane-level target according to the action of the traffic signal agent, obtain the observation value of the intelligent connected vehicle agent, select an action according to the observation value and calculate a reward value, and store the action trajectory into the intelligent connected vehicle experience replay pool;

[0016] S7, determine whether the actor network of the intelligent connected vehicle agent starts to update; if yes, batch sample the action trajectory from the intelligent connected vehicle experience replay pool, calculate the critic network loss function, and update the critic network parameters; otherwise, perform step S9;

[0017] S8, determine whether the target network of the intelligent connected vehicle agent starts to update; if yes, calculate the actor network target function of the intelligent connected vehicle agent, update the actor network parameters of the intelligent connected vehicle agent, and update the target network of the intelligent connected vehicle agent according to the updated actor; otherwise, perform step S9;

[0018] S9, determine whether the traffic signal agent and the environment interact reaches the maximum interaction times; if yes, perform step S10; otherwise, return to step S3;

[0019] S10, determine whether the maximum iteration times is reached; if yes, the process ends; otherwise, return to step S2.

[0020] As preferred, when creating the traffic signal agent, an action space is constructed respectively for the phase selection task and the phase timing task; wherein the phase selection task is represented by a discrete action p i represents four allowed traffic phases; the phase timing task is represented by a continuous action g min and g max respectively represent the minimum and maximum green light time allowed for a single phase;

[0021] According to the information of each lane and the historical traffic road information, the observation value of the signal agent at time t is defined as:

[0022]

[0023] wherein p represents one-hot encoding of the current phase, q l and n l respectively represent the number of queued vehicles and the total number of vehicles on lane l, L in represents all the incoming lanes of the intersection;

[0024] According to the difference between the total number of vehicles on the incoming lane and the outgoing lane, the intersection pressure is calculated, and the reward function of the traffic signal agent is established as:

[0025]

[0026] wherein c l represents the capacity of the lane l, n l represents the total number of vehicles on the lane l, L out represents all the off-lane of the intersection.

[0027] As preferred, when creating the intelligent connected vehicle agent, the intelligent connected vehicle agent action is defined as and respectively represent the minimum acceleration and the maximum acceleration allowed by the vehicle;

[0028] According to the intelligent connected vehicle's own driving information, the front vehicle information and the traffic signal information, the observation value of the intelligent connected vehicle agent at time t is defined as:

[0029]

[0030] wherein x t represents the distance of the vehicle to the intersection stop line, v t and p t represent the speed and acceleration of the vehicle, l t and m t respectively represent the distance and speed relative to the front vehicle, f t ∈{0,1} represents whether the signal light allows the current vehicle to pass, h t represents the remaining time of the current phase;

[0031] According to the safety reward p t , the comfort reward x t , the efficiency reward v t and the bad behavior punishment P t , the reward function of the intelligent connected vehicle agent is established as:

[0032] r t CAV = w1 p t + w2 x t + w3 v t - P t

[0033] wherein w1, w2, w3 represent the weight coefficients;

[0034] The safety reward p t is specifically:

[0035]

[0036] wherein TTC represents the time to collision;

[0037] The comfort reward xt Specifically,

[0038]

[0039] wherein, p max represents the maximum acceleration of the vehicle;

[0040] Efficiency reward v t is defined as:

[0041]

[0042] wherein, v max represents the maximum speed of the vehicle;

[0043] Bad behavior penalty P t is defined as:

[0044]

[0045]

[0046] wherein, v min represents the minimum speed of the vehicle, represents the acceleration allowed by the vehicle.

[0047] As a preferred, the traffic signal light agent and the intelligent connected vehicle agent adopt a hierarchical cooperation mechanism, and the traffic signal light manager agent sets the expected target for each lane, controls the expected state that the intelligent connected vehicle executor agent on each lane needs to reach, so as to communicate information and cooperate tasks;

[0048] According to the state observation value of the traffic signal light, the current lane state and the lane index value, the observation value of the state of the traffic signal light manager agent at time t is defined as:

[0049]

[0050] wherein, represents the state observation value of the traffic signal light, represents the state of the lane l, and l represents the lane index value;

[0051] According to the negative value of the average number of vehicles per lane per second, the reward function of the traffic signal light manager agent is established as:

[0052]

[0053] wherein, T represents the interval between two decisions of the manager, L in represents all the incoming lanes of the intersection, q lrepresents the number of queued vehicles on the lane l, k represents the total number of queued vehicles per second between two decisions summed up by the order number;

[0054] According to the state observation value of the intelligent connected vehicle agent, the expected speed of the current state and the expected speed of the next time period, the observation value of the state of the intelligent connected vehicle executor agent at time t is defined as:

[0055]

[0056] wherein, represents the state observation value of the intelligent connected vehicle agent, represents the expected speed of the current state, represents the expected speed of the next time period, and Δt represents the minimum control step of the intelligent connected vehicle, t represents the distance of the vehicle to the intersection stop line, t represents the speed of the vehicle;

[0057] According to the reward of the intelligent connected vehicle agent and the intrinsic reward of the target action, the reward function of the intelligent connected vehicle executor agent is established as:

[0058]

[0059] wherein, α represents the intrinsic reward weight factor, κ represents the normalization factor of the remaining green light countdown, and r t CAV represents the reward of the intelligent connected vehicle agent, t int represents the intrinsic reward of the target action.

[0060] As a preferred, when updating the reviewer network parameters, the reviewer network is updated by minimizing the loss of the reviewer network, which is represented as:

[0061]

[0062] wherein, E represents the expectation, s represents the current state, a represents the action, r represents the reward, s’ represents the next state, B represents a small batch of learning samples extracted from the experience replay buffer, and represents the Q value of taking action a under state s, θ i represents the parameters of the reviewer network i, and y represents the target value.

[0063] As a preferred, the actioner update order is randomly arranged for sequential update, which is specifically:

[0064]

[0065] wherein, φ i represents the actioner network parameters, denote the parameters of the updated actor network i, B denotes the number of mini-batch learning samples, denote the action value function fitted by the traffic light critic network, s t denote the global state at time t, denote the current actor policy not updated, according to the observation adopted action set, denote the current actor policy to be updated, according to the observation adopted action set.

[0066] As a preferred, the target network of the traffic light intelligent agent is updated according to the updated actor, specifically:

[0067]

[0068] wherein, denote the parameters of the updated traffic light critic target network i, τ denotes a soft update coefficient, θ i TS denote the parameters of the traffic light critic network i, denote the parameters of the updated traffic light actor target network j, φ j denote the parameters of the traffic light actor network j, p denotes the phase selection task, t denotes the phase timing task, and M denotes the target setting task.

[0069] As a preferred, the actor network parameters of the intelligent connected vehicle intelligent agent are updated, specifically:

[0070]

[0071] wherein, denote the gradient of the policy optimization objective function with respect to the parameter φ, E s denote the expected value over all states s, denote the gradient of the Q function with respect to the action a, π φ (s) denotes the action taken by the policy in state s, denote the gradient of the policy with respect to the parameter φ.

[0072] As a preferred, the target network of the intelligent connected vehicle intelligent agent is updated according to the updated actor, specifically:

[0073]

[0074] wherein, denote the parameters of the updated intelligent connected vehicle critic target network i, τ denotes a soft update coefficient, θ i CAV denote the parameters of the intelligent connected vehicle critic network i, φ represents an updated intelligent networked vehicle actor target network parameter v φ represents an intelligent networked vehicle actor network parameter.

[0075] The present application has the following beneficial effects:

[0076] The present application uses two types of agents to cooperatively control traffic lights and CAVs, and the two types of agents interact with each other to improve the overall traffic efficiency in a mixed traffic flow environment; the two types of agents effectively communicate information and cooperate in tasks through a hierarchical cooperation mechanism, further improving the traffic efficiency at the intersection and the safety and comfort of CAV driving; with the help of a heterogeneous multi-agent learning framework, heterogeneous multi-tasks are effectively learned, the problem of unstable training is solved, and the overall training effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The present application is a flowchart;

[0078] Figure 2 The present application is an architecture diagram;

[0079] Figure 3 The present application is a schematic diagram of a multi-heterogeneous agent learning framework based on double-delay deep deterministic policy gradient. DETAILED DESCRIPTION

[0080] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application as defined and determined by the appended claims, and all inventions utilizing the concept of the present application are within the scope of protection.

[0081] The application designs a traffic signal and intelligent networked vehicle cooperative control method based on heterogeneous layered reinforcement learning, which designs two types of intelligent agents of traffic signal and intelligent networked vehicle, which are trained through interaction with the environment, and finally achieve the effect of decision-making according to the current traffic state. The traffic signal agent is composed of three heterogeneous agents of phase agent, timing agent and manager agent, the phase agent selects the phase of the next decision step of the signal light, the timing agent controls the duration of the phase, and the manager agent constitutes a layered cooperation mechanism with the vehicle agent, wherein the manager agent of the signal light sets an expected target, as an executor agent, the vehicle agent considers the target during training, and comprehensively considers the target for decision-making during application. In addition, in order to ensure the effective cooperation of the three agents of the signal light, the algorithm uses a heterogeneous agent mirror learning framework, which shares the critic network, randomly selects the action network update order, and the latter action considers the former action, which promotes the cooperative training of the three types of agents and ensures the stability of the overall training.

[0082] With reference to Figure 1 and Figure 2 , the embodiment of the application provides a traffic signal and vehicle cooperative control method based on heterogeneous layered reinforcement learning, comprising the following steps:

[0083] S1, creating traffic signal agent and intelligent networked vehicle agent, initializing critic network, action network and corresponding target network;

[0084] In an optional embodiment of the application, step S1 first creates two types of intelligent agents of traffic signal agent and intelligent networked vehicle agent, initializes the parameters of each network in the two types of intelligent agents, including the critic network and the centralized critic network of the traffic signal agent phase action network timing action network management action network and the corresponding target network of the traffic signal agent; the critic network and the centralized critic network of the intelligent networked vehicle agent and the corresponding target network thereof. The action network is used to select the appropriate action according to the current state, the critic network is used to evaluate the goodness of the action in the current state, and the target network is an auxiliary network used to improve the training stability.

[0085] In this embodiment, the traffic signal at each time is controlled by the traffic signal agent, and the position of each intelligent networked vehicle in the driving lane and the distance from the front vehicle are controlled by the intelligent networked vehicle agent.

[0086] The traffic signal agent makes decisions at the end of each green phase, which has two parts: one is to select the next phase, and the other is to determine the duration of the selected phase. These two tasks have different action spaces but share the same state space and reward function, which are designed as follows:

[0087] (a) Action space design

[0088] The phase selection task has a discrete action space , where p i represents the four allowed traffic phases; the phase timing task has a continuous action space , where g min represents the green time of the selected phase. max and g min represent the minimum and maximum green time for a single phase, respectively, limiting the phase time within a reasonable range.

[0089] (b) State space design

[0090] Considering the information of each lane and historical traffic road information, the observation value of the signal agent at time t is defined as:

[0091]

[0092] where p represents the one-hot encoding of the current phase, q l and n l represent the number of queued vehicles and the total number of vehicles on lane l, respectively, both normalized by the road capacity, L in represents all the incoming lanes of the intersection.

[0093] (c) Reward function design

[0094] The reward function of the signal is mainly designed to consider the intersection pressure, which is defined as the difference between the total number of incoming and outgoing vehicles. Reducing the pressure is equivalent to maximizing the capacity, so the reward function is designed as:

[0095]

[0096] where c l represents the capacity of lane l, and n l represents the total number of vehicles on lane l.

[0097] The intelligent vehicle agent controls the longitudinal acceleration of the intelligent vehicle to cooperate with the signal for efficient and comfortable driving, which is designed as follows:

[0098] (a) Action space design

[0099] The agent action is designed as denotes the longitudinal acceleration of the CAV at time t, where and denote the minimum and maximum values allowed by the vehicle, respectively, to ensure safety and comfort during control.

[0100] (b) State space design

[0101] Considering the information of the intelligent connected vehicle itself, the front vehicle and the traffic signal, the observation of the intelligent connected vehicle agent at time t is defined as:

[0102]

[0103] where x t denotes the distance from the vehicle to the stop line at the intersection, v t and ρ t denote the speed and acceleration of the vehicle, λ t and μ t denote the distance and speed relative to the front vehicle, φ t ∈{0,1} denotes whether the signal light allows the current vehicle to pass, and η t denotes the remaining time of the current phase.

[0104] (c) Reward function design

[0105] The reward function of the intelligent connected vehicle agent is mainly composed of safety reward ψ t , comfort reward ξ t , efficiency reward v t and bad behavior penalty P t , and is expressed as:

[0106] r t CAV = ω1ψ t + ω2ξ t + ω3v t - P t

[0107] where ω1, ω2, ω3 are weight coefficients.

[0108] The safety reward ψ t is related to the Time-to-Collision (TCC), and a TCC less than four seconds is considered unsafe:

[0109]

[0110] The comfort reward ξ t is defined as the acceleration difference between time steps:

[0111]

[0112] Efficiency reward v t is defined as normalized speed:

[0113]

[0114] Bad behavior penalty P t consists of lazy behavior, inappropriate acceleration behavior and action mismatch behavior:

[0115]

[0116] where the lazy behavior penalty P t loaf is mainly for the case that the current speed of the CAV cannot pass the intersection within the remaining green time:

[0117]

[0118] The acceleration at the red light corresponds to the inappropriate acceleration penalty:

[0119]

[0120] The action mismatch penalty is used to correct and narrow the gap between the expected acceleration and the actual acceleration:

[0121]

[0122] where v min represents the minimum speed of the vehicle, represents the allowed acceleration of the vehicle.

[0123] The embodiment adopts a hierarchical cooperation mechanism of traffic signal light agent and intelligent connected vehicle agent, mainly composed of traffic signal light manager agent and intelligent connected vehicle executor agent, wherein the manager agent sets the target and specifies the expected state that the intelligent connected vehicle executor agent needs to achieve, to realize information communication and task cooperation.

[0124] The target in the hierarchical cooperation mechanism is used to deliver the manager's expectation to the executor, which plays a role in connecting the manager agent and the executor agent. The method of the present application is designed as a lane-level target, i.e. the manager sets a target for each lane to control all intelligent connected vehicles on the lane. The lane l target is defined as a function v l (x), which represents the relationship between the vehicle speed v and the distance x to the stop line. This function can be converted to and from the target state vector generated by the manager by sampling and interpolation method.

[0125] For each lane, the action of the manager is is the target state of the lane The lane-level target v can be obtained by interpolation method l (x), and then guide the decision-making of intelligent connected vehicles.

[0126] Unlike the phase and timing of the signal light agent, the traffic signal light manager agent uses more vehicle information, so the signal light state observation value is increased by the current lane state and the one-hot encoding of the lane index value l:

[0127]

[0128] The reward function of the traffic signal light manager agent is defined as the negative value of the average number of vehicles per lane per second, to promote cooperation between the signal light and the vehicle, and to reduce the number of stops and fuel consumption:

[0129]

[0130] The intelligent connected vehicle executor agent, on the basis of ensuring safety and driving efficiency, realizes the expected target of the manager for the lane through .

[0131] The state observation value of the intelligent connected vehicle executor agent is increased by the expected speed of the current state and the expected speed of the next time period to help the executor more efficiently understand the task:

[0132]

[0133] In order to encourage the consistency of the target actions of the manager and the executor, the executor agent incorporates the intrinsic reward function , which is the negative value of the L2 norm of the difference between the current lane state and the target lane state, and the reward function of the executor agent is defined as:

[0134]

[0135] where α is the intrinsic reward weight factor, and κ is the normalized factor of the remaining green light countdown, that is, as the remaining green light time decreases, the executor pays more and more attention to the intrinsic reward.

[0136] S2, initialize the traffic environment, road network environment and traffic flow;

[0137] In an optional embodiment of the present application, step S2 creates a traffic simulation environment and initializes, randomly sets the road network environment and traffic flow of the simulation environment, sets the signal light agent to control the traffic signal light in the simulation environment, sets the intelligent network connection car agent to control the intelligent network connection car, starts the simulation environment, and starts interaction with the agent.

[0138] S3, obtaining observation values of the traffic signal light agent interacting with the environment, selecting actions according to the observation values and calculating reward values, and storing an action trajectory into a traffic signal light experience replay pool;

[0139] In an optional embodiment of the present application, step S3 makes a decision at the end of the current green light phase, and the traffic signal light agent obtains observation values and selects actions:

[0140]

[0141] wherein σ TS is a random exploration noise, which ensures the stability of the overall training.

[0142] After selecting the action, the agent executes the action, and calculates the reward r t TS and r t M stores the action trajectory into the signal light experience replay pool B TS .

[0143] S4, judging whether the actor network of the traffic signal light agent starts to update; if yes, batch sampling an action trajectory from the traffic signal light experience replay pool, calculating a critic network loss function, and updating critic network parameters; otherwise, executing step S6;

[0144] In an optional embodiment of the present application, after the experience replay pool B TS stores a certain value, the action trajectory B is batch sampled, the critic network parameters are updated by minimizing the loss of the critic network and and the random exploration noise σ TS is reduced.

[0145] In this embodiment, the traffic signal light contains three agents to complete three tasks of signal light phase selection, phase timing and target setting for the executor respectively, and these tasks exist certain cooperative game, therefore, the present application designs a multi-heterogeneous agent learning framework based on double-delay deep deterministic policy gradient, referring to Figure 3The learning framework uses a decentralized execution, centralized training method, different executor networks share a centralized critic network to evaluate the goodness of the actions made by the actors, two critic networks are used here and Two estimated minimum values are used to reduce bias, and the critic network is updated by minimizing the loss:

[0146]

[0147] Where E denotes expectation, s denotes the current state, a denotes action, r denotes reward, s' denotes the next state, B denotes a small batch of learning samples drawn from the experience replay buffer, represents the Q value of taking action a in state s, θ i represents the parameters of critic network i, and y represents the target value.

[0148] Where the target value y is calculated as:

[0149]

[0150] Where, and are the parameters of the target network, and ò is the clipping noise added for target policy smoothing.

[0151] Unlike the traditional double-delay deep deterministic policy gradient algorithm, the centralized critic allows the actor to consider the information in the entire training process, and the update order of the actor is rearranged at each network parameter update, and the Q function of each actor considers the latest policy of the previously updated actor in the order of sequential update, and the actor network is updated using the deterministic policy gradient here:

[0152]

[0153] Where, represents the gradient of the policy optimization objective function with respect to the parameter φ, E s represents the expected value over all states s, represents the gradient of the Q function with respect to action a, π φ (s) represents the action taken by the policy in state s, represents the gradient of the policy with respect to the parameter φ.

[0154] S5, judge whether the target network of the traffic signal light agent starts to update; if yes, randomly arrange the update order of the actors for sequential update, and update the target network of the traffic signal light agent according to the updated actors; otherwise, execute step S6;

[0155] In an optional embodiment of the present application, step S5 is an interval fixed update step, updating the signal light actor network and the target network, first randomly arranging the update order of the three types of agents, and for the randomly arranged agent arrangement {p, t, M}, updating in turn according to the following formula:

[0156]

[0157] Then update the signal light agent target network according to the following formula:

[0158]

[0159] wherein, represents the parameter of the updated traffic signal light critic target network i, τ represents a soft update coefficient, θ i TS represents the parameter of the traffic signal light critic network i, represents the parameter of the updated traffic signal light actor target network j, φ j represents the parameter of the traffic signal light actor network j, p represents the phase selection task, t represents the phase timing task, and M represents the target setting task.

[0160] S6, calculate the lane-level target according to the action of the traffic signal light agent, obtain the observation value of the intelligent connected vehicle agent, select an action according to the observation value and calculate the reward value, and store the one-time action trajectory into the intelligent connected vehicle experience replay pool;

[0161] In an optional embodiment of the present application, step S6 calculates the lane-level target according to the action of the signal light management actor obtains the lane-level target state interpolates to obtain the CAV agent obtains the state observation value calculates to obtain and r t int , combined with random exploration noise σ CAV select an action:

[0162]

[0163] The CAV agent executes the action and calculates the reward r t W , and stores the one-time action trajectory into the CAV experience replay pool B CAV .

[0164] S7, determining whether the actor network of the intelligent connected vehicle agent starts to update; if yes, batch sampling action trajectories from the experience replay pool of the intelligent connected vehicle, calculating the critic network loss function, and updating the critic network parameters; otherwise, performing step S9;

[0165] In an optional embodiment of the present application, step S7 is performed at a fixed update interval CAV After storing a certain value, the batch sampling action trajectory B is used to update the critic network parameters by minimizing the loss of the critic network and and reduce the random exploration noise σ CAV .

[0166] S8, determining whether the target network of the intelligent connected vehicle agent starts to update; if yes, calculating the actor network target function of the intelligent connected vehicle agent, updating the actor network parameters of the intelligent connected vehicle agent, and updating the target network of the intelligent connected vehicle agent according to the updated actor; otherwise, performing step S9;

[0167] In an optional embodiment of the present application, step S8 is performed at a fixed update interval V and the target network parameters are updated according to the following formula:

[0168]

[0169] wherein, represents the updated parameters of the intelligent connected vehicle critic target network i, τ represents a soft update coefficient, and θ i CAV represents the parameters of the intelligent connected vehicle critic network i, represents the updated intelligent connected vehicle actor target network parameters, and φ v represents the parameters of the intelligent connected vehicle actor network.

[0170] S9, determining whether the maximum number of interactions between the traffic signal agent and the environment is reached; if yes, performing step S10; otherwise, returning to step S3;

[0171] S10, determining whether the maximum number of iterations is reached; if yes, the process is ended; otherwise, returning to step S2.

[0172] The embodiment of the present application uses two types of agents to cooperatively control traffic lights and CAVs, the two types of agents interact with each other to improve the overall traffic efficiency in a mixed traffic flow environment; the two types of agents effectively communicate information and cooperate tasks through a hierarchical cooperation mechanism, further improving the traffic efficiency of the intersection and the safety and comfort of CAV driving; with the help of a heterogeneous multi-agent learning framework, heterogeneous multi-task learning is effectively cooperated, the problem of unstable training is solved, and the overall training effect is improved.

[0173] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in one or more flows and / or blocks.

[0174] These computer program instructions can also be stored in a computer-readable memory capable of guiding the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in one or more flows and / or blocks.

[0175] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in one or more flows and / or blocks.

[0176] The principles and implementation modes of the present application are described in the specific embodiments, and the above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as the limitation of the present application.

[0177] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and that the inventive principles are not limited to these particular embodiments. Other variations and modifications can be made to the embodiments without departing from the spirit and scope of the inventive principles.

Claims

1. A traffic light and vehicle cooperative control method based on heterogeneous hierarchical reinforcement learning, characterized in that: The following steps are involved: S1. Create a traffic light agent and an intelligent connected vehicle agent, initialize the critic network, actor network, and corresponding target network. The traffic light agent and the intelligent connected vehicle agent adopt a hierarchical cooperation mechanism. The traffic light manager agent sets the expected goal for each lane and controls the desired state of the intelligent connected vehicle executor agent in each lane to facilitate information communication and task collaboration. According to the state observation value of the traffic light, the current lane state and the lane index value, the traffic light manager agent is defined as The observed value of the state at the moment is: in, represents the state observation value of the traffic light, Indicates lane l status, Indicates the lane index value; According to the negative value of the average number of vehicles queued per lane per second, the reward function of the traffic light manager agent is established as: in, T represents the interval between two decisions made by managers, Indicates all the incoming lanes at the intersection. Indicates lane The number of queued vehicles on k It means summing the total number of vehicles in queue per second between two decisions; According to the state observation value of the intelligent network vehicle agent, the expected speed of the current state and the expected speed of the next time period, the intelligent network vehicle executor agent is defined in The observed value of the state at the moment is: in, represents the state observation value of the intelligent connected vehicle agent, Indicates the expected speed of the current state, Indicates the expected speed in the next time period, represents the minimum control step of the intelligent connected vehicle, Indicates the distance from the vehicle to the stop line of the intersection. Indicates the speed of the vehicle; According to the reward of the intelligent connected vehicle agent and the intrinsic reward of the target action, the reward function of the intelligent connected vehicle actor agent is established as: in, represents the intrinsic reward weight factor, Indicates the normalization factor of the remaining green light countdown, represents the reward of the intelligent connected vehicle agent, represents the intrinsic reward of the target action; S2, initializing the traffic environment, road network environment and traffic flow conditions; S3. Obtain observations of the traffic light agent's interaction with the environment, select actions based on the observations, calculate rewards, and store the action trajectory in the traffic light experience replay pool. S4. Determine whether the actor network of the traffic light agent has started to update; if so, batch sample action trajectories from the traffic light experience replay pool, calculate the critic network loss function, and update the critic network parameters; otherwise, execute step S6; S5. Determine whether the target network of the traffic light agent has started to update; if so, randomly arrange the update order of the actors and update them in sequence, and update the target network of the traffic light agent based on the updated actors; otherwise, execute step S6; S6. Calculate lane-level targets based on the actions of the traffic light agent, obtain observations from the intelligent connected vehicle agent, select actions based on the observations, calculate rewards, and store the action trajectory in the intelligent connected vehicle experience replay pool. S7. Determine whether the actor network of the intelligent connected vehicle agent has started to update; if so, batch sample action trajectories from the intelligent connected vehicle experience replay pool, calculate the commentator network loss function, and update the commentator network parameters; otherwise, execute step S9; S8. Determine whether the target network of the intelligent connected vehicle agent has begun to be updated; if so, calculate the actor network objective function of the intelligent connected vehicle agent, update the actor network parameters of the intelligent connected vehicle agent, and update the target network of the intelligent connected vehicle agent based on the updated actor; otherwise, execute step S9; S9, determining whether the traffic light agent has interacted with the environment for a maximum number of times; if so, executing step S10; otherwise, returning to step S3; S10. Determine whether the maximum number of iterations has been reached; if so, the process ends; otherwise, return to step S2.

2. The method for cooperative control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: When creating a traffic light agent, the action space is constructed with phase selection task and phase timing task respectively; the phase selection task consists of a discrete action express, Represents four allowed traffic phases; the phase timing task consists of a continuous action express, and Respectively represent the minimum and maximum green light time allowed for a single phase; According to the information of each lane and historical traffic road information, the traffic light agent is defined The observed value of the state at the moment is: in, Represents the one-hot encoding of the current phase, and Represents lanes The number of queued vehicles and the total number of vehicles on Indicates all incoming lanes at the intersection; The intersection pressure is calculated based on the difference between the total number of vehicles entering and leaving the lane, and the reward function of the traffic light agent is established as: in, Indicates lane The traffic capacity, Indicates lane The total number of vehicles on Indicates all exit lanes at the intersection.

3. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: When creating an intelligent connected vehicle agent, define the intelligent connected vehicle agent action as , and Respectively represent the minimum acceleration and maximum acceleration allowed for the vehicle; According to the driving information of the intelligent connected vehicle itself, the information of the vehicle ahead and the traffic signal information, the intelligent connected vehicle agent is defined. The observed value of the state at the moment is: in, Indicates the distance from the vehicle to the stop line of the intersection. and represents the speed and acceleration of the vehicle, and Represent the distance and speed relative to the preceding vehicle, Indicates whether the traffic light allows the current vehicle to pass. Indicates the remaining time of the current phase; Reward based on safety , comfort bonus , efficiency rewards and punishment for bad behavior , the reward function for establishing the intelligent connected vehicle agent is: in, , , represents the weight coefficient; Security Rewards Specifically: Among them, TTC represents the time to collision; Comfort Bonus Specifically: in, Indicates the maximum acceleration of the vehicle; Efficiency Rewards Defined as: in, Indicates the maximum speed of the vehicle; Punishment for bad behavior Defined as: in, Indicates the minimum speed of the vehicle, Indicates the allowed acceleration of the vehicle.

4. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1, characterized in that: When updating the critic network parameters, the update is performed by minimizing the loss of the critic network, which is expressed as: in, Indicates expectation, s indicates the current state, a indicates the action, r indicates the reward, s' indicates the next state, and B indicates a small batch of learning samples extracted from the experience replay cache. represents the Q value of taking action a in state s, Representing the reviewer network i , y represents the target value.

5. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: Randomly arrange the update order of the actors and update them one by one as follows: in, Representing an actor network i Parameters, Represents the updated actor network i Parameters, B represents the number of small batch learning samples, represents the action-value function fitted by the traffic light critic network, represents the global state at time t, Indicates that the current actor strategy is not updated according to its observation The set of actions taken, Indicates the current actor strategy to be updated according to its observation The set of actions taken.

6. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: The target network for updating the traffic light agent based on the updated actor is: in, Represents the traffic light critic target network i Parameters, represents the soft update coefficient, Represents a traffic light critic network i Parameters, Representing the traffic light actor-target network j Parameters, Representing a traffic light actor network j , p represents the phase selection task, t represents the phase timing task, and M represents the goal setting task.

7. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: The specific network parameters of the actor to update the intelligent connected vehicle agent are: in, Represents the policy optimization objective function with respect to the parameter The gradient, represents the expected value on all states s, Represents the Q function for action a The gradient, represents the action taken by the strategy in state s, Indicates the policy parameters gradient.

8. The method for coordinated control of traffic lights and vehicles based on heterogeneous hierarchical reinforcement learning according to claim 1 is characterized in that: The target network for updating the intelligent connected vehicle agent based on the updated actors is: in, Represents the target network of intelligent connected car reviewers i Parameters, represents the soft update coefficient, Represents the Intelligent Connected Car Reviewer Network i Parameters, represents the target network parameters of the intelligent connected vehicle actor, Represents the network parameters of the intelligent connected vehicle actor.

Images