Networked hybrid multi-traffic-flow guide control method and device

Abstract

The invention relates to a network connection mixed multi-traffic-flow guiding control method and device, and the method comprises the steps: constructing a multi-traffic-flow guiding control scheme through employing a layering thought, and enabling a road side control center to decide an intersection signal lamp timing scheme according to the states and target directions of all vehicles at an intersection, the guidance information specifically comprises a lane changing instruction and a target speed of each networked autonomous vehicle, and after the networked autonomous vehicle receives the guidance information, based on the guidance information and a vehicle kinematics constraint, the guidance information of the networked autonomous vehicle is calculated, and then the guidance information is sent to the networked autonomous vehicle; and calculating a control instruction of the networked autonomous vehicle. Therefore, the problems that in the prior art, the lane changing requirement and the vehicle kinematics constraint of the networked automatic driving vehicle are not fully considered, the vehicle flow direction is difficult to dispatch and guide effectively, and the requirement of real-time application cannot be met are solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a method and device for guiding and controlling networked hybrid multi-vehicle flows. Background Art

[0002] With the continuous development of Intelligent Transportation Systems (ITS) and Internet of Vehicles (V2X) technologies, the ability to exchange information between vehicles and between vehicles and infrastructure has been significantly enhanced. Through the V2X communication platform, V2X technology enables vehicles to share important data such as road conditions, traffic signal status, and the speed and location of surrounding vehicles in real time, providing unprecedented opportunities for traffic management and control. Especially in mixed traffic environments, including those involving connected autonomous vehicles (CAVs) and traditional human-driven vehicles (HDVs), how to effectively dispatch and guide traffic has become a key issue that needs to be addressed in the field of traffic control.

[0003] Traditional traffic control methods, primarily based on fixed signal cycles and lane assignment strategies, often fail to fully account for the dynamic nature of traffic flow. This is particularly true in complex environments where different types of vehicles, such as CAVs and HDVs, coexist, often leading to traffic congestion, delays, and even safety hazards. Currently, coordinated control of CAVs with signals at intersections in mixed traffic flows, for example, has not yet been fully implemented. In particular, existing technologies for achieving coordinated control of signals and CAVs do not fully consider vehicle kinematic or dynamic constraints when applying control to CAVs.

[0004] In summary, existing technologies do not fully consider the lane-changing needs and vehicle kinematic constraints of CAV vehicles, making it difficult to effectively dispatch and guide vehicle flow and unable to meet the requirements of real-time applications, which urgently needs to be solved. Summary of the Invention

[0005] The present application provides a method and device for controlling connected hybrid multi-vehicle flow guidance to address the problems that the existing technology does not fully consider the lane-changing requirements and vehicle kinematic constraints of connected autonomous driving vehicles, making it difficult to effectively dispatch and guide vehicle flows and unable to meet the requirements of real-time applications.

[0006] The first aspect of the present application provides a method for guiding and controlling a networked hybrid multi-vehicle flow, comprising the following steps: determining a vehicle control area corresponding to a target intersection, and based on a preset deep reinforcement learning control strategy, constructing a signal light agent and at least one networked autonomous driving vehicle agent in the vehicle control area, and establishing a global shared reward function corresponding to the signal light agent and the at least one networked autonomous driving vehicle agent; initializing the local Q-value network, the global Q-value network, and the corresponding target network corresponding to the signal light agent and each networked autonomous driving vehicle agent based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, and performing an environment interaction operation on each agent based on the initialized Q-value network, the global Q-value network, and the target network to update the local Q-value network, the pre-constructed hyperparameter network, and the target network, and generating a target global Q-value corresponding to all agents through the updated local Q-value network, the hyperparameter network, and the target network; determining a control instruction corresponding to each networked autonomous driving vehicle according to the target global Q-value, and controlling each networked autonomous driving vehicle to perform a corresponding driving operation through the control instruction to pass through the target intersection.

[0007] Optionally, in one embodiment of the present application, the vehicle control area corresponding to the target intersection is determined, and based on a preset deep reinforcement learning control strategy, a signal light agent and at least one networked autonomous driving vehicle agent in the vehicle control area are constructed, and a global shared reward function corresponding to the signal light agent and the at least one networked autonomous driving vehicle agent is established, including: dividing the vehicle control area into equally spaced grids to obtain multiple grids, and assigning a value to each grid according to a preset grid assignment strategy to construct a corresponding first state space; determining a corresponding first action space based on a preset signal light four-phase control strategy, and obtaining the number of human-driven vehicles, the number of networked autonomous driving vehicles and the maximum speed limit of the road in the target intersection, so as to determine the corresponding first action space according to the number of human-driven vehicles, the number of networked autonomous driving vehicles and the maximum speed limit of the road. a reward function; constructing the traffic light agent based on the first state space, the first action space and the first reward function; constructing a second state space corresponding to each connected autonomous driving vehicle based on a preset state space expression, and calculating the target speed and target lane of each connected autonomous driving vehicle, so as to determine the second action space corresponding to each connected autonomous driving vehicle through the target speed and the target lane; obtaining the minimum speed limit of the target intersection, and determining a second reward function according to the number of connected autonomous driving vehicles, the minimum speed limit of the road and the maximum speed limit of the road, so as to construct a connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle based on the second state space, the second action space and the second reward function; constructing the global shared reward function based on the first reward function and the second reward function.

[0008] Optionally, in one embodiment of the present application, the local Q-value network, global Q-value network and corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent are initialized based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, and based on the initialized Q-value network, global Q-value network and the target network, each agent is subjected to environmental interaction operations to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, the hyperparameter network and the target network, including: initializing the local Q-value network corresponding to the traffic light agent and each connected autonomous driving vehicle agent to obtain the initialization network parameters corresponding to the local Q-value network; establishing the hybrid network based on the hyperparameter network, and performing a nonlinear combination of the output Q value of the local Q-value network through the hybrid network and the initialization network parameters to establish the global Q-value network; determining the corresponding target hyperparameter network and target local Q-value network based on the hyperparameter network and the local Q-value network, and performing the target hyperparameter network and the target local Q-value network on the local Q-value network; The target hyperparameter network and the target local Q-value network are initialized; a target global Q-value network is established through the initialized target hyperparameter network and the target local Q-value network, and multiple hyperparameters of the target global Q-value network are determined; based on the multiple hyperparameters, the initialized local Q-value network and the hybrid network and a preset ∈-greedy strategy, the random actions corresponding to the signal light agent and each of the connected autonomous driving vehicle agents are determined; the observation states corresponding to the signal light agent and each of the connected autonomous driving vehicle agents are obtained to calculate the target action based on the random action and the observation state, and the target action is executed by each agent to obtain the corresponding environmental reward; based on the random action, the environmental reward, the observation state and a preset discount factor, a target loss function is constructed, and the local Q-value network and the hyperparameter network are updated using the target loss function; the target hyperparameter network and the target local Q-value network are updated according to the updated local Q-value network and hyperparameter network to determine the target Q network parameters corresponding to each agent, so as to calculate the target global Q value corresponding to all agents through the target Q network parameters.

[0009] Optionally, in one embodiment of the present application, determining the control instructions corresponding to each connected autonomous driving vehicle based on the target global Q value includes: determining the vehicle state and target direction corresponding to each connected autonomous driving vehicle based on the target global Q value, and obtaining the traffic light timing strategy corresponding to the target intersection section, so as to determine the guidance information of each connected autonomous driving vehicle based on the vehicle state, the target direction and the traffic light timing strategy; sending the guidance information to each connected autonomous driving vehicle, so that each connected autonomous driving vehicle calculates the corresponding control instructions based on the guidance information and preset vehicle kinematic constraints.

[0010] Optionally, in one embodiment of the present application, the mathematical expression of the global shared reward function is:

[0011]

[0012] Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous driving vehicles in the target intersection during the current control cycle.

[0013] The second embodiment of the present application provides a networked hybrid multi-vehicle flow guidance control device, including: a construction module for determining a vehicle control area corresponding to a target intersection, and based on a preset deep reinforcement learning control strategy, constructing a signal light agent and at least one networked autonomous driving vehicle agent in the vehicle control area, and establishing a global shared reward function corresponding to the signal light agent and the at least one networked autonomous driving vehicle agent; a generation module for, based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, generating a local Q value network, a global Q value network corresponding to the signal light agent and each networked autonomous driving vehicle agent, and a global Q value network corresponding to the signal light agent and the at least one networked autonomous driving vehicle agent. The Q-value network and the corresponding target network are initialized, and based on the initialized Q-value network, the global Q-value network and the target network, each intelligent agent is subjected to environmental interaction operations to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all intelligent agents is generated through the updated local Q-value network, the hyperparameter network and the target network; an execution module is used to determine the control instructions corresponding to each connected autonomous driving vehicle according to the target global Q-value, and control each connected autonomous driving vehicle to perform corresponding driving operations through the control instructions to pass through the target intersection.

[0014] Optionally, in one embodiment of the present application, the construction module includes: a division unit, used to divide the vehicle control area into equally spaced grids to obtain multiple grids, and assign values ​​to each grid according to a preset grid assignment strategy to construct a corresponding first state space; a first determination unit, used to determine the corresponding first action space based on a preset traffic light four-phase control strategy, and obtain the number of human-driven vehicles, the number of networked autonomous driving vehicles and the maximum speed limit of the road in the target intersection, so as to determine the corresponding first reward function according to the number of human-driven vehicles, the number of networked autonomous driving vehicles and the maximum speed limit of the road; a first establishment unit, used to construct the traffic light agent based on the first state space, the first action space and the first reward function; a first calculation unit , used to construct a second state space corresponding to each connected autonomous driving vehicle based on a preset state space expression, and calculate the target speed and target lane of each connected autonomous driving vehicle, so as to determine the second action space corresponding to each connected autonomous driving vehicle through the target speed and the target lane; an acquisition unit, used to obtain the minimum speed limit of the target intersection, and determine a second reward function according to the number of connected autonomous driving vehicles, the minimum speed limit of the road, and the maximum speed limit of the road, so as to construct a connected autonomous driving vehicle intelligent body corresponding to each connected autonomous driving vehicle based on the second state space, the second action space and the second reward function; a second establishment unit, used to construct the global shared reward function based on the first reward function and the second reward function.

[0015] Optionally, in one embodiment of the present application, the generation module includes: a first initialization unit, used to initialize the local Q-value networks corresponding to the traffic light agent and each connected autonomous driving vehicle agent to obtain the initialized network parameters corresponding to the local Q-value networks; a nonlinear combination unit, used to establish the hybrid network based on the hyperparameter network, and perform a nonlinear combination of the output Q values ​​of the local Q-value networks through the hybrid network and the initialized network parameters to establish the global Q-value network; a second initialization unit, used to determine the corresponding target hyperparameter network and target local Q-value network based on the hyperparameter network and the local Q-value network, and initialize the target hyperparameter network and the target local Q-value network; a second determination unit, used to establish a target global Q-value network through the initialized target hyperparameter network and target local Q-value network, and determine multiple hyperparameters of the target global Q-value network; a third determination unit, used to determine the target global Q-value network based on the multiple hyperparameters number, the initialized local Q-value network and the hybrid network and the preset ∈-greedy strategy, to determine the random actions corresponding to the signal light agent and each of the networked autonomous driving vehicle agents; a second calculation unit, for obtaining the observation states corresponding to the signal light agent and each of the networked autonomous driving vehicle agents, to calculate the target action according to the random action and the observation state, and to execute the target action by each agent to obtain the corresponding environmental reward; an updating unit, for constructing a target loss function based on the random action, the environmental reward, the observation state and the preset discount factor, and using the target loss function to update the local Q-value network and the hyperparameter network; a third calculation unit, for updating the target hyperparameter network and the target local Q-value network according to the updated local Q-value network and hyperparameter network, to determine the target Q network parameters corresponding to each of the agents, so as to calculate the target global Q value corresponding to all agents through the target Q network parameters.

[0016] Optionally, in one embodiment of the present application, the execution module includes: a fourth determination unit, used to determine the vehicle state and target direction corresponding to each connected autonomous driving vehicle based on the target global Q value, and obtain the traffic light timing strategy corresponding to the target intersection section, so as to determine the guidance information of each connected autonomous driving vehicle based on the vehicle state, the target direction and the traffic light timing strategy; a sending unit, used to send the guidance information to each connected autonomous driving vehicle, so that each connected autonomous driving vehicle calculates the corresponding control instructions according to the guidance information and preset vehicle kinematic constraints.

[0017] Optionally, in one embodiment of the present application, the mathematical expression of the global shared reward function is:

[0018]

[0019] Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous driving vehicles in the target intersection during the current control cycle.

[0020] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the networked hybrid multi-vehicle flow guidance control method as described in the above embodiment.

[0021] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned networked hybrid multi-vehicle flow guidance control method.

[0022] The fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned networked hybrid multi-vehicle flow guidance control method.

[0023] Therefore, the embodiments of the present application have the following beneficial effects:

[0024] In an embodiment of the present application, a vehicle control area corresponding to a target intersection is determined, and based on a preset deep reinforcement learning control strategy, a signal light agent and at least one connected autonomous vehicle agent in the vehicle control area are constructed, and a global shared reward function corresponding to the signal light agent and at least one connected autonomous vehicle agent is established; based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, the global Q-value network and the corresponding target network corresponding to the signal light agent and each connected autonomous vehicle agent are initialized, and based on the initialized Q-value network, the global Q-value network and the target network, each agent is subjected to an environment interaction operation to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, the hyperparameter network and the target network; the control instructions corresponding to each connected autonomous vehicle are determined according to the target global Q-value, and each connected autonomous vehicle is controlled to perform the corresponding driving operation through the control instructions to pass the target intersection. This application can improve intersection efficiency at the system level by collaboratively controlling intersection lights and CAVs in a connected environment, targeting urban intersection traffic scenarios. This addresses the issues of existing technologies that fail to fully consider the lane-changing needs and kinematic constraints of connected autonomous vehicles, making it difficult to effectively dispatch and guide vehicle flows and unable to meet the requirements of real-time applications.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a networked hybrid multi-vehicle flow guidance control method provided according to an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a basic scenario setting provided for an embodiment of the present application;

[0029] Figure 3 A schematic diagram of a state space representation of a traffic light agent provided in one embodiment of the present application;

[0030] Figure 4 A schematic diagram of four-phase control of a signal light provided in one embodiment of the present application;

[0031] Figure 5A schematic diagram of a state space representation of a connected autonomous driving vehicle agent provided for one embodiment of the present application;

[0032] Figure 6 A schematic diagram of a hierarchical control architecture for multi-vehicle flow guidance provided in accordance with an embodiment of the present application;

[0033] Figure 7 This is an example diagram of a networked hybrid multi-vehicle flow guidance control device according to an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] Among them, 10-networked hybrid multi-vehicle flow guidance control device; 100-construction module, 200-generation module, 300-execution module; 801-memory, 802-processor, 803-communication interface. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the networked hybrid multi-vehicle flow guidance control method and device of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a connected hybrid multi-vehicle flow guidance and control method. In this method, by determining the vehicle control area corresponding to the target intersection section, and based on a preset deep reinforcement learning control strategy, a signal light agent and at least one connected autonomous vehicle agent in the vehicle control area are constructed, and a global shared reward function corresponding to the signal light agent and at least one connected autonomous vehicle agent is established; based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network and corresponding target network corresponding to the signal light agent and each connected autonomous vehicle agent are initialized, and based on the initialized Q-value network, global Q-value network and target network, each agent is subjected to an environment interaction operation to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, hyperparameter network and target network; the control instructions corresponding to each connected autonomous vehicle are determined according to the target global Q-value, and each connected autonomous vehicle is controlled to perform the corresponding driving operation through the control instructions to pass the target intersection section. This application can improve intersection efficiency at the system level by collaboratively controlling intersection lights and CAVs in a connected environment, targeting urban intersection traffic scenarios. This addresses the issues of existing technologies that fail to fully consider the lane-changing needs and kinematic constraints of connected autonomous vehicles, making it difficult to effectively dispatch and guide vehicle flows and unable to meet the requirements of real-time applications.

[0038] Specifically, Figure 1 A flowchart of a networked hybrid multi-vehicle flow guidance control method provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the networked hybrid multi-vehicle flow guidance control method includes the following steps:

[0040] In step S101, the vehicle control area corresponding to the target intersection is determined, and based on a preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous driving vehicle agent in the vehicle control area are constructed, and a global shared reward function corresponding to the traffic light agent and at least one connected autonomous driving vehicle agent is established.

[0041] The embodiment of the present application can first determine the vehicle control area corresponding to the target intersection section based on the basic setting of mixed multi-vehicle flow guidance control in the network environment, such as Figure 2 As shown, the area at a distance L from the stop line of the intersection is the vehicle control area.

[0042] Generally, the embodiment of the present application may assume a completely symmetrical intersection in four directions, with n s Lanes ( Figure 2 Chinese s =4), and the arrow indicates the direction in which vehicles are allowed to travel. There are two types of vehicles at this intersection: human-driven vehicles (HDVs) and connected autonomous vehicles (CAVs).

[0043] Then, the embodiments of the present application can be based on deep reinforcement learning control strategies to construct the traffic light agent in the vehicle control area and the connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle.

[0044] Optionally, in one embodiment of the present application, a vehicle control area corresponding to a target intersection is determined, and based on a preset deep reinforcement learning control strategy, a signal light agent and at least one networked autonomous driving vehicle agent in the vehicle control area are constructed, and a global shared reward function corresponding to the signal light agent and at least one networked autonomous driving vehicle agent is established, including: dividing the vehicle control area into equally spaced grids to obtain a plurality of grids, and assigning a value to each grid according to a preset grid assignment strategy to construct a corresponding first state space; determining a corresponding first action space based on a preset signal light four-phase control strategy, and obtaining the number of human-driven vehicles, the number of networked autonomous driving vehicles, and the maximum speed limit of the road in the target intersection, so as to obtain a reward function based on the number of human-driven vehicles, the number of networked autonomous driving vehicles, and the maximum speed limit of the road. The maximum speed limit determines the corresponding first reward function; based on the first state space, the first action space and the first reward function, a traffic light agent is constructed; based on the preset state space expression, a second state space corresponding to each connected autonomous driving vehicle is constructed, and the target speed and target lane of each connected autonomous driving vehicle are calculated to determine the second action space corresponding to each connected autonomous driving vehicle through the target speed and target lane; the minimum speed limit of the target intersection is obtained, and the second reward function is determined according to the number of connected autonomous driving vehicles, the minimum speed limit of the road and the maximum speed limit of the road, so as to construct a connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle based on the second state space, the second action space and the second reward function; based on the first reward function and the second reward function, a global shared reward function is constructed.

[0045] It should be noted that the embodiment of the present application can adopt the QMIX multi-agent reinforcement learning algorithm to define the relevant parameters of the corresponding deep reinforcement learning control strategy, determine the intelligent agent corresponding to the traffic light and each connected autonomous driving vehicle, and can observe some environmental conditions.

[0046] Specifically, the process of constructing the traffic light agent and the connected autonomous driving vehicle agent in the embodiment of the present application is as follows:

[0047] 1. Traffic Light Agent:

[0048] State space (i.e. first state space) S T :The vehicle control area is divided into equally spaced grids, and each lane contains a total of grid; for the traffic light agent in QMIX, Figure 2 Taking the traffic status of the eastbound intersection E as an example, the following Figure 3 The state space representation shown includes the position matrix P T , velocity matrix V T and signal state matrix TL T , that is, S T ={P T ,V T ,TL T}; Among them, for the position matrix P T For example, if the divided grid is occupied by a CAV, the grid is assigned a value of 2; if the divided grid is occupied by an HDV, the grid is assigned a value of 1; if no vehicle occupies the grid, the grid is assigned a value of 0.

[0049] It should be noted that for the velocity matrix V T , based on the speed information of CAV and HDV, assign values ​​to the speed matrix; for the signal state matrix TL T If the signal light status of the current lane is red, the signal status of the lane is assigned a value of 0; otherwise, the signal status of the lane is assigned a value of 1.

[0050] Action space (i.e. second action space) A T :The signal light adopts Figure 4 The four-phase scheme shown in the figure, the action space of the traffic light agent is represented as A T ={0,1}; 1 indicates switching to the next phase, and 0 indicates maintaining the current phase.

[0051] 2. CAV Agent:

[0052] State space (i.e., second state space) S C :For the CAV agent i in QMIX, its state space S Ci , the embodiment of the present application may adopt Figure 5 The state space representation shown includes the position matrix P Ci , relative velocity matrix RV Ci , signal state matrix TL Ci ,Right now Among them, the position matrix P Ci, Traffic light state matrix TL Ci and the traffic light agent's P T TL T remain consistent; in addition, the relative velocity matrix RV Ci It represents the speed difference between the CAV agent i and all surrounding vehicles.

[0053] Action space (i.e. second action space) A C : The action space of each CAV agent i is set as:

[0054]

[0055] Among them, V di ,L di denote the target speed and target lane of CAV agent i, respectively.

[0056] Based on the given target speed V of CAV agent i di and target lane L di In this embodiment, the model predictive control method can be used to calculate the acceleration acc of the CAV agent i. i and steering angle The specific process is as follows:

[0057] (1) In the embodiment of the present application, a vehicle dynamics model can be used to describe the kinematic characteristics of the vehicle controlled by the CAV agent i, as shown in the following formula:

[0058]

[0059] Among them, x i ,y i represents the horizontal and vertical coordinates of the vehicle controlled by CAV agent i; v i Indicates speed; represents the yaw angle; β i Indicates the sideslip angle of the center of mass; L fi , l ri Indicates the distance between the center of mass and the front and rear axles of the vehicle; acc i represents acceleration; δ fi Indicates the steering angle.

[0060] In order to facilitate those skilled in the art to understand the execution logic of this application, the embodiments of this application may define u i =[acc i ,δ fi ] T , then the above kinematic model can be simplified as:

[0061] xi,t+1 =f(x i,t ,u i,t )

[0062] Where f represents the vehicle dynamics model and t represents the time.

[0063] (2) The embodiment of the present application defines the model predictive control optimization objective function and constraints to solve the acceleration acc i and steering angle

[0064]

[0065] Among them, Q y ,Q l Represent the weight coefficients of tracking lane and tracking speed respectively.

[0066] In this embodiment of the present application, the prediction time domain may be set to T f , then the optimal control problem can be expressed as:

[0067]

[0068] Among them, u i,min ,u i,max represents the lower and upper bounds of the control input of the vehicle controlled by CAV agent i; the solution is After that, the embodiment of the present application can only execute the first action

[0069] Furthermore, the embodiment of the present application can also use the traffic light agent reward and the CAV agent reward to form a global shared reward, as described below:

[0070] 1. Traffic light agent reward function (i.e., first reward function) R T :

[0071] Those skilled in the art will appreciate that, since one of the goals of multi-vehicle flow guidance is to improve traffic efficiency, the reward function of the traffic light agent can be composed of the total delay time of all vehicles at the intersection, that is:

[0072]

[0073] Among them, N C , N H Indicates the number of CAVs and HDVs in the current control cycle; Indicates the time when vehicle i reaches the stop line of the intersection; Pos i represents the distance between vehicle i and the stop line of the intersection; v max Indicates the maximum speed limit on the road.

[0074] 2. CAV agent reward function (i.e., second reward function) R C :

[0075] In the embodiment of the present application, the reward function of each CAV agent i is composed of the efficiency term and security items composition, The specific formula is as follows:

[0076]

[0077] in, Represent the weight coefficients of efficiency and safety respectively.

[0078] also,

[0079]

[0080] Among them, N i represents all vehicles in the vehicle control area of ​​the lane controlled by the CAV agent i; v min , v max Respectively represent the minimum and maximum speed limits on the road.

[0081]

[0082] in, Indicates the penalty value when a collision occurs, Indicates that a collision occurs in the vehicle control zone of the lane where the vehicle controlled by CAV agent i is located.

[0083] Therefore, the embodiments of the present application provide reliable technical support for the subsequent model training and parameter updating by determining the traffic light agent and each connected autonomous driving vehicle agent.

[0084] Optionally, in one embodiment of the present application, the mathematical expression of the global shared reward function is:

[0085]

[0086] Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous vehicles in the target intersection during the current control cycle.

[0087] As an achievable approach, embodiments of the present application can utilize the traffic light agent reward function and the CAV agent reward function to construct a global shared reward R, as shown in the following formula:

[0088]

[0089] Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous vehicles in the target intersection during the current control cycle.

[0090] In step S102, based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network and corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent are initialized, and based on the initialized Q-value network, global Q-value network and target network, each agent is subjected to environmental interaction operations to update the local Q-value network, the pre-built hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, hyperparameter network and target network.

[0091] Furthermore, the embodiments of the present application also need to perform environmental interaction, local Q-value network and hybrid network parameter update operations based on the QMIX multi-agent reinforcement learning algorithm, so as to generate the target global Q value corresponding to all agents.

[0092] Optionally, in one embodiment of the present application, based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, the global Q-value network and the corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent are initialized, and based on the initialized Q-value network, the global Q-value network and the target network, each agent is subjected to environmental interaction operations to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, the hyperparameter network and the target network, including: initializing the local Q-value network corresponding to the traffic light agent and each connected autonomous driving vehicle agent to obtain the initialization network parameters corresponding to the local Q-value network; establishing a hybrid network based on the hyperparameter network, and performing a nonlinear combination of the output Q-value of the local Q-value network through the hybrid network and the initialization network parameters to establish a global Q-value network; determining the corresponding target hyperparameter network and target local Q-value network based on the hyperparameter network and the local Q-value network. , and initialize the target hyperparameter network and the target local Q-value network; establish a target global Q-value network through the initialized target hyperparameter network and the target local Q-value network, and determine multiple hyperparameters of the target global Q-value network; determine the random actions corresponding to the traffic light agent and each connected autonomous driving vehicle agent based on multiple hyperparameters, the initialized local Q-value network and the hybrid network and the preset ∈-greedy strategy; obtain the observation state corresponding to the traffic light agent and each connected autonomous driving vehicle agent to calculate the target action according to the random action and the observation state, and execute the target action through each agent to obtain the corresponding environmental reward; construct a target loss function based on the random action, environmental reward, observation state and the preset discount factor, and use the target loss function to update the local Q-value network and hyperparameter network; update the target hyperparameter network and the target local Q-value network according to the updated local Q-value network and hyperparameter network to determine the target Q network parameters corresponding to each agent, so as to calculate the target global Q value corresponding to all agents through the target Q network parameters.

[0093] In the specific implementation process, the embodiment of the present application can set the current road to have a total of 1 traffic light system and N C A CAV vehicle needs to be controlled, so the specific process of model training and parameter update in the embodiment of the present application is as follows:

[0094] 1. Initialization:

[0095] 1) Local Q value network initialization:

[0096] For the traffic light agent and each CAV agent i∈{1,2,…,N C}, initialize the local Q value network, including Q T (ST ,A T θ T )and Among them, θ T and denote the local Q-value network parameters of the traffic light agent and CAV agent i respectively;

[0097] 2) Global Q-value network initialization:

[0098] Building a hybrid network M , whose parameters are determined by the hyperparameter network f H Output: f M The traffic light agent is connected to the Q-value network Q of each CAV agent i. T (S T ,A T θ T ), The output Q values ​​are nonlinearly combined to form a global Q value network, as shown in the following formula:

[0099]

[0100] Where S=(S T ,S C ) represents the observed state of all agents; A=(A T ,A C ) represents the actions of all agents; θ H represents the hyperparameter network f H Parameters;

[0101] 3) Target network initialization:

[0102] Initialize the target Q value network Q of the traffic light agent target,T (S T ,A T θ target,T ), the target Q-value network of CAV agent i and target hyperparameters network f target,H ; Target global Q value network Q total,target (S,A) is composed of Q target,T (S T ,A T θ target,T ), With f target,H The composition is as shown below:

[0103]

[0104] in, denote the parameters of the target Q network of the traffic light agent and the target Q network of the CAV agent i; θ target,H Parameters representing the target hyperparameter network;

[0105] 4) Hyperparameter settings:

[0106] Set the learning rate α of the traffic light agent T , the learning rate α of CAV agent i i , the learning rate α of the hyperparameter network H , discount factor γ, exploration rate ∈, target network update step size τ.

[0107] 2. Environmental Interaction

[0108] In each round of interaction, the traffic light agent can choose action A according to its strategy T , using the ∈-greedy strategy, as shown below:

[0109]

[0110] Here, with probability∈ means selecting a random action with probability∈.

[0111] Similarly, in the embodiment of the present application, the CAV agent i selects an action according to its strategy Use the ∈-greedy strategy as shown below:

[0112]

[0113] At time t, the traffic light agent and the CAV agent are based on the observed state S t =(S T,t ,S C,t ), calculate action A t =(A T,t ,A C,t ), interact with the traffic environment and perform action A t Get environmental reward R t =(R T,t ,R C,t ) and the next state S t+1 =(S T,t+1 ,S C,t+1 ), and the experience (S t ,A t ,R t ,S t+1 ) exists in the experience replay pool B.

[0114] 3. Network parameter update:

[0115] Afterwards, the embodiment of the present application can construct the following loss function As shown in the following formula:

[0116]

[0117] Based on the above loss function, the embodiment of the present application can update the Q-value network parameters of the traffic light agent and the CAV agent.

[0118] For the Q-value network of the traffic light agent, it is updated as follows:

[0119]

[0120] in, Represents the loss function For parameter θ T Find the gradient.

[0121] For the Q-value network of CAV agent i, it is updated as follows:

[0122]

[0123] in, Represents the loss function For parameter θ i Find the gradient.

[0124] For the hyperparameter network, the update method is as follows:

[0125]

[0126] in, Represents the loss function For parameter θ H Find the gradient.

[0127] For the target Q-value network of the traffic light agent, it is updated as follows:

[0128] θ target,T ←τθ T +(1-τ)θ target,T (17)

[0129] For the target Q-value network of CAV agent i, it is updated as follows:

[0130]

[0131] For the target hyperparameter network, it is updated as follows:

[0132] θ target,H ←τθ H +(1-τ)θ target,H (19)

[0133] In step S103, the control instructions corresponding to each connected autonomous driving vehicle are determined according to the target global Q value, and each connected autonomous driving vehicle is controlled by the control instructions to perform corresponding driving operations to pass the target intersection.

[0134] It is understandable that the embodiment of the present application can adopt a layered approach to construct a multi-vehicle flow guidance control solution. The upper layer can adopt deep reinforcement learning to decide on the signal light control strategy and calculate the guidance information of the CAV vehicle, and send it to the CAV vehicle that needs to be controlled through the roadside control center; secondly, the lower layer uses model predictive control based on the guidance information to calculate the control instructions of the CAV, such as Figure 6 As shown, the computational complexity is reduced.

[0135] Optionally, in one embodiment of the present application, the control instructions corresponding to each connected autonomous driving vehicle are determined based on the target global Q value, including: determining the vehicle state and target direction corresponding to each connected autonomous driving vehicle based on the target global Q value, and obtaining the traffic light timing strategy corresponding to the target intersection, so as to determine the guidance information of each connected autonomous driving vehicle based on the vehicle state, target direction and traffic light timing strategy; sending the guidance information to each connected autonomous driving vehicle, so that each connected autonomous driving vehicle calculates the corresponding control instructions based on the guidance information and preset vehicle kinematic constraints.

[0136] In practice, the roadside control center determines the intersection signal timing based on the status and target directions of all vehicles at the intersection. It also calculates guidance information for CAVs and then sends it to the CAVs. This guidance information includes lane change instructions and target speeds for each CAV. After receiving the guidance information and combining it with the vehicle's kinematic constraints, the CAVs calculate control instructions.

[0137] Therefore, the embodiments of the present application fully consider the lane-changing requirements of CAV vehicles, and at the same time introduce the kinematic model of CAV vehicles through hierarchical control, thereby reducing the complexity of problem solving, improving computational efficiency, and meeting the requirements of real-time applications.

[0138] According to the connected hybrid multi-vehicle flow guidance control method proposed in an embodiment of the present application, by determining the vehicle control area corresponding to the target intersection section, and based on a preset deep reinforcement learning control strategy, constructing a signal light agent and at least one connected autonomous vehicle agent in the vehicle control area, and establishing a global shared reward function corresponding to the signal light agent and at least one connected autonomous vehicle agent; based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network and corresponding target network corresponding to the signal light agent and each connected autonomous vehicle agent are initialized, and based on the initialized Q-value network, global Q-value network and target network, each agent is subjected to an environment interaction operation to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, hyperparameter network and target network; the control instructions corresponding to each connected autonomous vehicle are determined based on the target global Q-value, and each connected autonomous vehicle is controlled to perform the corresponding driving operation through the control instructions to pass the target intersection section. In a connected environment, this application can improve the traffic efficiency of urban intersections at the system level by collaboratively controlling intersection lights and CAVs.

[0139] Secondly, the networked hybrid multi-vehicle flow guidance control device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0140] Figure 7 It is a block diagram of a networked hybrid multi-vehicle flow guidance control device according to an embodiment of the present application.

[0141] like Figure 7 As shown, the networked hybrid multi-vehicle flow guidance control device 10 includes: a construction module 100, a generation module 200 and an execution module 300.

[0142] Among them, the construction module 100 is used to determine the vehicle control area corresponding to the target intersection section, and based on the preset deep reinforcement learning control strategy, construct a traffic light agent and at least one networked autonomous driving vehicle agent in the vehicle control area, and establish a global shared reward function corresponding to the traffic light agent and at least one networked autonomous driving vehicle agent.

[0143] Generation module 200 is used to initialize the local Q-value network, global Q-value network and corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, and perform environmental interaction operations on each agent based on the initialized Q-value network, global Q-value network and target network to update the local Q-value network, pre-built hyperparameter network and target network, and generate the target global Q value corresponding to all agents through the updated local Q-value network, hyperparameter network and target network.

[0144] The execution module 300 is used to determine the control instructions corresponding to each connected autonomous driving vehicle based on the target global Q value, and control each connected autonomous driving vehicle to perform corresponding driving operations through the control instructions to pass the target intersection.

[0145] Optionally, in one embodiment of the present application, the construction module 100 includes: a division unit, a first determination unit, a first establishment unit, a first calculation unit, an acquisition unit, and a second establishment unit.

[0146] The division unit is used to divide the vehicle control area into equally spaced grids to obtain a plurality of grids, and assign a value to each grid according to a preset grid assignment strategy to construct a corresponding first state space.

[0147] The first determination unit is used to determine a corresponding first action space based on a preset four-phase control strategy of the traffic light, and obtain the number of human-driven vehicles, the number of connected autonomous driving vehicles, and the maximum speed limit of the road in the target intersection, so as to determine a corresponding first reward function according to the number of human-driven vehicles, the number of connected autonomous driving vehicles, and the maximum speed limit of the road.

[0148] The first building unit is used to build a traffic light agent based on the first state space, the first action space and the first reward function.

[0149] The first computing unit is configured to construct a second state space corresponding to each connected autonomous vehicle based on a preset state space expression, and calculate a target speed and a target lane for each connected autonomous vehicle, so as to determine a second action space corresponding to each connected autonomous vehicle through the target speed and the target lane.

[0150] an acquisition unit for obtaining the minimum speed limit of the target intersection and determining a second reward function based on the number of connected autonomous driving vehicles, the minimum speed limit of the road, and the maximum speed limit of the road, so as to construct a connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle based on the second state space, the second action space, and the second reward function.

[0151] The second establishing unit is configured to construct a global shared reward function based on the first reward function and the second reward function.

[0152] Optionally, in one embodiment of the present application, the generation module 200 includes: a first initialization unit, a nonlinear combination unit, a second initialization unit, a second determination unit, a third determination unit, a second calculation unit, an updating unit and a third calculation unit.

[0153] Among them, the first initialization unit is used to initialize the local Q-value network corresponding to the traffic light intelligent agent and each networked autonomous driving vehicle intelligent agent to obtain the initialization network parameters corresponding to the local Q-value network.

[0154] The nonlinear combination unit is used to establish a hybrid network based on the hyperparameter network, and to perform nonlinear combination on the output Q value of the local Q value network through the hybrid network and the initialized network parameters to establish a global Q value network.

[0155] The second initialization unit is used to determine the corresponding target hyperparameter network and target local Q value network based on the hyperparameter network and the local Q value network, and initialize the target hyperparameter network and the target local Q value network.

[0156] The second determining unit is used to establish a target global Q value network through the initialized target hyperparameter network and the target local Q value network, and determine multiple hyperparameters of the target global Q value network.

[0157] The third determination unit is used to determine the random actions corresponding to the traffic light agent and each connected autonomous driving vehicle agent based on multiple hyperparameters, the initialized local Q-value network and the hybrid network and the preset ∈-greedy strategy.

[0158] The second computing unit is used to obtain the observation states corresponding to the traffic light agent and each connected autonomous driving vehicle agent, calculate the target action based on the random action and the observation state, and execute the target action by each agent to obtain the corresponding environmental reward.

[0159] The update unit is used to construct a target loss function based on random actions, environmental rewards, observation states, and preset discount factors, and use the target loss function to update the local Q-value network and hyperparameter network.

[0160] The third computing unit is used to update the target hyperparameter network and the target local Q value network according to the updated local Q value network and hyperparameter network to determine the target Q network parameters corresponding to each intelligent agent, so as to calculate the target global Q value corresponding to all intelligent agents through the target Q network parameters.

[0161] Optionally, in one embodiment of the present application, the execution module 300 includes: a fourth determining unit and a sending unit.

[0162] Among them, the fourth determination unit is used to determine the vehicle state and target direction corresponding to each connected autonomous driving vehicle based on the target global Q value, and obtain the traffic light timing strategy corresponding to the target intersection section, so as to determine the guidance information of each connected autonomous driving vehicle based on the vehicle state, target direction and traffic light timing strategy.

[0163] The sending unit is used to send the guidance information to each networked autonomous driving vehicle, so that each networked autonomous driving vehicle calculates the corresponding control instructions based on the guidance information and preset vehicle kinematic constraints.

[0164] Optionally, in one embodiment of the present application, the mathematical expression of the global shared reward function is:

[0165]

[0166] Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous vehicles in the target intersection during the current control cycle.

[0167] It should be noted that the aforementioned explanation of the embodiment of the networked hybrid multi-vehicle flow guidance control method is also applicable to the networked hybrid multi-vehicle flow guidance control device of this embodiment, and will not be repeated here.

[0168] The networked hybrid multi-vehicle flow guidance control device proposed in the embodiment of the present application includes a construction module 100 for determining a vehicle control area corresponding to a target intersection, and based on a preset deep reinforcement learning control strategy, constructing a signal light agent and at least one networked autonomous driving vehicle agent in the vehicle control area, and establishing a global shared reward function corresponding to the signal light agent and at least one networked autonomous driving vehicle agent; a generation module 200 for generating a local Q value network corresponding to the signal light agent and each networked autonomous driving vehicle agent based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm. The system initializes the local Q-value network, the global Q-value network and the corresponding target network, and performs environmental interaction operations on each agent based on the initialized Q-value network, the global Q-value network and the target network to update the local Q-value network, the pre-built hyperparameter network and the target network, and generates the target global Q-value corresponding to all agents through the updated local Q-value network, the hyperparameter network and the target network; the execution module 300 is used to determine the control instructions corresponding to each networked autonomous driving vehicle according to the target global Q-value, and control each networked autonomous driving vehicle to perform the corresponding driving operation through the control instructions to pass the target intersection. In a connected environment, the present application can improve the traffic efficiency of intersections at the system level by collaboratively controlling intersection lights and CAVs for urban intersection traffic scenarios.

[0169] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0170] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0171] When the processor 802 executes the program, the networked hybrid multi-vehicle flow guidance control method provided in the above embodiment is implemented.

[0172] Furthermore, the electronic device further includes:

[0173] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0174] The memory 801 is used to store computer programs that can be run on the processor 802.

[0175] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0176] If the memory 801, processor 802, and communication interface 803 are implemented independently, the communication interface 803, memory 801, and processor 802 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0177] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0178] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0179] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned networked hybrid multi-vehicle flow guidance control method.

[0180] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned networked hybrid multi-vehicle flow guidance control method.

[0181] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0182] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0183] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0184] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0185] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0186] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0187] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0188] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A networked hybrid multi-vehicle flow guidance control method, characterized in that: The following steps are involved: Determine a vehicle control zone corresponding to the target intersection, and construct a signal light agent and at least one connected autonomous vehicle agent in the vehicle control zone based on a preset deep reinforcement learning control strategy, and establish a global shared reward function corresponding to the signal light agent and the at least one connected autonomous vehicle agent; Based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, the global Q-value network, and the corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent are initialized, and based on the initialized Q-value network, the global Q-value network, and the target network, each agent is subjected to an environment interaction operation to update the local Q-value network, the pre-constructed hyperparameter network, and the target network, and the target global Q-value corresponding to all agents is generated through the updated local Q-value network, the hyperparameter network, and the target network; The control instructions corresponding to each connected autonomous driving vehicle are determined based on the target global Q value, and each connected autonomous driving vehicle is controlled by the control instructions to perform corresponding driving operations to pass through the target intersection.

2. The method according to claim 1, characterized in that The determining of a vehicle control zone corresponding to a target intersection, constructing a signal light agent and at least one connected autonomous vehicle agent in the vehicle control zone based on a preset deep reinforcement learning control strategy, and establishing a global shared reward function corresponding to the signal light agent and the at least one connected autonomous vehicle agent, includes: Dividing the vehicle control area into equally spaced grids to obtain a plurality of grids, and assigning a value to each grid according to a preset grid assignment strategy to construct a corresponding first state space; Determining a corresponding first action space based on a preset four-phase traffic light control strategy, and obtaining the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection, so as to determine a corresponding first reward function based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road; constructing the traffic light agent based on the first state space, the first action space, and the first reward function; Based on a preset state space expression, construct a second state space corresponding to each connected autonomous vehicle, and calculate a target speed and a target lane for each connected autonomous vehicle, so as to determine a second action space corresponding to each connected autonomous vehicle based on the target speed and the target lane; Obtaining a minimum speed limit for the target intersection, and determining a second reward function based on the number of connected autonomous driving vehicles, the minimum speed limit, and the maximum speed limit, to construct a connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle based on the second state space, the second action space, and the second reward function; The global shared reward function is constructed based on the first reward function and the second reward function.

3. The method according to claim 2, characterized in that The global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm are used to initialize the local Q-value network, global Q-value network, and corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent. Based on the initialized Q-value network, global Q-value network, and target network, each agent is subjected to an environment interaction operation to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The target global Q-value corresponding to all agents is generated through the updated local Q-value network, the hyperparameter network, and the target network, including: Initializing the local Q-value networks corresponding to the traffic light agent and each of the networked autonomous driving vehicle agents to obtain initialized network parameters corresponding to the local Q-value networks; Based on the hyperparameter network, the hybrid network is established, and the output Q values ​​of the local Q value network are nonlinearly combined through the hybrid network and the initialized network parameters to establish the global Q value network; Based on the hyperparameter network and the local Q-value network, determining a corresponding target hyperparameter network and a target local Q-value network, and initializing the target hyperparameter network and the target local Q-value network; Establishing a target global Q-value network through the initialized target hyperparameter network and the target local Q-value network, and determining multiple hyperparameters of the target global Q-value network; Determining random actions corresponding to the traffic light agent and each of the connected autonomous driving vehicle agents based on the multiple hyperparameters, the initialized local Q-value network and the hybrid network, and a preset ∈-greedy strategy; Obtaining observation states corresponding to the traffic light agent and each of the connected autonomous driving vehicle agents, calculating a target action based on the random action and the observation states, and having each agent perform the target action to obtain a corresponding environmental reward; Constructing a target loss function based on the random action, the environmental reward, the observation state, and a preset discount factor, and updating the local Q-value network and the hyperparameter network using the target loss function; The target hyperparameter network and the target local Q value network are updated according to the updated local Q value network and hyperparameter network to determine the target Q network parameters corresponding to each intelligent agent, so as to calculate the target global Q value corresponding to all intelligent agents through the target Q network parameters.

4. The method according to claim 3, characterized in that Determining the control instructions corresponding to each connected autonomous driving vehicle according to the target global Q value includes: Determining a vehicle state and a target direction corresponding to each of the connected autonomous vehicles according to the target global Q value, and obtaining a traffic light timing strategy corresponding to the target intersection, so as to determine guidance information for each of the connected autonomous vehicles based on the vehicle state, the target direction, and the traffic light timing strategy; The guidance information is sent to each of the networked autonomous driving vehicles, so that each of the networked autonomous driving vehicles calculates corresponding control instructions based on the guidance information and preset vehicle kinematic constraints.

5. The method according to claim 2, characterized in that The mathematical expression of the global shared reward function is: Among them, R T represents the first reward function; R C,i represents the second reward function; N C Indicates the number of connected autonomous driving vehicles in the target intersection during the current control cycle.

6. A networked hybrid multi-vehicle flow guidance control device, characterized in that: include: A construction module is configured to determine a vehicle control zone corresponding to a target intersection, and based on a preset deep reinforcement learning control strategy, construct a signal light agent and at least one connected autonomous vehicle agent in the vehicle control zone, and establish a global shared reward function corresponding to the signal light agent and the at least one connected autonomous vehicle agent; A generation module is configured to initialize the local Q-value network, global Q-value network, and corresponding target network corresponding to the traffic light agent and each connected autonomous driving vehicle agent based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, and perform environmental interaction operations on each agent based on the initialized Q-value network, global Q-value network, and target network to update the local Q-value network, the pre-constructed hyperparameter network, and the target network, and generate target global Q-values ​​corresponding to all agents through the updated local Q-value network, the hyperparameter network, and the target network; An execution module is used to determine the control instructions corresponding to each connected autonomous driving vehicle based on the target global Q value, and control each connected autonomous driving vehicle to perform corresponding driving operations through the control instructions to pass the target intersection.

7. The device according to claim 6, characterized in that The building blocks include: a dividing unit, configured to divide the vehicle control area into equally spaced grids to obtain a plurality of grids, and assign a value to each grid according to a preset grid assignment strategy to construct a corresponding first state space; a first determining unit, configured to determine a corresponding first action space based on a preset four-phase control strategy for the traffic light, and obtain a number of human-driven vehicles, a number of connected autonomous vehicles, and a maximum speed limit of the road in the target intersection, so as to determine a corresponding first reward function based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road; a first building unit, configured to build the traffic light agent based on the first state space, the first action space, and the first reward function; a first computing unit, configured to construct a second state space corresponding to each connected autonomous vehicle based on a preset state space expression, and calculate a target speed and a target lane for each connected autonomous vehicle, so as to determine a second action space corresponding to each connected autonomous vehicle based on the target speed and the target lane; an acquiring unit, configured to acquire a minimum speed limit of the target intersection, and determine a second reward function based on the number of connected autonomous driving vehicles, the minimum speed limit of the road, and the maximum speed limit of the road, so as to construct a connected autonomous driving vehicle agent corresponding to each connected autonomous driving vehicle based on the second state space, the second action space, and the second reward function; The second establishing unit is configured to construct the global shared reward function based on the first reward function and the second reward function.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the networked hybrid multi-vehicle flow guidance control method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the networked hybrid multi-vehicle flow guidance control method as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that The computer program is executed to implement the networked hybrid multi-vehicle flow guidance control method as described in any one of claims 1 to 5.

Images