Connected Hybrid Multi-Traffic Guidance and Control Method and Device

Abstract

This application relates to a connected hybrid multi-traffic flow guidance and control method and device. The method includes: constructing a multi-traffic flow guidance and control scheme using a hierarchical approach. A roadside control center determines the traffic light timing scheme at the intersection based on the status and target direction of all vehicles, calculates guidance information for connected autonomous vehicles, and then sends this guidance information to the connected autonomous vehicles. The guidance information specifically includes lane-changing instructions and target speeds for each connected autonomous vehicle. Upon receiving the guidance information, the connected autonomous vehicles calculate control instructions based on the guidance information and vehicle kinematic constraints. This solves the problems of existing technologies that do not fully consider the lane-changing needs and vehicle kinematic constraints of connected autonomous vehicles, making it difficult to effectively schedule and guide vehicle flow, and failing to meet the requirements of real-time applications.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation technology, and in particular to a networked hybrid multi-vehicle flow guidance and control method and device. Background Technology

[0002] With the continuous development of Intelligent Transportation Systems (ITS) and Vehicle-to-Everything (V2X) technologies, the information exchange capabilities between vehicles and between vehicles and infrastructure have been significantly enhanced. V2X technology, through its communication platform, enables vehicles to share crucial data in real time, such as road conditions, traffic signal status, and the speed and location of surrounding vehicles, providing unprecedented opportunities for traffic management and control. Especially in mixed traffic flow environments, including both connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs), effectively scheduling and guiding traffic flow has become a critical issue that urgently needs to be addressed in the field of traffic control.

[0003] Traditional traffic control methods, primarily based on fixed signal cycles and lane allocation strategies, often fail to adequately consider the dynamic changes in traffic flow. This is especially problematic in complex environments where different types of vehicles (such as CAVs and HDVs) coexist, frequently leading to traffic congestion, delays, and even safety hazards. Currently, coordinated control of traffic lights and CAVs in mixed traffic flows at intersections is not fully realized. In particular, existing technologies, when applying control to CAVs to achieve coordinated control, do not adequately consider vehicle kinematics or dynamic constraints.

[0004] In summary, existing technologies do not fully consider the lane-changing needs of CAV vehicles and vehicle kinematic constraints, making it difficult to effectively schedule and guide vehicle flow, and thus failing to meet the requirements of real-time applications, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a connected hybrid multi-vehicle flow guidance and control method and device to solve the problems that the existing technology does not fully consider the lane changing needs and vehicle kinematic constraints of connected autonomous vehicles, making it difficult to effectively schedule and guide vehicle flow and meet the requirements of real-time applications.

[0006] The first aspect of this application provides a connected hybrid multi-traffic guidance and control method, comprising the following steps: determining the vehicle control zone corresponding to the target intersection segment, and constructing a traffic light agent and at least one connected autonomous vehicle agent in the vehicle control zone based on a preset deep reinforcement learning control strategy, and establishing a global shared reward function corresponding to the traffic light agent and the at least one connected autonomous vehicle agent; initializing the local Q-value network, global Q-value network, and corresponding target network corresponding to the traffic light agent and each connected autonomous vehicle agent based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, and performing environmental interaction operations on each agent based on the initialized Q-value network, global Q-value network, and target network to update the local Q-value network, the pre-constructed hyperparameter network, and the target network, and generating a target global Q-value corresponding to all agents through the updated local Q-value network, the hyperparameter network, and the target network; determining the control command corresponding to each connected autonomous vehicle according to the target global Q-value, and controlling each connected autonomous vehicle to perform corresponding driving operations through the control command to pass through the target intersection segment.

[0007] Optionally, in one embodiment of this application, determining the vehicle control area corresponding to the target intersection segment, and constructing a traffic light agent and at least one connected autonomous vehicle agent in the vehicle control area based on a preset deep reinforcement learning control strategy, and establishing a globally shared reward function corresponding to the traffic light agent and the at least one connected autonomous vehicle agent, includes: dividing the vehicle control area into equally spaced grids to obtain multiple grids, and assigning values ​​to each grid according to a preset grid assignment strategy to construct a corresponding first state space; determining the corresponding first action space based on a preset traffic light four-phase control strategy, and obtaining the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection segment, so as to determine the corresponding first action space based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road. A reward function is defined; based on the first state space, the first action space, and the first reward function, the traffic light agent is constructed; based on a preset state space expression, a second state space corresponding to each connected autonomous vehicle is constructed, and the target speed and target lane of each connected autonomous vehicle are calculated, so as to determine the second action space corresponding to each connected autonomous vehicle through the target speed and the target lane; the minimum speed limit of the target intersection is obtained, and a second reward function is determined according to the number of connected autonomous vehicles, the minimum speed limit, and the maximum speed limit, so as to construct the connected autonomous vehicle agent corresponding to each connected autonomous vehicle based on the second state space, the second action space, and the second reward function; based on the first reward function and the second reward function, a globally shared reward function is constructed.

[0008] Optionally, in one embodiment of this application, the initialization of the local Q-value network, global Q-value network, and corresponding target network for the traffic light agent and each connected autonomous vehicle agent based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, and the environmental interaction operation performed on each agent based on the initialized Q-value network, global Q-value network, and target network to update the local Q-value network, the pre-constructed hyperparameter network, and the target network, and the generation of the target global Q-value for all agents through the updated local Q-value network, the hyperparameter network, and the target network, includes: adjusting the local Q-values ​​of the traffic light agent and each connected autonomous vehicle agent. The network is initialized to obtain the initial network parameters corresponding to the local Q-value network; based on the hyperparameter network, the hybrid network is established, and the output Q-values ​​of the local Q-value network are nonlinearly combined using the hybrid network and the initial network parameters to establish the global Q-value network; based on the hyperparameter network and the local Q-value network, the corresponding target hyperparameter network and target local Q-value network are determined, and the target hyperparameter network and target local Q-value network are initialized; a target global Q-value network is established using the initialized target hyperparameter network and target local Q-value network, and multiple hyperparameters of the target global Q-value network are determined; based on the multiple hyperparameters, the initialized local Q-value network and the hybrid network, and a preset... The `-greedy` strategy determines the random actions corresponding to the traffic light agent and each connected autonomous vehicle agent; it acquires the observation states corresponding to the traffic light agent and each connected autonomous vehicle agent, calculates the target action based on the random actions and the observation states, and executes the target action by each agent to obtain the corresponding environmental reward; it constructs a target loss function based on the random actions, the environmental reward, the observation states, and a preset discount factor, and updates the local Q-value network and the hyperparameter network using the target loss function; it updates the target hyperparameter network and the target local Q-value network based on the updated local Q-value network and hyperparameter network to determine the target Q-network parameters corresponding to each agent, and calculates the target global Q-value corresponding to all agents using the target Q-network parameters.

[0009] Optionally, in one embodiment of this application, determining the control command corresponding to each connected autonomous vehicle based on the target global Q value includes: determining the vehicle state and target direction corresponding to each connected autonomous vehicle based on the target global Q value, and obtaining the traffic light timing strategy corresponding to the target intersection segment, so as to determine the guidance information of each connected autonomous vehicle based on the vehicle state, the target direction and the traffic light timing strategy; and sending the guidance information to each connected autonomous vehicle so that each connected autonomous vehicle can calculate the corresponding control command based on the guidance information and preset vehicle kinematic constraints.

[0010] Optionally, in one embodiment of this application, the mathematical expression of the globally shared reward function is:

[0011]

[0012] in, This represents the first reward function; This represents the second reward function; This indicates the number of connected autonomous vehicles in the target intersection segment during the current control cycle.

[0013] A second aspect of this application provides a connected hybrid multi-traffic guidance and control device, comprising: a construction module, configured to determine the vehicle control zone corresponding to a target intersection segment, and based on a preset deep reinforcement learning control strategy, construct a traffic light agent and at least one connected autonomous vehicle agent within the vehicle control zone, and establish a global shared reward function corresponding to the traffic light agent and the at least one connected autonomous vehicle agent; and a generation module, configured to, based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, generate the local Q-value network and global Q-value network corresponding to the traffic light agent and each connected autonomous vehicle agent. The Q-value network and the corresponding target network are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents. The execution module is used to determine the control command corresponding to each connected autonomous vehicle based on the target global Q-value, and control each connected autonomous vehicle to perform corresponding driving operations through the control command to pass through the target intersection.

[0014] Optionally, in one embodiment of this application, the construction module includes: a partitioning unit, configured to partition the vehicle control area into equally spaced grids to obtain multiple grids, and assign values ​​to each grid according to a preset grid assignment strategy to construct a corresponding first state space; a first determining unit, configured to determine a corresponding first action space based on a preset traffic light four-phase control strategy, and obtain the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection segment, so as to determine a corresponding first reward function based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road; a first building unit, configured to construct the traffic light agent based on the first state space, the first action space, and the first reward function; and a first calculation unit. The system comprises: a first state space for constructing a second state space corresponding to each connected autonomous vehicle based on a preset state space expression; a second action space for calculating the target speed and target lane of each connected autonomous vehicle; and an acquisition unit for acquiring the minimum speed limit of the target intersection segment and determining a second reward function based on the number of connected autonomous vehicles, the minimum speed limit, and the maximum speed limit, so as to construct a connected autonomous vehicle agent corresponding to each connected autonomous vehicle based on the second state space, the second action space, and the second reward function; and a second establishment unit for constructing the globally shared reward function based on the first reward function and the second reward function.

[0015] Optionally, in one embodiment of this application, the generation module includes: a first initialization unit, configured to initialize the local Q-value network corresponding to the traffic light agent and each connected autonomous vehicle agent, to obtain the initialization network parameters corresponding to the local Q-value network; a nonlinear combination unit, configured to establish the hybrid network based on the hyperparameter network, and to perform nonlinear combination of the output Q-value of the local Q-value network through the hybrid network and the initialization network parameters to establish the global Q-value network; a second initialization unit, configured to determine the corresponding target hyperparameter network and target local Q-value network based on the hyperparameter network and the local Q-value network, and initialize the target hyperparameter network and the target local Q-value network; a second determination unit, configured to establish a target global Q-value network through the initialized target hyperparameter network and target local Q-value network, and determine multiple hyperparameters of the target global Q-value network; and a third determination unit, configured to determine multiple hyperparameters, the initialized local Q-value network and the hybrid network, and a preset... A first unit employs a greedy strategy to determine the random actions corresponding to the traffic light agent and each connected autonomous vehicle agent; a second calculation unit acquires the observation states corresponding to the traffic light agent and each connected autonomous vehicle agent, calculates the target action based on the random actions and the observation states, and executes the target action by each agent to obtain the corresponding environmental reward; an update unit constructs a target loss function based on the random actions, the environmental reward, the observation states, and a preset discount factor, and updates the local Q-value network and the hyperparameter network using the target loss function; a third calculation unit updates the target hyperparameter network and the target local Q-value network based on the updated local Q-value network and hyperparameter network to determine the target Q-network parameters corresponding to each agent, and calculates the target global Q-value corresponding to all agents using the target Q-network parameters.

[0016] Optionally, in one embodiment of this application, the execution module includes: a fourth determining unit, configured to determine the vehicle state and target direction corresponding to each connected autonomous vehicle based on the target global Q value, and obtain the traffic light timing strategy corresponding to the target intersection segment, so as to determine the guidance information of each connected autonomous vehicle based on the vehicle state, the target direction and the traffic light timing strategy; and a sending unit, configured to send the guidance information to each connected autonomous vehicle, so that each connected autonomous vehicle calculates the corresponding control command based on the guidance information and preset vehicle kinematic constraints.

[0017] Optionally, in one embodiment of this application, the mathematical expression of the globally shared reward function is:

[0018]

[0019] in, This represents the first reward function; This represents the second reward function; This indicates the number of connected autonomous vehicles in the target intersection segment during the current control cycle.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the networked hybrid multi-traffic flow guidance control method as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described networked hybrid multi-traffic guidance and control method.

[0022] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described connected hybrid multi-traffic guidance and control method.

[0023] Therefore, the embodiments of this application have the following beneficial effects:

[0024] The embodiments of this application can determine the vehicle control zone corresponding to the target intersection segment, and construct a traffic light agent and at least one connected autonomous vehicle agent in the vehicle control zone based on a preset deep reinforcement learning control strategy, and establish a global shared reward function corresponding to the traffic light agent and at least one connected autonomous vehicle agent; based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, initialize the local Q-value network, global Q-value network and corresponding target network corresponding to the traffic light agent and each connected autonomous vehicle agent, and perform environmental interaction operations on each agent based on the initialized Q-value network, global Q-value network and target network to update the local Q-value network, the pre-constructed hyperparameter network and the target network, and generate the target global Q-value corresponding to all agents through the updated local Q-value network, hyperparameter network and target network; determine the control command corresponding to each connected autonomous vehicle according to the target global Q-value, and control each connected autonomous vehicle to perform the corresponding driving operation to pass through the target intersection segment through the control command. This application enables improved intersection efficiency at the system level by coordinating the control of traffic lights and CAVs at urban intersections in a connected environment. This addresses the shortcomings of existing technologies, such as insufficient consideration of lane-changing needs and kinematic constraints of connected autonomous vehicles, difficulty in effectively scheduling and guiding vehicle flow, and inability to meet real-time application requirements.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 This is a flowchart of a connected hybrid multi-traffic guidance and control method provided according to an embodiment of this application;

[0028] Figure 2 A schematic diagram illustrating a basic scenario setting for one embodiment of this application;

[0029] Figure 3A schematic diagram illustrating the state space representation of a traffic light agent, provided as an embodiment of this application;

[0030] Figure 4 A schematic diagram of a four-phase control for a traffic light is provided as an embodiment of this application;

[0031] Figure 5 A schematic diagram illustrating the state-space representation of an intelligent agent in a connected autonomous vehicle, provided as an embodiment of this application;

[0032] Figure 6 A schematic diagram of a multi-traffic flow guidance hierarchical control architecture is provided for one embodiment of this application;

[0033] Figure 7 This is an example diagram of a connected hybrid multi-traffic guidance and control device according to an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0035] Among them, 10-connected hybrid multi-vehicle flow guidance and control device; 100-construction module, 200-generation module, 300-execution module; 801-memory, 802-processor, 803-communication interface. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] The following description, with reference to the accompanying drawings, describes a networked hybrid multi-traffic guidance and control method and apparatus according to embodiments of this application. To address the problems mentioned in the background technology, this application provides a connected hybrid multi-traffic flow guidance and control method. In this method, by determining the vehicle control zone corresponding to the target intersection, and based on a preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous vehicle agent are constructed within the vehicle control zone. A global shared reward function is established for the traffic light agent and at least one connected autonomous vehicle agent. Based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network, and corresponding target network for the traffic light agent and each connected autonomous vehicle agent are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents. Control commands are determined for each connected autonomous vehicle based on the target global Q-value, and each connected autonomous vehicle is controlled to perform corresponding driving operations to pass through the target intersection. This application enables improved intersection efficiency at the system level by coordinating the control of traffic lights and CAVs at urban intersections in a connected environment. This addresses the shortcomings of existing technologies, such as insufficient consideration of lane-changing needs and kinematic constraints of connected autonomous vehicles, difficulty in effectively scheduling and guiding vehicle flow, and inability to meet real-time application requirements.

[0038] Specifically, Figure 1 This is a flowchart of a networked hybrid multi-traffic guidance and control method provided in an embodiment of this application.

[0039] like Figure 1 As shown, the connected hybrid multi-vehicle flow guidance and control method includes the following steps:

[0040] In step S101, the vehicle control zone corresponding to the target intersection is determined, and based on the preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous vehicle agent are constructed in the vehicle control zone, and a global shared reward function is established for the traffic light agent and at least one connected autonomous vehicle agent.

[0041] The embodiments of this application can first determine the vehicle control zone corresponding to the target intersection segment based on the basic settings of hybrid multi-flow guidance control in a connected environment, such as... Figure 2 As shown, the distance from the stop line at the intersection The area is the vehicle control zone.

[0042] Generally, embodiments of this application may assume a perfectly symmetrical intersection in four directions, with each direction having lanes ( Figure 2 middle The arrow indicates the permitted direction of travel for vehicles. This intersection accommodates both human-driven vehicles (HDVs) and connected autonomous vehicles (CAVs).

[0043] Subsequently, embodiments of this application can be based on deep reinforcement learning control strategies to construct traffic light agents in the vehicle control area and connected autonomous vehicle agents corresponding to each connected autonomous vehicle.

[0044] Optionally, in one embodiment of this application, a vehicle control zone corresponding to the target intersection is determined, and based on a preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous vehicle agent are constructed in the vehicle control zone. A globally shared reward function is established corresponding to the traffic light agent and at least one connected autonomous vehicle agent. This includes: dividing the vehicle control zone into equally spaced grids to obtain multiple grids, and assigning values ​​to each grid according to a preset grid assignment strategy to construct a corresponding first state space; determining the corresponding first action space based on a preset traffic light four-phase control strategy, and obtaining the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection, so as to determine the action space based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road. The maximum speed limit is determined by a first reward function; a traffic light agent is constructed based on a first state space, a first action space, and the first reward function; a second state space is constructed for each connected autonomous vehicle based on a preset state space expression, and the target speed and target lane of each connected autonomous vehicle are calculated to determine the second action space for each connected autonomous vehicle; the minimum speed limit of the target intersection is obtained, and a second reward function is determined based on the number of connected autonomous vehicles, the minimum speed limit, and the maximum speed limit, to construct a connected autonomous vehicle agent for each connected autonomous vehicle based on the second state space, the second action space, and the second reward function; a globally shared reward function is constructed based on the first and second reward functions.

[0045] It should be noted that the embodiments of this application may employ the QMIX multi-agent reinforcement learning algorithm to define the relevant parameters of the corresponding deep reinforcement learning control strategy, determine the traffic lights and the agents corresponding to each connected autonomous vehicle, and observe some environmental states.

[0046] Specifically, the process of constructing the traffic light intelligent agent and the connected autonomous vehicle intelligent agent in this application embodiment is as follows:

[0047] 1. Traffic light intelligent agent:

[0048] State space (i.e., the first state space) The vehicle control area is divided into equally spaced grids, with each lane containing a total of A grid; for the traffic light agent in QMIX, with Figure 2 The eastbound intersection shown Taking the traffic conditions as an example, adopting such Figure 3 The state-space representation shown includes the position matrix. Velocity matrix and signal state matrix ,Right now ; where, for the position matrix In this case, if a grid is occupied by a CAV, the grid is assigned a value of 2; if a grid is occupied by an HDV, the grid is assigned a value of 1; if no vehicle occupies the grid, the grid is assigned a value of 0.

[0049] It should be noted that for the velocity matrix Based on the speed information from CAV and HDV, values ​​are assigned to the speed matrix; for the signal state matrix... If the traffic light corresponding to the current lane is red, the signal status of that lane is assigned a value of 0; otherwise, the signal status of that lane is assigned a value of 1.

[0050] Action space (i.e., the first action space) Traffic lights adopt the following... Figure 4 The four-phase scheme shown is represented by the action space of the traffic light agent as follows: ;in, 0 indicates switching to the next phase, while 0 indicates maintaining the current phase.

[0051] 2. CAV agent:

[0052] State space (i.e., the second state space) For CAV agents in QMIX Its state space The embodiments of this application may employ, as follows: Figure 5 The state-space representation shown includes the position matrix. Relative velocity matrix Signal state matrix , , Among them, the position matrix Traffic light status matrix Respectively with traffic light intelligent agents , Maintain consistency; furthermore, the relative velocity matrix This indicates a CAV agent. The speed difference between the speed of the vehicle and the speed of all surrounding vehicles.

[0053] Action space (i.e., the second action space) Each CAV agent The motion space is set as follows:

[0054]

[0055] in, Representing CAV agents respectively Target speed and target lane .

[0056] Based on a given CAV agent Target speed and target lane The embodiments of this application can use model predictive control methods to calculate CAV agents. acceleration With steering angle The specific process is as follows:

[0057] (1) The embodiments of this application can use vehicle dynamics models to describe CAV intelligent agents. The kinematic characteristics of the controlled vehicle are shown in the following formula:

[0058] (1)

[0059] in, , Represents CAV intelligent agent The horizontal and vertical coordinates of the controlled vehicle; Indicates speed; Indicates the yaw angle; Indicates the centroid sideslip angle; , This indicates the distance of the center of gravity from the front and rear axles of the vehicle. Indicates acceleration; Indicates the steering angle.

[0060] To facilitate understanding of the execution logic of this application by those skilled in the art, embodiments of this application may define... , Then the above kinematic model can be simplified as follows:

[0061]

[0062] in, Represents the vehicle dynamics model; Indicates the time.

[0063] (2) The embodiments of this application define the objective function and constraints of model predictive control optimization in order to solve for the acceleration. With steering angle :

[0064] (2)

[0065] in, These represent the weighting coefficients for the tracking lane and the tracking speed, respectively.

[0066] The prediction time domain can be set in this embodiment of the application. The optimal control problem can then be expressed as:

[0067] (3)

[0068] in, , Represents CAV intelligent agent The lower and upper boundaries of the control input of the controlled vehicle are obtained by solving. Subsequently, in this embodiment of the application, only the first action may be performed. .

[0069] Furthermore, in this embodiment of the application, the reward for the traffic light agent and the reward for the CAV agent can be combined to form a globally shared reward, as described below:

[0070] 1. Traffic light agent reward function (i.e., the first reward function) :

[0071] As those skilled in the art will understand, since one of the purposes of multi-traffic guidance is to improve traffic efficiency, the reward function of the traffic light agent can be composed of the total delay time of all vehicles at the intersection, i.e.:

[0072] (4)

[0073] in, , This indicates the number of CAVs and HDVs within the current control cycle; Indicates vehicle Time taken to reach the stop line at the intersection; Indicates vehicle Distance from the stop line at the intersection; This indicates the maximum speed limit on the road.

[0074] 2. CAV agent reward function (i.e., the second reward function) :

[0075] In the embodiments of this application, each CAV agent The reward function is composed of the efficiency term and safety items composition, The specific formula is as follows:

[0076] (5)

[0077] in, , These represent the weighting coefficients for efficiency and safety, respectively.

[0078] also,

[0079] (6)

[0080] in, Represents CAV intelligent agent All vehicles in the vehicle control zone of the lane where the controlled vehicle is located; , These represent the minimum and maximum speed limits for the road, respectively.

[0081] (7)

[0082] in, This represents the penalty value when a collision occurs. Represents CAV intelligent agent A collision has occurred in the vehicle control zone of the lane where the controlled vehicle is located.

[0083] Therefore, the embodiments of this application provide reliable technical support for subsequent model training and parameter updates by determining the traffic light agent and each connected autonomous vehicle agent.

[0084] As one possible approach, embodiments of this application can construct a globally shared reward using two parts: a traffic light agent reward function and a CAV agent reward function. As shown in the following formula:

[0085] (8)

[0086] in, This represents the first reward function; This represents the second reward function; This indicates the number of connected autonomous vehicles in the target intersection segment during the current control period.

[0087] In step S102, based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network, and corresponding target network of the traffic light agent and each connected autonomous vehicle agent are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents.

[0088] Furthermore, embodiments of this application also require the use of the QMIX multi-agent reinforcement learning algorithm to perform environmental interaction, local Q-value network, and hybrid network parameter update operations, thereby generating the target global Q-value corresponding to all agents.

[0089] Optionally, in one embodiment of this application, based on a globally shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network, and corresponding target network for the traffic light agent and each connected autonomous vehicle agent are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are then used to generate the target global Q-value for all agents. This includes: initializing the local Q-value network, global Q-value network, and target network for each traffic light agent and each connected autonomous vehicle agent. The local Q-value network is initialized to obtain its initial network parameters. Based on the hyperparameter network, a hybrid network is established, and the output Q-values ​​of the local Q-value network are nonlinearly combined using the hybrid network and the initialized network parameters to establish the global Q-value network. Based on the hyperparameter network and the local Q-value network, the corresponding target hyperparameter network and target local Q-value network are determined, and the target hyperparameter network and target local Q-value network are initialized. The target global Q-value network is established using the initialized target hyperparameter network and target local Q-value network, and multiple hyperparameters of the target global Q-value network are determined. Based on multiple hyperparameters, the initialized local Q-value network, the hybrid network, and the preset parameters... The `-greedy` strategy determines the random actions corresponding to the traffic light agent and each connected autonomous vehicle agent; it acquires the observation states corresponding to the traffic light agent and each connected autonomous vehicle agent, calculates the target action based on the random actions and observation states, and executes the target action through each agent to obtain the corresponding environmental reward; based on the random actions, environmental reward, observation states, and a preset discount factor, it constructs a target loss function and updates the local Q-value network and hyperparameter network using the target loss function; it updates the target hyperparameter network and target local Q-value network according to the updated local Q-value network and hyperparameter network to determine the target Q-network parameters corresponding to each agent, and calculates the target global Q-value corresponding to all agents through the target Q-network parameters.

[0090] In specific implementation, embodiments of this application can be configured such that the current road has one traffic light system and Each CAV vehicle needs to be controlled; therefore, the specific process of model training and parameter updating in this embodiment is as follows:

[0091] 1. Initialization:

[0092] 1) Local Q-value network initialization:

[0093] For the traffic light agent and each CAV agent Initialize the local Q-value network, including and ;in, and These represent traffic light agents and CAV agents, respectively. Local Q-value network parameters;

[0094] 2) Global Q-value network initialization:

[0095] Building Hybrid Networks Its parameters are derived from the hyperparameter network. Output obtained; Connect the traffic light agent with each CAV agent Q-value network , ,…, ,…, The output Q values ​​are nonlinearly combined to form a global Q-value network, as shown in the following equation:

[0096] (9)

[0098] in, This represents the observation state of all agents; Represents the actions of all intelligent agents; Representing hyperparameter networks Parameters;

[0099] 3) Target network initialization:

[0100] Initialize the target Q-value network of the traffic light agent. CAV intelligent agent Target Q-value network and target hyperparameter network ;Target Global Q-value Network Depend on , and The structure is as follows:

[0101] (10)

[0102] in, Represent the target Q-network and CAV agent of the traffic light agent, respectively. The parameters of the target Q-network; These represent the parameters of the target hyperparameter network;

[0103] 4) Hyperparameter settings:

[0104] Set the learning rate of the traffic light agent. CAV intelligent agent learning rate The learning rate of hyperparameter networks Discount factor Exploration rate Target network update step size .

[0105] 2. Environmental Interaction:

[0106] In each round of interaction, the traffic light agent can select actions according to its strategy. ,use The -greedy strategy is shown in the following formula:

[0107] (11)

[0108] in, Indicates Probability-based selection of random actions.

[0109] Similarly, in the embodiments of this application, for CAV agents... Choose actions based on its strategy. ,use The -greedy strategy is shown in the following formula:

[0110] (12)

[0111] exist At any given moment, the traffic light agent and the CAV agent, based on the observed state... Calculate actions It interacts with the traffic environment and performs actions. Receive environmental awards and the next state and will experience Experience replay pool exists middle.

[0112] 3. Network parameter update:

[0113] Subsequently, the loss function can be constructed in the embodiments of this application as follows. As shown in the following formula:

[0114] (13)

[0115] Based on the above loss function, the Q-value network parameters of the traffic light agent and the CAV agent can be updated in this embodiment.

[0116] The Q-value network for the traffic light agent is updated as follows:

[0117] (14)

[0118] in, Represents the loss function For parameters Find the gradient.

[0119] For CAV agents The Q-value network is updated as follows:

[0120] (15)

[0121] in, Represents the loss function For parameters Find the gradient.

[0122] For hyperparameter networks, the update method is as follows:

[0123] (16)

[0124] in, Represents the loss function For parameters Find the gradient.

[0125] For the target Q-value network of the traffic light agent, the update method is as follows:

[0126] (17)

[0127] For CAV agents The target Q-value network is updated as follows:

[0128] (18)

[0129] For the target hyperparameter network, the update method is as follows:

[0130] (19)

[0131] In step S103, the control command corresponding to each connected autonomous vehicle is determined based on the target global Q value, and each connected autonomous vehicle is controlled to perform corresponding driving operations through the control command to pass through the target intersection.

[0132] It is understood that the embodiments of this application can adopt a hierarchical approach to construct a multi-traffic guidance control scheme. The upper layer can employ deep reinforcement learning to determine traffic light control strategies and calculate guidance information for CAVs, which is then sent to the CAVs requiring control via the roadside control center. Secondly, the lower layer, based on the guidance information, uses model predictive control to calculate control commands for the CAVs, such as... Figure 6 As shown, this reduces computational complexity.

[0133] Optionally, in one embodiment of this application, determining the control command corresponding to each connected autonomous vehicle based on the target global Q value includes: determining the vehicle state and target direction corresponding to each connected autonomous vehicle based on the target global Q value, and obtaining the traffic light timing strategy corresponding to the target intersection segment, so as to determine the guidance information of each connected autonomous vehicle based on the vehicle state, target direction and traffic light timing strategy; and sending the guidance information to each connected autonomous vehicle so that each connected autonomous vehicle can calculate the corresponding control command based on the guidance information and preset vehicle kinematic constraints.

[0134] In actual operation, the roadside control center can determine the traffic light timing scheme based on the status and target direction of all vehicles at the intersection, and calculate the guidance information for CAVs. This guidance information, including lane-changing instructions and target speeds for each CAV, is then sent to the CAVs. Upon receiving the guidance information, the CAVs can calculate their control commands based on the guidance information and vehicle kinematic constraints.

[0135] Therefore, the embodiments of this application fully consider the lane-changing requirements of CAV vehicles, and introduce the kinematic model of CAV vehicles through hierarchical control, thereby reducing the complexity of problem solving, improving computational efficiency, and meeting the requirements of real-time applications.

[0136] According to the connected hybrid multi-traffic guidance and control method proposed in this application, the vehicle control zone corresponding to the target intersection is determined, and based on a preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous vehicle agent are constructed in the vehicle control zone. A global shared reward function is established for the traffic light agent and at least one connected autonomous vehicle agent. Based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network, and corresponding target network for the traffic light agent and each connected autonomous vehicle agent are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents. Control commands are determined for each connected autonomous vehicle based on the target global Q-value, and each connected autonomous vehicle is controlled to perform corresponding driving operations to pass through the target intersection. This application can improve the traffic efficiency of urban intersections in a connected environment by coordinating the control of traffic lights and CAVs at the intersection level.

[0137] Secondly, the connected hybrid multi-vehicle flow guidance and control device proposed according to the embodiments of this application will be described with reference to the accompanying drawings.

[0138] Figure 7 This is a block diagram of a networked hybrid multi-traffic guidance and control device according to an embodiment of this application.

[0139] like Figure 7 As shown, the connected hybrid multi-vehicle flow guidance and control device 10 includes: a construction module 100, a generation module 200, and an execution module 300.

[0140] The construction module 100 is used to determine the vehicle control zone corresponding to the target intersection segment, and based on the preset deep reinforcement learning control strategy, construct the traffic light agent and at least one connected autonomous vehicle agent in the vehicle control zone, and establish the global shared reward function corresponding to the traffic light agent and at least one connected autonomous vehicle agent.

[0141] The generation module 200 is used to initialize the local Q-value network, global Q-value network, and corresponding target network of the traffic light agent and each connected autonomous vehicle agent based on the globally shared reward function and the preset QMIX multi-agent reinforcement learning algorithm. Based on the initialized Q-value network, global Q-value network, and target network, it performs environmental interaction operations on each agent to update the local Q-value network, the pre-built hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents.

[0142] The execution module 300 is used to determine the control command corresponding to each connected autonomous vehicle based on the target global Q value, and to control each connected autonomous vehicle to perform corresponding driving operations through the control command to pass through the target intersection.

[0143] Optionally, in one embodiment of this application, the construction module 100 includes: a partitioning unit, a first determining unit, a first establishing unit, a first calculating unit, an acquiring unit, and a second establishing unit.

[0144] The partitioning unit is used to divide the vehicle control area into equally spaced grids to obtain multiple grids, and to assign values ​​to each grid according to a preset grid assignment strategy to construct the corresponding first state space.

[0145] The first determining unit is used to determine the corresponding first action space based on the preset four-phase traffic light control strategy, and to obtain the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection segment, so as to determine the corresponding first reward function based on the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road.

[0146] The first establishment unit is used to construct a traffic light agent based on the first state space, the first action space, and the first reward function.

[0147] The first computing unit is used to construct a second state space corresponding to each connected autonomous vehicle based on a preset state space expression, and to calculate the target speed and target lane of each connected autonomous vehicle, so as to determine the second action space corresponding to each connected autonomous vehicle through the target speed and target lane.

[0148] The acquisition unit is used to acquire the minimum speed limit of the target intersection and determine the second reward function based on the number of connected autonomous vehicles, the minimum speed limit and the maximum speed limit of the road. Based on the second state space, the second action space and the second reward function, the intelligent agent of the connected autonomous vehicle corresponding to each connected autonomous vehicle is constructed.

[0149] The second establishment unit is used to construct a globally shared reward function based on the first reward function and the second reward function.

[0150] Optionally, in one embodiment of this application, the generation module 200 includes: a first initialization unit, a nonlinear combination unit, a second initialization unit, a second determination unit, a third determination unit, a second calculation unit, an update unit, and a third calculation unit.

[0151] The first initialization unit is used to initialize the local Q-value network corresponding to the traffic light agent and each connected autonomous vehicle agent, so as to obtain the initial network parameters corresponding to the local Q-value network.

[0152] Nonlinear combination units are used to establish hybrid networks based on hyperparameter networks, and to nonlinearly combine the output Q values ​​of local Q-value networks through hybrid networks and initialized network parameters to establish global Q-value networks.

[0153] The second initialization unit is used to determine the corresponding target hyperparameter network and target local Q-value network based on the hyperparameter network and the local Q-value network, and to initialize the target hyperparameter network and target local Q-value network.

[0154] The second determining unit is used to establish a target global Q-value network through the initialized target hyperparameter network and target local Q-value network, and to determine multiple hyperparameters of the target global Q-value network.

[0155] The third determining unit is used to determine the network based on multiple hyperparameters, the initialized local Q-value network, the hybrid network, and a preset parameter. The -greedy strategy determines the random actions corresponding to the traffic light agent and each connected autonomous vehicle agent.

[0156] The second computing unit is used to acquire the observation state corresponding to the traffic light agent and each connected autonomous vehicle agent, to calculate the target action based on the random action and the observation state, and to execute the target action through each agent to obtain the corresponding environmental reward.

[0157] The update unit is used to construct the target loss function based on random actions, environmental rewards, observation states, and preset discount factors, and to update the local Q-value network and hyperparameter network using the target loss function.

[0158] The third computational unit is used to update the target hyperparameter network and the target local Q-value network based on the updated local Q-value network and hyperparameter network, so as to determine the target Q-network parameters corresponding to each agent, and to calculate the target global Q-value corresponding to all agents through the target Q-network parameters.

[0159] Optionally, in one embodiment of this application, the execution module 300 includes a fourth determining unit and a sending unit.

[0160] The fourth determining unit is used to determine the vehicle state and target direction of each connected autonomous vehicle based on the target global Q value, and to obtain the traffic light timing strategy corresponding to the target intersection segment, so as to determine the guidance information of each connected autonomous vehicle based on the vehicle state, target direction and traffic light timing strategy.

[0161] The transmitting unit is used to send guidance information to each connected autonomous vehicle, so that each connected autonomous vehicle can calculate the corresponding control commands based on the guidance information and preset vehicle kinematic constraints.

[0162] Optionally, in one embodiment of this application, the mathematical expression of the globally shared reward function is:

[0163]

[0164] in, This represents the first reward function; This represents the second reward function; This indicates the number of connected autonomous vehicles in the target intersection segment during the current control period.

[0165] It should be noted that the foregoing explanation of the embodiment of the connected hybrid multi-vehicle flow guidance and control method also applies to the connected hybrid multi-vehicle flow guidance and control device of this embodiment, and will not be repeated here.

[0166] The connected hybrid multi-traffic guidance and control device proposed in this application includes a construction module 100, used to determine the vehicle control zone corresponding to the target intersection segment, and based on a preset deep reinforcement learning control strategy, construct a traffic light agent and at least one connected autonomous vehicle agent in the vehicle control zone, and establish a global shared reward function corresponding to the traffic light agent and at least one connected autonomous vehicle agent; and a generation module 200, used to generate a local Q-value network corresponding to the traffic light agent and each connected autonomous vehicle agent based on the global shared reward function and a preset QMIX multi-agent reinforcement learning algorithm. The system initializes the network, global Q-value network, and corresponding target network. Based on the initialized Q-value network, global Q-value network, and target network, it performs environmental interaction operations on each agent to update the local Q-value network, the pre-built hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are then used to generate the target global Q-value for all agents. The execution module 300 determines the control command corresponding to each connected autonomous vehicle based on the target global Q-value and controls each connected autonomous vehicle to perform corresponding driving operations to pass through the target intersection. This application can improve the traffic efficiency of intersections at the system level by coordinating the control of intersection traffic lights and CAVs in a connected environment, specifically for urban intersection traffic scenarios.

[0167] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0168] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0169] When the processor 802 executes the program, it implements the networked hybrid multi-vehicle flow guidance and control method provided in the above embodiments.

[0170] Furthermore, electronic devices also include:

[0171] Communication interface 803 is used for communication between memory 801 and processor 802.

[0172] The memory 801 is used to store computer programs that can run on the processor 802.

[0173] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0174] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0175] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0176] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0177] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described networked hybrid multi-vehicle flow guidance and control method.

[0178] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described connected hybrid multi-vehicle flow guidance and control method.

[0179] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0180] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0181] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0182] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0183] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0184] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0185] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0186] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A networked hybrid multi-traffic guidance and control method, characterized in that, Includes the following steps: The vehicle control zone corresponding to the target intersection is determined, and based on the preset deep reinforcement learning control strategy, a traffic light agent and at least one connected autonomous vehicle agent are constructed in the vehicle control zone, and a global shared reward function is established for the traffic light agent and the at least one connected autonomous vehicle agent. Based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, the local Q-value network, global Q-value network, and corresponding target network of the traffic light agent and each connected autonomous vehicle agent are initialized. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents. Based on the target global Q value, the control command corresponding to each connected autonomous vehicle is determined, and the control command is used to control each connected autonomous vehicle to perform corresponding driving operations in order to pass through the target intersection. The process involves determining the vehicle control zone corresponding to the target intersection segment, constructing a traffic light agent and at least one connected autonomous vehicle agent within the vehicle control zone based on a preset deep reinforcement learning control strategy, and establishing a globally shared reward function for the traffic light agent and the at least one connected autonomous vehicle agent, including: The vehicle control area is divided into equally spaced grids to obtain multiple grids, and each grid is assigned a value according to a preset grid assignment strategy to construct the corresponding first state space. Based on the preset four-phase traffic light control strategy, the corresponding first action space is determined, and the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road in the target intersection are obtained, so as to determine the corresponding first reward function according to the number of human-driven vehicles, the number of connected autonomous vehicles, and the maximum speed limit of the road. The traffic light agent is constructed based on the first state space, the first action space, and the first reward function; Based on a preset state space expression, a second state space corresponding to each connected autonomous vehicle is constructed, and the target speed and target lane of each connected autonomous vehicle are calculated, so as to determine the second action space corresponding to each connected autonomous vehicle through the target speed and the target lane. The minimum speed limit of the target intersection is obtained, and a second reward function is determined based on the number of connected autonomous vehicles, the minimum speed limit, and the maximum speed limit. Based on the second state space, the second action space, and the second reward function, a connected autonomous vehicle agent corresponding to each connected autonomous vehicle is constructed. Based on the first reward function and the second reward function, the globally shared reward function is constructed; The step of determining the control command corresponding to each connected autonomous vehicle based on the target global Q value includes: The vehicle state and target direction of each connected autonomous vehicle are determined based on the target global Q value, and the traffic light timing strategy corresponding to the target intersection is obtained. Based on the vehicle state, the target direction and the traffic light timing strategy, the guidance information of each connected autonomous vehicle is determined, wherein the guidance information includes lane change instructions and target speed. The guidance information is sent to each of the connected autonomous vehicles, so that each connected autonomous vehicle can use model predictive control to calculate the corresponding control command based on the guidance information and preset vehicle kinematic constraints. The control command includes acceleration and steering angle.

2. The method according to claim 1, characterized in that, The method, based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm, initializes the local Q-value network, global Q-value network, and corresponding target network for the traffic light agent and each connected autonomous vehicle agent. Based on the initialized Q-value network, global Q-value network, and target network, environmental interaction operations are performed on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. Furthermore, the updated local Q-value network, hyperparameter network, and target network are used to generate the target global Q-value for all agents, including: The local Q-value networks corresponding to the traffic light agent and each connected autonomous vehicle agent are initialized to obtain the initialization network parameters corresponding to the local Q-value networks. Based on the hyperparameter network, a hybrid network is established, and the output Q value of the local Q-value network is nonlinearly combined through the hybrid network and the initial network parameters to establish the global Q-value network; Based on the hyperparameter network and the local Q-value network, the corresponding target hyperparameter network and target local Q-value network are determined, and the target hyperparameter network and the target local Q-value network are initialized. A target global Q-value network is established by initializing the target hyperparameter network and the target local Q-value network, and multiple hyperparameters of the target global Q-value network are determined. Based on the aforementioned hyperparameters, the initialized local Q-value network and hybrid network, and the preset... -greedy strategy, to determine the random actions corresponding to the traffic light agent and each connected autonomous vehicle agent; The observation states corresponding to the traffic light agent and each connected autonomous vehicle agent are obtained, and the target action is calculated based on the random action and the observation state. The target action is then executed by each agent to obtain the corresponding environmental reward. Based on the random action, the environmental reward, the observation state, and the preset discount factor, a target loss function is constructed, and the target loss function is used to update the local Q-value network and the hyperparameter network. The target hyperparameter network and the target local Q-value network are updated based on the updated local Q-value network and hyperparameter network to determine the target Q-network parameters corresponding to each agent, so as to calculate the target global Q-value corresponding to all agents through the target Q-network parameters.

3. The method according to claim 1, characterized in that, The mathematical expression for the globally shared reward function is: in, This represents the first reward function; This represents the second reward function; This indicates the number of connected autonomous vehicles in the target intersection segment during the current control cycle.

4. A networked hybrid multi-traffic guidance and control device, characterized in that, include: The module is used to determine the vehicle control zone corresponding to the target intersection segment, and based on the preset deep reinforcement learning control strategy, construct the traffic light agent and at least one connected autonomous vehicle agent in the vehicle control zone, and establish the global shared reward function corresponding to the traffic light agent and the at least one connected autonomous vehicle agent. The generation module is used to initialize the local Q-value network, global Q-value network, and corresponding target network of the traffic light agent and each connected autonomous vehicle agent based on the global shared reward function and the preset QMIX multi-agent reinforcement learning algorithm. Based on the initialized Q-value network, global Q-value network, and target network, the module performs environmental interaction operations on each agent to update the local Q-value network, the pre-constructed hyperparameter network, and the target network. The module also generates the target global Q-value for all agents through the updated local Q-value network, the hyperparameter network, and the target network. The execution module is used to determine the control command corresponding to each connected autonomous vehicle based on the target global Q value, and control each connected autonomous vehicle to perform corresponding driving operations through the control command to pass through the target intersection. The building module includes: The partitioning unit is used to divide the vehicle control area into equally spaced grids to obtain multiple grids, and to assign values ​​to each grid according to a preset grid assignment strategy to construct the corresponding first state space. The first determining unit is used to determine the corresponding first action space based on a preset traffic light four-phase control strategy, and to obtain the number of human-driven vehicles, the number of connected autonomous vehicles and the maximum speed limit of the road in the target intersection segment, so as to determine the corresponding first reward function based on the number of human-driven vehicles, the number of connected autonomous vehicles and the maximum speed limit of the road. The first establishment unit is used to construct the traffic light agent based on the first state space, the first action space, and the first reward function; The first calculation unit is used to construct a second state space corresponding to each connected autonomous vehicle based on a preset state space expression, and to calculate the target speed and target lane of each connected autonomous vehicle, so as to determine the second action space corresponding to each connected autonomous vehicle through the target speed and the target lane. The acquisition unit is used to acquire the minimum speed limit of the target intersection, and determine a second reward function based on the number of connected autonomous vehicles, the minimum speed limit and the maximum speed limit of the road, so as to construct the connected autonomous vehicle intelligent agent corresponding to each connected autonomous vehicle based on the second state space, the second action space and the second reward function; The second establishment unit is used to construct the globally shared reward function based on the first reward function and the second reward function; The execution module includes: The fourth determining unit is used to determine the vehicle state and target direction corresponding to each connected autonomous vehicle based on the target global Q value, and to obtain the traffic light timing strategy corresponding to the target intersection segment, so as to determine the guidance information of each connected autonomous vehicle based on the vehicle state, the target direction and the traffic light timing strategy, wherein the guidance information includes lane change instructions and target speed; The sending unit is used to send the guidance information to each connected autonomous vehicle, so that each connected autonomous vehicle can use model predictive control to calculate the corresponding control command based on the guidance information and preset vehicle kinematic constraints. The control command includes acceleration and steering angle.

5. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the connected hybrid multi-flow guidance control method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the connected hybrid multi-flow guidance control method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the connected hybrid multi-traffic guidance and control method as described in any one of claims 1-3.

Images