Different-intelligence network-connected vehicle collaborative decision-making method and system for mixed traffic scene

Abstract

The invention discloses a mixed traffic scene-oriented different-intelligence network-connected vehicle collaborative decision-making method and system, and relates to the technical field of multi-vehicle collaborative decision-making, and the method comprises the specific steps: building a mixed traffic simulation environment for multi-agent reinforcement learning training; constructing a different intelligence network vehicle physical model, and performing function module configuration based on the grade of each vehicle; the different-intelligence networked vehicle physical model comprises a sensing module, a decision planning module, a control module and a communication module. A heterogeneous multi-agent deep reinforcement learning method is adopted to carry out cooperative training on the different-intelligence network vehicle simulation queue based on a mixed traffic simulation environment, and an optimal cooperative decision strategy is obtained; and deploying the optimal collaborative decision-making strategy to the different intelligence network connection vehicle queue for collaborative decision-making information generation. Through the parameter sharing and independent optimization mechanism of the high-level and low-level intelligent vehicle strategy and state value networks, hierarchical cooperation between different-intelligence vehicles is realized; and the overall traffic efficiency in the complex mixed traffic scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-vehicle collaborative decision-making technology, and more specifically to a collaborative decision-making method and system for intelligent connected vehicles in mixed traffic scenarios. Background Art

[0002] Currently, intelligent connected vehicle technology has entered the stage of large-scale testing and application. Vehicle cooperative control technology based on V2X communication has significantly improved the safety and efficiency of convoy driving. Existing technical solutions often use isomorphism assumptions to construct collaborative decision-making systems, optimizing vehicle group decision-making through methods such as multi-agent reinforcement learning. Typical functional applications such as formation cruising and cooperative obstacle avoidance have been implemented in closed test scenarios. In particular, in the field of highway convoy control, collaborative methods based on global state perception and centralized decision-making have formed a complete technical system. Some solutions reduce computational complexity through parameter sharing mechanisms and employ algorithms such as proximal policy optimization to achieve rapid policy convergence.

[0003] However, existing technologies have three core flaws: First, collaborative decision-making models generally presuppose that fleet vehicles have the same level of intelligence and perception capabilities, and fail to consider the mixed traffic scenarios of heterogeneous vehicles caused by differences in the penetration rate of intelligent equipment in actual traffic environments, resulting in decision-making failures when low-level intelligent vehicles are connected; second, traditional parameter sharing mechanisms are difficult to adapt to the functional heterogeneity of heterogeneous intelligent vehicles, and there is a parameter space mismatch between the complex strategy network of high-level vehicles and the lightweight execution modules of low-level vehicles; third, existing communication delay compensation methods are mainly designed for homogeneous vehicles, and do not consider the asymmetric delay characteristics of the decision-execution link between heterogeneous intelligent vehicles, resulting in a significantly increased risk of disorder in the collaborative action sequence.

[0004] Therefore, how to build a collaborative decision-making framework for connected vehicles that supports heterogeneous intelligence levels in a mixed traffic environment and achieve safe and efficient collaboration of heterogeneous vehicle fleets under conditions of limited perception, heterogeneous decision-making, and time delay differences is an urgent problem that technical personnel in this field need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a collaborative decision-making method and system for intelligent connected vehicles in mixed traffic scenarios, which overcomes the above-mentioned defects.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A collaborative decision-making method for intelligent connected vehicles in mixed traffic scenarios, with the following specific steps:

[0008] Establish a mixed traffic simulation environment for multi-agent reinforcement learning training, which includes a road model and randomly distributed traffic vehicles;

[0009] Constructing a physical model of intelligent connected vehicles and configuring functional modules based on the level of each vehicle; the physical model of intelligent connected vehicles includes a perception module, a decision-making and planning module, a control module, and a communication module;

[0010] A heterogeneous multi-agent deep reinforcement learning method is used to collaboratively train a simulated platoon of heterogeneous connected vehicles based on the mixed traffic simulation environment to obtain the optimal collaborative decision-making strategy.

[0011] The optimal collaborative decision-making strategy is deployed to a fleet of intelligent connected vehicles to generate collaborative decision-making information.

[0012] Furthermore, the levels of vehicles are divided into high-level and low-level categories. High-level intelligent vehicles are equipped with a perception module, a decision-making and planning module, a control module and a communication module; low-level intelligent vehicles are equipped with a control module and a communication module.

[0013] Furthermore, the steps for obtaining the optimal collaborative decision-making strategy are:

[0014] Step 31: Initialize the simulation environment and reinforcement learning parameter space, where the parameter space includes the global state spliced from local observations of high-level intelligent vehicles, the action set of the simulated platoon of heterogeneous intelligent connected vehicles, rewards, policy networks, and value networks.

[0015] Step 32: Collect state-action-reward data trajectories through the interaction between the simulated fleet of intelligent connected vehicles and the simulation environment;

[0016] Step 33: Calculate a loss function based on the collected data, and optimize the parameters of the collaborative strategy of the intelligent connected vehicles according to the loss function;

[0017] Step 34: Repeat steps 32 and 33 until the convergence condition is reached and the optimal collaborative decision-making strategy is obtained.

[0018] Furthermore, the expression of the global state is:

[0019]

[0020] Where, O Hi represents the local observation of the road state by the i-th high-level intelligent vehicle; O Lj represents the local observation of the road state by the jth low-level intelligent vehicle; m represents the number of high-level intelligent vehicles; n represents the number of low-level intelligent vehicles; Represents the global state outside the observation range of all high-level intelligent vehicles.

[0021] Furthermore, the reward is expressed as:

[0022]

[0023] Where R Hi represents the reward of the i-th high-level intelligent vehicle; R Lj represents the reward for the j-th low-level intelligent vehicle; m represents the number of high-level intelligent vehicles; n represents the number of low-level intelligent vehicles.

[0024] Furthermore, the expression of the state-action-reward data trajectory is:

[0025]

[0026] Where, τ k represents the data of the kth vehicle from time step t = 0 to time step T; s t and r t Represent the global state and reward of step t respectively; s t+1 Represents the global state at step t+1; and They represent the observation and action of the kth vehicle at time t respectively; m represents the number of high-level intelligent vehicles; and n represents the number of low-level intelligent vehicles.

[0027] Furthermore, the policy network adopts the same level parameter sharing mechanism, specifically:

[0028] The sharing mechanism expression of the policy network of the high-level intelligent vehicle is:

[0029]

[0030] The sharing mechanism expression of the policy network of the low-level intelligent vehicle is:

[0031]

[0032] Where θ Hi and θ Lj They represent the policy network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively; θ H and θ L Represent the policy network parameters shared by high-level intelligent vehicles and low-level intelligent vehicles respectively.

[0033] Furthermore, the value network adopts the same level parameter sharing mechanism, specifically:

[0034] The sharing mechanism expression of the value network of the high-level intelligent vehicle is:

[0035]

[0036] The sharing mechanism expression of the value network of the high-level intelligent vehicle is:

[0037]

[0038] Where, ω Hi and ω Lj They represent the state network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively; ω H and ω L Represent the value network parameters shared by high-level intelligent vehicles and low-level intelligent vehicles respectively.

[0039] Furthermore, the loss function includes a joint loss function, which is expressed as:

[0040] L H (θ,ω)=L clip,H (θ)+c1·L value,H (ω)-c2·L entropy,H (θ);

[0041] L L (θ,ω)=L clip,L (θ)+c1·L value,L (ω)-c2·L entropy,L (θ);

[0042] Where, L H (θ, ω) represents the high-level joint loss function; L L (θ, ω) represents the low-level joint loss function; L clip，H (θ) represents the high-level strategy optimization loss; L clip，L (θ) represents the low-level policy optimization loss; L value，H (ω) represents the high-level value function loss; L value，L (ω) represents the low-level value function loss; L entropy，H (θ) represents the high-level policy entropy loss; L entropy，H (θ) represents the low-level policy entropy loss; c1 and c2 are the weights of the value function loss and policy entropy loss, respectively.

[0043] A collaborative decision-making system for intelligent connected vehicles in mixed traffic scenarios, including:

[0044] A simulation environment construction module is used to build a mixed traffic simulation environment for multi-agent reinforcement learning training. The simulation environment includes a road model and randomly distributed traffic vehicles;

[0045] A vehicle model configuration module is used to construct a physical model of a connected vehicle and configure functional modules based on the level of each vehicle; the physical model of the connected vehicle includes a perception module, a decision-making and planning module, a control module, and a communication module;

[0046] A reinforcement learning training module is used to collaboratively train a simulated platoon of heterogeneous connected vehicles based on the mixed traffic simulation environment using a heterogeneous multi-agent deep reinforcement learning method to obtain an optimal collaborative decision-making strategy;

[0047] The strategy deployment module is used to deploy the optimal collaborative decision-making strategy to the intelligent connected vehicle fleet to generate collaborative decision-making information.

[0048] The above technical solutions show that the present invention provides a method and system for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios. Compared with the existing technology, it has the following beneficial effects:

[0049] 1. Through parameter sharing and independent optimization of high- and low-level intelligent vehicle strategies and state-value networks, hierarchical collaboration between vehicles with different intelligence is achieved. High-level vehicles guide platoon collaborative decision-making based on perception information, while low-level vehicles adapt to dynamic environmental changes by responding to decision instructions. This effectively reduces inter-vehicle decision-making conflicts, improves the overall intelligence level of the platoon, optimizes the spatiotemporal distribution of traffic flow, and significantly enhances overall traffic efficiency in complex mixed traffic scenarios.

[0050] 2. Through a hierarchical sharing mechanism, the multiple independent policy networks of traditional heterogeneous multi-agent reinforcement learning are compressed into two shared policy networks, reducing the number of parameters to be trained while retaining the heterogeneous characteristics, improving training efficiency and policy convergence stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses a collaborative decision-making method for heterogeneous intelligent connected vehicles in mixed traffic scenarios. Figure 1 As shown, the specific steps are:

[0055] Step 1: Establish a mixed traffic simulation environment for multi-agent reinforcement learning training, which includes a road model and randomly distributed traffic vehicles;

[0056] Step 2: Build a physical model of the intelligent connected vehicle and configure functional modules based on the level of each vehicle; the physical model of the intelligent connected vehicle includes a perception module, a decision-making and planning module, a control module, and a communication module;

[0057] Step 3: Use a heterogeneous multi-agent deep reinforcement learning method to collaboratively train a simulated platoon of heterogeneous connected vehicles in a mixed traffic simulation environment to obtain the optimal collaborative decision-making strategy;

[0058] Step 4: Deploy the optimal collaborative decision-making strategy to the intelligent connected vehicle fleet to generate collaborative decision-making information.

[0059] Furthermore, a road model is established in the mixed traffic simulation environment in step 1, including road types such as highways and urban expressways, and traffic vehicles are added to the simulation environment according to a certain number and vehicle density. The traffic vehicles travel at a certain initial velocity and have a certain degree of behavioral randomness.

[0060] The heterogeneous intelligent connected vehicle models established in Step 2 exhibit physical differences in their perception, decision-making, and control modules. High-level intelligent vehicles possess environmental perception, autonomous decision-making, and real-time control. Low-level intelligent vehicles lack these capabilities and can control their vehicles based on decisions issued by high-level intelligent vehicles, but their actions experience delays due to communication. Furthermore, all participating heterogeneous intelligent vehicles possess network communication capabilities.

[0061] In step 3, a heterogeneous multi-agent deep reinforcement learning method is used to simulate a platoon of heterogeneous connected vehicles. The heterogeneous connected vehicles are considered to be multiple heterogeneous agents, and the multi-vehicle collaborative process is considered a fully cooperative relationship. The heterogeneous multi-agent deep reinforcement learning algorithm is initialized. The heterogeneous vehicles interact with the environment to collect data. The loss function and error are calculated for the reinforcement learning strategy. The parameters of the heterogeneous multi-agent are updated using gradient descent. These steps are repeated, iterating continuously to achieve the optimal collaborative decision-making strategy.

[0062] In step 4, a heterogeneous multi-agent reinforcement learning method is used to deploy the strategy to the heterogeneous intelligent connected vehicle. After the training is completed, the strategy is deployed to the heterogeneous intelligent connected vehicle and the strategy parameters are no longer updated, and the strategy is evaluated.

[0063] In one embodiment, the levels of vehicles are divided into high-level and low-level categories. High-level intelligent vehicles are equipped with a perception module, a decision-making and planning module, a control module, and a communication module; low-level intelligent vehicles are equipped with a control module and a communication module.

[0064] Furthermore, the physical model of the heterogeneous intelligent connected vehicle in step 2 refers to: vehicles of different intelligence levels have functional modules with different capabilities due to the different hardware devices they carry. Specifically, they include perception modules, decision-making and planning modules, control modules, and communication modules. High-level intelligent vehicles are equipped with at least cameras and high-performance computing platform equipment, which can realize the recognition and tracking of traffic vehicles in the environment; equipped with a high-performance computing platform to realize the calculation of decision-making and planning algorithms; equipped with a chassis system with wire control capabilities to execute continuous control signals in real time; equipped with a communication unit to carry out simple communication with other vehicles. Low-level intelligent vehicles are not equipped with cameras or high-performance computing platforms, and cannot realize the recognition and tracking of traffic vehicles in the environment; at the same time, they cannot realize the calculation of decision-making and planning algorithms; equipped with a chassis system with wire control capabilities to execute continuous control signals or enter a low-level assisted driving mode; equipped with a communication unit to carry out simple communication with other vehicles.

[0065] In one embodiment, the steps for obtaining the optimal collaborative decision-making strategy are:

[0066] Step 31: Initialize the simulation environment and reinforcement learning parameter space. The parameter space includes the global state composed of local observations of high-level intelligent vehicles, the action set of the simulated platoon of heterogeneous intelligent connected vehicles, rewards, policy networks, and value networks.

[0067] Step 32: Collect state-action-reward data trajectories through the interaction between the simulated fleet of intelligent connected vehicles and the simulation environment;

[0068] Step 33: Calculate a loss function based on the collected data, and optimize the parameters of the collaborative strategy of the intelligent connected vehicles according to the loss function;

[0069] Step 34: Repeat steps 32 and 33 until the convergence condition is reached and the optimal collaborative decision-making strategy is obtained.

[0070] Furthermore, in step 3, the multi-agent reinforcement learning centralized training method is used to train the simulated platoon of connected vehicles with different intelligence. Taking the same-strategy reinforcement learning method as an example, the specific steps include:

[0071] In step 31, the simulated traffic training environment is initialized, and the simulated queue of hetero-intelligent connected vehicles performs initial observations of the environment; specifically, the initialization of the simulated traffic training environment includes resetting the simulated road state, resetting the state of the traffic vehicles, setting the initial state of the simulated queue of hetero-intelligent connected vehicles, and initializing the parameter space of multi-agent reinforcement learning according to the state of the simulated queue of hetero-intelligent connected vehicles.

[0072] Intelligent connected vehicles are randomly generated within a certain distance range, and traffic vehicles are randomly distributed in front of and behind the simulated queue of intelligent connected vehicles. The reinforcement learning parameter space should at least include: state S, action A, reward R, and policy π.

[0073] The state S represents the global state of the simulated platoon of heterogeneous intelligent connected vehicles, which is composed of local observations of high-level intelligent vehicles:

[0074]

[0075] Where, O Hi represents the local observation of the road state by the i-th high-level intelligent vehicle, O Lj represents the local observation of the road state by the jth low-level intelligent vehicle. In reality, low-level intelligent vehicles do not have perception capabilities, and their local observations are obtained by cutting and splicing the local observations of nearby high-level intelligent vehicles. m represents the number of high-level intelligent vehicles, n represents the number of low-level intelligent vehicles, It represents the global state outside the observation range of all high-level intelligent vehicles and can be used to improve the accuracy of value estimation.

[0076] In this example, the high-level intelligent vehicle observation dimension is defined as (8, 5), which allows observation of the motion parameters of eight vehicles. Each vehicle's motion parameters include the presence of a vehicle at that location, its longitudinal relative position, its lateral relative position, its longitudinal relative velocity, and its lateral relative velocity. The global state dimension is defined as (50, 5), representing the motion parameters of 50 vehicles in the environment.

[0077] Action A represents the action of the simulated platoon of intelligent connected vehicles:

[0078] A=[A H1 ,…,A Hm ,A L1 ,…,A Ln ] (2);

[0079] Where A Hi represents the action of the i-th high-level intelligent vehicle, A Lj represents the action of the jth low-level intelligent vehicle. Vehicle actions can be continuous-valued vehicle control signals, such as steering wheel angle, throttle opening, brake opening, etc., or discrete vehicle control action elements, such as changing lanes left, driving straight at a constant speed, changing lanes right, accelerating straight, and decelerating. Action elements can be further decomposed into vehicle control signals by the control system.

[0080] The reward R represents the environmental reward obtained by the simulated platoon of intelligent connected vehicles after performing an action. In this embodiment, the relationship between intelligent vehicles is assumed to be fully cooperative, and the reward is defined as follows:

[0081]

[0082] Where R Hi represents the reward of the i-th high-level intelligent vehicle, R Lj represents the reward of the j-th low-level intelligent vehicle.

[0083] In this example, the reward is set to include the collision penalty r collision , safety distance penalty r safe_distance , traffic efficiency reward r efficiency The collision penalty is used to determine whether a collision has occurred in the environment. The type is bool. The safety distance penalty is used to determine whether there are other vehicles too close. It is used to warn of collisions and improve training efficiency. The calculation formula is:

[0084]

[0085] Where, d th represents the safety distance threshold, which is 15m in this example, d represents the distance between the nearest vehicle and the vehicle, and k norm is the normalization coefficient, which is 10 in this example.

[0086] Traffic efficiency rewards are used to encourage vehicles to travel at high speeds within the target speed range. The calculation formula is as follows:

[0087]

[0088] Where, v represents the vehicle speed; v low and v high Indicates the speed threshold.

[0089] In summary, the reward calculation formula at a certain moment in this example is as follows:

[0090]

[0091] Among them, ω c ,ω s ,ω e They represent the weights of the three rewards, which are -0.8, -0.2, and 0.5 respectively in this example.

[0092] π represents the agent's strategy; that is, how each intelligent vehicle chooses an action based on its current state. Taking discrete reinforcement learning as an example, it is expressed as:

[0093] π(A H1 |O H1 θ H1 ),…,π(A Hm |O Hm θ Hm ),π(AL1 |O L1 θ L1 ),…,π(A Ln |O Ln θ Ln ) (7);

[0094] Where, π(θ Hi ) and π(θ Lj ) represent the strategies of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle, θ Hi and θ Lj represent the policy network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively.

[0095] In this embodiment, heterogeneous multi-agents have partial isomorphism, that is, vehicles with the same intelligence level share strategies, which can be expressed as:

[0096]

[0097] Where θ H and θ L Represent the policy network parameters shared by high-level and low-level intelligent vehicles respectively.

[0098] In addition, using the state value function V π (s) evaluates the state of the agent. Each agent uses a neural network to estimate the state value function, which can be expressed as:

[0099] v(s;ω H1 ),…,v(s;ω Hm ),v(s;ω L1 ),…,v(s;ω Ln )(10);

[0100] In the formula, v(ω Hi ) and v(ω Lj ) represent the state value network of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle, ω Hi and ω Lj They represent the state value network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively.

[0101] In this embodiment, vehicles with the same intelligence level also share the same state value function, which is expressed as:

[0102]

[0103] Among them, ω H and ω L Represent the state value network parameters shared by high-level and low-level intelligent vehicles respectively.

[0104] Similar to the observation process, low-level intelligent vehicles lack decision-making capabilities. Their policy networks are deployed on nearby high-level intelligent vehicles. These vehicles make decisions on their behalf based on observation information and communicate their results to the low-level intelligent vehicles for execution. Unless otherwise specified, in this embodiment, when discussing decision-making by low-level intelligent vehicles, it is assumed that the nearby high-level intelligent vehicles perform policy network calculations and transmit the results to the low-level intelligent vehicles.

[0105] After establishing the reinforcement learning parameter space, the observation and policy parameters of each agent are initialized according to the vehicle position.

[0106] In step 32, the intelligent vehicle interacts with the environment and collects data. Taking the same-strategy method as an example, each vehicle in the intelligent connected vehicle simulation fleet adopts the current strategy π(θ Hi ) and π(θ Lj ) Make continuous decisions and take actions in the environment. The environment determines the next state and feedback rewards based on the actions of each agent and the randomness of the environment, and repeats this process. The state-action-reward data trajectory can be collected:

[0107]

[0108] Where, τ k represents the data of the kth vehicle from time step t = 0 to time step T; s t and r t Represent the global state and reward of step t, s t+1 represents the global state at step t+1, shared by all agents; and represents the observation and action of the k-th vehicle at time t.

[0109] In step 33, the loss is calculated for the heterogeneous multi-agent reinforcement learning strategy and value network, taking the heterogeneous multi-agent proximal policy optimization algorithm as an example:

[0110] First, the advantage function of each vehicle is calculated using the generalized advantage estimate. The advantage function of the kth vehicle at time t is:

[0111]

[0112] In the formula, γ is the discount factor, λ is the generalized advantage estimation coefficient, represents the time series difference error of the kth vehicle at time t+l, which is defined as:

[0113]

[0114] Where, v(st ;ω k ) represents the value estimation network of the kth agent, which is composed of parameters ω k constitute.

[0115] Next, we calculate the policy loss; for each agent, its policy ratio is:

[0116]

[0117] Then the clip proximal strategy optimizes the loss L clip (θ) is:

[0118]

[0119] Among them, ε is the clipping threshold of the proximal policy optimization algorithm, min and clip are the minimization function and clipping function respectively.

[0120] Then, calculate the value function loss. Use the mean square error to update the value function of each car, the value function loss L value (ω) is defined as:

[0121]

[0122] Furthermore, the policy entropy loss is calculated. Using the policy entropy as the regularization term, the policy entropy H(s;θ) is defined as:

[0123]

[0124] Then the policy entropy loss L entropy for:

[0125] L entropy,H (θ)=E s,i [H(s;θ i )] (twenty two);

[0126] L entropy,L (θ)=E s,j [H(s;θ j )] (twenty three);

[0127] In summary, the joint loss function L(θ,ω) is:

[0128] L H (θ,ω)=L clip,H (θ)+c1·L value,H (ω)-c2·L entropy,H (θ) (24);

[0129] L L (θ,ω)=L clip,L (θ)+c1·L value,L (ω)-c2·Lentropy,L (θ) (25);

[0130] Where c1 and c2 are the weights of the value function loss and policy entropy loss, respectively.

[0131] Finally, the parameters of the heterogeneous multi-agent are updated by gradient descent. Specifically, the Adam optimizer is used to update the policy and value networks respectively. The update formula is as follows:

[0132]

[0133] Among them, α is the learning rate.

[0134] In step 34, repeat steps 32 and 33 until the collaborative decision-making strategy reaches the convergence condition. Specifically, the total reward of the round is Convergence, policy network parameter convergence, etc.

[0135] Based on the above methods, the strategy network and value network are defined.

[0136] The policy network is used to generate vehicle actions based on vehicle perception information. In this embodiment, due to the varying levels of vehicle intelligence, each established agent has an independent policy network and different perception information. Vehicle perception information should at least include information that characterizes the vehicle's surroundings and influences vehicle behavioral decisions, such as the lateral and longitudinal positions and lateral and longitudinal speeds of surrounding vehicles. For different types of perception information, the policy network uses different encoders to encode the information into feature vectors. A fully connected network and activation function are then established based on the set action mode to construct a mapping from feature vectors to vehicle actions.

[0137] The value network is used to evaluate vehicle policies based on global environmental information. In this method, each agent has its own independent value network, as vehicles have different mission objectives, but they share the same global environmental information. This global environmental information should at least include all vehicle perception information, and additional information that cannot be perceived is added to improve evaluation. For different types of global environmental information, the value network uses different encoders to encode this information into feature vectors. A fully connected network is then used to construct a mapping from feature vectors to values.

[0138] On the other hand, this embodiment discloses a collaborative decision-making system for intelligent connected vehicles in mixed traffic scenarios, including:

[0139] A simulation environment construction module is used to build a mixed traffic simulation environment for multi-agent reinforcement learning training. The simulation environment includes a road model and randomly distributed traffic vehicles;

[0140] The vehicle model configuration module is used to build the physical model of intelligent connected vehicles and configure functional modules based on the level of each vehicle; the physical model of intelligent connected vehicles includes a perception module, a decision-making and planning module, a control module, and a communication module;

[0141] A reinforcement learning training module is used to collaboratively train a simulated platoon of connected vehicles using a heterogeneous multi-agent deep reinforcement learning method in a mixed traffic simulation environment to obtain the optimal collaborative decision-making strategy;

[0142] The strategy deployment module is used to deploy the optimal collaborative decision-making strategy to the intelligent connected vehicle fleet to generate collaborative decision-making information.

[0143] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0144] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A collaborative decision-making method for intelligent connected vehicles in mixed traffic scenarios, characterized by: The specific steps are: Establish a mixed traffic simulation environment for multi-agent reinforcement learning training, which includes a road model and randomly distributed traffic vehicles; Constructing a physical model of intelligent connected vehicles and configuring functional modules based on the level of each vehicle; the physical model of intelligent connected vehicles includes a perception module, a decision-making and planning module, a control module, and a communication module; A heterogeneous multi-agent deep reinforcement learning method is used to collaboratively train a simulated platoon of heterogeneous connected vehicles based on the mixed traffic simulation environment to obtain the optimal collaborative decision-making strategy. The optimal collaborative decision-making strategy is deployed to a fleet of intelligent connected vehicles to generate collaborative decision-making information.

2. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 1 is characterized in that: Vehicle levels are divided into high-level and low-level categories. High-level intelligent vehicles are equipped with perception modules, decision-making and planning modules, control modules and communication modules; low-level intelligent vehicles are equipped with control modules and communication modules.

3. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 2 is characterized in that: The steps for obtaining the optimal collaborative decision-making strategy are: Step 31: Initialize the simulation environment and reinforcement learning parameter space, where the parameter space includes the global state spliced from local observations of high-level intelligent vehicles, the action set of the simulated platoon of heterogeneous intelligent connected vehicles, rewards, policy networks, and value networks. Step 32: Collect state-action-reward data trajectories through the interaction between the simulated fleet of intelligent connected vehicles and the simulation environment; Step 33: Calculate a loss function based on the collected data, and optimize the parameters of the collaborative strategy of the intelligent connected vehicles according to the loss function; Step 34: Repeat steps 32 and 33 until the convergence condition is reached and the optimal collaborative decision-making strategy is obtained.

4. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The expression of the global state is: S=[O H1 ,…THE Hm ,THE L1 ,…,THE Ln ,S e ]; Where, O Hi represents the local observation of the road state by the i-th high-level intelligent vehicle; O Lj represents the local observation of the road state by the jth low-level intelligent vehicle; m represents the number of high-level intelligent vehicles; n represents the number of low-level intelligent vehicles; S e Represents the global state outside the observation range of all high-level intelligent vehicles.

5. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The reward expression is: Where R Hi represents the reward of the i-th high-level intelligent vehicle; R Lj represents the reward for the j-th low-level intelligent vehicle; m represents the number of high-level intelligent vehicles; n represents the number of low-level intelligent vehicles.

6. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The expression of the state-action-reward data trajectory is: Where, τ k represents the data of the kth vehicle from time step t = 0 to time step T; s t and r t Represent the global state and reward of step t respectively; s t+1 represents the global state at step t+1; and They represent the observation and action of the kth vehicle at time t respectively; m represents the number of high-level intelligent vehicles; and n represents the number of low-level intelligent vehicles.

7. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The policy network adopts the same level parameter sharing mechanism, specifically: The sharing mechanism expression of the policy network of the high-level intelligent vehicle is: The sharing mechanism expression of the policy network of the low-level intelligent vehicle is: Where θ Hi and θ Lj They represent the policy network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively; θ H and θ L Represent the policy network parameters shared by high-level intelligent vehicles and low-level intelligent vehicles respectively.

8. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The value network adopts the same level parameter sharing mechanism, specifically: The sharing mechanism expression of the value network of the high-level intelligent vehicle is: The sharing mechanism expression of the value network of the high-level intelligent vehicle is: Where, ω Hi and ω Lj They represent the state network parameters of the i-th high-level intelligent vehicle and the j-th low-level intelligent vehicle respectively; ω H and ω L Represent the value network parameters shared by high-level intelligent vehicles and low-level intelligent vehicles respectively.

9. The method for collaborative decision-making of intelligent connected vehicles in mixed traffic scenarios according to claim 3 is characterized in that: The loss function includes a joint loss function, which is expressed as: L H (θ,ω)=L clip,H (θ)+c1·L value,H (ω)-c2·L entropy,H (i); L L (θ,ω)=L clip,L (θ)+c1·L value,L (ω)-c2·L entropy,L (i); Where, L H (θ, ω) represents the high-level joint loss function; L L (θ, ω) represents the low-level joint loss function; L clip，H (θ) represents the high-level strategy optimization loss; L clip,L (θ) represents the low-level policy optimization loss; L value,H (ω) represents the high-level value function loss; L value,L (ω) represents the low-level value function loss; L entropy,H (θ) represents the high-level policy entropy loss; L entropy，L (θ) represents the low-level policy entropy loss; c1 and c2 are the weights of the value function loss and policy entropy loss, respectively.

10. A collaborative decision-making system for intelligent connected vehicles in mixed traffic scenarios, characterized by: include: A simulation environment construction module is used to build a mixed traffic simulation environment for multi-agent reinforcement learning training. The simulation environment includes a road model and randomly distributed traffic vehicles; A vehicle model configuration module is used to construct a physical model of a connected vehicle and configure functional modules based on the level of each vehicle; the physical model of the connected vehicle includes a perception module, a decision-making and planning module, a control module, and a communication module; A reinforcement learning training module is used to collaboratively train a simulated platoon of heterogeneous connected vehicles based on the mixed traffic simulation environment using a heterogeneous multi-agent deep reinforcement learning method to obtain an optimal collaborative decision-making strategy; The strategy deployment module is used to deploy the optimal collaborative decision-making strategy to the intelligent connected vehicle fleet to generate collaborative decision-making information.

Images