Decision-making method and device for multi-vehicle reinforcement learning, storage medium and equipment

Abstract

The invention discloses a decision-making method and device for multi-vehicle reinforcement learning, a storage medium and equipment. The decision-making method comprises the steps of distributing a priority index for each vehicle controlled based on a multi-vehicle reinforcement learning model; traversing each vehicle according to the priority index from high to low; for the traversed current vehicle, acquiring exploratory actions in the ith time step output by the multi-vehicle reinforcement learning model, and judging whether the current vehicle has a collision risk with other vehicles in the ith time step or not; if not, taking the exploratory action in the ith time step as the actual action of the current vehicle in the ith time step; otherwise, based on the safety margin of the current vehicle at the ith time step, screening out the actual action of the current vehicle in the ith time step from the effective action set; and obtaining new i after i + 1, and returning to execute the step of obtaining the exploratory action in the ith time step until the actual action of the current vehicle in each time step in the preset time range is determined, and continuing to traverse the next vehicle.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a decision-making method, device, storage medium and equipment for multi-vehicle reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence, reinforcement learning algorithms have gradually become an important tool for optimizing vehicle decision-making in the field of autonomous driving. These algorithms enable vehicles to make autonomous decisions in complex and dynamic traffic scenarios by learning and feeding back a large amount of environmental data. Especially in multi-vehicle cooperative driving, the interaction and coordination between vehicles is crucial. Multi-vehicle reinforcement learning provides an adaptive solution that enables multiple vehicles to adjust their strategies in real time to deal with emergencies.

[0003] In complex traffic scenarios, vehicle flow is highly dynamic, vehicle speeds vary greatly, and driver behavior is unpredictable, increasing the risk of collision. Therefore, existing reinforcement learning algorithms may lead to unstable decisions in such scenarios, making it difficult to take into account the safety needs of multiple vehicles. The "black box" nature of reinforcement learning algorithms makes their decision-making process lack of explainability, which is particularly prominent in multi-vehicle environments. Current research focuses on how to maximize the benefits of the overall system and ignores the safety of individual vehicles. For example, some Q-learning-based algorithms focus on equilibrium decisions in collaborative or competitive scenarios, often producing unpredictable behaviors in dynamic and complex traffic situations, and fail to effectively address potential collision risks in multi-vehicle interactions.

[0004] Therefore, how to improve vehicle safety based on multi-vehicle reinforcement learning decision-making is one of the urgent problems to be solved. Summary of the invention

[0005] The present application provides a decision-making method, device, storage medium and equipment for multi-vehicle reinforcement learning, which can improve the vehicle safety based on multi-vehicle reinforcement learning decision-making.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect, an embodiment of the present application provides a decision-making method for multi-vehicle reinforcement learning, the method comprising:

[0008] According to the preset priority allocation rules, a priority index is assigned to each autonomous driving vehicle controlled by the multi-vehicle reinforcement learning model, wherein the probability of collision risk of the autonomous driving vehicle is positively correlated with the priority index;

[0009] Traversing each of the autonomous driving vehicles in order of priority index from high to low;

[0010] For the traversed current autonomous driving vehicle, obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, and judge whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of other autonomous driving vehicles, wherein when the other autonomous driving vehicle is an untraversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle in the most recent driving, and when the other autonomous driving vehicle is an already traversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle in the i-th time step;

[0011] If there is no collision risk, the exploratory action in the i-th time step is used as the actual action of the current autonomous driving vehicle in the i-th time step;

[0012] If there is a collision risk, based on the safety margin of the current autonomous driving vehicle at the i-th time step, select the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step;

[0013] After i+1, a new i is obtained, and the execution step is returned to obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual action of the current autonomous driving vehicle in each time step within the preset time range is determined, and then the traversal of the next autonomous driving vehicle is continued.

[0014] In a possible implementation, according to a preset priority allocation rule, a priority index is allocated to each autonomous driving vehicle controlled based on a multi-vehicle reinforcement learning model, including:

[0015] For each of the autonomous driving vehicles controlled based on the multi-vehicle reinforcement learning model, respectively calculating a merging priority, a lane end priority, and a headway priority of the autonomous driving vehicle;

[0016] Performing a weighted summation on the merging priority, the lane end priority, and the headway priority of the autonomous driving vehicle to obtain a priority index of the autonomous driving vehicle;

[0017] The merging priority of the autonomous driving vehicle located in the merging lane is higher than that of the autonomous driving vehicle located in the main lane;

[0018] For multiple autonomous driving vehicles on a merging lane, the closer they are to the end point of the merging lane, the higher the priority of the end point of the lane;

[0019] The smaller the headway time is, the higher the headway time priority of the autonomous driving vehicle.

[0020] In a possible implementation, based on the safety margin of the current autonomous driving vehicle at the i-th time step, selecting the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step includes:

[0021] For each valid action in the valid action set, respectively determine the safety margin of the current automatic driving vehicle at the i-th time step;

[0022] The effective action with the largest safety margin is determined as the optimal action, and the optimal action is determined as the actual action of the current autonomous driving vehicle in the i-th time step.

[0023] In a possible implementation, the method further includes:

[0024] In the process of training the multi-vehicle reinforcement learning model, when the current state spaces of the multiple reinforcement learning vehicles are input into the policy network, the policy network obtains an initial action set in the environment and a label set after the effectiveness of each action in the initial action set is labeled, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training;

[0025] Obtaining a valid action set according to the initial action set and the annotation set;

[0026] The policy network processes the current state space and selects the current action spaces of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after performing value evaluation on the current action space based on the value network.

[0027] In a possible implementation, before determining whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles, the method further includes:

[0028] In the case where there are manually driven vehicles around the autonomous driving vehicle, predicting the driving trajectories of the manually driven vehicles within the preset time range based on the manually driven vehicle behavior prediction model to obtain a predicted trajectory of each of the manually driven vehicles;

[0029] The determining whether there is a collision risk between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles includes:

[0030] Based on the trajectory predicted based on the exploratory action within the i-th time step, the trajectory predicted based on the target action of other autonomous driving vehicles and the predicted trajectory of the manually driven vehicle, it is determined whether there is a risk of collision between the current autonomous driving vehicle and other surrounding vehicles within the i-th time step.

[0031] In a possible implementation, the manually driven vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model;

[0032] Predicting the driving trajectory of the manually driven vehicle within the preset time range based on the manually driven vehicle behavior prediction model to obtain the predicted trajectory of each manually driven vehicle includes:

[0033] Simulating the real-time driving behavior of each of the manually driven vehicles based on the car-following model;

[0034] Obtaining a target lane for the manually driven vehicle according to the real-time driving behavior and lane change decision model of the manually driven vehicle;

[0035] Based on the vehicle kinematic model and the target lane, the driving trajectory of the manually driven vehicle within the preset time range is predicted to obtain the predicted trajectory of the manually driven vehicle.

[0036] In a second aspect, an embodiment of the present application provides a decision-making device for multi-vehicle reinforcement learning, the device comprising:

[0037] An allocating unit, configured to allocate a priority index to each autonomous driving vehicle controlled based on the multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of a collision risk of the autonomous driving vehicle is positively correlated with the priority index;

[0038] A traversal unit, used to traverse each of the autonomous driving vehicles in order of priority index from high to low;

[0039] The traversal unit comprises:

[0040] An action acquisition module, used for acquiring, for the traversed current autonomous driving vehicle, the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model;

[0041] a judgment module, configured to judge whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to a trajectory predicted based on the exploratory action in the i-th time step and a trajectory predicted based on the target action of the other autonomous driving vehicles, wherein when the other autonomous driving vehicles are untraversed autonomous driving vehicles, the target action of the other autonomous driving vehicles is the actual action of the other autonomous driving vehicles most recently driven, and when the other autonomous driving vehicles are traversed autonomous driving vehicles, the target action of the other autonomous driving vehicles is the actual action of the other autonomous driving vehicles in the i-th time step;

[0042] a determination module, configured to use the exploratory action in the i-th time step as the actual action of the current autonomous driving vehicle in the i-th time step if there is no collision risk;

[0043] a screening module, configured to screen out the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step based on the safety margin of the current autonomous driving vehicle at the i-th time step if there is a collision risk;

[0044] The traversal unit is used to obtain a new i after i+1, and return to the execution step to obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual action of the current autonomous driving vehicle in each time step within a preset time range is determined, and then continue to traverse the next autonomous driving vehicle.

[0045] In a possible implementation, the allocation unit includes:

[0046] a calculation module, configured to calculate, for each of the autonomous driving vehicles controlled based on the multi-vehicle reinforcement learning model, a merging priority, a lane end priority, and a headway priority of the autonomous driving vehicle;

[0047] a weighting module, configured to perform a weighted summation on the merging priority, the lane end priority and the headway priority of the autonomous driving vehicle to obtain a priority index of the autonomous driving vehicle;

[0048] The merging priority of the autonomous driving vehicle located in the merging lane is higher than that of the autonomous driving vehicle located in the main lane;

[0049] For multiple autonomous driving vehicles on a merging lane, the closer they are to the end point of the merging lane, the higher the priority of the end point of the lane;

[0050] The smaller the headway time is, the higher the headway time priority of the autonomous driving vehicle.

[0051] In one possible implementation, the screening module is used to determine the safety margin of the current autonomous driving vehicle at the i-th time step for each valid action in the valid action set; determine the valid action with the largest safety margin as the optimal action, and determine the optimal action as the actual action of the current autonomous driving vehicle in the i-th time step.

[0052] In a possible implementation, the device further includes:

[0053] A training unit is used for, during the process of training the multi-vehicle reinforcement learning model, when the current state space of multiple reinforcement learning vehicles is input into the policy network, the policy network obtains an initial action set in the environment and a labeled set after the validity is labeled for each action in the initial action set, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training; a valid action set is obtained according to the initial action set and the labeled set; the policy network processes the current state space and filters out the current action space of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after the current action space is valued based on the value network.

[0054] In a possible implementation, the device further includes:

[0055] a prediction unit, configured to, before determining whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles, predict the driving trajectory of the manually driven vehicles within the preset time range based on the manually driven vehicle behavior prediction model in the case where there are manually driven vehicles around the autonomous driving vehicle, so as to obtain a predicted trajectory of each of the manually driven vehicles;

[0056] The judgment module is also used to judge whether there is a risk of collision between the current autonomous driving vehicle and other surrounding vehicles within the i-th time step based on the trajectory predicted based on the exploratory action within the i-th time step, the trajectory predicted based on the target action of other autonomous driving vehicles and the predicted trajectory of the manually driven vehicle.

[0057] In a possible implementation, the manually driven vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model;

[0058] The prediction unit comprises:

[0059] A simulation module, used for simulating the real-time driving behavior of each of the manually driven vehicles based on the car-following model;

[0060] A lane-changing decision module, used for obtaining a target lane of the manually driven vehicle according to the real-time driving behavior of the manually driven vehicle and the lane-changing decision model;

[0061] The prediction module is used to predict the driving trajectory of the manually driven vehicle within the preset time range based on the vehicle kinematic model and the target lane, so as to obtain the predicted trajectory of the manually driven vehicle.

[0062] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any possible implementation manner of the first aspect.

[0063] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0064] one or more processors;

[0065] The processor is coupled to a storage device, and the storage device is used to store one or more programs;

[0066] When the one or more programs are executed by the one or more processors, the electronic device implements the method described in any possible implementation manner of the first aspect.

[0067] The decision-making method, device, storage medium and equipment for multi-vehicle reinforcement learning provided by the embodiment of the present application can first assign a higher priority index to the autonomous driving vehicle with a higher probability of collision risk, and determine the actual action of each autonomous driving vehicle in the future preset time range in order from high to low priority index. Specifically, after obtaining the exploratory action of each autonomous driving vehicle at the i-th time step output by the multi-vehicle reinforcement learning model, the exploratory action is not directly used as the actual action, but first judged whether it will collide with other vehicles if driving based on the exploratory action. Only when there is no risk of collision, the exploratory action will be used as the actual action, and when there is a risk of collision, the safety margin of the current autonomous driving vehicle at the i-th time step will be combined to select the best action from the effective action set as the actual action of the i-th time step. It can be seen that the embodiment of the present application can efficiently manage computing resources and ensure real-time decision-making capabilities in multi-vehicle scenarios by introducing a priority-based computing resource allocation mechanism, thereby solving the problem of rapid increase in computing complexity in multi-vehicle environments. By combining multi-step predictions and introducing a safety assurance mechanism in multi-vehicle scenarios, the explainability of the decision-making process is effectively improved. It is able to predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0068] In addition, the embodiments of the present application can also achieve the following technical effects:

[0069] 1. The embodiment of the present application determines the priority index of each autonomous driving vehicle by combining multiple factors such as merging priority, lane end priority and headway priority, thereby improving the accuracy and reliability of the priority index.

[0070] 2. In the process of training the multi-vehicle reinforcement learning model, the embodiment of the present application annotates the validity of each action in the initial action set in advance. When training the policy network, the effective action set can be quickly screened out from the initial action set through the annotated set, and decisions can be made based on the effective action set. It can be seen that the embodiment of the present application excludes invalid or unsafe actions in multi-vehicle reinforcement learning by adopting an invalid action masking mechanism, ensuring that the decision-making process is safer and more effective, and also accelerating the convergence of the multi-vehicle reinforcement learning model.

[0071] 3. In the scenario where autonomous driving and manual driving are combined, the embodiments of the present application predict the trajectory of the manually driven vehicle within a preset time range through the manually driven vehicle behavior prediction model, and when performing multi-step predictions for each autonomous driving vehicle, in addition to considering the risk of collision with other autonomous driving vehicles, the risk of collision with manually driven vehicles is also considered, thereby further improving the accuracy of driving decisions and thus improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0073] Figure 1 A flowchart of a decision-making method for multi-vehicle reinforcement learning provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of priority allocation parameters provided in an embodiment of the present application;

[0075] Figure 3 An example diagram of a trajectory conflict provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of a safety margin definition for an autonomous driving vehicle whose lane change action is provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of a safety margin definition for an autonomous driving vehicle whose action is straight driving is provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a neural network architecture that introduces action masking provided in an embodiment of the present application;

[0079] Figure 7 A block diagram of a decision-making device for multi-vehicle reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of the present application and the accompanying drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0082] Figure 1 The present invention is a flowchart of a decision-making method for multi-vehicle reinforcement learning. The method can be applied to electronic devices, specifically to autonomous driving vehicles or cloud servers interacting with autonomous driving vehicles. The method includes:

[0083] S110: According to a preset priority assignment rule, a priority index is assigned to each autonomous driving vehicle controlled based on the multi-vehicle reinforcement learning model.

[0084] In actual applications, there can be multiple preset priority allocation rules, and different preset priority allocation rules can be set for different road scenarios. However, all preset priority allocation rules must meet the following conditions: the greater the possibility of a collision risk, the higher the corresponding priority index of the autonomous driving vehicle, that is, the probability of a collision risk for an autonomous driving vehicle is positively correlated with the priority index.

[0085] The following takes the ramp merging scenario as an example to describe the priority index allocation rules in detail:

[0086] For each autonomous driving vehicle controlled based on the multi-vehicle reinforcement learning model, the merging priority, lane end priority and headway priority of the autonomous driving vehicle are calculated respectively; the merging priority, lane end priority and headway priority of the autonomous driving vehicle are weightedly summed to obtain the priority index of the autonomous driving vehicle.

[0087] The merging priority of the autonomous vehicles in the merging lane is higher than that of the autonomous vehicles in the main lane. This is because the autonomous vehicles in the merging lane face a time-sensitive merging task (because the end of the merging lane is impassable) and need to complete the merging operation more quickly. By setting the autonomous vehicles in the merging lane to a higher priority, they can be ensured to merge more smoothly into the traffic flow.

[0088] For multiple autonomous vehicles on a merging lane, the closer they are to the end of the merging lane, the higher the priority of the lane end. This is because vehicles closer to the end of the merging lane are more likely to cause collisions and deadlocks. Autonomous vehicles on a merging lane near the end of the merging lane are more susceptible to traffic, so they require more attention in terms of safety. By giving these vehicles a higher priority, potential collision risks can be avoided more effectively, improving the safety of the entire system.

[0089] The smaller the headway distance, the higher the headway distance priority of the autonomous vehicle. This is because vehicles with smaller headway distances are more likely to collide with the vehicle in front. At intersections and merging areas, smaller headway distances may increase the probability of crossing between adjacent vehicles, thereby increasing the risk of collision. By increasing the priority of these vehicles, their actions can be arranged more carefully, reducing potential crossing conflicts, thereby improving the smoothness and safety of the overall traffic system.

[0090] Based on the above settings, the priority index p of the autonomous driving vehicle i controlled by multi-vehicle reinforcement learning is i The formula is as follows:

[0091] p i =α1p m +α2p d +α3p h +σ i

[0092] Among them, α1, α2, and α3 are the merge priorities p m , Lane end priority p d , headway priority p h The weight factor of is a small random variable that prevents two autonomous vehicles from having the same priority index.

[0093] Merge priority p m :

[0094]

[0095] Lane end priority p d :

[0096]

[0097] Headway priority p h :

[0098]

[0099] Among them, Figure 2As shown in Figure 2, x represents the distance that the autonomous vehicle travels in the merging lane, and L represents the length of the merging lane. headway represents the distance corresponding to the headway of the autonomous vehicle, t h represents the headway safety threshold set in advance, v t Indicates the current speed of the autonomous vehicle.

[0100] exist Figure 2 In the figure, both black rectangles and white rectangles represent vehicles, and black rectangles represent autonomous driving vehicles, while white rectangles can represent autonomous driving vehicles or manually driven vehicles. The embodiment of the present application only focuses on the relationship between the autonomous driving vehicle to be assigned a priority and the surrounding vehicles and the road on which it is located, and does not focus on the type of vehicles surrounding the autonomous driving vehicle to be assigned a priority.

[0101] S120: Traverse each autonomous driving vehicle in order of priority index from high to low.

[0102] S130: For the current autonomous driving vehicle being traversed, obtain the exploratory action of the current autonomous driving vehicle in the i-th time step in the future output by the multi-vehicle reinforcement learning model, and determine whether there is a risk of collision between the current autonomous driving vehicle and other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of other autonomous driving vehicles.

[0103] Among them, when the other autonomous driving vehicles are untraversed autonomous driving vehicles, the target action of other autonomous driving vehicles is the actual action of other autonomous driving vehicles driving recently, and when the other autonomous driving vehicles are traversed autonomous driving vehicles, the target action of other autonomous driving vehicles is the actual action of other autonomous driving vehicles in the i-th time step. Both exploratory actions and actual actions refer to driving actions, such as turning left, turning right, driving straight at a constant speed, accelerating straight, decelerating straight, idling straight, etc.

[0104] i is a positive integer. When i=1, the input data of the multi-vehicle reinforcement learning model includes the real state space of multiple autonomous driving vehicles when the regulation and control begins, and the output data includes the exploratory action of the current autonomous driving vehicle in the first time step in the future; when i>1, the input data of the multi-vehicle reinforcement learning model includes the state space corresponding to the actual action (planned actual action) of the current autonomous driving vehicle in the i-1th time step and the state space corresponding to the target action of other autonomous driving vehicles when planning the action of the current autonomous driving vehicle in the i-1th time step, and the output data includes the exploratory action of the current autonomous driving vehicle in the i-th time step in the future.

[0105] When judging whether there is a collision risk between the current autonomous driving vehicle and other autonomous driving vehicles in the i-th time step, as long as there is another autonomous driving vehicle whose distance from the current autonomous driving vehicle is less than the preset safety distance, it is determined that there is a collision risk. That is, when the distance between the current autonomous driving vehicle and all other autonomous driving vehicles is greater than or equal to the preset safety distance, it is determined that there is no collision risk.

[0106] S140: If there is no collision risk, the exploratory action in the i-th time step is used as the actual action of the current autonomous driving vehicle in the i-th time step.

[0107] If there is no collision risk, it means that the exploratory action in the i-th time step is safe and reliable, and can be directly used as the actual action of the current autonomous driving vehicle in the i-th time step.

[0108] S150: If there is a risk of collision, based on the safety margin of the current autonomous driving vehicle at the i-th time step, the best action is selected from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step.

[0109] If there is a risk of collision, the exploratory action in the i-th time step is unsafe, and a safer action needs to be found to replace the exploratory action. In order to obtain a safer action, the embodiment of the present application can select the best action from the valid action set based on the safety margin of the current autonomous driving vehicle at the i-th time step as the actual action of the current autonomous driving vehicle in the i-th time step. Figure 3 As shown in the figure, if the autonomous driving vehicle represented by the black rectangle wants to switch to the left lane, it may collide with the vehicle in the left lane at time steps k=3, 4, and 5. Therefore, the exploratory actions of the autonomous driving vehicle represented by the black rectangle at the 3rd, 4th, and 5th time steps are unreasonable, and other actions need to be re-decided.

[0110] Specifically, the safety margin of the current autonomous driving vehicle at the i-th time step can be determined for each valid action in the valid action set; then the valid action with the largest safety margin is determined as the optimal action, and the optimal action is determined as the actual action of the current autonomous driving vehicle in the i-th time step.

[0111] The valid action set may include left turn, right turn, acceleration, idling and deceleration. If the valid action is left turn or right turn, the safety margin of the current autonomous driving vehicle at the i-th time step is: the minimum distance between the current autonomous driving vehicle and the vehicles in front and behind the target lane to which the lane needs to be changed. The minimum distance refers to the distance between the two vehicle heads along the road driving direction, such as Figure 4As shown in the figure, when the autonomous driving vehicle represented by the black rectangle wants to turn left, the minimum distance between the autonomous driving vehicle and the vehicles in front and behind on the left lane can be determined respectively. The minimum distance in the figure is d sm If the effective action is acceleration, idling or deceleration, the safety margin of the current autonomous driving vehicle at the i-th time step is: the minimum distance between the current autonomous driving vehicle and the front and rear vehicles in the same lane, such as Figure 5 As shown in the figure, when the autonomous driving vehicle represented by the black rectangle wants to continue to go straight (accelerate, idle, decelerate, etc.), the minimum distance between the autonomous driving vehicle and the vehicles in front and behind it in the lane can be determined respectively. The minimum distance in the figure is d sm .

[0112] S160: Obtain a new i after i+1, and return to the execution step to obtain the exploratory actions of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual actions of the current autonomous driving vehicle in each time step within the preset time range are determined, and then continue to traverse the next autonomous driving vehicle.

[0113] After i+1, a new i is obtained, and the execution of steps S130-S150 is returned to determine the actual action of the current autonomous driving vehicle in the new i-th time step, until the actual action of the current autonomous driving vehicle in each time step within the preset time range is completed, the current autonomous driving vehicle is completed, and the traversal continues to the next autonomous driving vehicle.

[0114] The decision-making method for multi-vehicle reinforcement learning provided by the embodiment of the present application can first assign a higher priority index to the autonomous driving vehicle with a higher probability of collision risk, and determine the actual action of each autonomous driving vehicle within the preset time range in the future in order of priority index from high to low. Specifically, after obtaining the exploratory action of each autonomous driving vehicle at the i-th time step output by the multi-vehicle reinforcement learning model, the exploratory action is not directly used as the actual action, but first determines whether it will collide with other vehicles if driving based on the exploratory action. Only when there is no risk of collision, the exploratory action will be used as the actual action. When there is a risk of collision, the safety margin of the current autonomous driving vehicle at the i-th time step will be combined to select the best action from the effective action set as the actual action of the i-th time step. It can be seen that the embodiment of the present application can efficiently manage computing resources and ensure real-time decision-making capabilities in multi-vehicle scenarios by introducing a priority-based computing resource allocation mechanism, thereby solving the problem of rapid increase in computing complexity in multi-vehicle environments. By combining multi-step predictions and introducing a safety assurance mechanism in multi-vehicle scenarios, the explainability of the decision-making process is effectively improved. It is able to predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0115] In one possible implementation, during the training of a multi-vehicle reinforcement learning model, there are some problems in the process of sampling actions. The core of the problem lies in the two main problems that may be encountered when sampling actions through random strategies. The first is that invalid or unsafe actions are assigned a non-zero probability, which means that during the training process, the model may choose to execute these unsafe actions, resulting in poor behavior of the system or even system crashes. The second is that since invalid actions are also involved in policy updates, the learning of invalid policies leads to invalidation of the training process because the experience associated with invalid actions is meaningless. In order to solve these problems, the embodiment of the present application can introduce a method called invalid action masking during the training of a multi-vehicle reinforcement learning model. This method ensures that only valid and safe actions are included in the training of the model, thereby accelerating the convergence of the model and improving the efficiency of model training.

[0116] Specifically, in the process of training a multi-vehicle reinforcement learning model, when the current state space of multiple reinforcement learning vehicles is input into the policy network (i.e., the Actor network), the policy network obtains an initial action set in the environment and a labeled set after the validity is labeled for each action in the initial action set, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training; a valid action set is obtained based on the initial action set and the labeled set; the policy network processes the current state space and filters out the current action space of multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after performing value evaluation on the current action space based on the value network (i.e., the Critic network).

[0117] When marking the validity of each action in the initial action set, you can first set the rule that it must be an invalid action, and then mark the remaining actions as valid actions. The rules for invalid actions include but are not limited to the following two:

[0118] (1) The action of the ego vehicle attempting to change lane to a non-existent lane is an invalid action;

[0119] (2) When the vehicle's speed has reached the predefined maximum or minimum speed, the vehicle attempts to accelerate or decelerate, which is an invalid action.

[0120] In the embodiment of the present application, valid actions can be marked as "1" and invalid actions can be marked as "0", and then the initial action set and the marked set can be directly multiplied to see the valid action set. Figure 6As shown, the state information (including position and speed) of other autonomous driving vehicles and the state information (including position and speed) of the target autonomous driving vehicle can be respectively input into the Actor network after passing through FC (Fully Connected Network). In the Actor network, the logical value sequence of the initial action set ([5,5,5,5,5]) and the logical value sequence of the labeled set ([1,1,1,0,1]) can be multiplied, and the logical value sequence of the valid action set can be obtained by sampling and evaluating each action. This makes the probability corresponding to invalid actions become very close to zero after passing through the Softmax layer of the neural network. Therefore, in the sampling stage, the model will hardly select these invalid actions.

[0121] In the process of training the multi-vehicle reinforcement learning model, the embodiment of the present application labels the validity of each action in the initial action set in advance. When training the policy network, the effective action set can be quickly screened out from the initial action set through the label set, and decisions can be made based on the effective action set. It can be seen that the embodiment of the present application excludes invalid or unsafe actions in multi-vehicle reinforcement learning by adopting an invalid action masking mechanism, ensuring that the decision-making process is safer and more effective, and also accelerating the convergence of the multi-vehicle reinforcement learning model.

[0122] In a possible implementation, in actual applications, in the same road segment, there may be not only autonomous driving vehicles but also manually driven vehicles. Therefore, in the scenario where there are manually driven vehicles, when predicting the collision between each autonomous driving vehicle and surrounding vehicles, it is also necessary to consider whether the autonomous driving vehicle may have a collision risk with the manually driven vehicles. Therefore, before judging whether there is a collision risk between the current autonomous driving vehicle and other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of other autonomous driving vehicles, in the case where there are manually driven vehicles around the autonomous driving vehicle, the driving trajectory of the manually driven vehicle within a preset time range is predicted based on the manually driven vehicle behavior prediction model to obtain the predicted trajectory of each manually driven vehicle. Then, it can be judged whether there is a collision risk between the current autonomous driving vehicle and other surrounding vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step, the trajectory predicted based on the target action of other autonomous driving vehicles, and the predicted trajectory of the manually driven vehicle.

[0123] Among them, the manually driven vehicle behavior prediction model includes the following model, the lane changing decision model and the vehicle kinematics model.

[0124] The method for predicting the trajectory of a manually driven vehicle may include: simulating the real-time driving behavior of each manually driven vehicle based on a car-following model; obtaining a target lane of the manually driven vehicle according to the real-time driving behavior of the manually driven vehicle and a lane-changing decision model; and predicting the driving trajectory of the manually driven vehicle within a preset time range based on a vehicle kinematic model and the target lane to obtain a predicted trajectory of the manually driven vehicle.

[0125] Among them, the following model can select the IDM (Intelligent Driver Model) model, or other models.

[0126] The specific mathematical formula of the IDM model is expressed as a set of equations that can quantitatively describe the acceleration and deceleration behavior of vehicles in traffic flow, making the simulation closer to real driving scenarios. The specific formula of the IDM model is as follows:

[0127]

[0128] The symbols in the above formula are explained as follows:

[0129] a HV represents the acceleration of the target vehicle HV; v HV Indicates the speed of the target vehicle HV; s HV Indicates the actual following distance of the target vehicle HV; Δv HV represents the speed difference between the target vehicle HV and its preceding vehicle; δ′ represents the acceleration index; v0 represents the expected speed of the target vehicle; s * (v HV , Δv HV ) indicates the expected following distance.

[0130] The lane changing decision model may select the MOBIL (Minimize Overall Braking Induced by LaneChange) model or other models.

[0131] The application of the MOBIL model and the use of incentive criteria help evaluate the overall benefits of lane-changing behavior and ensure that such behavior does not have adverse effects on surrounding vehicles, thereby promoting the stability and efficiency of the traffic system. The incentive criterion model of the MOBIL model is expressed as follows:

[0132]

[0133] The symbols in the above formula are explained as follows:

[0134] u HV represents the total lane change benefit of the target vehicle HV; a HVrepresents the acceleration of the target vehicle HV before changing lanes; represents the acceleration of the target vehicle HV after changing lanes; a PFV Indicates the acceleration of the following vehicle in the target lane before the target vehicle changes lanes to the target lane; Indicates the acceleration of the following vehicle in the target lane after the target vehicle changes lanes; Indicates the acceleration of the vehicle behind the target vehicle in the original lane after the target vehicle changes lanes; a FV represents the acceleration of the vehicle behind it in the original lane before the target vehicle changes lanes; p represents the courtesy coefficient; Δa th Represents the lane-changing benefit threshold.

[0135] The vehicle kinematic model may be a bicycle model or other models.

[0136] When the MOBIL model gives a lane change decision, it will output a target lane and pass it to the vehicle kinematics model. The vehicle kinematics model uses the classic bicycle model, and then outputs the front wheel angle signal through the vehicle kinematics model to achieve the purpose of predicting human driving behavior. The expression of the bicycle model includes:

[0137]

[0138] The symbols in the above formula are explained as follows:

[0139] Δy represents the lateral deviation from the target vehicle to the center line of the target lane; Indicates the yaw angular velocity required for lane change; l f , l r are the distances from the target vehicle's center of mass to the center of its front and rear axles; β represents the target vehicle's center of mass sideslip angle; v represents the target vehicle's speed; ψ represents the target vehicle's yaw angle; and δ represents the target vehicle's front wheel turning angle.

[0140] In a scenario where autonomous driving and manual driving are combined, the embodiments of the present application predict the trajectory of a manually driven vehicle within a preset time range through a manually driven vehicle behavior prediction model, and when performing multi-step predictions for each autonomous driving vehicle, in addition to considering the risk of collision with other autonomous driving vehicles, the risk of collision with manually driven vehicles is also considered, thereby further improving the accuracy of driving decisions and thereby improving driving safety.

[0141] Based on the above method embodiment, another embodiment of the present application provides a decision-making device for multi-vehicle reinforcement learning, such as Figure 7 As shown, the device comprises:

[0142] An allocating unit 210 is used to allocate a priority index to each autonomous driving vehicle controlled based on the multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of a collision risk of the autonomous driving vehicle is positively correlated with the priority index;

[0143] A traversal unit 220, configured to traverse each of the autonomous driving vehicles in descending order of priority index;

[0144] The traversal unit 220 includes:

[0145] An action acquisition module 221 is used to acquire, for the traversed current autonomous driving vehicle, the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model;

[0146] A judgment module 222 is used to judge whether there is a collision risk between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles, wherein when the other autonomous driving vehicle is an untraversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle most recently driven, and when the other autonomous driving vehicle is a traversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle in the i-th time step;

[0147] A determination module 223 is configured to use the exploratory action in the i-th time step as the actual action of the current autonomous driving vehicle in the i-th time step if there is no collision risk;

[0148] A screening module 224 is configured to screen out the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step if there is a collision risk;

[0149] The traversal unit 220 is used to obtain a new i after i+1, and return to the execution step to obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual action of the current autonomous driving vehicle in each time step within a preset time range is determined, and then continue to traverse the next autonomous driving vehicle.

[0150] In a possible implementation, the allocation unit 210 includes:

[0151] a calculation module, configured to calculate, for each of the autonomous driving vehicles controlled based on the multi-vehicle reinforcement learning model, a merging priority, a lane end priority, and a headway priority of the autonomous driving vehicle;

[0152] a weighting module, configured to perform a weighted summation on the merging priority, the lane end priority and the headway priority of the autonomous driving vehicle to obtain a priority index of the autonomous driving vehicle;

[0153] The merging priority of the autonomous driving vehicle located in the merging lane is higher than that of the autonomous driving vehicle located in the main lane;

[0154] For multiple autonomous driving vehicles on a merging lane, the closer they are to the end point of the merging lane, the higher the priority of the end point of the lane;

[0155] The smaller the headway time is, the higher the headway time priority of the autonomous driving vehicle.

[0156] In one possible implementation, the screening module is used to determine the safety margin of the current autonomous driving vehicle at the i-th time step for each valid action in the valid action set; determine the valid action with the largest safety margin as the optimal action, and determine the optimal action as the actual action of the current autonomous driving vehicle in the i-th time step.

[0157] In a possible implementation, the device further includes:

[0158] A training unit is used for, during the process of training the multi-vehicle reinforcement learning model, when the current state space of multiple reinforcement learning vehicles is input into the policy network, the policy network obtains an initial action set in the environment and a labeled set after the validity is labeled for each action in the initial action set, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training; a valid action set is obtained according to the initial action set and the labeled set; the policy network processes the current state space and filters out the current action space of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after the current action space is valued based on the value network.

[0159] In a possible implementation, the device further includes:

[0160] a prediction unit, configured to, before determining whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles, predict the driving trajectory of the manually driven vehicles within the preset time range based on the manually driven vehicle behavior prediction model in the case where there are manually driven vehicles around the autonomous driving vehicle, so as to obtain a predicted trajectory of each of the manually driven vehicles;

[0161] The judgment module is also used to judge whether there is a risk of collision between the current autonomous driving vehicle and other surrounding vehicles within the i-th time step based on the trajectory predicted based on the exploratory action within the i-th time step, the trajectory predicted based on the target action of other autonomous driving vehicles and the predicted trajectory of the manually driven vehicle.

[0162] In a possible implementation, the manually driven vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model;

[0163] The prediction unit comprises:

[0164] A simulation module, used for simulating the real-time driving behavior of each of the manually driven vehicles based on the car-following model;

[0165] A lane-changing decision module, used for obtaining a target lane of the manually driven vehicle according to the real-time driving behavior of the manually driven vehicle and the lane-changing decision model;

[0166] The prediction module is used to predict the driving trajectory of the manually driven vehicle within the preset time range based on the vehicle kinematic model and the target lane, so as to obtain the predicted trajectory of the manually driven vehicle.

[0167] The decision-making device for multi-vehicle reinforcement learning provided by the embodiment of the present application can first assign a higher priority index to the autonomous driving vehicle with a higher probability of collision risk, and determine the actual action of each autonomous driving vehicle within the preset time range in the future in order of priority index from high to low. Specifically, after obtaining the exploratory action of each autonomous driving vehicle at the i-th time step output by the multi-vehicle reinforcement learning model, the exploratory action is not directly used as the actual action, but first determines whether it will collide with other vehicles if driving based on the exploratory action. Only when there is no risk of collision, the exploratory action will be used as the actual action. When there is a risk of collision, the safety margin of the current autonomous driving vehicle at the i-th time step will be combined to select the best action from the effective action set as the actual action of the i-th time step. It can be seen that the embodiment of the present application can efficiently manage computing resources and ensure real-time decision-making capabilities in multi-vehicle scenarios by introducing a priority-based computing resource allocation mechanism, thereby solving the problem of rapid increase in computing complexity in multi-vehicle environments. By combining multi-step predictions and introducing a safety assurance mechanism in multi-vehicle scenarios, the explainability of the decision-making process is effectively improved. It is able to predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0168] In addition, the embodiments of the present application can also achieve the following technical effects:

[0169] 1. The embodiment of the present application determines the priority index of each autonomous driving vehicle by combining multiple factors such as merging priority, lane end priority and headway priority, thereby improving the accuracy and reliability of the priority index.

[0170] 2. In the process of training the multi-vehicle reinforcement learning model, the embodiment of the present application annotates the validity of each action in the initial action set in advance. When training the policy network, the effective action set can be quickly screened out from the initial action set through the annotated set, and decisions can be made based on the effective action set. It can be seen that the embodiment of the present application excludes invalid or unsafe actions in multi-vehicle reinforcement learning by adopting an invalid action masking mechanism, ensuring that the decision-making process is safer and more effective, and also accelerating the convergence of the multi-vehicle reinforcement learning model.

[0171] 3. In the scenario where autonomous driving and manual driving are combined, the embodiments of the present application predict the trajectory of the manually driven vehicle within a preset time range through the manually driven vehicle behavior prediction model, and when performing multi-step predictions for each autonomous driving vehicle, in addition to considering the risk of collision with other autonomous driving vehicles, the risk of collision with manually driven vehicles is also considered, thereby further improving the accuracy of driving decisions and thus improving driving safety.

[0172] Based on the above method embodiments, another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method described in any possible implementation manner above is implemented.

[0173] Based on the above method embodiment, another embodiment of the present application provides an electronic device, the electronic device comprising:

[0174] one or more processors;

[0175] The processor is coupled to a storage device, and the storage device is used to store one or more programs;

[0176] When the one or more programs are executed by the one or more processors, the electronic device implements the method described in any one of the embodiments.

[0177] The above electronic device embodiments all correspond to the system embodiments and have the same technical effects. For specific descriptions, please refer to the system embodiments. A person skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present application.

[0178] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A decision-making method for multi-vehicle reinforcement learning, characterized in that: The method comprises: According to the preset priority allocation rules, a priority index is assigned to each autonomous driving vehicle controlled by the multi-vehicle reinforcement learning model, wherein the probability of collision risk of the autonomous driving vehicle is positively correlated with the priority index; Traversing each of the autonomous driving vehicles in order of priority index from high to low; For the traversed current autonomous driving vehicle, obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, and judge whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of other autonomous driving vehicles, wherein when the other autonomous driving vehicle is an untraversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle in the most recent driving, and when the other autonomous driving vehicle is an already traversed autonomous driving vehicle, the target action of the other autonomous driving vehicle is the actual action of the other autonomous driving vehicle in the i-th time step; If there is no collision risk, the exploratory action in the i-th time step is used as the actual action of the current autonomous driving vehicle in the i-th time step; If there is a collision risk, based on the safety margin of the current autonomous driving vehicle at the i-th time step, select the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step; After i+1, a new i is obtained, and the execution step is returned to obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual action of the current autonomous driving vehicle in each time step within the preset time range is determined, and then the traversal of the next autonomous driving vehicle is continued.

2. The method according to claim 1, characterized in that According to the preset priority allocation rules, a priority index is assigned to each autonomous driving vehicle controlled by the multi-vehicle reinforcement learning model, including: For each of the autonomous driving vehicles controlled based on the multi-vehicle reinforcement learning model, respectively calculating a merging priority, a lane end priority, and a headway priority of the autonomous driving vehicle; Performing a weighted summation on the merging priority, the lane end priority, and the headway priority of the autonomous driving vehicle to obtain a priority index of the autonomous driving vehicle; The merging priority of the autonomous driving vehicle located in the merging lane is higher than that of the autonomous driving vehicle located in the main lane; For multiple autonomous driving vehicles on a merging lane, the closer they are to the end point of the merging lane, the higher the priority of the end point of the lane; The smaller the headway time is, the higher the headway time priority of the autonomous driving vehicle.

3. The method according to claim 1, characterized in that Based on the safety margin of the current autonomous driving vehicle at the i-th time step, selecting the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step includes: For each valid action in the valid action set, respectively determine the safety margin of the current automatic driving vehicle at the i-th time step; The effective action with the largest safety margin is determined as the optimal action, and the optimal action is determined as the actual action of the current autonomous driving vehicle in the i-th time step.

4. The method according to claim 1, characterized in that The method further comprises: In the process of training the multi-vehicle reinforcement learning model, when the current state spaces of the multiple reinforcement learning vehicles are input into the policy network, the policy network obtains an initial action set in the environment and a label set after the effectiveness of each action in the initial action set is labeled, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training; Obtaining a valid action set according to the initial action set and the annotation set; The policy network processes the current state space and selects the current action spaces of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after performing value evaluation on the current action space based on the value network.

5. The method according to any one of claims 1 to 4, characterized in that Before determining whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles, the method further includes: In the case where there are manually driven vehicles around the autonomous driving vehicle, predicting the driving trajectories of the manually driven vehicles within the preset time range based on the manually driven vehicle behavior prediction model to obtain a predicted trajectory of each of the manually driven vehicles; The determining whether there is a collision risk between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of the other autonomous driving vehicles includes: Based on the trajectory predicted based on the exploratory action within the i-th time step, the trajectory predicted based on the target action of other autonomous driving vehicles and the predicted trajectory of the manually driven vehicle, it is determined whether there is a risk of collision between the current autonomous driving vehicle and other surrounding vehicles within the i-th time step.

6. The method according to claim 5, characterized in that The artificial driving vehicle behavior prediction model includes a car-following model, a lane-changing decision model and a vehicle kinematics model; Predicting the driving trajectory of the manually driven vehicle within the preset time range based on the manually driven vehicle behavior prediction model to obtain the predicted trajectory of each manually driven vehicle includes: Simulating the real-time driving behavior of each of the manually driven vehicles based on the car-following model; Obtaining a target lane for the manually driven vehicle according to the real-time driving behavior and lane change decision model of the manually driven vehicle; Based on the vehicle kinematic model and the target lane, the driving trajectory of the manually driven vehicle within the preset time range is predicted to obtain the predicted trajectory of the manually driven vehicle.

7. A decision-making device for multi-vehicle reinforcement learning, characterized in that: The device comprises: An allocating unit, configured to allocate a priority index to each autonomous driving vehicle controlled based on the multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of a collision risk of the autonomous driving vehicle is positively correlated with the priority index; A traversal unit, used to traverse each of the autonomous driving vehicles in order of priority index from high to low; The traversal unit comprises: An acquisition module, configured to acquire, for a traversed current autonomous driving vehicle, an exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model; a judgment module, configured to judge whether there is a risk of collision between the current autonomous driving vehicle and the other autonomous driving vehicles in the i-th time step according to a trajectory predicted based on the exploratory action in the i-th time step and a trajectory predicted based on the target action of the other autonomous driving vehicles, wherein when the other autonomous driving vehicles are untraversed autonomous driving vehicles, the target action of the other autonomous driving vehicles is the actual action of the other autonomous driving vehicles most recently driven, and when the other autonomous driving vehicles are traversed autonomous driving vehicles, the target action of the other autonomous driving vehicles is the actual action of the other autonomous driving vehicles in the i-th time step; a determination module, configured to use the exploratory action in the i-th time step as the actual action of the current autonomous driving vehicle in the i-th time step if there is no collision risk; a screening module, configured to screen out the best action from the valid action set as the actual action of the current autonomous driving vehicle in the i-th time step based on the safety margin of the current autonomous driving vehicle at the i-th time step if there is a collision risk; The traversal unit is used to obtain a new i after i+1, and return to the execution step to obtain the exploratory action of the current autonomous driving vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual action of the current autonomous driving vehicle in each time step within a preset time range is determined, and then continue to traverse the next autonomous driving vehicle.

8. The device according to claim 7, characterized in that The distribution unit comprises: a calculation module, configured to calculate, for each of the autonomous driving vehicles controlled based on the multi-vehicle reinforcement learning model, a merging priority, a lane end priority, and a headway priority of the autonomous driving vehicle; a weighting module, configured to perform a weighted summation on the merging priority, the lane end priority and the headway priority of the autonomous driving vehicle to obtain a priority index of the autonomous driving vehicle; The merging priority of the autonomous driving vehicle located in the merging lane is higher than that of the autonomous driving vehicle located in the main lane; For multiple autonomous driving vehicles on a merging lane, the closer they are to the end point of the merging lane, the higher the priority of the end point of the lane; The smaller the headway time is, the higher the headway time priority of the autonomous driving vehicle.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; The processor is coupled to a storage device, and the storage device is used to store one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the method according to any one of claims 1 to 6.

Images