Decision-making methods, devices, storage media and equipment for multi-vehicle reinforcement learning

Abstract

This application discloses a decision-making method, apparatus, storage medium, and device for multi-vehicle reinforcement learning, comprising: assigning a priority index to each vehicle controlled based on a multi-vehicle reinforcement learning model; traversing each vehicle in descending order of priority index; for the current vehicle being traversed, obtaining the exploratory action output by the multi-vehicle reinforcement learning model within the i-th time step, and determining whether there is a collision risk between the current vehicle and other vehicles within the i-th time step; if not, taking the exploratory action within the i-th time step as the actual action of the current vehicle within the i-th time step; otherwise, based on the safety margin of the current vehicle at the i-th time step, selecting the actual action of the current vehicle within the i-th time step from the set of valid actions; incrementing i by 1 to obtain a new i, and returning to execute the step of obtaining the exploratory action within the i-th time step, until the actual action of the current vehicle at each time step within a preset time range is determined, and then continuing to traverse the next vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a decision method and device for multi-vehicle reinforcement learning, a storage medium and equipment. BACKGROUND

[0002] With the rapid development of artificial intelligence, in the field of autonomous driving, reinforcement learning algorithms have gradually become an important tool for optimizing vehicle decision-making. These algorithms learn and feedback on a large amount of environmental data, enabling vehicles to make autonomous decisions in complex and dynamic traffic scenarios. Especially in multi-vehicle cooperative driving, vehicle interaction and coordination are crucial, and multi-vehicle reinforcement learning provides an adaptive solution that enables multiple vehicles to adjust strategies in real time to respond to unexpected situations.

[0003] In complex traffic scenarios, vehicle flow is highly dynamic, with large differences in vehicle speed and unpredictable driver behavior, increasing the risk of collisions. Therefore, existing reinforcement learning algorithms may lead to unstable decision-making in such scenarios, making it difficult to balance the safety needs of multiple vehicles. The "black box" nature of reinforcement learning algorithms makes their decision-making process lack explainability, which is particularly prominent in multi-vehicle environments. Current research focuses on maximizing overall system benefits, ignoring single-vehicle safety considerations. For example, some Q-learning-based algorithms focus on balancing decisions in cooperative or competitive scenarios, often producing unpredictable behavior in dynamic and complex traffic situations, failing to effectively address potential collision risks in multi-vehicle interactions.

[0004] Therefore, how to improve the safety of vehicles based on multi-vehicle reinforcement learning decision-making is one of the problems to be solved. SUMMARY

[0005] The present application provides a decision method and device for multi-vehicle reinforcement learning, a storage medium and equipment, which can improve the safety of vehicles based on multi-vehicle reinforcement learning decision-making.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect, the present application provides a decision method for multi-vehicle reinforcement learning, comprising:

[0008] According to a preset priority allocation rule, a priority index is assigned to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model, wherein the probability of collision risk of the autonomous vehicle is positively correlated with the priority index;

[0009] According to the order of priority index from high to low, each of the autonomous vehicles is traversed;

[0010] For the currently traversed autonomous vehicle, the exploratory actions of the current autonomous vehicle in the future i-th time step are obtained from the output of the multi-vehicle reinforcement learning model. Based on the trajectory predicted by the exploratory actions in the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, it is determined whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step. When the other autonomous vehicle is an untraversed autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle that has been driven the most recently. When the other autonomous vehicle is a traversed autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle in the i-th time step.

[0011] If there is no risk of collision, the exploratory action in the i-th time step shall be regarded as the actual action of the current autonomous vehicle in the i-th time step.

[0012] If there is a collision risk, the best action is selected from the set of valid actions based on the safety margin of the current autonomous vehicle at the i-th time step, and is taken as the actual action of the current autonomous vehicle in the i-th time step.

[0013] After incrementing i by 1, a new i is obtained, and the process returns to the step of obtaining the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model. This continues until the actual actions of the current autonomous vehicle in each time step within a preset time range are determined, and then the process continues to traverse the next autonomous vehicle.

[0014] In one possible implementation, a priority index is assigned to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model according to a preset priority allocation rule, including:

[0015] For each of the autonomous vehicles controlled based on the multi-vehicle reinforcement learning model, the merging priority, lane end priority, and headway priority of the autonomous vehicle are calculated respectively.

[0016] The priority index of the autonomous vehicle is obtained by weighted summing of the merging priority, the lane end priority, and the headway priority.

[0017] Among them, the merging priority of the autonomous vehicles located in the merging lane is higher than that of the autonomous vehicles located in the main lane;

[0018] For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher the priority of the end of the lane.

[0019] The smaller the headway, the higher the headway priority of the autonomous vehicle.

[0020] In one possible implementation, based on the safety margin of the current autonomous vehicle at the i-th time step, the optimal action is selected from the set of valid actions as the actual action of the current autonomous vehicle in the i-th time step, including:

[0021] For each valid action in the set of valid actions, determine the safety margin of the current autonomous vehicle at the i-th time step;

[0022] The effective action with the greatest safety margin is determined as the optimal action, and the optimal action is determined as the actual action of the current autonomous vehicle in the i-th time step.

[0023] In one possible implementation, the method further includes:

[0024] During the training of the multi-vehicle reinforcement learning model, when the current state space of multiple reinforcement learning vehicles is input into the policy network, the policy network obtains an initial action set in the environment and a label set after valid labeling for each action in the initial action set, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training.

[0025] Based on the initial action set and the annotation set, a valid action set is obtained;

[0026] The policy network processes the current state space and selects the current action space of the multiple reinforcement learning vehicles from the set of effective actions, so that the policy network can continue to be trained after the current action space is evaluated based on the value network.

[0027] In one possible implementation, before determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory actions within the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, the method further includes:

[0028] When there are human-driven vehicles around the autonomous vehicle, the driving trajectory of the human-driven vehicles within the preset time range is predicted based on the human-driven vehicle behavior prediction model to obtain the predicted trajectory of each human-driven vehicle.

[0029] The step of determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory actions in the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles includes:

[0030] Based on the trajectory predicted by the exploratory actions in the i-th time step, the trajectory predicted by the target actions of other autonomous vehicles, and the predicted trajectory of the human-driven vehicle, it is determined whether there is a collision risk between the current autonomous vehicle and other surrounding vehicles in the i-th time step.

[0031] In one possible implementation, the artificial driving vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model.

[0032] Based on a human-driven vehicle behavior prediction model, the driving trajectory of the human-driven vehicle within the preset time range is predicted to obtain the predicted trajectory of each human-driven vehicle, including:

[0033] The real-time driving behavior of each of the manually driven vehicles is simulated based on the car-following model.

[0034] Based on the real-time driving behavior and lane-changing decision model of the manually driven vehicle, the target lane of the manually driven vehicle is obtained.

[0035] Based on the vehicle kinematics model and the target lane, the driving trajectory of the manually driven vehicle within the preset time range is predicted to obtain the predicted trajectory of the manually driven vehicle.

[0036] Secondly, embodiments of this application provide a decision-making apparatus for multi-vehicle reinforcement learning, the apparatus comprising:

[0037] The allocation unit is used to assign a priority index to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of an autonomous vehicle causing a collision is positively correlated with the priority index.

[0038] A traversal unit is used to traverse each of the autonomous vehicles in descending order of priority index;

[0039] The traversal unit includes:

[0040] The action acquisition module is used to acquire the exploratory actions of the current autonomous vehicle in the future i-th time step, as output by the multi-vehicle reinforcement learning model, for the current autonomous vehicle being traversed.

[0041] The judgment module is used to determine whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory action in the i-th time step and the trajectory predicted by the target action of other autonomous vehicles. When the other autonomous vehicle is an unvisited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle that has been driven the most recently. When the other autonomous vehicle is a visited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle in the i-th time step.

[0042] The determination module is used to determine the exploratory action in the i-th time step as the actual action of the current autonomous vehicle in the i-th time step if there is no collision risk.

[0043] The filtering module is used to select the best action from the set of valid actions based on the safety margin of the current autonomous vehicle at the i-th time step if there is a collision risk, and use it as the actual action of the current autonomous vehicle in the i-th time step.

[0044] The traversal unit is used to obtain a new i by adding i+1, and return to the execution step to obtain the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual actions of the current autonomous vehicle in each time step within the preset time range are determined, and then continue to traverse the next autonomous vehicle.

[0045] In one possible implementation, the allocation unit includes:

[0046] The calculation module is used to calculate the merging priority, lane end priority, and headway priority of each autonomous vehicle controlled based on the multi-vehicle reinforcement learning model.

[0047] The weighting module is used to perform a weighted summation of the merging priority, lane end priority, and headway priority of the autonomous vehicle to obtain the priority index of the autonomous vehicle.

[0048] Among them, the merging priority of the autonomous vehicles located in the merging lane is higher than that of the autonomous vehicles located in the main lane;

[0049] For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher the priority of the end of the lane.

[0050] The smaller the headway, the higher the headway priority of the autonomous vehicle.

[0051] In one possible implementation, the filtering module is configured to determine the safety margin of the current autonomous vehicle at the i-th time step for each valid action in the set of valid actions; determine the valid action with the largest safety margin as the optimal action, and determine the optimal action as the actual action of the current autonomous vehicle in the i-th time step.

[0052] In one possible implementation, the device further includes:

[0053] The training unit is configured to, during the training of the multi-vehicle reinforcement learning model, when the current state spaces of multiple reinforcement learning vehicles are input into the policy network, obtain an initial action set in the environment and a label set after valid annotation for each action in the initial action set, wherein the reinforcement learning vehicles are autonomous vehicles used for reinforcement learning training; obtain a valid action set based on the initial action set and the label set; process the current state space and filter the current action spaces of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after evaluating the value of the current action spaces based on the value network.

[0054] In one possible implementation, the device further includes:

[0055] The prediction unit is used to predict the driving trajectory of the human-driven vehicle within a preset time range based on the trajectory predicted by the exploratory action within the i-th time step and the trajectory predicted by the target action of other autonomous vehicles, in the case where there are human-driven vehicles around the autonomous vehicle, before determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles within the i-th time step.

[0056] The judgment module is further configured to determine whether there is a collision risk between the current autonomous vehicle and other surrounding vehicles in the i-th time step based on the trajectory predicted by the exploratory action in the i-th time step, the trajectory predicted by the target action of other autonomous vehicles, and the predicted trajectory of the human-driven vehicle.

[0057] In one possible implementation, the artificial driving vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model.

[0058] The prediction unit includes:

[0059] The simulation module is used to simulate the real-time driving behavior of each of the manually driven vehicles based on the car-following model.

[0060] The lane change decision module is used to obtain the target lane of the manually driven vehicle based on the real-time driving behavior of the manually driven vehicle and the lane change decision model.

[0061] The prediction module is used to predict the driving trajectory of the manually driven vehicle within the preset time range based on the vehicle kinematics model and the target lane, so as to obtain the predicted trajectory of the manually driven vehicle.

[0062] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any possible implementation of the first aspect.

[0063] Fourthly, embodiments of this application provide an electronic device, the electronic device comprising:

[0064] One or more processors;

[0065] The processor is coupled to a storage device for storing one or more programs;

[0066] When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any possible implementation of the first aspect.

[0067] The decision-making method, apparatus, storage medium, and device for multi-vehicle reinforcement learning provided in this application can first assign higher priority indices to autonomous vehicles with higher collision risk probabilities, and then determine the actual actions of each autonomous vehicle within a preset future time range according to the order of priority indices from high to low. Specifically, after obtaining the exploratory action of each autonomous vehicle at time step i from the output of the multi-vehicle reinforcement learning model, the exploratory action is not directly used as the actual action. Instead, it is first determined whether driving based on the exploratory action will result in a collision with other vehicles. Only when there is no collision risk will the exploratory action be used as the actual action. If there is a collision risk, the best action is selected from the set of effective actions based on the safety margin of the current autonomous vehicle at time step i. Thus, this application, by introducing a priority-based computing resource allocation mechanism, can efficiently manage computing resources, ensure real-time decision-making capabilities in multi-vehicle scenarios, and thus solve the problem of rapidly increasing computational complexity in multi-vehicle environments. By combining multi-step prediction and introducing a safety assurance mechanism in multi-vehicle scenarios, the interpretability of the decision-making process is effectively improved. It can predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0068] Furthermore, the embodiments of this application can also achieve the following technical effects:

[0069] 1. The embodiments of this application comprehensively determine the priority index of each autonomous vehicle by combining multiple factors such as merging priority, lane end priority, and headway priority, which can improve the accuracy and reliability of the priority index.

[0070] 2. In the training of the multi-vehicle reinforcement learning model, this embodiment of the application pre-labels each action in the initial action set for validity. During the training of the policy network, it can quickly filter out the valid action set from the initial action set using the labeled set, and make decisions based on the valid action set. Therefore, this embodiment of the application, by employing an invalid action masking mechanism, eliminates invalid or unsafe actions in multi-vehicle reinforcement learning, ensuring a safer and more effective decision-making process, and also accelerating the convergence of the multi-vehicle reinforcement learning model.

[0071] 3. In scenarios combining autonomous driving and manual driving, the embodiments of this application predict the trajectory of a manually driven vehicle within a preset time range through a manual driving vehicle behavior prediction model. Furthermore, when making multi-step predictions for each autonomous vehicle, in addition to considering the collision risk with other autonomous vehicles, the collision risk with manually driven vehicles is also considered, thereby further improving the accuracy of driving decisions and thus improving driving safety. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0073] Figure 1 A flowchart illustrating a decision-making method for multi-vehicle reinforcement learning provided in an embodiment of this application;

[0074] Figure 2 A schematic diagram of priority allocation parameters provided in an embodiment of this application;

[0075] Figure 3 An example diagram illustrating a trajectory conflict provided in an embodiment of this application;

[0076] Figure 4 A schematic diagram illustrating the definition of safety margin for an autonomous vehicle whose action is lane changing, provided in an embodiment of this application;

[0077] Figure 5 A schematic diagram illustrating the definition of safety margin for an autonomous vehicle whose action is straight-line driving, provided for an embodiment of this application;

[0078] Figure 6 A schematic diagram of a neural network architecture that incorporates action occlusion, provided as an embodiment of this application;

[0079] Figure 7 This is a block diagram of a decision-making device for multi-vehicle reinforcement learning provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0081] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0082] Figure 1 This is a flowchart illustrating a decision-making method based on reinforcement learning for multiple vehicles. This method can be applied to electronic devices, specifically to autonomous vehicles or cloud servers interacting with autonomous vehicles. The method includes:

[0083] S110: Assign a priority index to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model according to a preset priority allocation rule.

[0084] In practical applications, there can be multiple preset priority allocation rules, and different preset priority allocation rules can be set for different road scenarios. However, all preset priority allocation rules must meet the following conditions: the greater the probability of a collision, the higher the priority index of the corresponding autonomous vehicle. That is, the probability of an autonomous vehicle colliding is positively correlated with the priority index.

[0085] The following uses a ramp merging scenario as an example to describe the priority index allocation rules in detail:

[0086] For each autonomous vehicle controlled by a multi-vehicle reinforcement learning model, the merging priority, lane end priority, and headway priority of the autonomous vehicle are calculated respectively. The merging priority, lane end priority, and headway priority of the autonomous vehicle are weighted and summed to obtain the priority index of the autonomous vehicle.

[0087] In this scenario, autonomous vehicles located in merging lanes have a higher merging priority than those located in main lanes. This is because autonomous vehicles in merging lanes face time-sensitive merging tasks (due to impassable ends of the merging lanes) and need to complete the merging operation more quickly. By prioritizing autonomous vehicles in merging lanes, it is possible to ensure that they integrate more smoothly into the traffic flow.

[0088] For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher their priority at the end of the lane. This is because vehicles closer to the end of the merging lane are more likely to cause collisions and lock-up situations. Autonomous vehicles in the merging lane near the end of the merging lane are more susceptible to traffic disruptions, thus requiring greater attention in terms of safety. Giving these vehicles higher priority can more effectively avoid potential collision risks and improve the safety of the entire system.

[0089] The smaller the headway, the higher the headway priority for autonomous vehicles. This is because vehicles with smaller headway are more likely to collide with the vehicle in front. At intersections and merging areas, smaller headway distances can increase the probability of intersections between adjacent vehicles, thus increasing the risk of collisions. By prioritizing these vehicles, their actions can be more carefully planned, reducing potential intersection conflicts and improving the overall smoothness and safety of the traffic system.

[0090] Based on the above settings, the priority index p of the autonomous vehicle i in the multi-vehicle reinforcement learning control in this embodiment of the application is... i The formula is as follows:

[0091] p i =α1p m +α2p d +α3p h +σ i

[0092] Where α1, α2, and α3 are the merging priorities p, respectively. m Lane End Priority p d Priority of headway p h The weighting factor, and It is a small random variable that can prevent two autonomous vehicles from having the same priority index. Specifically:

[0093] Merge priority p m :

[0094]

[0095] Lane End Priority p d :

[0096]

[0097] Headway priority p h :

[0098]

[0099] Among them, such as Figure 2As shown, x represents the distance the autonomous vehicle travels in the merging lane, and L represents the length of the merging lane. headway t represents the distance t represents the headway of an autonomous vehicle. h This indicates the pre-set safe headway threshold, v t This indicates the current speed of the autonomous vehicle.

[0100] exist Figure 2 In this diagram, both black and white rectangles represent vehicles. Black rectangles represent autonomous vehicles, while white rectangles can represent either autonomous or manually driven vehicles. This application's embodiments focus only on the relationship between the autonomous vehicle requiring priority allocation and surrounding vehicles and the road in which it is located, without considering the type of vehicles surrounding the autonomous vehicle requiring priority allocation.

[0101] S120: Traverse each autonomous vehicle in descending order of priority index.

[0102] S130: For the current autonomous vehicle being traversed, obtain the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model. Based on the trajectory predicted by the exploratory actions in the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, determine whether there is a collision risk between the current autonomous vehicle and other autonomous vehicles in the i-th time step.

[0103] Specifically, when other autonomous vehicles are unvisited, their target action is the most recently driven action of that vehicle. When other autonomous vehicles are already visited, their target action is the actual action taken by that vehicle within the i-th time step. Both exploratory and actual actions refer to driving actions, such as turning left, turning right, driving straight at a constant speed, accelerating straight, decelerating straight, and idling straight.

[0104] i is a positive integer. When i = 1, the input data of the multi-vehicle reinforcement learning model includes the real state space of multiple autonomous vehicles when they begin to control the vehicle, and the output data includes the exploratory actions of the current autonomous vehicle in the first time step in the future. When i > 1, the input data of the multi-vehicle reinforcement learning model includes the state space corresponding to the actual actions (planned actual actions) of the current autonomous vehicle in the (i-1)th time step, and the state space corresponding to the target actions of other autonomous vehicles when planning the actions of the current autonomous vehicle in the (i-1)th time step. The output data includes the exploratory actions of the current autonomous vehicle in the i-th time step in the future.

[0105] When determining whether there is a collision risk between the current autonomous vehicle and other autonomous vehicles in the i-th time step, a collision risk is determined to exist as long as there is a distance between the current autonomous vehicle and any other autonomous vehicle that is less than the preset safe distance. That is, no collision risk is determined only when the distance between the current autonomous vehicle and all other autonomous vehicles is greater than or equal to the preset safe distance.

[0106] S140: If there is no collision risk, the exploratory action in the i-th time step shall be taken as the actual action of the current autonomous vehicle in the i-th time step.

[0107] If there is no risk of collision, it means that the exploratory action in the i-th time step is safe and reliable, and can be directly taken as the actual action of the current autonomous vehicle in the i-th time step.

[0108] S150: If there is a collision risk, the best action is selected from the set of valid actions based on the safety margin of the current autonomous vehicle at time step i, and this action is taken as the actual action of the current autonomous vehicle at time step i.

[0109] If a collision risk exists, the exploratory action in time step i is unsafe, and a safer action needs to be found to replace it. To obtain a safer action, embodiments of this application can select the optimal action from the set of valid actions based on the safety margin of the current autonomous vehicle at time step i, and use this as the actual action of the current autonomous vehicle in time step i. For example... Figure 3 As shown, if the autonomous vehicle represented by the black rectangle in the figure wants to switch to the left lane, it may collide with the vehicle in the left lane at time steps k=3, 4, and 5. Therefore, the exploratory actions of the autonomous vehicle represented by the black rectangle at time steps 3, 4, and 5 are unreasonable and other actions need to be re-determined.

[0110] Specifically, for each valid action in the set of valid actions, the safety margin of the current autonomous vehicle at time step i can be determined first; then the valid action with the largest safety margin can be determined as the optimal action, and the optimal action can be determined as the actual action of the current autonomous vehicle in time step i.

[0111] The set of valid actions can include left turn, right turn, acceleration, idling, and deceleration. If the valid action is a left turn or right turn, the safety margin of the current autonomous vehicle at time step i is: the minimum distance between the current autonomous vehicle and the vehicles in front and behind it in the target lane to which it needs to change lanes. This minimum distance refers to the distance between the two vehicle heads along the direction of travel on the road, such as... Figure 4As shown, when the autonomous vehicle, represented by the black rectangle, wants to change lanes to the left, the minimum distance between the autonomous vehicle and the vehicles in front and behind it in the left lane can be determined. The minimum distance in the figure is d. sm If the effective action is acceleration, idling, or deceleration, then the safety margin of the current autonomous vehicle at time step i is: the minimum distance between the current autonomous vehicle and the vehicles in front and behind it in the same lane, such as... Figure 5 As shown, when the autonomous vehicle, represented by the black rectangle, wants to continue driving straight (accelerating, idling, and decelerating, etc.), the minimum distance between the autonomous vehicle and the vehicles in front and behind it in the same lane can be determined. The minimum distance in the figure is d. sm .

[0112] S160: After adding i+1, obtain the new i, and return to the execution step to obtain the exploratory actions of the current autonomous vehicle in the i-th time step in the future, until the actual actions of the current autonomous vehicle in each time step within the preset time range are determined, and then continue to traverse the next autonomous vehicle.

[0113] After incrementing i by 1, obtain the new i, and return to execute steps S130-S150 to determine the actual action of the current autonomous vehicle in the new i-th time step. Continue until the current autonomous vehicle has completed its actual action in each time step within the preset time range, then complete the process for the current autonomous vehicle and continue to traverse the next autonomous vehicle.

[0114] The decision-making method for multi-vehicle reinforcement learning provided in this application first assigns higher priority indices to autonomous vehicles with higher collision risk probabilities, and then determines the actual actions of each autonomous vehicle within a preset time range in descending order of priority indices. Specifically, after obtaining the exploratory action of each autonomous vehicle at time step i from the multi-vehicle reinforcement learning model, the exploratory action is not directly used as the actual action. Instead, it first determines whether driving based on the exploratory action will result in a collision with other vehicles. Only when there is no collision risk will the exploratory action be used as the actual action. If there is a collision risk, the best action is selected from the set of valid actions based on the safety margin of the current autonomous vehicle at time step i. Thus, this application, by introducing a priority-based computing resource allocation mechanism, can efficiently manage computing resources, ensure real-time decision-making capabilities in multi-vehicle scenarios, and solve the problem of rapidly increasing computational complexity in multi-vehicle environments. By combining multi-step prediction and introducing a safety assurance mechanism in multi-vehicle scenarios, the interpretability of the decision-making process is effectively improved. It can predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0115] In one possible implementation, several issues arise during the action sampling process in the training of a multi-vehicle reinforcement learning model. The core problems lie in two main challenges that may arise when sampling actions using a random policy. First, invalid or unsafe actions are assigned non-zero probabilities, meaning that during training, the model might choose to execute these unsafe actions, leading to poor system behavior or even system crashes. Second, because invalid actions also participate in policy updates, they lead to the learning of invalid policies, rendering the training process ineffective, as experience associated with invalid actions is meaningless. To address these issues, embodiments of this application introduce a method called invalid action masking during the training of the multi-vehicle reinforcement learning model. This method ensures that only valid and safe actions are included in the model's training, thereby accelerating the model's convergence speed and improving training efficiency.

[0116] Specifically, during the training of the multi-vehicle reinforcement learning model, when the current state spaces of multiple reinforcement learning vehicles are input into the policy network (i.e., the Actor network), the policy network obtains an initial action set in the environment and a label set after valid annotation for each action in the initial action set. Here, the reinforcement learning vehicle is the autonomous vehicle used for reinforcement learning training. Based on the initial action set and the label set, a valid action set is obtained. The policy network processes the current state space and selects the current action spaces of multiple reinforcement learning vehicles from the valid action set. After the current action space is evaluated based on the value network (i.e., the Critic network), the policy network continues to be trained.

[0117] When labeling the validity of each action in the initial action set, a rule can be first set that all actions must be invalid, and then the remaining actions can be labeled as valid. The rules for invalid actions include, but are not limited to, the following two:

[0118] (1) An attempt by a vehicle to change lanes to a non-existent lane is an invalid action;

[0119] (2) When the speed of the vehicle has reached the predefined maximum or minimum speed, the vehicle attempts to accelerate or decelerate, which is an invalid action.

[0120] In this embodiment, valid actions can be marked as "1" and invalid actions as "0". Then, the initial action set and the marked set can be directly multiplied to obtain the valid action set. Figure 6As shown, the state information (including position and speed) of other autonomous vehicles and the state information (including position and speed) of the target autonomous vehicle can be fed into the Actor network after passing through the FC (Fully Connected Network). In the Actor network, the logical value sequence of the initial action set ([5,5,5,5,5]) and the logical value sequence of the label set ([1,1,1,0,1]) can be multiplied, and the logical value sequence of the effective action set can be obtained by sampling and evaluating each action. This makes the probability of corresponding invalid actions very close to zero after passing through the Softmax layer of the neural network. Therefore, the model will hardly select these invalid actions during the sampling stage.

[0121] In the training of a multi-vehicle reinforcement learning model, this embodiment pre-labels each action in the initial action set for validity. During policy network training, the label set allows for rapid selection of valid actions from the initial action set, and decision-making is based on this valid action set. Therefore, this embodiment employs an invalid action masking mechanism to eliminate invalid or unsafe actions in multi-vehicle reinforcement learning, ensuring a safer and more effective decision-making process and accelerating the convergence of the multi-vehicle reinforcement learning model.

[0122] In one possible implementation, in practical applications, not only autonomous vehicles but also manually driven vehicles may exist on the same road segment. Therefore, in scenarios where manually driven vehicles are present, when predicting collisions between each autonomous vehicle and surrounding vehicles, it is also necessary to consider whether there is a potential collision risk between the autonomous vehicle and the manually driven vehicles. Thus, before determining whether there is a collision risk between the current autonomous vehicle and other autonomous vehicles in the i-th time step based on the trajectory predicted by exploratory actions within the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, when manually driven vehicles are present around the autonomous vehicle, the driving trajectory of the manually driven vehicles within a preset time range is predicted based on a human behavior prediction model, obtaining the predicted trajectory for each manually driven vehicle. Then, based on the trajectory predicted by exploratory actions within the i-th time step, the trajectory predicted by the target actions of other autonomous vehicles, and the predicted trajectory of the manually driven vehicles, it can be determined whether there is a collision risk between the current autonomous vehicle and other surrounding vehicles in the i-th time step.

[0123] Among them, the human-driven vehicle behavior prediction model includes the car-following model, the lane-changing decision model, and the vehicle kinematics model.

[0124] The method for predicting the trajectory of manually driven vehicles may include: simulating the real-time driving behavior of each manually driven vehicle based on a car-following model; obtaining the target lane of the manually driven vehicle based on its real-time driving behavior and lane-changing decision model; and predicting the driving trajectory of the manually driven vehicle within a preset time range based on the vehicle kinematics model and the target lane, thereby obtaining the predicted trajectory of the manually driven vehicle.

[0125] Among them, the following model can be selected as the IDM (Intelligent Driver Model) model or other models.

[0126] The IDM model's mathematical formula is represented by a set of equations that quantitatively describe the acceleration and deceleration behavior of vehicles in traffic flow, making the simulation closer to real-world driving scenarios. The specific formulas of the IDM model are as follows:

[0127]

[0128] The symbols in the above formula are explained as follows:

[0129] a HV The velocity v represents the acceleration of the target vehicle, HV. HV Indicates the speed (HV) of the target vehicle; s HV Δv represents the actual following distance of the target vehicle (HV). HV δ' represents the speed difference between the target vehicle HV and the vehicle in front; δ′ represents the acceleration exponent; v0 represents the desired speed of the target vehicle; s * (v HV Δv HV () indicates the desired following distance.

[0130] The lane change decision model can be either the MOBIL (Minimize Overall Braking Induced by LaneChange) model or other models.

[0131] The application of the MOBIL model and its incentive criteria helps evaluate the overall benefits of lane-changing behavior and ensures that this behavior does not adversely affect surrounding vehicles, thereby promoting the stability and efficiency of the traffic system. The incentive criteria of the MOBIL model are expressed as follows:

[0132]

[0133] The symbols in the above formula are explained as follows:

[0134] u HV Represents the total lane-changing benefit of the target vehicle HV; a HVThis indicates the acceleration of the target vehicle (HV) before it changes lanes; Indicates the acceleration of the target vehicle HV after lane change; a PFV This indicates the acceleration of vehicles following in the target lane before the target vehicle changes lanes to the target lane; This indicates the acceleration of the vehicle following in the target lane after the target vehicle changes lanes; This represents the acceleration of the vehicle following the target vehicle in its original lane after the target vehicle changes lanes; a FV Δa represents the acceleration of the vehicle behind it in the original lane before the target vehicle changes lanes; p represents the yield coefficient; Δa represents the acceleration of the vehicle behind it in the original lane before the target vehicle changes lanes. th This represents the threshold for lane-changing benefits.

[0135] The vehicle kinematics model can be a bicycle model or other models.

[0136] When the MOBIL model makes a lane-changing decision, it outputs a target lane and passes it to the vehicle kinematics model. The vehicle kinematics model uses the classic bicycle model, and then outputs the front wheel steering angle signal to predict human driving behavior. The expression of the bicycle model includes:

[0137]

[0138] The symbols in the above formula are explained as follows:

[0139] Δy represents the lateral deviation of the target vehicle from the center line of the target lane; Indicates the yaw rate required for a lane change; l f , l r , respectively, represent the distances from the center of mass of the target vehicle to the centers of its front and rear axles; β represents the sideslip angle of the target vehicle's center of mass; v represents the speed of the target vehicle; ψ represents the yaw angle of the target vehicle; δ represents the front wheel steering angle of the target vehicle.

[0140] In scenarios combining autonomous driving and manual driving, this application embodiment predicts the trajectory of a manually driven vehicle within a preset time range using a manual driving vehicle behavior prediction model. Furthermore, when making multi-step predictions for each autonomous vehicle, it considers not only the collision risk with other autonomous vehicles but also the collision risk with manually driven vehicles, thereby further improving the accuracy of driving decisions and thus enhancing driving safety.

[0141] Based on the above method embodiments, another embodiment of this application provides a decision-making device for multi-vehicle reinforcement learning, such as... Figure 7 As shown, the device includes:

[0142] Allocation unit 210 is used to assign a priority index to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of an autonomous vehicle causing a collision is positively correlated with the priority index.

[0143] Traversal unit 220 is used to traverse each of the autonomous vehicles in descending order of priority index;

[0144] The traversal unit 220 includes:

[0145] Action acquisition module 221 is used to acquire, for the current autonomous vehicle being traversed, the exploratory actions of the current autonomous vehicle output by the multi-vehicle reinforcement learning model in the future i-th time step.

[0146] The judgment module 222 is used to determine whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted based on the exploratory action in the i-th time step and the trajectory predicted based on the target action of other autonomous vehicles. When the other autonomous vehicle is an unvisited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle that has been driven the most recently. When the other autonomous vehicle is a visited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle in the i-th time step.

[0147] The determination module 223 is used to determine the exploratory action in the i-th time step as the actual action of the current autonomous vehicle in the i-th time step if there is no collision risk.

[0148] The filtering module 224 is used to filter the best action from the set of valid actions based on the safety margin of the current autonomous vehicle at the i-th time step if there is a collision risk, and use it as the actual action of the current autonomous vehicle in the i-th time step.

[0149] The traversal unit 220 is used to obtain a new i by adding i+1 and return to the execution step to obtain the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual actions of the current autonomous vehicle in each time step within the preset time range are determined, and then continue to traverse the next autonomous vehicle.

[0150] In one possible implementation, the allocation unit 210 includes:

[0151] The calculation module is used to calculate the merging priority, lane end priority, and headway priority of each autonomous vehicle controlled based on the multi-vehicle reinforcement learning model.

[0152] The weighting module is used to perform a weighted summation of the merging priority, lane end priority, and headway priority of the autonomous vehicle to obtain the priority index of the autonomous vehicle.

[0153] Among them, the merging priority of the autonomous vehicles located in the merging lane is higher than that of the autonomous vehicles located in the main lane;

[0154] For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher the priority of the end of the lane.

[0155] The smaller the headway, the higher the headway priority of the autonomous vehicle.

[0156] In one possible implementation, the filtering module is configured to determine the safety margin of the current autonomous vehicle at the i-th time step for each valid action in the set of valid actions; determine the valid action with the largest safety margin as the optimal action, and determine the optimal action as the actual action of the current autonomous vehicle in the i-th time step.

[0157] In one possible implementation, the device further includes:

[0158] The training unit is configured to, during the training of the multi-vehicle reinforcement learning model, when the current state spaces of multiple reinforcement learning vehicles are input into the policy network, obtain an initial action set in the environment and a label set after valid annotation for each action in the initial action set, wherein the reinforcement learning vehicles are autonomous vehicles used for reinforcement learning training; obtain a valid action set based on the initial action set and the label set; process the current state space and filter the current action spaces of the multiple reinforcement learning vehicles from the valid action set, so as to continue training the policy network after evaluating the value of the current action spaces based on the value network.

[0159] In one possible implementation, the device further includes:

[0160] The prediction unit is used to predict the driving trajectory of the human-driven vehicle within a preset time range based on the trajectory predicted by the exploratory action within the i-th time step and the trajectory predicted by the target action of other autonomous vehicles, in the case where there are human-driven vehicles around the autonomous vehicle, before determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles within the i-th time step.

[0161] The judgment module is further configured to determine whether there is a collision risk between the current autonomous vehicle and other surrounding vehicles in the i-th time step based on the trajectory predicted by the exploratory action in the i-th time step, the trajectory predicted by the target action of other autonomous vehicles, and the predicted trajectory of the human-driven vehicle.

[0162] In one possible implementation, the artificial driving vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model.

[0163] The prediction unit includes:

[0164] The simulation module is used to simulate the real-time driving behavior of each of the manually driven vehicles based on the car-following model.

[0165] The lane change decision module is used to obtain the target lane of the manually driven vehicle based on the real-time driving behavior of the manually driven vehicle and the lane change decision model.

[0166] The prediction module is used to predict the driving trajectory of the manually driven vehicle within the preset time range based on the vehicle kinematics model and the target lane, so as to obtain the predicted trajectory of the manually driven vehicle.

[0167] The decision-making device for multi-vehicle reinforcement learning provided in this application can first assign higher priority indices to autonomous vehicles with higher collision risk probabilities, and then determine the actual actions of each autonomous vehicle within a preset time range in descending order of priority indices. Specifically, after obtaining the exploratory action of each autonomous vehicle at time step i from the multi-vehicle reinforcement learning model, it does not directly use the exploratory action as the actual action. Instead, it first determines whether driving based on the exploratory action will result in a collision with other vehicles. Only when there is no collision risk will the exploratory action be used as the actual action. If there is a collision risk, the device will combine the safety margin of the current autonomous vehicle at time step i and select the best action from the set of valid actions as the actual action for time step i. Thus, this application, by introducing a priority-based computing resource allocation mechanism, can efficiently manage computing resources, ensure real-time decision-making capabilities in multi-vehicle scenarios, and solve the problem of rapidly increasing computational complexity in multi-vehicle environments. By combining multi-step prediction and introducing a safety assurance mechanism in multi-vehicle scenarios, the interpretability of the decision-making process is effectively improved. It can predict potential collision risks in real time and adjust unsafe behaviors in advance, thereby improving the predictability and stability of the system.

[0168] Furthermore, the embodiments of this application can also achieve the following technical effects:

[0169] 1. The embodiments of this application comprehensively determine the priority index of each autonomous vehicle by combining multiple factors such as merging priority, lane end priority, and headway priority, which can improve the accuracy and reliability of the priority index.

[0170] 2. In the training of the multi-vehicle reinforcement learning model, this embodiment of the application pre-labels each action in the initial action set for validity. During the training of the policy network, it can quickly filter out the valid action set from the initial action set using the labeled set, and make decisions based on the valid action set. Therefore, this embodiment of the application, by employing an invalid action masking mechanism, eliminates invalid or unsafe actions in multi-vehicle reinforcement learning, ensuring a safer and more effective decision-making process, and also accelerating the convergence of the multi-vehicle reinforcement learning model.

[0171] 3. In scenarios combining autonomous driving and manual driving, the embodiments of this application predict the trajectory of a manually driven vehicle within a preset time range through a manual driving vehicle behavior prediction model. Furthermore, when making multi-step predictions for each autonomous vehicle, in addition to considering the collision risk with other autonomous vehicles, the collision risk with manually driven vehicles is also considered, thereby further improving the accuracy of driving decisions and thus improving driving safety.

[0172] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above possible implementations.

[0173] Based on the above method embodiments, another embodiment of this application provides an electronic device, the electronic device comprising:

[0174] One or more processors;

[0175] The processor is coupled to a storage device for storing one or more programs;

[0176] When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any embodiment.

[0177] The above-described electronic device embodiments correspond to the system embodiments and have the same technical effects. For detailed descriptions, please refer to the system embodiments. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0178] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A decision-making method for multi-vehicle reinforcement learning, characterized in that, The method includes: According to the preset priority allocation rules, a priority index is assigned to each autonomous vehicle controlled by the multi-vehicle reinforcement learning model. The probability of an autonomous vehicle causing a collision is positively correlated with the priority index. Each of the autonomous vehicles is traversed in descending order of priority index; For the currently traversed autonomous vehicle, the exploratory actions of the current autonomous vehicle in the future i-th time step are obtained from the output of the multi-vehicle reinforcement learning model. Based on the trajectory predicted by the exploratory actions in the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, it is determined whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step. When the other autonomous vehicle is an untraversed autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle that has been driven the most recently. When the other autonomous vehicle is a traversed autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle in the i-th time step. If there is no risk of collision, the exploratory action in the i-th time step shall be regarded as the actual action of the current autonomous vehicle in the i-th time step. If there is a collision risk, the best action is selected from the set of valid actions based on the safety margin of the current autonomous vehicle at the i-th time step, and is taken as the actual action of the current autonomous vehicle in the i-th time step. After incrementing i by 1, a new i is obtained, and the process returns to the execution step to obtain the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model. This continues until the actual actions of the current autonomous vehicle in each time step within the preset time range are determined, and then the process continues to traverse the next autonomous vehicle. According to a preset priority allocation rule, a priority index is assigned to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model, including: For each of the autonomous vehicles controlled based on the multi-vehicle reinforcement learning model, the merging priority, lane end priority, and headway priority of the autonomous vehicle are calculated respectively. The priority index of the autonomous vehicle is obtained by weighted summing of the merging priority, the lane end priority, and the headway priority. Among them, the merging priority of the autonomous vehicles located in the merging lane is higher than that of the autonomous vehicles located in the main lane; For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher the priority of the end of the lane. The smaller the headway, the higher the headway priority of the autonomous vehicle.

2. The method according to claim 1, characterized in that, Based on the safety margin of the current autonomous vehicle at the i-th time step, the optimal action is selected from the set of valid actions as the actual action of the current autonomous vehicle in the i-th time step, including: For each valid action in the set of valid actions, determine the safety margin of the current autonomous vehicle at the i-th time step; The effective action with the greatest safety margin is determined as the optimal action, and the optimal action is determined as the actual action of the current autonomous vehicle in the i-th time step.

3. The method according to claim 1, characterized in that, The method further includes: During the training of the multi-vehicle reinforcement learning model, when the current state space of multiple reinforcement learning vehicles is input into the policy network, the policy network obtains an initial action set in the environment and a label set after valid labeling for each action in the initial action set, wherein the reinforcement learning vehicle is an autonomous driving vehicle used for reinforcement learning training. Based on the initial action set and the annotation set, a valid action set is obtained; The policy network processes the current state space and selects the current action space of the multiple reinforcement learning vehicles from the set of effective actions, so that the policy network can continue to be trained after the current action space is evaluated based on the value network.

4. The method according to any one of claims 1-3, characterized in that, Before determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory actions within the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles, the method further includes: When there are human-driven vehicles around the autonomous vehicle, the driving trajectory of the human-driven vehicles within the preset time range is predicted based on the human-driven vehicle behavior prediction model to obtain the predicted trajectory of each human-driven vehicle. The step of determining whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory actions in the i-th time step and the trajectory predicted by the target actions of other autonomous vehicles includes: Based on the trajectory predicted by the exploratory actions in the i-th time step, the trajectory predicted by the target actions of other autonomous vehicles, and the predicted trajectory of the human-driven vehicle, it is determined whether there is a collision risk between the current autonomous vehicle and other surrounding vehicles in the i-th time step.

5. The method according to claim 4, characterized in that, The artificial driving vehicle behavior prediction model includes a car-following model, a lane-changing decision model, and a vehicle kinematics model. Based on a human-driven vehicle behavior prediction model, the driving trajectory of the human-driven vehicle within the preset time range is predicted to obtain the predicted trajectory of each human-driven vehicle, including: The real-time driving behavior of each of the manually driven vehicles is simulated based on the car-following model. Based on the real-time driving behavior and lane-changing decision model of the manually driven vehicle, the target lane of the manually driven vehicle is obtained; Based on the vehicle kinematics model and the target lane, the driving trajectory of the manually driven vehicle within the preset time range is predicted to obtain the predicted trajectory of the manually driven vehicle.

6. A decision-making device for multi-vehicle reinforcement learning, characterized in that, The device includes: The allocation unit is used to assign a priority index to each autonomous vehicle controlled by a multi-vehicle reinforcement learning model according to a preset priority allocation rule, wherein the probability of an autonomous vehicle causing a collision is positively correlated with the priority index. A traversal unit is used to traverse each of the autonomous vehicles in descending order of priority index; The traversal unit includes: The acquisition module is used to acquire, for the current autonomous vehicle being traversed, the exploratory actions of the current autonomous vehicle output by the multi-vehicle reinforcement learning model in the future i-th time step. The judgment module is used to determine whether there is a collision risk between the current autonomous vehicle and the other autonomous vehicles in the i-th time step based on the trajectory predicted by the exploratory action in the i-th time step and the trajectory predicted by the target action of other autonomous vehicles. When the other autonomous vehicle is an unvisited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle that has been driven the most recently. When the other autonomous vehicle is a visited autonomous vehicle, the target action of the other autonomous vehicle is the actual action of the other autonomous vehicle in the i-th time step. The determination module is used to determine the exploratory action in the i-th time step as the actual action of the current autonomous vehicle in the i-th time step if there is no collision risk. The filtering module is used to select the best action from the set of valid actions based on the safety margin of the current autonomous vehicle at the i-th time step if there is a collision risk, and use it as the actual action of the current autonomous vehicle in the i-th time step. The traversal unit is used to obtain a new i after i+1, and return to the execution step to obtain the exploratory actions of the current autonomous vehicle in the future i-th time step output by the multi-vehicle reinforcement learning model, until the actual actions of the current autonomous vehicle in each time step within the preset time range are determined, and then continue to traverse the next autonomous vehicle. The allocation unit includes: The calculation module is used to calculate the merging priority, lane end priority, and headway priority of each autonomous vehicle controlled based on the multi-vehicle reinforcement learning model. The weighting module is used to perform a weighted summation of the merging priority, lane end priority, and headway priority of the autonomous vehicle to obtain the priority index of the autonomous vehicle. Among them, the merging priority of the autonomous vehicles located in the merging lane is higher than that of the autonomous vehicles located in the main lane; For multiple autonomous vehicles in a merging lane, the closer they are to the end of the merging lane, the higher the priority of the end of the lane. The smaller the headway, the higher the headway priority of the autonomous vehicle.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-5.

Images