An unmanned driving decision-making method based on multi-agent, single environment and multiple environments

Abstract

The application discloses an unmanned driving decision-making method based on multi-agent, single environment and multiple environments, relates to the field of artificial intelligence control, and constructs a multi-agent decision-making model, performs single-environment training first, and then performs multiple-environment training after convergence; in the single-environment training process, vehicle elements in the environment are converted into trainable agents, according to the Nash equilibrium thought of virtual self-game, an opponent model is added to the traditional actor-critic model, the single-agent model is changed into a multi-agent model, single-agent decision-making training is changed into multi-agent decision-making training, the diversity of training data in the single environment is increased, the training efficiency is improved, meanwhile, the mutual restriction between agents avoids the decision-making method from falling into overfitting; in the multiple-environment training process, a multiple-environment training distribution mechanism based on Dirichlet distribution is established, the mutual influence between environments is suppressed, and the generalization ability of the decision-making method is improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to an autonomous driving decision-making method based on multiple intelligent agents, single environment, and multiple environment. Background Technology

[0002] Autonomous driving technology is a key technology for solving the safety and efficiency problems of point-to-point vehicle traffic, and the decision-making system in autonomous driving technology is crucial to its successful application. In dealing with complex autonomous driving environments, the generalization performance of the decision-making method determines the adaptability of the decision-making system, making it extremely important. When the front-end decision-making method mainly uses reinforcement learning methods based on deep learning, such as... Figure 1 As shown. Reinforcement learning methods rely on interaction with the environment to acquire data and for training. Most methods use single-environment or multi-environment training, which presents two main problems. In single-environment training, data acquisition efficiency is low. When the amount and variety of data are insufficient, the trained decision system often performs poorly. To improve performance, longer training times are often required, leading to overfitting, decreased adaptability to different environments, and poor generalization performance. To improve generalization performance, multiple different environments are typically used for training.

[0003] However, due to the inherent "catastrophic forgetting" defect of neural networks, the generalization performance of decision-making methods often fails to improve normally. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by proposing an autonomous driving decision-making method based on multiple agents, single environment, and multiple environment. This method adds an adversary model to the traditional actor-critic model, transforming the single-agent model into a multi-agent model, and converting single-agent decision-making training into multi-agent decision-making training. This increases the diversity of training data in a single environment, improves training efficiency, and the mutual constraints between agents prevent the decision-making method from falling into overfitting. In the multi-environment training process, a multi-environment training allocation mechanism based on Dirichlet distribution is established to curb the mutual influence between environments while improving the generalization ability of the decision-making method.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] An autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment approaches is proposed. This method constructs a multi-agent reinforcement learning decision-making model, which includes single-environment training and multi-environment training, and utilizes an improved single-environment training method for multi-environment training. The specific steps include:

[0007] Step 1: Construct a multi-agent reinforcement learning decision-making model;

[0008] Step 2, training in a single environment: The vehicle elements in the environment are transformed into trainable agents. Based on the Nash equilibrium idea of ​​virtual self-game, an opponent model is added to the traditional actor-critic model, transforming the single agent model into a multi-agent model. Single agent decision training is transformed into multi-agent decision training, increasing the diversity of training data in a single environment to improve training efficiency. At the same time, the mutual constraints between agents also prevent the decision method from falling into overfitting.

[0009] Step 3: In the multi-environment training process, a multi-environment training allocation mechanism based on Dirichlet distribution is established to improve the generalization ability of the decision-making method while curbing the mutual influence between environments.

[0010] As a further preferred embodiment of the autonomous driving decision-making method based on multiple agents, single environment, and multiple environment of the present invention, in step 1, a reinforcement learning decision-making model based on multi-layer perceptron (MLP) is constructed, using an opponent-actor-critic model.

[0011] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment of the present invention, step 1 specifically includes the following steps;

[0012] Step 1.1: Assume that each environment has at most two intelligent vehicle agents;

[0013] Step 1.2: The opponent model, actor model, and critic model all contain 3 layers. Each layer of the opponent model, actor model, and critic model is composed of a neural network with nodes of 128-64-3, 128-64-3, and 128-64-1, respectively.

[0014] Step 1.3: The opponent model takes the current environment state as input and outputs the predicted longitudinal position, lateral position, and speed of the opponent vehicle at the next moment; the actor model takes the current environment state and the prediction results of the opponent model as input and outputs the predicted longitudinal position, lateral target position, and target speed of the actor vehicle at the next moment; the critic model takes the current environment state, the decision results of the actor model, and the immediate reward from the environment as input and outputs the discounted cumulative action value function.

[0015] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment principles of the present invention, step 2 specifically includes the following steps:

[0016] Step 2.1: Based on the idea of ​​virtual self-game, all vehicles in the environment make independent decisions by the decision-making model. The decision-making process is separated, and all vehicles use the same model to interact, play games, and train against each other in the same environment.

[0017] Step 2.2: The adversary model in the decision model predicts the decision strategies of other vehicles based on the vehicle's current partial observation state, and makes the optimal decision response based on the partial observation state of each vehicle and the predicted decision strategies of other vehicles.

[0018] Step 2.3: The environment provides immediate rewards and feedback on the next state based on the decision results;

[0019] Step 2.4, repeat steps 2.2 and 2.3 until the end of this round.

[0020] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment principles of the present invention, step 3 specifically includes the following steps.

[0021] Step 3.1: Construct N autonomous driving decision-making environments that need to be trained and their reward functions R;

[0022] Step 3.2: Sort the autonomous driving decision-making environments according to the principle of difficulty.

[0023] Step 3.3, set the parameters α = {α1, α2, ..., α3} for the j-th group (j∈[0,1,…N-1]). N}, where α i = (Nj)×10; Construct an N-dimensional Dirichlet environment distribution based on parameter α, and sample from the distribution to obtain environment e;

[0024] Step 3.4: The decision model undergoes a single-environment training in environment e.

[0025] Step 3.5, repeat step 3.4, until the decision model converges to the environmental distribution under this parameter;

[0026] Step 3.6: Update parameter j = j + 1, and repeat steps 3.4 to 3.6 until j = N - 1.

[0027] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment of the present invention, in step 3.1, six autonomous driving decision-making environments that need to be trained are constructed.

[0028] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment of the present invention, the reward function R is: task completion +1, timeout -0.5, collision -1.

[0029] As a further preferred embodiment of the autonomous driving decision-making method based on multiple intelligent agents, single environment, and multiple environment of the present invention, in step 3.2, the order from difficult to easy is: according to the number of intelligent agents vehicles from few to many, according to straight road to curve to intersection, and according to the distance of environmental tasks from short to long, and the average continuous numbering is performed in the interval (0-1).

[0030] As a further preferred embodiment of the autonomous driving decision-making method based on multiple intelligent agents, single environment, and multiple environment in this invention, let N=10, the interval 0-0.1 represents the simplest environment, and the interval 0.9-1 represents the most complex environment.

[0031] As a further preferred embodiment of the autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment of the present invention, step 3.4 involves the decision model undergoing a single-environment training in environment e.

[0032] Step 3.41: The decision model is input into the environmental state observed by the blue and orange intelligent vehicles respectively, and outputs the longitudinal target position, lateral target position and target speed of the two vehicles at the next moment.

[0033] Step 3.42: The environment provides immediate rewards and feedback on the next state based on the decision results;

[0034] Step 3.43: Collect the current environmental status, decision output, and immediate reward R for each of the two intelligent vehicles;

[0035] Step 3.44, repeat steps 3.42 and 3.43 until the end of this round.

[0036] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0037] 1. This invention uses the same decision-making model to control different vehicle agents to interact, play games, and compete in the same environment, which greatly enriches the data for training reinforcement learning models and improves training efficiency. At the same time, the mutual constraints between agents also prevent the decision-making method from falling into overfitting and improve generalization performance.

[0038] 2. In the beginning, the probability of training in simple environments is high, which ensures that the decision model can be trained smoothly. At the same time, a few complex environments are added to avoid the decision model overfitting to simple environments. Then, the probability of training in complex environments is gradually increased, but training in simple environments is not completely abandoned. This reduces the catastrophic forgetting problem of neural networks and further enhances the generalization ability of the decision model.

[0039] 3. The decision model of this invention does not make significant changes to the traditional reinforcement learning decision model. Based on the idea of ​​virtual self-game, it only adds an opponent model to fit the average strategy of the external vehicle and predict the strategy of the external vehicle. The actor model makes the best decision response accordingly. At the same time, the Nash equilibrium solution of the decision model has convergence guarantee. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is the current decision-making framework diagram of the present invention;

[0042] Figure 2 This is a flowchart of an autonomous driving decision-making method based on multiple intelligent agents, single environment, and multiple environment according to the present invention;

[0043] Figure 3 This is a decision model diagram of an autonomous driving decision-making method based on multiple intelligent agents, single environment, and multiple environment according to the present invention;

[0044] Figure 4 These are the six classic environments of this invention, in which blue and orange represent two intelligent vehicles. Detailed Implementation

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0047] The purpose of this invention is to address the shortcomings of existing solutions by proposing a multi-agent, single-environment, and multi-environment-based decision-making method for autonomous driving, applicable to the field of autonomous driving decision-making and planning. The main technical solution of this invention is to construct a multi-agent reinforcement learning decision-making model, which includes single-environment training and multi-environment training, and relies on an improved single-environment training method for multi-environment training. The overall process is as follows:

[0048] 1. Construct a multi-agent reinforcement learning decision-making model: adopt the Opponent-Actor-Critic model;

[0049] 2. Training in a single environment, such as Figure 2 As shown in the left figure:

[0050] (1) Unlike traditional single-agent training that only controls the master vehicle, based on the idea of ​​virtual self-game, all vehicles in the environment of this invention are independently decided by the decision model, with the decision-making process separated. All vehicles use the same model to interact, play games, and engage in adversarial training in the same environment. This allows the decision model to simultaneously collect data on different situations of vehicles in the same environment, enhancing data diversity and improving data efficiency and training efficiency.

[0051] (2) The adversary model in the decision-making model predicts the decision-making strategies of other vehicles based on the current partial observation state of the vehicle, and makes the optimal decision response based on the partial observation state of each vehicle and the predicted decision-making strategies of other vehicles.

[0052] (3) The environment provides immediate rewards and feedback for the next state based on the decision results;

[0053] (4) Repeat steps (2) and (3) until the end of this round. Begin backpropagation reinforcement learning training using the collected data;

[0054] 3. Training in multiple environments, such as Figure 2 As shown in the image on the right:

[0055] (1) Construct N autonomous driving decision environments that need to be trained and their reward functions R;

[0056] (2) The environments are sorted according to the following difficulty classification principles, from difficult to easy: according to the number of intelligent vehicles from few to many, according to straight road to curve to intersection, and according to the distance of the environmental task from short to long, and are numbered continuously on average in the (0-1) interval. For example, assuming N=10, the numbering interval 0-0.1 represents the simplest environment, and the numbering interval 0.9-1 represents the most complex environment.

[0057] (3) Set the j-th group (j∈[0,1,…N-1]) parameter α={α1,α2,…,α N}, where α i = (Nj)×10. Construct an N-dimensional Dirichlet environment distribution based on the parameter α, and sample from the distribution to obtain environment e. With this setting, the maximum value of the probability density function of this distribution is initially biased towards simpler environments, and then gradually biased towards more difficult environments;

[0058] (4) The decision model undergoes a single-environment training in environment e;

[0059] (5) Repeat step (4) until the decision model converges to the environmental distribution under the parameter; update parameter j = j + 1, repeat steps (4)-(6) until j = N-1.

[0060] Its implementation process, such as Figure 2 As shown, specifically:

[0061] Construct a reinforcement learning decision model based on a multilayer perceptron (MLP), such as... Figure 3 As shown, it is assumed that each environment has at most two intelligent vehicle agents. The adversary model, actor model, and critic model are all composed of three-layer neural networks, with each layer having 128-64-3, 128-64-3, and 128-64-1 nodes, respectively. The adversary model takes the current environment state as input and outputs the predicted longitudinal position, lateral position, and predicted speed of the adversary vehicle at the next moment. The actor model takes the current environment state and the prediction result of the adversary model as input and outputs the predicted longitudinal position, lateral target position, and target speed of its own vehicle at the next moment. The critic model takes the current environment state, the decision result of the actor model, and the immediate reward from the environment as input and outputs a discounted cumulative action value function.

[0062] Build as Figure 4 The six environments shown are sorted and numbered according to the number of intelligent vehicles from fewest to most, from straight road to curve to intersection, and from shortest to longest distance for the environmental task. The reward function is: +1 for task completion, -0.5 for timeout, and -1 for collision.

[0063] Let the j-th group (j∈[0,1,…5]) parameter α={α1,α2,…,α6}, where α i = (6-j)×10. Construct an N-dimensional Dirichlet environment distribution based on parameter α, and sample from the distribution to obtain environment e; by setting it up in this way, the maximum value of the probability density function of the distribution is initially biased towards simpler environments, and will gradually bias towards more difficult environments later;

[0064] The decision model undergoes a single-environment training in environment e, specifically as follows:

[0065] 1) The decision model takes into account the environmental conditions observed by the blue and orange intelligent vehicles respectively, and outputs the longitudinal target position, lateral target position and target speed of the two vehicles at the next moment.

[0066] 2) The environment provides immediate rewards and feedback on the next state based on the decision-making results;

[0067] 3) Collect the current environmental status, decision output, and immediate reward R of the two intelligent vehicles respectively;

[0068] 4) Repeat steps 2) and 3) until the end of this round. Begin training the reinforcement learning neural network using backpropagation based on the collected data;

[0069] Repeat step 4 until the decision model converges to the environmental distribution under these parameters;

[0070] Update parameter j = j + 1, repeat steps 3-6 until j = N-1, and complete the training.

[0071] This invention utilizes a multi-agent decision-making model to fully leverage and extend each training process in a single environment, enriching the data diversity of each training session, improving data efficiency, and enhancing the generalization ability of the decision-making model.

[0072] This invention utilizes a multi-environment distribution based on the Dirichlet distribution to comprehensively utilize different environments. In each multi-environment training iteration, the decision model collects data from multiple environments, and this data can be adjusted according to the difficulty of the environment and the current training status, mitigating the catastrophic forgetting problem of neural networks and further enhancing the generalization ability of the decision model.

[0073] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0074] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A decision-making method for autonomous driving based on multi-agent systems, single-environment systems, and multi-environment systems, characterized in that: A multi-agent reinforcement learning decision-making model is constructed, and training includes both single-environment and multi-environment training. An improved single-environment training method is used for multi-environment training. The specific steps include: Step 1: Construct a multi-agent reinforcement learning decision-making model; Step 2, training in a single environment: The vehicle elements in the environment are transformed into trainable agents. Based on the Nash equilibrium idea of ​​virtual self-game, an opponent model is added to the traditional actor-critic model, transforming the single agent model into a multi-agent model. Single agent decision training is transformed into multi-agent decision training, increasing the diversity of training data in a single environment to improve training efficiency. At the same time, the mutual constraints between agents also prevent the decision method from falling into overfitting. Step 3: In the multi-environment training process, a multi-environment training allocation mechanism based on Dirichlet distribution is established to improve the generalization ability of the decision-making method while curbing the mutual influence between environments. In step 1, a reinforcement learning decision model based on multilayer perceptron (MLP) is constructed, using an opponent-actor-critic model; Step 1 specifically includes the following steps; Step 1.1: Assume that each environment has at most two intelligent vehicle agents; Step 1.2: The opponent model, actor model, and critic model all contain 3 layers. Each layer of the opponent model, actor model, and critic model is composed of a neural network with nodes of 128-64-3, 128-64-3, and 128-64-1, respectively. Step 1.3: The opponent model takes the current environment state as input and outputs the predicted longitudinal position, lateral position, and speed of the opponent vehicle at the next moment; the actor model takes the current environment state and the prediction results of the opponent model as input and outputs the predicted longitudinal position, lateral target position, and target speed of the actor vehicle at the next moment; the critic model takes the current environment state, the decision results of the actor model, and the immediate reward from the environment as input and outputs the discounted cumulative action value function.

2. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 1, characterized in that: Step 2 specifically includes the following steps. Step 2.1: Based on the idea of ​​virtual self-game, all vehicles in the environment make independent decisions by the decision-making model. The decision-making process is separated, and all vehicles use the same model to interact, play games, and train against each other in the same environment. Step 2.2: The adversary model in the decision model predicts the decision strategies of other vehicles based on the vehicle's current partial observation state, and makes the optimal decision response based on the partial observation state of each vehicle and the predicted decision strategies of other vehicles. Step 2.3: The environment provides immediate rewards and feedback on the next state based on the decision results; Step 2.4, repeat steps 2.2 and 2.3 until the end of this round.

3. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 1, characterized in that: Step 3 specifically includes the following steps. Step 3.1: Construct N autonomous driving decision-making environments that need to be trained and their reward functions R; Step 3.2: Sort the autonomous driving decision-making environments according to the principle of difficulty. Step 3.3, set the j-th group parameter ,in, According to parameters Construct an N-dimensional Dirichlet environment distribution, and sample from the distribution to obtain environment e; Step 3.4: The decision model undergoes a single-environment training in environment e. Step 3.5, repeat step 3.4, until the decision model converges to the environmental distribution under this parameter; Step 3.6, Update parameters Repeat steps 3.4 to 3.6 until... .

4. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 3, characterized in that: In step 3.1, six autonomous driving decision-making environments that need to be trained are constructed.

5. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 4, characterized in that: The reward function R is: +1 for completing the task, -0.5 for timeout, and -1 for collision.

6. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 5, characterized in that: In step 3.2, the order from difficult to easy is as follows: from few to many intelligent vehicles, from straight roads to curves to intersections, and from short to long distances of environmental tasks, and then the average continuous numbering is performed within the (0-1) interval.

7. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 6, characterized in that: Let N=10, with the interval 0-0.1 representing the simplest environment and the interval 0.9-1 representing the most complex environment.

8. The autonomous driving decision-making method based on multi-agent, single-environment, and multi-environment scenarios according to claim 7, characterized in that: Step 3.4: The decision model undergoes a single-environment training in environment e. Step 3.41: The decision model is input into the respective environmental states observed by the blue and orange intelligent vehicles, and outputs the environmental states of the two vehicles. The longitudinal target position, lateral target position, and target velocity of the vehicle at the next moment; Step 3.42: The environment provides immediate rewards and feedback on the next state based on the decision results; Step 3.43: Collect the current environmental status, decision output, and immediate reward R for each of the two intelligent vehicles; Step 3.44, repeat steps 3.42 and 3.43 until the end of this round.

Images