Multi-agent unmanned driving decision-making method and system for intersection scene without signal lamp

By introducing importance sampling and dynamic noise mechanisms in multi-agent unmanned driving systems, the limitations of single-agent reinforcement learning in complex multi-vehicle scenarios and insufficient research on multi-agent deep reinforcement learning in signal-free intersections is solved. The problem has improved the decision-making efficiency and traffic success rate of unmanned vehicles, and reduced the collision rate between vehicles.

CN118918720AInactive Publication Date: 2024-11-08BEIJING UNION UNIVERSITY

Patent Information

Application Number
CN202410948308.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing single agent reinforcement learning algorithm has limitations in complex multi-vehicle intersection decision scenarios, and cannot effectively handle the coordination and competition of multi-agent strategies, and there are few researches on deep reinforcement learning in signal-free intersection scenarios. , centralized learning has poor expansion, and independent learning faces environmental non-stationarity problems.

Method used

A multi-agent unmanned driving decision-making method based on the importance sampling module and dynamic noise mechanism was designed. By sampling empirical samples based on the importance weight in the experience playback pool, the dynamic noise mechanism is introduced to make the agent gradually rely on the learned strategy. , build decision models to improve learning efficiency and robustness.

Benefits of technology

The learning efficiency and robustness of multi-agent reinforcement learning algorithms have been improved, the decision-making efficiency and traffic success rate of unmanned vehicles at complex dynamic signal-free intersections have been improved, and the collision rate between vehicles has been reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918720A_ABST
    Figure CN118918720A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-agent unmanned driving decision-making method and system for a non-signal-lamp intersection scene, and the method comprises the steps: setting a multi-agent parameter space based on an unmanned driving scene of a non-signal intersection; a dynamic noise mechanism is designed for the multiple agents, so that the multiple agents accumulate experience in a parameter space; a sampling rule of an experience playback pool is designed for the multiple agents, so that the multiple agents effectively learn accumulated experience; constructing a decision model based on a parameter space, a dynamic noise mechanism and a sampling rule; and generating the multi-agent unmanned driving decision-making method by using the decision-making model. According to the invention, the unmanned vehicle can pass through a complex signal-free intersection. The method not only improves the problem of low learning efficiency of a multi-agent algorithm model caused by a complex algorithm model of a multi-agent reinforcement learning algorithm, but also improves the problems of unstable decision strategy and poor robustness of final training, and reduces the collision rate of unmanned vehicles at a non-signalized intersection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving, and in particular to a multi-agent unmanned driving decision-making method and system for a scene at an intersection without signal lights. Background Art

[0002] Single-agent reinforcement learning algorithms, such as Q-learning, DQN, DDPG, PPO and other algorithms, have been widely used in the field of autonomous driving. Single-agent reinforcement learning algorithms are widely used in autonomous driving intersection decision-making scenarios, but they only train one agent and cannot handle more complex multi-agent strategies such as coordination and competition. Compared with multi-agent reinforcement learning, single-agent reinforcement learning has limitations. For example, single-agent reinforcement learning algorithms usually assume that the driving behavior of other cars is fixed and follows certain rules. However, in the real world, drivers of other vehicles may adjust their decisions based on observed driving behavior, which brings uncertainty and unpredictability.

[0003] For complex decision-making scenarios such as intersections involving multiple vehicles, it is worth considering the use of multi-agent deep reinforcement learning to achieve better decision-making results. Multi-agent reinforcement learning algorithms involve multiple agents (unmanned vehicles) learning simultaneously in a shared environment and constantly adjusting their strategies. This instability destroys the steady state of the environment. Although it hinders the learning process, it is more in line with complex scenarios in the real world. At present, there are relatively few studies on multi-agent deep reinforcement learning unmanned driving decision-making algorithms based on unsignaled intersection scenarios. Based on the improvement of multi-agent deep reinforcement learning algorithms, it has a broad research prospect for practical application in unmanned driving decision-making in intersection scenarios.

[0004] For multi-agent reinforcement learning algorithms, centralized learning has poor scalability in large-scale agent environments, and some agents may learn negative strategies; independent learning faces the problem of environmental non-stationarity; "centralized training and distributed execution" multi-agent reinforcement learning algorithms are more feasible, and the problem becomes how to train independent strategies for each agent from a global perspective. Summary of the invention

[0005] In order to solve the above technical problems, the present invention designs a multi-agent unmanned driving decision-making method based on an importance sampling module and a dynamic noise mechanism. For the experience stored in the experience replay pool, the sampling frequency of the experience sample is determined according to the importance weight, so that learning is more efficient. The dynamic noise mechanism is introduced so that the fixed noise of the agent changes with the increase of training rounds, gradually reducing the introduction of action noise, allowing the agent to rely more on the learned strategies to complete the task. The learning efficiency and robustness of the decision-making model of the multi-intelligence reinforcement learning algorithm are improved, the decision-making efficiency and success rate of unmanned vehicles facing complex and dynamic signal-free intersections are improved, and collisions between vehicles are alleviated.

[0006] To achieve the above object, the present invention provides a multi-agent unmanned driving decision-making method for a scene without a signal light intersection, the steps comprising:

[0007] Based on the unmanned driving scenario at an unsignalized intersection, set the parameter space of the multi-agent;

[0008] Designing a dynamic noise mechanism for the multi-agent so that the multi-agent accumulates experience in the parameter space;

[0009] Designing a sampling rule for the experience replay pool for the multi-agent so that the multi-agent can effectively learn the accumulated experience;

[0010] Constructing a decision model based on the parameter space, the dynamic noise mechanism and the sampling rule;

[0011] The decision model is used to generate a multi-agent unmanned driving decision-making method.

[0012] Preferably, the parameter space includes: a state space and an action space;

[0013] The state space is defined as follows:

[0014] S=(V ego , V 1 , V 2 ...V n , D 1 , D 2 ...D n , Dest ego )

[0015] Where S represents the state space; V ego represents the speed of the ego vehicle; V 1 , V 2 ...V n Indicates the speed of the remaining vehicles; D 1 , D 2 ...D nIndicates the relative distance between the remaining vehicles and the self-vehicle; Dest ego Indicates the distance between the destination end point and the self vehicle;

[0016] The action space is defined as follows:

[0017] A=(Throttle ego , Brake ego , Steer ego )

[0018] Where A represents the action space; Throttle ego Indicates throttle; Brake ego Indicates brake; Steer ego represents the vehicle turning angle; the action space of each intelligent vehicle in the scene contains these parameters.

[0019] Preferably, dynamic noise is introduced into the actions output by the agent strategy network to increase the randomness of the agent's behavior, which decreases accordingly as the number of training rounds increases; the agent is encouraged to explore more at the beginning, and by accumulating experience, the agent gradually makes more use of its learned strategy.

[0020] Preferably, the noise value is calculated based on the initial noise, the final noise and the remaining percentage of training rounds. The specific calculation method includes: based on the final noise value, adding the difference between the initial noise value and the final noise value and multiplying it by the percentage of remaining training rounds.

[0021] Preferably, the sampling rules include: giving higher priority to experiences that cause significant changes in the agent's behavior or are close to the optimal solution; the sampling strategy is to select experience samples to be replayed based on the importance of the experience, and during training, the decision model focuses more on samples that have a greater impact on the learning process, thereby improving the efficiency and speed of the learning process and enabling the decision model to converge to a better strategy more quickly.

[0022] Preferably, the training method of the decision model includes: after initializing the network parameters of each agent, the agent explores the environment based on the dynamic noise mechanism and accumulates experience; after the experience stored in the experience replay pool exceeds a set threshold, the agent samples based on the sampling rules of the experience replay pool to train the decision model.

[0023] Preferably, the reward function of the decision model includes: local reward and global reward;

[0024] The local reward is given based on the vehicle's safe and fast crossing of the intersection from the starting point to the target point, as well as the vehicle speed and time spent as reward evaluation criteria; a certain reward is given based on the distance from the target point, and a penalty is given for conflicts between vehicles; whether the vehicle has reached the target point is judged based on the distance from the target point, and if it arrives successfully, a task completion reward is given;

[0025] If all vehicles safely reach the expected destination, a global reward will be given to promote task collaboration among vehicles.

[0026] The present invention also provides a multi-agent unmanned driving decision-making system for non-signal intersection scenarios, the system is used to implement the above method, including: a space design module, a noise design module, a rule design module, a model building module and a decision generation module;

[0027] The spatial design module is used to set the parameter space of multiple agents based on the unmanned driving scenario at an unsignalized intersection;

[0028] The noise design module is used to design a dynamic noise mechanism for the multi-agent so that the multi-agent can accumulate experience in the parameter space;

[0029] The rule design module is used to design a sampling rule of the experience replay pool for the multi-agent so that the multi-agent can effectively learn the accumulated experience;

[0030] The model building module is used to build a decision model based on the parameter space, the dynamic noise mechanism and the sampling rule;

[0031] The decision generation module is used to generate a multi-agent unmanned driving decision method using the decision model.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention can enable unmanned vehicles to pass through complex unsignaled intersections. The method not only improves the problem of low learning efficiency of the multi-agent algorithm model due to the complexity of the algorithm model of the multi-agent reinforcement learning algorithm, but also improves the problem of insufficient stability and poor robustness of the final training decision strategy, reduces the collision rate of unmanned vehicles at unsignaled intersections, and improves the vehicle traffic rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a simulation of a non-signalized intersection scenario according to an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of sampling rules according to an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the decision model structure of an embodiment of the present invention;

[0039] Figure 5 The figure is an overall flow chart of the decision-making method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Embodiment 1

[0043] like Figure 1 FIG. 1 is a schematic diagram of the method flow of this embodiment, and the steps include:

[0044] S1. Set the parameter space of multi-agents based on the unmanned driving scenario at an unsignalized intersection.

[0045] In this embodiment, the agent can be regarded as the unmanned vehicle itself or the algorithm model representing these vehicles, and the multi-agent refers to multiple agents.

[0046] The method of this embodiment is based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. Each agent in the MADDPG algorithm uses the same Actor-Critic architecture as DDPG. The experience samples in the experience replay pool in the algorithm are composed of {s, a 1 , a 2 , ..., a N , r 1 , ..., r N,s',d}.

[0047] To build an autonomous driving decision model, it is necessary to set the corresponding parameter space according to the autonomous driving scenario. In this embodiment, the parameter space includes: state space and action space. In the training phase, the state space input of each agent includes all other agent participants in the scenario; at the same time, continuous action output is used to solve the continuous behavior space problem in the autonomous driving decision problem. Figure 2 The specific steps include:

[0048] S1.1. Design the state space of intelligent vehicles. Take a typical two-way single-lane unsignaled intersection in a multi-agent scenario as an example. The state space needs to contain the state data of all vehicles acquired by the vehicles in the scene. Specifically, taking any intelligent vehicle in the scene as an example, the state space is defined as follows:

[0049] S=(V ego , V 1 , V 2 ...V n , D 1 , D 2 ...D n , Dest ego )

[0050] Where S represents the state space; V ego represents the speed of the ego vehicle; V 1 , V 2 ...V n Indicates the speed of the remaining vehicles; D 1 , D 2 ...D n Indicates the relative distance between the remaining vehicles and the self-vehicle; Dest ego Indicates the distance between the destination end point and the self vehicle.

[0051] S1.2. Design the action space of the intelligent vehicle. The action space is the set of all possible actions that the intelligent agent can take. The behavior of the intelligent agent is defined by the action space. The correct definition of the action space can facilitate the learning process and enable the intelligent agent to effectively explore and utilize experience to achieve the established goals.

[0052] The decision model proposed in this embodiment is more suitable for solving the continuous behavior space problem in the autonomous driving decision problem. The design of its action space is continuous. Taking any intelligent vehicle ego in the scene as an example, its action space is defined as follows:

[0053] A=(Throttle ego , Brake ego , Steer ego )

[0054] Where A represents the action space; Throttle ego Indicates throttle; Brake ego Indicates brake; Steer ego represents the vehicle turning angle; the action space of each intelligent vehicle in the scene contains these parameters.

[0055] These three are continuous-value control signals, which the intelligent vehicle uses to accelerate, decelerate, and turn, and then successfully pass through the intersection and reach the destination.

[0056] S2. Design a dynamic noise mechanism for multi-agents to enable them to accumulate experience in the parameter space.

[0057] In order to encourage the agent to explore the environment, noise is introduced in this embodiment to increase the randomness of the agent's behavior.

[0058] The noise changes dynamically with the number of training rounds and decreases accordingly.

[0059] The noise value is calculated based on the initial noise, final noise, and the remaining training round percentage, as shown in the formula:

[0060]

[0061] In the formula, Explr remain It indicates the percentage of remaining training rounds; exploration_eps indicates the total number of training rounds; ep_i indicates the current round number.

[0062] The noise introduced in the current round is equal to the final noise value plus the difference between the initial noise value and the final noise value multiplied by the percentage of the remaining training rounds:

[0063] Noise=noise final +(noise init -noise final )

[0064] In the formula, Noise represents the noise introduced in the current round; init Indicates the initial noise value; noise final Represents the final noise value.

[0065] As the number of exploration cycles decreases, the noise value gradually transitions from the initial value to the final value, encouraging the agent to explore more in the beginning and gradually exploit its learned strategy more as it gains experience.

[0066] S3. Design sampling rules for the experience replay pool for multi-agents so that they can effectively learn from the accumulated experience.

[0067] In the scenario of multi-agent reinforcement learning, the instability of dynamic changes in the environment leads to slow convergence of the decision model during training, making it difficult to learn the optimal strategy. In order to improve the learning efficiency and convergence speed of the model, this embodiment introduces experience importance sampling. By designing sampling rules for the experience replay pool for multi-agents, experiences that cause significant changes in the behavior of the agent or are close to the optimal solution are given higher priority. The sampling rules are as follows: Figure 3 shown.

[0068] The sampling strategy is to select experience samples to be replayed based on the importance of the experience. The priority of the experience is measured by the TD error or the reward prediction error. The larger the error, the less accurate the model's prediction of the experience, the higher the priority of the experience, and the more it needs to be learned.

[0069] Prediction error δ t It can be expressed by the following formula:

[0070] δ t =|r t +γ·V t+1 -Q(s t , a t )|

[0071] In the formula, r t represents the current reward; γ represents the discount factor; V t+1 represents the value in the next state calculated using the target network; Q(s t , a t ) indicates that action a is executed in the current state t The value of a t =π(s t ) represents the action selection in the current state; π represents the strategy that the current agent has learned; s t Indicates the current status.

[0072] Specifically, the value V in the next state calculated by the target network t+1 It is expressed as:

[0073] V t+1 =Q′(st +1, π′(s t+1 ))

[0074] Where Q′ is the target Q network; π′ is the target policy network; s t+1 Indicates the next state;

[0075] Priority weight of experience sample iThe definition is as follows:

[0076] piror i =(δ t +ε) α

[0077] In the formula, ε is a small constant used to ensure that the priority is not zero; α is a hyperparameter that controls the priority weight to prevent high-priority experience from being over-sampled and causing training deviation.

[0078] The probability of each experience sample being selected is calculated according to its priority weight, which is defined as follows:

[0079]

[0080] In the formula, p i Represents the probability of each experience sample being selected.

[0081] S3. Construct a decision model based on parameter space, dynamic noise mechanism and sampling rules.

[0082] Decision model such as Figure 4 As shown, when the decision model samples experience samples during training, it focuses more on samples that have a greater impact on the learning process, utilizes samples in the experience buffer more effectively, improves the efficiency and speed of the learning process, and helps the reinforcement learning algorithm converge to a better strategy faster.

[0083] In this embodiment, the reward function of the decision model includes local rewards and global rewards. Specifically, the local rewards are given based on whether the vehicle reaches the target point safely and quickly through the intersection, as well as the vehicle speed and time spent as reward evaluation criteria; a certain reward is given for the distance from the target point, and a penalty is given for conflicts between vehicles; whether the vehicle reaches the target point is judged based on the distance from the target point, and if it arrives successfully, a task completion reward is given; if all vehicles arrive safely at the expected target point, a global reward is given to promote task collaboration between vehicles.

[0084] The training method of the decision model includes: after initializing the network parameters of each agent, the agent explores the environment and accumulates experience based on the dynamic noise mechanism that makes the agent gradually use more of the strategy it has learned; after the experience stored in the experience replay pool exceeds the set threshold, the agent samples based on the sampling rules of the experience replay pool that makes the agent gradually use more of the strategy it has learned to train the decision model. The specific steps include:

[0085] Initialize the Actor network of all intelligent vehicles and parameter ω i ;

[0086] Initialize the Critic network of all intelligent vehicles and parameter θ i ;

[0087] Initialize the target Actor network for all intelligent vehicles and its parameter ω′ i ;

[0088] Initialize the target critic network of all intelligent vehicles and its parameter θ′ i ;

[0089] Initialize the experience playback buffer D;

[0090] Training a multi-agent autonomous driving decision model based on dynamic noise mechanism and importance sampling:

[0091] According to the current state of the environment, each intelligent vehicle selects an action to explore the environment through the strategy network to obtain an initial observation value;

[0092] At each time step, the following steps are performed:

[0093] Each intelligent vehicle obtains an action a based on the current observation value and its Actor network.

[0094] Execute the actions of all intelligent vehicles, obtain the next observation value s′ and reward r, and determine whether the passage task of all vehicles is completed d.

[0095] The observation value s, action a(a 1 ...a N ), reward r(r 1 ...r N ), the next observation value s′ and whether it is ended d are stored in the experience replay pool (experience replay buffer) D.

[0096] D←(s,a 1 , a 2 , ..., a N , r 1 , ..., r N , s′, d)

[0097] Update the importance weights of the experience samples in the experience replay pool and update the sampling probability p of each experience sample i If the experience replay pool has enough experience, each intelligent vehicle performs the following steps:

[0098] Extract a batch of experiences (s, a) from the experience replay buffer 1 , a 2 , ..., a N , r1 , ..., r N , s′, d).

[0099] Based on the extracted experience and the current Critic network, the target Q value and actual Q value of each agent are calculated.

[0100] The mean square error calculates the Critic's loss.

[0101] Gradient descent updates the Critic network

[0102] Calculate the Actor's loss based on the current Actor network and Critic network.

[0103] Updating the Actor Network by Gradient Descent

[0104] Use a soft update strategy to update the target Actor network for each intelligent vehicle. Parameter ω′ i :

[0105] ω′ i =τ·ω i +(1-τ)·ω′ i

[0106] Where τ is a proportional coefficient between 0 and 1, which indicates the influence of the current network parameters on the update of the target network parameters.

[0107] Use the soft update strategy to update the target critic network of each intelligent vehicle Parameter θ′ i :

[0108] θ′ i =τ·θ i +(1-τ)·θ′ i

[0109] The performance of the decision model is evaluated based on the convergence speed of the training model and the collision probability, passage success rate, and reward value of the intelligent vehicle in the test scenario.

[0110] Specifically, the collision rate is defined as the proportion of vehicle collisions per 100 training runs:

[0111]

[0112] Where CollisionCounts represents the total number of collisions between vehicles during the training process; TotalNumber is equal to 100.

[0113] The success rate is defined as the proportion of the number of times that all intelligent vehicles successfully pass through the intersection without collision and successfully reach the target point in every 100 training times:

[0114]

[0115] Where SuccessCounts represents the total number of times that there is no collision between vehicles during the training process and all vehicles successfully pass through the intersection and arrive at the destination; TotalNumber is equal to 100.

[0116] Use decision-making models to generate multi-agent unmanned driving decision-making methods.

[0117] The process of using decision-making models to generate multi-agent unmanned driving decision-making methods is as follows: Figure 5 shown.

[0118] Embodiment 2

[0119] This embodiment also provides a multi-agent unmanned driving decision system for non-signal intersection scenarios, including: a space design module, a noise design module, a rule design module, a model construction module and a decision generation module; the space design module is used to set the parameter space of the multi-agent based on the unmanned driving scenario of the non-signal intersection; the noise design module is used to design a dynamic noise mechanism for the multi-agent so that the multi-agent accumulates experience in the parameter space; the rule design module is used to design sampling rules for the experience replay pool for the multi-agent so that the multi-agent effectively learns the accumulated experience; the model construction module is used to construct a decision model based on the parameter space, the dynamic noise mechanism and the sampling rules; the decision generation module is used to generate a multi-agent unmanned driving decision method using the decision model.

[0120] The following will describe in detail how the present invention solves the technical problems in practical work in conjunction with this embodiment.

[0121] First, the spatial design module is used for the unmanned driving scenario based on the unsignalized intersection to set the parameter space of the multi-agent. In this embodiment, the parameter space includes: state space and action space;

[0122] The state space is defined as follows:

[0123] S=(V ego , V 1 , V 2 ...V n , D 1 , D 2 ...D n , Dest ego )

[0124] Where S represents the state space; Vego represents the speed of the ego vehicle; V 1 , V 2 ...V n Indicates the speed of the remaining vehicles; D 1 , D 2 ...D n Indicates the relative distance between the remaining vehicles and the self-vehicle; Dest ego Indicates the distance between the destination end point and the self vehicle;

[0125] The action space is defined as follows:

[0126] A=(Throttle ego ,Broke ego , Steer ego )

[0127] Where A represents the action space; Throttle ego Indicates throttle; Brake ego Indicates brake; Steer ego represents the vehicle turning angle; the action space of each intelligent vehicle in the scene contains these parameters.

[0128] The noise design module introduces dynamic noise into the actions output by the agent’s policy network, increasing the randomness of the agent’s behavior, which decreases accordingly as the number of training rounds increases; it encourages the agent to explore more at the beginning, and gradually makes more use of its learned strategy through accumulated experience;

[0129] The noise value is calculated based on the initial noise, the final noise and the percentage of the remaining training rounds. The specific calculation method includes: based on the final noise value, adding the difference between the initial noise value and the final noise value and multiplying it by the percentage of the remaining training rounds.

[0130] The rule design module is used to design sampling rules for the experience replay pool for multi-agents, so that multi-agents can effectively learn the accumulated experience. The sampling rules include: giving higher priority to experiences that cause significant changes in the behavior of the agent or are close to the optimal solution; the sampling strategy is to select the experience samples to be replayed based on the importance of the experience. During the training period, the decision model focuses more on samples that have a greater impact on the learning process, improves the efficiency and speed of the learning process, and enables the decision model to converge to a better strategy faster.

[0131] The training process of the decision model includes: after initializing the network parameters of each agent, the agent explores the environment based on the dynamic noise mechanism and accumulates experience; after the experience stored in the experience replay pool exceeds the set threshold, the agent samples based on the sampling rules of the experience replay pool to train the decision model.

[0132] The reward function of the decision model includes: local reward and global reward;

[0133] Local rewards are given based on whether the vehicle can travel from the starting point to the target point and pass through the intersection safely and quickly, as well as the vehicle speed and time spent as reward evaluation criteria; a certain reward is given based on the distance from the target point, and penalties are imposed on conflicts between vehicles; whether the vehicle has reached the target point is judged based on the distance from the target point, and a task completion reward is given if it arrives successfully; if all vehicles arrive safely at the expected target point, a global reward is given to promote task collaboration between vehicles.

[0134] Finally, the decision generation module uses the decision model to generate a multi-agent unmanned driving decision method.

[0135] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-agent unmanned driving decision-making method for a non-signal intersection scenario, characterized in that the steps include: Based on the unmanned driving scenario at an unsignalized intersection, set the parameter space of the multi-agent; Designing a dynamic noise mechanism for the multi-agent so that the multi-agent accumulates experience in the parameter space; Designing a sampling rule for the experience replay pool for the multi-agent so that the multi-agent can effectively learn the accumulated experience; Constructing a decision model based on the parameter space, the dynamic noise mechanism and the sampling rule; The decision model is used to generate a multi-agent unmanned driving decision-making method.

2. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 1 is characterized in that: The parameter space includes: a state space and an action space; The state space is defined as follows: S=(V ego ,V1,V2...V n ,D1,D2...D n ,Dest ego ) Where S represents the state space; V ego represents the speed of the ego vehicle; V1, V2...V n Indicates the speed of the remaining vehicles; D1, D2...D n Indicates the relative distance between the remaining vehicles and the self-vehicle; Dest ego Indicates the distance between the destination end point and the self vehicle; The action space is defined as follows: A=(Throttle ego ,Broke ego ,Steer ego ) Where A represents the action space; Throttle ego Indicates throttle; Brake ego Indicates brake; Steer ego represents the vehicle turning angle; the action space of each intelligent vehicle in the scene contains these parameters.

3. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 1 is characterized in that: Dynamic noise is introduced into the actions output by the agent's policy network to increase the randomness of the agent's behavior, which decreases accordingly as the number of training rounds increases; the agent is encouraged to explore more at the beginning, and by accumulating experience, the agent gradually makes more use of its learned strategy.

4. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 3 is characterized in that: The noise value is calculated based on the initial noise, the final noise and the percentage of the remaining training rounds. The specific calculation method includes: based on the final noise value, adding the difference between the initial noise value and the final noise value and multiplying it by the percentage of the remaining training rounds.

5. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 1 is characterized in that: The sampling rules include: giving higher priority to experiences that cause significant changes in the behavior of the intelligent agent or are close to the optimal solution; the sampling strategy is to select experience samples to be replayed based on the importance of the experience. During training, the decision model focuses more on samples that have a greater impact on the learning process, thereby improving the efficiency and speed of the learning process and enabling the decision model to converge to a better strategy more quickly.

6. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 1 is characterized in that: The training method of the decision model includes: after initializing the network parameters of each agent, the agent explores the environment based on the dynamic noise mechanism and accumulates experience; after the experience stored in the experience replay pool exceeds the set threshold, the agent samples based on the sampling rules of the experience replay pool to train the decision model.

7. The multi-agent unmanned driving decision-making method for non-signal intersection scenarios according to claim 1 is characterized in that: The reward function of the decision model includes: local reward and global reward; The local reward is given based on the vehicle's safe and fast crossing of the intersection from the starting point to the target point, as well as the vehicle speed and time spent as reward evaluation criteria; a certain reward is given based on the distance from the target point, and a penalty is given for conflicts between vehicles; whether the vehicle has reached the target point is judged based on the distance from the target point, and if it arrives successfully, a task completion reward is given; If all vehicles safely reach the expected destination, a global reward will be given to promote task collaboration among vehicles.

8. A multi-agent unmanned driving decision-making system for non-signal intersection scenarios, the system being used to implement the method described in any one of claims 1 to 7, characterized in that: include: Space design module, noise design module, rule design module, model building module and decision making module; The spatial design module is used to set the parameter space of multiple agents based on the unmanned driving scenario at an unsignalized intersection; The noise design module is used to design a dynamic noise mechanism for the multi-agent so that the multi-agent can accumulate experience in the parameter space; The rule design module is used to design a sampling rule of the experience replay pool for the multi-agent so that the multi-agent can effectively learn the accumulated experience; The model building module is used to build a decision model based on the parameter space, the dynamic noise mechanism and the sampling rule; The decision generation module is used to generate a multi-agent unmanned driving decision method using the decision model.

Citation Information

Patent Citations

  • Traffic signal control method and device and readable storage medium

    CN113257016A

  • Multi-intersection traffic signal lamp control method and device, electronic equipment and storage medium

    CN115294784A

  • Intelligent agent reinforcement learning method and device for controlling vehicle driving

    CN116176606A

  • Intersection decision-making method based on multi-agent deep reinforcement learning

    CN117496707A

Cited By

  • Intelligent vehicle decision-making method for no-signal crossroad

    CN119889073A

  • An intelligent vehicle decision-making method for signal-free intersections

    CN119889073B

  • Lane-constraint-free intersection automatic driving decision-making method based on near-end strategy optimization

    CN121305918A