A control method for autonomous vehicles based on multi-agent reinforcement learning

Abstract

The present invention relates to a method for controlling an autonomous driving vehicle based on multi-agent reinforcement learning, and belongs to the field of artificial intelligence and autonomous driving technology. For autonomous driving vehicle scenarios, the present method proposes a multi-agent adaptive decision-making framework for AVs. First, the personalized needs of passengers are considered, and different driving styles are formulated for AVs for use in the subsequent decision-making modeling process. By utilizing coalition game and V2V communication, the multi-agent system is divided with each connected autonomous driving vehicle CAV as the center, thereby constructing a cooperative decision-making area. Then, based on the rolling time domain in model predictive control, a deep reinforcement learning decision-making framework is constructed to realize the motion decision-making of CAV in the cooperative area. In the scenario of fully autonomous vehicles, the present invention can not only ensure the safety and efficiency on the main lanes of the highway, but also ensure the safety, comfort and efficiency of the ramp merging area.

Description

Technical Field

[0001] The present invention relates to a control method for an autonomous driving vehicle, and specifically to an autonomous behavior control method for an autonomous driving vehicle based on multi-agent deep reinforcement learning, belonging to the field of artificial intelligence and autonomous driving technology. Background Art

[0002] Autonomous vehicles (AVs) are intelligent systems that integrate advanced communications, computing, networking, and control technologies to achieve real-time, continuous vehicle control. AV systems offer a variety of functions, including automatic vehicle wake-up and sleep, automatic parking, automatic cleaning, automatic driving, automatic parking, automatic door opening and closing, and automatic fault recovery. They also have multiple operating modes, including normal operation, degraded operation, and interrupted operation. AV technology is an inevitable requirement and result of the three major technological development trends of electrification, intelligence, and connectivity. When AVs reach advanced stages, they will greatly liberate human drivers, especially long-distance driving, which relies heavily on energy, physical strength, experience, and intelligence.

[0003] In recent years, autonomous driving has been widely researched and applied in scenarios such as intelligent transportation, smart cities, and smart highways. Highways, in particular, have become a research hotspot in autonomous driving decision-making and control due to their simplicity. Highways are characterized by high complexity and high dynamics, making safe, comfortable, and efficient highway driving for autonomous vehicles a key research priority.

[0004] To achieve fully autonomous driving capabilities, vehicles should develop safe, controllable, and robust driving strategies. This strategy may involve low-level control of specific driving maneuvers (such as acceleration, deceleration, and steering) or higher-level strategic decision-making (such as lane changes, overtaking, and turning). Considering that the ultimate goal of autonomous vehicles (AVs) is to completely replace human-driven cars on highways in the future, how to integrate all AVs into the current traffic ecosystem in which humans drive and minimize the driving efficiency and driving risks of AVs are the most worthy issues to explore in the field of autonomous driving. Because passengers have different driving characteristics (polite, rude, peaceful) and different destinations, they have different demands on AVs. Therefore, how AVs can adaptively provide personalized needs for different passengers is another issue worth exploring.

[0005] Currently, there are four main types of multi-agent deep reinforcement learning methods:

[0006] 1. Analytical methods for emergent behavior, i.e., evaluating single-agent DRL algorithms in multi-agent scenarios;

[0007] 2. Communication learning methods, where agents learn communication protocols to solve cooperative tasks;

[0008] 3. Cooperative learning methods, where agents learn to cooperate using only actions and (partial) observations;

[0009] 4. Agent modeling agent, that is, modeling and inferring the opponent agent to complete the task.

[0010] However, these methods are primarily used in scenarios such as Atari games, social dilemmas, and 3D competitive games. In the field of autonomous driving, most methods are simply applied without in-depth analysis and application based on actual autonomous driving scenarios. Summary of the Invention

[0011] The purpose of this invention is to solve technical problems such as how to integrate autonomous driving vehicles (AVs) into the current transportation system to achieve safer, more comfortable and more efficient driving control, and how to adaptively provide personalized needs for different passengers to achieve more intelligent autonomous driving control. A control method for autonomous driving vehicles based on multi-agent reinforcement learning is creatively proposed.

[0012] The innovation of the present invention is that for a high-speed scenario entirely for autonomous vehicles (i.e., human-driven vehicles are not considered in this scenario), this method proposes a multi-agent adaptive decision-making framework for AVs. First, considering the personalized needs of passengers, different driving styles (polite, rude, peaceful) are formulated for AVs for use in the subsequent decision modeling process. By using coalition games and V2V communication, the multi-agent system is divided with each connected automated vehicle (CAV) as the center to construct a cooperative decision-making area. Then, based on the rolling horizon in model predictive control, a deep reinforcement learning decision-making framework is constructed to realize the motion decision-making of CAVs in the cooperative area.

[0013] The present invention is implemented by adopting the following technical solutions.

[0014] In machine learning, reinforcement learning is used to describe and solve the problem of an intelligent agent maximizing rewards or achieving specific goals by learning strategies during its interaction with the environment. Reinforcement learning is usually modeled as a Markov decision process (MDP), where a tuple (S, A, P, R, γ) with a measurable state S, an action space A, an unknown reward function R, and unknown dynamics P is used to solve the MDP. The goal of reinforcement learning is to maximize the future discounted reward. Among them, r t+k is the reward at time t+k, γ represents the discount rate, r t Represents the reward of feedback at time t.

[0015] In the reinforcement learning framework, the action value function Vπ Expressed as:

[0016]

[0017] Where s0 = s, a t ~π(a t |s t ), s t+1 ~P(s t+1 |s t , a t ), π represents the policy function, P represents the state transition probability distribution, Represents the expectation of solution.

[0018] State value function Q π Expressed as:

[0019]

[0020] Among them, s′ represents the state at the next moment.

[0021] Finally, the optimal action value function Q * =maxQ π (s) satisfies the Bellman optimality equation, which is expressed as:

[0022]

[0023] A method for controlling an autonomous vehicle based on multi-agent reinforcement learning comprises the following steps:

[0024] Step 1: Define the cooperation and alliance model for CAVs. CAVs refer to connected and autonomous vehicles.

[0025] There are two types of vehicle driving areas in the highway ramp area: one is the vehicles driving on the main road, and the other is the vehicles driving at the entrance of the ramp.

[0026] like Figure 1 As shown, the vehicle is located at different locations on different roads. Considering that human drivers are also affected by the limited field of view, the observation area is only within the set area. Figure 1 In the figure, the black dotted box is used as the field of view of the autonomous driving. The size of the area is set to l*n, where l represents the length of the area and n represents the width of the area. The area is set to be larger than the field of view of a human driver. In addition, in certain situations, such as when the distance to the vehicle ahead is close and overtaking is impossible, the human driver will default to following the vehicle. Therefore, for Figure 1 As shown in the black oval box, the distance between the two cars is Less than the threshold Δr0, V i 、V iRepresent the i-th and j-th vehicles respectively. At this time, the two are united, that is, they have the same control strategy.

[0027] Step 2: Construct the CAVs multi-agent subsystem.

[0028] like Figure 1 As shown in the figure, it is a multi-agent system, and the box area after the alliance in step 1 is defined as the constructed multi-agent subsystem.

[0029] Different from other multi-agent deep reinforcement learning methods, the multi-agent subsystem is applied by one of the agents ( Figure 2 a V4 in the oval box) or multiple agents when multiple agents achieve coalition decision-making through step 1 ( Figure 2 c) V7 and V8 in the oval box serve as the research subjects for multi-agent reinforcement learning. The other agents in the square box provide real-time observation information to the research subjects based on V2V wireless communication technology.

[0030] Step 3: Design a CAVs multi-agent deep reinforcement learning decision-making algorithm.

[0031] Human drivers typically make three different decisions: accelerating to overtake, slowing down to avoid, and following. At the same time, decisions also need to be made about the vehicle's behavior, such as turning left, turning right, or going straight. Considering that human drivers typically only need to use observations within their field of view to make the best decision, deep reinforcement learning combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning. For autonomous driving multi-agent scenarios, all agents can directly return acceleration (acceleration or deceleration), steering angles, and behavioral strategies (overtaking, avoiding, or following) based on observations for control. This is an artificial intelligence method that is closer to human thinking.

[0032] The CAV decision-making algorithm architecture of multi-agent deep reinforcement learning is as follows: Figure 3 As shown in the figure, it is an end-to-end decision-making architecture consisting of four main parts: input layer, neural network layer, output layer, and environment interaction layer.

[0033] Among them, the input layer is composed of the perception information of CAVs at the current moment and the strategy adopted at the previous moment and returns The neural network architecture consists of a convolutional neural network (CNN), a long short-term memory (LSTM) neural network, and a fully connected network (FCN). LSTM is a time-recurrent neural network that generally performs better than recursive neural networks and hidden Markov models (HMMs).

[0034] The details of the entire neural network architecture can be found in Figure 3 The output layer is the vehicle V at time t i Final action taken The role of the environment interaction layer is: the agent takes action at time t After that, it enters the state of the next moment by interacting with the environment. After receiving the new state, the agent enters the next decision cycle.

[0035] Step 4: Construct the state space, action space, and reward function of CAVs.

[0036] State space construction Through V2V wireless communication technology, CAVs can exchange observation information such as speed, acceleration, position, and heading.

[0037] Assume that the research object of a multi-agent subsystem obtained in step 2 is agent n, and the state space expression of the agent is Then the observation space of the system is expressed as: o = o n ,...,o k , r n , where o k Represents other agents in the multi-agent subsystem, excluding agents n, r n represents the reward of agent n. In other words, the observed state of agent n contains the state information of all other agents in the multi-agent subsystem, and at the same time provides rich feature information for the input of the neural network in step 3, thereby improving the decision accuracy.

[0038] The action space consists of three parts: the change in acceleration in the x-direction, where a positive value indicates acceleration and a negative value indicates deceleration; the change in heading angle, where a positive value indicates an increase in heading angle and a negative value indicates a decrease; and whether to turn, such as left, right, or go straight. The strategy that agent n ultimately executes, i.e., the optimal strategy, is:

[0039] Safety, comfort, and efficiency are the three most important performance indicators in the field of autonomous driving. Therefore, the reward function is constructed based on these three indicators and is specifically expressed as: in, Represents agent V i safety incentives, Represents agent V i Comfort incentives, Represents agent V i efficiency incentives, and and are their respective weight factors.

[0040] Step 5: CAVs control based on multi-agent reinforcement learning to achieve safe, comfortable and efficient vehicle driving.

[0041] The optimal strategy obtained by the multi-agent deep reinforcement learning decision-making algorithm is used as the control input of the autonomous driving vehicle to achieve safe, comfortable and efficient driving of the vehicle.

[0042] Beneficial effects

[0043] The method of the present invention has the following advantages over the prior art:

[0044] 1. This paper adopts a human-like approach to enable CAVs to collaborate and form alliances, proposing a novel approach to building a multi-agent system. By utilizing V2V communication technology to define the CAV learning area, the quality of the observation space is improved while significantly reducing resource utilization.

[0045] 2. This method proposes a deep reinforcement learning decision-making approach for CAVs within the constructed multi-agent subsystem. At time t, other agents transmit velocity, acceleration, position, and heading information via vehicle-to-vehicle communication. This information is then integrated with the state of the agent itself, along with the reward and action at time t-1 as input. This decision is ultimately made using an LSTM neural network with historical memory.

[0046] In the context of fully autonomous vehicles, this method not only ensures safety and efficiency on highway trunk lanes, but also ensures safety, comfort, and efficiency in ramp merging areas. Therefore, the proposed method is more universally applicable to the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the highway ramp area with cooperative decision-making of CAVs.

[0048] Figure 2 These are the four cooperative decision-making modes proposed by the present invention for CAVs during highway driving.

[0049] Figure 3 This is the architecture diagram of the CAV decision algorithm based on the rolling time domain proposed in the present invention.

[0050] Figure 4 The present invention proposes a new interactive learning method between multiple agents and the environment.

[0051] Figure 5 Schematic diagram of the driving direction X and the lane perpendicular direction Y of CAVs in the scenario set by the present invention.

[0052] Figure 6The figure is a schematic diagram of the monorail vehicle dynamics model and the symbols used in the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example

[0055] like Figure 1 As shown, the method of the present invention is explained by taking the autonomous decision-making and control of multiple self-driving cars in a high-speed scene with forks through cooperation as an example.

[0056] Step 1: Set up the cooperation and alliance methods of CAVs.

[0057] like Figure 2 As shown in Figure 3, the cooperative decision-making of four types of CAVs while driving on the highway.

[0058] in, Figure 2 (a) Figure 2 In (c), V4 and V8 represent the cooperative decision-making system of all CAVs driving on Lane 3 and surrounding vehicles. In addition, Figure 2 The oval box in (c) represents V7 and V8 making alliance decisions, and V7's decision will follow V8's; Figure 2 In (b), V6 represents the cooperative decision-making system of all CAVs traveling on Lane 2 and surrounding vehicles; (d) V8 represents the cooperative decision-making system of all CAVs traveling on Lane 2 and surrounding vehicles. Figure 2 The box in (d) is the same as Figure 2 (c) The principles are the same. The above decision-making is also affected by the driver's characteristics (politeness, rudeness, and peace).

[0059] The process of cooperation and alliance is as follows:

[0060] Let the alliance set be Set a threshold Δr0, in the multi-agent subsystem, judge Is it less than Δr0? If so, then Otherwise V j Do not join the alliance set The final alliance set is

[0061] Step 2: Construction of CAVs multi-agent subsystem.

[0062] Considering the limited vision of human drivers during driving, the multi-agent subsystem area is divided based on V2V communication and vision threshold. Then, after the cooperation and alliance in step 1, the alliance is obtained. Figure 1In the given scenario, each agent can build a multi-agent subsystem with itself as the research object. Therefore, there are as many multi-agent subsystems as there are agents, which makes it easier for each autonomous vehicle to achieve autonomous decision-making and control. Based on step 1, the algorithm flow for establishing a multi-agent subsystem is as follows:

[0063] Assume the subsystem set is: Set a collection Set a threshold Δd, in the multi-agent subsystem, judge is less than Δd, then and V η+1 Otherwise, V η and V η+1 from Remove it, then c i Add to Finally, we get the subsystem

[0064] Step 3: Design of CAVs multi-agent deep reinforcement learning decision algorithm.

[0065] The input information of the multi-agent deep reinforcement learning decision algorithm consists of two parts: the i-th autonomous driving car V i , the input maneuver information at t = 0 is and the jth autonomous vehicle V j The maneuver information is In addition, V j V i The surrounding vehicles in the multi-agent subsystem centered on , the construction of the multi-agent subsystem is shown in step 2, and the process is as follows:

[0066] The multi-agent subsystem is represented as: The system parameters are initialized to Then the state, action, and reward of the agent at time t are Then the value function is optimized by the neural network optimizer. Then, the agent V i From the action Start simulating (not actually executing) m times to get And predict the next action p times through neural network There will be p action decisions Finally choose the first action Control the agent as the optimal strategy.

[0067] Step 4: Construct the state space, action space, and reward function of CAVs.

[0068] like Figure 4As shown, for agent 1, its observation space is o1,...,o i , r1, mainly composed of its own state space and reward function, as well as the state space of other agents in the subsystem. The action space is represented as

[0069] The reward function is mainly divided into three parts.

[0070] First, CAVs must ensure absolute safety during driving. In a multi-agent system, the agent must maintain a safe distance from the vehicle in front during learning and must not collide or drive out of bounds. The specific incentive function is as follows:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] Here, σ is a minimum value used to prevent the denominator from being zero.

[0082] The comfort of CAVs is defined by the derivative of acceleration j, and the specific expression is as follows:

[0083]

[0084] The efficiency function of CAVs is defined by the inverse square of the difference between the initial time and the time when the vehicle safely reaches the end point. The specific expression is as follows:

[0085]

[0086] The larger the difference between the two moments, the lower the driving efficiency of the autonomous vehicle.

[0087] Step 5: Implement CAVs control based on multi-agent reinforcement learning.

[0088] Typically, a 2-DOF bicycle model and a 1-DOF longitudinal model are used to design the controller instead of a four-wheeled vehicle model. This model is sufficient for high-speed and shallow-angle driving conditions. Figure 6 The diagram of the vehicle dynamics model is shown in Figure 2. Through the vehicle dynamics formula, the control quantity u=[a x ,δ f ] T Directly related to the state variable s.

[0089] In the expressway scenario, the steering angle θ of the vehicle's front wheels is usually set to be very small, that is, sinθ≈0.

[0090] like Figure 6 As shown in the figure, the single-track vehicle dynamics model and the symbols used are described. The longitudinal, lateral and yaw angle dynamics equations of the vehicle are expressed as follows:

[0091] ma x =F xf cosδ f +F xr

[0092] ma y =F yf +F yr

[0093]

[0094] in, F represents the inertial acceleration at the center of mass of the vehicle in the y-axis direction; yf and F yr Represents the lateral force of the tires on the front and rear wheels of the vehicle respectively. y Factors include the acceleration of motion along the y-axis Centripetal acceleration Therefore, the equation for the lateral translational motion of the vehicle is:

[0095]

[0096] When the slip angle is small, the lateral force of the tire is proportional to the slip angle. vf and θ vr are the front and rear wheel speed angles respectively, δ is the front wheel steering angle, and the front and rear wheel slip angles are expressed as:

[0097] α f =δ-θ vf , α r =-θ vr

[0098] The lateral forces on the front and rear wheels of the vehicle are expressed as:

[0099] F yf =C af (δ-θ vf ), F yr =C ar (-θ vr )

[0100] Among them, C αf and C αr The cornering stiffness of the front and rear wheels respectively. The velocity angle θ of the front and rear wheels of the vehicle vf ,θ vr Using the small angle approximation:

[0101]

Claims

1. A method for controlling an autonomous vehicle based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Step 1: Set the cooperation and alliance mode of CAVs, where CAVs refer to connected autonomous vehicles; the vehicles are located at different locations on different roads, and the observation area is set within the set area. The size of the set area is set to l*n, where l represents the length of the area and n represents the width of the area. The area is set to be larger than the field of view of a human driver; the distance between the two vehicles is set to Less than the threshold Δr0, V i 、V i Represent the i-th and j-th vehicles respectively. At this time, the two are combined into an alliance and have the same control strategy; Let the alliance set be And set a threshold Δr0, in the multi-agent subsystem, judge Is it less than Δr0? If so, then Otherwise V j Do not join the alliance set The final alliance set is Step 2: Construct CAVs multi-agent subsystem; Divide the multi-agent subsystem area based on V2V communication and field of view threshold; Step 3: Design a CAVs multi-agent deep reinforcement learning decision-making algorithm; The CAV decision-making algorithm of multi-agent deep reinforcement learning is an end-to-end decision-making architecture, including input layer, neural network layer, output layer and environment interaction layer; Among them, the input layer is composed of the perception information of CAVs at the current moment and the strategy adopted at the previous moment and returns The neural network structure includes convolutional neural network (CNN), long short-term memory (LSTM) artificial neural network (ANN), and fully connected network (FCN). The output layer of the neural network architecture is the vehicle V at time t i Final action taken The role of the environment interaction layer is: the agent takes action at time t After that, it enters the state of the next moment by interacting with the environment. After receiving the new state, the agent enters the next decision cycle; Step 4: Construct the state space, action space, and reward function of CAVs; Assume that the research object of a multi-agent subsystem obtained in step 2 is agent n, and the state space expression of the agent is Then the observation space of the system is expressed as: o = o n ,...o k , r n , where o k Represents other agents in the multi-agent subsystem, excluding agents n, r n Represents the reward of agent n. The observed state of agent n contains the state information of all other agents in the multi-agent subsystem, and provides rich feature information for the input of the neural network in step 3. The action space includes: the change in acceleration in the x-direction, where a positive value indicates acceleration and a negative value indicates deceleration; the change in heading angle, where a positive value indicates an increase in heading angle and a negative value indicates a decrease; whether to turn, such as left, right, or go straight; and the strategy that agent n ultimately executes, i.e., the optimal strategy is: The reward function is constructed based on the three indicators of safety, comfort and efficiency, and is expressed as: in, Represents agent V i safety incentives, Represents agent V i Comfort incentives, Represents agent V i efficiency incentives, and and are their respective weight factors; Step 5: Use the optimal strategy obtained by the multi-agent deep reinforcement learning decision-making algorithm as the control input of the autonomous driving vehicle.

2. The autonomous driving vehicle control method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step 2, let the subsystem set be: Set a collection Set a threshold Δd, in the multi-agent subsystem, judge Is it less than Δd? If so, then and V η+1 Otherwise, V η and V η+1 from Remove it, and then Add to Finally, we get the subsystem

Images