Collaborative lane changing control method for unmanned vehicle

Abstract

The invention relates to the technical field of intelligent driving, and discloses an unmanned vehicle cooperative lane changing control method, which comprises the following steps: constructing a Dueling Double DQN network; the method comprises the following steps: constructing a single vehicle transverse controller based on a Dueling Double DQN network, and making a lane change decision for a current vehicle; based on the intelligent driver model, a single vehicle longitudinal controller is constructed, and the longitudinal speed of the current vehicle is adjusted; and constructing a multi-agent reinforcement learning framework for making a multi-vehicle lane changing decision. According to the invention, in an uncertain driving environment, on the basis of the obtained own vehicle state and the surrounding vehicle state, lanes on which all vehicles run at the next moment are decided through a multi-agent reinforcement learning algorithm, and the own vehicle speed is adjusted according to an intelligent driver model, so that collision with a front vehicle is avoided; a plurality of unmanned vehicles can safely change lanes in an uncertain driving environment, so that the passing efficiency is improved, and the traffic jam is reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a cooperative lane-changing control method for unmanned vehicles. Background Technology

[0002] Lane changing is a key technology for autonomous driving, playing a crucial role in improving traffic conditions. Effective lane changing maneuvers can alleviate traffic congestion and reduce travel time, while unsuccessful lane changing attempts can lead to serious traffic accidents. In complex and dynamic driving environments, especially with multiple autonomous vehicles, lane changing control becomes exceptionally complex.

[0003] Current automatic lane change control methods are mainly divided into two categories: rule-based and learning-based. Rule-based methods rely on specific models and assumptions, are applicable to specific lane-changing scenarios, and typically include two sequential steps: trajectory planning and trajectory tracking. While rule-based methods are relatively effective within predefined scenarios or model constraints, their lane-changing trajectories lack adaptability and flexibility to dynamic environments. Therefore, the likelihood of lane-changing failure may increase when encountering unforeseen circumstances. Learning-based methods utilize deep learning to train lane-changing strategies. However, most existing research only considers the behavior of individual vehicles, addressing relatively simple traffic flow tasks. In multi-agent environments, how to achieve multi-vehicle cooperative lane changing while ensuring safety and improving traffic efficiency has become one of the key issues in intelligent driving technology research. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for cooperative lane-changing control of unmanned vehicles.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for cooperative lane-changing control of autonomous vehicles includes the following steps:

[0007] Step 1: Construct and train a Dueling Double DQN network by fusing the Dueling Double DQN network; build a single-vehicle lateral controller based on the Dueling Double DQN network to make lane change decisions for the current vehicle.

[0008] Step 2: Based on the intelligent driver model, construct a single-vehicle longitudinal controller to adjust the current longitudinal speed of the vehicle;

[0009] Step 3: Construct a multi-agent reinforcement learning framework to make lane-changing decisions for multiple vehicles and achieve coordinated lane changing among multiple vehicles.

[0010] Furthermore, the autonomous vehicle can use V2V technology for vehicle-to-vehicle communication or V2I technology for vehicle-to-infrastructure communication to share the vehicle's current status information, including vehicle speed, vehicle position, and vehicle heading angle.

[0011] Furthermore, in step one, the single-vehicle lateral controller utilizes the Dueling Double DQN network and makes lane change decisions for the next moment based on the current environmental information.

[0012] Furthermore, in step two, the single-vehicle longitudinal controller uses an intelligent driver model and adjusts the longitudinal speed of the current vehicle based on the relative position and speed information of the current vehicle and the vehicle in front.

[0013] Furthermore, in step three, the constructed multi-agent reinforcement learning framework is trained to generate a network that iteratively outputs the lane change decisions for all vehicles at the next moment.

[0014] Furthermore, in step one, during the training process of the Dueling Double DQN network, each vehicle is treated as a vehicle agent, and the action space of each vehicle agent includes discrete actions of [0, 1, 2]; where 0 represents keeping the lane unchanged, 1 represents changing lanes to the left, and 2 represents changing lanes to the right.

[0015] The reward function during training is r = r1 + r2 + r3 + r4 + r5;

[0016] in,

[0017] r1 = -w1|vv d |;

[0018] r2 = -w2n;

[0019]

[0020]

[0021] Among them, r1 is a speed penalty term used to encourage the agent to increase vehicle speed; r2 is a lane-changing penalty term; r3 is a lane-center-line deviation penalty term to encourage the vehicle to stay within the lane centerline; r4 is a safety penalty term used to punish unsafe driving behavior; and r5 is a comfort penalty term; w1, w2, w3, w4, w5, and w6 are the corresponding weighting coefficients; and v represents the current vehicle speed. d Let x represent the desired maximum speed, n=0 represent lane keeping, n=1 represent lane changing, and x and y represent the horizontal and vertical coordinates of the vehicle's current position. * and y *It is the position of the lane center line. and The magnitude of acceleration changes in the x and y directions, respectively.

[0022] Compared with the prior art, the beneficial technical effects of the present invention are:

[0023] In uncertain driving environments, this invention allows each vehicle to acquire its own state and the states of surrounding vehicles at any given moment. A multi-agent reinforcement learning algorithm then determines the lanes for all vehicles in the next moment. Simultaneously, the vehicle adjusts its speed based on an intelligent driver model to avoid collisions with the vehicle in front. This technology enables safe lane changing for multiple autonomous vehicles in uncertain driving environments, effectively improving traffic efficiency and alleviating traffic congestion compared to other methods. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the autonomous vehicle cooperative lane-changing control method of the present invention.

[0025] Figure 2 This is the architecture diagram of the Dueling Double DQN network;

[0026] Figure 3 This is a schematic diagram of a multi-agent reinforcement learning framework used to make lane-changing decisions for multiple vehicles.

[0027] Figure 4 This is a schematic diagram of the training results of the multi-vehicle lane-changing decision algorithm. Detailed Implementation

[0028] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] like Figure 1 As shown, this embodiment discloses a cooperative lane-changing control method for autonomous vehicles, including the following steps:

[0030] Step 1: Integrate the advantages of Dueling DQN networks and Double DQN networks, such as... Figure 2 As shown, a Dueling Double DQN network is constructed as the lateral controller of a single vehicle. The motion space and reward function are further constructed, and the Dueling Double DQN network is trained. The lateral control of the single vehicle is achieved through the trained Dueling Double DQN network.

[0031] During the training of the Dueling Double DQN network, the action space of each vehicle agent includes discrete actions of [0, 1, 2]. Here, 0 represents staying in the same lane, 1 represents changing lanes to the left, and 2 represents changing lanes to the right.

[0032] The reward function r during the training of the Dueling Double DQN network is:

[0033] r = r1 + r2 + r3 + r4 + r5;

[0034] in,

[0035] r1 = -w1|vv d |;

[0036] r2 = -w2n;

[0037]

[0038]

[0039] Where w1, w2, w3, w4, w5, and w6 are the corresponding weighting coefficients, and v represents the vehicle speed at the current moment. d Let x represent the desired maximum speed, n=0 represent lane keeping, n=1 represent lane changing, and x and y represent the horizontal and vertical coordinates of the vehicle's current position. * and y * It is the position of the lane center line. and The magnitude of acceleration changes in the x and y directions, respectively.

[0040] Step 2: Based on the intelligent driver model, construct a single-vehicle longitudinal controller. Specifically, in the longitudinal direction, at each moment, the vehicle calculates an appropriate acceleration based on the state information of itself and the vehicle in front, including relative position and relative speed. By executing acceleration or deceleration, it maintains a suitable distance from the vehicle in front, thereby achieving collision avoidance and adjusting the desired distance.

[0041] Step 3: Construct a multi-agent reinforcement learning framework to make lane-changing decisions for multiple vehicles.

[0042] like Figure 3 As shown, this invention constructs a multi-agent reinforcement learning framework centered on the current vehicle (the self-vehicle). Specifically, each vehicle on the road calculates and outputs the lane it should maintain in the next time step based on the multi-agent reinforcement learning framework. Through iteration, each vehicle participates in the calculation as a self-vehicle, thus deciding the lane each vehicle should travel in the next moment before executing an action. This method aims to enable all vehicles to change lanes in a timely and safe manner, thereby improving traffic efficiency.

[0043] The training process of the Dueling Double DQN network specifically includes:

[0044] First, initialize the Dueling Double DQN neural network.

[0045] Secondly, based on the state information of its own vehicle and surrounding vehicles, the vehicle uses a neural network to decide its next action and receives corresponding rewards or penalties. The current state, next state, actions taken, and rewards / penalties of all agents are stored in an experience pool.

[0046] Finally, during training, samples are drawn from the experience pool using a priority experience replay method to train the evaluation network. The parameters of the evaluation network are updated by minimizing the loss function, and the parameters of the corresponding policy network are updated using gradient descent. Soft updates are then used to update the target policy network and the target evaluation network.

[0047] The priority experience replay during the training process involves storing the current state, next state, actions taken, and rewards / penalties of all agents in the experience pool in the form of tuples. During training, more important experiences are extracted first for training, thereby accelerating convergence.

[0048] like Figure 4 As shown, the present invention designed a simulation experiment. The training results show that the autonomous vehicle cooperative lane-changing control method of the present invention (i.e., the DRL-MVLC_DDDQN algorithm in the figure) shows higher rewards compared with the Double DQN algorithm and the Dueling DQN algorithm, and shows a larger average vehicle speed compared with the MOBIL algorithm.

[0049] This invention constructs a Dueling Double DQN network based on Dueling DQN and Double DQN networks to achieve lateral control of a single vehicle. Then, based on an intelligent driver model, a longitudinal controller for a single vehicle is built to achieve collision avoidance with the vehicle in front and adjustment of the desired distance. Finally, a multi-agent reinforcement learning framework centered on the ego vehicle is constructed. Through iteration, each vehicle participates in the calculation as its own and outputs the lane it should maintain in the next time step. This invention utilizes deep reinforcement learning to improve traffic efficiency while ensuring safety, enabling safe lane changing for multiple autonomous vehicles in uncertain driving environments.

[0050] Uncertain driving environments refer to driving environments where multiple driverless vehicles exist on the road, each capable of making autonomous decisions. Therefore, for any given vehicle, the driving status of the surrounding vehicles at the next moment is unknown.

[0051] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0052] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for controlling a coordinated lane change of an unmanned vehicle, characterized in that: The following steps are involved: Step 1: By integrating the Dueling DQN network and the Double DQN network, a Dueling Double DQN network is constructed and trained; Build a single-vehicle lateral controller based on the Dueling Double DQN network to make lane change decisions for the current vehicle; Step 2: Based on the intelligent driver model, a single-vehicle longitudinal controller is constructed to adjust the longitudinal speed of the current vehicle; Step three, build a multi-agent reinforcement learning framework to make multi-vehicle lane changing decisions and realize coordinated lane changing of multiple vehicles.

2. The method for controlling the coordinated lane change of an unmanned vehicle according to claim 1, characterized in that: The unmanned vehicle can use V2V technology to communicate between vehicles, or use V2I technology to communicate between vehicles and infrastructure, and share the vehicle's current status information; the status information includes vehicle speed, vehicle position and vehicle heading angle.

3. The method for controlling the coordinated lane change of an unmanned vehicle according to claim 1, characterized in that: In step 1, the single-vehicle lateral controller uses the Dueling Double DQN network and makes a lane change decision at the next moment according to the environmental information at the current moment.

4. The method for controlling the coordinated lane change of an unmanned vehicle according to claim 1, characterized in that: In step 2, the single-vehicle longitudinal controller uses the intelligent driver model and adjusts the longitudinal speed of the current vehicle according to the relative position and speed information between the current vehicle and the vehicle in front.

5. The method for controlling the coordinated lane change of an unmanned vehicle according to claim 1, characterized in that: In step three, the constructed multi-agent reinforcement learning framework is trained to cyclically output the lane change decisions of all vehicles at the next moment.

6. The method for controlling the coordinated lane change of an unmanned vehicle according to claim 1, characterized in that: In step 1, during the training of the Dueling Double DQN network, each vehicle is regarded as a vehicle agent, and the action space of each vehicle agent includes discrete actions of [0, 1, 2]; wherein 0 indicates that the lane remains unchanged, 1 indicates changing lanes to the left, and 2 indicates changing lanes to the right; The reward function during training is r=r1+r2+r3+r4+r5; in, r1=-w1|vv d |; r2=-w2n; Among them, r1 is the penalty term for speed, r2 is the penalty term for lane changing, r3 is the penalty term for deviation from the lane centerline, r4 is the safety term, and r5 is the penalty term for comfort; w1, w2, w3, w4, w5, w6 are the corresponding weight coefficients, v represents the current speed, v d represents the expected maximum speed, n = 0 represents lane keeping, n = 1 represents lane change, x and y are the horizontal and vertical coordinates of the current position of the vehicle, x * and * is the position of the lane centerline, and The acceleration changes in the x and y directions respectively.

Images