A deep reinforcement learning-based overtaking decision-making method for autonomous driving vehicles

Through deep reinforcement learning network and feature splicing operations, complex tasks are decomposed into subtasks, and reward functions are designed, which solves the problem of insufficient extraction of shallow network features and sparse rewards in decision-making of autonomous vehicles, and achieves collision-free and continuous high-speed overtaking Efficient decision-making.

CN114655211BActive Publication Date: 2024-12-03SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210332505.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-12-03
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

Among the existing decision-making methods of autonomous driving vehicles, rules-based methods are difficult to adapt to complex and dynamic traffic environments. Based on imitation learning, a large amount of labeled data is required. The shallow network feature extraction based on reinforcement learning is insufficient and sparse rewards lead to low training efficiency and strategies. Not convergent.

Method used

Using the overtaking decision-making method of autonomous driving vehicles based on deep reinforcement learning, a reinforcement learning network is built through the actor and critic architecture, including a policy network, a value network and a Q-value network, and the complex tasks are decomposed into two subtasks, using deep fully connected networks and feature stitching. Operation, design reward function to guide learning.

Benefits of technology

End-to-end learning control for collision-free and continuous high-speed overtaking tasks is achieved, the learning efficiency and strategy accuracy of reinforcement learning are improved, and the dependence of manual rules and large amounts of labeled data is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114655211B_ABST
    Figure CN114655211B_ABST
Patent Text Reader

Abstract

The present invention discloses an overtaking decision method for an autonomous driving vehicle based on deep reinforcement learning. The method comprises: constructing a reinforcement learning network based on an actor-critic architecture, the reinforcement learning network comprising a policy network, a value network and a Q-value network, wherein the policy network generates continuous driving actions by observing the driving environment; the vehicle applies the continuous driving actions generated by the policy network to the driving decision task, and through interaction with the driving environment, performs autonomous driving under the guidance of a reward function with the goal of achieving collision-free and continuous high-speed overtaking, wherein the driving decision task comprises a first subtask and a second subtask, the first subtask is to achieve high-speed driving of an autonomous driving vehicle in a single-vehicle scenario, the second subtask is to achieve an overtaking task in a multi-vehicle environment, and the initial strategy of the second subtask is the optimal strategy obtained by the first subtask. The present invention can achieve collision-free and continuous high-speed overtaking tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving and intelligent decision-making technology, and more specifically, to an overtaking decision method for an autonomous driving vehicle based on deep reinforcement learning. Background Art

[0002] Autonomous driving vehicles (intelligent vehicles) have changed the control method of traditional vehicles, freeing people from the "vehicle-road-people" closed-loop control mode, thereby improving the efficiency and safety of the transportation system and alleviating many problems caused by vehicle use, such as traffic congestion, energy shortages and environmental pollution.

[0003] In the prior art, the methods for solving the decision-making problem of autonomous vehicles mainly include rule-based decision-making, imitation learning-based decision-making, and reinforcement learning-based decision-making. Rule-based methods usually use manually coded rules for each specific scenario, while imitation-based methods require a large amount of labeled data in the supervised learning process to cover as many driving scenarios as possible. For rule-based decision-making, a typical framework is to design states and actions, using a complete finite state machine to select actions such as keeping lanes, changing lanes, and overtaking, and the action switching is adapted to different driving scenarios.

[0004] Imitation learning-based decision making learns driving actions in a supervised learning form, without manually formulating rules. For example, DAgger uses the current strategy to collect data once in each iteration, and uses all the collected data sets to train the next strategy. GAIL is also a model-free imitation learning method that builds a generative model, which is a random strategy that produces an expert-like action. On this basis, InfoGAIL learns an abstract variable feature generation model to reproduce the expert's actions and generate a semantically meaningful observation space. Imitation-based methods require a large amount of labeled data in the supervised learning process to cover as many driving scenarios as possible. These data are usually provided by human experts. Since they are expert data from users with different skills and habits, these data have significant differences. The final effect of imitation learning depends on the quality of the expert data.

[0005] The decision-making based on reinforcement learning aims to maximize the cumulative reward. The policy network is optimized by sampling data through the interaction between the autonomous driving vehicle and the environment. It is a self-learning method through trial and error, without the need for manually formulated rules and manually labeled data. In the networks based on reinforcement learning methods, most of them adopt shallow network structures, which have the problem of insufficient feature extraction. In the face of complex tasks, the sparse rewards obtained by reinforcement learning will lead to slow training efficiency and even non-convergence of the final strategy.

[0006] Rule-based methods usually use manually coded rules for each specific scenario. As the number of scenarios increases, the manually coded rules will become more and more complex, and the entire rule base will be difficult to maintain. Therefore, manually designed rules are not suitable for the dynamic and random nature of the traffic environment. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an overtaking decision method for an autonomous driving vehicle based on deep reinforcement learning. The method comprises the following steps:

[0008] A reinforcement learning network is constructed based on the actor-critic architecture, which includes a policy network, a value network and a Q-value network, wherein the policy network generates continuous driving actions by observing the driving environment; the policy network, the value network and the Q-value network are trained based on the sampling information of the experience pool, and the experience pool stores the correspondence between observations and actions;

[0009] The vehicle applies the continuous driving actions generated by the policy network to the driving decision-making task, and through interaction with the driving environment, performs autonomous driving under the guidance of the reward function with the goal of achieving collision-free and continuous high-speed overtaking. The driving decision-making task includes a first subtask and a second subtask. The first subtask is to achieve high-speed driving of the autonomous driving vehicle in a single-vehicle scenario, and the second subtask is to achieve overtaking in a multi-vehicle environment. The initial strategy of the second subtask is the optimal strategy obtained from the first subtask.

[0010] Compared with the prior art, the advantages of the present invention are that an end-to-end learning control method is proposed, in which the interaction between the intelligent agent (unmanned vehicle) and the environment is guided by the reward function to achieve collision-free, continuous high-speed overtaking tasks. In addition, the problem of insufficient feature extraction of shallow neural networks in reinforcement learning and the problem of sparse rewards in complex scenarios are solved, the learning efficiency of reinforcement learning is improved, and the accuracy of the obtained driving strategy is guaranteed.

[0011] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 is a schematic diagram of a deep reinforcement learning network framework according to an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of information in a state space according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a network structure according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram for explaining the structure of a policy network according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of the arrangement of 12 vehicles according to one embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of overtaking distance and time for overtaking at different departure intervals according to an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of 12 vehicles arranged at intervals of 50 m according to an embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of 12 vehicles arranged at intervals of 80 m according to an embodiment of the present invention;

[0021] Fig. 9 It is a schematic diagram of 12 vehicles arranged at intervals of 150 m according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0023] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0024] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0025] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0026] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0027] See also Figure 1As shown in the figure, the reinforcement learning network provided is built on the actor-critic architecture, which includes a policy network, a value network, and a Q-value network. The policy network is used as an actor to generate continuous driving actions (or driving strategies) by observing the driving environment. The critic is a value function, which is used to estimate the value function of the current driving strategy. Figure 1 There are two estimates in the network, namely the value network and the Q-value network. The agent (unmanned vehicle) applies continuous driving actions to enter the next state, obtains reward feedback, and stores the correspondence between observations and actions in the experience pool. The experience pool is sampled and the sampling information is used to update the policy network, value network, and Q-value network.

[0028] It should be noted that in order to solve the problem of slow training or non-convergence caused by sparse rewards, Figure 1 In the embodiment, the complex overtaking task (i.e., the driving decision task of the agent) is decomposed into two consecutive subtasks, including two stages of curriculum learning. The goal of the first stage is to learn how to drive as fast as possible in a single-car environment, and then the goal of the second stage is to learn how to overtake other vehicles as quickly as possible in a multi-car environment without any collisions.

[0029] Specifically, still combined with Figure 1 As shown in the figure, in the continuous high-speed overtaking scenario, the task is divided into two subtasks. The first subtask is to complete the high-speed driving of the autonomous driving vehicle in a single-vehicle scenario, and the second subtask is to complete the fast overtaking task in a multi-vehicle environment. The initial strategy of the second subtask is the optimal strategy obtained in the first subtask, that is, the second subtask is the progression of the first subtask, and the strategy obtained in the first subtask is used to continue to optimize the strategy to achieve the final continuous high-speed overtaking task goal.

[0030] In the following, the idea of ​​the present invention is described by taking the simulation environment as an example. For example, the open source autonomous driving simulator TORCS is used as the test platform. The state information can be directly read from the TORCS environment. Figure 2 As shown, θ k is the angle between the autonomous vehicle and the centerline of the lane, d k is the distance between the autonomous vehicle and the lane boundary, bi k is the vertical distance between the center of the autonomous vehicle and the centerline of the lane, v k is the speed of the autonomous vehicle, w k is the speed of the wheels of the autonomous vehicle, rpm k is the engine speed of the autonomous vehicle, δ k-1 is the turning action at the previous moment, f c is the collision mark between the autonomous driving vehicle and other vehicles, f wIt is the collision mark between the autonomous vehicle and the lane edge, op k is the distance between the autonomous vehicle and other vehicles.

[0031] In one embodiment, the action output by the reinforcement learning policy network includes turning δ k , Acceleration k and brake k One or more of the following.

[0032] In order to obtain efficient feature extraction, the policy network, value network and Q-value network, for example, use a deep fully connected network, each including an input layer, multiple hidden layers and an output layer. In order to solve the problem of gradient disappearance or explosion, the policy network, value network and Q-value network can use the method of direct connection of hidden layer features, but considering that this feature direct connection cannot strengthen the direct transmission of input features from the input layer to the hidden layer. Therefore, further, a feature splicing operation is designed to input the input state to each hidden layer.

[0033] For example, for the framework based on the Soft Actor-Critic (SAC) network, the two-layer fully connected structure in SAC is improved to a four-layer fully connected structure, and the direct connection and feature concatenation of features are added. Figure 3 Take the policy network as an example. For the specific network structure explanation, see Figure 4 , the input data dimension is set to 68, X1 is the output of the first fully connected layer, the input of the second fully connected layer is [S, X1], X2 is the output of the second fully connected layer, the input of the third fully connected layer is [S, X1+X2], X3 is the output of the third fully connected layer, the input of the fourth fully connected layer is [S, X2+X3], and X4 is the output of the fourth fully connected layer. Through this direct connection and splicing operation, the input state can be passed to each hidden layer, so as to fully extract the data features.

[0034] In summary, the deep reinforcement learning (or deep curriculum reinforcement) method adopted in the present invention collects data and trains online, and obtains the optimal driving strategy under the guidance of the reward function. A deep network structure is used to extract effective features, and complex tasks are decomposed into multiple continuous and progressive subtasks, and the problem of sparse rewards is overcome by gradual training. This method does not require manual setting of rules and is suitable for decision-making in complex scenarios; it does not require a large amount of manually labeled data and is not affected by differences in data distribution. It is particularly suitable for decision-making in complex scenarios in the field of autonomous driving, such as highway scenarios with multiple vehicles.

[0035] In order to verify the effectiveness of the present invention, it is applied to the open source autonomous driving simulator TORCS. In complex scenarios, a large number of decision rule methods are required, which is not suitable for manually pre-setting rules. A learning method is needed to learn the characteristics of the scene and make reasonable decision actions.

[0036] The deep course reinforcement learning method adopted in the embodiment of the present invention decomposes the task of continuous overtaking into two subtasks, the first of which is how to drive quickly and the second is how to overtake quickly. A key part of the reinforcement learning algorithm is to design a reward function. For example, the reward function for fast driving is set to:

[0037] R d =k1v+k2△sc w k3v 2 (1)

[0038] Where v is the driving speed. This item rewards the smart car for fast driving. △s = s k -s k-1 This reward smart car to drive forward along the road, s k is the length of the intelligent car driving along the center line of the road at the kth moment, c w = {0,1} indicates whether a collision with the road edge occurs, k i ,i=1,2,3 are the weight coefficients of each item.

[0039] In one embodiment, the reward function for fast overtaking is set to:

[0040] R o =R d +k p (Np)-c c k4v 2 (2)

[0041] Where N is the total number of vehicles, p is the ranking of the autonomous driving vehicle at the start time, and c c = {0,1} indicates whether a collision with other vehicles occurs, k p and k4 are the weight coefficients of each item.

[0042] Specifically, for example, in a multi-vehicle environment, 12 vehicles are arranged, such as Figure 5 As shown, the vehicle corresponding to the upper left corner is controlled by the reinforcement learning algorithm, and the remaining 11 vehicles are controlled by the built-in AI algorithm. At the start time, the vehicle controlled by the reinforcement learning algorithm is placed in the last position, that is, the ranking position p=12. Figure 6 These are the test results obtained at different departure intervals. The time and driving distance of each car overtaken were recorded in the experiment. The speed was not high in the initial acceleration stage but accelerated quickly, and then reached 150km / h.

[0043] In the experiment, we compared it with other existing methods. Figures 7 to 9 The circle in the middle represents the method of the embodiment of the present invention, and the square represents the improved Soft Actor-critic method. Experimental results show that the method proposed by the present invention can complete the continuous overtaking task on the highway faster and more efficiently.

[0044] In summary, the present invention designs a continuous overtaking strategy based on a deep course reinforcement learning algorithm. Through interaction with the environment, the intelligent agent learns the skills of high-speed driving and continuous overtaking, avoiding a large number of labeled samples and manually set rules; the designed feature direct connection and feature splicing are introduced into the strategy network, value network and Q-value network, which effectively extracts data features and is conducive to the final strategy of reinforcement learning; the complex task is decomposed into two consecutive sub-tasks, which solves the problem of sparse rewards, improves the exploration efficiency of the early stage of the strategy, and accelerates the convergence of reinforcement learning.

[0045] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0046] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0047] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0048] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0049] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0050] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0051] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0052] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0053] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for overtaking decision of an autonomous driving vehicle based on deep reinforcement learning, comprising the following steps: A reinforcement learning network is constructed based on the actor-critic architecture, which includes a policy network, a value network and a Q-value network, wherein the policy network generates continuous driving actions by observing the driving environment; the policy network, the value network and the Q-value network are trained based on the sampling information of the experience pool, and the experience pool stores the correspondence between observations and actions; The vehicle applies the continuous driving actions generated by the policy network to the driving decision-making task, and through interaction with the driving environment, performs automatic driving under the guidance of the reward function with the goal of achieving collision-free and continuous high-speed overtaking. The driving decision-making task includes a first subtask and a second subtask. The first subtask aims to achieve high-speed driving of the vehicle in a single-vehicle scenario, and the second subtask aims to achieve overtaking in a multi-vehicle environment. The initial strategy of the second subtask is the optimal strategy obtained from the first subtask.

2. The method according to claim 1, characterized in that The interaction between the vehicle and the driving environment includes the action of obtaining state space information and the output of the strategy network, wherein the state space information includes the angle between the vehicle and the center line of the lane, the distance between the vehicle and the lane boundary, the vertical distance between the center of the vehicle and the center line of the lane, the speed of the vehicle, the speed of the vehicle wheels, the speed of the vehicle engine, the steering action of the vehicle at the last moment, the collision mark between the vehicle and other vehicles, the collision mark between the vehicle and the edge of the lane, and the distance between the vehicle and other vehicles.

3. The method according to claim 1, characterized in that The continuous driving actions generated by the policy network include one or more of steering, acceleration and braking.

4. The method according to claim 1, characterized in that: The policy network, the value network and the Q-value network each include multiple hidden layers, and the input state is passed to each hidden layer.

5. The method according to claim 1, characterized in that The reward function of the first subtask is set to: R d =k1v+k2△s-c w k3v 2 Where v is the driving speed, △s = s k -s k-1 Used to reward a vehicle for driving forward along a road, s k is the length of the vehicle driving along the center line of the road at the kth moment, c w Indicates whether the vehicle collides with the road edge, k i ,i=1,2,3 are the weight coefficients of the relevant items.

6. The method according to claim 5, characterized in that The reward function of the second subtask is set to: R o =R d +k p (N-p)-c c k4v 2 Among them, p is the ranking at the start time, N is the total number of vehicles, c c = {0,1} indicates whether a collision with other vehicles occurs, k p and k4 are the weight coefficients of each item.

7. The method according to claim 1, characterized in that In the reinforcement learning process, different departure intervals are set for each target vehicle, and the corresponding relationship between observation and action under different departure intervals is obtained and stored in the experience pool.

8. The method according to claim 4, characterized in that The strategy network, the value network and the Q-value network are all provided with four hidden layers of fully connected structure, X1 is the output of the first fully connected layer, the input of the second fully connected layer is [S, X1], X2 is the output of the second fully connected layer, the input of the third fully connected layer is [S, X1+X2], X3 is the output of the third fully connected layer, the input of the fourth fully connected layer is [S, X2+X3], X4 is the output of the fourth fully connected layer, wherein S represents the input data.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.