An integrated horizontal and vertical decision-making method for autonomous driving based on deep reinforcement learning

Abstract

The present invention provides a method, system, storage medium and electronic device for integrated horizontal and vertical decision-making of autonomous driving based on deep reinforcement learning, and relates to the field of autonomous driving technology. The present invention includes real-time acquisition of the state of the autonomous driving vehicle and the surrounding environment; using the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to obtain the action space; according to the reinforcement learning decision model, with the goal of maximizing the expected reward, selecting and executing the action with the highest evaluation; repeating the above steps until the target point is reached. By collecting and processing a large amount of driving data, automatically learning and optimizing the intelligent decision-making network, it is possible to quickly adapt to various driving scenarios and tasks without retraining the model, thereby improving the accuracy and stability of decision-making, and laying a solid foundation for the commercial application of autonomous driving technology.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically to a method, system, storage medium, and electronic device for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning. Background Art

[0002] Traditional reinforcement learning algorithms are often designed to solve specific driving scenarios or tasks, such as car-following and obstacle avoidance for autonomous vehicles on highways. The decision-making models these algorithms generate after training are only suitable for specific driving scenarios or tasks and are not well adapted to different driving tasks or scenarios. This leads to a significant gap between traditional reinforcement learning algorithms and human drivers, who can operate vehicles with ease in a wide variety of driving scenarios and tasks.

[0003] However, during the autonomous driving process, traffic conditions on the road are often changeable. How to accurately predict these complex traffic conditions to formulate the best driving plans and decisions and drive to the destination safely and efficiently is a problem that autonomous driving intelligent decision-making needs to solve. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides an integrated lateral and longitudinal decision-making method, system, storage medium and electronic device for autonomous driving based on deep reinforcement learning, which solves the technical problem of being unable to accurately predict complex traffic conditions.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning, comprising pre-training an intelligent decision-making network comprising a deep learning model and a reinforcement learning decision-making model; the method comprising:

[0009] S1, real-time collection of the status of the autonomous driving vehicle and its surrounding environment;

[0010] S2. Using the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to obtain an action space;

[0011] S3. Select and execute the action with the highest evaluation according to the reinforcement learning decision model with the goal of maximizing the expected reward;

[0012] S4. Repeat S1 to S3 until the target point is reached.

[0013] Preferably, the state of the autonomous driving vehicle and the surrounding environment in S1 includes:

[0014]

[0015]

[0016]

[0017] Among them, s t represents the state of the autonomous driving vehicle and its surrounding environment at time t; Represents the state of the vehicle at time t, including the speed of the vehicle at time t Car position Represents the target state at time t, using the target position p at time t t goal express; represents the status of other vehicles at time t; Indicates other vehicle status.

[0018] Preferably, the states of the other vehicles are acquired in real time by sensors and represented by a grid occupancy map;

[0019] The grid occupied map specifically refers to:

[0020] A 7×40 binary matrix is ​​used to represent the occupancy status of the 12m×120m area around the vehicle.

[0021] The grid occupancy map is defined in the Frenet coordinate system, using the variable s to represent the front and rear position of the vehicle along the road, and the variable d to represent the left and right position of the vehicle on the road;

[0022] Along the s-axis, the area 100m in front of the vehicle and 20m behind the vehicle is used as the sampling area; along the d-axis, the area 6m to the left and right of the vehicle is used as the sampling area.

[0023] Preferably, the action space a in S2 t It includes five discrete behaviors: acceleration, deceleration, left lane change, right lane change, and state maintenance;

[0024] Among them, acceleration is defined as increasing the current speed by 2m / s, deceleration is defined as decreasing the current speed by 2m / s, left lane change is defined as changing to the left lane, right lane change is defined as changing to the right lane, and holding state is defined as driving along the lane at the current speed for a fixed period of time.

[0025] Preferably, the deep learning model adopts a three-layer stacked LSTM network.

[0026] Preferably, the reinforcement learning decision model is trained using an Actor-Critic method;

[0027] Among them, the policy network π(a|s;θ) is used to input the state information s at time t t , after the deep learning model operation output action space a t , the reinforcement learning decision model selects the optimal action according to the current decision strategy π The value network q(s,a;ω) is used to evaluate the t Take action in the event of How good or bad is it?

[0028] Execute an action After the autonomous driving vehicle forms a new state t+1 , and get the feedback reward r from the environment t ; The reinforcement learning decision model is based on the reward function r t , continuously maximize rewards and learn the optimal decision strategy π * .

[0029] Preferably, the reward function r t Specifically:

[0030]

[0031]

[0032] Among them, disr r represents the vehicle position at time t and the target position p t goal the distance between them; represents the x-coordinate of the vehicle at time t, represents the x-coordinate of the target point at time t, represents the y coordinate of the vehicle at time t, Represents the y coordinate of the target point at time t; represents the vehicle speed at time t

[0033] An integrated lateral and longitudinal decision-making system for autonomous driving based on deep reinforcement learning, comprising a pre-trained intelligent decision-making network comprising a deep learning model and a reinforcement learning decision-making model; the system comprises:

[0034] The acquisition module is used to collect the status of the autonomous driving vehicle and its surrounding environment in real time;

[0035] An acquisition module, configured to use the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to acquire an action space;

[0036] A selection module is used to select and execute the action with the highest evaluation based on the reinforcement learning decision model with the goal of maximizing the expected reward;

[0037] The repetition module is used to transfer to the acquisition module, the acquisition module and the selection module and execute them repeatedly until the target point is reached.

[0038] A storage medium storing a computer program for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning, wherein the computer program enables a computer to execute the autonomous driving intelligent decision-making method as described above.

[0039] An electronic device, comprising:

[0040] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the autonomous driving intelligent decision-making method as described above.

[0041] (3) Beneficial effects

[0042] The present invention provides a method, system, storage medium, and electronic device for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning. Compared with existing technologies, it has the following advantages:

[0043] The present invention involves collecting the real-time status of the autonomous vehicle and its surroundings; using the current state of the autonomous vehicle and its surroundings as input to a deep learning model to obtain an action space; selecting and executing the highest-rated action based on the reinforcement learning decision model with the goal of maximizing expected reward; and repeating these steps until the target point is reached. By collecting and processing large amounts of driving data, the intelligent decision-making network is automatically learned and optimized, enabling rapid adaptation to various driving scenarios and tasks without requiring model retraining. This improves the accuracy and stability of decision-making and lays a solid foundation for the commercial application of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A technical roadmap for an integrated lateral and longitudinal decision-making method for autonomous driving based on deep reinforcement learning, provided by an embodiment of the present invention;

[0046] Figure 2 A block diagram of an integrated lateral and longitudinal decision-making method for autonomous driving based on deep reinforcement learning provided by an embodiment of the present invention;

[0047] Figure 3 An example diagram of a grid occupancy map provided by an embodiment of the present invention;

[0048] Figure 4 This is an example diagram of another grid occupancy map provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0050] The embodiments of the present application solve the technical problem of being unable to accurately predict complex traffic conditions by providing an integrated lateral and longitudinal decision-making method, system, storage medium and electronic device for autonomous driving based on deep reinforcement learning.

[0051] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0052] like Figure 1 As shown in the figure, the present invention proposes an intelligent decision-making method for autonomous driving based on deep reinforcement learning. This method uses sensors to obtain the status of the vehicle and other surrounding vehicles in real time. t , input into the deep learning network model to generate decision candidates (i.e. action space) a t , the reinforcement learning model selects the optimal action The vehicle performs the optimal action The environment gives a reward function r t , and reaches the next state s t+1 , repeating this process until the target point is reached and the current task is completed. By collecting and processing large amounts of driving data, automatically learning and optimizing the intelligent decision-making network, it can quickly adapt to various driving scenarios and tasks without retraining the model, thereby improving the accuracy and stability of decision-making and laying a solid foundation for the commercial application of autonomous driving technology.

[0053] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] Example:

[0055] like Figure 2 As shown, an embodiment of the present invention provides an integrated lateral and longitudinal decision-making method for autonomous driving based on deep reinforcement learning, wherein an intelligent decision-making network is pre-trained, and the intelligent decision-making network includes a deep learning model and a reinforcement learning decision-making model; the method includes:

[0056] S1, real-time collection of the status of the autonomous driving vehicle and its surrounding environment;

[0057] S2. Using the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to obtain an action space;

[0058] S3. Select and execute the action with the highest evaluation according to the reinforcement learning decision model with the goal of maximizing the expected reward;

[0059] S4. Repeat S1 to S3 until the target point is reached.

[0060] The embodiments of the present invention collect and process large amounts of driving data, automatically learn and optimize intelligent decision-making networks, and can quickly adapt to various driving scenarios and tasks without retraining the model, thereby improving the accuracy and stability of decision-making and laying a solid foundation for the commercial application of autonomous driving technology.

[0061] The following is a detailed introduction to the various steps of the above technical solution:

[0062] First of all, it should be noted that the intelligent decision-making network pre-trained based on the deep reinforcement learning algorithm in the implementation of the present invention is specifically composed of two parts, a deep learning model and a reinforcement learning decision model.

[0063] Deep learning models use multi-layer neural networks to learn complex relationships between input data. Deep learning models are often used to learn from state s t Extract features and generate decision candidates.

[0064] Reinforcement learning decision-making models aim to teach an agent how to make decisions in an unknown environment. In reinforcement learning, the agent interacts with the environment and receives rewards or penalties from it. Reinforcement learning decision-making models optimize the agent's strategy to maximize future rewards.

[0065] In step S1, the state of the autonomous driving vehicle and its surrounding environment is collected in real time.

[0066] The status of the autonomous driving vehicle and its surrounding environment in this step includes:

[0067]

[0068]

[0069]

[0070] Among them, s t represents the state of the autonomous driving vehicle and its surrounding environment at time t; Represents the state of the vehicle at time t, including the speed of the vehicle at time t Car position Represents the target state at time t, using the target position p at time t t goal express; represents the status of other vehicles at time t; Indicates other vehicle status.

[0071] In particular, the status of other vehicles is acquired by sensors in real time and represented using a grid occupancy map;

[0072] Among them, Figures 3-4 As shown, the grid occupied map specifically refers to:

[0073] A 7×40 binary matrix is ​​used to represent the occupancy of the 12m×120m area around the vehicle, for example

[0074] The grid occupancy map is defined in the Frenet coordinate system, using the variable s to represent the front-to-back position of the vehicle along the road (i.e., longitudinal displacement), and the variable d to represent the left-to-right position of the vehicle on the road (i.e., lateral offset).

[0075] Along the s-axis, the area 100m in front of the vehicle and 20m behind the vehicle is used as the sampling area; along the d-axis, the area 6m to the left and right of the vehicle is used as the sampling area.

[0076] Compared to other representation methods based on raw data like lasers or images, raster maps have a relatively low dimensionality, effectively reducing the complexity of the state space, lowering the algorithm's computational workload and training time. Furthermore, they can be widely applied in diverse scenarios and environments, such as cities and highways. Compared to other representation methods, raster maps have better generalization performance and are more adaptable to diverse environments.

[0077] The use of grid occupancy maps, an input state that is not much different between the simulation and the actual environment, ensures that the state domain is not much different, and the training model of the simulation environment can be easily transferred to the real vehicle environment.

[0078] In step S2, the current state of the autonomous driving vehicle and the surrounding environment are used as inputs to the deep learning model to obtain the action space.

[0079] In autonomous driving, vehicles need to analyze historical data to predict future dynamic states and make appropriate driving decisions. Decisions based on single-frame observations often miss important state information. Therefore, embodiments of the present invention incorporate multi-frame data into a deep learning model and employ an LSTM network to process time-series inputs. The LSTM network can model historical states and predict current states to guide decision-making. By incorporating LSTM, intelligent vehicles can more accurately predict dynamic changes on the road, enabling more precise driving decisions and further improving decision-making accuracy.

[0080] Specifically, the deep learning model described in this step adopts a three-layer stacked LSTM network. The input layer, hidden layer, and output layer respectively constitute the input, processing, and output parts of the neural network. In the hidden layer, each LSTM unit contains three gates and a cell state, including a forget gate, an input gate, and an output gate, as well as a cell state that maintains the current state. In order to improve the expressive power of the network, the number of cells in the LSTM network of the present invention is 128, and a structure of three recurrent layers is adopted. A dropout rate of 0.2 is adopted between different layers to avoid overfitting problems. In addition, the embodiment of the present invention adopts the Adam optimizer, and sets the learning rate to a=0.0005 and the decay rate to 0.9 to accelerate the convergence speed of network training.

[0081] The selection of the input length of the LSTM network is crucial to the performance of the model. An input length that is too short will cause the model to be unable to capture the inherent mathematical characteristics of the input data, thereby reducing the accuracy of feature extraction; while an input length that is too long will increase the amount of model calculation, thereby increasing the time cost of training and inference. In order to balance the accuracy of feature extraction and computational efficiency, the present invention selects a relatively long input length in the model design, that is, 20 frames of historical data (s t ,s t-1 …s t-19 ) as the input layer, the last layer outputs action a t .

[0082] By adding LSTM to the network structure to implement the memory function, previous operations and previous rewards are used as input to provide more complete information and improve the speed of model convergence.

[0083] In addition, as shown in Table 1, the action space a in this step t It includes five discrete behaviors: acceleration, deceleration, left lane change, right lane change and state maintenance.

[0084]

[0085] Table 1

[0086] Among them, acceleration is defined as increasing the current speed by 2m / s, deceleration is defined as decreasing the current speed by 2m / s, left lane change is defined as changing to the left lane, right lane change is defined as changing to the right lane, and holding state is defined as driving along the lane at the current speed for a fixed period of time.

[0087] In step S3, according to the reinforcement learning decision model, the action with the highest evaluation is selected and executed with the goal of maximizing the expected reward.

[0088] It is supplemented that the embodiment of the present invention adopts the Actor-Critic method to train the reinforcement learning decision model;

[0089] Among them, the policy network π(a|s;θ) is used to input the state information s at time t t , after the deep learning model operation output action space a t , the reinforcement learning decision model selects the optimal action according to the current decision strategy π The value network q(s,a;ω) is used to evaluate the t Take action in the event of How good or bad is it?

[0090] Execute an action After the autonomous driving vehicle forms a new state t+1 , and get the feedback reward r from the environment t ; The reinforcement learning decision model is based on the reward function r t , continuously maximize rewards and learn the optimal decision strategy π * .

[0091] The reward function r t Reaching the destination is rewarded with a large positive reward, while collisions are penalized accordingly. During driving, in order to prevent the vehicle from driving too slowly, relatively high speeds are rewarded to improve the efficiency of completing the driving task. Specifically:

[0092]

[0093]

[0094] Among them, dist t represents the vehicle position at time t and the target position p t goal the distance between them; represents the x-coordinate of the vehicle at time t, represents the x-coordinate of the target point at time t, represents the y coordinate of the vehicle at time t, Represents the y coordinate of the target point at time t;

[0095] represents the vehicle speed at time t

[0096] Overtaking is defined as a vehicle taking an action to change lanes left or right. Overtaking can improve driving efficiency, especially in congested traffic, by shortening travel time and reducing congestion. However, overtaking also carries certain risks, so a smaller positive reward of 5 is assigned to the overtaking action.

[0097] A collision state is defined as a situation where a vehicle overlaps with another vehicle in the grid occupancy map, meaning two or more vehicles occupy the same grid. This situation represents a traffic accident and is therefore given a large negative reward.

[0098] At the same time, the experience gained (s t ,a t ,s t+1 ,r t ) is stored in the experience pool, and the parameters of the policy network are continuously updated and new experiences are collected through priority experience replay to obtain the optimal strategy, so that the current value network has the highest evaluation of the current action.

[0099] When the amount of experience reaches the preset capacity of the experience pool, a set number of experience vectors of batch size are extracted from the experience pool according to the principle of priority experience replay to learn the parameters of the policy network a(a|s;θ) and the value network q(s,a;ω).

[0100] The principle of prioritizing experience replay prioritizes experience vectors in the experience pool that contain a large amount of TD error. TD error refers to the difference between the target value and the predicted value. A larger TD error indicates a greater impact of the experience vector on network updates. Therefore, experience vectors with high TD error are prioritized for training, thereby improving model efficiency and performance.

[0101] The goal of the policy network π(a|s;θ) is to maximize the expected reward and improve the performance of the policy by updating the parameters of the policy network. The update direction of the policy network can be calculated using the backpropagation algorithm.

[0102] For example, the loss function of the policy network is defined as:

[0103]

[0104] Where θ is the parameter of the policy network, q(s,a;ω) is the value network based on the current state s t The score scored, is the back-propagation gradient of the policy network.

[0105] The goal of the value network q(s, a; ω) is to accurately estimate the value function of the state action and improve the accuracy of the estimation by updating the parameters of the value network. The update direction of the value network can be calculated by calculating the error between the predicted value and the target value and adjusting the network parameters using the backpropagation algorithm.

[0106] For example, the loss function of the value network is defined as:

[0107]

[0108] Among them, ω is the parameter of the value network, q(s t ,a t ;ω) is the network’s state s at time t t and action a t The score, q(s t+1 ,a t+1 ;ω) is the network’s state s at time t+1 t+1 and action a t+1 The score given, γ is the discount coefficient.

[0109] According to the trained reinforcement learning decision model, with the goal of maximizing the expected reward, the action with the highest evaluation is selected and executed in the action space obtained in S2.

[0110] In step S4, steps S1 to S3 are repeated until the target point is reached.

[0111] An embodiment of the present invention provides an integrated lateral and longitudinal decision-making system for autonomous driving based on deep reinforcement learning. The system includes a pre-trained intelligent decision-making network comprising a deep learning model and a reinforcement learning decision-making model. The system comprises:

[0112] The acquisition module is used to collect the status of the autonomous driving vehicle and its surrounding environment in real time;

[0113] An acquisition module, configured to use the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to acquire an action space;

[0114] A selection module is used to select and execute the action with the highest evaluation based on the reinforcement learning decision model with the goal of maximizing the expected reward;

[0115] The repetition module is used to transfer to the acquisition module, the acquisition module and the selection module and execute them repeatedly until the target point is reached.

[0116] An embodiment of the present invention provides a storage medium storing a computer program for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning, wherein the computer program enables a computer to execute the autonomous driving intelligent decision-making method as described above.

[0117] An electronic device, comprising:

[0118] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a program for executing the autonomous driving intelligent decision-making method as described above,

[0119] It is understandable that the autonomous driving lateral and longitudinal integrated decision-making system based on deep reinforcement learning, storage medium and electronic device provided in the embodiments of the present invention correspond to the autonomous driving lateral and longitudinal integrated decision-making method based on deep reinforcement learning provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the autonomous driving intelligent decision-making method, and will not be repeated here.

[0120] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0121] 1. By collecting and processing large amounts of driving data, the embodiments of the present invention automatically learn and optimize intelligent decision-making networks, enabling rapid adaptation to various driving scenarios and tasks without the need to retrain the model. This improves the accuracy and stability of decision-making and lays a solid foundation for the commercial application of autonomous driving technology.

[0122] 2. Unlike the method of using camera images or radar raw data as network input, the use of grid occupancy maps, which are input states that are not much different from the simulation and the actual environment, ensures that the state domain is not much different, and can easily transfer the training model of the simulation environment to the real vehicle environment.

[0123] 3. By adding LSTM to the network structure to implement the memory function, previous operations and previous rewards are used as input to provide more complete information and improve the speed of model convergence.

[0124] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning, characterized in that: An intelligent decision network is pre-trained, wherein the intelligent decision network includes a deep learning model and a reinforcement learning decision model; the method includes: S1, real-time collection of the status of the autonomous driving vehicle and its surrounding environment; S2. Using the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to obtain an action space; S3. Select and execute the action with the highest evaluation according to the reinforcement learning decision model with the goal of maximizing the expected reward; S4, repeat S1 to S3 until reaching the target point; The autonomous driving vehicle and surrounding environment states in S1 include: Among them, s t represents the state of the autonomous driving vehicle and its surrounding environment at time t; Represents the state of the vehicle at time t, including the speed of the vehicle at time t Car position Represents the target state at time t, using the target position p at time t t goal express; represents the status of other vehicles at time t; Indicates other vehicle status; The states of the other vehicles are acquired in real time by sensors and represented by a grid occupancy map; The grid occupied map specifically refers to: A 7×40 binary matrix is ​​used to represent the occupancy status of the 12m×120m area around the vehicle. The grid occupancy map is defined in the Frenet coordinate system, using the variable s to represent the front and rear position of the vehicle along the road, and the variable d to represent the left and right position of the vehicle on the road; Along the s-axis, the area 100m in front of the vehicle and 20m behind the vehicle is used as the sampling area; along the d-axis, the area 6m to the left and right of the vehicle is used as the sampling area; Using the Actor-Critic method to train the reinforcement learning decision model; Among them, the policy network π(a|s;θ) is used to input the state information s at time t t , after the deep learning model operation output action space a t , the reinforcement learning decision model selects the optimal action according to the current decision strategy π The value network q(s,a;ω) is used to evaluate the t Take action in the event of How good or bad is it? Execute an action After the autonomous driving vehicle forms a new state t+1 , and get the feedback reward r from the environment t ; The reinforcement learning decision model is based on the reward function r t , continuously maximize rewards and learn the optimal decision strategy π * ; The reward function r t Specifically: Among them, dist t represents the vehicle position at time t and the target position p t goal the distance between them; represents the x-coordinate of the vehicle at time t, represents the x-coordinate of the target point at time t, represents the y coordinate of the vehicle at time t, Represents the y coordinate of the target point at time t; represents the vehicle speed at time t 2. The method for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that: The action space a in S2 t It includes five discrete behaviors: acceleration, deceleration, left lane change, right lane change, and state maintenance; Among them, acceleration is defined as increasing the current speed by 2m / s, deceleration is defined as decreasing the current speed by 2m / s, left lane change is defined as changing to the left lane, right lane change is defined as changing to the right lane, and holding state is defined as driving along the lane at the current speed for a fixed period of time.

3. The method for integrated lateral and longitudinal decision-making for autonomous driving based on deep reinforcement learning according to claim 1, characterized in that: The deep learning model adopts a three-layer stacked LSTM network.

4. An integrated lateral and longitudinal decision-making system for autonomous driving based on deep reinforcement learning, characterized by: An intelligent decision network is pre-trained, wherein the intelligent decision network includes a deep learning model and a reinforcement learning decision model; the system includes: The acquisition module is used to collect the status of the autonomous driving vehicle and its surrounding environment in real time; An acquisition module, configured to use the current state of the autonomous driving vehicle and the surrounding environment as input to the deep learning model to acquire an action space; A selection module is used to select and execute the action with the highest evaluation based on the reinforcement learning decision model with the goal of maximizing the expected reward; The repetition module is used to transfer to the acquisition module, the acquisition module and the selection module and execute them repeatedly until the target point is reached; The autonomous driving vehicle and surrounding environment states include: Among them, s t represents the state of the autonomous driving vehicle and its surrounding environment at time t; Represents the state of the vehicle at time t, including the speed of the vehicle at time t Car position Represents the target state at time t, using the target position p at time t t goal express; represents the status of other vehicles at time t; Indicates other vehicle status; The states of the other vehicles are acquired in real time by sensors and represented by a grid occupancy map; The grid occupied map specifically refers to: A 7×40 binary matrix is ​​used to represent the occupancy status of the 12m×120m area around the vehicle. The grid occupancy map is defined in the Frenet coordinate system, using the variable s to represent the front and rear position of the vehicle along the road, and the variable d to represent the left and right position of the vehicle on the road; Along the s-axis, the area 100m in front of the vehicle and 20m behind the vehicle is used as the sampling area; along the d-axis, the area 6m to the left and right of the vehicle is used as the sampling area; Using the Actor-Critic method to train the reinforcement learning decision model; Among them, the policy network π(a|s;θ) is used to input the state information s at time t t , after the deep learning model operation output action space a t , the reinforcement learning decision model selects the optimal action according to the current decision strategy π The value network q(s,a;ω) is used to evaluate the t Take action in the event of How good or bad is it? Execute an action After the autonomous driving vehicle forms a new state t+1 , and get the feedback reward r from the environment t ; The reinforcement learning decision model is based on the reward function r t , continuously maximize rewards and learn the optimal decision strategy π * ; The reward function r t Specifically: Among them, dist t represents the vehicle position at time t and the target position p t goal the distance between them; represents the x-coordinate of the vehicle at time t, represents the x-coordinate of the target point at time t, represents the y coordinate of the vehicle at time t, Represents the y coordinate of the target point at time t; represents the vehicle speed at time t 5. A storage medium, characterized in that: It stores a computer program for lateral and longitudinal integrated decision-making of autonomous driving based on deep reinforcement learning, wherein the computer program enables the computer to execute the lateral and longitudinal integrated decision-making method of autonomous driving based on deep reinforcement learning as described in any one of claims 1 to 3.

6. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the deep reinforcement learning-based integrated lateral and longitudinal decision-making method for autonomous driving as described in any one of claims 1 to 3.

Images