An efficient learning method for deep reinforcement learning
By building a dual-channel parallel network model and multi-objective reward function, combined with the priority experience replay strategy, the low learning efficiency and convergence difficulties of deep reinforcement learning in large-size map path planning are solved, and the efficiency and stability of unmanned vehicle path planning are achieved.
Patent Information
- Application Number
- CN202211656057.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2025-08-19
- Estimated Expiration
- 2042-12-22
AI Technical Summary
The existing path planning method based on deep reinforcement learning has problems such as low network learning efficiency and convergence difficulties when planning larger-sized maps.
A new dual-channel parallel network model is constructed, including dynamic global channels and dynamic local channels, input global map and local map observation information respectively, and adopt multi-objective reward function and deep reinforcement learning exploration strategy based on priority experience replay to improve the utilization rate of training data through experience buffer pool.
The correlation between observation status and action rewards of deep reinforcement learning methods in three-dimensional terrain has been significantly improved, network learning efficiency and convergence are improved, and the efficiency of unmanned vehicle path planning is promoted.
Smart Images
Figure CN115979287B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned vehicle path planning, and in particular to an efficient learning method for deep reinforcement learning. Background Art
[0002] In recent years, deep reinforcement learning (DRL) has gained widespread development. Research has demonstrated its ability to solve decision-making problems in complex, high-dimensional state spaces. DeepMind's AlphaGo, a resounding success in the game of Go, has propelled DRL research to new heights. DRL has been extensively studied in fields such as video games, robotics, intelligent driving, and recommender systems. Furthermore, DRL has been shown to have significant potential in solving NP-hard problems, such as the Traveling Salesman Problem (TSP).
[0003] Path planning, as a typical NP-hard problem, has attracted widespread attention from scholars. However, DRL-based path planning methods suffer from low network learning efficiency and difficulty in convergence when planning large-scale maps. Summary of the Invention
[0004] In view of this, an embodiment of the present invention provides an efficient learning method for deep reinforcement learning to solve the problems of low network learning efficiency and difficult convergence when planning large-scale maps using the DRL-based path planning method in the existing technology.
[0005] The present invention provides an efficient learning method for deep reinforcement learning, including:
[0006] Input global map observation information into the dynamic global channel;
[0007] Inputting local map observation information into the dynamic local channel;
[0008] The energy consumption and driving time of the autonomous vehicle are evaluated through a multi-objective reward function;
[0009] In each training process, several historical experience data of reaching the target location are extracted from the experience buffer pool as a training set;
[0010] Among them, the dynamic global channel and the dynamic local channel are two parallel input channels of the deep reinforcement learning model; the input layer of the dynamic global channel is set to three-layer image input; the first layer of image input is used to store the location information of the current planning point; the second layer of image input is used to store the target location information; the third layer of image input is used to store three-dimensional map data; the input layer of the dynamic local channel is set to one layer of image input; the part of the rectangle formed by expanding 10 unit steps outward with the current planning point as the center in the original map is the local map observation information; the deep reinforcement learning model adopts a deep reinforcement learning exploration strategy based on priority experience replay.
[0011] Optionally, the convolution backbone layer of the dynamic global channel consists of a first convolution layer and a second convolution layer; a maximum pooling is set between the first convolution layer and the second convolution layer;
[0012] The convolutional backbone layer of the dynamic local channel consists of the third and fourth convolutional layers.
[0013] Optionally, the input size of the dynamic global channel is 100*100*3; the convolution kernel size of the first convolution layer is 8*(3*3*3), and the convolution kernel size of the second convolution layer is 16*(3*3*8); the pooling kernel size of the maximum pooling is 2*2, and the stride is set to 3;
[0014] The input size of the dynamic local channel is 20*20*1; the size of the convolution kernel of the third convolution layer is 4*(3*3*1), and the size of the convolution kernel of the fourth convolution layer is 10*(3*3*4);
[0015] The activation functions of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer all use the ReLu function.
[0016] Optionally, after the output of the convolutional trunk layer of the dynamic global channel and the output of the convolutional trunk layer of the dynamic local channel are spliced through a fully connected layer, a first branch output and a second branch output are derived; the first branch output and the second branch output are linearly connected, and the output node formed is used to represent the Q value of each orientation action;
[0017] Among them, the first layer of the first branch output is set with 128 nodes; the second layer of the first branch output is set with 8 nodes; the first branch output is used to evaluate the action quality in 8 directions; the first layer of the second branch output is set with 128 nodes; the second layer of the second branch output is set with 1 node; the second branch output is used to predict the state value; the activation function of the first layer of the first branch output and the first layer of the second branch adopts the ReLu function; the activation function of the second layer of the first branch output and the second layer of the second branch adopts the linear function.
[0018] Optionally, the current planning point is represented by an image of size 6*6*1; the target position is represented by an image of size 5*5*1.
[0019] Optionally, obtaining the three-dimensional map data includes:
[0020] A rectangular area is formed with the line connecting the current planning point position and the target position as the diagonal line; the map within the rectangular area is centered and each boundary of the map is expanded outward by 10 unit steps; the expanded map is padded with 0s to form a 100*100*1 map image.
[0021] Optionally, the multi-objective reward function includes: an energy consumption reward function, a time consumption reward function, a wall collision reward function and a target position reward function.
[0022] Optionally, the wall-hitting reward function includes:
[0023] The penalty items in the horizontal and vertical directions are calculated according to the intensity factor and sensitivity value of the wall collision penalty.
[0024] Optionally, the deep reinforcement learning exploration strategy based on prioritized experience replay includes:
[0025] Obtain training samples of the unmanned vehicle in a three-dimensional environment; the training samples are state-action sequences;
[0026] The training samples of the unmanned vehicle during the training process are stored in the first experience buffer pool; the training samples that have reached the target position are stored in the second experience buffer pool; wherein the judgment condition for the training sample that has reached the target position is: the reward score of the current training sample is greater than 0;
[0027] When the number of training samples in the first experience buffer pool / the second experience buffer pool reaches the upper limit, the earliest stored training sample data is deleted and the latest training sample data is saved;
[0028] When the number of training samples in the first experience buffer pool reaches a preset value, network training begins.
[0029] Beneficial effects of the embodiments of the present invention:
[0030] This paper proposes an efficient deep reinforcement learning method for path planning in autonomous vehicles. By constructing a novel dual-channel parallel network model consisting of two channels: a dynamic global channel and a dynamic local channel, this method significantly improves the deep reinforcement learning method's ability to focus on three-dimensional terrain from both a global and local perspective, enhancing the correlation between observation state and action reward. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:
[0032] Figure 1 A flowchart of an efficient learning method for deep reinforcement learning in an embodiment of the present invention is shown;
[0033] Figure 2 A schematic diagram of a novel dual-channel parallel network framework for an efficient learning method for deep reinforcement learning according to an embodiment of the present invention is shown;
[0034] Figure 3 A schematic diagram illustrating the input configuration of a novel dual-channel parallel network for an efficient learning method for deep reinforcement learning according to an embodiment of the present invention is shown;
[0035] Figure 4 A schematic diagram of an experience buffer pool for an efficient learning method for deep reinforcement learning according to an embodiment of the present invention is shown;
[0036] Figure 5 A schematic diagram of a training strategy for an efficient learning method for deep reinforcement learning in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0037] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.
[0038] The embodiment of the present invention provides an efficient learning method of deep reinforcement learning, which is applied to the path planning of unmanned vehicles, such as Figure 1 As shown, including:
[0039] Step S10: input global map observation information into the dynamic global channel.
[0040] Step S20: inputting the local map observation information into the dynamic local channel.
[0041] In this example, a novel dual-channel parallel network model is constructed. This network consists of two channels: a dynamic local channel and a dynamic global channel. The inputs to this network are global map observations from the dynamic global channel and local map observations from the dynamic local channel, guiding the deep reinforcement learning method to focus on the three-dimensional terrain.
[0042] In a specific embodiment, the dynamic global channel and the dynamic local channel serve as two parallel input channels for the deep reinforcement learning model. The dynamic global channel's input layer consists of three layers of image input: the first layer stores the current planning point's location information, the second layer stores the target location information, and the third layer stores the 3D map data. The dynamic local channel's input layer consists of a single layer of image input. The local map observation information is captured by a rectangle extending outward by 10 unit steps from the current planning point.
[0043] Step S30: Evaluate the energy consumption and driving time of the unmanned vehicle using a multi-objective reward function.
[0044] In this embodiment, a reward function is designed by comprehensively considering the energy consumption and driving time of the unmanned vehicle in the three-dimensional map. The reward function includes four parts: energy consumption reward, time reward, wall collision reward and target position reward.
[0045] Step S40: In each training process, extracting a number of historical experience data of reaching the target location from the experience buffer pool as a training set.
[0046] In this embodiment, a deep reinforcement learning exploration strategy based on prioritized experience replay is designed, and an experience buffer pool is designed to collect the experience of reaching the target position. In each training process, the parallel network extracts a batch of experience training from the buffer pool, which promotes network learning by improving the utilization of effective data.
[0047] In this example, a novel dual-channel parallel network model is constructed. This model contains two channels: a dynamic global channel and a dynamic local channel. This model significantly improves the deep reinforcement learning method's ability to focus on the three-dimensional terrain from both a global and local map perspective, increasing the correlation between observation state and action reward.
[0048] As an optional implementation, Figure 2 As shown in the figure, the convolutional backbone of the dynamic global channel consists of the first and second convolutional layers; a maximum pooling layer is set between the first and second convolutional layers; the convolutional backbone of the dynamic local channel consists of the third and fourth convolutional layers. Since the role of the dynamic local channel is to provide the network with raw terrain information, it does not have a pooling layer.
[0049] As an optional implementation, the input size of the dynamic global channel is 100*100*3; the convolution kernel size of the first convolution layer is 8*(3*3*3), and the convolution kernel size of the second convolution layer is 16*(3*3*8); the pooling kernel size of the maximum pooling is 2*2, and the step size is set to 3.
[0050] The input size of the dynamic local channel is 20*20*1; the size of the convolution kernel of the third convolution layer is 4*(3*3*1), and the size of the convolution kernel of the fourth convolution layer is 10*(3*3*4).
[0051] The activation functions of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer all use the ReLu function.
[0052] As an optional implementation, after the output of the convolutional trunk layer of the dynamic global channel and the output of the convolutional trunk layer of the dynamic local channel are spliced through the fully connected layer, the first branch output and the second branch output are derived; the first branch output and the second branch output are linearly connected, and the output nodes formed are used to represent the Q value of each direction action. Among them, the first layer of the first branch output is set with 128 nodes; the second layer of the first branch output is set with 8 nodes; the first branch output is used to evaluate the action quality of 8 directions; the first layer of the second branch output is set with 128 nodes; the second layer of the second branch output is set with 1 node; the second branch output is used to predict the state value; the activation function of the first layer of the first branch output and the first layer of the second branch adopts the ReLu function; the activation function of the second layer of the first branch output and the second layer of the second branch adopts the linear function.
[0053] The fully connected layer has three parts: the first part connects the last layers of the two channels, concatenating the outputs of the two channels. This concatenation then leads to two branches, each with two layers. The first branch has 128 nodes in the first layer and 8 nodes in the second layer, used to evaluate the quality of actions in eight directions. The other branch is used to predict state values, with 128 nodes in the first layer and 1 node in the second layer. The activation functions for the first and second layers of the two groups are Reinforced Lu (ReLU) and linear functions, respectively. Finally, the two branches are linearly connected, outputting 8 nodes representing the Q-values of the actions in the eight directions.
[0054] As an optional implementation, the current planning point is represented by an image of 6*6*1 size; the target position is represented by an image of 5*5*1 size.
[0055] As an optional implementation, Figure 3 As shown, the acquisition of three-dimensional map data includes:
[0056] A rectangular area is formed with the line connecting the current planning point position and the target position as the diagonal line.
[0057] Center the map within the rectangular area and expand each border of the map outward by 10 unit steps. Fill the expanded map with zeros to form a 100*100*1 map image.
[0058] The two-layer image (100*100*1) containing the current planning point and the target location is combined with the aforementioned map image to form a 100*100*3 input image. As the current planning point moves, the input image of the global dynamic channel is obtained until the training is completed.
[0059] As an optional implementation, the multi-objective reward function includes: an energy consumption reward function, a time consumption reward function, a wall collision reward function, and a target position reward function. The wall collision reward function includes: calculating penalty terms for the horizontal and vertical axes based on the intensity factor and sensitivity value of the wall collision penalty.
[0060] In this embodiment, the formula for calculating the reward function is as follows:
[0061] R=R e +R T +R w +R t
[0062] Where R represents the total reward of the autonomous vehicle traveling from its current position to the adjacent position.
[0063] R e Represents energy consumption reward, which is calculated as follows:
[0064] R e =-E(p i ,p i+1 )·f e
[0065] Where, f e is the intensity factor of the energy penalty, f e =15, E(p i ,p i+1 ) indicates that the unmanned vehicle moves from its current position p i (x i ,y i ,z i ) Travel to the adjacent position p i+1 (x i+1 ,y i+1 ,z i+1 ) is shown in the following formula:
[0066]
[0067] Where, H3(p i ,p i+1 ) indicates that the unmanned vehicle moves from its current position p i (x i ,y i ,z i ) to the adjacent position p i+1 (x i+1 ,yi+1 ,z i+1 ), which is calculated as follows:
[0068]
[0069] In the formula, (x i ,y i ,z i ) is the three-dimensional coordinate of the i-th planning point in the three-dimensional map.
[0070] In the formula, when z i+1 >z i When the unmanned vehicle is on an uphill slope, the unmanned vehicle overcomes gravity, rolling friction, and wind force to perform work. The calculation formula is:
[0071]
[0072] Where, e us is the ratio of the energy consumption of the unmanned vehicle going uphill to that on a flat road, where subscript f indicates that the unmanned vehicle is driving on a flat road, subscript u indicates that the unmanned vehicle is driving uphill, subscript d indicates that the unmanned vehicle is driving downhill, subscript r indicates the rolling resistance of the tire, subscript g indicates the weight of the unmanned vehicle, and subscript w indicates the weight of the unmanned vehicle.
[0073] E g The energy consumption of the unmanned vehicle to overcome gravity, E g =m·g·sin(β)·s, where β is the position point p in the three-dimensional map i (x i ,y i ,z i ) and position point p i+1 (x i+1 ,y i+1 ,z i+1 The slope between ) is calculated as follows:
[0074]
[0075] E ur Energy consumption for unmanned vehicles to overcome tire rolling friction:
[0076] E ur =m·g·cos(β)·ρ·s
[0077] Where ρ represents the rolling friction coefficient, ρ = 0.01.
[0078] E w To overcome wind resistance:
[0079]
[0080] Where Ca is the air resistance coefficient, ρ a is the air density, S is the frontal area of the unmanned vehicle, and the unmanned vehicle is assumed to be moving at a constant speed, v is the vehicle speed, and s is the distance traveled. a =0.45, S=2.04m 2 , ρ a =1.39kg / m, the mass of the unmanned vehicle m = 1670kg. According to the proportional relationship between the air resistance and mechanical resistance of the unmanned vehicle, assuming the average speed of the unmanned vehicle v = 45km / h, the air resistance is 1 / 4 of the mechanical resistance, that is:
[0081]
[0082] Therefore, the ratio of energy consumption on uphill and flat roads is:
[0083]
[0084] When z i+1 <z i When the unmanned vehicle is traveling downhill, the calculation formula is:
[0085]
[0086] Where, e ds It is the ratio of the uphill energy consumption of the unmanned vehicle to the energy consumption on flat roads.
[0087] R T Represents the time reward. Set it to a negative number. Otherwise, the path planning method based on deep reinforcement learning will tend to plan a path that takes longer time. The time reward is as follows:
[0088] R T =-T(p i ,p i+1 )·f t
[0089] Where, f t is the intensity factor of the time penalty, f t =100, T(p i ,p i+1 ) indicates that the unmanned vehicle moves from its current position p i (x i ,y i ,z i ) Travel to the adjacent position p i+1 (x i+1 ,y i+1 ,z i+1 ) time, T(p i ,p i+1 ) is calculated as follows:
[0090]
[0091] Where, H3(p i ,p i+1 ) indicates that the unmanned vehicle moves from its current position p i (x i ,y i ,z i ) to the adjacent position p i+1 (x i+1 ,y i+1 ,z i+1 ), v e Indicates the speed of the autonomous vehicle per unit step.
[0092] R w Represents the wall collision reward. Set it to a negative number. If you do not set a wall collision penalty, the agent will end the task with a stagnant strategy at the wall, making the path unable to reach the goal and planning failure. The wall collision penalty calculation formula is as follows:
[0093] r w =-2·(r wx +r wy )·γ
[0094] Where γ is the intensity factor of the wall-hitting penalty, γ = 15, representing the penalty in the horizontal and vertical directions, respectively. The calculation formulas for each penalty term are as follows:
[0095]
[0096]
[0097] Where s v Indicates the sensitivity value of the wall penalty, s v =3, l1 and l2 are the specific locations where the wall penalty starts, p x and p y is the position of the unmanned vehicle, p x is the horizontal axis, p y Is the vertical axis.
[0098] R t Represents the reward for the autonomous vehicle to reach the target location. When the autonomous vehicle reaches the target location, the DRL will receive a larger arrival reward. The arrival reward is set to a positive number to encourage the DRL to remember the experience of reaching the target location. t =195, this parameter is related to the size of the map.
[0099] like Figure 4 and Figure 5 As shown in the figure, the data collected by the unmanned vehicle in the three-dimensional environment is a state-action sequence, that is, a four-tuple (s t ,at ,r t ,s t+1 ), this data has a high degree of contextual correlation. Using these state-action sequences for training makes neural networks difficult to converge and unstable. Storing the collected data in a buffer pool and extracting data from the buffer pool for training using uniform random sampling overcomes the problems of contextual correlation and uneven distribution of sample data. Because samples are reused multiple times, data utilization is high. Specifically, deep reinforcement learning exploration strategies based on prioritized experience replay include:
[0100] S1, obtain training samples of the unmanned vehicle in a three-dimensional environment; the training samples are state-action sequences.
[0101] S2, storing the training samples of the unmanned vehicle during the training process in the first experience buffer pool; storing the training samples that have reached the target position in the second experience buffer pool; wherein, the judgment condition for the training samples that have reached the target position is: the reward score of the current training sample is greater than 0.
[0102] In this embodiment, the training samples of the unmanned vehicle during the training process are stored in the experience buffer pool memory, and the training samples that have reached the target position are stored in another buffer pool memory_d. The judgment condition for the sample reaching the target position is:
[0103] scort>0
[0104] Where scort is the reward of the autonomous vehicle in a training round. From the aforementioned reward function, we can see that the energy consumption reward, time reward, and wall collision reward are all negative. Only when the autonomous vehicle reaches the target position will the reward be positive.
[0105] S3: When the number of training samples in the first experience buffer pool / the second experience buffer pool reaches an upper limit, the earliest stored training sample data is deleted and the latest training sample data is saved.
[0106] Set the size of the two experience buffer pools to 10,000. When the number of samples in the experience pool reaches 10,000, delete the oldest sample and save the new one. As training progresses, the autonomous vehicle's planning strategy will become better and better, and the samples generated will have higher rewards.
[0107] S4: When the number of training samples in the first experience buffer pool reaches a preset value, network training begins.
[0108] When the number of samples in the buffer pool memory is equal to 500, the network training begins, and the sample batch size used for each training is N batch =256, while ensuring that there is a 50% probability of selecting samples that reach the target position to improve the utilization of effective experience, that is:
[0109] N d =0.5N batch
[0110] Where N d Represents samples that reach the target position.
[0111] In order to solve the problems of low network learning efficiency and difficult convergence when planning large-scale maps in existing DRL-based path planning methods, the present invention provides an efficient learning method for deep reinforcement learning for path planning. By constructing a new dual-channel parallel network model, the new dual-channel parallel network contains two channels, namely a dynamic global channel and a dynamic local channel. From the two perspectives of global map and local map, the deep reinforcement learning method is significantly improved in focusing on three-dimensional terrain, the correlation between observation state and action reward is improved, and efficient network learning is promoted. The method is also used for path planning of unmanned vehicles in three-dimensional terrain, providing new impetus for the development of this industry.
[0112] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.
Claims
1. An efficient deep reinforcement learning method for unmanned vehicle path planning in a three-dimensional large-scale terrain environment, characterized by: include: Input global map observation information into the dynamic global channel; Inputting local map observation information into the dynamic local channel; The energy consumption and driving time of the autonomous vehicle are evaluated through a multi-objective reward function; In each training process, several historical experience data of reaching the target location are extracted from the experience buffer pool as a training set; Among them, the dynamic global channel and the dynamic local channel are two parallel input channels of the deep reinforcement learning model; the input layer of the dynamic global channel is set to three-layer image input; the first layer of image input is used to store the position information of the current planning point; the second layer of image input is used to store the target position information; the third layer of image input is used to store three-dimensional map data; the input layer of the dynamic local channel is set to one layer of image input; the part of the rectangle formed by expanding outward by 10 unit steps with the current planning point as the center in the original map is the local map observation information; the deep reinforcement learning model adopts a deep reinforcement learning exploration strategy based on priority experience replay.
2. The efficient learning method for deep reinforcement learning according to claim 1, characterized in that The convolutional backbone layer of the dynamic global channel consists of a first convolutional layer and a second convolutional layer; a maximum pooling is set between the first convolutional layer and the second convolutional layer; The convolutional backbone layer of the dynamic local channel consists of a third convolutional layer and a fourth convolutional layer.
3. The efficient learning method for deep reinforcement learning according to claim 2, characterized in that The input size of the dynamic global channel is 100*100*3; the convolution kernel size of the first convolution layer is 8*(3*3*3), and the convolution kernel size of the second convolution layer is 16*(3*3*8); the pooling kernel size of the maximum pooling is 2*2, and the stride is set to 3; The input size of the dynamic local channel is 20*20*1; the size of the convolution kernel of the third convolution layer is 4*(3*3*1), and the size of the convolution kernel of the fourth convolution layer is 10*(3*3*4); The activation functions of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer all adopt ReLu functions.
4. The efficient learning method for deep reinforcement learning according to claim 3, characterized in that After the output of the convolutional trunk layer of the dynamic global channel and the output of the convolutional trunk layer of the dynamic local channel are spliced through a fully connected layer, a first branch output and a second branch output are derived; the first branch output and the second branch output are linearly connected to form an output node for representing the Q value of each azimuth action; Among them, the first layer of the first branch output is set with 128 nodes; the second layer of the first branch output is set with 8 nodes; the first branch output is used to evaluate the action quality of 8 directions; the first layer of the second branch output is set with 128 nodes; the second layer of the second branch output is set with 1 node; the second branch output is used to predict the state value; the activation function of the first layer of the first branch output and the first layer of the second branch adopts the ReLu function; the activation function of the second layer of the first branch output and the second layer of the second branch adopts the linear function.
5. The efficient learning method for deep reinforcement learning according to claim 1, characterized in that The current planning point is represented by an image of size 6*6*1; the target position is represented by an image of size 5*5*1.
6. The efficient learning method for deep reinforcement learning according to claim 1, characterized in that The acquisition of the three-dimensional map data includes: A rectangular area is formed with the line connecting the current planning point position and the target position as the diagonal line; the map within the rectangular area is centered, and each boundary of the map is expanded outward by 10 unit steps; the expanded map is padded with zeros to form a 100*100*1 map image.
7. The efficient learning method for deep reinforcement learning according to claim 1, characterized in that The multi-objective reward function includes: an energy consumption reward function, a time consumption reward function, a wall collision reward function and a target position reward function.
8. The efficient learning method for deep reinforcement learning according to claim 7, characterized in that The wall-hitting reward function includes: The penalty items in the horizontal and vertical directions are calculated according to the intensity factor and sensitivity value of the wall collision penalty.
9. The efficient learning method for deep reinforcement learning according to claim 1, characterized in that Deep reinforcement learning exploration strategies based on prioritized experience replay include: Obtaining a training sample of the unmanned vehicle in a three-dimensional environment; the training sample is a state-action sequence; The training samples of the unmanned vehicle during the training process are stored in a first experience buffer pool; the training samples that have reached the target position are stored in a second experience buffer pool; wherein the judgment condition for the training sample that has reached the target position is: the reward score of the current training sample is greater than 0; When the number of training samples in the first experience buffer pool / the second experience buffer pool reaches an upper limit, deleting the earliest stored training sample data and saving the latest training sample data; When the number of training samples in the first experience buffer pool reaches a preset value, network training begins.