Training method for deep reinforcement learning of unmanned aerial vehicle

Abstract

The invention relates to the technical field of unmanned aerial vehicle training methods, and discloses an unmanned aerial vehicle deep reinforcement learning training method, which comprises the steps of constructing an imitation learning network, training the imitation learning network, simulating a training process, obtaining a simulation data module, constructing a Faster R-CNN network, and obtaining a flight attitude control module. Building a deep simulation reinforcement learning model through the simulation data module and the flight attitude control module, and training the deep simulation reinforcement learning model; the decision-making ability is improved by performing simulation correction on the same scene for many times, confirming the optimal flight attitude interval and improving the autonomous decision-making selection interval, meanwhile, crystallization obstacle generation is performed on the basis of an initial cylinder in simulation training, the generated obstacle boundary is relatively smooth, and the method is more suitable for simulating obstacles in natural scenes and has high practical value. And the obstacle avoidance flight attitude of the unmanned aerial vehicle can be known more conveniently in analog simulation close to nature.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle training, and in particular to a training method for deep reinforcement learning of unmanned aerial vehicles. Background Art

[0002] With the rapid development and application of drones in many fields, autonomous flight decision-making technology for drones has become one of the research directions that the industry and academia are eager to break through, requiring drones to have autonomous obstacle avoidance, path planning and other functions. Deep reinforcement learning optimizes strategies through the interaction between intelligent agents and the environment. It has a good ability to solve high-dimensional decision-making problems and is widely used in intelligent decision-making and optimization control problems. In the field of autonomous flight decision-making for drones, deep reinforcement learning has achieved good results and can be applied to situations with continuous state space and action space, but the problem of how deep reinforcement learning can balance exploration and utilization still exists.

[0003] The mathematical description structure of the current UAV autonomous decision-making algorithm is simple, the resulting flight trajectory fluctuates greatly, the autonomous decision-making range is wide, which can easily cause the path to be unflyable, and the simulation is not natural enough during simulation training. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a training method for deep reinforcement learning of unmanned aerial vehicles to solve the problems existing in the above-mentioned background technology.

[0005] The present invention provides the following technical solution: a training method for deep reinforcement learning of an unmanned aerial vehicle, characterized in that it includes:

[0006] Build an imitation learning network;

[0007] Training the imitation learning network;

[0008] Simulate the training process to obtain a simulation data module;

[0009] Build the Faster R-CNN network and obtain the flight attitude control module;

[0010] Through the simulation data module and the flight attitude control module, the construction of the deep imitation reinforcement learning model is completed;

[0011] Training the deep imitation reinforcement learning model;

[0012] Furthermore, the imitation learning network inputs a processed image of 64x64 pixels, and outputs quadcopter control information, including predicted steering angle, predicted driving information, and predicted braking information.

[0013] Furthermore, the imitation learning network includes five convolutional layers and four fully connected layers, wherein the convolutional layers are used to extract features, and the fully connected layers are used to predict steering angles, motors, and posture control; the five convolutional layers use 5x5 convolution kernels, and maximum pooling layers and Dropout layers are added to optimize the network; the five convolutional layers and the first three fully connected layers all use the Relu activation function, and the last fully connected layer is the output layer, which includes three fully connected networks, using tanh, sigmoid, and sigmoid activation functions, respectively, corresponding to the output of three actions: steering, acceleration, and braking.

[0014] Furthermore, the imitation learning network is trained, and the human operation data is collected using the simulation driving simulator Prepar3D. The underlying flight logic of the UAV simulation flight is simulated based on the human operation steps, and is continuously optimized through simulation training.

[0015] Furthermore, during the simulation process, obstacles are randomly generated in the simulation scene. The obstacles are divided into several parts from top to bottom. The size and shape of each part gradually change. The number of obstacles gradually increases with the number of simulated flights. Initially, it is a cylinder. Some random points are selected on the cylinder as the starting point. Crystallization grows outward with a specified color at a speed of four or eight templates. The intersection of the crystals is an edge of the Voronoi diagram. The number and position of the selected points change with the increase of the simulation cycle. A simulation cycle is when the current obstacle can be successfully passed by manual operation until the obstacle and the drone reach a safe distance after the deflection angle is modified. Obstacles in real life are simulated in this way. The entire flight process is realized through digital simulation. The variability of obstacles is more suitable for simulating various obstacles in real life.

[0016] Furthermore, a safety distance S1 and a maximum distance S2 are set during the simulated flight. During the simulated flight, obstacles are first generated, and then simulated obstacle avoidance is performed based on human operation. During the simulation, posture correction is performed, and the correction is divided into two parts: inward correction and outward correction. Through the correction, a more efficient and safer flight posture can be achieved.

[0017] Furthermore, the inward correction is performed during several simulated flights, based on the manually operated flight attitude, to correct the deflection angle to the side of the obstacle, through several continuous offsets of the minimum unit, until the distance between the obstacle and the simulated UAV model is less than the safety distance S1, the simulation is stopped, and the current obstacle status, UAV flight attitude and steering angle are recorded.

[0018] Furthermore, the outward correction is performed during several simulated flights, based on the manually operated flight attitude, to correct the deflection angle toward the side of the outward obstacle, through several continuous offsets of the minimum unit, until the distance between the obstacle and the simulated UAV model is equal to or greater than the maximum distance S2, and then the simulation is stopped, and the current obstacle status, UAV flight attitude and steering angle are recorded.

[0019] Furthermore, a UAV maneuver model is constructed, and the OXYZ coordinate system is set as the three-dimensional space coordinate system where the UAV is located, where the origin O represents the center of the UAV activity area, the X axis points to the due north, the Z axis points to the due east, and the Y axis points to the vertical upward direction;

[0020] Considering the drone as a point mass, the modified motion equation of the drone is as follows:

[0021]

[0022] Y t+dT =Y t +V t+dT ×sinθ t+dT ×D

[0023]

[0024] V t+dT =V t +d v ×d T

[0025]

[0026] Where t is the current time, d T is the integration step length of the UAV, [X t ,Y t ,Z t ],[X t +d T ,Y t +d T ,Z t +d T ] are respectively time t and t+d T The coordinate position component of the drone at the moment, V t 、V t +d T They are respectively the UAV at time t and t+d T The velocity at the moment, the pitch angle θ t ,θ t +d T They are respectively the UAV at time t and t+d T The angle between the velocity vector and the XOZ plane at the moment, the heading angle is the UAV at time t and t+d T The angle between the projection vector of the velocity vector on the XOZ plane and the positive direction of the X axis, d v is the acceleration of the drone, d θ is the pitch angle change of the UAV, is the change in the heading angle of the UAV, is the correction amount, i.e. the minimum unit amount of the UAV deflection angle;

[0027] Let (a, b, c) be the position of the point on the obstacle closest to the drone when the drone first detects the obstacle, (a+ε, b+ε, c+ε) be the position of the point on the obstacle closest to the drone when the drone is moving, S represents the distance between the point and the drone, And S2≥S≥S1.

[0028] In each simulation cycle, the UAV model first conducts a preliminary flight based on human operation, and subsequent flights will perform inward and outward correction flights, gradually changing the flight obstacle avoidance angle until the UAV reaches the maximum obstacle avoidance distance or safe distance, and records the flight data of the maximum distance and safe distance. The obstacle shape is then changed to conduct the next cycle of flight simulation training. Through training, it can deeply learn the best obstacle avoidance flight posture, adopt a more reasonable obstacle avoidance method, supplement the autonomous decision-making database through simulation training, and expand the scope of autonomous decision-making options.

[0029] Beneficial effects:

[0030] 1. The training method of deep reinforcement learning of the drone is as follows: during the simulation process, obstacles are randomly generated in the simulation scene. The obstacles are divided into several parts from top to bottom. The size and shape of each part gradually change. The number of obstacles gradually increases with the number of simulated flights. Initially, it is a cylinder. Some random points are selected on the cylinder as the starting point. Crystallization grows outward with a specified color at a speed of four or eight templates. The intersection of the crystals is an edge of the Voronoi diagram. The number and position of the selected points change with the increase of the simulation cycle. The current obstacle can be smoothly passed by manual operation until the obstacle and the drone reach a safe distance after the deflection angle is modified. This is a simulation cycle to simulate obstacles in real life. The entire flight process is realized through digital simulation. The variability of obstacles is more in line with various obstacles in real life.

[0031] 2. The training method of deep reinforcement learning of the UAV is that in each simulation cycle, the UAV model first conducts a preliminary flight based on human operation, and the subsequent flight will perform inward and outward correction flights, gradually changing the flight obstacle avoidance angle until the UAV reaches the maximum obstacle avoidance distance or safe distance, and records the flight data of the maximum distance and safe distance. Then, the obstacle shape is changed and the next cycle of flight simulation training is carried out. Through training, the best obstacle avoidance flight posture can be learned through deep learning, and a more reasonable obstacle avoidance method can be adopted. Through simulation training, the autonomous decision-making database can be supplemented to expand the scope of autonomous decision-making options. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an overall flow chart of a training method for deep reinforcement learning of unmanned aerial vehicles proposed by the present invention;

[0033] Figure 2 This is a training flow chart of a training method for deep reinforcement learning of unmanned aerial vehicles proposed by the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] See also Figure 1-Figure 2 , a training method for deep reinforcement learning of unmanned aerial vehicles, characterized by comprising:

[0037] Build an imitation learning network;

[0038] Train the imitation learning network;

[0039] Simulate the training process to obtain a simulation data module;

[0040] Build the Faster R-CNN network and obtain the flight attitude control module;

[0041] Through the simulation data module and the flight attitude control module, the construction of the deep imitation reinforcement learning model is completed;

[0042] Training deep imitation reinforcement learning models;

[0043] The imitation learning network inputs a processed image of size 64x64 pixels and outputs the control information of the quadcopter, including predicted steering angle, predicted driving information, and predicted braking information.

[0044] The imitation learning network consists of five convolutional layers and four fully connected layers. The convolutional layers are used to extract features, and the fully connected layers are used to predict steering angles, motors, and posture control. The five convolutional layers use 5x5 convolution kernels, and the maximum pooling layer and Dropout layer are added to optimize the network. The five convolutional layers and the first three fully connected layers all use the Relu activation function. The last fully connected layer is the output layer, which includes three fully connected networks, using tanh, sigmoid, and sigmoid activation functions, respectively, corresponding to the output actions of steering, acceleration, and braking.

[0045] Train the imitation learning network, use the simulation driving simulator Prepar3D to collect human operation data, use human operation steps to simulate the underlying flight logic of the UAV flight, and continuously optimize through simulation training.

[0046] During the simulation, obstacles are randomly generated in the simulation scene. The obstacles are divided into several parts from top to bottom. The size and shape of each part gradually change. The number of obstacles gradually increases with the number of simulated flights. Initially, it is a cylinder. Some random points are selected on the cylinder as the starting point. Crystallization grows outward with the specified color at the speed of four or eight templates. The intersection of the crystals is an edge of the Voronoi diagram. The number and position of the selected points change with the increase of the simulation cycle. The current obstacle can be smoothly passed by manual operation until the obstacle and the drone reach a safe distance after the deflection angle is modified. This is a simulation cycle to simulate obstacles in real life. The entire flight process is realized through digital simulation. The variability of obstacles is more suitable for simulating various obstacles in real life.

[0047] Embodiment 2

[0048] See also Figure 1-Figure 2 During the simulated flight, a safety distance S1 and a maximum distance S2 are set. During the simulated flight, obstacles are first generated, and then simulated obstacle avoidance is performed based on human operation. During the simulation, posture correction is performed. The correction is divided into two parts: inward correction and outward correction. Through correction, a more efficient and safer flight posture can be achieved.

[0049] Inward correction is to make attitude correction based on the manually operated flight attitude during several simulated flights, and correct the deflection angle to the side of the obstacle. Through several continuous offsets of the smallest unit, the simulation is stopped until the distance between the obstacle and the simulated UAV model is less than the safety distance S1, and the current obstacle status, UAV flight attitude and steering angle are recorded.

[0050] Outward correction is to perform attitude correction based on the manually operated flight attitude during several simulated flights, and correct the deflection angle to the side of the outward obstacle through several continuous offsets of the smallest unit until the distance between the obstacle and the simulated UAV model is equal to or greater than the maximum distance S2, then stop the simulation and record the current obstacle status, UAV flight attitude and steering angle.

[0051] Construct a UAV maneuver model, and set the OXYZ coordinate system as the three-dimensional space coordinate system where the UAV is located, where the origin O represents the center of the UAV activity area, the X axis points to the due north, the Z axis points to the due east, and the Y axis points vertically upward;

[0052] Considering the drone as a point mass, the modified motion equation of the drone is as follows:

[0053]

[0054] Y t+dT =Y t +V t+dT ×sinθ t+dT ×D

[0055]

[0056] V t+dT =V t +d v ×d T

[0057]

[0058] Where t is the current time, d T is the integration step length of the UAV, [X t ,Y t ,Z t ],[X t +d T ,Y t +d T ,Z t +d T ] are respectively time t and t+d T The coordinate position component of the drone at the moment, V t 、V t +d T They are respectively the UAV at time t and t+d T The velocity at the moment, the pitch angle θ t ,θ t +d T They are respectively the UAV at time t and t+d T The angle between the velocity vector and the XOZ plane at the moment, the heading angle is the UAV at time t and t+dT The angle between the projection vector of the velocity vector on the XOZ plane and the positive direction of the X axis, d v is the acceleration of the drone, d θ is the pitch angle change of the UAV, is the change in the heading angle of the UAV, is the correction amount, i.e. the minimum unit amount of the UAV deflection angle;

[0059] Let (a, b, c) be the position of the point on the obstacle closest to the drone when the drone first detects the obstacle, (a+ε, b+ε, c+ε) be the position of the point on the obstacle closest to the drone when the drone is moving, S represents the distance between the point and the drone, And S2≥S≥S1.

[0060] In each simulation cycle, the UAV model first conducts a preliminary flight based on human operation, and subsequent flights will perform inward and outward correction flights, gradually changing the flight obstacle avoidance angle until the UAV reaches the maximum obstacle avoidance distance or safe distance, and records the flight data of the maximum distance and safe distance. The obstacle shape is then changed to conduct the next cycle of flight simulation training. Through training, it can deeply learn the best obstacle avoidance flight posture, adopt a more reasonable obstacle avoidance method, supplement the autonomous decision-making database through simulation training, and expand the scope of autonomous decision-making options.

[0061] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A training method for deep reinforcement learning of unmanned aerial vehicles, characterized in that: include: Build an imitation learning network; Training the imitation learning network; Simulate the training process to obtain a simulation data module; Build the Faster R-CNN network and obtain the flight attitude control module; Through the simulation data module and the flight attitude control module, the construction of the deep imitation reinforcement learning model is completed; Train the deep imitation reinforcement learning model.

2. The method for training deep reinforcement learning of a drone according to claim 1, characterized in that: The imitation learning network inputs a processed image of 64x64 pixels and outputs control information of the quadcopter, including predicted steering angle, predicted driving information and predicted braking information.

3. The method for training deep reinforcement learning of a drone according to claim 1, characterized in that: The imitation learning network includes five convolutional layers and four fully connected layers, wherein the convolutional layers are used to extract features, and the fully connected layers are used to predict steering angles, motors, and posture control; the five convolutional layers use 5x5 convolution kernels, and a maximum pooling layer and a Dropout layer are added to optimize the network; the five convolutional layers and the first three fully connected layers all use the Relu activation function, and the last fully connected layer is the output layer, which includes three fully connected networks, using tanh, sigmoid, and sigmoid activation functions, respectively, corresponding to the output of three actions: steering, acceleration, and braking.

4. The method for training deep reinforcement learning of a drone according to claim 1, characterized in that: The imitation learning network is trained, and the human operation data is collected using the simulation driving simulator Prepar3D. The underlying flight logic of the UAV simulation flight is simulated based on the human operation steps, and is continuously optimized through simulation training.

5. The method for training deep reinforcement learning of a drone according to claim 4, characterized in that: During the simulation, obstacles are randomly generated in the simulation scene. The obstacles are divided into several parts from top to bottom. The size and shape of each part gradually change, and the number of obstacles gradually increases with the number of simulated flights.

6. The method for training deep reinforcement learning of a drone according to claim 5, characterized in that: During the simulated flight, a safe distance S1 and a maximum distance S2 are set. During the simulated flight, obstacles are first generated, and then simulated obstacle avoidance is performed based on human operation. During the simulation, posture corrections are performed. The corrections are divided into two parts: inward correction and outward correction. Through corrections, a more efficient and safer flight posture can be achieved.

7. The method for training deep reinforcement learning of a drone according to claim 6, characterized in that: The inward correction is to perform attitude correction based on the manually operated flight attitude during several simulated flights, and correct the deflection angle to the side of the obstacle. Through several continuous offsets of the minimum unit, the simulation is stopped until the distance between the obstacle and the simulated drone model is less than the safety distance S1, and the current obstacle status, drone flight attitude and steering angle are recorded.

8. The method for training deep reinforcement learning of a drone according to claim 7, characterized in that: The outward correction is to perform attitude correction based on the manually operated flight attitude during several simulated flights, and correct the deflection angle to the side of the outward obstacle through several continuous offsets of the minimum unit until the distance between the obstacle and the simulated drone model is equal to or greater than the maximum distance S2, then stop the simulation, and record the current obstacle status, drone flight attitude and steering angle.

9. The method for training deep reinforcement learning of a drone according to claim 8, characterized in that: Construct a UAV maneuver model, and set the OXYZ coordinate system as the three-dimensional space coordinate system where the UAV is located, where the origin O represents the center of the UAV activity area, the X axis points to the due north, the Z axis points to the due east, and the Y axis points vertically upward; Considering the drone as a point mass, the modified motion equation of the drone is as follows: AND t+dT =And t +V t+dT ×sinθ t+dT ×dT V t+dT =V t +d v ×d T Where t is the current time, d T is the integration step length of the UAV, [X t ,Y t ,Z t ],[X t +d T ,Y t +d T ,Z t +d T ] are respectively time t and t+d T The coordinate position component of the drone at the moment, V t 、V t +d T They are respectively the UAV at time t and t+d T The velocity at the moment, the pitch angle θ t ,θ t +d T They are respectively the UAV at time t and t+d T The angle between the velocity vector and the XOZ plane at the moment, the heading angle is the UAV at time t and t+d T The angle between the projection vector of the velocity vector on the XOZ plane and the positive direction of the X axis, d v is the acceleration of the drone, d θ is the pitch angle change of the UAV, is the change in the heading angle of the UAV, is the correction amount, i.e. the minimum unit amount of the UAV deflection angle; Let (a, b, c) be the position of the point on the obstacle closest to the drone when the drone first detects the obstacle, (a+ε, b+ε, c+ε) be the position of the point on the obstacle closest to the drone when the drone is moving, S represents the distance between the point and the drone, And S2≥S≥S1.

Images