UAV control method, device, equipment and medium based on deep reinforcement learning

Abstract

The present application discloses a method, device, equipment and medium for controlling a UAV based on deep reinforcement learning, and relates to the field of UAV flight control. The method for controlling a UAV based on deep reinforcement learning includes: constructing a six-degree-of-freedom dynamic equation for a long-flight UAV; training a deep reinforcement learning model using a proximal policy optimization algorithm according to the six-degree-of-freedom kinematic equation to obtain a trained deep reinforcement learning model; predicting a control strategy based on a policy network in the trained deep reinforcement learning model; and controlling a long-flight UAV based on the predicted control strategy. The present application improves the stability of low-altitude, long-flight UAVs in complex flight environments.

Description

Technical Field

[0001] The present application relates to the field of drone flight control, and in particular to a drone control method, device, equipment and medium based on deep reinforcement learning. Background Art

[0002] As an emerging aviation platform, low-altitude and long-flight UAVs have the characteristics of long-term flight and are suitable for performing continuous surveillance, reconnaissance and communication tasks. They have become a hot topic in research and application.

[0003] In order to increase the flight time, low-altitude and long-flight UAVs usually have the characteristics of large aspect ratio, light structure weight, and low structural stiffness, which brings significant dynamic nonlinearity. Existing flight control methods mainly rely on linear models, which cannot effectively deal with the impact of nonlinear aerodynamic forces and torques on flight control. For example, although the traditional PID controller is simple and effective, it performs poorly when faced with nonlinear dynamics and external interference, and it is difficult to ensure flight stability and control accuracy.

[0004] In recent years, Deep Reinforcement Learning (DRL), as an emerging control method, has been gradually applied to the field of UAV flight control due to its advantages in dealing with nonlinear systems and high-dimensional state spaces. However, existing research still has the problem of unstable control effect when applying DRL to the flight control of low-altitude and long-endurance UAVs. Summary of the invention

[0005] The purpose of this application is to provide a UAV control method, device, equipment and medium based on deep reinforcement learning, which can improve the stability of low-altitude and long-flight UAVs in complex flight environments.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In the first aspect, the present application provides a UAV control method based on deep reinforcement learning, and the UAV control method based on deep reinforcement learning includes: constructing a six-degree-of-freedom dynamic equation for a long-flight UAV; the six-degree-of-freedom dynamic equation includes nonlinear aerodynamic characteristics and torque characteristics; according to the six-degree-of-freedom kinematic equation, a proximal policy optimization algorithm is used to train a deep reinforcement learning model to obtain a trained deep reinforcement learning model; based on the policy network in the trained deep reinforcement learning model, a control strategy is predicted; and the long-flight UAV is controlled based on the predicted control strategy.

[0008] In the second aspect, a UAV control device based on deep reinforcement learning is provided, which applies the above-mentioned UAV control method based on deep reinforcement learning, and the UAV control device based on deep reinforcement learning includes: a dynamic equation construction module, which is used to construct a six-degree-of-freedom dynamic equation of a long-flight UAV; the six-degree-of-freedom dynamic equation contains nonlinear aerodynamic characteristics and torque characteristics; a model training module, which is used to train a deep reinforcement learning model according to the six-degree-of-freedom kinematic equation using a proximal strategy optimization algorithm to obtain a trained deep reinforcement learning model; a prediction module, which is used to predict the control strategy based on the policy network in the trained deep reinforcement learning model; and a control module, which is used to control the long-flight UAV based on the predicted control strategy.

[0009] In a third aspect, a computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned drone control method based on deep reinforcement learning.

[0010] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned drone control method based on deep reinforcement learning.

[0011] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0012] This application provides a method, device, equipment and medium for controlling a drone based on deep reinforcement learning. By establishing a six-degree-of-freedom dynamics model including nonlinear aerodynamic and torque characteristics, and adopting a deep reinforcement learning control method based on a proximal strategy optimization algorithm, the airspeed and attitude of the drone can be precisely controlled, solving the problem that low-altitude and long-flight drones are difficult to maintain a stable flight attitude in complex flight environments due to the nonlinear characteristics of the dynamics model and external interference, and improving the stability of low-altitude and long-flight drones in complex flight environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A flowchart of a drone control method based on deep reinforcement learning is provided in one embodiment of the present application.

[0015] Figure 2 A schematic diagram of a drone control method based on deep reinforcement learning provided in one embodiment of the present application.

[0016] Figure 3 A framework diagram of a deep reinforcement learning control algorithm provided in one embodiment of the present application.

[0017] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In an exemplary embodiment, Figure 1 As shown, a drone control method based on deep reinforcement learning is provided, including the following steps 101 to 104.

[0021] Step 101, constructing a six-degree-of-freedom dynamics equation for a long-flight UAV; the six-degree-of-freedom dynamics equation includes nonlinear aerodynamic characteristics and torque characteristics.

[0022] Step 102: According to the six-degree-of-freedom kinematic equation, a proximal strategy optimization algorithm is used to train the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0023] Step 103: predict the control strategy based on the policy network in the trained deep reinforcement learning model.

[0024] Step 104: Control the long-flight UAV based on the predicted control strategy.

[0025] Implementing the above steps 101 to 104 can solve the control problem of low-altitude and long-flight UAVs being difficult to maintain a stable flight attitude in complex flight environments due to the nonlinear characteristics of the dynamics model and external interference, thereby improving the stability of low-altitude and long-flight UAVs in complex flight environments.

[0026] like Figure 2As shown, in another exemplary embodiment of the present application, in order to realize the construction of the six-degree-of-freedom dynamic equation, the above step 101 includes: the process of dynamic modeling and simulation environment construction, and the six-degree-of-freedom nonlinear model can be obtained in the dynamic modeling process.

[0027] 1.1. Dynamic modeling.

[0028] First, based on the Newton-Euler dynamics equation, the six-degree-of-freedom dynamics equation of the long-flight UAV is established. Based on actual needs, the equation is established based on the following assumptions.

[0029] (1) Modeling The long-flight UAV is a rigid body with a mass of m.

[0030] (2) When establishing the ground coordinate system, the Earth’s rotation, revolution and the curvature of the Earth’s surface are ignored.

[0031] The reference coordinate system adopted is defined as follows.

[0032] Ground coordinate system (referred to as ground system) Using the definition of Northeast Earth (NED), the coordinate origin Located at any fixed point on the ground, the axis Pointing east in the horizontal plane; axis Perpendicular to the ground plane, pointing downward; axis Also in the ground plane and perpendicular to Flat surface, pointing north.

[0033] Airflow coordinate system (referred to as airflow system) The origin of the coordinates Coincident with the center of mass of the long-flight UAV, axis The positive direction is the same as the airspeed direction of the long-flight UAV; axis and Vertical, pointing toward the ground in the plane of symmetry of the LEF drone; axis Perpendicular to The plane points to the right.

[0034] Aircraft coordinate system (abbreviated as aircraft system) The origin of the coordinates Coincident with the center of mass of the long-flight UAV, pointing to the axis of the nose Parallel to the fuselage axis and located in the symmetry plane of the long-flight UAV; axis and Vertical, pointing toward the ground in the plane of symmetry of the LEF drone; axis Perpendicular to Flat surface, pointing to the right.

[0035] There are 12 variables describing the status of long-flight drones, which are defined as: .

[0036] in, The three-axis components representing the position of the long-flight UAV in the ground coordinate system; It is the three-axis attitude angle of the long-flight UAV in the ground coordinate system, defined by Euler angles; It is the three-axis component of the velocity of the long-flight UAV in the body coordinate system; It is the three-axis component of the angular velocity of the long-flight UAV in the body coordinate system.

[0037] There are 4 control inputs for long-flight UAVs, which are specifically defined as: .

[0038] in, is distributed in The throttle input between , , They are distributed in Elevator, aileron and rudder inputs between.

[0039] Since the velocity component of the long-flight UAV is defined in the body coordinate system and the center of mass position is defined in the ground coordinate system, the center of mass motion equation can be obtained based on the transformation matrix from the ground coordinate system to the body coordinate system: The transformation matrix and the center of mass motion equation are shown below.

[0040]

[0041] .

[0042] .

[0043] in, Indicates the position of the long-flight UAV in the ground coordinate system The derivative of , , are the three-axis components of the position of the long-flight UAV in the ground coordinate system, is the three-axis component of the velocity of the long-flight UAV in the body coordinate system, It is the transformation matrix from the ground coordinate system to the body coordinate system.

[0044] The angular velocity of the long-flight UAV is defined in the body coordinate system, while the attitude angle is defined in the ground coordinate system. According to the Euler angle conversion matrix of the two coordinate systems, the angular motion equation of the long-flight UAV can be obtained as follows.

[0045] .

[0046] in, is the three-axis attitude angle of the long-flight UAV in the ground coordinate system The derivative of are the three-axis components of the angular velocity of the long-flight UAV in the body coordinate system.

[0047] The forces acting on long-flight UAVs include gravity, aerodynamic force and thrust, and the torques include aerodynamic torque and thrust torque. Among them, aerodynamic force includes lift ,resistance , lateral force , aerodynamic moment includes rolling moment , pitch moment and yaw moment ,Right now, ,in, is the aerodynamic moment in the body coordinate system.

[0048] In order to obtain the dynamic equations of the long-flight UAV, it is necessary to project the forces and moments into the body coordinate system, as follows.

[0049] Project gravity from the ground system to the host system: ,in, is the gravity projected to the body coordinate system, is the acceleration due to gravity.

[0050] Project the aerodynamic force from the airflow system to the aircraft system to obtain the aerodynamic force projected to the aircraft coordinate system As shown in the following formula.

[0051] .

[0052] in, is the angle of attack, is the sideslip angle.

[0053] Aerodynamic torque, thrust and thrust moment are defined in the context of the machine system.

[0054] Therefore, the center of mass dynamic equation of the long-flight UAV in the body coordinate system is as follows.

[0055] .

[0056] in, is the three-axis component of the velocity of the long-flight UAV in the body coordinate system The derivative of is the three-axis component of the angular velocity of the long-flight UAV in the body coordinate system The derivative of is the gravity projected to the body coordinate system, is the aerodynamic force projected to the body coordinate system, is the thrust in the body coordinate system, and m is the mass of the long-flight UAV.

[0057] The angular dynamics equation of the long-flight UAV can be obtained in the body coordinate system. The angular dynamics equation of the long-flight UAV is shown as follows.

[0058] .

[0059] in, is the inertial matrix of the long-flight UAV, is the aerodynamic moment in the body coordinate system; due to the structure of the long-flight UAV The plane has symmetry, so the body's inertia and is always 0, and the inertia matrix is ​​as follows.

[0060] .

[0061] The present application uses dimensionless aerodynamic coefficients to calculate aerodynamic forces and aerodynamic moments as follows.

[0062] .

[0063] .

[0064] .

[0065] .

[0066] .

[0067] .

[0068] .

[0069] in, is the dynamic pressure, is the airspeed, S is the wing area, is the air density, For wingspan, is the average aerodynamic chord length, is the lift coefficient, is the lateral force coefficient, is the drag coefficient, is the rolling moment coefficient, is the pitching moment coefficient, is the yaw moment coefficient.

[0070] In order to model the lift and drag at high angles of attack, the nonlinear Newtonian plate theory at high angles of attack and the linear lift model at small angles of attack are combined, thereby extending the calculation range of lift and drag to a range greater than the stall angle of attack.

[0071] The lift coefficient is considered as a function of the angle of attack, and the linear expression at small angles of attack is as follows.

[0072] .

[0073] in, is the lift coefficient when the angle of attack is not greater than the stall angle of attack, is the coefficient of the first constant term, is the coefficient of the first linear term, is the angle of attack of the long-flight UAV.

[0074] The Newtonian plate model used for the lift coefficient at high angles of attack beyond the stall angle of attack is as follows.

[0075] .

[0076] in, is the lift coefficient when the angle of attack is greater than the stall angle of attack, is a symbolic function.

[0077] A sigmoid function is used to weight the linear segment and the nonlinear segment to achieve a smooth transition of the lift coefficient from the linear segment to the nonlinear segment at the stall angle of attack. The expression of the nonlinear lift coefficient is as follows.

[0078] .

[0079] in, is the lift coefficient, is the sigmoid function, and and stall angle of attack The relationship is as follows.

[0080] .

[0081] in, is the coefficient used to control the curve shape of the sigmoid function.

[0082] The drag coefficient at small angles of attack is modeled as a quadratic function of the angle of attack as follows.

[0083] .

[0084] in, is the drag coefficient when the angle of attack is not greater than the stall angle of attack, is the second constant coefficient, i.e. the constant coefficient of the drag coefficient of the long-flight UAV, is the second first-order coefficient, that is, the first-order component coefficient of the derivative of the angle of attack of the long-flight UAV drag coefficient. is the coefficient of the quadratic term, that is, the coefficient of the quadratic component of the constant coefficient of the drag coefficient of the long-flight UAV with respect to the derivative of the angle of attack. The drag coefficient is expanded using the same method as that used to expand the angle of attack, and the drag expression of the Newtonian plate model used is as follows.

[0085] .

[0086] in, is the drag coefficient when the angle of attack is greater than the stall angle of attack.

[0087] The expanded drag coefficient as a function of the angle of attack is as follows.

[0088] .

[0089] in, for Drag coefficient.

[0090] 1.2. Build a simulation environment, that is, build a deep reinforcement learning simulation environment for low-altitude and long-flight UAVs.

[0091] First, the flight dynamics part is written. This part implements the wind model and the six-degree-of-freedom model of the long-flight UAV based on the six-degree-of-freedom dynamics model derived in the first step. The wind model includes the simulation of the steady wind field in the northeast coordinate system and the Dryden turbulence disturbance in the body coordinate system, and is combined into the final six-degree-of-freedom nonlinear dynamics model of the low-altitude long-flight UAV.

[0092] Then the post-processing part is written, which implements the functional interfaces such as step(), reset(), seed(), and render() for interacting with the dynamics model and the deep reinforcement learning controller. reset() is used to reset the state of the simulation environment to the initial state. seed() is used to set the random seed to ensure reproducibility. render() is used to render the simulation results and calculate performance indicators for evaluating the control effectiveness. The corresponding performance indicators are calculated only when the controller controls the drone to achieve the set control goals in a simulation. In step(), the interaction between the deep reinforcement learning controller and the environment is implemented, that is, the environment gives a state, the deep reinforcement learning controller generates an action based on the state, and the environment calls the flight dynamics part to simulate one step based on the action, obtains the state at the next moment, applies data enhancement to the state, and calculates the reward, such as Figure 3 shown.

[0093] In another exemplary embodiment of the present application, Figure 2As shown, in order to realize the training of the deep reinforcement learning model, the above step 102 includes the process of control algorithm design and control algorithm training based on PPO (Proximal Policy Optimization).

[0094] 2.1. Design of control algorithm based on PPO.

[0095] In this simulation environment, the state space and action space are designed as follows.

[0096] For low-altitude, long-endurance UAV airspeed and attitude control tasks, the sensor data that the deep reinforcement learning controller can obtain and is relevant to the control task is a subset of the state variables used for dynamic modeling: .

[0097] In order to drive the intelligent agent to control the drone to the target state, the difference between the target value and the current value of the controlled quantity is added to the state space: .

[0098] in, , , Represents the difference between the current roll angle, pitch angle and airspeed and the target roll angle, pitch angle and airspeed respectively.

[0099] Taking into account the timing delay of sensors in actual tasks, the target value part of the state space adds an integral term related to the previous moment: .

[0100] in, Represents the time step The difference between the target value and the current value of the controlled quantity, is the attenuation coefficient, is the time step The target value part of the state space, is the time step Error in state space.

[0101] The action space is the set of actions that the controller takes in a given environment that conforms to the environment. . Normalize the action space to Between, expressed as: , , i=1, 2, 3, 4, when i is equal to 1, express , when i is equal to 2, express , when i is equal to 3, express , when i is equal to 4, express .

[0102] Finally, the sensor data, target value and the action at the previous moment are spliced ​​to obtain the total state space for: .

[0103] In this simulation environment, the data enhancement measures used during training are designed as follows.

[0104] In order to maximize the simulation of the flight state of long-flight UAVs under real conditions and avoid overfitting of the deep reinforcement learning algorithm to the training environment, this application adopts composite data enhancement measures, including observation noise, dynamic model parameter errors, and rudder assembly errors.

[0105] The sensor observation noise follows the Ornstein-Uhlenbeck process, defined as: .

[0106] in, is the process value at time step t, is the long-term mean of the noise regression, is the regression rate of the noise, is the volatility of the process, It is a Brownian motion process.

[0107] The model parameter error is defined as: .

[0108] in, is the original dynamic parameter of the long-flight UAV, is the dynamic parameter of the domain random initialization sampling, is the crop factor.

[0109] The rudder surface assembly error is defined as: .

[0110] in, Represents the zero point of the rudder surface, is the rudder surface zero point offset.

[0111] 2.2. Control algorithm training.

[0112] The Proximal Policy Optimization (PPO) algorithm is used to train the deep reinforcement learning controller, and the generalized advantage estimation (GAE) is combined to improve the training efficiency. The Actor-Critic neural network structure used by the PPO algorithm is a combination of a one-dimensional convolutional layer and a fully connected layer. The algorithm framework used is as follows Figure 3 shown.

[0113] The PPO algorithm formula used for deep reinforcement learning controller training is as follows.

[0114] .

[0115] .

[0116] .

[0117] .

[0118] .

[0119] .

[0120] .

[0121] .

[0122] in, is the total loss function, is the clipping objective function, is the value function loss, is the entropy reward, and are the parameters of the first policy network and the value function network respectively, , are weight coefficients, is the state at time step t, The parameters are The first strategy network; For expectations, is the generalized advantage function, For reward, To limit The cut-off coefficient, The parameters used are The value function of the network generates the value at time step t, is the target value, For the shear operation, is the number of steps accumulated from the current time step t, which is used to gradually accumulate the TD error of future time steps. is the discount factor, To balance the weight of variance and bias, The parameters used are The value function network generates the value at time step t+1, The parameters used are The action at time step t+1 generated by the first policy network is The parameters used are The action at time step t+1 generated by the second policy network is , Respectively about , ' gradient.

[0123] In this application, the Critic neural network is used to implement ; Using probability ratio; Actor neural network is used in this application and .

[0124] The Actor-Critic neural network structure used in the deep reinforcement learning controller of this application is a combination of a one-dimensional convolutional neural network (Conv1D) and a fully connected layer (MLP). Network input vector ,yes The observation vector of time steps in the time series The MLP network model is: .

[0125] in, is the output vector, ReLU is the activation function, is the network weight vector, is the bias vector. The Conv1D network model is: .

[0126] in, It is The output feature map is at time step The value of It is 'The input feature map is in The convolution kernel weights, It is The input feature map is at time step The value of For the The bias of the convolution kernel, is the convolution kernel size, where the convolution kernel and the output feature Figure 1 One to one correspondence.

[0127] The training process is: first initialize the strategy parameters and , and then each Actor network uses the current strategy Run T time steps in the environment and collect the state at each time step ,action ,award and the next state , and then calculate the generalized advantage function , and then calculate the shear objective function , value function loss and Entropy Reward To get the total loss , and finally update the network parameters , and Repeat the above steps to iterate the training until convergence, and obtain the airspeed and attitude deep reinforcement learning controller of low-altitude long-flight UAV.

[0128] In another exemplary embodiment of the present application, in order to realize the control of the UAV, the above steps 103 and 104 specifically include: after the low-altitude long-flight UAV takes off, the current airspeed and attitude data are first obtained through the airborne sensor to initialize the state space. After ensuring that the sensor data is stable, the deep reinforcement learning controller is started to start real-time control of the airspeed and attitude of the UAV. The controller receives the state information fed back by the airborne sensor in real time, including the current airspeed, pitch angle, roll angle, etc., and updates the state space. According to the updated state space, the controller calculates the optimal control action for the next step and outputs the corresponding control surface and throttle control signal. The actuator adjusts the aileron, elevator, throttle and rudder according to the control signal, and controls the attitude of the UAV in real time until the UAV reaches the set airspeed and attitude target range, so as to achieve stable low-altitude long-flight flight control. When encountering interference such as atmospheric turbulence, wind field and observation delay, the controller maintains effective control of the UAV to ensure flight stability.

[0129] like Figure 2 As shown, in order to verify the effect of the control method of the present application, the present application also sets up a control algorithm simulation test process, specifically: testing the trained deep reinforcement learning controller in a simulation environment, and performing a variety of tests to verify the effectiveness of the controller, including different interference conditions, observation delays, and multi-target value continuous tracking tasks. Use render() to view the performance indicators to obtain the control efficiency evaluation results of the deep reinforcement learning controller. For example, by comparing the control efficiency of the deep reinforcement learning controller and the PID controller under turbulent disturbances, the following conclusions can be drawn: the trajectory of the PID controller is relatively sparse, and there are many obvious divergences, while the trajectory of the DRL controller tracking the target is relatively concentrated, and there is no control divergence, which is more conducive to the stable flight of low-altitude and long-endurance UAVs under interference. It proves that the control method of the present application can maintain the stable flight and efficient task execution of the UAV in a complex flight environment.

[0130] The control method proposed in this application has the following beneficial effects.

[0131] Precise control: By establishing a six-degree-of-freedom dynamic model that includes nonlinear aerodynamic and torque characteristics, the flight dynamics characteristics of the UAV can be simulated more accurately, achieving precise control of airspeed and attitude.

[0132] Efficient training: The deep reinforcement learning method based on the proximal policy optimization (PPO) algorithm is combined with the generalized advantage estimation (GAE) technology to significantly improve the efficiency and effect of controller training.

[0133] Strong generalization capability: By performing data augmentation and domain randomization in a simulation environment, the generalization capability of the controller is improved, and it can maintain stable control performance under different flight conditions and interference environments.

[0134] High robustness: A variety of reward functions are designed, combined with the action oscillation reward function, to encourage the agent to take smooth control actions, improve the robustness of the controller, and maintain good control effects under conditions such as observation noise and sensor delay.

[0135] Strong adaptability: The effectiveness of the controller under various interference conditions, including atmospheric turbulence, wind fields, and observation delays, was verified in simulation experiments, demonstrating its potential for application in actual flight missions.

[0136] In summary, the control method of the present application can achieve stable flight and efficient mission execution of UAVs in complex flight environments, and provides an efficient and reliable solution for the flight control of long-flight UAVs.

[0137] In an exemplary embodiment, a drone control device based on deep reinforcement learning is provided, comprising:

[0138] The dynamic equation construction module is used to construct the six-degree-of-freedom dynamic equation of the long-flight UAV; the six-degree-of-freedom dynamic equation includes nonlinear aerodynamic characteristics and torque characteristics.

[0139] The model training module is used to train the deep reinforcement learning model according to the six-degree-of-freedom kinematic equation using a proximal strategy optimization algorithm to obtain a trained deep reinforcement learning model.

[0140] The prediction module is used to predict the control strategy based on the policy network in the trained deep reinforcement learning model.

[0141] The control module is used to control the long-flight UAV based on the predicted control strategy.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data determined by complementarity. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a drone control method based on deep reinforcement learning is implemented.

[0143] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned embodiment of the drone control method based on deep reinforcement learning are implemented.

[0145] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the steps in the above-mentioned drone control method embodiment based on deep reinforcement learning.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0147] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, etc., but is not limited thereto.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A drone control method based on deep reinforcement learning, characterized in that: The UAV control method based on deep reinforcement learning includes: Constructing a six-degree-of-freedom dynamics equation for a long-flight UAV; the six-degree-of-freedom dynamics equation includes nonlinear aerodynamic characteristics and torque characteristics; According to the six-degree-of-freedom kinematic equation, a proximal strategy optimization algorithm is used to train a deep reinforcement learning model to obtain a trained deep reinforcement learning model; Predict control strategies based on the policy network in the trained deep reinforcement learning model; Control the long-flight UAV based on the predicted control strategy; The six-degree-of-freedom dynamic equations include: center-of-mass motion equations, angular motion equations, center-of-mass dynamic equations and angular dynamic equations; The center of mass motion equation is: ； in, Indicates the position of the long-flight UAV in the ground coordinate system The derivative of , , are the three-axis components of the position of the long-flight UAV in the ground coordinate system, is the three-axis component of the velocity of the long-flight UAV in the body coordinate system, is the transformation matrix from the ground coordinate system to the body coordinate system, the coordinate origin of the body coordinate system Coincident with the center of mass of the long-flight UAV, the x-axis of the body coordinate system The z-axis of the body coordinate system is directed toward the nose of the long-flight UAV and parallel to the axis of the fuselage of the long-flight UAV. The y-axis of the body coordinate system is pointed to the ground and perpendicular to the x-axis in the symmetry plane of the long-flight UAV. Perpendicular to Flat and pointing to the right; The angular motion equation is: ； in, is the three-axis attitude angle of the long-flight UAV in the ground coordinate system The derivative of are the three-axis components of the angular velocity of the long-flight UAV in the body coordinate system; The center-of-mass dynamic equation is: ； in, is the three-axis component of the velocity of the long-flight UAV in the body coordinate system The derivative of is the three-axis component of the angular velocity of the long-flight UAV in the body coordinate system The derivative of is the gravity projected to the body coordinate system, is the aerodynamic force projected to the body coordinate system, is the thrust in the body coordinate system, m is the mass of the long-flight UAV; The angular dynamics equation is: ； in, is the inertial matrix of the long-flight UAV, is the aerodynamic moment in the body coordinate system; Aerodynamic force projected to the body coordinate system and the aerodynamic moment in the body coordinate system Obtained by using dimensionless aerodynamic coefficients; The dimensionless aerodynamic coefficients include: lift coefficient and drag coefficient; The calculation formula of lift coefficient is: ； ； ； in, is the lift coefficient, is the angle of attack of the long-flight UAV, is the sigmoid function, is the lift coefficient when the angle of attack is not greater than the stall angle of attack, is the lift coefficient when the angle of attack is greater than the stall angle of attack, is the coefficient of the first constant term, is the coefficient of the first linear term, is a symbolic function; The calculation formula of the drag coefficient is: ； ； ； in, is the drag coefficient, is the drag coefficient when the angle of attack is not greater than the stall angle of attack, is the drag coefficient when the angle of attack is greater than the stall angle of attack, is the coefficient of the second constant term, is the coefficient of the second linear term, is the coefficient of the quadratic term.

2. The drone control method based on deep reinforcement learning according to claim 1, characterized in that: According to the six-degree-of-freedom kinematic equation, a proximal strategy optimization algorithm is used to train a deep reinforcement learning model to obtain a trained deep reinforcement learning model, specifically including: Constructing an Actor-Critic neural network model; the Actor-Critic neural network model includes: a first strategy network, a second strategy network and a value function network; Based on the state and action at time step t, using the first policy network to generate an action at time step t+1; Based on the state and action at time step t, using the first policy network and the second policy network, calculate the reward at time step t; Based on the state at time step t, use the value function network to calculate the value of time step t; Based on the state at time step t and the action at time step t+1, the state at time step t+1 is generated based on the six-degree-of-freedom dynamic equation; Calculate the value of the total loss function according to the reward at time step t, the value at time step t, and the state at time step t+1; the total loss function includes the clipping objective function, the value function loss, and the entropy reward; Based on the value of the total loss function, update the parameters of the first policy network, the second policy network, and the value function network, increase the value of t by 1, return the state and action based on time step t, and use the first policy network to generate the action of time step t+1 until the iteration end condition is met. Output the first policy network after the last parameter update as the policy network in the trained deep reinforcement learning model.

3. The UAV control method based on deep reinforcement learning according to claim 2 is characterized in that: The total loss function is: ； ； ； ； ； in, is the total loss function, is the clipping objective function, is the value function loss, is the entropy reward, and are the parameters of the first policy network and the value function network respectively, , are weight coefficients, is the state at time step t, The parameters are The first strategy network; For expectations, is the generalized advantage function, For reward, To limit The cut-off coefficient, The parameters used are The value function of the network generates the value at time step t, is the target value, For the shear operation, is the number of steps accumulated from the current time step t, is the discount factor, To balance the weight of variance and bias, The parameters used are The value function network generates the value at time step t+1, The parameters used are The action at time step t+1 generated by the first policy network is The parameters used are The second policy network generates the action for time step t+1.

4. A drone control device based on deep reinforcement learning, characterized in that: The UAV control device based on deep reinforcement learning applies the UAV control method based on deep reinforcement learning according to any one of claims 1 to 3. The UAV control device based on deep reinforcement learning includes: A dynamic equation building module, used to build a six-degree-of-freedom dynamic equation for a long-flight UAV; the six-degree-of-freedom dynamic equation includes nonlinear aerodynamic characteristics and torque characteristics; A model training module, used to train the deep reinforcement learning model using a proximal strategy optimization algorithm according to the six-degree-of-freedom kinematic equation to obtain a trained deep reinforcement learning model; The prediction module is used to predict the control strategy based on the policy network in the trained deep reinforcement learning model; The control module is used to control the long-flight UAV based on the predicted control strategy.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the drone control method based on deep reinforcement learning as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the drone control method based on deep reinforcement learning described in any one of claims 1 to 3 is implemented.

Images