A UAV control method based on multi-objective reinforcement learning
Through the multi-objective reinforcement learning method, the multi-objective weight in drone control is adaptively adjusted, which solves the problems of complex controller design and time-consuming adjustment of reward weights in the prior art, and achieves efficient drone control effect.
Patent Information
- Application Number
- CN202411341728.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-25
AI Technical Summary
The existing UAV control method needs to be linearized, resulting in complex controller design and not suitable for different balance points. The reinforcement learning-based method requires manual adjustment of reward weights, which is time-consuming and labor-intensive, making it difficult to effectively solve multiple goals. trade-offs.
Adopt the UAV control method based on multi-objective reinforcement learning, design the reinforcement learning observation space and action space by establishing the UAV model and simulation environment, use Lagrangian relaxation technology to solve the multi-objective optimization problem, and adaptively adjust the control target. The weight of the controller design bypasses the dependence of the model.
It realizes rapid design of controllers, reduces the dependence of reward function design, improves control effect, is suitable for fixed-wing and multi-rotor drones, avoids manual trial and error processes, and improves the unity of control effects.
Smart Images

Figure CN118884837B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle flight control, and in particular, relates to a unmanned aerial vehicle control method based on multi-objective reinforcement learning. Background Art
[0002] The complex six-degree-of-freedom UAV model has the characteristics of strong nonlinearity and state variable coupling. The traditional control method needs to linearize the model at the balancing point and use the obtained linearized model to design the controller. Due to the errors introduced by the linearized model, these controllers only guarantee convergence near the balancing point. In order to ensure the control effect within the full envelope range, it is also necessary to readjust the controller parameters at various balancing points, which makes the controller design time-consuming and labor-intensive. In addition, the control methods developed for specific unmanned machine models cannot be directly used on other unmanned machine models.
[0003] In recent years, the application of reinforcement learning technology to solve UAV control problems has received increasing attention. Reinforcement learning methods can not only bypass the controller design’s dependence on the model, but also have a high upper limit on control effects.
[0004] Existing reinforcement learning-based drone control methods design rewards as the weighted sum of rewards for multiple different control targets. In order to find a reward suitable for the control task, the weight coefficients need to be manually optimized through trial and error, which is time-consuming.
[0005] Multi-objective reinforcement learning is used to solve learning problems with multiple objectives. Different objectives may have different importance, so it is necessary to weigh different objectives and find the optimal solution that meets the requirements. Therefore, how to define the multi-objective optimization structure according to the UAV control requirements and adaptively adjust the weight coefficients of each control objective based on multi-objective reinforcement learning is a problem that needs to be solved at present. Summary of the invention
[0006] In view of the above defects, the present invention provides a UAV control method based on multi-objective reinforcement learning, comprising the following steps:
[0007] S1. Build a drone model and set up a simulation environment:
[0008] S1-1. Define the coordinate system;
[0009] S1-2, model construction, including construction of a translational dynamics model of a six-degree-of-freedom fixed-wing model, construction of a rotational dynamics model of a six-degree-of-freedom fixed-wing model, construction of a translational kinematics model of a six-degree-of-freedom fixed-wing model, construction of a rotational kinematics model of a six-degree-of-freedom fixed-wing model and construction of an engine model as a first-order delay model;
[0010] S1-3, build a simulation environment, including an initialization module, an algorithm action output module, a model solution module and a simulation end judgment module;
[0011] S2. The control requirements of the UAV are to achieve the desired yaw, altitude, speed and roll in sequence while ensuring safety and saving energy;
[0012] S3. Design the reinforcement learning observation space, action space, reward functions for yaw, altitude, speed, and roll control targets, and penalty functions for energy and safety control targets based on the UAV model and control requirements, and specify the maximum limit that the sum of the single-game simulation penalties is not allowed to exceed;
[0013] S4, designing a multi-objective reinforcement learning algorithm and performing training according to the control requirements specified in step S2;
[0014] S4-1. According to the control requirements of step S2, define the optimization problem of multi-objective reinforcement learning;
[0015] S4-2. Select the basic reinforcement learning algorithm that multi-objective reinforcement learning relies on;
[0016] S4-3, using Lagrangian relaxation techniques to solve the unconstrained dual problem of multi-objective reinforcement learning optimization problems;
[0017] S5. Use the multi-objective reinforcement learning algorithm Actor network model as the controller deployment algorithm.
[0018] Furthermore, the translational dynamics model of the six-degree-of-freedom fixed-wing model in step S1-2 is:
[0019]
[0020] In the above formula, are the differential forms of u, v, and w, respectively. u, v, and w are the velocity components in the fixed-wing body axis system, where u is the forward velocity, v is the lateral velocity, and w is the vertical velocity. p, q, and r are the angular velocity components in the fixed-wing body axis system, where p is the angular velocity around the X axis, q is the angular velocity around the Y axis, and r is the angular velocity around the Z axis. g is the acceleration of gravity, θ is the yaw angle, and φ is the roll angle. is the incoming flow pressure, S is the wing reference area, C X ,C Y ,C Z is the aerodynamic coefficient in the three directions of the fuselage axis, T is the engine thrust, and m is the fixed wing mass;
[0021] The rotational dynamics model of the six-degree-of-freedom fixed-wing model is:
[0022]
[0023] In the above formula, I is the moment of inertia, I X ,I Y ,I Z and I XZ ,I XY ,I YZ are the moments of inertia in the X, Y, Z axis directions and in the XZ, XY, YZ planes, respectively. are the differential forms of p, q, and r respectively. p, q, and r are the angular velocity components under the fixed wing body axis system, where p is the angular velocity around the X axis, q is the angular velocity around the Y axis, and r is the angular velocity around the Z axis. is the average geometric chord length of the wing, is the incoming flow pressure, C l , C m , C n are the aerodynamic moment coefficients in the three directions of the body axis, h eng is the angular momentum of the engine's rotating mass, S is the wing reference area, and b is the wing span;
[0024] v t , the relationship between α, β and u, v, w is as follows:
[0025]
[0026] In the above formula, v t is the airspeed, α and β are the angle of attack and sideslip angle respectively, u, v and w are the velocity components under the fixed wing body axis system respectively, where u is the forward speed, v is the lateral speed and w is the vertical speed;
[0027] The translational kinematic model of the six-degree-of-freedom fixed-wing model is:
[0028]
[0029] In the above formula, x E ,y E , h are the north position, east position and height in the plane geodetic coordinate system respectively, x E ,y E , the differential form of h, u, v, w are the velocity components under the fixed wing body axis system, where u is the forward velocity, v is the lateral velocity, and w is the vertical velocity;
[0030] The rotational kinematics model of the six-degree-of-freedom fixed-wing model is:
[0031]
[0032] In the above formula, p, q, and r are the angular velocity components of the fixed-wing body axis system, where p is the angular velocity around the X axis, q is the angular velocity around the Y axis, r is the angular velocity around the Z axis, θ is the yaw angle, ψ is the pitch angle, and φ is the roll angle. are the differential forms of θ, ψ, and φ respectively;
[0033] The engine is modeled as a first-order delay model:
[0034]
[0035] In the above formula, P c is the desired thrust level of the engine, P a is the engine delayed thrust level, P a The differential form of a is a variable, δ t is the throttle, h is the height in the plane geodetic coordinate system, M a is the Mach number, T is the engine thrust, τ eng is the engine time constant.
[0036] Furthermore, the initialization module in step S1-3 is used to set the initial state variable value and expected control output of the UAV at the beginning of the simulation, the algorithm action output module calculates the control variables of the UAV through a multi-objective reinforcement learning algorithm, the model solution module uses the current state variables and control variables of the UAV as input to obtain the state variables at the next moment, and the simulation end judgment module is used to determine whether the current simulation is ended.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] 1. Based on the multi-objective reinforcement learning method, according to the designed UAV control requirements, the multi-objective weights in the UAV control are adaptively adjusted, which reduces the dependence of the reward function design on expert knowledge, avoids the process of repeated attempts on the multi-objective weights in the reward function design, and can quickly obtain the controller;
[0039] 2. The multi-objective reinforcement learning method bypasses the controller design’s dependence on the model, can be applied to fixed-wing and multi-rotor controller design, and has a high upper limit on control effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flow chart of the control method based on multi-objective reinforcement learning of the present invention;
[0041] Figure 2 It is a simulation environment flow chart of the present invention;
[0042] Figures 3 to 6 They are respectively the reward training curve diagrams of yaw, altitude, speed and roll in the embodiments of the present invention;
[0043] Figures 7 to 10 They are respectively energy (elevator), energy (aileron), and energy (rudder) penalty training curve diagrams in the embodiment of the present invention;
[0044] Figures 11 to 14 They are respectively diagrams of yaw, altitude, speed and roll control effects in the embodiments of the present invention. DETAILED DESCRIPTION
[0045] In order to facilitate the understanding of the present invention, the device of the present invention will be described more fully below with reference to the relevant drawings. The accompanying drawings provide embodiments of the control method. However, the control method can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0046] Example
[0047] like Figure 1-2 As shown, this embodiment provides a drone control method based on multi-objective reinforcement learning, comprising the following steps:
[0048] S1. Build a drone model and set up a simulation environment:
[0049] S1-1. Define the coordinate system:
[0050] The center of mass of the fixed wing is located at the coordinate origin O, the OX axis points forward and is parallel to the longitudinal centerline of the fixed wing, the OY axis points to the right and is perpendicular to the symmetry plane, the OZ axis points upward and is perpendicular to the OX axis in the symmetry plane, and the forces acting on the fixed wing include aerodynamic force, thrust and gravity. The thrust direction is along the body axis X direction, and the curvature of the earth is ignored;
[0051] S1-2. Model construction:
[0052] The translational dynamics model of the six-degree-of-freedom fixed-wing model is:
[0053]
[0054] In the above formula, are the differential forms of u, v, and w respectively (the same applies to the following), u, v, and w are the velocity components in the fixed-wing body axis system, where u is the forward velocity, v is the lateral velocity, w is the vertical velocity, p, q, and r are the angular velocity components in the fixed-wing body axis system, where p is the angular velocity (roll) around the X axis, q is the angular velocity (pitch) around the Y axis, r is the angular velocity (yaw) around the Z axis, and g is the gravitational acceleration. θ is the yaw angle, ψ is the pitch angle, φ is the roll angle, is the incoming flow pressure, and C X ,C Y ,C Z is the aerodynamic coefficient in the three directions of the fuselage axis, T is the engine thrust, m is the fixed wing mass, m=9298.65kg, S is the wing reference area, S=27.87m 2 , ρ is the air density, v t is the airspeed, ρ and v t Airspeed is determined according to the International Standard Atmosphere;
[0055] The rotational dynamics model of the six-degree-of-freedom fixed-wing model is:
[0056]
[0057] In the above formula, p, r, q are the angular velocity components of the fixed wing body axis system, are the differential forms of p, r, q respectively, I is the moment of inertia, I X ,I Y ,I Z and I XZ ,I XY ,I YZ are the moments of inertia in the X, Y, Z axis directions and in the XZ, XY, YZ planes, respectively, and I X =12874.85kg·m 2 , I Y =75673.62kg·m 2 , I Z =85552.11kg·m 2 , I XZ =1331.41kg·m 2 , I XY =IYZ=0, is the average geometric chord length of the wing, For the incoming flow pressure, take C l , C m , C n are the aerodynamic moment coefficients in the three directions of the fuselage axis, and S is the reference area of the wing, taking S = 27.87m 2 , b is the wing span, b = 9.14m, C X ,C Y ,C Z ,C l ,C m ,C n The Morelli polynomial fitting scheme was used to determine h eng is the angular momentum of the engine's rotating mass, take
[0058] v t , the relationship between α, β and u, v, w is as follows:
[0059]
[0060] In the above formula, v t is the airspeed, α and β are the angle of attack and sideslip angle respectively, u, v and w are the velocity components under the fixed wing body axis system respectively, where u is the forward speed, v is the lateral speed and w is the vertical speed;
[0061] The translational kinematic model of the six-degree-of-freedom fixed-wing model is:
[0062]
[0063] In the above formula, x E ,y E , h are the north position, east position and height in the plane geodetic coordinate system respectively, x E ,y E , the differential form of h, u, v, w are the velocity components under the fixed wing body axis system, where u is the forward velocity, v is the lateral velocity, and w is the vertical velocity;
[0064] The rotational kinematics model of the six-degree-of-freedom fixed-wing model is:
[0065]
[0066] In the above formula, p, q, and r are the angular velocity components of the fixed-wing body axis system, where p is the angular velocity around the X axis, q is the angular velocity around the Y axis, r is the angular velocity around the Z axis, θ is the yaw angle, ψ is the pitch angle, and φ is the roll angle. are the differential forms of θ, ψ, and φ respectively;
[0067] The engine is modeled as a first-order delay model:
[0068]
[0069] In the above formula, P c is the desired thrust level of the engine, P a is the engine delayed thrust level, P a The differential form of a is a variable, δ t is the throttle, h is the height in the plane geodetic coordinate system, Ma is the Mach number, T is the engine thrust, τ eng is the engine time constant.
[0070] The state variable of the six-degree-of-freedom fixed-wing model is x = [vt ,α,β,φ,θ,ψ,p,q,r,x E ,y E ,h,P a ] T , v t is the airspeed, α, β are the angle of attack and sideslip angle respectively, θ is the yaw angle, ψ is the pitch angle, φ is the roll angle, p, q, r are the angular velocity components under the fixed wing body axis system, where p is the angular velocity around the X axis, q is the angular velocity around the Y axis, r is the angular velocity around the Z axis, x E ,y E , h are the north position, east position and height in the plane geodetic coordinate system, P a is the delayed thrust level of the engine, and the control input is u=[δ t ,δ e ,δ a ,δ r ] T , δ t ,δ e ,δ a ,δ r They are throttle, elevator angle, aileron angle and rudder angle, and their value ranges are δ t ∈[0,1],δ e ∈[-25°,+25°],δ a ∈[-21.5°,+21.5°],δ r ∈[-30°,+30°], the state equation of the six-degree-of-freedom fixed-wing model can be written as It should be noted that the model data used in this embodiment is F-16 aircraft data, and the parameters and functions can be determined according to "Brian LS, Frank L L. Aircraft control and simulation. John Wiley & Sons, Inc., Hoboken, New Jersey, 2003.";
[0071] S1-3. Construction of simulation environment:
[0072] The simulation environment includes an initialization module, an algorithm action output module, a model solution module and a simulation end judgment module;
[0073] The initialization module is used to set the initial state variable values and expected control output of the drone at the beginning of the simulation. The initial state value range is ψ~Uniform(-π,+π), h~Uniform(3,6)km, α=β=0, p=q=r=0, φ=θ=0, P aRandomly select values between 7, 8, 9, and 10. The expected output range of yaw, altitude, and speed is *+Uniform(-Δ* max ,+Δ* max ), * is a variable, where variable *={ψ,h,v t}, that is, * can be ψ,h,v t Any value in Δh max =50m, The expected roll angle output is always 0, where α and β are the angle of attack and sideslip angle respectively, and P a is the delayed engine thrust level, S is the wing reference area, m is the fixed wing mass, ψ is the pitch angle, φ is the roll angle, and h is the height in the plane geodetic coordinate system;
[0074] The algorithm action output module calculates the control variables of the UAV through the multi-objective reinforcement learning algorithm;
[0075] The model solving module uses the current state variables and control variables of the UAV as input to obtain the state variables at the next moment. In order to speed up the model solving speed, the Euler method is used for solving, and the simulation step length is Δt = 0.05 seconds;
[0076] The simulation end judgment module is used to judge whether the current simulation is ended. The simulation end conditions include reaching the maximum simulation step length of 300 steps or the fixed-wing overload n>10;
[0077] The simulation workflow of each module mentioned above is as follows: Figure 2 As shown in the figure, after the simulation operation starts, the initial state variable value and expected control output of the UAV are first transmitted through the initialization module, and then the state variables and control variables of the current UAV are transmitted to the simulation end judgment module through the algorithm action output module and the model solution module in sequence. The simulation end judgment module makes a judgment according to the set conditions. If the simulation conditions are met (ie, yes), the simulation ends. If the simulation conditions are not met (ie, no), it returns to the algorithm action output module.
[0078] Its role is that the design process of the multi-objective reinforcement learning algorithm does not depend on the specific drone model, and the algorithm does not depend on the specific forms of the observation space, action space, reward and penalty functions, so the applicable drone equation f can include fixed-wing and multi-rotor drones.
[0079] S2. The control requirements of the UAV are to achieve the desired yaw, altitude, speed and roll in sequence while ensuring safety and saving energy;
[0080] In the prior art, when macro-application reinforcement learning is used to solve unmanned control problems, control objectives such as yaw, altitude, speed, roll, energy, and safety are generally linearly combined into a scalar reward value through a set of weights, and then the scalar reward value is used to guide the update of the reinforcement learning algorithm parameters. This step can avoid the above multi-objective weight determination process, and by stipulating the optimization structure of all drone control objectives as the control requirements of step S2, the problem can be solved with the paradigm of multi-objective reinforcement learning.
[0081] At the same time, the control requirements can be abstracted to ensure that one set of control objectives does not exceed the maximum limit value, and another set of control objectives are achieved in sequence. The sequence can generally be sorted according to the difficulty of the tasks. The multi-objective reinforcement learning algorithm is designed to solve the abstracted control requirements, which does not depend on the number and type of the two sets of control objectives.
[0082] S3. Design the reinforcement learning observation space, action space, reward functions for yaw, altitude, speed, and roll control targets, and penalty functions for energy and safety control targets based on the UAV model and control requirements, and specify the maximum limit that the sum of the single-game simulation penalties is not allowed to exceed;
[0083] Specifically, the desired yaw, altitude, and speed of the UAV are defined as ψ expceted ,h expcated ,v t,expceted , the expected roll angle φ expceted ≡0, the maximum expected yaw, altitude, and speed increment of the UAV is defined as Δψ max ,Δh max ,Δv t,max , Δφ max =π, the difference between the actual value and the expected value of yaw, altitude, speed, and roll is defined as e ψ ,e h , e φ ,
[0084] The observation space of reinforcement learning is designed as
[0085] in, and This means replacing the column vector with a row vector:
[0086] The action space is designed as The values of all control variables are normalized to the interval [-1, +1]. In order to reduce the learning difficulty of the reinforcement learning algorithm, each action is discretized into 50 parts to form a four-dimensional discrete action space.
[0087] The reward functions for the yaw, altitude, speed, and roll control objectives are designed as:
[0088]
[0089] in:
[0090] * is a variable, variable*={ψ,h,v t ,φ}, that is, * can be ψ,h,v t , any value of φ, For example:
[0091] Where abs is the absolute function value,
[0092]
[0093] There are three penalty functions for the energy control objective, corresponding to the three moving control surfaces, namely the elevator, aileron and rudder, which are used to suppress the excessive vibration of the control surfaces and are designed as follows:
[0094]
[0095] where δ = {δ e ,δ a ,δ r}, δ e ,δ a and δ r are the elevator deflection angle, aileron deflection angle and rudder deflection angle respectively, and Δt is the simulation step length;
[0096] The penalty function c of the safety control objective safe (t) is:
[0097]
[0098] The maximum limit value that the sum of energy penalties for a single game simulation is not allowed to exceed is 1, and the maximum limit value that the sum of safety penalties for a single game simulation is not allowed to exceed is 0.01, which means that the situation where n>10 is not allowed to occur.
[0099] S4. Design a multi-objective reinforcement learning algorithm and perform training according to the control requirements of step S2:
[0100] S4-1. According to the control requirements of step S2, define the optimization problem of multi-objective reinforcement learning;
[0101] Define the reward functions for the yaw angle, height, speed, and roll angle that need to be optimized in order as r 1 ,r 2 ,r 3 ,r 4 , define the δ that needs to be constrained below the maximum limit value e ,δ a ,δr The corresponding energy control objective penalty functions are c 1 ,c 2 ,c 3 , define the maximum limit value of the energy control target penalty function as α1=α2=α3=1, and define the penalty function of the safety control target that needs to be constrained below the maximum limit value as c 4 , define the maximum limit value of the security control objective penalty function as α4 = 0.01;
[0102] Assume that the current time t target has converged, and its maximum values are B1, B2, …, B i-1 ,and To ensure convergence to the target does not decrease, and Under the condition of It can be formulated as the following sub-optimization problem:
[0103]
[0104] For all Solving the above sub-optimization problem completes the solution process of the entire optimization problem;
[0105] S4-2. Select the basic reinforcement learning algorithm that multi-objective reinforcement learning relies on. It should be noted that the basic reinforcement learning algorithm can be any Actor-Critic reinforcement learning algorithm, including but not limited to the PPO algorithm and the SAC algorithm;
[0106] The basic reinforcement learning algorithm selected in this embodiment is the PPO algorithm. The neural network of the PPO algorithm includes an Actor network for outputting policy actions and a Critic network for evaluating state value functions. Both types of networks are set as multi-layer feedforward neural networks. In addition, GRU (Gated Recurrent Unit) neural networks are used in these two types of networks to learn the time dependency in decision sequences.
[0107] Both the Actor network and the Critic network are divided into three parts, namely the feature extraction layer, the recurrent neural network layer and the output layer. The feature extraction layer of the Actor network and the Critic network are both two-layer feedforward neural networks with 128 neurons, the recurrent neural network layer is a GRU with 128 neurons, and the output layer is a two-layer feedforward neural network with 128 neurons. The difference is that the Actor network maps the output to a multi-dimensional classification distribution, representing the distribution satisfied by actions in each dimension, while the Critic network maps the output to a single-value output, representing the value of a certain state. In addition, the activation layers between all neural network layers are ReLU activation functions.
[0108] S4-3, using Lagrangian relaxation techniques to solve the unconstrained dual problem of multi-objective reinforcement learning optimization problems;
[0109] The Actor network parameters, Critic network parameters, and Lagrange multipliers obtained in step S4-2 are optimized synchronously by a multi-scale stochastic approximation method until all reward functions converge. Specifically:
[0110] Solve the unconstrained dual of an atomic optimization problem using Lagrangian relaxation techniques:
[0111]
[0112] in is the Lagrangian function, λ is the Lagrangian multiplier. In order to speed up the training, a multi-scale random approximation method is used to synchronously update all parameters to solve the unconstrained dual problem, and all Update the parameter θ and optimize all Parameter θ;
[0113] When the i-th target After convergence, let the learning rate And set the learning rate to:
[0114]
[0115] Finally, the multi-objective reinforcement learning problem can be solved by the following gradient update formula:
[0116]
[0117] In the above formula, V w For the Critic network, w is updated using temporal difference learning. In this example, we use the PPO algorithm. in is the advantage function, and the generalized advantage estimator (GAE) method is used for calculation, with γ = 0.99 and ε = 0.2 as hyper parameters.
[0118]
[0119] The pseudo code of the multi-objective reinforcement learning algorithm is shown in the following table
[0120]
[0121] like Figures 3 to 6 The change curves of each reward function in the training process of the multi-objective reinforcement learning algorithm in this embodiment are respectively shown, and the training curves of yaw, altitude, speed, and roll rewards are shown, indicating that the functions continue to increase as the training progresses; Figures 7 to 10 The variation curves of energy (elevator) penalty, energy (aileron) penalty, energy (rudder) penalty and safety penalty in the training process of the multi-objective reinforcement learning algorithm in this embodiment are respectively shown. For the convenience of display, the ordinates of the curves are drawn in logarithmic coordinates, and the four penalty functions are all restricted to the vicinity of the constraint values represented by the dotted lines. Figures 3 to 10 It can be seen that the multi-objective reinforcement learning algorithm proposed in the present invention successfully optimizes each control objective according to the control requirements, and there is no manual trial and error determination process of each control objective weight in this process.
[0122] S5: Use the multi-objective reinforcement learning algorithm Actor network model as the controller for algorithm nominal deployment;
[0123] In actual use, after loading the Actor network model, the fixed-wing current observation space variable As input, the output is
[0124] Further inverse transformation obtains the fixed-wing control variable [δ t ,δ e ,δ a ,δ r ].
[0125] Figures 11 to 14 They represent the situation when the Actor network model is used as the controller and the initial state is α=β=0,w B =0,φ=θ=0,ψ=-1.91rad,h=4025.38km,P a =7, the expected output is ψ expceted =-1.65rad,h expceted =3999.71m, The yaw, altitude, speed and roll control effects diagram.
[0126] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.
Claims
1. A drone control method based on multi-objective reinforcement learning, characterized in that: The following steps are involved: S1. Build a drone model and set up a simulation environment: S1-1. Define the coordinate system; S1-2, model construction, including construction of a translational dynamics model of a six-degree-of-freedom fixed-wing model, construction of a rotational dynamics model of a six-degree-of-freedom fixed-wing model, construction of a translational kinematics model of a six-degree-of-freedom fixed-wing model, construction of a rotational kinematics model of a six-degree-of-freedom fixed-wing model and construction of an engine model as a first-order delay model; S1-3, build a simulation environment, including an initialization module, an algorithm action output module, a model solution module and a simulation end judgment module; S2. The control requirements of the UAV are to achieve the desired yaw, altitude, speed and roll in sequence while ensuring safety and saving energy; S3. Design the reinforcement learning observation space, action space, reward functions for yaw, altitude, speed, and roll control targets, and penalty functions for energy and safety control targets based on the UAV model and control requirements, and specify the maximum limit that the sum of the single-game simulation penalties is not allowed to exceed; S4, designing a multi-objective reinforcement learning algorithm and performing training according to the control requirements specified in step S2; S4-1. According to the control requirements of step S2, define the optimization problem of multi-objective reinforcement learning; S4-2. Select the basic reinforcement learning algorithm that multi-objective reinforcement learning relies on; S4-3, using Lagrangian relaxation techniques to solve the unconstrained dual problem of multi-objective reinforcement learning optimization problems; S5. Use the multi-objective reinforcement learning algorithm Actor network model as the controller deployment algorithm.
2. The method for controlling a drone based on multi-objective reinforcement learning according to claim 1, characterized in that: The translational dynamics model of the six-degree-of-freedom fixed-wing model in step S1-2 is: ; In the above formula, They are The differential form of are the velocity components of the fixed-wing body axis system, among which is the forward speed, is the lateral velocity, is the vertical speed, is the angular velocity component of the fixed-wing body axis system, where is the angular velocity about the X axis, is the angular velocity around the Y axis, is the angular velocity about the Z axis, is the acceleration due to gravity, is the yaw angle, is the roll angle, is the incoming flow pressure, is the wing reference area, is the aerodynamic coefficient in the three directions of the body axis, is the engine thrust, is the fixed-wing mass; The rotational dynamics model of the six-degree-of-freedom fixed-wing model is: ; In the above formula, is the moment of inertia, , , and , , are the moments of inertia in the X, Y, Z axis directions and in the XZ, XY, YZ planes, respectively. , , They are The differential form of is the angular velocity component of the fixed-wing body axis system, where is the angular velocity about the X axis, is the angular velocity around the Y axis, is the angular velocity about the Z axis, is the average geometric chord length of the wing, is the incoming flow pressure, are the aerodynamic moment coefficients in the three directions of the fuselage axis, is the angular momentum of the engine's rotating mass, is the wing reference area, is the wing span; and The relationship is as follows: ; In the above formula, is the airspeed, are the angle of attack and sideslip angle, are the velocity components of the fixed-wing body axis system, among which is the forward speed, is the lateral velocity, is the vertical velocity; The translational kinematic model of the six-degree-of-freedom fixed-wing model is: ; In the above formula, are the north position, east position and height in the plane geodetic coordinate system, They are The differential form of are the velocity components of the fixed-wing body axis system, among which is the forward speed, is the lateral velocity, is the vertical velocity; The rotational kinematics model of the six-degree-of-freedom fixed-wing model is: ; In the above formula, is the angular velocity component of the fixed-wing body axis system, where is the angular velocity about the X axis, is the angular velocity around the Y axis, is the angular velocity about the Z axis, is the yaw angle, is the pitch angle, is the roll angle, , , They are , , The differential form of The engine is modeled as a first-order delay model: , , , , , In the above formula, is the desired thrust level of the engine, is the engine delayed thrust level, for The differential form of is a variable, For the throttle, is the height in the plane geodetic coordinate system, is the Mach number, is the engine thrust, is the engine time constant.
3. The method for controlling a drone based on multi-objective reinforcement learning according to claim 1, characterized in that: The initialization module in step S1-3 is used to set the initial state variable value and expected control output of the drone at the beginning of the simulation. The algorithm action output module calculates the control variables of the drone through a multi-objective reinforcement learning algorithm. The model solution module uses the current state variables and control variables of the drone as input to obtain the state variables at the next moment. The simulation end judgment module is used to judge whether the current simulation is ended.
Citation Information
Patent Citations
High-speed under-actuated AUV motion control method based on GRU maximum entropy reinforcement learning
CN118311868A
Wind driven generator blade unmanned aerial vehicle inspection method based on deep reinforcement learning algorithm
CN118601807A
Flight trajectory planning method and system based on random event capture, and storage medium
WO2024164498A1
