A variable impedance flight control method for UAV based on reinforcement learning

Abstract

The present invention relates to a variable-impedance flight control method for unmanned aerial vehicles (UAVs) based on reinforcement learning, and belongs to the technical field of UAVs. A reinforcement learning algorithm is used to establish a reinforcement learning network 1 for outputting flight control commands in a dynamics model and a reinforcement learning network 2 for outputting impedance parameters in an impedance control model. The flight control commands are input into the dynamics model to obtain the UAV's current actual state, and the impedance parameters are input into the impedance control model to obtain the UAV's estimated state error. The difference between the actual state, the estimated state error, and the desired state is then applied simultaneously to reinforcement learning network 1 and reinforcement learning network 2. This method achieves compliant control of the UAV during flight, thereby preventing damage to the UAV's rigid body while consuming less energy. The method has promising application prospects in the field of UAV flight control.

Description

Technical Field

[0001] The present invention relates to a variable impedance flight control method for an unmanned aerial vehicle (UAV) based on reinforcement learning, and belongs to the technical field of UAVs. Background Art

[0002] Traditional flight control methods rely heavily on various dynamic models and parameters, requiring complex mathematical proofs and a significant amount of time to adjust parameters to achieve stable flight. Furthermore, current flight control methods lack the ability to proactively adjust the flight compliance of the drone based on the current external environment during flight, potentially failing to cope with various emergencies and potentially causing damage to the drone itself. Summary of the Invention

[0003] In response to the shortcomings of current flight control methods, the present invention provides a variable impedance flight control method for unmanned aerial vehicles based on reinforcement learning. Based on the combination of reinforcement learning and variable impedance, the intelligent agent continuously interacts with the environment and autonomously learns impedance parameters instead of taking fixed parameters. This method can not only protect the rigid body of the unmanned aerial vehicle from damage, but also consume less energy and achieve the effect of smooth operation.

[0004] The objectives of the present invention are achieved through the following technical solutions.

[0005] A variable impedance flight control method for unmanned aerial vehicles (UAVs) based on reinforcement learning is proposed. A reinforcement learning algorithm is used to establish a reinforcement learning network 1 that outputs flight control commands in a dynamics model and a reinforcement learning network 2 that outputs impedance parameters in an impedance control model. The flight control commands are input into the dynamics model to obtain the UAV's current actual state, and the impedance parameters are input into the impedance control model to obtain the UAV's estimated state error. The difference between the actual state, the estimated state error, and the desired state is then applied simultaneously to reinforcement learning networks 1 and 2, achieving compliant control of the UAV during flight.

[0006] Among them, reinforcement learning learns the mapping relationship between the state error s and the optimal action a to be performed, that is, finding a strategy (function / logical rule) so that the decision made by this strategy in a given state can ultimately bring the maximum reward. That is, in this process, the learner is not told what action to take, but through continuous interaction with the environment, they discover what action can produce the highest reward value.

[0007] Furthermore, the specific operations of the variable impedance flight control method for UAV based on reinforcement learning are as follows:

[0008] (1) Construct a reinforcement learning network 1 with a dynamic model as the interactive environment. The target state error is used as the input of the reinforcement learning network 1. The output of the reinforcement learning network 1 is the speed in the three directions of x, y, and z and a yaw angular velocity. The output of the reinforcement learning network 1 (i.e., the speed in the three directions and a yaw angular velocity) is used as the input of the dynamic environment (or dynamic model). The output of the dynamic environment is the current actual state of the UAV.

[0009] (2) Repeat the reinforcement learning network_1 with reference to step (1) until the reward_1 in the reinforcement learning network_1 no longer increases (i.e., at this time, the reward_1 is the minimization of the error between the current state of the drone and the target state and the minimization of the number of collisions of the drone);

[0010] (3) Constructing a reinforcement learning network 2 with the impedance control model as the interactive environment, taking the target state error as the input of the reinforcement learning network 2, and correspondingly the output of the reinforcement learning network 2 is the damping parameter and stiffness parameter of the impedance control model; taking the output of the reinforcement learning network 2 as the input of the impedance environment (or the second-order system environment or the impedance control model), and correspondingly the output of the impedance environment is the estimated state error;

[0011] (4) The difference between the current actual state output corresponding to the time when reward_1 in step (2) no longer increases and the estimated state error and the expected state value output in step (3) is applied to the reinforcement learning network_1 and the reinforcement learning network_2 respectively, so that the reinforcement learning network_1 refers to step (1) and the reinforcement learning network_2 refers to step (3) and runs simultaneously. At this time, the difference between the current actual state output based on the reinforcement learning network_1 and the estimated state error and the expected state value output based on the reinforcement learning network_2 is used to run in the reinforcement learning network_1 and the reinforcement learning network_2 respectively, and so on, until the reward_1 in the reinforcement learning network_1 and the reward_2 in the reinforcement learning network_2 no longer increase, that is, the control of the flight state of the UAV in the variable impedance environment is completed;

[0012] Among them, the target state error includes position error, velocity error, rotation matrix error and angular velocity error; the reward_1 set in the reinforcement learning network_1 is to achieve stable flight of the UAV. The reward_1 is set to the error value between the current state of the UAV and the target state and the number of collisions of the UAV; the estimated state error refers to the state error value caused by external force and the original target position; the reward_2 set in the reinforcement learning network_2 is to achieve variable impedance operation. When the external force is large, the UAV follows the direction of the external force to produce a larger displacement. When the external force is small, the UAV follows the direction of the external force to produce a smaller displacement. The attitude change of the UAV changes according to the external torque as above. In addition, the energy consumption is related to the impedance parameter value. The larger the impedance parameter value, the greater the energy consumption. The reward_2 is set to the impedance parameter value and the displacement value generated by the UAV under the action of the external force.

[0013] Furthermore, the overall structures of reinforcement learning network_1 and reinforcement learning network_2 are both based on the PPO model architecture, and the PPO model is established based on the Garage framework.

[0014] Furthermore, the actor network and critic network in the reinforcement learning network_1 and reinforcement learning network_2 based on the PPO model architecture are both two-layer fully connected layers.

[0015] Furthermore, when both the actor network and the critic network are two fully connected layers, the number of neurons is 64×64.

[0016] Furthermore, the impedance control model adopts a second-order impedance model, and the specific formula is shown in (1.1):

[0017]

[0018] In formula (1.1), the F value on the right is the 6-dimensional vector composed of the corresponding external force and external torque; k1 is the target inertia matrix, k2 is the target damping matrix, k3 is the target stiffness matrix; y is the position error value, is the speed error value, is the acceleration error value.

[0019] Use the fourth-order Runge-Kutta formula to solve the second-order differential equation, and the fourth-order Runge-Kutta formula for the second-order differential equation is shown in (1.2):

[0020]

[0021] Introduce the variable z, let Then the original equation can be transformed into a first-order equation system as shown in formula (1.3):

[0022]

[0023] The fourth-order RK method solves the above equations, as shown in formula (1.4). When x takes different values, y and z are obtained:

[0024]

[0025] In formula (1.4), h is a hyperparameter, y_n is the estimated trajectory error value (original expected trajectory - estimated trajectory error value = current trajectory target position, the current trajectory target position is recorded as the expected state value);

[0026] Since the second-order impedance system should meet the requirements of stable second-order system parameters, the second-order linear system function is shown as (1.5):

[0027]

[0028] The relationship between the damping factor ζ and the frequency w is shown in formula (1.6):

[0029]

[0030] Formula (1.1) can be converted into formula (1.7):

[0031]

[0032]

[0033] In formula (1.7), as well as Both are 3×3 matrices;

[0034] To simplify the calculation, k1 can be taken as the unit matrix, then formula (1.8) can be transformed into formula (1.9):

[0035] k2=2ζω

[0036] k3=ω 2 (1.9)

[0037] Reinforcement learning is used to train the various parameter values of formula (1.1). The training goal is to minimize the overall energy consumption and the damage to the rigid body. From this, the target reward_2 in the optimized impedance control model can be designed, as shown in formula (1.10):

[0038] Cb=1-e -1 / 2||goal-real_state||

[0039] Ce=w 2

[0040] R=r1Cb+r2Ce (1.10)

[0041] In formula (1.10), goal is the target state value, real_state is the current actual state value, w is the frequency value in formula (1.7), r1 and r2 are fixed hyperparameters, and R is reward_2. Optimizing reward_2 is optional but not limited to formula (1.10).

[0042] Beneficial effects:

[0043] (1) The present invention adopts a reinforcement learning network based on a reinforcement learning algorithm and combines it with a dynamic environment to achieve stable flight control of the UAV. It can avoid the tedious parameter adjustment process of traditional UAV control methods and does not require repeated experiments to determine the parameters. Once the strategy network is generated, the control instructions of the UAV will be directly generated according to the current state and the target state value.

[0044] (2) The present invention adopts a reinforcement learning network established by a reinforcement learning algorithm and combines it with the impedance environment to achieve variable impedance control, so that the drone can adapt to the environment, thereby protecting the drone itself and other objects in the current environment.

[0045] (3) The present invention combines reinforcement learning with variable impedance. Under the action of uncontrollable external forces, the variable impedance model trained by reinforcement learning can change the target state value according to the magnitude of the external force. At the same time, with the cooperation of the dynamic model trained by reinforcement learning, the smooth control of the UAV flight process can be achieved, achieving the effect of reducing energy consumption and meeting the target displacement.

[0046] (4) The control method of the present invention is simple to operate and avoids the tedious parameter adjustment process. The control method can actively adjust the flexibility of the UAV flight according to the current external environment during the flight, which can not only avoid damage to the UAV rigid body itself, but also consume less energy. It has good application prospects in the field of UAV flight control. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of the variable impedance flight control method for unmanned aerial vehicle based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to specific embodiments, wherein the methods are conventional methods unless otherwise specified, and the raw materials can be obtained from public commercial channels unless otherwise specified.

[0049] Example 1

[0050] The specific operations of the variable impedance flight control method for UAV based on reinforcement learning are as follows:

[0051] (1) Construct a reinforcement learning network 1 with a dynamic model as the interactive environment. The target state error is used as the input of the reinforcement learning network 1. The output of the reinforcement learning network 1 is the speed in the three directions of x, y, and z and a yaw angular velocity. The output of the reinforcement learning network 1 (i.e., the speed in the three directions and a yaw angular velocity) is used as the input of the dynamic environment (or dynamic model). The output of the dynamic environment is the current actual state of the UAV.

[0052] (2) Repeat the reinforcement learning network_1 with reference to step (1) until the reward_1 in the reinforcement learning network_1 no longer increases (i.e., at this time, the reward_1 is the minimization of the error between the current state of the drone and the target state and the minimization of the number of collisions of the drone);

[0053] (3) Constructing a reinforcement learning network 2 with the impedance control model as the interactive environment, taking the target state error as the input of the reinforcement learning network 2, and correspondingly the output of the reinforcement learning network 2 is the damping parameter and stiffness parameter of the impedance control model; taking the output of the reinforcement learning network 2 as the input of the impedance environment (or the second-order system environment or the impedance control model), and correspondingly the output of the impedance environment is the estimated state error;

[0054] (4) The difference between the current actual state output corresponding to the time when reward_1 in step (2) no longer increases and the estimated state error and the expected state value output in step (3) is applied to the reinforcement learning network_1 and the reinforcement learning network_2 respectively, so that the reinforcement learning network_1 refers to step (1) and the reinforcement learning network_2 refers to step (3) and runs simultaneously. At this time, the difference between the current actual state output based on the reinforcement learning network_1 and the estimated state error and the expected state value output based on the reinforcement learning network_2 is used to run in the reinforcement learning network_1 and the reinforcement learning network_2 respectively, and so on, until the reward_1 in the reinforcement learning network_1 and the reward_2 in the reinforcement learning network_2 no longer increase, that is, the control of the flight state of the UAV in the variable impedance environment is completed;

[0055] Among them, the target state error includes position error, velocity error, rotation matrix error and angular velocity error; the reward_1 set in the reinforcement learning network_1 is to achieve stable flight of the drone. The reward_1 is set to the error value between the current state of the drone and the target state and the number of collisions of the drone; the estimated state error refers to the state error value caused by the external force and the original target position; the reward_2 set in the reinforcement learning network_2 is to achieve variable impedance operation. When the external force is large, the drone follows the direction of the external force to produce a larger displacement. When the external force is small, the drone follows the direction of the external force to produce a larger displacement. To produce a smaller displacement, the posture change of the drone is the same as above according to the change of external torque. In addition, the amount of energy consumption is related to the impedance parameter value. The larger the impedance parameter value, the greater the energy consumption. The reward_2 is set to the impedance parameter value and the displacement value generated by the drone under the action of external force. In addition, the formula for optimizing reward_2 is shown in formula (1.10); the overall structure of reinforcement learning network_1 and reinforcement learning network_2 are both based on the PPO model architecture established on the basis of the garage framework, in which the actor network and the critic network are both two-layer fully connected layers with a number of neurons of 64×64.

[0056] Cb=1-e -1 / 2||goal-real_state||

[0057] Ce=w 2

[0058] R=r1Cb+r2Ce (1.10)

[0059] In formula (1.10), goal is the target state value, real_state is the current actual state value, w is the frequency, r1 and r2 are fixed hyperparameters, and R is reward_2.

[0060] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A variable impedance flight control method for unmanned aerial vehicles based on reinforcement learning, characterized by: Using a reinforcement learning algorithm, a reinforcement learning network 1 is established to output flight control commands in the dynamics model, and a reinforcement learning network 2 is established to output impedance parameters in the impedance control model. The flight control commands are input into the dynamics model to obtain the current actual state of the UAV, and the impedance parameters are input into the impedance control model to obtain the estimated state error of the UAV. The difference between the actual state, the estimated state error, and the desired state is then applied to both reinforcement learning networks 1 and 2. This allows the UAV to achieve compliant control during flight. The specific operations of the variable impedance flight control method for UAV based on reinforcement learning are as follows: (1) Construct a reinforcement learning network 1 with the dynamic model as the interactive environment. The target state error is used as the input of the reinforcement learning network 1. The output of the reinforcement learning network 1 is the speed in the three directions of x, y, and z and a yaw angular velocity. The output of the reinforcement learning network 1 is used as the input of the dynamic model. The output of the dynamic model is the current actual state of the UAV. (2) Repeat the reinforcement learning network_1 according to step (1) until reward_1 in the reinforcement learning network_1 no longer increases; (3) Constructing a reinforcement learning network 2 with the impedance control model as the interactive environment, taking the target state error as the input of the reinforcement learning network 2, and correspondingly the output of the reinforcement learning network 2 is the damping parameter and stiffness parameter of the impedance control model; taking the output of the reinforcement learning network 2 as the input of the impedance control model, and correspondingly the output of the impedance control model is the estimated state error; (4) The difference between the current actual state output corresponding to the time when reward_1 in step (2) no longer increases and the estimated state error and the expected state value output in step (3) is applied to the reinforcement learning network_1 and the reinforcement learning network_2 respectively, so that the reinforcement learning network_1 refers to step (1) and the reinforcement learning network_2 refers to step (3) and runs simultaneously. At this time, the difference between the current actual state output based on the reinforcement learning network_1 and the estimated state error and the expected state value output based on the reinforcement learning network_2 is used to run in the reinforcement learning network_1 and the reinforcement learning network_2 respectively, and so on, until the reward_1 in the reinforcement learning network_1 and the reward_2 in the reinforcement learning network_2 no longer increase, that is, the control of the flight state of the UAV in the variable impedance environment is completed; Among them, the target state error includes position error, velocity error, rotation matrix error and angular velocity error; reward_1 is set to the error value between the current state of the drone and the target state and the number of collisions of the drone; the estimated state error refers to the state error value caused by external force compared with the original target position; reward_2 is set to the impedance parameter value and the displacement value generated by the drone under the action of external force.

2. The variable impedance flight control method for unmanned aerial vehicle based on reinforcement learning according to claim 1, characterized in that: The overall structure of reinforcement learning network_1 and reinforcement learning network_2 are both based on the PPO model architecture, and the PPO model is built on the basis of the Garage framework.

3. The variable impedance flight control method for unmanned aerial vehicle based on reinforcement learning according to claim 2, characterized in that: The actor network and critic network in the reinforcement learning network_1 and reinforcement learning network_2 based on the PPO model architecture are both two-layer fully connected layers.

4. The variable impedance flight control method for unmanned aerial vehicle based on reinforcement learning according to claim 3, characterized in that: When both the actor network and the critic network have two fully connected layers, the number of neurons is 64×64.

5. The variable impedance flight control method for unmanned aerial vehicle based on reinforcement learning according to claim 1, characterized in that: The formula for the target reward_2 in the optimized impedance control model is as follows: Cb=1-e -1 / 2||goal-real_state|| What=w 2 R=r1Cb+r2Ce Where goal is the target state value, real_state is the current actual state value, w is the frequency, r1 and r2 are fixed hyperparameters, and R is reward_2.

Images