Advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning

Abstract

The invention discloses an advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning, and the method comprises the steps: firstly obtaining an effective flight sample for the construction of a helicopter intelligent flight control system through the parameter selection and multi-source data integration of the multi-source data according to the obtained actual flight multi-source data of a helicopter; the samples are controlled based on helicopter flight. The invention designs a helicopter flight control method based on advanced active learning, and aims to extract a flight control rule from a limited flight sample and improve the performance of a flight control and decision-making system through adaptive model adjustment and self-supervision auxiliary tasks.

Description

Technical Field

[0001] The invention relates to the technical field of helicopter maneuver flight control, in particular to an advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning. Background Art

[0002] Since helicopter maneuvering flight control in maneuvering or special situations is complex and highly uncertain, it is difficult to express analytically, and active learning and optimization are required based on the current control operation and performance. This project proposes an advanced active learning flight control method based on PID and reinforcement learning. Through adaptive adjustment of the model mechanism and self-supervised auxiliary task learning, the potential laws of flight control laws are extracted from limited helicopter flight samples, thereby improving the performance of helicopter flight control and decision-making systems.

[0003] Helicopter multi-source data is semantically rich. Combining the complementary information of multiple data sources can effectively improve the control performance, but the resulting data uncertainty also brings challenges to subsequent analysis. In order to solve this problem, data uncertainty will be considered from multiple levels such as features and samples, and helicopter pilots' control experience and flight mechanics prior knowledge will be embedded through constraints, in order to learn a more effective unified feature representation. Specifically, the fusion representation of helicopter multi-source data will be studied based on subspace learning, deep learning and other technologies, and feature representation learning methods that use prior knowledge and representation learning methods that mine and utilize the relationship between samples and features will be studied. In order to effectively learn efficient data feature representation from noisy information, a unified learning framework combining feature space learning with inter-sample relationship learning will be established. In particular, joint modeling of data from both the feature layer and the sample layer will be considered, and feature learning and inter-sample relationship learning will be organically integrated into a systematic algorithm. Feature representation learning methods based on subspace learning or manifold learning will be studied, representation learning methods that incorporate constraints on inter-sample relationships (such as geometric structure, sparse or low-rank prior information) and feature representation methods based on deep learning will be studied, in order to obtain a more robust control model. In addition, the collection cost and difficulty of helicopter flight samples are high, resulting in a very limited number of flight control samples, which poses a challenge to the discovery of helicopter maneuver flight control rate. This project proposes a flight control method based on advanced active learning to achieve autonomous multiple iterative optimization of helicopter maneuver flight process, continuously explore the potential laws of helicopter flight control, and realize helicopter control and decision modeling.

[0004] In order to solve the above problems, an advanced active learning method based on PID and reinforcement learning for helicopter trajectory tracking under small sample conditions is proposed. First, according to the multi-source data of actual helicopter flight, through parameter selection and integration of multi-source data, effective flight samples are obtained for the construction of helicopter intelligent flight control system. Based on the helicopter flight control sample, the present invention designs a helicopter flight control method based on advanced active learning, which aims to extract flight control laws from limited flight samples through adaptive model adjustment and self-supervised auxiliary tasks, and improve the performance of flight control and decision-making system. Helicopter flight control is difficult to analyze under complex and uncertain conditions, and active learning and optimization are required according to current operation and performance. Reinforcement learning can effectively combine operation and environment, so it is selected to realize advanced active learning of flight control law. However, due to the slow training and frequent triggering of training failure conditions in the early training process of reinforcement learning, it is considered to use PID control to intervene in the early stage to assist reinforcement learning training, set weights for PID and reinforcement learning control quantities, use high-weight PID control quantities to ensure the stability of training in the early stage of training, and use high-weight reinforcement learning to represent more complex flight actions in the later stage. Summary of the invention

[0005] The object of the present invention is to provide an advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The invention proposes an advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning. The method comprises the following steps: constructing an accurate flight dynamics model to describe the physical behavior of the helicopter under different flight conditions, including six degrees of freedom and related dynamic equations. The flight dynamics model includes aerodynamic characteristics and a control system part, and reproducing various flight operations and conditions of the helicopter in a simulation environment for reinforcement learning. In reinforcement learning, the state space should fully describe the flight state of the helicopter, and the action space defines all operations that the helicopter can perform. The reward function is the core of reinforcement learning, and it is necessary to balance flight stability, task completion, energy consumption and safety factors, select a suitable reinforcement learning algorithm, combine experience replay and target network and other technologies for training, and optimize the strategy to improve its performance in a real environment. Through the above steps, combined with the flight dynamics model and reinforcement learning, multiple iterative optimizations of helicopter flight control can be achieved, the automatic driving task can be gradually realized, and an intelligent flight control method for helicopters based on advanced active learning can be constructed.

[0008] Preferably, the reinforcement learning algorithm in the reinforcement learning is tested using a soft actor-critic algorithm (Soft Actor-Critic, SAC) and a twin delayed-deterministic policy gradient algorithm (Twin Delayed Deep Deterministic policy gradient, TD3).

[0009] Preferably, the SAC algorithm includes an Actor network and four Critic networks, wherein the Critic networks are respectively a state value estimation V network, a Target V network, and an action-state value estimation Q0 and Q1 network. The input of the Actor network is the flight state s of the helicopter at the current moment. t , the output is the action probability distribution parameter (continuous action space); the input of the Critic network is the flight state s of the helicopter at the current moment t , the output of the V Critic network is the estimate v(s) of the value of the helicopter’s flight state, and the output of the Q Critic network is the estimate q(s,a) of the value of the action-state pair.

[0010] Preferably, the SAC algorithm adopts a clipped double-Q technique similar to the TD3 algorithm, using two Q networks, and avoiding overestimation caused by maximization by taking the minimum Q value, and using a delayed value network (through Polyak averaging) to alleviate the infinite transmission of overestimation caused by bootstraping, training through policy network sampling actions, recording transition (state, action, reward, next state) in the play buffer; then obtaining training data from the play buffer, calculating the policy network and two Q networks, updating the Q network through the TD algorithm, using the smaller Q value to update the V network, and then using gradient ascent to update the obtained V value, and finally using Polyak averaging to update the target V network.

[0011] Preferably, the dual-delay-deterministic policy gradient algorithm (TD3) selects the smaller value as the estimated value during training.

[0012] Preferably, the reward function is specifically set as follows (index is the sequence number of the current target point):

[0013] rewardposition=-|△x|-|△y|-|△z|

[0014]

[0015]

[0016] reward success .

[0017] Preferably, a variable step_switch needs to be defined during the training process to measure the intervention timing of PID control and reinforcement learning control.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1) This paper proposes a novel advanced active learning method based on PID and reinforcement learning for helicopter trajectory tracking. This method studies the helicopter trajectory tracking task based on small samples, solves the problem that it is difficult to train a generalized reinforcement learning model using a small number of samples, and can achieve high-precision helicopter trajectory tracking and attitude control.

[0020] 2) The present invention comprehensively considers the data acquisition conditions in actual application scenarios, uses only the expected path as input, and solves the expected control amount at each moment according to the expected path as a benchmark for PID control error calculation and reinforcement learning reward function calculation.

[0021] 3) The present invention uses the PID-assisted reinforcement learning method to solve the problem that failure conditions are often triggered in the early stage of reinforcement learning training under the condition of a small number of samples, making training difficult.

[0022] 4) The present invention designs a fusion mechanism of PID and reinforcement learning control quantity, which gradually increases the usage weight of the reinforcement learning control quantity during the reinforcement learning training process. When the effect of reinforcement learning training is poor in the early stage, the PID control quantity is mainly used to play a stabilizing role. When the effect of reinforcement learning training gradually improves in the later stage, the reinforcement learning control quantity is mainly used to express more complex actions.

[0023] 5) This paper proposes a helicopter trajectory tracking method using PID-assisted reinforcement learning. Experimental results under multiple paths and multiple helicopter models show that this method can achieve helicopter trajectory tracking and attitude control tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the network structure diagram of the SAC algorithm in the present invention;

[0025] Figure 2 This is a block diagram of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The present invention provides an advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning, which is as follows:

[0028] Due to the strong coupling and weak stability characteristics of helicopter rotors, flight control requirements are very high, and advanced intelligent systems are urgently needed to reduce the difficulty of pilot control. However, the flight action data in the early stage of research is limited, especially in special conditions such as engine shutdown and tail rotor control jam, and it is impossible to directly train the deep neural network model through a large amount of data. Therefore, it is necessary to collect and integrate multi-source flight sample data, and use advanced active learning to develop digitally driven flight control technology to autonomously discover environmental laws and optimize flight control schemes. In order to realize advanced active learning of helicopter flight control laws based on reinforcement learning, an accurate flight dynamics model is first required to describe the physical behavior of the helicopter under different flight conditions, including six degrees of freedom and related dynamic equations. The model should contain aerodynamic characteristics and control system information, and reproduce various flight operations and conditions of the helicopter in a simulated environment. In reinforcement learning, the state space should fully describe the flight state of the helicopter, such as position, speed, and attitude; the action space defines all operations that the helicopter can perform, such as the adjustment of the pitch lever and pedals. The reward function is the core of reinforcement learning, which needs to balance factors such as flight stability, task completion, energy consumption, and safety. Select a suitable reinforcement learning algorithm, combine it with experience replay and target network techniques for training, and optimize the strategy to improve its performance in the real environment. Through the above steps, combined with the flight dynamics model and reinforcement learning, multiple iterative optimizations of helicopter flight control can be achieved, the automatic driving task can be gradually realized, and a helicopter intelligent flight control method based on advanced active learning can be constructed.

[0029] In terms of reinforcement learning algorithm selection, this project uses the Soft Actor-Critic (SAC) algorithm and the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm for testing. The SAC algorithm is an off-policy reinforcement learning algorithm that can solve reinforcement learning problems in discrete and continuous action spaces. The core feature of this algorithm is entropy regularization. The training of the strategy makes a trade-off between maximizing expected returns and maximizing entropy. The increase in entropy will lead to more exploration during the training process, which can accelerate subsequent learning while preventing the strategy from converging to a bad local optimum too early. The SAC algorithm consists of an Actor network and four Critic networks. The Critic networks are the state value estimation V network, the Target V network, and the action-state value estimation Q0 and Q1 networks. The input of the Actor network is the flight state s of the helicopter at the current moment. t , the output is the action probability distribution parameter (continuous action space); the input of the Critic network is the flight state s of the helicopter at the current moment t The output of the V Critic network is the estimated value of the helicopter flight state v(s), and the output of the Q Critic network is the estimated value of the action-state pair q(s,a). The specific network structure is as follows Figure 1 shown.

[0030] In order to prevent the overestimation problem from affecting the training, it is necessary to solve the overestimation caused by conduction and maximization. The SAC network used adopts a clipped double-Q technique similar to the TD3 algorithm. Figure 1 As shown in the figure, two Q networks are used, and the overestimation caused by maximization is avoided by taking the minimum Q value, and the delayed value network (through Polyak averaging) is used to alleviate the overestimation caused by bootstraping. In the specific implementation, the policy network samples actions for training, and records transition (state, action, reward, next state) in the play buffer; then the training data is obtained from the μlay buffer, the policy network and two Q networks are calculated, the Q network is updated through the td algorithm, and the smaller Q value is used to update the V network. After the V value is obtained, the gradient ascent is used to update the policy network, and finally the target V network is updated using Polyak averaging.

[0031] The action value function is estimated, and the smaller value is selected as the estimated value during training. When implementing the TD3 algorithm, an Actor network and two Critic networks are first initialized, and then the optimal strategy is found through repeated iterations. In order to smooth the noise added in the early stage, noise a~π is added when selecting the action value in each iteration. φ (s)+∈,∈~N(0,σ). After the Critic network is updated, the Actor network adopts a delayed update. Generally, the Critic is updated twice and the Actor is updated once. The Actor network is updated by gradient ascent and the Critic network is updated by soft update.

[0032] In the process of reinforcement learning training, TD3 and SAC reinforcement learning models are used to train and test the helicopter model. The action space is the total pitch, pedal, lateral periodic pitch change and longitudinal periodic pitch change. The action space value range is [-1, 1], and the state space is composed of the state quantities provided by the helicopter sensor. The success judgment condition is that the helicopter successfully arrives near the last target point, and the failure judgment condition is that the shortest distance between the helicopter and all target points is greater than 2 meters. In the reward function setting, a large number of combination attempts and coefficient fine-tuning are made for the above state quantities. The specific settings of the reward function are as follows (index is the serial number of the current target point):

[0033] rewardposition=-|△x|-|△y|-|△z|

[0034]

[0035]

[0036]

[0037] The working principle of PID controller (proportional-integral-differential controller) is based on feedback control theory. It calculates the error between the system output and the target value, and adjusts the control amount according to the current value, past value and change trend of the error, so that the system output is as close to the target value as possible. PID controller consists of three parts: proportional control (P), integral control (I) and differential control (D). Proportional control (P) is the core part of PID control. Its function is to adjust the control output according to the current error. Taking altitude control as an example, the error is defined as the difference between the set altitude value (target altitude value) and the actual altitude value. The proportional controller multiplies the altitude error by a proportional gain coefficient (K p ), and get the control output. The formula for proportional control is:

[0038] P(t)=K p ·e(t)

[0039] Where P(t) is the output of the proportional control part, K p is the proportional gain, that is, the coefficient of proportional control, and e(t) is the current altitude error. Proportional control can respond quickly to system errors and make adjustments immediately. However, when proportional control is used alone, the system often has a steady-state error, that is, the actual output of the system cannot fully reach the target value. This is because when the altitude error is very small, the output of the proportional control is also very small. When the lift provided by the control amount given by the PID is equal to the gravity, the aircraft will maintain a steady state. The error that exists at this time is called a steady-state error, so proportional control is not enough to completely eliminate the error.

[0040] Integral control (I) adjusts the control output by accumulating the error over time. The integral controller calculates the sum of the errors from the start of control to the current moment and multiplies it by the integral gain factor (K i ), and get the control output. The formula for integral control is:

[0041]

[0042] Where I(t) is the output of the integral control part, K i is the integral gain, i.e. the coefficient of integral control, and e(τ) is the error over time. Integral control can eliminate steady-state errors because, even if the error is small, the integral part will continue to accumulate over time, thereby increasing the control output until the error is zero. However, excessive integral gain may cause the system to respond too slowly or produce oscillations.

[0043] Derivative control (D) adjusts the control output by predicting the changing trend of the error. The derivative controller calculates the rate of change of the error (i.e., the derivative of the error) and multiplies it by the derivative gain coefficient (K d ), and get the control output. The formula for differential control is:

[0044]

[0045] Where D(t) is the output of the differential control part, K d is the differential gain, which is the coefficient of differential control, is the error change rate. Differential control can foresee the change of error and make adjustments in advance, thereby improving the dynamic response characteristics of the system and reducing overshoot and oscillation. However, differential control is sensitive to noise because rapid changes in noise can cause the derivative of the error to change rapidly, causing instability in the control output. The PID controller combines the three parts of proportional, integral and differential control and comprehensively utilizes their advantages to achieve precise control of the system. The total output of the PID controller is the sum of the above three parts:

[0046] u(t)=P(t)+I(t)+D(t)

[0047]

[0048] In PID control, the desired acceleration and velocity direction and magnitude are calculated through the input trajectory and corrected as the quantity controlled by PID, ultimately achieving a complete tracking effect.

[0049] Specifically, a variable step_switch needs to be defined during the training process to measure the timing of intervention of PID control and reinforcement learning control. For example, when step_switch = 1000, it means that reinforcement learning control will replace PID control when the model is trained for 1000 rounds. For the specific value of the quaternary in the action space, the PID control pedal and rudder control inputs are always used, and after exceeding the number of conversion rounds, reinforcement learning is used to replace the collective distance originally controlled by PID to achieve the trajectory tracking task of PID-assisted reinforcement learning. Finally, when the number of training rounds does not reach step_switch, the action quaternary is defined as: [PID collective distance control amount, PID longitudinal control amount-a, PID lateral control amount, PID pedal control amount]; when the number of training rounds reaches step_switch, the action quaternary is defined as: [a* reinforcement learning collective distance control amount-a+a*PID collective distance control amount, a* reinforcement learning lateral control amount-a+a*PID longitudinal control amount, PID lateral control amount, PID pedal control amount]. Then the determined action quaternary is input into the helicopter model to obtain the state quantity at the next moment, and the control of the helicopter is simulated in this way. For the sake of simplicity, the weights of reinforcement learning and PID control are replaced by a. In actual implementation, the weight of reinforcement learning control can be increased and the weight of PID control can be reduced as the number of training rounds increases.

[0050] In the reinforcement learning process, failure judgment conditions include but are not limited to: the distance from the target trajectory exceeds 2m; after the action quadruple is input into the model, a Nan value is obtained (an unknown error occurs or the aircraft attitude cannot be controlled and loses control); the helicopter attitude and speed exceed the critical value. The reinforcement learning rate decreases with the number of training rounds, making the reinforcement learning model more robust.

[0051] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0052] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning, characterized by: The method is as follows: an accurate flight dynamics model is constructed to describe the physical behavior of a helicopter under different flight conditions, including six degrees of freedom and related dynamic equations. The flight dynamics model includes aerodynamic characteristics and a control system part, and various flight operations and conditions of the helicopter are reproduced in a simulation environment to perform reinforcement learning. In reinforcement learning, the state space should fully describe the flight state of the helicopter, and the action space defines all operations that the helicopter can perform. A reward function is set in reinforcement learning, and the reward function is the core of reinforcement learning. It is necessary to balance flight stability, task completion, energy consumption and safety factors, select a suitable reinforcement learning algorithm, combine experience replay and target network and other technologies for training, and optimize the strategy to improve its performance in a real environment. Through the above steps, combined with the flight dynamics model and reinforcement learning, multiple iterative optimizations of helicopter flight control can be achieved, automatic driving tasks can be gradually achieved, and a helicopter intelligent flight control method based on advanced active learning can be constructed.

2. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm in the reinforcement learning is tested using the Soft Actor-Critic (SAC) algorithm and the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm.

3. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 2, characterized in that: The SAC algorithm includes an Actor network and four Critic networks. The Critic networks are the state value estimation V network, the Target V network, the action-state value estimation Q0 and Q1 networks. The input of the Actor network is the flight state s of the helicopter at the current moment. t , the output is the action probability distribution parameter (continuous action space); the input of the Critic network is the flight state s of the helicopter at the current moment t , the output of the V Critic network is the estimate v(s) of the value of the helicopter’s flight state, and the output of the Q Critic network is the estimate q(s,a) of the value of the action-state pair.

4. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 3, characterized in that: The SAC algorithm adopts a clipped double-Q technique similar to the TD3 algorithm, using two Q networks and taking the minimum Q value to avoid overestimation caused by maximization, and using a delayed value network (through Polyak averaging) to alleviate the infinite transmission of overestimation caused by bootstraping. The policy network samples actions for training and records transitions (state, action, reward, next state) in the play buffer. Then, the training data is obtained from the playbuffer, the policy network and two Q networks are calculated, the Q network is updated through the TD algorithm, the smaller Q value is used to update the V network, and the obtained V value is then updated using gradient ascent to update the policy network, and finally the target V network is updated using Polyak averaging.

5. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 1, characterized in that: The dual delayed-deterministic policy gradient algorithm (TD3) selects the smaller value as the estimated value during training.

6. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 1, characterized in that: The reward function is specifically set as follows (index is the current target point number): reward position =-|△x|-|△y|-△z| 7. The advanced active learning method for helicopter trajectory tracking based on PID and reinforcement learning according to claim 1, characterized in that: During the training process, a variable step_switch needs to be defined to measure the timing of intervention of PID control and reinforcement learning control.

Images