PID adaptive control method for parafoil motion

Abstract

The present invention provides a PID adaptive control method for parafoil motion, comprising the following steps: Step 1: Selecting P key characteristic factors that affect the dynamic performance of the parafoil to construct a state space S, and dividing the state space into multiple state subspaces; Step 2: Using a deep reinforcement learning method to obtain effective PID parameters for each state subspace; Step 3: Selecting K state subspaces that are most similar to the current state of the parafoil flight, and using the effective PID parameters in the K state subspaces to synthesize the PID control parameters for the parafoil in the current flight state, thereby completing the motion control of the parafoil system. The present invention utilizes a deep reinforcement learning method to assist the PID control strategy in synthesizing the PID control parameters for the parafoil in the current flight state, thereby completing the motion control of the parafoil system. This achieves the complementary advantages of deep reinforcement learning and PID controllers, while avoiding their respective shortcomings.

Description

Technical Field

[0001] The present invention belongs to the technical field of parafoil motion control, and in particular relates to a parafoil motion PID adaptive control method assisted by deep reinforcement learning. Background Art

[0002] How to achieve precise and stable motion control of the parafoil system is a technical difficulty. The reasons are: first, the parafoil system is a complex under-actuated system with the characteristics of strong coupling, nonlinearity, large time lag, uncertainty and nonlinear disturbance, and is affected by multiple constraints; second, the main operation method of the parafoil is to pull down the left and right trailing edges to achieve steering. Compared with helicopters, quadcopters and fixed-wing UAVs, its unique ramjet structure aerodynamic shape and parachute rope control structure determine its limited controllability, poor maneuverability, and obvious interference from the external environment.

[0003] The PID controller is one of the earliest controller types and is widely used in industrial process control. It is currently the primary controller used in paraglider motion control. As a non-model-based controller, the PID controller has the advantages of simple structure and easy adjustment, which are specifically manifested in the following four aspects: First, the PID algorithm contains the key information of the past, present, and future in the dynamic control process, and its configuration is almost optimal; Second, PID control is adaptable and robust, and can be applied to various industrial applications to varying degrees; Third, the PID algorithm is simple and clear, and each control parameter is relatively independent, making parameter selection relatively simple, forming a complete design and parameter adjustment method that is easily mastered by engineering technicians; Fourth, PID control has been improved to address its own shortcomings according to different requirements, resulting in a series of improved PID algorithms. However, PID control is only suitable for relatively simple applications and cannot control complex objects with large inertia and large hysteresis. Its shortcomings are mainly manifested in the following three aspects: First, the initial error of PID control can be large, which easily causes overshoot, making the method of generating error in PID control unreasonable; Second, the differential signal of the error cannot be generated very well; Third, the introduction of error integral feedback has side effects. Existing PID control methods for parafoil motion are primarily based on parafoil aerodynamic models, utilizing simulation and real-world experimental data to determine PID control parameters. However, the parafoil system is a complex, nonlinear control system with significant time lag. During flight, it is susceptible to external uncertainties, exhibiting nonlinear characteristics and coupling. In practice, any small external disturbance can alter the parafoil's aerodynamic shape. Simulation models and limited real-world or simulated experimental data alone cannot accurately and completely describe the actual trajectory of a parafoil's motion.

[0004] Reinforcement learning (RL) originates from principles of animal psychology and mimics the trial-and-error learning mechanisms of humans and animals. It is a method that learns the mapping between states and behaviors through interaction with the environment to maximize cumulative expected rewards. Furthermore, RL is an important approach to achieving general artificial intelligence (capable of handling a wide range of tasks and adapting to various situations). This method is widely used to train control strategies for various unmanned systems, such as drones, autonomous vehicles, and robots. However, its application in parafoil motion control is relatively limited. Deep neural networks (DNNs) can directly learn optimal policies without traversing all possible states of the system and are currently the most commonly used control strategy model for RL. DNNs can fit highly nonlinear phenomena without explicit models. Control strategies derived from DNNs trained online or offline in real or simulated environments using RL can effectively predict and respond to parafoil motion. The main drawbacks of this control strategy are: first, training a blank initial model to convergence requires a large amount of training data and time, and the more complex the model, the greater the training data and time required. Second, DNN control strategies lack interpretability, hindering their widespread and in-depth application in numerous cost- and safety-sensitive fields. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and provide a PID adaptive control method for parafoil motion. The method uses a deep reinforcement learning method to assist the PID control strategy in synthesizing the PID control parameters of the parafoil under the current flight state, completing the motion control of the parafoil system, realizing the complementary advantages of deep reinforcement learning and PID controller, and avoiding their respective shortcomings.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A PID adaptive control method for parafoil motion includes the following steps:

[0008] Step 1: Select P key characteristic factors that affect the dynamic performance of the parafoil to construct a state space S, and divide the state space into multiple state subspaces;

[0009] Step 2: Use deep reinforcement learning method to obtain effective PID parameters for each state subspace;

[0010] Step 3: Select the K state subspaces that are most similar to the current state of the parafoil flight, and use the effective PID parameters under the K state subspaces to synthesize the PID control parameters under the current flight state of the parafoil to complete the motion control of the parafoil system.

[0011] Furthermore, the method for constructing the state space in step 1 is:

[0012] Select P key characteristic factors that affect the dynamic performance of the parafoil to form the state space P∈{1,2,3,4,5,6}.

[0013] Furthermore, the key characteristic factors in step 1 include but are not limited to the wind direction angle, wind force, and load multiple of the wind field in the parafoil flight environment.

[0014] Furthermore, the method for dividing the state subspace in step 1 is:

[0015] Divide the state space S into N state subspaces to form a state subspace set

[0016] Divide the j (j = 1, 2, 3, ... P) dimension value range of the state space S into n j subintervals, then the number of state subspaces divided by the state space S is N=n1*n2*n3*...*n P ;

[0017] In each state subspace Select a state As Representative states in the state subspace.

[0018] Furthermore, in step 2, the Actor-Critic deep reinforcement learning method is used to obtain the effective PID parameters of each state subspace.

[0019] Furthermore, step 2 specifically includes the following steps:

[0020] Step 2.1: Assume that the representative states and effective PID parameters of L state subspaces are known, and let the set of L state subspaces be G = {S i G} i＝1,2,3,...,L , where L≥1, the valid PID parameter set is {PID i G} i＝1,2,3,...,L ;

[0021] Step 2.2: Use Actor-Critic deep reinforcement learning to train an Actor model and a Critic model for each state subspace in the set G.

[0022] Step 2.3: For each state subspace Do the following: If the state subspace Valid PID parameters have been set, then end step 2.3; if the state subspace If no valid PID parameters are set, search the adjacent state subspace. If no valid PID parameters are set in the adjacent state subspace, end step 2.3. If the state subspace If no valid PID parameters are set, and at least one state subspace in its adjacent state subspace has valid PID parameters set, first select an adjacent state subspace that has valid PID parameters set, and take the Actor model and Critic model of the adjacent state subspace as the initial Actor model and initial Critic model of the current state subspace; then, in the state subspace Actor-Critic reinforcement learning method is used to train and Model until both performance converges; finally, take out The model parameters are used as the current state subspace Valid PID parameters;

[0023] Step 2.4: Repeat step 2.3 until all state subspaces have been set with valid PID parameters.

[0024] Furthermore, the specific method of step 2.2 is:

[0025] First, for any state subspace S in the set G i G (i=1,2,3,...,L), using parameter PID i G (i=1,2,3,...,L) build PID model as Actor i G Model, using deep neural network to build initial critic i G Model; secondly, fixed Actor i G The model parameters remain unchanged, in the state subspace S i G In the code, we use the Actor-Critic deep reinforcement learning method to train the Critic i G model until its performance converges.

[0026] Furthermore, the specific implementation steps in step 3 are:

[0027] Step 3.1, O t ∈R P Indicates the current state of the parafoil during flight;

[0028] Step 3.2, calculate the current state O t and state subspace The similarity between

[0029]

[0030] Step 3.3, take the current state O t The most similar K state subspaces The effective PID parameters constitute a set The corresponding K similarities constitute a set The current state is O t The synthesis results of the PID control parameters are as follows:

[0031]

[0032] Step 3.4: During the motion of the parafoil system, steps 3.1, 3.2, and 3.3 are iteratively executed in sequence to synthesize the PID parameters of the current state of the parafoil system and complete the motion control of the parafoil system.

[0033] Furthermore, the calculation formula for similarity in step 3.2 is:

[0034]

[0035] Where, represents the vector inner product, "||·||" represents the modulus of the orientation quantity, Represents the state subspace Representative status.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1) Reduce the labor cost of obtaining multiple sets of valid PID parameters; the present invention only needs to provide valid PID control parameters in at least one state subspace, and can use deep reinforcement learning methods to obtain valid PID control parameters in all other state subspaces;

[0038] 2) Adaptive PID control of complex nonlinear parafoil systems is achieved. Since general PID control uses a fixed single set of parameters, it cannot adapt to complex nonlinear parafoil systems. The present invention divides the complex nonlinear parafoil system into multiple approximately linear parafoil systems by dividing the state subspace.

[0039] 3) It saves data and time for deep reinforcement learning model training. Generally, deep reinforcement learning requires a large amount of training data and training time to train a blank initial model until performance convergence. The more complex the model, the more training data and training time are required. However, the present invention uses a simple PID model as the actor model in actor-critic reinforcement learning. Each time the model in a new state subspace is trained, the converged actor model and critic model in the adjacent state subspace are migrated and used as the initial model. This can greatly reduce the demand for training data and training time.

[0040] 4) The upgrade requirements for the existing parafoil control system are low, and the performance is improved at a low cost. The final control strategy of the present invention is still PID, so there is no need to update the existing PID control system hardware or add high-performance parallel computing hardware. It only needs to store multiple groups of representative states of state subspaces and valid PID parameters for calculating vector similarity and vector weighted summation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the PID adaptive control method for parafoil motion proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other.

[0044] The present invention will be further described below with reference to specific examples, but they are not intended to limit the present invention.

[0045] The embodiment of the present invention is an example of applying a parafoil system to perform simulated flight control and actual flight control of the parafoil. In this embodiment, a deep reinforcement learning agent interacts with the parafoil simulation environment to learn the effective PID parameters of the parafoil in each state subspace, and implements motion control of the parafoil system in the simulation environment and the real environment respectively. The parafoil motion PID adaptive control method provided by the embodiment of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0046] Step 1: Construct the state space; select P key characteristic factors that affect the dynamic performance of the parafoil to form the state space P∈{1,2,3,4,5,6}. In this embodiment, the wind direction angle (defined as the angle between the wind vector and the parafoil trajectory direction, in radians) θ∈[0,2π], the wind force (in meters per second) w∈[0,8], and the load multiple (defined as the multiple of the normal load of the parafoil system) z∈[0,2] in the parafoil flight environment are selected to form a three-dimensional state space with P=3.

[0047] Step 2: Divide the state subspace; divide the state space into N subspaces to form a state subspace set In this embodiment, the j (j=1, 2, 3, ...P) dimension value range of the state space S is divided into n j (In this embodiment, n j =4, in other embodiments, subintervals are set according to actual needs, so that the state space S is divided into N=n1*n2*n3=4*4*4=64 state subspaces. In each state subspace Select the central state point of the state subspace As Representative states in the state subspace.

[0048] Step 3: Use deep reinforcement learning to obtain valid PID parameters in the state subspace. The number of state subspaces in this embodiment is very large, i.e., N=64. The feasibility and practical effect of using manual debugging to obtain all valid PID parameters are poor. However, this embodiment uses a deep reinforcement learning-assisted PID adaptive control method to obtain the PID parameters in the state subspace with higher efficiency and accuracy. The specific implementation steps are as follows:

[0049] 1) Assume that the representative states and effective PID parameters of L (L≥1) state subspaces are known; let the set of L state subspaces be G={S i G} i＝1,2,3,...,L , state subspace S i G The representative state set of (i=1,2,3,...,L) is {T i G} i＝1,2,3,...,L , the valid PID parameter set is {PID i G} i＝1,2,3,...,L In this embodiment, the representative states and effective PID parameters of L (L=4) state subspaces are given by manual debugging. i G} i＝1,2,3,4 represents a set of 4 state subspaces, {T i G} i＝1,2,3,4Indicates the corresponding representative state set, {PID i G} i＝1,2,3,4 Indicates the corresponding valid PID parameter set; where T1 G =(θ=π / 4, w=1, z=1.25), T2 G =(θ=3π / 4, w=1, z=1.25), T3 G =(θ=5π / 4, w=1, z=1.25), T4 G =(θ=7π / 4, w=1, z=1.25);

[0050] 2) Use Actor-Critic deep reinforcement learning to train the Actor model and Critic model for each state subspace in the set G; this embodiment uses the TD3 deep reinforcement learning method. First, for any state subspace S in the set G, i G (i=1,2,3,4), using parameter PID i G (i=1,2,3,4) Build PID model as Actor i G Model, using deep neural network to build initial critic i G Model; secondly, fixed Actor i G The model parameters remain unchanged, in the state subspace S i G In the code, we use the Actor-Critic deep reinforcement learning method to train the Critic i G model until its performance converges;

[0051] 3) For the state subspace set in step 2 Each state subspace in Do the following: If the state subspace Valid PID parameters have been set, then end step 3); if the state subspace If no valid PID parameters are set, search the adjacent state subspace. If no valid PID parameters are set in the adjacent state subspace, end step 3); if the state subspace If no valid PID parameters are set, and there is ≥1 state subspace in its adjacent state subspace that has valid PID parameters set, first select an adjacent state subspace that has valid PID parameters set, and take the Actor model and Critic model of the adjacent state subspace as the initial Actor model and initial Critic model of the current state subspace; then, in the state subspace Actor-Critic reinforcement learning method is used to train and Model until they all converge; finally, take out The model parameters are used as the current state subspace Valid PID parameters;

[0052] 4) Repeat step 3) until all state subspaces have been set with valid PID parameters.

[0053] Step 4: Synthesize the PID control parameters of the parafoil in the current flight state. The specific implementation steps are as follows:

[0054] 1) Take O t ∈R P Indicates the current state of the parafoil during flight;

[0055] 2) Calculate the current state O t With the state subspace in step 2 The similarity between them is calculated as shown in equation (1):

[0056]

[0057] In equation (1) represents the vector inner product, "||·||" represents the modulus of the orientation quantity, Represents the state subspace Representative status;

[0058] 3) Get the current state O t The effective PID parameters of the most similar K state subspaces (K=5 in this embodiment, and selected as needed in other embodiments) constitute a set The corresponding K similarities constitute a set The current state is O t The synthesis results of the PID control parameters are as follows:

[0059]

[0060] Step 5, iteratively executing step 4 during the motion of the parafoil system, and using the PID parameters synthesized in step 4 to perform PID control of the parafoil motion. In this embodiment, step 5 is implemented in a simulation environment and a real environment respectively.

[0061] The above are only preferred embodiments of the present invention and do not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the present invention specification should be included in the protection scope of the present invention.

Claims

1. A PID adaptive control method for parafoil motion, characterized in that: The steps include: Step 1: Select P key characteristic factors that affect the dynamic performance of the parafoil to construct the state space S , and divide the state space into multiple state subspaces; Step 2: Use deep reinforcement learning method to obtain effective PID parameters for each state subspace; Step 3: Select K state subspaces that are most similar to the current state of the parafoil flight, and use the effective PID parameters under the K state subspaces to synthesize the PID control parameters under the current flight state of the parafoil to complete the motion control of the parafoil system; Among them, the specific implementation steps in step 3 are: Step 3.1, O t ∈ R P Indicates the current state of the parafoil during flight; Step 3.2: Calculate the current state O t and state subspace S i I The similarity between ,in, i =1,2,3,..., N ; Step 3.3, get the current state O t Most similar K state subspace S j I The valid PID parameters constitute the set { PID j I } j=1,2,3,...,K ,in, j =1,2,3,..., K , corresponding to K Similarities constitute the set { d ( O t , S j I )} j=1,2,3,...,K ， The current state O t The synthesis results of the PID control parameters are as follows: ； Step 3.4: During the motion of the parafoil system, steps 3.1, 3.2, and 3.3 are iteratively executed in sequence to synthesize the PID parameters of the current state of the parafoil system and complete the motion control of the parafoil system.

2. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: The method for constructing the state space in step 1 is: Select the factors that affect the dynamic performance of the parafoil P Key characteristic factors constitute the state space S ⊂ R P , P ∈{1,2,3,4,5,6}.

3. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: The key characteristic factors in step 1 include but are not limited to the wind direction angle, wind force, and load multiple of the wind field in the parafoil flight environment.

4. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: The method for dividing the state subspace in step 1 is: The state space S Divided into N state subspaces, forming a state subspace set I ={ S i I } i=1,2,3,...,N ; The state space S of j The dimension value range is divided into n j subintervals, where j =1,2,3,... P , then the state space S Number of divided state subspaces N = n 1* n 2* n 3*...* n P ; In each state subspace S i I Select a state T i I ∈ R P ,in, i =1,2,3,..., N， As S i I Representative states in the state subspace.

5. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: In step 2, the Actor-Critic deep reinforcement learning method is used to obtain the effective PID parameters for each state subspace.

6. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Assume that L The representative state and effective PID parameters of the state subspace, where L≥1, let the L The state subspace set is G ={ S i G } i=1,2,3,...,L , the valid PID parameter set is { PID i G } i=1,2,3,...,L ; Step 2.2: Use Actor-Critic Deep Reinforcement Learning as a Set G Each state subspace in trains the Actor model and the Critic model; Step 2.3: For each state subspace S i I Do the following: If the state subspace S i I Valid PID parameters have been set, then end step 2.3; if the state subspace S i I If no valid PID parameters are set, search the adjacent state subspace. If no valid PID parameters are set in the adjacent state subspace, end step 2.

3. If the state subspace S i I If no valid PID parameters are set, and at least one state subspace in its adjacent state subspace has valid PID parameters set, first select an adjacent state subspace that has valid PID parameters set, and take the Actor model and Critic model of the adjacent state subspace as the initial Actor model and initial Critic model of the current state subspace; then, in the state subspace S i I Actor-Critic reinforcement learning method is used to train Actor i I and Critic i I Model until both performance converges; finally, take out Actor i I The model parameters are used as the current state subspace S i I Valid PID parameters; Step 2.4: Repeat step 2.3 until all state subspaces have been set with valid PID parameters.

7. The PID adaptive control method for parafoil motion according to claim 6, characterized in that: The specific method of step 2.2 is: First, the set G Any state subspace in S i G ,in i =1,2,3,..., L , using the parameter PID i G Constructing PID model as Actor i G Model, in, i =1,2,3,..., L Using deep neural networks to build initial Critic i G Model; secondly, fixed Actor i G The model parameters remain unchanged in the state subspace S i G Using Actor-Critic deep reinforcement learning method to train Critic i G model until its performance converges.

8. The PID adaptive control method for parafoil motion according to claim 1, characterized in that: The calculation formula for similarity in step 3.2 is: ； Where, " represents the vector inner product, " " represents the modulus of the orientation quantity, T i I Represents the state subspace S i I Representative status.

Images