A fixed-wing attitude control method based on equivariant reinforcement learning

Abstract

The invention discloses a fixed-wing attitude control method based on equivariant reinforcement learning, comprising the following steps: S1: establishing a fixed-wing model based on fixed-wing kinematic and dynamic equations under a plane earth assumption; S2: setting assumption conditions to establish an approximate heading mirror-symmetric structure of the fixed-wing model in step S1; S3: defining an observation space, an action space and a reward function of reinforcement learning; S4: selecting an Actor-Critic architecture reinforcement learning algorithm as a training algorithm, and designing a reinforcement learning neural network structure by using an approximate heading mirror-symmetric structure; S5: building a simulation environment, training the reinforcement learning algorithm, and finally using the Actor neural network model in the reinforcement learning algorithm as a controller deployment algorithm; the invention alleviates the problem of low efficiency of reinforcement learning samples, and can be combined with an Actor-Critic reinforcement learning algorithm of any depth to accelerate its training speed in fixed-wing control problems.

Description

Technical Field

[0001] The present invention belongs to the technical field of fixed-wing attitude control, and in particular, relates to a fixed-wing attitude control method based on equivariant reinforcement learning. Background Art

[0002] In recent years, more and more studies have begun to apply reinforcement learning to solve the problem of fixed-wing attitude control. Compared with traditional control methods, reinforcement learning does not require gain scheduling, can directly train the controller within the full envelope, and has a high upper limit on the control effect. However, the reinforcement learning method fits the strategy and value function through a deep neural network, which has the problem of low sample utilization efficiency and requires a large amount of simulated interaction data to train the algorithm network. For underactuated fixed-wing aircraft that are difficult to control, it is crucial to improve the training efficiency of the reinforcement learning algorithm and complete the learning safely with the least number of trials.

[0003] Equivariance means that there exists a transformation g. For a function h, h(g(x))=g(h(x)), then h is said to be equivariant with respect to g, that is, applying a transformation to the input of a function will also be reflected in the output. Equivariant reinforcement learning aims to utilize the equivariant properties of the environmental model so that the output actions of the intelligent agent also satisfy certain transformation properties when facing equivariant environmental transformations. For example, for a geometrically symmetric environment, when the state space undergoes a symmetric transformation, in order to maintain the overall geometric symmetry of the environment, the action of the intelligent agent also needs to undergo a corresponding transformation in order to maintain an invariant equivalence relationship with the state space.

[0004] Although this field is still in the research stage, some results have shown that equivariant reinforcement learning has potential advantages in improving model generalization and accelerating algorithm training speed, especially in the field where the state space is an image. Discovering and utilizing the equivariant structure in fixed-wing attitude control, accelerating the algorithm training speed based on equivariant reinforcement learning, and reducing the demand for interactive data are the problems that need to be solved at present. Summary of the invention

[0005] In view of the above defects, the present invention provides a fixed-wing attitude control method based on equivariant reinforcement learning, comprising the following steps:

[0006] S1: Under the flat earth assumption, a fixed-wing model is established based on the fixed-wing kinematic and dynamic equations;

[0007] S2: setting assumptions and establishing an approximate heading mirror-symmetric structure of the fixed-wing model in step S1;

[0008] S3: Define the observation space, action space, and reward function of reinforcement learning;

[0009] S4: Select the Actor-Critic architecture reinforcement learning algorithm as the training algorithm, and use the approximate heading mirror symmetric structure to design the reinforcement learning neural network structure;

[0010] S5: Build a simulation environment, train the reinforcement learning algorithm, and finally use the Actor neural network model in the reinforcement learning algorithm as the controller to deploy the algorithm.

[0011] Furthermore, the fixed-wing model in step S1 is a six-degree-of-freedom fixed-wing model under the flat earth assumption.

[0012] Furthermore, the assumptions of step S2 include:

[0013] Assume that the angular momentum generated by rotating parts such as the engine is ignored;

[0014] It is assumed that the aerodynamic force coefficients and moment coefficients accurately conform to the physical laws and there is no error.

[0015] Furthermore, the observation space of the reinforcement learning in step S3 is defined as:

[0016]

[0017] In the above formula, v t is the fixed-wing airspeed, α is the angle of attack, β is the sideslip angle, φ is the roll angle, θ is the pitch angle, p is the roll angle rate, q is the pitch angle rate, r is the yaw angle rate, h is the height in the fixed-wing body axis coordinate system, P a is the actual thrust level, e φ is the roll angle error, e θ is the pitch angle error, e ψ is the yaw angle error;

[0018] The action space of reinforcement learning is defined as:

[0019] a=[δ t , δ e , δ a , δ r ];

[0020] In the above formula, δ t is the throttle amount, δ e is the elevator deflection angle, δ a is the aileron deflection angle, δ r is the rudder deflection angle;

[0021] The reward function for reinforcement learning is defined as:

[0022] R(t)=r φ (t)+r θ (t)+r ψ (t),

[0023]

[0024] In the above formula, * = {φ, θ, ψ},

[0025] Δ* max is the maximum expected control output increment,

[0026] Furthermore, the Actor-Critic architecture reinforcement learning algorithm adopted in step S4 may be any deep Actor-Critic algorithm.

[0027] Furthermore, the Actor-Critic architecture reinforcement learning algorithm in step S4 includes an Actor neural network and a Critic neural network. The Actor neural network and the Critic neural network only fit the strategy and value function on one side of the heading symmetry plane, and the strategy and value function on the other side of the heading symmetry plane are calculated from the fitting side according to the equivariant structure of the fixed-wing model. The working processes of the Actor and the Critic are respectively:

[0028] The Actor neural network first judges the input. If e ψ ≥0, then directly The output is calculated by neural network 1 and neural network 2 in turn Otherwise, for the input After M transformation, we get After neural network 1 and neural network 3, we get Then after M transformation, the output is obtained

[0029] Critic neural network first judges the input. If e ψ ≥0, then directly or Input neural network 4 to get the value function, otherwise, or After M transformation, we get or Input neural network 4 to obtain the value function;

[0030] The M transformation is used to transform the relevant variables from one side of the heading symmetry plane to the other side;

[0031] Neural network 1 is a fully connected neural network with two layers of 128 nodes and a GRU neural network with one layer of 128 nodes, and the activation function is ReLU. Neural network 2 and neural network 3 are fully connected neural networks with two layers of 128 nodes, and the activation function is ReLU. Neural network 4 is a fully connected neural network with two layers of 128 nodes, a GRU neural network with one layer of 128 nodes, and a fully connected neural network with two layers of 128 nodes, and the activation function is ReLU.

[0032] Furthermore, the neural network structure in step S4 may include a feedforward neural network and / or a recurrent neural network.

[0033] Furthermore, the simulation environment in step S5 includes an environment initialization module for reinforcement learning and simulation environment parameter setting, a single-game simulation link for simulation training, and an algorithm training completion judgment module for judging whether the current algorithm training is completed;

[0034] The single-game simulation link includes an environment reset module, an action selection module, a model solution module, an algorithm update module and a current simulation end judgment module.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The approximate heading mirror-symmetric structure in fixed-wing control was discovered, and the equivariant properties of the model brought by the approximate heading mirror-symmetric structure were used to speed up the training speed of reinforcement learning, alleviate the problem of low efficiency of reinforcement learning samples, and can be combined with arbitrary depth Actor-Critic reinforcement learning algorithm to speed up its training speed in fixed-wing control problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the fixed-wing attitude control method based on equivariant reinforcement learning in the present invention;

[0038] Figure 2 It is a schematic diagram of the approximate heading mirror symmetry in the present invention;

[0039] Figure 3 This is a structural diagram of a reinforcement learning neural network in the present invention;

[0040] Figure 4 It is a flow chart of the operation of the simulation environment in the present invention;

[0041] Figure 5 A comparison chart of the training reward curves of the algorithm in the present invention and the common reinforcement learning algorithm;

[0042] Figure 6 is a schematic diagram of the roll angle control effect in the present invention;

[0043] Figure 7 A schematic diagram of the pitch angle control effect in the present invention;

[0044] Figure 8 It is a schematic diagram of the yaw angle control effect in the present invention. DETAILED DESCRIPTION

[0045] In order to facilitate the understanding of the present invention, the device of the present invention will be described more fully below with reference to the relevant drawings. Embodiments of the device are given in the drawings. However, the device can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Example

[0048] like Figure 1-2 As shown, this embodiment provides a fixed-wing attitude control method based on equivariant reinforcement learning, comprising the following steps:

[0049] S1: Under the flat earth assumption, a fixed-wing model is established based on the fixed-wing kinematic and dynamic equations;

[0050] For the attitude control of the fixed wing, the horizontal position of the fixed wing is not of concern, so the horizontal position equation of the fixed wing is ignored. At the same time, it is assumed that the earth is a plane, which is more suitable for studying maneuvering flight on a small area of ​​the earth. It is also applicable to the design of the attitude controller in this embodiment, where:

[0051] The kinematic and dynamic equations of the fixed wing are:

[0052]

[0053] It should be noted that the origin of the fixed-wing body axis coordinate system is located at the center of mass of the model, the OX axis is in the fixed-wing body symmetry plane, parallel to the longitudinal centerline of the fixed wing and points forward, the OZ axis is also in the body symmetry plane and perpendicular to the OX axis, and the OY axis is perpendicular to the body symmetry plane and points to the right;

[0054] v B =[v x , v y , v z ]T is the velocity in the fixed-wing body axis coordinate system, w B =[p,q,r] T is the angular velocity in the fixed-wing body axis coordinate system, Φ = [φ, θ, ψ] T is the Euler angle describing the fixed-wing attitude, h is the height in the fixed-wing body axis coordinate system, They are v B , w B , the derivative of Φ,h;

[0055] is the rotation matrix from the north-east ground coordinate system to the fixed-wing body axis system, w B arrive The transformation matrix of

[0056] m is the mass of the fixed wing, g is the acceleration due to gravity, g = [0, 0, g] T is the gravity acceleration vector in the north-east coordinate system, J is the inertia matrix, which implies that the fixed wing is symmetrical about the plane formed by the X-axis and Z-axis of the body axis system, which is true for general fixed-wing aircraft. -1 is the inverse matrix of J, x =∫(y 2 +z 2 )dm,J y =∫(x 2 +z 2 )dm,J z =∫(x 2 +y 2 )dm is the moment of inertia of the three coordinate axes of the body axis system, J xz =∫xzdm is the inertia product between the x and z axes of the body axis system, H B =[H x , H y , H z ] T is the angular momentum generated by the engine rotation. For a single-engine small jet aircraft in maneuvering flight, H B is often not negligible, F B =[F x , F y , F z ] T , T B =[T x , T y , T z ] T are the forces and moments in the fixed-wing body axis coordinate system respectively;

[0057] The engine thrust T is determined by the actual thrust level P a, h and Mach number Ma, the direction points forward along the body axis X direction, in order to fit the actual situation, the actual thrust level P a With the expected thrust level P c The relationship between is modeled as a first-order delay model:

[0058]

[0059] Where P c By δ t OK, δ t is the throttle amount, τ eng is the time constant.

[0060] F B , T B The calculation is done using the following formula:

[0061]

[0062] in ρ is the air density, v t is the fixed-wing airspeed, S is the wing reference area, b is the wing span, is the average geometric chord length of the wing, C x , C y , C z is the aerodynamic coefficient in three directions of the fixed wing body axis coordinate system, C l , C m , C n is the aerodynamic moment coefficient in three directions of the fixed wing body axis coordinate system, C x , C y , C z , C l , C m , C n The determination of is carried out by using the Morelli polynomial fitting scheme, which can be determined according to "Morelli E A. Global nonlinear parametric modelling with application to F-16 aerodynamics [C] / / Proceedings of the 1998 American Control Conference. IEEE, 1998, 2: 997-1001.";

[0063] Fixed wing airspeed v t , the angle of attack α and the sideslip angle β are related to v B =[v x , v y , v z ] T The relationship is as follows:

[0064]

[0065] The fixed-wing state variable is x = [v t , α, β, φ, θ, ψ, p, q, r, h, P a ] T , the control variable is u=[δ t , δ e , δ a , δ r ] T , where δ e , δ a , δ r They are the elevator deflection angle, aileron deflection angle and rudder deflection angle respectively. The fixed-wing kinematic and dynamic equations can be written as first-order differential equations Further write the fixed-wing state transfer equation as x t+1 =F(x t ,u t ), the state transfer equation can be obtained by integrating the differential equation;

[0066] In this embodiment, the fixed wing mass m = 9298.65 kg, the gravity acceleration Wing reference area S = 27.87m 2 , wing span b = 9.14m, wing average geometric chord length Substituting into the above equation we get:

[0067] The inertia matrix is:

[0068]

[0069] The control input range is δ t ∈[0, 1], δ e ∈[-25°, +25°],δ a ∈[-21.5°, +21.5°],δ r ∈[-30°, +30°];

[0070]

[0071] It should be noted that the aircraft model data in this embodiment uses the publicly available F-16 aircraft data;

[0072] S2: Set up assumptions and establish an approximate heading mirror-symmetric structure for fixed-wing aircraft;

[0073] The mirror mapping M of the heading symmetry plane is used to transform the variables on one side of the heading symmetry plane to the other side.

[0074] Assume that the angular momentum generated by the rotating parts such as the engine is ignored, that is, HB =[0,0,0] T ; Assuming the aerodynamic coefficient C x , C y , C z and moment coefficient C l , C m , C n It conforms to the laws of physics accurately and has no error, that is, M(C y )=-C y ,M(C l )=-C l ,M(C n )=-C n Based on the above assumptions, the fixed-wing model is symmetrical about the plane composed of the desired heading and the vertical direction, that is, the mirror plane of the heading symmetry plane. The heading mirror symmetry structure is defined as an approximate heading mirror symmetry structure. See the schematic diagram. Figure 2 .

[0075] If the state variables and control inputs that are symmetric about the heading symmetry plane at the current moment are still symmetric about the heading symmetry plane after solving the fixed-wing state transfer equation, it can be explained that there is an approximate heading mirror symmetric structure.

[0076] The desired attitude control output is φ reference ,θ reference , ψ reference , define the mapping of state variable x and control variable u on the heading symmetry plane as Then there is an approximate heading mirror symmetry structure in the fixed-wing attitude control: under the above two assumptions, for x on either side of the heading symmetry plane t ,u t ,like but

[0077] The approximately heading mirror-symmetric structure shows that the fixed-wing state transfer equation is equivariant with respect to the mapping M.

[0078] In applying reinforcement learning to solve the problem of fixed-wing attitude control, the algorithm needs to learn the mapping from observations to fixed-wing control inputs. The equivariant property of the model brought by the approximate heading mirror-symmetric structure shows that if the control input that the fixed-wing on one side of the heading symmetry plane should take from the current observation to the desired heading is found, then the control input that the fixed-wing on the other side that is mirror-symmetric should take differs from the control input on the current side by a mapping M. Therefore, for the reinforcement learning algorithm, it only needs to learn the control input on one side of the symmetry plane, recorded as the learning side, and the control input on the other side only needs to be obtained by mapping M using the control input obtained from the corresponding learning side. This is equivalent to simplifying the original control learning problem. Compared with learning the control input on the entire observation space, the number of samples required is smaller, which improves the sample utilization efficiency and can speed up the algorithm training speed.

[0079] S3: Define the observation space, action space, and reward function of reinforcement learning:

[0080] The observation space of reinforcement learning is designed as:

[0081] o=[v t ,α,β,φ,θ,p,q,r,h,P a , e φ , e θ , e ψ ];

[0082] In order to avoid large differences in the magnitude of different parameters, the observation space is t ,h,P a After scaling, we get:

[0083]

[0084] In the above formula, v t is the fixed-wing airspeed, α is the angle of attack, β is the sideslip angle, φ is the roll angle, θ is the pitch angle, p is the roll angle rate, q is the pitch angle rate, r is the yaw angle rate, h is the height in the fixed-wing body axis coordinate system, P a is the actual thrust level, e φ is the roll angle error, e θ is the pitch angle error, e ψ is the yaw angle error,

[0085] The angles in the observation space are all in radians, and the error e φ =φ-φ reference , e θ =θ-θ reference , e ψ =ψ-ψ reference ,

[0086] The action space of reinforcement learning is designed as: a = [δ t , δ e , δ a , δ r ],

[0087] In the above formula, δ t is the throttle amount, δ e is the elevator deflection angle, δ a is the aileron deflection angle, δ r is the rudder deflection angle,

[0088] After scaling up, we get

[0089] Mapping of observation space:

[0090] M(o)=[v t ,α,-β,-φ,θ,-p,q,-r,h,P a , -e φ , e θ , -e ψ ],

[0091] Mapping of action space:

[0092] M(a)=[δ t , δ e , -δ q , -δ r ],

[0093] The goal of reinforcement learning is to find a strategy in the strategy space to maximize the expected cumulative discounted reward. In order to accelerate algorithm training by using the approximate heading mirror symmetric structure, it is necessary to give the same reward signal to the state-action trajectory symmetric about the heading symmetry plane. Therefore, the design of the reward function R(·) must satisfy R(o, a) = R(M(o), M(a)). In addition, R(o, a) is designed to be equal to abs(e φ ), abs(e θ ), abs(e ψ ) is inversely proportional to the size of abs(e φ ), abs(e θ ), abs(e ψ ) is smaller, the larger the reward R(0, a) is. According to this rule, the reward function is designed as:

[0094] R(t)=r φ (t)+r θ (t)+r ψ (t),

[0095]

[0096] where * = {φ, θ, ψ},

[0097] Δ* max is the maximum expected control output increment,

[0098] S4: Select the Actor-Critic architecture reinforcement learning algorithm as the training algorithm, and use the approximate heading mirror symmetric structure to design the reinforcement learning neural network structure:

[0099] It should be noted that Actor-Critic is a mainstream reinforcement learning algorithm architecture. The algorithm parameterizes both the policy function and the value function, which act as actors and critics respectively. The value function is updated by the data collected from the interaction between the policy function and the environment, and the policy function is updated by the evaluation guidance of the value function. The two are updated alternately until the algorithm converges. Common algorithms include the proximal policy optimization algorithm (i.e., the Proximal Policy Optimization, PPO algorithm) and the soft Actor-Critic (Soft Actor-Critic, SAC) algorithm. In this embodiment, the PPO algorithm is used for training. The training parameters of the PPO algorithm are shown in the following table:

[0100] parameter Value Reward discount silver 0.99 Learning Rate 0.00003 Number of parallel environments 32 Batch size 500 Entropy coefficient 0.01

[0101] This embodiment uses the fixed-wing approximate heading mirror symmetric structure to accelerate the training speed of the reinforcement learning algorithm based on equivariant reinforcement learning. The equivariant reinforcement learning algorithm only changes the neural network structure of the Actor-Critic architecture reinforcement learning algorithm, and does not change other parts. Therefore, any Actor-Critic architecture deep reinforcement learning algorithm can be selected for simulation training, specifically:

[0102] There are two assumptions for the fixed-wing approximate heading mirror symmetry structure: ignoring the angular momentum generated by rotating parts such as the engine; the aerodynamic coefficient C x , C y , C z and moment coefficient C l , C m , C n Accurately conform to the laws of physics; the two assumptions are difficult to meet in actual situations. Based on the fixed-wing approximate heading mirror symmetry structure, the reinforcement learning neural network structure is designed, and the neural network is improved to make the approximate heading mirror symmetry structure free from the constraints of the two assumptions;

[0103] The neural network structure of mainstream Actor-Critic architecture reinforcement learning consists of two parts: Actor neural network and Critic neural network. Actor neural network is used to fit the policy function and establish the mapping between observation o and action a. Critic neural network is used to fit the value function and establish the mapping between observation o or observation action pair (o, a) and value.

[0104] The state variables on both sides of the heading symmetry plane can be calculated according to e ψ ≥0 is used to distinguish whether it is true or not. The basic idea of ​​designing the reinforcement learning neural network structure based on the fixed-wing approximate heading mirror symmetric structure is to only focus on e as much as possible. ψ ≥0 input for fitting, for e ψ The input of <0 uses the equivariant property of the model to get the corresponding output. The structure diagram of Actor neural network and Critic neural network is as follows Figure 3 As shown;

[0105] That is, the Actor neural network first judges the input. If e ψ ≥0, then directly The output is calculated by neural network 1 and neural network 2 in turn Otherwise, for the input After M transformation, we get After neural network 1 and neural network 3, we get Then after M transformation, the output is obtained In order to make the most of the equivariant properties of the model, the Actor neural network makes the input and The neural network 1 is shared. At the same time, in order to alleviate the problems caused by the two assumptions, and In the input situation, neural network 2 and neural network 3 were used separately. It should be noted that neural network 1 is a fully connected neural network including two layers of 128 nodes and a GRU (Gated Recurrent Unit) neural network with 128 nodes in sequence, and the activation function is ReLU. Neural network 2 and neural network 3 are two layers of fully connected neural networks with 128 nodes, and the activation function is ReLU.

[0106] Critic neural network first judges the input. If e ψ ≥0, then directly or Input neural network 4 to get the value function, otherwise, or After M transformation, we get or Input neural network 4 to obtain the value function; in order to make the most of the equivariant properties of the model, the Critic neural network is equivalent to only or It should be noted that the neural network 4 includes, in sequence, a two-layer fully connected neural network with 128 nodes, a one-layer GRU neural network with 128 nodes, and a two-layer fully connected neural network with 128 nodes, and the activation function is ReLU;

[0107] It should be noted that the M transformation is used to transform the relevant variables from one side of the heading symmetry plane to the other side;

[0108] Among them, neural network 1, neural network 2, neural network 3 and neural network 4 are used to establish a mapping relationship between high-dimensional continuous vector spaces in fixed-wing attitude control, and the neural network structure may include a feedforward neural network and / or a recurrent neural network.

[0109] S5: Build a simulation environment, train the reinforcement learning algorithm, and finally use the Actor neural network model in the reinforcement learning algorithm as the controller to deploy the algorithm:

[0110] The simulation environment includes an environment initialization module, a single-game simulation link, and an algorithm training end judgment module, wherein the single-game simulation link includes an environment reset module, an action selection module, a model solution module, an algorithm update module, and a current simulation end judgment module;

[0111] Initialize the module for reinforcement learning and simulation environment parameter settings:

[0112] Fixed wing initial state variables The remaining state variables α, β, φ, θ, p, q, and r are all 0, and the expected roll, pitch, and yaw output values ​​are in the range of *+Uniform(-Δ* max , +Δ* max ), where *={φ, θ, ψ},

[0113] A single round of simulation is carried out in the single-round simulation link, in which the environment reset module sets the fixed-wing state to the initial state at the beginning of each round of simulation, the action selection module selects an action from the action space according to the reinforcement learning algorithm, the model solving module solves the state variables of the fixed-wing at the next moment according to the selected action and the state variables of the fixed-wing at the previous moment, and returns the reward signal, the algorithm update module updates the reinforcement learning algorithm, the current simulation end judgment module is used to check whether the current simulation reaches the termination condition; the algorithm training end judgment module is used to judge whether the current algorithm training is finished, in which the model solving module adopts the Euler method for model solving, and the simulation step length is 0.05 seconds; the simulation end judgment module includes two simulation end judgments, namely, reaching the maximum simulation step length of 100 steps, and the fixed-wing overload is greater than 10, and the fixed-wing overload greater than 10 means that the fixed-wing reaches an extreme state and crashes.

[0114] The simulation environment workflow is as follows Figure 4 As shown, after the simulation starts, it enters the single-game simulation link after passing through the initialization module. The single-game simulation link passes through the environment reset module, the action selection module, the model solution module, the algorithm update module, and the current simulation end judgment module in sequence. If the simulation end judgment module returns yes, then the single-game simulation link is exited, otherwise it enters the action selection module, and after exiting from the single-game simulation module, it enters the algorithm training end judgment module. If the algorithm training end judgment module returns yes, then the simulation is ended, otherwise it re-enters the single-game simulation link;

[0115] The comparison of the reward curve of the PPO algorithm training using the approximate heading mirror symmetric structure and the reward curve of the PPO algorithm training without the approximate heading mirror symmetric structure is shown in the figure below. Figure 5 As shown, it can be seen that compared with the ordinary PPO algorithm, the use of an approximate heading mirror symmetric structure can speed up the training of the algorithm, and the final convergence reward value is also relatively large.

[0116] The final controller model is the trained equivariant reinforcement learning algorithm Actor neural network model. In actual use, the Actor neural network model is loaded, the current observation of the algorithm is used as input, and the output is the fixed-wing control input [δ t , δ e , δ a , δ r ],like Figure 6-Figure 8 As shown, the expected output is φ reference =1.0295rad,θ reference =-0.0317rad,ψ reference =0.1558rad, the control effect of the roll angle, pitch angle and yaw angle is improved. It can be seen that the algorithm can quickly reach the expected output, which verifies the control effect of the algorithm.

[0117] It should be noted that the structure described in the present invention can be implemented in a variety of different forms and is not limited to the described embodiments. Any equivalent transformations made by ordinary technicians in this field using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A fixed-wing attitude control method based on equivariant reinforcement learning, characterized in that: The following steps are involved: S1: Under the flat earth assumption, a fixed-wing model is established based on the fixed-wing kinematic and dynamic equations; S2: Set assumptions, establish an approximate heading mirror symmetric structure of the fixed-wing model in step S1, and design a mapping M to transform the state and control variables on both sides of the heading symmetry plane, including: The mirror mapping M of the heading symmetry plane is used to transform the variables on one side of the heading symmetry plane to the other side; Assuming that the angular momentum generated by the rotating parts of the engine is ignored, and assuming that the aerodynamic force coefficient and the moment coefficient accurately conform to the physical laws and there is no error, based on the above assumptions, the fixed-wing model is symmetrical about the plane composed of the desired heading and the vertical direction, that is, the mirror plane of the heading symmetry plane, then the heading mirror symmetric structure is defined as an approximate heading mirror symmetric structure; The approximately heading mirror-symmetric structure shows that the fixed-wing state transfer equation is equivariant with respect to the mapping M; S3: Define the observation space, action space, and reward function of reinforcement learning; S4: Select the Actor-Critic architecture reinforcement learning algorithm as the training algorithm, and use the approximate heading mirror symmetric structure to design the reinforcement learning neural network structure, including: S4-1: The Actor neural network first judges the input. If e ψ ≥0, then directly The output is calculated by neural network I and neural network II in turn. Otherwise, for the input After M transformation, we get After neural network I and neural network III, we can get After M transformation, the output is S4-2: Critic neural network first judges the input. If e ψ ≥0, then directly Input neural network IV to get the value function, otherwise, After M transformation, we get Input neural network IV to get the value function; Wherein, neural network I, neural network II, neural network III and neural network IV are used to establish a mapping relationship between high-dimensional continuous vector spaces in fixed-wing attitude control, and the neural network structure may include a feedforward neural network and / or a recurrent neural network; S5: Build a simulation environment, train the reinforcement learning algorithm, and finally use the Actor neural network model in the reinforcement learning algorithm as the controller to deploy the algorithm.

2. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 1, characterized in that: The fixed-wing model in step S1 is a six-degree-of-freedom fixed-wing model under the assumption of a flat earth.

3. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 1, characterized in that: The observation space of the reinforcement learning in step S3 is defined as: In the above formula, v t is the fixed-wing airspeed, α is the angle of attack, β is the sideslip angle, φ is the roll angle, θ is the pitch angle, p is the roll angle rate, q is the pitch angle rate, r is the yaw angle rate, h is the height in the fixed-wing body axis coordinate system, P a is the actual thrust level, e φ is the roll angle error, e θ is the pitch angle error, e ψ is the yaw angle error; The action space of reinforcement learning is defined as: a=[δ t ,d e ,d a ,d r ]; In the above formula, δ t is the throttle amount, δ e is the elevator deflection angle, δ a is the aileron deflection angle, δ r is the rudder deflection angle; The reward function for reinforcement learning is defined as: R(t)=r φ (t)+r θ (t)+r ψ (t), In the above formula, *={φ,θ,ψ}, Δ* max is the maximum expected control output increment, 4. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 1, characterized in that: The Actor-Critic architecture reinforcement learning algorithm adopted in step S4 is any deep Actor-Critic algorithm.

5. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 4, characterized in that: The Actor-Critic architecture reinforcement learning algorithm in step S4 includes an Actor neural network and a Critic neural network, and the Actor neural network and the Critic neural network only fit the strategy and value function on one side of the heading symmetry plane, and the strategy and value function on the other side of the heading symmetry plane are calculated from the fitting side according to the fixed-wing model equivariant structure; Neural network I includes, in sequence, two layers of fully connected neural networks with 128 nodes and one layer of GRU neural network with 128 nodes, and the activation function is ReLU. Neural network II and neural network III are two layers of fully connected neural networks with 128 nodes, and the activation function is ReLU. Neural network IV includes, in sequence, two layers of fully connected neural networks with 128 nodes, one layer of GRU neural network with 128 nodes, and two layers of fully connected neural networks with 128 nodes, and the activation function is ReLU.

6. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 1, characterized in that: The neural network structure in step S4 includes a feedforward neural network and / or a recurrent neural network.

7. The fixed-wing attitude control method based on equivariant reinforcement learning according to claim 1, characterized in that: The simulation environment in step S5 includes an environment initialization module for reinforcement learning and simulation environment parameter setting, a single-game simulation link for simulation training, and an algorithm training completion judgment module for judging whether the current algorithm training is completed; The single-game simulation link includes an environment reset module, an action selection module, a model solution module, an algorithm update module and a current simulation end judgment module.

Images