Variable configuration aircraft intelligent attitude control method and system based on multi-agent reinforcement learning

Abstract

The invention provides a variable configuration aircraft intelligent attitude control method and system based on multi-agent reinforcement learning, and belongs to the field of intelligent control. The method aims to solve the problems that under a complex control task, a variable-configuration hypersonic aircraft model is complex and nonlinear, and attitude control performance is not fully exerted. According to the multi-agent system-based intelligent attitude control method, end-to-end variable-configuration aircraft attitude control is performed based on multi-agent reinforcement learning, and a multi-agent cooperation mode is adopted to further improve the exploration capability and control the elasticity; training and learning are carried out in the same environment, and distributed decision making is carried out according to respective states, so that an effective intelligent control mode is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a method and system for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning. Background Art

[0002] In recent years, with the growing demand for round-trip transportation between the earth and the sky, traditional fixed-configuration hypersonic aircraft have difficulty adapting to various mission requirements at different flight ranges and altitudes due to their limited performance range and single mission types. Variable-configuration hypersonic aircraft can adjust their configuration according to different mission requirements and flight environments by changing their own shape, thereby maintaining the best aerodynamic shape throughout the flight, achieving an optimal balance of key parameters such as lift, flight speed, and energy consumption, while better meeting external constraints such as heat flow and dynamic pressure. This will help support future efficient transportation missions at different altitudes inside and outside the atmosphere.

[0003] However, while variable-configuration hypersonic vehicles have a larger flight envelope and stronger maneuverability, the changes in inertia and aerodynamic parameters caused by their structural changes also pose new challenges to the control system. The attitude control problems of such aircraft include methods of designing separate controllers for the longitudinal and lateral planes to simplify the model complexity, and using fuzzy disturbance observers, neural network observers and other technologies to deal with model uncertainty and environmental disturbances. In addition, nonlinear control methods such as sliding mode controllers and backstepping methods have also been successfully applied to the attitude control of such aircraft.

[0004] In order to further improve the performance boundary and adaptive capability of attitude control of variable-configuration hypersonic vehicles, various optimization methods and intelligent methods have been introduced. Reinforcement learning has been widely used in this field due to its powerful exploration and learning capabilities and data-driven characteristics that do not rely on models.

[0005] Most existing attitude control methods rely on a single agent to make decisions. In order to cope with more complex control tasks, the use of multi-agent collaboration can further improve exploration capabilities and control flexibility. Therefore, this paper proposes an intelligent attitude control method based on a multi-agent system, which regards multiple control channels of a variable-configuration hypersonic vehicle as independent agents, trains and learns in the same environment, and makes distributed decisions based on their respective states to achieve a new type of intelligent control method. Summary of the invention

[0006] The technical problems to be solved by the present invention are:

[0007] In order to solve the problems of complex nonlinearity and insufficient attitude control performance of variable configuration hypersonic vehicle models under complex control tasks.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] The present invention provides a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning, comprising the following steps:

[0010] S100, establishing a training network for a variable configuration aircraft attitude control intelligent agent;

[0011] S200, with attitude control as the goal, determine the observation space and action space of the three-channel attitude control of the variable-configuration aircraft, regard the three control channels as three separate agents, and realize multi-agent distributed control of attitude control through interactive training in the same environment, and then merge the observation quantities of the three agents corresponding to the three channels respectively, and finally obtain their own independent observation spaces;

[0012] S300, a variable-structure aircraft attitude control decision network based on observation data training of multi-agent reinforcement learning, with design rewards including attitude convergence rewards Stable attitude reward Attitude Divergence Penalty and servo jitter penalty

[0013] Furthermore, in step S100, it specifically includes:

[0014] The soft actor-critic algorithm is used to train the intelligent posture controller; the action value function of each agent is expressed as Among them, θ represents the parameters of the action value function neural network; the strategy function is Among them, φ represents the parameters of the policy function neural network; o i represents the observation of agent i, a i Represents the execution action of agent i;

[0015] The strategy optimization process of the SAC algorithm is:

[0016]

[0017] in, represents the optimized strategy, ρ π Representative in Strategy The state transition distribution under Represents the intelligent agent i from the observation Next action Rewards received, Representative strategy The entropy of h Represents the entropy weight coefficient, which is used to balance the impact of reward and action entropy on the strategy; The observation (o) and action (a) of the ith agent at time t follow the state transition distribution, that is, the action is obtained after the state transition according to the observation; The · in it represents all possible actions;

[0018] During the execution of the SAC algorithm, the state value function of the observation is estimated as follows:

[0019]

[0020] in, Represents the value function of the ith agent when the observation value is o at time t; Represents the action amount of the i-th agent at time t that follows the strategy

[0021] Action-Value Function The estimation method is:

[0022]

[0023] Among them, γ represents the discount factor; Represents that the observation of the i-th agent at time t+1 follows the state transition distribution;

[0024] When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function:

[0025]

[0026] Among them, J Q (θ) represents the loss function; y i Represents the predicted action-value function of the ith agent; represents the reward of the i-th agent at time t; D represents the experience pool, which stores the experience of agent i The neural network parameters representing the target Q function are updated as follows:

[0027]

[0028] Where τ represents the soft update rate of the target Q function neural network parameters;

[0029] In the SAC algorithm, there are two Q functions: The two networks will be trained simultaneously, computing Use smaller values ​​to improve training stability;

[0030] The policy neural network parameters are learned by minimizing the following loss function:

[0031]

[0032] Further, in step S200: the observation space includes the configuration of the aircraft, the desired attitude angle, the current attitude angle, the current angular velocity and the steering gear instruction at the previous moment, and the action space includes the equivalent steering gear instruction of the aircraft.

[0033] Further, in step S300, the posture convergence reward It is used to encourage the aircraft's attitude to converge to the desired attitude. The closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, They are σ e |、|β e |、|α e |, where |σ e | is the absolute value of the roll angle deviation, |β e | is the absolute value of the sideslip angle deviation, |α e | is the absolute value of the angle of attack deviation; the specific design of this reward is:

[0034]

[0035] Further, in step S300, the posture stability reward It is used to make the attitude difference of the aircraft smaller to avoid excessive attitude oscillation. When the attitude angular velocity is less than a certain value This reward is provided when the reward is provided; this reward is designed to be coupled with the first reward, and the specific design of this reward is:

[0036]

[0037] Further, in step S300, the posture divergence penalty It is used to prevent the attitude controller from exploring dangerous areas. When the attitude error is greater than a certain range, this penalty is provided and the round ends at the same time. The danger value of the attitude angle error is set to The specific design of this reward is:

[0038]

[0039] Further, in step S300, the servo vibration penalty Used to prevent the servo from repeatedly shaking and reduce the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment Provide penalties based on the change in the servo state. The specific design is:

[0040]

[0041] Among them, δ i,max Represents the maximum value of the ith rudder deviation.

[0042] A configuration-changing aircraft intelligent attitude control system based on multi-agent reinforcement learning, the system has a program module corresponding to the above steps, and executes the steps in the configuration-changing aircraft intelligent attitude control method based on multi-agent reinforcement learning when running.

[0043] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning when called by a processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention regards the three-channel controller of the aircraft attitude as the interaction between the intelligent agent and the environment to learn the control strategy. Aiming at the complex nonlinearity of the variable configuration hypersonic aircraft model and the insufficient performance of attitude control, a variable configuration aircraft attitude control method based on multi-agent reinforcement learning is designed. Among them, each intelligent agent has an independent observation space and makes different rudder deviation instructions respectively, and the overall attitude stability control is achieved through distributed control. Through the performance verification of the configuration fixed / variable configuration process controller and the comparative analysis with various methods, the superiority of the method proposed in the present invention is verified.

[0046] The present invention can perform stable attitude control for the complex nonlinearity of the variable configuration hypersonic aircraft model, thereby ensuring the stable flight of the variable configuration hypersonic aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a framework diagram of a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning in an embodiment of the present invention;

[0048] Figure 2 This is a channel strategy network framework diagram in an embodiment of the present invention;

[0049] Figure 3 is a reward value curve diagram during the training process in an embodiment of the present invention;

[0050] Figure 4 is a step attitude angle command tracking curve diagram under a fixed configuration in an embodiment of the present invention;

[0051] Figure 5is a graph of angular velocity of a step attitude angle command under a fixed configuration in an embodiment of the present invention;

[0052] Figure 6 is a curve diagram of the step attitude angle command rudder deflection under the fixed configuration in an embodiment of the present invention;

[0053] Figure 7 is a continuous attitude angle command tracking curve diagram under a fixed configuration in an embodiment of the present invention;

[0054] Figure 8 is a graph showing the posture angle maintenance during the conformational transformation in an embodiment of the present invention;

[0055] Fig. 9 is a graph of equivalent rudder deflection angles during the conformational change process in an embodiment of the present invention;

[0056] Fig.10 is a graph of attitude angle response during the conformational change process in an embodiment of the present invention;

[0057] Fig.11 It is a comparative analysis diagram of the posture response of the conformational change process under different controllers in the embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] Specific implementation plan 1: Combine Figure 1 and Figure 2 As shown, the present invention provides a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning, comprising the following steps:

[0060] S100, establish a training network for the attitude control agent of a variable configuration aircraft,

[0061] The attitude control of the aircraft is a process of continuous action change. The Soft Actor Critic (SAC) algorithm is used to train the intelligent attitude controller. The action value function of each agent is expressed as Among them, θ is the parameter of the action value function neural network; the strategy function is Among them, φ is the parameter of the policy function neural network; o i represents the observation of agent i, a i represents the execution action of agent i;

[0062] The SAC algorithm is a reinforcement learning algorithm based on maximum entropy, and its strategy optimization process is:

[0063]

[0064] in, represents the optimized strategy, ρ π Representative in Strategy The state transition distribution under Represents the observation quantity of agent i Next action Rewards received, For strategy The entropy of h is the entropy weight coefficient, which is used to balance the impact of reward and action entropy on the strategy; The observation (o) and action (a) of the ith agent at time t follow the state transition distribution, that is, the action is obtained after the state transition according to the observation; The · in it represents all possible actions;

[0065] During the execution of the SAC algorithm, the state value function of the observation is estimated as follows:

[0066]

[0067] in, Represents the state value function of the ith agent when the observation value is o at time t; Represents the action amount of the i-th agent at time t that follows the strategy

[0068] Action-Value Function The estimation method is:

[0069]

[0070] Among them, γ represents the discount factor; Represents that the observation of the i-th agent at time t+1 follows the state transition distribution;

[0071] When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function:

[0072]

[0073] Among them, J Q (θ) represents the loss function; y i Represents the predicted action-value function of the ith agent; represents the reward of the i-th agent at time t; D represents the experience pool, which stores the experience of agent i The neural network parameters representing the target Q function are updated as follows:

[0074]

[0075] Where τ represents the soft update rate of the target Q function neural network parameters;

[0076] In the SAC algorithm, there are two Q functions: The two networks will be trained simultaneously, computing Smaller values ​​will be used to improve the stability of training;

[0077] The policy neural network parameters are learned by minimizing the following loss function:

[0078]

[0079] S200, with attitude control as the goal, determining the observation space and action space of the three-channel attitude control of the variable configuration aircraft, that is, taking the three control channels as separate agents, and implementing multi-agent distributed control of attitude control through interactive training in the same environment, the three agents corresponding to the three channels respectively merge their own observation quantities, and finally obtain their own independent observation spaces; the observation space includes the configuration of the aircraft, the expected attitude angle, the current attitude angle, the current angular velocity and the steering gear command at the previous moment, and the action space includes the equivalent steering gear command of the aircraft;

[0080] S300, a variable-structure aircraft attitude control decision network based on observation data training of multi-agent reinforcement learning, with design rewards including attitude convergence rewards Stable attitude reward Attitude Divergence Penalty and servo jitter penalty Specifically include:

[0081] S310, Attitude Convergence Reward The aircraft's attitude will be encouraged to converge to the desired attitude, and the closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, They are |σ e |、|β e |、|α e |, where |σ e | is the absolute value of the roll angle deviation, |β e | is the absolute value of the sideslip angle deviation, |α e | is the absolute value of the angle of attack deviation; the specific design of this reward is:

[0082]

[0083] The above formula includes two parts: linear reward and step reward. The linear reward can make the posture error always approach 0, and continue to grow in the process of posture error elimination, and reach the maximum after reaching 0; the step reward can promote the agent to explore more quickly and reach the small error area, which can effectively improve the controller's learning ability for posture convergence;

[0084] S320, Stable Attitude Reward When the attitude difference of the aircraft is small, avoid excessive attitude oscillation. When the attitude angular velocity is less than a certain value At the same time, in order to prevent the aircraft from prematurely converging without completely eliminating the error, this reward is designed to be coupled with the first reward, increasing with the increase of the first reward, to avoid poor training results caused by unreasonable weighting coefficients between the two rewards; the specific design of this reward is:

[0085]

[0086] S330, Posture Divergence Penalty Prevent the attitude controller from exploring dangerous areas. When the attitude error is greater than a certain range, this penalty is provided and the round ends to prevent the agent from continuing to explore. Set the danger value of the attitude angle error to The specific design of this reward is:

[0087]

[0088] S340, Servo vibration penalty Prevents the servo from repeatedly shaking and reduces the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment Provide penalties based on the change in the servo state. The specific design is:

[0089]

[0090] Among them, δ i,max represents the maximum value of the ith rudder deviation;

[0091] S400, use the trained decision network to perform attitude control of variable-configuration aircraft, establish simulation scenarios based on the Python environment, complete the training and testing of aircraft clusters, and test the formation control performance through a variety of random environments;

[0092] In order to evaluate the attitude control capability of the transformable aircraft under the multi-agent control of the present invention, the basic performance verification of the controller, the performance verification of the transformable process control, and the simulation experiments of comparative analysis of various controllers are designed. The model parameters of the transformable aircraft are: the initial flight environment is selected as the flight altitude h = 40km, and the flight speed v = 3000m / s. The reward function coefficients in the multi-agent system are set as shown in the following table.

[0093] Table 1

[0094]

[0095] The hyperparameters of the multi-agent reinforcement learning algorithm used in this example are shown in the following table.

[0096] Table 2

[0097] parameter Numeric parameter Numeric Discount factor γ 0.99 Soft update rate τ 0.01 <![CDATA[Entropy weight coefficient α h > 0.2 Batch size B 256

[0098] In order for the agent to obtain a strategy with good generalization in learning, it is necessary to randomize the expected attitude angle of the aircraft and the aircraft configuration during the training process to ensure that the agent learns from a sufficiently wide range of data samples. The aircraft state of the training environment is set as shown in the following table. The aircraft tracks a random fixed attitude angle each time, and the configuration changes continuously during the process.

[0099] Table 3

[0100] Parameter name Parameter Symbols Numeric Initial angle of attack <![CDATA[α0]]> 0deg Initial sideslip angle <![CDATA[β0]]> 0deg Initial roll angle <![CDATA[σ0]]> 0deg Desired angle of attack <![CDATA[α d ]]> 0~20 degrees Expected sideslip angle <![CDATA[β d ]]> 0deg Expected roll angle <![CDATA[σ d ]]> -30~30 degrees Configuration folding angle <![CDATA[δ f ]]> 6*tdeg

[0101] The maximum simulation step length of each round of the training process is designed to be 5.00s, the simulation step length is 0.01s for the initial state of the aircraft in the training process, and the total number of training rounds is 15000. In order to compare with the original single-agent reinforcement learning algorithm, a single-agent controller is designed at the same time, whose observation quantity is the union of the observation quantity of the multi-agent system, its action quantity is three rudder deviations, and its reward function adopts the average value of the reward function of each of the three agents in the multi-agent system.

[0102] The training results are as follows Figure 3 As shown in the figure. The solid line is the curve of the average reward value change of the agent in the adjacent 100 rounds, and the light-colored blocks of the corresponding colors are the reward variances within the 100 rounds. It can be seen that in the multi-agent system, the three agents all learned effective experience quickly, and the rewards began to increase around 1000 rounds, reaching a high value at 4000 rounds, and converged at 7000 rounds; the learning speed of the single agent was slow at the beginning, and the reward value was always around 0 before 2500 rounds, and then began to increase slowly, converging at 8000 rounds. After convergence, the average reward fluctuation of the multi-agent system was small, while the single agent always had certain fluctuations (such as at 10700 rounds and 12200 rounds).

[0103] In addition, through the reward fluctuations of agent A1 and agent A2 at 5000 rounds, it can be considered that the agents in the rolling and yaw channels have greater coupling, and the decisions of the two agents will directly affect the reward value of the other agent.

[0104] In order to evaluate the basic performance of the controller, this example sets the aircraft control mode to BTT (bank to turn) control, so the desired sideslip angle is always 0°, and the fixed configuration folding angle parameter is δ f =15°, the initial parameters of the aircraft are α0=5°, β0=0°, σ0=20°, and the expected attitude at t=0s is α d =12°, σ0=23°, the expected attitude at t=1.5s is α d =15°, σ0=27°, the expected attitude at t=3.5s is α d =20°, σ0=30°, the tracking curve of the attitude angle is as follows Figure 4 As shown, the angular velocity curve is Figure 5 As shown, the rudder deflection command is as follows Figure 6 shown.

[0105] Among them, Figure 4 In the figure, the solid red line is the expected value, and the dotted black line is the actual value. The angle of attack and the roll angle can quickly respond to the expected value with almost no overshoot, and the sideslip angle can quickly return to 0 in each attitude response process. From the angular velocity and rudder deflection curves, it can be seen that the attitude change process of the aircraft is relatively smooth, and it can remain stable when reaching a new state.

[0106] In addition, the controller is tested for its ability to track continuously changing attitude angles under a fixed configuration. The folding angle parameter of the fixed configuration is also δ f =15°, the initial parameters of the aircraft are α0=10°, β0=0°, σ0=23°, and the expected attitude angle is:

[0107]

[0108] The simulation results are as follows Figure 7 As shown in the figure, during the continuous angle tracking process, the maximum error of the roll angle is 0.27°, the maximum error of the angle of attack is 0.13°, and the maximum deviation of the sideslip angle is 0.012°. The aircraft has a good response effect to the continuously changing angle of attack and roll angle, and the sideslip angle can always be kept near 0.

[0109] In order to measure the attitude control ability of the aircraft during the configuration change process, the aircraft is set to always maintain a fixed attitude angle, specifically set to α0 = α d =10°、σ0=σ d=7°, the configuration of the aircraft begins to change continuously, and the change process is:

[0110]

[0111] The process of maintaining the attitude angle is as follows: Figure 8 As shown in the figure, the roll angle fluctuated slightly at the beginning, with the maximum error reaching 0.055°, and then the roll angle error continued to decrease and stabilized within 0.02°. The angle of attack also dropped from 10° to 9.965° at the beginning, and then the error gradually decreased to within 0.02°. During this period, the sideslip angle was always less than 0.0025°, with good accuracy.

[0112] The rudder angle change curve is as follows Fig. 9 As shown in the figure, during the process of continuous configuration change, the rudder angle response is very stable, and only when the configuration change mode is switched (by δ f Constantly changing switch to δ f becomes smaller and δ f The rudder angle response has a small jitter and then stabilizes quickly. In addition, the pitch channel rudder angle δ z The change process shows that the aerodynamic parameters are constantly changing during the configuration change process, the required trim angle is also constantly changing, and the attitude of the aircraft can always remain stable.

[0113] Further considering the attitude tracking capability of the conformational process, the initial parameters of the designed aircraft are α0 = 5°, β0 = 0°, σ0 = 15°, and the desired attitude angle is α d =10°, σ0=20°. The conformational change process of the aircraft is:

[0114] (δ f ) d =(6×t)°

[0115] The attitude angle response process is as follows: Fig.10 As shown in the figure, both the angle of attack and the roll angle can respond to commands quickly and stabilize in about 0.2s, with almost no overshoot. The sideslip angle has a small oscillation, but can also quickly maintain stability, and the maximum error is no more than 0.25°.

[0116] In order to measure the performance of the multi-agent controller of this method, a comparative analysis is conducted. The configuration change of the designed aircraft is the same as above, with initial parameters of α0 = 10°, β0 = 1°, σ0 = 0°, and the desired attitude angle is α d =20°, σ0=20°. Considering the control effects of PID controller, backstepping controller, single agent controller and multi-agent controller proposed in this paper under this working condition, the attitude response curve is as follows: Fig.11As shown, "PID" represents PID controller, "BackStepping" represents backstepping controller, "DRL" represents single-agent controller, and "MADRL" represents the controller based on multi-agent reinforcement learning proposed in the present invention.

[0117] from Fig.11 From the mid-roll angle response curve, it can be seen that without overshoot, the response time of the backstepping controller is 0.35s, the response time of the PID controller is 1.0s, and the response speeds of the DRL controller and MADRL are similar, both around 0.2s, but the DRL controller has a certain overshoot and a certain tracking error after 0.25s, and converges at nearly 1.0s. It can be seen that the response speed of MADRL is better than that of the traditional controller, and the stability is better than that of the DRL controller.

[0118] from Fig.11 From the angle of attack tracking curve, it can be found that the response speeds of the backstepping controller and PID controller are 0.35s and 0.6s respectively, and the response speeds of the DRL controller and MADRL controller are 0.25s and 0.26s respectively. The DRL controller also has the problems of insufficient control accuracy and poor stability, and a large deviation occurs after reaching the desired angle of attack.

[0119] for Fig.11 In terms of the sideslip angle tracking curve, the MADRL controller has a faster response speed, and different intelligent agents can make reasonable control decisions. The maximum deviation of the sideslip angle is 0.65°. The sideslip angle deviations of the DRL controller and the backstepping controller are relatively small, which are 0.3° and 0.15° respectively. In terms of response speed, the backstepping controller, DRL controller, and MADRL controller all converge at 0.4s, while the PID controller has shortcomings in response speed and tracking error compared with the first three controllers.

[0120] Specific implementation scheme two: The present invention is a transformable aircraft intelligent attitude control system based on multi-agent reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned transformable aircraft intelligent attitude control method based on multi-agent reinforcement learning during operation.

[0121] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0122] Specific implementation scheme three: The present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to implement the steps of a variable configuration aircraft intelligent attitude control method based on multi-agent reinforcement learning when called by a processor.

[0123] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0124] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S100, establishing a training network for a variable configuration aircraft attitude control intelligent agent; S200, with attitude control as the goal, determine the observation space and action space of the three-channel attitude control of the variable-configuration aircraft, regard the three control channels as three separate agents, and realize multi-agent distributed control of attitude control through interactive training in the same environment, and then merge the observation quantities of the three agents corresponding to the three channels respectively, and finally obtain their own independent observation spaces; S300, a variable-structure aircraft attitude control decision network based on observation data training of multi-agent reinforcement learning, with design rewards including attitude convergence rewards Stable attitude reward Attitude Divergence Penalty and servo jitter penalty 2. According to claim 1, a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning is characterized in that: In step S100, it specifically includes: The soft actor-critic algorithm is used to train the intelligent posture controller; the action value function of each agent is expressed as Among them, θ represents the parameters of the action value function neural network; the strategy function is Among them, φ represents the parameters of the policy function neural network; o i represents the observation of agent i, a i Represents the execution action of agent i; The strategy optimization process of the SAC algorithm is: in, represents the optimized strategy, ρ π Representative in Strategy The state transition distribution under Represents the intelligent agent i from the observation Next action Rewards received, Representative strategy The entropy of h Represents the entropy weight coefficient, which is used to balance the impact of reward and action entropy on the strategy; The observation (o) and action (a) of the ith agent at time t follow the state transition distribution, that is, the action is obtained after the state transition according to the observation; The · in it represents all possible actions; During the execution of the SAC algorithm, the state value function of the observation is estimated as follows: in, Represents the value function of the ith agent when the observation value is o at time t; Represents the action amount of the i-th agent at time t that follows the strategy Action-Value Function The estimation method is: Among them, γ represents the discount factor; Represents that the observation of the i-th agent at time t+1 follows the state transition distribution; When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function: Among them, J Q (θ) represents the loss function; y i represents the predicted action-value function of the ith agent; represents the reward of the i-th agent at time t; D represents the experience pool, which stores the experience of agent i The neural network parameters representing the target Q function are updated as follows: Where τ represents the soft update rate of the target Q function neural network parameters; In the SAC algorithm, there are two Q functions: The two networks will be trained simultaneously, computing Use smaller values ​​to improve training stability; The policy neural network parameters are learned by minimizing the following loss function:

3. The method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning according to claim 2, characterized in that: In step S200: the observation space includes the configuration of the aircraft, the expected attitude angle, the current attitude angle, the current angular velocity and the steering gear instruction at the last moment, and the action space includes the equivalent steering gear instruction of the aircraft.

4. The method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning according to claim 3, characterized in that: In step S300, the posture convergence reward It is used to encourage the aircraft's attitude to converge to the desired attitude. The closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, They are |σ e |, β e |、|α e |, where |σ e | is the absolute value of the roll angle deviation, |β e | is the absolute value of the sideslip angle deviation, |α e | is the absolute value of the angle of attack deviation; the specific design of this reward is:

5. The method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning according to claim 4, characterized in that: In step S300, the posture stability reward It is used to make the attitude difference of the aircraft smaller to avoid excessive attitude oscillation. When the attitude angular velocity is less than a certain value This reward is provided when the reward is provided; this reward is designed to be coupled with the first reward, and the specific design of this reward is:

6. The method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning according to claim 5, characterized in that: In step S300, the posture divergence penalty It is used to prevent the attitude controller from exploring dangerous areas. When the attitude error is greater than a certain range, this penalty is provided and the round ends at the same time. The danger value of the attitude angle error is set to The specific design of this reward is:

7. The method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning according to claim 6, characterized in that: In step S300, the servo vibration penalty Used to prevent the servo from repeatedly shaking and reduce the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment Provide penalties based on the change in the servo state. The specific design is: Among them, δ i,max Represents the maximum value of the ith rudder deviation.

8. A configuration-variable aircraft intelligent attitude control system based on multi-agent reinforcement learning, characterized in that: The system has a program module corresponding to the steps of any one of claims 1 to 7 above, and executes the steps in the above-mentioned method for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning when running.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the intelligent attitude control method of a variable configuration aircraft based on multi-agent reinforcement learning described in any one of claims 1 to 7 when called by a processor.

Images