A method and system for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning

Abstract

The present invention provides a method and system for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning, which belongs to the field of intelligent control. In order to solve the problems of complex nonlinearity of variable-configuration hypersonic aircraft models and insufficient attitude control performance under complex control tasks. The present invention performs end-to-end attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning, and adopts a multi-agent collaborative approach to further improve exploration capabilities and control flexibility. The intelligent attitude control method based on a multi-agent system regards multiple control channels of a variable-configuration hypersonic aircraft as independent agents, trains and learns in the same environment, and makes distributed decisions based on their respective states to achieve an effective intelligent control method.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a method and system for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning. Background Art

[0002] In recent years, with the growing demand for space-to-earth transportation, traditional fixed-configuration hypersonic vehicles, due to their limited performance range and limited mission types, have struggled to adapt to diverse mission requirements across different flight ranges and altitudes. Variable-configuration hypersonic vehicles, by changing their shape, can adapt to varying mission requirements and flight environments. This allows them to maintain an optimal aerodynamic shape throughout flight, achieving an optimal balance between key parameters such as lift, flight speed, and energy consumption, while better meeting external constraints such as heat flow and dynamic pressure. This will help support future efficient transportation missions at different altitudes within and beyond the atmosphere.

[0003] However, while variable-configuration hypersonic vehicles offer a wider flight envelope and greater maneuverability, the changes in inertia and aerodynamic parameters brought about by their structural changes also pose new challenges to control systems. Attitude control issues for these vehicles include designing separate controllers for the longitudinal and lateral planes to simplify model complexity, and utilizing techniques such as fuzzy disturbance observers and neural network observers to address model uncertainties and environmental disturbances. Furthermore, nonlinear control methods such as sliding mode controllers and backstepping have also been successfully applied to attitude control for these vehicles.

[0004] To further enhance the performance and adaptability of attitude control for variable-configuration hypersonic vehicles, various optimization and intelligent methods have been introduced. Reinforcement learning, with its powerful exploration and learning capabilities and model-independent, data-driven nature, has been widely used in this field.

[0005] Most existing attitude control methods rely on a single agent to make decisions. To address more complex control tasks, multi-agent collaboration can further improve exploration capabilities and control flexibility. Therefore, this paper proposes an intelligent attitude control method based on a multi-agent system. This method treats the multiple control channels of a variable-configuration hypersonic vehicle as independent agents, trains and learns in the same environment, and makes distributed decisions based on their respective states, thus achieving a new type of intelligent control. Summary of the Invention

[0006] The technical problems to be solved by the present invention are:

[0007] In order to solve the problems of complex nonlinearity and insufficient attitude control performance of variable-configuration hypersonic aircraft models under complex control tasks.

[0008] The present invention is to solve the above technical problems using the following technical solutions:

[0009] The present invention provides a method for intelligent attitude control of a configuration-variable aircraft based on multi-agent reinforcement learning, comprising the following steps:

[0010] S100, establishing a training network for a configuration-variable aircraft attitude control agent;

[0011] S200, with attitude control as the goal, determine the observation space and action space for the three-channel attitude control of the variable-state vehicle. Treat the three control channels as three separate agents, and implement multi-agent distributed control of attitude control through interactive training in the same environment. Then, merge the observations of the three agents corresponding to the three channels, ultimately obtaining their own independent observation spaces.

[0012] S300, a variable-structure aircraft attitude control decision network trained with multi-agent reinforcement learning based on observation data, with design rewards including attitude convergence rewards Stable posture reward Posture Divergence Penalty and servo vibration penalty

[0013] Furthermore, in step S100, it specifically includes:

[0014] The soft actor-critic algorithm is used to train the intelligent posture controller; the action value function of each agent is expressed as Among them, θ represents the parameters of the action value function neural network; the policy function is Among them, φ represents the parameters of the policy function neural network; o i represents the observation of agent i, a i Represents the execution action of agent i;

[0015] The strategy optimization process of the SAC algorithm is:

[0016]

[0017] in, represents the optimized strategy, ρ π Representative in Strategy The state transition distribution under Represents agent i from the observation Next action Rewards received, Representative Strategy Entropy, α h Represents the entropy weight coefficient, which is used to balance the impact of rewards and action entropy on the strategy; The observation amount (o) and action amount (a) of the i-th agent at time t follow the state transition distribution, that is, according to the observation amount, the action is obtained after the state transition; The · in it represents all possible actions;

[0018] During the execution of the SAC algorithm, the state value function of the observation is estimated as follows:

[0019]

[0020] in, Represents the value function of the i-th agent when the observation quantity at time t is o; Represents the action amount of the i-th agent at time t that follows the strategy

[0021] Action-value function The estimation method is:

[0022]

[0023] Where γ represents the discount factor; Represents that the observation quantity of the i-th agent at time t+1 follows the state transition distribution;

[0024] When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function:

[0025]

[0026] Among them, J Q (θ) represents the loss function; y i represents the predicted action-value function of the i-th agent; represents the reward of the i-th agent at time t; D represents the experience pool, which stores the experience of agent i The neural network parameters representing the target Q function are updated as follows:

[0027]

[0028] Where τ represents the soft update rate of the target Q-function neural network parameters;

[0029] In the SAC algorithm, there are two Q functions: The two networks will be trained simultaneously, Use smaller values ​​to improve training stability;

[0030] The policy neural network parameters are learned by minimizing the following loss function:

[0031]

[0032] Furthermore, in step S200: the observation space includes the configuration, desired attitude angle, current attitude angle, current angular velocity and the servo command of the aircraft at the previous moment, and the action space includes the equivalent rudder deflection command of the aircraft.

[0033] Furthermore, in step S300, the posture convergence reward It is used to encourage the aircraft's attitude to converge to the desired attitude. The closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, are σ e |、|β e |、|α e |, where |σ e | is the absolute value of the roll angle deviation, |β e | is the absolute value of the sideslip angle deviation, |α e | is the absolute value of the angle of attack deviation; the specific design of this reward is:

[0034]

[0035] Furthermore, in step S300, the posture stability reward Used to make the attitude difference of the aircraft smaller, to avoid excessive attitude oscillation, when the attitude angular velocity is less than a certain value This reward is provided when ; this reward is designed to be coupled with the first reward, and the specific design of this reward is:

[0036]

[0037] Furthermore, in step S300, the posture divergence penalty It is used to prevent the attitude controller from exploring dangerous areas. When the attitude error is greater than a certain range, this penalty is provided and the round ends at the same time. The danger value of the attitude angle error is set to The reward is specifically designed as follows:

[0038]

[0039] Furthermore, in step S300, the servo vibration penalty Used to prevent the servo from repeatedly shaking and reduce the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment Provides penalties based on the change in servo status. The specific design is:

[0040]

[0041] Among them, δ i,max Represents the maximum value of the ith rudder deviation.

[0042] A configuration-shifting aircraft intelligent attitude control system based on multi-agent reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned configuration-shifting aircraft intelligent attitude control method based on multi-agent reinforcement learning during operation.

[0043] A computer-readable storage medium stores a computer program, which is configured to implement the steps of a method for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning when called by a processor.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This paper designs a multi-agent reinforcement learning-based attitude control method for a variable-configuration hypersonic vehicle by treating the three-channel aircraft attitude controller as an agent interacting with the environment to learn the control strategy. This method addresses the complex nonlinearities of variable-configuration hypersonic vehicle models and the insufficient performance of attitude control. Each agent has an independent observation space and independently issues different rudder deflection commands, achieving overall attitude stability control through distributed control. The superiority of the proposed method was verified through performance verification of the controller in the fixed / variable configuration process and comparative analysis with multiple methods.

[0046] The present invention can perform stable attitude control for the complex nonlinearity of the variable-configuration hypersonic aircraft model, thereby ensuring the stable flight of the variable-configuration hypersonic aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a framework diagram of a method for intelligent attitude control of a configuration-variable aircraft based on multi-agent reinforcement learning in an embodiment of the present invention;

[0048] Figure 2 This is a channel strategy network framework diagram in an embodiment of the present invention;

[0049] Figure 3 is a reward value curve diagram during the training process in an embodiment of the present invention;

[0050] Figure 4 is a step attitude angle command tracking curve diagram under a fixed configuration in an embodiment of the present invention;

[0051] Figure 5is a graph of angular velocity of a step attitude angle command under a fixed configuration in an embodiment of the present invention;

[0052] Figure 6 is a graph showing the step attitude angle command rudder deflection under a fixed configuration in an embodiment of the present invention;

[0053] Figure 7 is a graph of continuous attitude angle command tracking under a fixed configuration in an embodiment of the present invention;

[0054] Figure 8 is a graph showing the posture angle maintenance during the conformational transformation in an embodiment of the present invention;

[0055] Figure 9 is a graph showing an equivalent rudder deflection angle during the conformational transformation process according to an embodiment of the present invention;

[0056] Figure 10 is a graph showing the attitude angle response curve during the conformational transformation in an embodiment of the present invention;

[0057] Figure 11 3 is a comparative analysis diagram of the posture response of the conformational change process under different controllers in the embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] Specific implementation plan 1: Combined Figure 1 and Figure 2 As shown, the present invention provides a method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning, comprising the following steps:

[0060] S100, establish a training network for the attitude control agent of a variable configuration aircraft,

[0061] The attitude control of an aircraft is a process of continuous action change. The Soft Actor Critic (SAC) algorithm is used to train the intelligent attitude controller. The action value function of each agent is expressed as Among them, θ is the parameter of the action value function neural network; the policy function is Among them, φ is the parameter of the policy function neural network; o i represents the observation quantity of agent i, a i represents the execution action of agent i;

[0062] The SAC algorithm is a reinforcement learning algorithm based on maximum entropy. Its strategy optimization process is:

[0063]

[0064] in, represents the optimized strategy, ρ π Representative in Strategy The state transition distribution under Represents the agent i from the observation Next action Rewards received, For strategy Entropy, α h is the entropy weight coefficient, which is used to balance the impact of rewards and action entropy on the strategy; The observation amount (o) and action amount (a) of the i-th agent at time t follow the state transition distribution, that is, according to the observation amount, the action is obtained after the state transition; The · in it represents all possible actions;

[0065] During the execution of the SAC algorithm, the state value function of the observation is estimated as follows:

[0066]

[0067] in, Represents the state value function of the i-th agent when the observation value at time t is o; Represents the action amount of the i-th agent at time t that follows the strategy

[0068] Action-value function The estimation method is:

[0069]

[0070] Where γ represents the discount factor; Represents that the observation quantity of the i-th agent at time t+1 follows the state transition distribution;

[0071] When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function:

[0072]

[0073] Among them, J Q (θ) represents the loss function; y i represents the predicted action-value function of the i-th agent; represents the reward of the i-th agent at time t; D represents the experience pool, which stores the experience of agent i The neural network parameters representing the target Q function are updated as follows:

[0074]

[0075] Where τ represents the soft update rate of the target Q-function neural network parameters;

[0076] In the SAC algorithm, there are two Q functions: The two networks will be trained simultaneously, A smaller value will be used to improve the stability of training;

[0077] The policy neural network parameters are learned by minimizing the following loss function:

[0078]

[0079] S200, with attitude control as the goal, determining the observation space and action space for three-channel attitude control of a variable-state aircraft. Specifically, the three control channels are treated as separate agents, and multi-agent distributed control of attitude control is implemented through interactive training in the same environment. The three agents corresponding to the three channels each merge their own observations to ultimately obtain independent observation spaces. The observation space includes the aircraft's configuration, desired attitude angle, current attitude angle, current angular velocity, and the previous moment's servo command. The action space includes the aircraft's equivalent rudder deflection command.

[0080] S300, a variable-structure aircraft attitude control decision network trained with multi-agent reinforcement learning based on observation data, with design rewards including attitude convergence rewards Stable posture reward Posture Divergence Penalty and servo vibration penalty Specifically include:

[0081] S310, Posture Convergence Reward The aircraft's attitude will be encouraged to converge to the desired attitude, and the closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, They are |σ e |、|β e |、|α e |, where |σ e | is the absolute value of the roll angle deviation, |β e | is the absolute value of the sideslip angle deviation, |α e | is the absolute value of the angle of attack deviation; the specific design of this reward is:

[0082]

[0083] The above formula includes two parts: linear reward and step reward. The linear reward can make the posture error always close to 0, and continue to grow during the posture error elimination process, reaching its maximum after reaching 0. The step reward can promote the agent to explore more quickly and reach the small error area, which can effectively improve the controller's learning ability for posture convergence.

[0084] S320, Stable Attitude Reward When the attitude difference of the aircraft is small, avoid excessive attitude oscillation. When the attitude angular velocity is less than a certain value At the same time, in order to prevent the aircraft from prematurely converging without completely eliminating the error, this reward is coupled with the first reward and increases with the increase of the first reward, avoiding poor training results caused by unreasonable weighting coefficients between the two rewards. The specific design of this reward is as follows:

[0085]

[0086] S330, Posture Divergence Penalty Prevent the posture controller from exploring dangerous areas. When the posture error is greater than a certain range, the penalty is provided and the round ends to prevent the agent from continuing to explore. The danger value of the posture angle error is set to The reward is specifically designed as follows:

[0087]

[0088] S340, Servo vibration penalty Prevents the servo from repeatedly shaking and reduces the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment Provides penalties based on the change in servo status. The specific design is:

[0089]

[0090] Among them, δ i,max represents the maximum value of the ith rudder deviation;

[0091] S400, using the trained decision network to perform attitude control of variable-mode aircraft, building simulation scenarios based on the Python environment, completing the training and testing of aircraft clusters, and testing formation control performance in a variety of random environments;

[0092] To evaluate the attitude control capabilities of the proposed multi-agent controlled transformable aircraft, simulation experiments were designed to verify the basic controller performance, verify the transformable process control performance, and compare and analyze various controllers. The transformable aircraft model parameters were: an initial flight environment with an altitude of h = 40 km and a speed of v = 3000 m / s. The reward function coefficients for the multi-agent system were set as shown in the table below.

[0093] Table 1

[0094]

[0095] The hyperparameters of the multi-agent reinforcement learning algorithm used in this example are shown in the following table.

[0096] Table 2

[0097] parameter Numerical parameter Numerical Discount factor γ 0.99 Soft update rate τ 0.01 <![CDATA[Entropy weight coefficient α h > 0.2 Batch size B 256

[0098] To ensure that the agent learns a generalizable strategy, it's necessary to randomize the desired attitude angle and configuration of the aircraft during training to ensure that the agent learns from a sufficiently broad range of data samples. The training environment aircraft states are set as shown in the table below. The aircraft tracks a random fixed attitude angle each time, and the configuration changes continuously during this process.

[0099] Table 3

[0100] Parameter name Parameter Symbol Numerical Initial angle of attack <![CDATA[α0]]> 0deg Initial sideslip angle <![CDATA[β0]]> 0deg Initial roll angle <![CDATA[σ0]]> 0deg Desired angle of attack <![CDATA[α d ]]> 0~20 degrees Desired sideslip angle <![CDATA[β d ]]> 0deg Desired roll angle <![CDATA[σ d ]]> -30~30deg Configuration folding angle <![CDATA[δ f ]]> 6*tdeg

[0101] The maximum simulation step size for each training round was designed to be 5.00 seconds, the simulation step size was 0.01 seconds for the initial state of the aircraft during the training process, and the total number of training rounds was 15,000. To compare with the original single-agent reinforcement learning algorithm, a single-agent controller was designed. Its observations were the union of the multi-agent system's observations, its actions were the three rudder deflections, and its reward function was the average of the reward functions of the three agents in the multi-agent system.

[0102] The training results are as follows Figure 3 As shown in the figure. The solid line represents the average reward curve for the agents over 100 consecutive rounds, and the corresponding light-colored blocks represent the reward variance within those 100 rounds. It can be seen that in the multi-agent system, all three agents quickly learned effective experience, with rewards beginning to increase around round 1000, reaching a high value around round 4000, and converging around round 7000. The single agent's learning speed was slower at the beginning, with the reward remaining around 0 for the first 2500 rounds, then slowly increasing, and converging around round 8000. After convergence, the average reward of the multi-agent system fluctuated slightly, while the single agent's reward still fluctuated to some extent (such as at rounds 10700 and 12200).

[0103] In addition, the reward fluctuations of agents A1 and A2 at 5000 rounds indicate that the agents in the roll and yaw channels are more coupled, and the decisions of two agents will directly affect the reward value of the other agent.

[0104] To evaluate the basic performance of the controller, this example sets the aircraft control mode to BTT (bank to turn) control, so the desired sideslip angle is always 0°, and the fixed configuration folding angle parameter is δ f =15°, the initial parameters of the aircraft are α0=5°, β0=0°, σ0=20°, and the expected attitude at t=0s is α d =12°, σ0=23°, and the expected attitude at t=1.5s is α d =15°, σ0=27°, and the expected attitude at t=3.5s is α d =20°, σ0=30°, the tracking curve of the attitude angle is as follows Figure 4 As shown, the angular velocity curve is as follows Figure 5 As shown, the rudder deflection command is as follows Figure 6 shown.

[0105] Among them, Figure 4 In the figure, the solid red line represents the expected value, and the dashed black line represents the actual value. Both the angle of attack and the bank angle quickly respond to the expected value with almost no overshoot, and the sideslip angle quickly returns to zero during each attitude response. The angular velocity and rudder deflection curves show that the aircraft's attitude changes smoothly and remains stable once it reaches a new state.

[0106] In addition, the controller is tested for its ability to track continuously changing attitude angles under a fixed configuration. The folding angle parameter of the fixed configuration is also δ f =15°, the initial parameters of the aircraft are α0=10°, β0=0°, σ0=23°, and the expected attitude angle is:

[0107]

[0108] The simulation results are as follows Figure 7 As shown in the figure, during the continuously changing angle tracking process, the maximum error in the roll angle is 0.27°, the maximum error in the angle of attack is 0.13°, and the maximum deviation in the sideslip angle is 0.012°. The aircraft responds well to continuously changing angles of attack and roll, and the sideslip angle is always maintained near zero.

[0109] In order to measure the attitude control ability of the aircraft during the transformation process, the aircraft is set to always maintain a fixed attitude angle, specifically set to α0 = α d =10°、σ0=σ d=7°, the aircraft's configuration begins to change continuously, and the change process is:

[0110]

[0111] The process of maintaining the attitude angle is as follows Figure 8 As shown, the roll angle initially fluctuated slightly, reaching a maximum error of 0.055°. The error then decreased and stabilized within 0.02°. The angle of attack also initially dropped from 10° to 9.965°, before gradually decreasing to within 0.02°. The sideslip angle remained consistently below 0.0025°, demonstrating excellent accuracy.

[0112] The rudder angle change curve is as follows Figure 9 As shown in the figure, during the process of continuous configuration change, the rudder angle response is very stable, and only when the configuration change mode is switched (by δ f Constantly changing switch to δ f Continuously decreasing, and δ f The rudder angle response has a small jitter and then quickly stabilizes. In addition, the pitch channel rudder angle δ z The change process shows that the aerodynamic parameters and the required trim angles are constantly changing during the configuration change process, but the attitude of the aircraft can always remain stable.

[0113] Further considering the attitude tracking capability of the conformational process, the initial parameters of the designed aircraft are α0=5°, β0=0°, σ0=15°, and the desired attitude angle is α d =10°, σ0=20°. The conformational change process of the aircraft is:

[0114] (δ f ) d =(6×t)°

[0115] The attitude angle response process is as follows: Figure 10 As shown in the figure, both the angle of attack and the roll angle can quickly respond to commands and stabilize in about 0.2s with almost no overshoot. The sideslip angle has small oscillations but can also quickly maintain stability, and the maximum error is no more than 0.25°.

[0116] In order to measure the performance of the multi-agent controller of this method, a comparative analysis is conducted. The configuration change of the designed aircraft is the same as above, with the initial parameters α0=10°, β0=1°, σ0=0°, and the desired attitude angle is α d =20°, σ0=20°. Considering the control effects of PID controller, backstepping controller, single agent controller and multi-agent controller proposed in this paper under this working condition, the attitude response curve is as follows: Figure 11As shown, "PID" represents PID controller, "BackStepping" represents backstepping controller, "DRL" represents single-agent controller, and "MADRL" represents the controller based on multi-agent reinforcement learning proposed in the present invention.

[0117] from Figure 11 The mid-roll angle response curves show that, without overshoot, the backstepping controller has a response time of 0.35s, while the PID controller has a response time of 1.0s. The DRL and MADRL controllers have comparable response speeds, both around 0.2s. However, the DRL controller exhibits some overshoot and tracking error after 0.25s, before converging at nearly 1.0s. This indicates that the MADRL has a faster response than traditional controllers and offers superior stability.

[0118] from Figure 11 The angle-of-attack tracking curves show that the response speeds of the backstepping and PID controllers are 0.35s and 0.6s, respectively, while those of the DRL and MADRL controllers are 0.25s and 0.26s, respectively. The DRL controller also suffers from insufficient control accuracy and poor stability, and exhibits significant deviations after reaching the desired angle of attack.

[0119] for Figure 11 In terms of the sideslip angle tracking curve, the MADRL controller achieves a faster response speed, enabling different agents to make reasonable control decisions. The maximum sideslip angle deviation is 0.65°. The DRL and backstepping controllers have relatively small sideslip angle deviations of 0.3° and 0.15°, respectively. In terms of response speed, the backstepping, DRL, and MADRL controllers all converge in 0.4s. However, the PID controller has shortcomings in response speed and tracking error compared to the previous three controllers.

[0120] Specific implementation scheme two: The present invention provides a variable-configuration aircraft intelligent attitude control system based on multi-agent reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned variable-configuration aircraft intelligent attitude control method based on multi-agent reinforcement learning during operation.

[0121] Other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0122] Specific implementation scheme three: The present invention provides a computer-readable storage medium, which stores a computer program. The computer program is configured to implement the steps of a variable-configuration aircraft intelligent attitude control method based on multi-agent reinforcement learning when called by a processor.

[0123] Other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0124] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art of the present invention may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for intelligent attitude control of a variable configuration aircraft based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S100: Establishing a training network for a configuration-variable aircraft attitude control agent; specifically, including: The soft actor-critic algorithm is used to train the intelligent posture controller; the action value function of each agent is expressed as ,in, Represents the parameters of the action-value function neural network; the policy function is ,in, Represents the parameters of the policy function neural network; Representative Agent The observed quantity, Representative Agent Execution action; The strategy optimization process of the SAC algorithm is: ； in, represents the optimized strategy, Representative in Strategy The state transition distribution under Representative Agent From the observation Next action Rewards received, Representative Strategy The entropy of Represents the entropy weight coefficient, which is used to balance the impact of rewards and action entropy on the strategy; The observation amount (o) and action amount (a) of the i-th agent at time t follow the state transition distribution, that is, according to the observation amount, the action is obtained after the state transition; The · in it represents all possible actions; During the execution of the SAC algorithm, the state value function of the observation is estimated as follows: ； in, Represents the value function of the i-th agent when the observation quantity at time t is o; Represents the action amount of the i-th agent at time t that follows the strategy ; Action-value function The estimation method is: ； Where γ represents the discount factor; Represents that the observation quantity of the i-th agent at time t+1 follows the state transition distribution; When using a neural network to estimate the above formula, the action value function The way a neural network learns is to minimize the following loss function: ； in, represents the loss function; represents the predicted action-value function of the i-th agent; Represents the reward of the i-th agent at time t; Represents the experience pool, which stores the agent Experience , Representative Target The neural network parameters of the function are updated as follows: ； in, Representative Target Function: soft update rate of neural network parameters; In the SAC algorithm, there are two The functions are 、 , the two networks will be trained simultaneously, in the calculation Use smaller values ​​to improve training stability; The policy neural network parameters are learned by minimizing the following loss function: ； S200, with attitude control as the goal, determine the observation space and action space for the three-channel attitude control of the variable-state vehicle. Treat the three control channels as three separate agents, and implement multi-agent distributed control of attitude control through interactive training in the same environment. Then, merge the observations of the three agents corresponding to the three channels, ultimately obtaining their own independent observation spaces. S300, a variable-structure aircraft attitude control decision network trained with multi-agent reinforcement learning based on observation data, with design rewards including attitude convergence rewards , Stable Posture Reward , Posture Divergence Penalty and servo vibration penalty ; The pose convergence reward It is used to encourage the aircraft's attitude to converge to the desired attitude. The closer the actual attitude of the aircraft is to the desired attitude, the greater the reward. , then the reward is not provided; is the absolute value of an attitude angle error in the attitude vector, 、 、 They are 、 、 ,in, is the absolute value of the roll angle deviation, is the absolute value of the sideslip angle deviation, is the absolute value of the angle of attack deviation; the specific design of this reward is: ； The attitude stability reward Used to make the attitude difference of the aircraft smaller, to avoid excessive attitude oscillation, when the attitude angular velocity is less than a certain value This reward is provided when ; this reward is designed to be coupled with the first reward, and the specific design of this reward is: ； The posture divergence penalty It is used to prevent the attitude controller from exploring dangerous areas. When the attitude error is greater than a certain range, this penalty is provided and the round ends at the same time. The danger value of the attitude angle error is set to The specific design of the reward is: ； The servo vibration penalty Used to prevent the servo from repeatedly shaking and reduce the usage of the servo. This reward needs to take the current action into consideration. and the action at the previous moment , provides penalties based on the change in servo state, specifically designed as follows: ； in, Represents the maximum value of the ith rudder deviation.

2. The method for intelligent attitude control of a configuration-adaptable aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S200: the observation space includes the configuration, desired attitude angle, current attitude angle, current angular velocity and the servo command of the aircraft at the last moment, and the action space includes the equivalent rudder deflection command of the aircraft.

3. A multi-agent reinforcement learning-based intelligent attitude control system for a variable-configuration aircraft, characterized by: The system has a program module corresponding to the steps of claim 1 or 2 above, and executes the steps in the above-mentioned method for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning when running.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is configured to implement the steps of the method for intelligent attitude control of a variable-configuration aircraft based on multi-agent reinforcement learning according to claim 1 or 2 when called by a processor.

Images