An aircraft attitude tracking intelligent allocation method based on multi-agent reinforcement learning

Abstract

This invention discloses an intelligent attitude assignment method for aircraft tracking based on multi-agent reinforcement learning, comprising: constructing an aircraft formation state space and action space; performing collaborative training based on a multi-agent reinforcement learning framework to obtain an attitude pointing assignment strategy; secondly, using a policy network to output the initial desired attitude of each aircraft, and introducing illumination and target direction constraints through a forward-facing angle calculation module to correct the attitude to generate the final desired attitude that meets the mission requirements; finally, combining an adaptive PID control algorithm considering attitude angular velocity constraints to achieve closed-loop attitude tracking control. This method achieves collaborative attitude assignment and high-precision tracking control of multiple aircraft under complex constraints, exhibiting good real-time performance, stability, and environmental adaptability.

Description

Technical Field

[0001] This invention belongs to the field of aircraft cooperative control technology, specifically relating to an intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous development of space technology and the diversification and complexity of mission requirements, spacecraft, due to their high maneuverability, wide coverage, and adaptability to multiple mission scenarios, have been widely used in missions such as collaborative inspection, logistics delivery, on-orbit servicing, and deep space exploration. To meet the requirements of wide coverage, high mission execution efficiency, and system robustness and redundancy, space mission execution methods are gradually shifting from single-vehicle operations to multi-vehicle collaborative formations. Multi-vehicle collaborative methods significantly improve the overall efficiency and safety of mission execution through division of labor, cooperation, and information sharing among multiple spacecraft. In multi-vehicle collaborative observation and control missions, each spacecraft must complete rapid attitude pointing and precise attitude tracking control of the target under multiple constraints, including safe distance, lighting conditions (such as the angle of illumination), energy constraints, and communication link constraints. This process involves multiple coupled links such as attitude dynamics modeling, mission pointing allocation, and attitude control law design. Due to the high dimensionality, strong nonlinearity, and high real-time requirements of multi-vehicle collaborative systems, attitude pointing allocation and optimal control have become the core challenges in collaborative missions.

[0003] Existing technologies mainly include two types of paths: The first category is based on centralized or distributed optimization methods, which achieve attitude pointing assignment through auction, matching or consensus algorithms, and combine quaternion control, SE(3) geometric control or optimal control to achieve attitude tracking. This type of method is stable in deterministic environments, but under strong nonlinear and uncertain conditions, it relies on accurate modeling and complex tuning, and it is difficult to simultaneously satisfy global optimality and real-time performance. The second type of method is based on end-to-end control strategies using single-agent reinforcement learning. It achieves adaptive optimization of attitude control through policy learning, but it is difficult to extend to multi-aircraft cooperative scenarios and lacks global coordination and system stability guarantees.

[0004] Furthermore, CN119828728A discloses an intelligent attitude control method and system for variable-configuration hypersonic vehicles based on multi-agent reinforcement learning. By treating multiple control channels of a variable-configuration hypersonic vehicle as independent agents for collaborative training, it improves exploration capabilities and control flexibility to a certain extent. However, this method still has significant limitations: 1. Lack of explicit handling of task-level constraints - Existing methods mainly focus on end-to-end learning of aircraft attitude control, making it difficult to embed task constraints such as illumination angle, target pointing assignment, and safety distance during the learning process. This may result in the control strategy not meeting the global task requirements under complex multi-constraint tasks. 2. Insufficient coordination between attitude tracking closed loop and pointing assignment - Existing methods separate agent control and task assignment, lacking a closed-loop coordination mechanism for attitude tracking accuracy and task pointing assignment of multiple aircraft, which may lead to large actual tracking errors or attitude conflicts. 3. Insufficient stability and scalability of multi-vehicle collaboration – In the multi-agent framework, existing technologies rely on fixed structures for training policy networks and value evaluation networks, lacking robust designs for multi-constraint and multi-vehicle dynamic interactions, making it difficult to guarantee stable convergence and real-time execution in high-dimensional nonlinear environments.

[0005] Therefore, there is an urgent need to build a technical system that integrates optimal attitude orientation allocation, attitude tracking control, and task constraint closed-loop fusion. Through multi-agent reinforcement learning, intelligent optimization of multi-aircraft formation attitude control can be achieved, which can simultaneously ensure task constraint satisfaction, system closed-loop stability, real-time performance, and scalability, thereby improving the overall performance of multi-aircraft collaborative operations in complex space environments. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning.

[0007] The objective of this invention can be achieved through the following technical solutions: This invention provides an intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning, comprising the following steps: Construct the state space and action space of the aircraft. Determine the state space based on the position, velocity and attitude information of each aircraft in the multi-aircraft formation, and determine the corresponding action space based on the attitude adjustment requirements. A multi-agent reinforcement learning framework is constructed and trained based on the state space and action space. The attitude pointing allocation strategy under multi-aircraft cooperative tasks is obtained through interactive learning. The multi-agent includes a policy network and a value evaluation network. The trained policy network is used to make decisions about the current state of the aircraft and outputs the initial expected attitude for each aircraft. The initial desired posture is input into the forward angle calculation module for constraint correction. Combined with the lighting conditions and target direction information, the final desired posture that meets the constraint requirements is generated. Based on the desired final attitude, an adaptive PID control algorithm that considers the attitude angular velocity amplitude constraint is used to achieve attitude closed-loop tracking control.

[0008] Furthermore, the state space includes the state space of aircraft formation A and the state space of aircraft formation B; The state space is represented as follows: in, For state space, Indicates the first i The state space of an aircraft; in, For the first i The three-dimensional position coordinates of the aircraft; For the first i The three-dimensional velocity vector of the aircraft; For the first i The attitude Euler angles of the aircraft.

[0009] Furthermore, the action space is the action vector allocated and adjusted by each individual aircraft in aircraft formation A to the attitude orientation of each individual aircraft in aircraft formation B, expressed as: in, Indicates the first in formation A of aircraft i The attitude adjustment amount of an aircraft. and These represent the adjustment components of the Euler angles in the three directions corresponding to the attitude.

[0010] Furthermore, the input to the policy network includes the state space of aircraft formation A and the state space of aircraft formation B, and the output is the action space of each individual aircraft in aircraft formation A. This is used to generate the initial desired attitude of each aircraft; The objective function of the policy network is expressed as: in, The parameters represent the policy network. Representing the state space, Represents the action space, express The reward value at any moment, Indicates the discount factor. Indicates the time step; This represents the expected value.

[0011] Furthermore, the input to the value assessment network includes the state space of each individual aircraft in aircraft formation A. With the corresponding action space The output is the value function of the state-action pair. This is used to evaluate the quality of the actions generated by the policy network; The objective function of the value assessment network is expressed as: in, The parameters representing the value assessment network, Represents the state space at the next time step. This indicates that the target policy network is in state The actions generated below, The parameters represent the target value assessment network. Indicates the reward value. Indicates the discount factor. This represents the expectation for the state transition sample.

[0012] Furthermore, the reward function during the training process of the multi-agent system is expressed as: in, Indicates the first The reward value at any given moment; Reward for posture adjustment time; Rewards are based on relative distance. Rewards for attitude tracking performance; The attitude adjustment time reward is represented as: in, Indicates the time step. Indicates the time weighting coefficient; The relative distance reward is expressed as: in, Indicates the first in formation A of aircraft The position vectors of each aircraft. Indicates the first in formation B of the aircraft The position vectors of each aircraft. This represents the distance weighting coefficient; The attitude tracking performance reward is expressed as: in, Indicates the first in formation A of aircraft The current attitude Euler angles of the aircraft Indicates the first The desired attitude of an aircraft This represents the performance weighting coefficient.

[0013] Furthermore, the training process for the multi-agent system includes: Initialize the parameters of the policy network and the value evaluation network, as well as the parameters of the corresponding target policy network and target value evaluation network; Sample the current state from the environment. And perform the action To obtain the next state and reward value R; The experience obtained from sampling Store in the experience replay buffer; A batch of experiences is randomly sampled from the experience replay buffer, the parameters of the value evaluation network are updated, and the objective function of the policy network is maximized by minimizing the objective function of the value evaluation network and updating the parameters of the policy network. Update the target network parameters using a soft update method; Repeat the sampling and update steps until the policy network converges.

[0014] Furthermore, the step of inputting the initial desired attitude into the forward-facing angle calculation module for constraint correction, and combining the lighting conditions and target direction information to generate the final desired attitude that meets the constraint requirements, specifically includes: Determine the inputs to the front-light angle calculation module, including the initial desired attitudes of each aircraft in aircraft formation A. Current solar direction vector and the direction vectors of each target in aircraft formation B ; Based on the initial desired posture Calculate the reference solar vector for each spacecraft: in, Indicates the first b The spacecraft references the solar vector; This represents the rotation matrix generated by the Euler angles of the initial attitude; By optimizing the rotation angle The initial desired attitude is corrected so that the angle between the corrected solar vector and the target direction vector satisfies the forward angle constraint. An optimization problem is then constructed using geometric constraints: in, The optimized attitude Euler angles, This is a rotation superposition operation; The optimized rotation angle Compared with the initial desired posture Synthesize and generate the final desired pose. .

[0015] Furthermore, the adoption of an adaptive PID control algorithm considering attitude angular velocity amplitude constraints to achieve attitude closed-loop tracking control based on the final desired attitude specifically includes: Based on the desired final attitude, the current attitude error is calculated using the following formula: in, Indicates the final desired posture, Indicates the current attitude of the aircraft. Indicates the current attitude error; limits the pitch angle range. ; Based on the calculated current attitude error The control output is generated by adaptive PID control, and is expressed as follows: in, To control the output; and These are the adaptive proportional gain, integral gain, and differential gain matrices, respectively. and These are the desired angular velocity and the current angular velocity vectors, respectively.

[0016] Furthermore, the adaptive law of the adaptive PID is: in, , and These represent the adaptive adjustment rates of the corresponding gain matrices; The error surface is defined as follows: ; This is the error surface adjustment coefficient; and Adaptive learning rate; and This is the anti-drift coefficient; and This is the initial gain value for the PID controller.

[0017] Compared with the prior art, the present invention has the following advantages: (1) In the prior art, the cooperative attitude pointing allocation of multiple aircraft suffers from control conflicts and difficulties in global coordination. Existing methods often fail to achieve intelligent division of labor and cooperation among multiple aircraft, resulting in low task execution efficiency and frequent attitude conflicts. To address this technical problem, this invention constructs a state space and action space for multiple aircraft and trains it based on a multi-agent reinforcement learning framework (MADDPG algorithm) with centralized training and distributed execution, thereby achieving cooperative decision-making and optimal attitude pointing allocation among aircraft. This technical feature can effectively avoid attitude pointing conflicts, improve the overall tracking accuracy and task adaptability of multiple aircraft, and at the same time ensure the convergence and cooperative efficiency of the algorithm in high-dimensional nonlinear environments.

[0018] (2) Existing technologies struggle to simultaneously consider aircraft attitude tracking time, relative distance, and tracking performance in multi-constraint tasks, resulting in slow task execution or insufficient attitude tracking accuracy. To address this issue, this invention designs a multi-agent reward function that includes attitude tracking time rewards, aircraft relative distance rewards, and attitude tracking performance rewards, enabling multiple aircraft to comprehensively consider tracking efficiency, spatial safety distance, and attitude accuracy during attitude allocation. This technical feature enables rapid attitude adjustment and efficient tracking of aircraft in highly constrained environments, while maintaining a safe distance between aircraft, thus improving overall task completion efficiency.

[0019] (3) Existing technologies directly incorporate lighting conditions (such as the direction of the sun) into the action dimension, leading to an expansion of the action space dimension and difficulties in learning and exploration, and making it difficult to adapt to dynamic lighting changes. To address this technical problem, this invention designs a front-light angle calculation module to decouple the lighting geometric constraints from the MADDPG algorithm and calculate the front-light angle independently. This technical feature enables the generation of feasible attitudes under dynamic lighting conditions without expanding the action dimension, and stably outputs the reference attitude, significantly shortening the training and inference latency, while enhancing the aircraft's response capability to sudden changes in lighting.

[0020] (4) Existing technologies lack guarantees for rapid response and stable output in attitude tracking closed-loop control, leading to large tracking errors or output jitter in aircraft. To address this technical problem, this invention models the attitude dynamics of the aircraft and combines it with an adaptive PID controller that considers the amplitude constraints of attitude angular velocity to achieve attitude closed-loop tracking. This technical feature ensures that the aircraft can quickly respond to desired attitude changes, achieve real-time adjustment and stable control of attitude errors, and effectively solve the closed-loop stability problem in the process of multi-aircraft cooperative tracking.

[0021] (5) Existing technologies struggle to formally embed task constraints (such as safety distance and lighting conditions) into the learning framework during multi-agent collaborative training, potentially causing the learning strategy to fail or become infeasible in complex environments. To address this issue, this invention achieves stable convergence of multi-agent collaborative training by using an experience replay buffer and soft-update policy network and value evaluation network parameters. Furthermore, it ensures that task constraints are effectively considered during training through a reward function and a forward-facing angle constraint. These technical features enable the learned strategy to be feasible and stable in complex, multi-constraint environments, significantly improving the overall execution reliability and adaptability of multi-aircraft collaborative tasks.

[0022] (6) Existing technologies lack an effective coordination mechanism between distributed decision-making and mission-level closed-loop control for multiple aircraft, resulting in unsatisfactory actual tracking accuracy and mission execution performance. To address this technical problem, this invention combines the initial desired attitude generated by MADDPG with the front-light angle calculation module and the adaptive PID closed-loop control module to achieve closed-loop fusion of attitude pointing allocation, mission constraint calculation, and attitude tracking control. This technical feature not only improves the satisfaction of mission constraints and attitude tracking accuracy but also significantly enhances the stability and execution efficiency of multiple aircraft in dynamic environments, achieving global optimization of mission execution and system scalability. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the intelligent allocation method for aircraft attitude tracking of the present invention; Figure 2 An attitude rose diagram of an aircraft employing the attitude tracking and allocation method described in this invention; Figure 3 A diagram showing the attitude assignment and pointing relationship of an aircraft using the present invention; Figure 4 This is a graph showing the change in reward value of the MADDPG algorithm in this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] Example 1: This embodiment provides an intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning, such as... Figure 1 This includes the following steps: Step S1: Construct the aircraft state space and motion space. The state space is determined based on the position, velocity, and attitude information of each aircraft in the multi-aircraft formation, and the corresponding motion space is determined based on attitude adjustment requirements. Specifically, this includes: S11. Establish the state space of aircraft formation A: In this application, aircraft formation A includes N There are 10 aircraft. The state parameters of each aircraft i include its position coordinates. Velocity vector Euler angles represent the attitude. The three-dimensional position coordinates of the aircraft and velocity vector It is represented using an Earth-Centered Inertial (ECI) coordinate system, with the origin located at the Earth's center, the Z-axis pointing to the North Pole, and the X-axis pointing to the vernal equinox.

[0026] S12. Establish the B state space for the aircraft formation: In this embodiment, the aircraft formation B contains M aircraft, and its state space includes positions. ,speed Coordinates of individuals in aircraft formation B Also represented using the ECI coordinate system. Incorporating the target formation's position and velocity information into the state space helps aircraft formation A perceive dynamic changes in the target during decision-making, enabling more reasonable attitude pointing allocation. For example... Figure 2 As shown, this is an attitude angle rose diagram formed by the change of attitude angles of each aircraft over time when using the intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning as described in this invention. It is used to characterize the angle distribution characteristics and convergence of the aircraft attitude adjustment process. S13. Establish the A-stage maneuver space for the aircraft formation: The action space of aircraft formation A is for attitude and pointing allocation adjustment. .

[0027] S14. Construct the reward function for the pose tracking and assignment algorithm: To ensure safe, effective, and efficient attitude tracking, the MADDPG algorithm reward function is constructed by comprehensively considering attitude tracking time, relative distance, and attitude tracking performance. To encourage rapid attitude tracking completion, the designed attitude tracking time reward is: in, For time step, This is a weighting coefficient. This term penalizes the increase in time, guiding the strategy to complete attitude adjustment in a shorter time, thereby improving the system's response speed.

[0028] To encourage maintaining an appropriate distance from individuals in aircraft formation B, the relative distance reward is designed as follows: in, The distance between aircraft numbered i in aircraft formation A and aircraft numbered j in aircraft formation B; This is a weighting coefficient. This term is used to constrain the relative positional relationships between aircraft, avoiding safety risks caused by excessively close proximity, while ensuring a reasonable observation or coordination distance, thereby improving system safety and mission effectiveness.

[0029] To encourage accurate attitude tracking, the designed attitude tracking performance reward function is as follows: in, It is the deviation between the current attitude of an individual in aircraft formation A and the attitude of the target. This is a weighting coefficient. This term penalizes attitude errors, guiding the aircraft to achieve high-precision attitude tracking, thereby improving mission execution quality.

[0030] The total reward function designed by combining the above factors for: like Figure 4 This is a graph showing the change in reward value with the number of training steps during the multi-agent reinforcement learning training process in this invention. It is used to characterize the convergence process and learning stability of the algorithm under the constructed reward function.

[0031] S15. Construct the objective function for the attitude tracking assignment algorithm: The Actor objective function is designed to maximize the cumulative reward: in, These are the parameters of the Actor network. It is a discount factor; The Critic objective function is to minimize the Bellman error: in, These are the parameters of the Critic network. These are the parameters of the target Critic network.

[0032] Step S2: Construct and train a multi-agent reinforcement learning framework based on the state space and action space. Obtain the attitude pointing assignment policy for multi-aircraft cooperative tasks through interactive learning. The multi-agent framework includes a policy network and a value evaluation network, specifically comprising: In this step, a multi-agent deep deterministic policy gradient (MADDPG) framework with centralized training and distributed execution is adopted to achieve collaborative decision-making and stable learning among multiple aircraft. Multi-aircraft systems are characterized by strong coupling and non-stationarity. Traditional single-agent methods are prone to policy oscillations in multi-agent interactive environments, while the centralized evaluation mechanism can introduce global information during the training phase, improving convergence stability and collaborative performance.

[0033] S21. Constructing the MADDPG algorithm framework: Construct a strategy network and a value assessment network. For the first... i Each aircraft has a policy network whose input is the joint state information of aircraft formation A and aircraft formation B, used to characterize the cooperative mission environment; the output is an action vector. , used to represent the adjustment amount in the three pose directions. The policy network adopts a multi-layer fully connected structure with an input layer dimension of 9. The hidden layer uses the nonlinear activation function ReLU to enhance the network's ability to express complex nonlinear relationships. The output layer uses the Tanh function for normalization to ensure that the output action meets the constraints of a limited range, thereby avoiding system instability caused by excessive pose adjustment.

[0034] For the Critic value evaluation network, the input is the joint state and joint action of all agents, and the output is the corresponding state-action value function. This structure can explicitly model the coupling relationships between multiple agents, allowing each agent to consider the behavior of other agents when updating its policy, thereby improving overall collaborative capabilities. The Critic network also employs a multi-layer fully connected structure, with ReLU functions in the hidden layers and linear functions in the output layer to ensure the continuity and differentiability of the value function.

[0035] S22. This embodiment uses the following steps to perform reinforcement learning training for attitude pointing assignment: Step 1: Initialization: Initialize the parameters of the Actor network and Critic network, and initialize the parameters of the target Actor network and target Critic network; Step 2: Sampling: Sampling the state from the environment and actions Perform the action to obtain the next state. and rewards ; Step 3: Storage: Storing the experience Store in the experience replay buffer; Step 4: Update: Sample a batch of data from the experience replay buffer and update the Critic network: Update the Actor network: Update the target network: ; Step 5: Repeat steps 2-4 until convergence.

[0036] Step S3: Use the trained policy network to make decisions about the current state of the aircraft and output the initial expected attitude for each aircraft. Using the MADDPG algorithm constructed above, individuals in aircraft formation A can be directed to individuals in aircraft formation B with a better attitude, generating the initial desired attitude for individuals in aircraft formation A to track individuals in aircraft formation B.

[0037] Step S4: Input the initial desired attitude into the forward-lighting angle calculation module for constraint correction. Combine the lighting conditions and target direction information to generate the final desired attitude that meets the constraint requirements, including: S41. Determine the input to the module, i.e., the MADDPG algorithm module mentioned above generates the initial desired pose. Current solar direction vector The direction vector of an individual in aircraft formation B .

[0038] S42. Based on the initial attitude Calculate the current solar vector .

[0039] S43. To meet the constraints of observation in direct sunlight, the observation direction of the spacecraft needs to maintain a preset angle with the direction of the sun. Therefore, a minor correction is made to the initial desired attitude, introducing an attitude increment. The attitude is optimized and adjusted. The optimal attitude correction is solved by constructing the following optimization problem: S44, Output ,satisfy This relationship is used to ensure that the final attitude meets the preset requirements for the front-light observation angle.

[0040] Step S4 achieves the goal of maintaining the initial desired posture as much as possible. Without causing significant deviations, minimize the attitude correction amount. This design ensures that the solar direction and the target observation direction in the spacecraft coordinate system satisfy a preset angle constraint, thus balancing mission pointing requirements and illumination constraints. This design avoids directly introducing complex geometric constraints into reinforcement learning, thereby reducing the difficulty of policy learning and improving convergence speed. It decouples illumination geometric constraints from the reinforcement learning decision-making process, ensuring attitude adjustment continuity and minimal perturbation while enabling the spacecraft to stably meet the front-lighting conditions in imaging or observation missions, improving the system's adaptability to dynamic lighting environments and increasing mission success rate.

[0041] Step S5: Based on the final desired attitude, an adaptive PID control algorithm considering attitude angular velocity amplitude constraints is used to implement attitude closed-loop tracking control, specifically including: S51. The equations for the attitude dynamics of the aircraft are as follows: In the above formula, This represents the angular velocity of the aircraft's coordinate system relative to the inertial frame. This represents the moment of inertia of the aircraft. The control torque is generated by the reaction wheel or thruster. The aircraft attitude is represented using Euler angles. The attitude dynamics equations describe the change of the angular velocity vector over time under the action of the external torque. In this embodiment of the invention, only the control torque is considered, ignoring other interfering factors, and these factors are fixed to the aircraft coordinate system.

[0042] S52. The attitude tracking error is reduced to zero using an adaptive PID controller that considers the attitude angular velocity amplitude constraint. The specific implementation process is as follows: S521. The input of the adaptive PID controller considering the attitude angular velocity amplitude constraint is the desired attitude. With actual posture error To avoid gimbal lock, the range of Euler angle values ​​needs to be limited. .

[0043] S522, the control output of the adaptive PID is: S523, The adaptive law of adaptive PID is: in, , and Indicates the adaptive learning rate. For error surface, , and It is the anti-drift modification factor. and This represents the initial values ​​of the controller parameters. This control law improves response speed through a proportional term, eliminates steady-state deviation through an integral term, and suppresses overshoot through a derivative term, thereby achieving fast and smooth attitude tracking. For example... Figure 3The diagram shown illustrates the attitude pointing assignment relationship between aircraft formation A and aircraft formation B after applying the method of the present invention, and is used to explain the target pointing correspondence and collaborative assignment effect among multiple aircraft. Through the above steps, a closed-loop control process from the final desired attitude to the control torque output is achieved, enabling the aircraft to quickly and stably track the target attitude in complex space environments. Simultaneously, the adaptive mechanism improves the system's robustness to model uncertainties and environmental changes, effectively connecting with the front-end multi-agent decision-making module, thereby enhancing the overall performance and reliability of the multi-aircraft cooperative attitude control system.

[0044] In summary, this invention proposes an intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning. This method uses a reinforcement learning algorithm to learn the optimal pointing allocation strategy for each aircraft. Based on the algorithm's results, the initial desired attitude is obtained and used as input to the forward beam angle calculation module to generate the final desired attitude, which is then passed to the adaptive PID control module. Based on the desired attitude and the current state, the adaptive PID control module generates specific attitude control actions, achieving optimal attitude pointing tracking of individuals in aircraft formation A towards individuals in aircraft formation B.

[0045] Example 2: This embodiment provides an intelligent attitude tracking allocation system for aircraft based on multi-agent reinforcement learning, including a state construction module, a multi-agent decision-making module, a front-light angle calculation module, and an adaptive attitude control module. The modules are connected through data interaction to form a complete attitude pointing allocation and control closed loop.

[0046] The state construction module is used to acquire the operational state information of multiple aircraft formations and construct a unified state space and action space. This module receives the position, velocity, and attitude information of each aircraft in aircraft formation A, and simultaneously receives the position and velocity information of the target aircraft in aircraft formation B. It processes and expresses the multi-source data in a unified manner, providing basic input for subsequent intelligent decision-making.

[0047] The multi-agent decision-making module generates aircraft attitude orientation assignment strategies based on multi-agent reinforcement learning. This module includes a policy network and a value evaluation network. By jointly modeling the state information of aircraft formation A and aircraft formation B, it outputs the attitude adjustment actions of each aircraft, thereby obtaining the initial desired attitude. Through training and learning, this module achieves collaborative decision-making among multiple aircraft, enabling each aircraft to consider both overall mission requirements and individual states during mission execution, thus achieving reasonable attitude orientation assignment.

[0048] The front-light angle calculation module is used to constrain and correct the initial desired attitude. This module combines illumination conditions and target direction information to apply geometric constraints to the attitude orientation. While meeting the requirements for front-light observation, it adjusts the attitude to generate the final desired attitude that satisfies the constraints. By decoupling illumination constraints from the decision-making process, the learning difficulty of the agent can be reduced, and the system's adaptability to complex lighting environments can be improved.

[0049] The adaptive attitude control module is used to achieve closed-loop control of the aircraft's attitude based on the desired final attitude. This module generates corresponding control commands based on the error between the aircraft's current attitude and the target attitude, enabling real-time attitude adjustment and stable tracking. Simultaneously, by introducing an adaptive adjustment mechanism, the controller parameters can dynamically change according to the system state, thereby improving control accuracy and system robustness.

[0050] Through the coordinated work of the above modules, a complete processing flow from state perception, intelligent decision-making, constraint correction to closed-loop control is realized, enabling multi-aircraft formations to achieve efficient and stable attitude pointing allocation and tracking control under complex environments and multiple constraints.

[0051] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent attitude tracking allocation of aircraft based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Construct the state space and action space of the aircraft. Determine the state space based on the position, velocity and attitude information of each aircraft in the multi-aircraft formation, and determine the corresponding action space based on the attitude adjustment requirements. A multi-agent reinforcement learning framework is constructed and trained based on the state space and action space. An attitude pointing allocation strategy for multi-aircraft cooperative tasks is obtained through interactive learning. The multi-agent system includes a policy network and a value evaluation network; The trained policy network is used to make decisions about the current state of the aircraft and outputs the initial expected attitude for each aircraft. The initial desired posture is input into the forward angle calculation module for constraint correction. Combined with the lighting conditions and target direction information, the final desired posture that meets the constraint requirements is generated. Based on the desired final attitude, an adaptive PID control algorithm that considers the attitude angular velocity amplitude constraint is used to achieve attitude closed-loop tracking control.

2. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that, The state space includes the state space of aircraft formation A and the state space of aircraft formation B; The state space is represented as follows: in, For state space, Indicates the first i The state space of an aircraft; in, For the first i The three-dimensional position coordinates of the aircraft; For the first i The three-dimensional velocity vector of the aircraft; For the first i The attitude Euler angles of the aircraft.

3. The intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning according to claim 1, characterized in that, The action space refers to the action vectors that each individual aircraft in aircraft formation A assigns to adjust the attitude and orientation of each individual aircraft in aircraft formation B, expressed as: in, Indicates the first in formation A of aircraft i The attitude adjustment amount of an aircraft. and These represent the adjustment components of the Euler angles in the three directions corresponding to the attitude.

4. The intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning according to claim 1, characterized in that, The input to the policy network includes the state space of aircraft formation A and the state space of aircraft formation B, and the output is the action space of each individual aircraft in aircraft formation A. This is used to generate the initial desired attitude of each aircraft; The objective function of the policy network is expressed as: in, The parameters represent the policy network. Representing the state space, Represents the action space, express The reward value at any moment, Indicates the discount factor. Indicates the time step; This represents the expected value.

5. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that, The input to the value assessment network includes the state space of each individual aircraft in aircraft formation A. With the corresponding action space The output is the value function of the state-action pair. This is used to evaluate the quality of the actions generated by the policy network; The objective function of the value assessment network is expressed as: in, The parameters representing the value assessment network, Represents the state space at the next time step. This indicates that the target policy network is in state The actions generated below, The parameters represent the target value assessment network. Indicates the reward value. Indicates the discount factor. This represents the expectation for the state transition sample.

6. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that, The reward function for the multi-agent training process is expressed as: in, Indicates the first The reward value at any given moment; Reward for posture adjustment time; Rewards are based on relative distance. Rewards for attitude tracking performance; The attitude adjustment time reward is represented as: in, Indicates the time step. Indicates the time weighting coefficient; The relative distance reward is expressed as: in, Indicates the first in formation A of aircraft The position vectors of each aircraft. Indicates the first in formation B of the aircraft The position vectors of each aircraft. This represents the distance weighting coefficient; The attitude tracking performance reward is expressed as: in, Indicates the first in formation A of aircraft The current attitude Euler angles of the aircraft Indicates the first The desired attitude of an aircraft This represents the performance weighting coefficient.

7. The intelligent allocation method for aircraft attitude tracking based on multi-agent reinforcement learning according to claim 1, characterized in that, The training process for the multi-agent system includes: Initialize the parameters of the policy network and the value evaluation network, as well as the parameters of the corresponding target policy network and target value evaluation network; Sample the current state from the environment. And perform the action To obtain the next state and reward value R; The experience obtained from sampling Store in the experience replay buffer; A batch of experiences is randomly sampled from the experience replay buffer, the parameters of the value evaluation network are updated, and the objective function of the policy network is maximized by minimizing the objective function of the value evaluation network and updating the parameters of the policy network. Update the target network parameters using a soft update method; Repeat the sampling and update steps until the policy network converges.

8. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that, The step of inputting the initial desired attitude into the forward angle calculation module for constraint correction, and combining the illumination conditions and target direction information to generate the final desired attitude that meets the constraint requirements, specifically includes: Determine the inputs to the front-light angle calculation module, including the initial desired attitudes of each aircraft in aircraft formation A. Current solar direction vector and the direction vectors of each target in aircraft formation B ; Based on the initial desired posture Calculate the reference solar vector for each spacecraft: in, Indicates the first b The spacecraft references the solar vector; This represents the rotation matrix generated by the Euler angles of the initial attitude; By optimizing the rotation angle The initial desired attitude is corrected so that the angle between the corrected solar vector and the target direction vector satisfies the forward angle constraint. An optimization problem is then constructed using geometric constraints: in, The optimized attitude Euler angles, This is a rotation superposition operation; The optimized rotation angle Compared with the initial desired posture Synthesize and generate the final desired pose. .

9. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 1, characterized in that, The attitude closed-loop tracking control, based on the final desired attitude, employs an adaptive PID control algorithm that considers attitude angular velocity amplitude constraints. Specifically, this includes: Based on the desired final attitude, the current attitude error is calculated using the following formula: in, Indicates the final desired posture, Indicates the current attitude of the aircraft. Indicates the current attitude error; limits the pitch angle range. ; Based on the calculated current attitude error The control output is generated by adaptive PID control, and is expressed as follows: in, To control the output; and These are the adaptive proportional gain, integral gain, and differential gain matrices, respectively. and These are the desired angular velocity and the current angular velocity vectors, respectively.

10. The intelligent attitude tracking allocation method for aircraft based on multi-agent reinforcement learning according to claim 9, characterized in that, The adaptive law of the adaptive PID is: in, , and These represent the adaptive adjustment rates of the corresponding gain matrices; The error surface is defined as follows: ; This is the error surface adjustment coefficient; and Adaptive learning rate; and This is the anti-drift coefficient; and This is the initial gain value for the PID controller.

Images