Spacecraft attitude control method based on reinforcement learning

Abstract

The invention discloses a spacecraft attitude control method based on reinforcement learning, and belongs to the technical field of spacecraft attitude control. The method aims at solving the problem that an existing attitude control method cannot carry out fault reconstruction when fault information is unknown, and the control precision is affected. Comprising the following steps: constructing a strategy network based on two full-connection linear layers and two branch linear layers; constructing a value network based on the two full-connection linear layers; the two networks take an error quaternion and an error angular velocity as state input quantities, the strategy network outputs beta distribution parameters of four flywheel actions of the spacecraft, and then the four flywheel actions are calculated based on the beta distribution parameters; the value network outputs the state prediction value of the current flywheel action; the learnable parameters of the value network are adjusted with the purpose of minimizing the error between the state prediction value and the state real value; adjusting learnable parameters of the strategy network by taking maximization of a training target constructed by the dominant function as a target; according to the invention, intelligent attitude control of the in-situ fault reconstruction capability is realized.

Description

Technical Field

[0001] This invention relates to a spacecraft attitude control method based on reinforcement learning, belonging to the field of spacecraft attitude control technology. Background Technology

[0002] Existing spacecraft attitude control methods include proportional-derivative-integral (PID) control, observer-based sliding mode control, direct adaptive control, and reinforcement learning control based on value methods.

[0003] The PID control principle suffers from slow convergence, difficulty in constructing the integral element, and the need to adjust parameters based on prior spacecraft inertia using the root locus method. It exhibits poor accuracy and the potential for divergence in attitude control scenarios for modern complex spacecraft with variable inertia and flexibility. A fractional-order finite-time control method for flexible spacecraft was designed by adding a fractional-order element to the PID controller. The stability of the control was rigorously proven using a modified Lyapunov function, and simulations showed that the controller demonstrated stronger anti-interference capabilities. However, the controller's mathematical structure is extremely complex, making practical engineering difficult.

[0004] Treating dynamic characteristics such as flexible vibrations, inertia changes, and environmental torques that do not conform to the rigid body attitude dynamics model as total disturbances, and observing and compensating for them through observers and parameter identification, is the most common approach to attitude control problems of complex spacecraft with flexible variable inertia. Linear or nonlinear extended state observers (ESOs) have become the most common disturbance observation method in recent years. Based on the design of a pyramid-shaped single-frame control moment gyroscope (SGCMGs) and a flywheel system integrated control law, a sliding mode attitude control law has been proposed. This control law observes spacecraft flexible accessory disturbances through an extended state observer, possesses finite-time stable provable characteristics, and has been verified by a ground-based semi-physical simulation system. Disturbance observers (DOBs) are intuitive in principle and simple in design, and still have some applications in recent research on the attitude control of complex spacecraft. To eliminate the effects of system parameter uncertainties and external torques, a finite-time integral sliding mode disturbance observer was designed. Used in conjunction with time-varying sliding mode control, it achieved high-precision attitude control of a flexible liquid-filled spacecraft. Furthermore, flexible vibrations, liquid sloshing, and external disturbances were treated as lumped disturbances, and the fuzzy terminal sliding mode control strategy of the disturbance observer effectively suppressed system jitter. While the above method achieved attitude control of complex spacecraft, it also suffers from problems such as convergence dependence on observer design, poor fault reconstructive capability, complex mathematical structure, and difficult programming. Moreover, sliding mode control is a variable structure control, and the problem of chattering in the control output after the system reaches the sliding surface cannot be fundamentally solved.

[0005] Using adaptive control with automatic control gain adjustment capability is another simpler approach to spacecraft attitude controller design. An auxiliary signal can be used to establish the relationship between the boundary conditions and internal state of the flexible spacecraft dynamics system. Based on this, a direct adaptive method can be adopted to achieve boundary-tolerant control, ensuring asymptotic stability of attitude control. To address the attitude control problem of flexible spacecraft with time-varying external disturbances, an adaptive law is used to estimate the upper bound of the disturbance and compensate for it in the predetermined-time sliding mode control law. This method considers the dynamic characteristics of SGCMGs. Due to the existence of the adaptive law, the work of adjusting the controller parameters based on the prior inertia before spacecraft launch is greatly reduced, and no observation loop needs to be designed. However, the adaptive law has an upper bound on the allowable disturbance. Once the inertia change, environmental disturbance, or system fault exceeds the upper bound, the control efficiency will drop sharply, or even cause control divergence.

[0006] There have been attempts to use reinforcement learning methods to solve spacecraft attitude control problems. For example, two value-based reinforcement learning methods, DDPG (Deep Deterministic Policy Gradient) and TD3 (Dual Delayed Deep Deterministic Policy Gradient), have been used to train neural networks to achieve attitude control of rigid spacecraft. However, the agents trained using these methods can only obtain action values ​​based on the current spacecraft attitude and select the action with the highest value. The lack of explicit actions leads to poor end-point control accuracy, a lack of engineering feasibility, and potential generalization problems.

[0007] Existing attitude control algorithms rely on prior satellite inertia data for parameter tuning, have poor resistance to environmental disturbances and control system errors, cannot reconstruct faults when the fault information is unknown, and cannot adapt to high-dynamic continuous target tracking and observation tasks where the actuators are at risk of failure. Summary of the Invention

[0008] To address the problem that existing attitude control methods cannot reconstruct faults when there is no known fault information, thus affecting control accuracy, this invention provides a spacecraft attitude control method based on reinforcement learning.

[0009] The present invention provides a spacecraft attitude control method based on reinforcement learning, comprising:

[0010] A policy network is constructed based on two consecutive fully connected linear layers and two parallel branched linear layers. The policy network takes error quaternions and error angular velocity as state inputs and outputs the beta distribution parameters of the four flywheel actions of the spacecraft. The four flywheel actions are then calculated based on the beta distribution parameters.

[0011] A value network is constructed based on two consecutive fully connected linear layers; the value network uses error quaternions and error angular velocity as state inputs and outputs the state prediction value of the current flywheel action.

[0012] During training, the learnable parameters of the value network are adjusted with the goal of minimizing the error between the predicted state value and the actual state value. Based on the predicted state value output by the value network, the advantage function is calculated, and the learnable parameters of the policy network are adjusted with the goal of maximizing the training objective constructed by the advantage function. Finally, the obtained policy network outputs the current control quantity of the four flywheel actions based on the current error quaternion and the current error angular velocity.

[0013] The beneficial effects of this invention are as follows: The method of this invention uses a small-scale fully connected neural network as the attitude control agent. Through an improved proximal policy optimization algorithm, the network is trained to master a generalized, disturbance-resistant attitude tracking control strategy. Based on the error quaternion and error angular velocity, the agent can manipulate the flywheel system to achieve high-precision attitude maneuvering and attitude tracking control under conditions of uncertain satellite inertia, large environmental torque disturbances, large attitude observation errors, inaccurate flywheel output torque, and unknown flywheel faults.

[0014] A highly convergent training method is adopted: by improving the PPO algorithm (proximal policy optimization algorithm) and the phased training scheme, the convergence of reinforcement learning training is greatly improved, and a convergent agent can be obtained with only a few attempts to adjust hyperparameters and random seeds.

[0015] Intelligent attitude control with in-situ fault reconfiguration capability: The agent can adapt to various flywheel faults and failures without fault information, program structure, or agent parameters, and complete the attitude control task without losing control accuracy. Attached Figure Description

[0016] Figure 1 This is a flowchart of the reinforcement learning training process for the spacecraft attitude control method based on reinforcement learning described in this invention.

[0017] Figure 2 This is a schematic diagram of a strategy network structure;

[0018] Figure 3 This is a schematic diagram of a value network structure;

[0019] Figure 4 This is a schematic diagram of error quaternions during the intelligent attitude maneuvering process of a spacecraft.

[0020] Figure 5 This is a schematic diagram of the error angular velocity during the intelligent attitude maneuver of a spacecraft;

[0021] Figure 6 This is a schematic diagram of the expected output torque of the intelligent agent during the intelligent attitude maneuver of a spacecraft;

[0022] Figure 7 This is a schematic diagram of the spacecraft flywheel rotation speed during the intelligent attitude maneuver of the spacecraft;

[0023] Figure 8 This is a schematic diagram of the quaternion results of the random target shooting error in the intelligent attitude control of a spacecraft.

[0024] Figure 9 This is a schematic diagram of the angular velocity results of random target shooting error in the intelligent attitude control of a spacecraft. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Specific Implementation Method 1: Combination Figures 1 to 3 As shown, this invention provides a spacecraft attitude control method based on reinforcement learning, including:

[0027] A policy network is constructed based on two consecutive fully connected linear layers and two parallel branched linear layers. The policy network takes error quaternions and error angular velocity as state inputs and outputs the beta distribution parameters of the four flywheel actions of the spacecraft. The four flywheel actions are then calculated based on the beta distribution parameters.

[0028] A value network is constructed based on two consecutive fully connected linear layers; the value network uses error quaternions and error angular velocity as state inputs and outputs the state prediction value of the current flywheel action.

[0029] During training, the learnable parameters of the value network are adjusted with the goal of minimizing the error between the predicted state value and the actual state value. Based on the predicted state value output by the value network, the advantage function is calculated, and the learnable parameters of the policy network are adjusted with the goal of maximizing the training objective constructed by the advantage function. Finally, the obtained policy network outputs the current control quantity of the four flywheel actions based on the current error quaternion and the current error angular velocity.

[0030] This implementation proposes a complete scheme for the design, training, and engineering deployment of an intelligent attitude control agent. First, the error quaternion between the satellite's current attitude and the target attitude, and the relationship between the error quaternion and the satellite attitude dynamics equations, are derived. All internal and external disturbances to the control system are modeled as the basis for reinforcement learning training. The structures of the attitude control policy network and value network are designed, and a complete reinforcement learning training process and training reward for the proximal policy optimization (PPO) algorithm are constructed. Finally, an on-board engineering deployment procedure for the attitude control agent is designed to ensure that it can achieve the expected control effect in actual missions.

[0031] As an example, combined Figure 2 and Figure 3 As shown, the two fully connected linear layers of the policy network are each configured with 150 neurons, and the outputs of the two fully connected linear layers are nonlinearly activated using the hyperbolic tangent function.

[0032] Each of the two branch linear layers is configured with 3 neurons, and the outputs of the two branch linear layers are activated using flexible rectified functions.

[0033] This implementation selects quaternions—the attitude output format most suitable for star sensors, sun sensors, gyroscopes, and other sensors—to describe satellite attitude in the most convenient way for engineering practice. To achieve tasks such as attitude maneuvering and tracking control from any attitude to any other attitude, an error quaternion is proposed.

[0034] Furthermore, the state input is represented as :

[0035] ,

[0036] In the formula For error quaternions, The error angular velocity in the satellite's body coordinate system. Indicates transpose;

[0037] ,

[0038] ,

[0039] In the formula For target quaternions conjugate, Let be the quaternion at time t. Represents the multiplication of quaternions.

[0040] The attitude kinematics equations describe the relationship between the error quaternion and the angular velocity using the derivative of the quaternion.

[0041] The calculation method for the four flywheel motions is as follows:

[0042] The basic values ​​of flywheel motion are obtained by randomly sampling from the beta distribution parameters. The basic values ​​of flywheel motion are then mapped to the range of flywheel output torque to obtain the expected output torque of flywheel motion.

[0043]

[0044] in, This represents the base motion value for the i-th flywheel. Indicates the beta distribution, Let be the success parameters of the i-th flywheel beta distribution. Let be the failure parameter of the i-th flywheel beta distribution. and The numerical range of all values ​​is [0,1].

[0045] The expected output torque of the i-th flywheel for:

[0046] ,

[0047] In the formula This represents the maximum output torque of the i-th flywheel.

[0048] Reinforcement learning training process:

[0049] A two-layer fully connected neural network was trained using reinforcement learning. This network learned a strategy from attitude maneuvering mission simulations and manipulated the satellite's four flywheels to perform tasks such as attitude maneuver reversal and attitude tracking control. The reinforcement learning process is as follows: Figure 1 As shown. The policy network directly controls the satellite, serving as the training target, and its structure is designed as follows. Figure 2 As shown.

[0050] Figure 2 Input quantity based on the current state The decision is made for the next control step, outputting four sets of beta distribution parameters for the action, with the desired output torque from... and The determined beta distribution is sampled and scaled to fit within the flywheel output torque range. The policy network is then configured based on the state input. The statistical distribution of the output action 'a' is called the strategy. The trainable parameters θ within the policy network determine the specific form of this distribution; therefore, the process of the policy neural network making an action can be written as:

[0051] .

[0052] This approach helps neural networks better explore the action space and accelerates training. The value network guides the policy network to learn in the right direction; its structure is designed as follows... Figure 3 As shown.

[0053] The value network is based on the satellite's current state input. The decision-making process categorizes the current state as good or bad, outputting its value. The learnable parameter ω within the value network determines the magnitude of the state's value. Furthermore, since the policy network and value network are trained synchronously, the policy network, having a policy... At that time, the value network outputs the state value under the current policy. .

[0054] Figure 2 and Figure 3 middle, For the error quaternion label, The first element of the error quaternion is the imaginary number. The second element of the imaginary quaternion is the error quaternion. The third element of the quaternion is the imaginary number of the error quaternion. The x-axis error angular velocity, The y-axis error angular velocity, The z-axis error angular velocity.

[0055] The process of completing one round of training is as follows: the agent first interacts with the attitude dynamics simulation for a certain number of steps, and records all process data in the data playback pool. It is a function used to guide the agent to perform correct actions and control its posture in the correct direction. It has different forms depending on the current training stage.

[0056] The Proximal Policy Optimization (PPO) algorithm is used to train both the policy network and the value network simultaneously. The policy network is updated by solving an optimization problem.

[0057] The method for calculating the advantage function based on the state prediction value output of the value network is as follows:

[0058] ,

[0059] In the formula Let t be the timing error difference component at step t. For the single-step reward at step t, As a value discount factor, Value for state prediction These are the learnable parameters of the value network. For the policy of the policy network, These are the learnable parameters of the policy network. For learnable parameters strategy, For policy-based and learnable parameters State prediction value Let t be the state input quantity at step t. Let be the dominance function at step t. This is the timing error discount factor. and Hyperparameters that determine the importance of future states.

[0060] The method to maximize the training objective constructed from the advantage function is as follows:

[0061] ,

[0062] In the formula As expected, State input and state prediction value Expectation under the given conditions State input Next strategy Output flywheel motion Expectation under the given conditions for A set; Learnable parameters Updated to The subsequent strategy For strategy The advantage function under the given conditions This is a numerical truncation function. Hyperparameters that determine the cutoff range;

[0063] set up , , ,

[0064] get: .Bundle Limited to Within the range.

[0065] Furthermore, the value network learnable parameters through gradient descent. Update:

[0066] ,

[0067] In the formula Learnable parameters gradient, Indicates loss, Let be the state input at step t.

[0068] Before training a neural network, making certain improvements to the PPO algorithm can effectively improve data utilization efficiency, enhance convergence, prevent gradient explosion caused by backpropagation, and avoid catastrophic forgetting during training. If no improvements are made to the algorithm and training is performed directly, the neural network converges very slowly and is prone to getting stuck in suboptimal solutions.

[0069] During training, the single-step reward is calculated after each time the spacecraft executes the desired output torque. All process data is stored in a data replay pool to accumulate training samples; then batch data is extracted from the data replay pool for the learnable parameters of the policy network. and the learnable parameters of the value network Update;

[0070] Single-step reward The numerical values ​​vary significantly under different task states. If directly used to generate the loss function, the algorithm cannot effectively represent the advantages and disadvantages of different poses and actions, resulting in poor training efficiency. Therefore, before starting network training, the single-step reward... Normalization is performed.

[0071] Single-step rewards in the data replay pool The normalization method is as follows:

[0072] ,

[0073] In the formula The average of all rewards. The total number of data steps. The variance of all rewards, To avoid using 0 as the minimum divisor, we can take... .

[0074] Current single-step reward Projecting the data onto a space with a mean of 0 and a variance of 1, we obtain the updated single-step reward. Then used in the advantage function at step t. Calculation;

[0075] Normalizing the advantage function values ​​generated by the normalized reward and value network output values ​​during training using the same scaling method can effectively remove outliers and further improve training efficiency.

[0076] The calculated advantage function at step t The normalization method is as follows:

[0077] ,

[0078] In the formula For the advantage function The mean, For the advantage function The variance;

[0079] The normalized dominance function at step t The computation used to maximize the training objective constructed from the advantage function.

[0080] After normalization, the multiple trajectories in the current replay pool are divided into fixed-length mini-batches while preserving the temporal order of data within each mini-batch. This is equivalent to cutting the original long trajectory into multiple short trajectories. The order of the mini-batches is randomly shuffled, and the strategy and value network are trained using one mini-batch at a time. Training is complete only after all mini-batches have been used. This improvement effectively avoids the overfitting problem caused by long-term training of neural networks on a single task and, to some extent, avoids the generation of suboptimal solutions.

[0081] As an example, the policy network and value network are trained in four stages, with each stage configured with a different reward function and task initial reset conditions:

[0082] The first phase of rewards references the existing reward structure but incorporates improvements to significantly enhance training efficiency.

[0083] Phase 1, Step t, Single-Step Reward The calculation method is as follows:

[0084] ,

[0085] In the formula The error quaternion label for step t, The imaginary number is the quaternion for error. As an additional reward, used to indicate that the attitude maneuvering task is about to be completed, it guides the agent to remain at a certain position. Near the position close to 1;

[0086] In the first training phase, no angular velocity penalty or flywheel torque penalty is introduced into the reward function; otherwise, the network would hardly learn any policy, and the actions would diverge randomly. (First phase training) The initial policy of the policy network is obtained through 1000 rounds. ;

[0087] Additional rewards in the first phase This is a static value; regardless of the accuracy of the satellite's attitude maneuvers, it cannot further guide the agent to reduce the terminal steady-state error. When training enters the second phase, the terminal reward is used to enhance the task endpoint guidance, as shown in the following formula, while keeping the other parts of the reward the same as in the first phase.

[0088] Phase 2, Step t, Single-Step Reward Additional rewards will be added to the calculation. The calculation method is replaced with:

[0089] ;

[0090] The second phase involves 1000 rounds of training;

[0091] In the first two stages, in order to simplify the task and reduce the difficulty of training the policy network, the attitude maneuvering tasks performed by the agent in interaction with the environment all start from a fixed attitude starting point and a fixed flywheel angular momentum.

[0092] The initial value of the error quaternion in the first two stages Initial value of error angular velocity and initial values ​​of angular momentum vector Set to:

[0093] ;

[0094] This method causes the agent to fail to execute effective policies when the task starting point changes. The third stage of training randomly resets the starting point within a certain range to enhance the generalization ability of the policy network. After training for a certain number of steps, the agent can perfectly complete the attitude maneuvering task from arbitrary attitude to arbitrary attitude based on the attitude error quaternion within the complete task space of ±180° on each of the three axes of the initial attitude deviation.

[0095] The third stage focuses on training to enhance the generalization ability of the policy network, while the fourth stage employs a high-precision dynamic simulation program that closely approximates the timing trigger sequence of a real satellite control system.

[0096] The error quaternion in the third stage Error angular velocity The initial values ​​of the angular momentum vector are set as follows:

[0097] ,

[0098] In the formula The initial value of the first element of the error quaternion imaginary number is... The initial value of the second element of the imaginary quaternion is the error quaternion. The initial value of the third element of the error quaternion imaginary number. The initial value of the error quaternion is the standard part. The initial value of the error angular velocity along the j-axis. Let be the initial value of the angular momentum vector of the i-th flywheel. Indicates random uniform sampling;

[0099] Phase 3 Training step;

[0100] After the first three stages of training are completed, when the agent completes the posture control task, the posture will converge to... The accuracy could not be improved further because the additional bonus designed in the second stage was limited. At that time, it is basically impossible to generate any gain.

[0101] Phase 4 Additional Rewards The calculation method is replaced with:

[0102] ,

[0103] In the formula The error is indicated by the four elements.

[0104] In the first three training phases, the attitude control step size should be set to 0.2s or shorter, and all system noise and environmental torques should be ignored. This allows the agent to operate in a state that closely approximates an ideal continuous control system, training it to master the dynamic characteristics and basic strategies of the attitude control process. In the fourth phase, a dynamic simulation program with the highest possible precision, closely resembling the timing trigger sequence of a real satellite control system, should be used for training. Particular attention should be paid to using the same control step size and time delay as a real satellite, allowing the reinforcement learning algorithm to fine-tune the agent's strategy to achieve high-precision control under real system conditions.

[0105] Since the reward structure changes throughout the four training phases, the learning rate of the value network should be lower than that of the policy network to avoid overfitting to the state value at any given stage. Regardless of the training phase, the satellite flywheel will be set to never saturate, possessing infinite release angular momentum and always capable of outputting torque. Because the policy network's input does not include the current flywheel speed, if the flywheel stops outputting torque due to saturation during a maneuver, the agent cannot understand why the control action has ceased, leading to training failure to converge.

[0106] This implementation method employs a phased attitude control reinforcement learning training approach: first, the agent is trained with a fixed attitude maneuvering task to master basic flywheel control strategies; then, in subsequent phases, internal and external disturbances are gradually added, the initialization method is changed, and the agent is gradually guided to learn generalized high-performance strategies.

[0107] An improved reinforcement learning training reward system was adopted, based on the classic approach: by refining the reward parameters of existing successful studies, and according to... The novel phased end-point reward design for convergence accuracy can guide the agent to continuously improve attitude control accuracy, reduce the convergence steady-state error to within the range of attitude observation error, and achieve high-precision attitude control.

[0108] Application scenarios:

[0109] 1. High-Reliability Disturbance-Resistant Attitude Control for Microsatellites and Nanosatellites: In complex mission scenarios such as deep space exploration, ultra-low orbit remote sensing, and rendezvous and docking, microsatellites and nanosatellites are subject to various unknown environmental disturbances. Furthermore, due to limitations in size, weight, and cost, existing attitude and orbit control systems exhibit poor measurement and execution accuracy. The intelligent controller described in this invention can effectively resist internal and external disturbances, achieving high-precision and high-reliability attitude control.

[0110] 2. Spacecraft Attitude Control Fault Reconstruction: When a satellite flywheel experiences additive, multiplicative, or even complete failure, existing attitude and orbit control systems may be unable to detect the specific fault, causing the torque distribution of traditional attitude control algorithms to fail and increasing attitude control errors. The intelligent attitude controller of this invention has fuzzy decision-making capabilities. For a flywheel system with three upright and one inclined configuration, even if any flywheel fails completely, the attitude control task can still be completed without sacrificing control accuracy.

[0111] Verification experiment: Based on the actual mission requirements of the spacecraft, a high-precision dynamic environment simulation was conducted, considering all environmental disturbances and uncertainties as well as flywheel saturation, to verify the stability of the attitude control agent strategy.

[0112] Attitude redirection maneuver:

[0113] The spacecraft performed a large-angle attitude maneuver and reorientation, moving from the error quaternion [0.0185, 0.7070, -0.0062, 0.7070] to [1, 0, 0, 0] in 110 seconds. After attitude stabilization, the X and Y axis attitude errors were <0.025°, and the Z axis attitude error was <0.08°. Specific results are as follows... Figures 4 to 7 As shown. Figure 7 middle, Let x be the angular momentum of the x-th flywheel. Let y be the angular momentum of the y-th flywheel. Let z be the angular momentum of the z-th flywheel. Let be the angular momentum of the s-th flywheel.

[0114] Large-scale random target practice demonstrates that the agent employing the method of this invention possesses a highly generalizable strategy, capable of converging the error quaternion to [1,0,0,0] from any starting point, thereby achieving attitude maneuver control from any angle to any angle. Specific results are as follows: Figure 8 and Figure 9 As shown.

[0115] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A spacecraft attitude control method based on reinforcement learning, characterized by Comprising, a policy network is constructed based on two successive fully connected linear layers and two parallel branch linear layers; the policy network takes error quaternion and error angular velocity as state input, outputs beta distribution parameters of four flywheel actions of the spacecraft, and calculates four flywheel actions based on the beta distribution parameters; a value network is constructed based on two successive fully connected linear layers; the value network takes error quaternion and error angular velocity as state input, and outputs state prediction value of the current flywheel action; in the training process, the learnable parameters of the value network are adjusted to minimize the error between the state prediction value and the real state value; the advantage function is calculated based on the state prediction value output by the value network, and the learnable parameters of the policy network are adjusted to maximize the training target constructed by the advantage function; finally, the policy network outputs the current control quantity of the four flywheel actions according to the current error quaternion and the current error angular velocity.

2. The spacecraft attitude control method based on reinforcement learning according to claim 1, wherein, the two fully connected linear layers of the policy network are respectively configured with 150 neurons, and the outputs of the two fully connected linear layers are respectively subjected to hyperbolic tangent function for nonlinear activation; the two branch linear layers are respectively configured with 3 neurons, and the outputs of the two branch linear layers are respectively subjected to soft rectifier function for activation.

3. The spacecraft attitude control method based on reinforcement learning according to claim 2, wherein, The state input quantity is expressed as : ， wherein is the error quaternion, is the error angular velocity, denotes the transpose; ， ， wherein is the conjugate of the target quaternion is the conjugate of the target quaternion is the quaternion at time t.

4. The spacecraft attitude control method based on reinforcement learning according to claim 3, wherein, the calculation method of the four flywheel actions is: a flywheel action basic value is randomly sampled in the beta distribution parameters, and the flywheel action basic value is mapped to the flywheel output torque range to obtain the expected output torque of the flywheel action; ， wherein, is the action base value for the i-th flywheel, denotes a Beta distribution, is the success parameter of the Beta distribution for the i-th flywheel, is the failure parameter of the Beta distribution for the i-th flywheel, and both have a numerical range of [0, 1]; The i-th flywheel desired output torque is: ， In the formula is the maximum output torque of the i-th flywheel.

5. The spacecraft attitude control method based on reinforcement learning according to claim 4, wherein, the method for calculating the advantage function based on the state prediction value output by the value network is: ， wherein is a tth step timing error difference component, is a tth step one-step reward, is a value discount factor, is a state prediction value, are learnable parameters of the value network, is a policy of the policy network, are learnable parameters of the policy network, is a policy based on learnable parameters of the policy network, is a state prediction value based on policy and learnable parameters of the value network, is a tth step state input quantity, is a tth step advantage function, is a timing error discount factor.

6. The spacecraft attitude control method based on reinforcement learning according to claim 5, wherein, the method for maximizing the training target constructed by the advantage function is: ， where is the expected, is the state input and state prediction value is the expected, is the state input is the lower policy is the output flywheel action is the expected, is the set of is the set of is the learnable parameter is updated to is the policy after is the policy is the advantage function under is the numerical clipping function, is the hyperparameter that determines the clipping range; Set , , , Obtained: .

7. The spacecraft attitude control method based on reinforcement learning according to claim 6, wherein, The value network updates the learnable parameters by gradient descent ​ ， wherein is a learnable parameter is the gradient of represents the loss is the tth step state input 8. The spacecraft attitude control method based on reinforcement learning according to claim 7, wherein, During training, the spacecraft computes a one-step reward after each execution of the desired output torque ; All process data is stored into the data replay pool to accumulate training samples; then batch data is extracted from the data replay pool for updating the learnable parameters of the strategy network and the value network . Single-step rewards in data replay pools The normalization method for the single-step rewards is: ， where is the mean of all rewards, is the total number of data steps, is the variance of all rewards, is a small value to avoid division by zero; Projecting the current one-step reward into a space with mean 0 and variance 1 to get the updated one-step reward for use in the calculation of the t-step advantage function ; The normalization processing method of the obtained tth step advantage function is: ， wherein is the mean of the advantage function is the variance of the advantage function is the mean of the advantage function is the variance of the advantage function the normalized t-th step advantage function computations for maximizing the training objective constructed from the advantage function.

9. The spacecraft attitude control method based on reinforcement learning according to claim 8, wherein, the policy network and the value network are trained in four stages, and different reward functions and task initial reset conditions are configured in each stage: First stage step t reward The calculation method is: ， wherein is the tth step error quaternion, is the imaginary part of the error quaternion, is the additional reward; first stage training obtaining an initial policy of the policy network in a first stage training ; Second stage step t single step reward In the calculation of the additional reward The calculation method of the additional reward is replaced by ； the second stage is trained for 1000 rounds; initial values of the error quaternion initial values of the error angular velocity and the angular momentum vector are set to: ； the third stage is trained to enhance the generalization ability of the policy network, and the fourth stage is trained by using a high-precision dynamic simulation program close to the real satellite control system time sequence triggering sequence.

10. The spacecraft attitude control method based on reinforcement learning according to claim 9, wherein, error quaternion of the third stage error angular velocity and angular momentum vector initial value are set as: ， wherein is an initial value of the first element of the error quaternion imaginary number, is an initial value of the second element of the error quaternion imaginary number, is an initial value of the third element of the error quaternion imaginary number, is an initial value of the error quaternion real part, is an initial value of the j-axis of the error angular velocity, is an initial value of the i-th flywheel angular momentum vector, denotes a random uniform sampling; third stage training step Additional rewards for the fourth stage The calculation method is replaced by: ， In the formula is the error quaternion.

Images