Spacecraft attitude control method based on reinforcement learning

Abstract

The application discloses a spacecraft attitude control method based on reinforcement learning and belongs to the technical field of spacecraft attitude control.The application aims at the problem that the existing attitude control method cannot perform fault reconstruction when unknown fault information exists, and the control precision is affected.The application comprises the following steps: constructing a strategy network based on two fully connected linear layers and two branch linear layers;constructing a value network based on two fully connected linear layers;taking error quaternions and error angular velocities as state input quantities;outputting beta distribution parameters of four flywheel actions of the spacecraft by the strategy network;calculating the four flywheel actions based on the beta distribution parameters;outputting state prediction values of the current flywheel actions by the value network;adjusting the learnable parameters of the value network by taking the error between the state prediction values and real values as a target;adjusting the learnable parameters of the strategy network by taking the training target constructed by an advantage function as a target;and the application realizes intelligent attitude control with in-situ fault reconstruction capability.

Description

TECHNICAL FIELD

[0001] The present application relates to a spacecraft attitude control method based on reinforcement learning, and belongs to the technical field of spacecraft attitude control. BACKGROUND

[0002] The existing spacecraft attitude control methods include proportional-integral-derivative (PID) control, observer-based sliding mode control, direct adaptive control, and reinforcement learning control based on value method.

[0003] The PID control principle has problems such as slow convergence speed, difficulty in constructing an integral element, and the need to match parameters according to prior spacecraft inertia by root locus method. In the modern complex spacecraft attitude control scenarios such as variable inertia and flexibility, the precision is poor, and divergence may occur. A flexible spacecraft fractional order finite time control method is designed by adding a fractional order element to the PID controller. The control stability is strictly proved by using a modified Lyapunov function. Simulation shows that the controller exhibits stronger anti-interference ability, but the mathematical structure of the controller is very complex, and engineering practice is difficult.

[0004] The dynamics characteristics such as flexible vibration, inertia variation, and environmental torque that do not conform to the rigid body attitude dynamics model are regarded as total disturbance. The disturbance is observed and compensated by means such as observers and parameter identification, which is the most common method for flexible variable inertia spacecraft attitude control. The linear or nonlinear extended state observer (ESO) is the most common disturbance observation method in recent years. Based on the design of a single-frame control moment gyroscope (SGCMG) and a flywheel system comprehensive control law in a pyramid layout, a sliding mode attitude control law is proposed. The control law observes the spacecraft flexible accessory disturbance by using an extended state observer, has a provable feature of finite time stability, and has been verified by a ground semi-physical simulation system. The disturbance observer (DOB) principle is intuitive and simple to design, and still has some applications in recent complex spacecraft attitude control research. To eliminate the influence of system parameter uncertainty and external torque, a finite time integral sliding mode disturbance observer is designed. The disturbance observer is used in combination with a time-varying sliding mode control to achieve high-precision attitude control of a flexible liquid-filled spacecraft. Further, the flexible vibration, liquid sloshing, and external disturbance are regarded as lumped disturbance, and a fuzzy terminal sliding mode control strategy using a disturbance observer is used to effectively suppress system jitter. Although the above methods achieve attitude control of complex spacecraft, they also have problems such as dependence on observer design for convergence, poor fault reconstruction ability, complex mathematical structure, and difficult program compilation. Moreover, sliding mode control is a variable structure control, and the chattering problem of control output after the system reaches the sliding mode surface cannot be fundamentally solved.

[0005] Another simpler idea for spacecraft attitude controller design is to use adaptive control with automatic adjustment of control gain. An auxiliary signal can be used to establish the relationship between the boundary conditions of the flexible spacecraft dynamic system and the internal state, and a direct adaptive method is used to achieve boundary fault-tolerant control based on this, which ensures the asymptotic stability of the attitude control. To solve the problem of flexible spacecraft attitude control with time-varying external disturbance, an adaptive law is used to estimate the upper bound of the disturbance and compensate it in the predetermined time sliding mode control law, which takes into account the dynamic characteristics of SGCMGs. Due to the existence of the adaptive law, the spacecraft launch controller parameter adjustment work according to the prior inertia is greatly reduced, and there is no need to design an observation section. However, the adaptive law has a disturbance allowed upper bound, and once the inertia changes, the environmental disturbance, and the system fault exceed the upper bound, the control efficiency will be severely reduced, and even the control will be divergent.

[0006] There have been attempts to use reinforcement learning methods to solve spacecraft attitude control problems: for example, using DDPG (deep deterministic policy gradient) and TD3 (double delay deep deterministic policy gradient) two value-based reinforcement learning methods to train neural networks, which have achieved the attitude control task of rigid spacecraft. The agent trained by these two methods can only obtain the action value according to the current spacecraft attitude and select the action with the highest value, which does not have an explicit action. The control end precision is poor, it lacks engineering realizability, and there may be generalization problems.

[0007] Existing attitude control algorithms rely on satellite prior inertia data for parameter adjustment, have poor resistance to environmental disturbances and control system errors, cannot perform fault reconstruction when fault information is unknown, and cannot adapt to high-dynamic target continuous tracking observation tasks with the risk of actuator failure. SUMMARY

[0008] In view of the problem that the existing attitude control method cannot perform fault reconstruction when fault information is unknown, affecting the control precision, the present application provides a spacecraft attitude control method based on reinforcement learning.

[0009] The spacecraft attitude control method based on reinforcement learning provided by the present application comprises,

[0010] A policy network is constructed based on two consecutive fully connected linear layers and two parallel branch linear layers; the policy network takes error quaternion and error angular velocity as state input, outputs beta distribution parameters of four spacecraft flywheel actions, and calculates four flywheel actions based on the beta distribution parameters;

[0011] A value network is constructed based on two consecutive fully connected linear layers; the value network takes error quaternion and error angular velocity as state input, and outputs the state prediction value of the current flywheel action;

[0012] In the training process, the learnable parameters of the value network are adjusted to minimize the error between the state predicted value and the state real value; the state predicted value output by the value network is used to calculate the advantage function, and the learnable parameters of the policy network are adjusted to maximize the training target constructed by the advantage function; finally, the policy network obtained according to the current error quaternion and the current error angular velocity outputs the current control amount of the four flywheel actions.

[0013] The method of the application uses a small-scale fully connected neural network as a posture control agent, trains the network through an improved proximal policy optimization algorithm, so that the network masters a general anti-disturbance posture tracking control strategy. The agent can control the flywheel system to achieve high-precision attitude maneuvering and attitude tracking control under the conditions of satellite inertia uncertainty, large environmental torque disturbance, large attitude observation error, inaccurate flywheel output torque and unknown flywheel failure, according to the error quaternion and the error angular velocity.

[0014] Highly convergent training method: through the PPO algorithm (proximal policy optimization algorithm) and the phased training scheme, the convergence of the reinforcement learning training is greatly improved, and only a few times of super parameter and random seed trial and error are needed to obtain the training convergent agent.

[0015] Intelligent posture control with in-situ fault reconstruction capability: the agent does not need fault information and does not need to change the program structure and agent parameters, so it can adapt to various flywheel faults and failures without losing control accuracy and completing the attitude control task. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the reinforcement learning training flowchart of the spacecraft attitude control method based on reinforcement learning according to the application;

[0017] Figure 2 is a schematic diagram of the policy network structure;

[0018] Figure 3 is a schematic diagram of the value network structure;

[0019] Figure 4 is a schematic diagram of the error quaternion in the spacecraft intelligent attitude maneuvering process;

[0020] Figure 5 is a schematic diagram of the error angular velocity in the spacecraft intelligent attitude maneuvering process;

[0021] Figure 6 is a schematic diagram of the expected output torque output by the agent in the spacecraft intelligent attitude maneuvering process;

[0022] Figure 7 is a schematic diagram of the spacecraft flywheel speed in the spacecraft intelligent attitude maneuvering process;

[0023] Figure 8 is a schematic diagram of a random targeting error quaternion result of intelligent attitude control of a spacecraft;

[0024] Figure 9 is a schematic diagram of a random targeting error angular velocity result of intelligent attitude control of a spacecraft. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0026] DETAILED DESCRIPTION Figures 1 to 3 As shown in the accompanying drawings, the present application provides a spacecraft attitude control method based on reinforcement learning, comprising,

[0027] A policy network is constructed based on two consecutive fully connected linear layers and two parallel branch linear layers; the policy network takes error quaternion and error angular velocity as state input quantity, and outputs beta distribution parameters of four flywheel actions of the spacecraft, and four flywheel actions are calculated based on the beta distribution parameters;

[0028] A value network is constructed based on two consecutive fully connected linear layers; the value network takes error quaternion and error angular velocity as state input quantity, and outputs state prediction value of the current flywheel action;

[0029] In the training process, the learnable parameters of the value network are adjusted to minimize the error between the state prediction value and the real state value; the advantage function is calculated based on the state prediction value output by the value network, and the learnable parameters of the policy network are adjusted to maximize the training target constructed by the advantage function; finally, the policy network outputs the current control quantity of the four flywheel actions according to the current error quaternion and the current error angular velocity.

[0030] The present embodiment proposes a complete intelligent attitude intelligent agent design, training and engineering deployment scheme. Firstly, the relationship between the error quaternion of the current attitude of the satellite and the target attitude, and the error quaternion and the satellite attitude dynamics equation is derived, and all the control system internal and external disturbances are modeled as the basis for reinforcement learning training. The structure of the attitude control policy network and the value network is designed, and a complete proximal policy optimization (PPO) algorithm reinforcement learning training process and training reward is constructed. Finally, the on-board engineering deployment program of the attitude control intelligent agent is designed to ensure that it can play the expected control effect in the actual task.

[0031] As an example, combined Figure 2 and Figure 3 As shown, the two fully connected linear layers of the policy network are each configured with 150 neurons, and the outputs of the two fully connected linear layers are nonlinearly activated using the hyperbolic tangent function.

[0032] Each of the two branch linear layers is configured with 3 neurons, and the outputs of the two branch linear layers are activated using flexible rectified functions.

[0033] This implementation selects quaternions—the attitude output format most suitable for star sensors, sun sensors, gyroscopes, and other sensors—to describe satellite attitude in the most convenient way for engineering practice. To achieve tasks such as attitude maneuvering and tracking control from any attitude to any other attitude, an error quaternion is proposed.

[0034] Furthermore, the state input is represented as :

[0035] ,

[0036] In the formula For error quaternions, The error angular velocity in the satellite's body coordinate system. Indicates transpose;

[0037] ,

[0038] ,

[0039] In the formula For target quaternions conjugate, Let be the quaternion at time t. Represents the multiplication of quaternions.

[0040] The attitude kinematics equations describe the relationship between the error quaternion and the angular velocity using the derivative of the quaternion.

[0041] The calculation method for the four flywheel motions is as follows:

[0042] The basic values ​​of flywheel motion are obtained by randomly sampling from the beta distribution parameters. The basic values ​​of flywheel motion are then mapped to the range of flywheel output torque to obtain the expected output torque of flywheel motion.

[0043]

[0044] in, This represents the base motion value for the i-th flywheel. Indicates the beta distribution, Let be the success parameters of the i-th flywheel beta distribution. Failure parameter of the i-th flywheel beta distribution, and The numerical range of both is [0, 1];

[0045] The i-th flywheel expected output torque is:

[0046]

[0047] In the formula is the maximum output torque of the i-th flywheel.

[0048] Reinforcement learning training process:

[0049] A two-layer fully connected neural network is trained by a reinforcement learning method, so that the neural network learns a strategy from a simulation of a pose maneuvering task, and manipulates the four flywheels of a satellite to complete tasks such as pose maneuvering reorientation and pose tracking control. The flow of reinforcement learning is shown in Figure 1 . The policy network directly controls the satellite, and the structure design is shown in Figure 2 .

[0050] Figure 2 According to the current state input , the next control step is decided, and the four sets of beta distribution parameters of the action are output. The expected output torque is obtained by sampling from the beta distribution determined by and , and is scaled and mapped into the range of flywheel output torque. The statistical distribution of the action a output by the policy network according to the state input is called the policy . The trainable parameters θ inside the policy network determine the specific form of this distribution, so the process of the policy neural network making an action can be written as:

[0051] .

[0052] This way can help the neural network better explore the action space and speed up the training. The value network is used to guide the policy network to learn in the right direction, and the structure design is shown in Figure 3 .

[0053] The value network decides the goodness of the current state according to the current state input of the satellite, and outputs the value of the current state. The learnable parameters ω inside the value network determine the size of the state value, and since the policy network and the value network are trained synchronously, when the policy network has a policy , the value network also outputs the state value under the current policy .

[0054] Figure 2 and​Figure 3 middle, For the error quaternion label, The first element of the error quaternion is the imaginary number. The second element of the imaginary quaternion is the error quaternion. The third element of the quaternion is the imaginary number of the error quaternion. The x-axis error angular velocity, The y-axis error angular velocity, The z-axis error angular velocity.

[0055] The process of completing one round of training is as follows: the agent first interacts with the attitude dynamics simulation for a certain number of steps, and records all process data in the data playback pool. It is a function used to guide the agent to perform correct actions and control its posture in the correct direction. It has different forms depending on the current training stage.

[0056] The Proximal Policy Optimization (PPO) algorithm is used to train both the policy network and the value network simultaneously. The policy network is updated by solving an optimization problem.

[0057] The method for calculating the advantage function based on the state prediction value output of the value network is as follows:

[0058] ,

[0059] In the formula Let t be the timing error difference component at step t. For the single-step reward at step t, As a value discount factor, Value for state prediction For the learnable parameters of the value network, For the policy of the policy network, These are the learnable parameters of the policy network. For learnable parameters strategy, For policy-based and learnable parameters State prediction value, Let t be the state input at step t. Let be the dominance function at step t. This is the timing error discount factor. and Hyperparameters that determine the importance of future states.

[0060] The method to maximize the training objective constructed from the advantage function is as follows:

[0061] ,

[0062] In the formula As expected, State input and state prediction value Expectation under the given conditions State input Next strategy Output flywheel motion Expectation under the given conditions for A set; Learnable parameters Updated to The subsequent strategy For strategy The advantage function under the given conditions This is a numerical truncation function. Hyperparameters that determine the cutoff range;

[0063] set up , , ,

[0064] get: .Bundle Limited to Within the range.

[0065] Furthermore, the value network learnable parameters through gradient descent. Update:

[0066] ,

[0067] In the formula Learnable parameters gradient, Indicates loss, Let be the state input at step t.

[0068] Before training a neural network, making certain improvements to the PPO algorithm can effectively improve data utilization efficiency, enhance convergence, prevent gradient explosion caused by backpropagation, and avoid catastrophic forgetting during training. If no improvements are made to the algorithm and training is performed directly, the neural network converges very slowly and is prone to getting stuck in suboptimal solutions.

[0069] During training, the single-step reward is calculated after each time the spacecraft executes the desired output torque. All process data is stored in a data replay pool to accumulate training samples; then batch data is extracted from the data replay pool for the learnable parameters of the policy network. and the learnable parameters of the value network Update;

[0070] Single-step reward The numerical difference is large in different task states. If it is directly used to generate a loss function, the algorithm cannot effectively represent the advantages and disadvantages of different poses and actions, and the training efficiency is poor. Therefore, before starting to train the network, the single-step reward is normalized.

[0071] The normalization method of the single-step reward in the data playback pool is:

[0072] ,

[0073] In the formula, is the mean of all rewards, is the total number of steps, is the variance of all rewards, is a small value to avoid division by zero; it can be taken .

[0074] The current single-step reward is projected into a space with a mean of 0 and a variance of 1 to obtain the updated single-step reward , which is used to calculate the t-step advantage function ;

[0075] The normalized rewards and the advantage function values generated by the value network output during training are all normalized in the same scaling manner, which can effectively remove outliers and further improve the training efficiency.

[0076] The normalization method of the t-step advantage function obtained by calculation is:

[0077] ,

[0078] In the formula, is the mean of the advantage function , and is the variance of the advantage function .

[0079] The normalized t-step advantage function is used to calculate the training target constructed by the advantage function.

[0080] After the normalization process, the multiple trajectories in the current playback pool are divided into mini-batches with fixed length, and the time sequence relationship of the data in each mini-batch is maintained, which is equivalent to cutting the original long trajectory into multiple short trajectories. The order relationship between the mini-batches is randomly disturbed, and a mini-batch is used to train the strategy and value networks each time. The training is completed after all the mini-batches are used. This improvement effectively avoids the overfitting problem caused by long-term training of neural networks on a single task, and to some extent, avoids the generation of suboptimal solutions.

[0081] As an example, the strategy network and the value network are trained in four stages, and different reward functions and task initial reset conditions are configured in each stage.

[0082] The first stage reward refers to the existing reward structure and is improved to greatly improve the training efficiency.

[0083] The first stage single-step reward at step t is calculated as:

[0084] ,

[0085] In the formula, is the real part of the error quaternion at step t, is the imaginary part of the error quaternion, is an additional reward, which represents that the attitude maneuver task is about to be completed, and guides the agent to stay near the attitude close to 1;

[0086] In the first training stage, no angular velocity penalty and flywheel torque penalty are introduced into the reward function, otherwise the network cannot learn any strategy, and the action will randomly diverge. The initial strategy of the strategy network is obtained by training for 1000 rounds in the first stage ;

[0087] The additional reward in the first stage is a static value, which cannot further guide the agent to reduce the end static error regardless of the accuracy of the satellite attitude maneuver. When the training enters the second stage, the end reward is enhanced to guide the task endpoint, while keeping the other parts of the reward the same as in the first stage.

[0088] The second stage single-step reward at step t is calculated as: The calculation method of the additional reward

[0089] ;

[0090] ​​The second stage is trained for 1000 rounds;

[0091] In the first two stages, to simplify the task and reduce the difficulty of training the policy network, the attitude maneuver task performed by the agent interacting with the environment starts from a fixed attitude starting point and a fixed angular momentum of flywheel.

[0092] In the first two stages, the initial values of the error quaternion , the initial value of the error angular velocity and the initial value of the angular momentum vector are set as:

[0093] ;

[0094] This method causes the agent to be unable to perform an effective strategy when the starting point of the task changes. The third stage of training randomly resets the starting point within a certain range to enhance the generalization ability of the policy network. After steps of training, the agent can perfectly complete the attitude maneuver task from any attitude to any attitude based on the attitude error quaternion within the complete task space range of the initial attitude deviation of three axes ±180°.

[0095] The third stage is trained to enhance the generalization ability of the policy network, and the fourth stage is trained using a high-precision dynamics simulation program that is close to the time-triggered sequence of the real satellite control system.

[0096] The initial values of the error quaternion , the error angular velocity and the angular momentum vector in the third stage are set as:

[0097] ,

[0098] wherein is the initial value of the first imaginary element of the error quaternion, is the initial value of the second imaginary element of the error quaternion, is the initial value of the third imaginary element of the error quaternion, is the initial value of the real part of the error quaternion, is the initial value of the error angular velocity j-axis, is the initial value of the i-th flywheel angular momentum vector, denotes random uniform sampling;

[0099] The third stage is trained for steps;

[0100] After the first three stages of training are completed, when the agent completes the attitude control task, the attitude will converge to , and the accuracy cannot be further improved, because the additional reward designed in the second stage cannot be further improved. The gain is basically unable to be generated.

[0101] The fourth stage additional reward The calculation method is replaced by:

[0102]

[0103] In the formula Is the error four-element vector.

[0104] In the training of the first three stages, the attitude control step should be set to 0.2s or shorter, and all system noise and environmental moments are ignored, so that the agent works in the state closest to the ideal continuous control system to train it to master the dynamics characteristics and basic strategies of the attitude control process. In the fourth stage, a high-precision dynamics simulation program close to the real satellite control system timing trigger sequence should be used for training, especially attention should be paid to using the same control step and time delay length as the real satellite, so that the reinforcement learning algorithm fine-tunes the agent strategy to achieve high-precision control in the real system.

[0105] Because the reward structure will change in the four training stages, the learning rate of the value network should be less than that of the strategy network to avoid overfitting the state value of a certain stage. Regardless of which stage the training is in, the satellite flywheel will be set to never be saturated, with unlimited releasable angular momentum, and can always output torque. Since the input of the strategy network does not include the current flywheel speed, if the flywheel stops outputting torque due to saturation in the maneuver task, the agent cannot understand why the control action disappears at this time. This situation will lead to the training unable to converge.

[0106] This embodiment adopts a phased attitude control reinforcement learning training method: first, train the agent with a fixed attitude maneuver task to enable it to master the basic flywheel manipulation strategy, and then gradually increase the internal and external disturbances in subsequent stages, change the initialization method, and gradually guide the agent to learn a general high-performance strategy.

[0107] An improved reinforcement learning training reward is adopted based on the classical scheme: by improving the reward parameters of the existing successful research, and according to The new type of phased end reward is designed to improve the convergence accuracy, which can guide the agent to continuously improve the attitude control accuracy, reduce the convergence static error to the attitude observation error range, and realize high-precision attitude control.

[0108] Application scenarios:

[0109] ​1. High-Reliability Disturbance-Resistant Attitude Control for Microsatellites and Nanosatellites: In complex mission scenarios such as deep space exploration, ultra-low orbit remote sensing, and rendezvous and docking, microsatellites and nanosatellites are subject to various unknown environmental disturbances. Furthermore, due to limitations in size, weight, and cost, existing attitude and orbit control systems exhibit poor measurement and execution accuracy. The intelligent controller described in this invention can effectively resist internal and external disturbances, achieving high-precision and high-reliability attitude control.

[0110] 2. Spacecraft Attitude Control Fault Reconstruction: When a satellite flywheel experiences additive, multiplicative, or even complete failure, existing attitude and orbit control systems may be unable to detect the specific fault, causing the torque distribution of traditional attitude control algorithms to fail and increasing attitude control errors. The intelligent attitude controller of this invention has fuzzy decision-making capabilities. For a flywheel system with three upright and one inclined configuration, even if any flywheel fails completely, the attitude control task can still be completed without sacrificing control accuracy.

[0111] Verification experiment: Based on the actual mission requirements of the spacecraft, a high-precision dynamic environment simulation was conducted, considering all environmental disturbances and uncertainties as well as flywheel saturation, to verify the stability of the attitude control agent strategy.

[0112] Attitude redirection maneuver:

[0113] The spacecraft performed a large-angle attitude maneuver and reorientation, moving from the error quaternion [0.0185, 0.7070, -0.0062, 0.7070] to [1, 0, 0, 0] in 110 seconds. After attitude stabilization, the X and Y axis attitude errors were <0.025°, and the Z axis attitude error was <0.08°. Specific results are as follows... Figures 4 to 7 As shown. Figure 7 middle, Let x be the angular momentum of the x-th flywheel. Let y be the angular momentum of the y-th flywheel. Let z be the angular momentum of the z-th flywheel. Let be the angular momentum of the s-th flywheel.

[0114] Large-scale random target practice demonstrates that the agent employing the method of this invention possesses a highly generalizable strategy, capable of converging the error quaternion to [1,0,0,0] from any starting point, thereby achieving attitude maneuver control from any angle to any angle. Specific results are as follows: Figure 8 and Figure 9 As shown.

[0115] While the application has been described with reference to particular embodiments thereof, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present application. It will be apparent to those skilled in the art that numerous modifications can be made within the scope of the present application as defined by the appended claims. It is intended that all such modification fall within the spirit and scope of the present application. It will be understood that the features described in connection with one embodiment can be used in connection with another embodiment.

Claims

1. A spacecraft attitude control method based on reinforcement learning, characterized by Comprising, a policy network is constructed based on two successive fully connected linear layers and two parallel branch linear layers; the policy network takes error quaternion and error angular velocity as state input, outputs beta distribution parameters of four flywheel actions of the spacecraft, and calculates four flywheel actions based on the beta distribution parameters; a value network is constructed based on two successive fully connected linear layers; the value network takes error quaternion and error angular velocity as state input, and outputs state prediction value of the current flywheel action; in the training process, the learnable parameters of the value network are adjusted to minimize the error between the state prediction value and the real state value; the advantage function is calculated based on the state prediction value output by the value network, and the learnable parameters of the policy network are adjusted to maximize the training target constructed by the advantage function; finally, the policy network outputs the current control quantity of the four flywheel actions according to the current error quaternion and the current error angular velocity.

2. The spacecraft attitude control method based on reinforcement learning according to claim 1, wherein, the two fully connected linear layers of the policy network are respectively configured with 150 neurons, and the outputs of the two fully connected linear layers are respectively subjected to hyperbolic tangent function for nonlinear activation; the two branch linear layers are respectively configured with 3 neurons, and the outputs of the two branch linear layers are respectively subjected to soft rectifier function for activation.

3. The spacecraft attitude control method based on reinforcement learning according to claim 2, wherein, The state input quantity is expressed as : ， wherein is the error quaternion, is the error angular velocity, denotes the transpose; ， ， wherein is the conjugate of the target quaternion is the conjugate of the target quaternion is the quaternion at time t.

4. The spacecraft attitude control method based on reinforcement learning according to claim 3, wherein, the calculation method of the four flywheel actions is: a flywheel action basic value is randomly sampled in the beta distribution parameters, and the flywheel action basic value is mapped to the flywheel output torque range to obtain the expected output torque of the flywheel action; ， wherein, is the action base value for the i-th flywheel, denotes a Beta distribution, is the success parameter of the Beta distribution for the i-th flywheel, is the failure parameter of the Beta distribution for the i-th flywheel, and both have a numerical range of [0, 1]; ith flywheel desired output torque is: ， In the formula is the maximum output torque of the i-th flywheel.

5. The spacecraft attitude control method based on reinforcement learning according to claim 4, wherein, the method for calculating the advantage function based on the state prediction value output by the value network is: ， wherein is a tth step timing error difference component, is a tth step one-step reward, is a value discount factor, is a state prediction value, are learnable parameters of the value network, is a policy of the policy network, are learnable parameters of the policy network, is a policy based on learnable parameters of the policy network, is a state prediction value based on policy and learnable parameters of the value network, is a tth step state input quantity, is a tth step advantage function, is a timing error discount factor.

6. The spacecraft attitude control method based on reinforcement learning according to claim 5, wherein, the method for maximizing the training target constructed by the advantage function is: ， where is the expected, is the state input and state prediction value is the expected, is the state input is the lower policy is the output flywheel action is the expected, is the set of is the set of is the learnable parameter is updated to is the policy after is the policy is the advantage function under is the numerical clipping function, is the hyperparameter that determines the clipping range; Set , , , Obtained: .

7. The spacecraft attitude control method based on reinforcement learning according to claim 6, wherein, The value network updates the learnable parameters by gradient descent ​ ， wherein is a learnable parameter is the gradient of represents the loss is the tth step state input 8. The spacecraft attitude control method based on reinforcement learning according to claim 7, wherein, During training, the spacecraft computes a one-step reward after each execution of the desired output torque ; All process data is stored into the data replay pool to accumulate training samples; then batch data is extracted from the data replay pool for updating the learnable parameters of the strategy network and the value network . Single-step rewards in data replay pools The normalization method for the single-step rewards is: ， where is the mean of all rewards, is the total number of data steps, is the variance of all rewards, is a small value to avoid division by zero; Projecting the current one-step reward into a space with mean 0 and variance 1 to get the updated one-step reward for use in the calculation of the t-step advantage function ; The normalization processing method of the obtained tth step advantage function is: ， wherein is the mean of the advantage function is the variance of the advantage function is the mean of the advantage function is the variance of the advantage function the normalized t-th step advantage function computations for maximizing the training objective constructed from the advantage function.

9. The spacecraft attitude control method based on reinforcement learning according to claim 8, wherein, the policy network and the value network are trained in four stages, and different reward functions and task initial reset conditions are configured in each stage: First stage step t reward The calculation method is: ， wherein is the tth step error quaternion, is the imaginary part of the error quaternion, is the additional reward; first stage training obtaining an initial policy of the policy network in a first stage training ; Second stage step t single step reward In the calculation of the additional reward The calculation method of the additional reward is replaced by ； the second stage is trained for 1000 rounds; initial values of the error quaternion initial values of the error angular velocity and the angular momentum vector are set to: ； the third stage is trained to enhance the generalization ability of the policy network, and the fourth stage is trained by using a high-precision dynamic simulation program close to the real satellite control system time sequence triggering sequence.

10. The spacecraft attitude control method based on reinforcement learning according to claim 9, wherein, error quaternion of the third stage error angular velocity and angular momentum vector initial value are set as: ， wherein is an initial value of the first element of the error quaternion imaginary number, is an initial value of the second element of the error quaternion imaginary number, is an initial value of the third element of the error quaternion imaginary number, is an initial value of the error quaternion real part, is an initial value of the j-axis of the error angular velocity, is an initial value of the i-th flywheel angular momentum vector, denotes a random uniform sampling; Third stage training Step; Additional rewards for the fourth stage The calculation method is replaced by: ， In the formula is the error quaternion.

Images