Micro-nano satellite attitude control method based on deterministic strategy gradient and PID

Abstract

The invention relates to the technical field of aerospace, and discloses a micro-nano satellite attitude control method based on deterministic strategy gradient and PID, and the method comprises the following steps: obtaining the pitching theta, the rolling psi and the yaw angle # imgabs0 # of a satellite through a sensitive sensor; the required control moment is converted into the attitude angle speed # imgabs 1 #, the angles theta t and psi t of the satellite in the current state are obtained according to an attitude kinematics model, and the satellite attitude quaternion error qe is determined according to the difference of the satellite attitude angles determined by a sensitive sensor through # imgabs 2 #. According to the micro-nano satellite attitude control method based on the deterministic strategy gradient and the PID, a better control strategy can be learned by adopting a deterministic strategy gradient algorithm, and by combining the deterministic strategy gradient algorithm with the traditional PID, more accurate adjustment can be carried out for the complex attitude change of the micro-nano satellite, and attitude deviation is reduced.

Description

Technical Field

[0001] The present invention relates to the field of aerospace technology, and in particular to a micro-nano satellite attitude control method based on deterministic policy gradient and PID. Background Art

[0002] With the development of modern aerospace technology, the tasks undertaken by satellites are becoming more and more diverse, and the performance requirements of satellite attitude control systems are becoming higher and higher. At present, there are more and more technical researches in the direction of satellite attitude control, especially the researches on artificial intelligence and satellite attitude control algorithms. At present, the control methods in the field of satellite attitude mainly adopt PID control, sliding mode control and other methods. For PID control, PID control has the advantages of simple structure, robustness to model errors and easy operation. Reinforcement learning is a kind of model-based and model-free. In the model-based RL, dynamic programming algorithm can be used to solve the value of each state and derive the strategy. The model-free RL finds the strategy with the highest benefit in the initial state through trial and error. Deterministic policy gradient It learns the optimal control strategy by establishing a dynamic policy network. The DDPG algorithm adopts a dual neural network architecture, in which one network is used to estimate the state value function and the other network is used to learn the control strategy. By continuously interacting with the environment, the DDPG algorithm can gradually approach the optimal control strategy, thereby optimizing the PID control parameters and realizing the attitude control of micro-nano satellites.

[0003] PID control is widely used in industrial process control. The difficulty of PID control is not programming implementation, but controller parameter setting. Controller parameters directly affect the performance of PID controller. Due to the complexity of the motion system of the controlled object satellite and the perturbation of the environment, traditional PID control needs to rely on manual experience to adjust the PID parameters, which generally needs to be done slowly according to the object. Summary of the invention

[0004] The purpose of the present invention is to provide a micro-nano satellite attitude control method based on deterministic policy gradient and PID, so as to solve the problem raised in the above background technology that due to the complexity of the motion system of the controlled object satellite and the perturbation of the environment, the traditional PID control needs to rely on artificial experience to adjust the PID parameters.

[0005] To achieve the above object, the present invention provides the following technical solution: a micro-nano satellite attitude control method based on deterministic policy gradient and PID, comprising the following steps:

[0006] Step 1: Obtain the satellite's pitch θ, roll ψ, and yaw angle through sensitive sensors

[0007] Step 2: Use the satellite attitude dynamics model to convert the required control torque into attitude angle and angular velocity

[0008] Step 3: According to the attitude kinematics model, obtain the angle θ of the satellite in the current state t , ψ t , The satellite attitude quaternion error q is determined by the difference in the satellite attitude angle determined by the sensitive sensor e , fed back into the satellite attitude control model;

[0009] Step 4: By adding a PID controller to the controller, the initial gain K is given p , K d ;

[0010] Step 5: Add a deep reinforcement learning model to the PID controller and adaptively tune the PID controller gain K through reinforcement learning p , K i , K d , to achieve optimized control of a certain attitude angle, so that it can adjust the attitude angle faster and more steadily than the initial gain.

[0011] By adopting the above technical solution, a dynamic attitude planning algorithm for micro-nano satellites is developed through the joint learning of deep reinforcement learning model and PID controller.

[0012] Preferably, the micro-nano satellite attitude is based on the formula Calculate the quaternion projection of the satellite's attitude in the stellar coordinate system b relative to the orbital coordinate system o

[0013] The above technical solution is used to solve the satellite in the stellar coordinate system b relative to the orbital coordinate system.

[0014] Preferably, the is the current state quaternion of the satellite attitude, and the quaternion Q of the target attitude angle t , calculate the error attitude quaternion Q e .

[0015] The above technical solution is adopted to provide corresponding labels for different posture data.

[0016] Preferably, the dynamic equation outputs the angular velocity ω of the satellite in the inertial coordinate system ib .

[0017] The above technical solution is used to output an inertial coordinate system for the dynamic equation.

[0018] Preferably, after matrix operation is performed on the posture matrix, the following formula is used:

[0019]

[0020] It is the satellite rotation angle of Euler axis and quaternion representation.

[0021] With the above technical solution, the formula is used to perform matrix operations on the satellite rotation angle attitude matrix of the Euler axis.

[0022] Preferably, the PID parameter K p , K i , K d Stored as action a, and the satellite's attitude error quaternion Q e Discount it to a constant according to the discount factor, and stipulate that Q e The smaller the reward coefficient, the higher the storage.

[0023] The above technical solution is used to calculate the data corresponding to the work plan of the deep reinforcement learning model.

[0024] Preferably, the deep reinforcement learning model in step 5 performs the following cyclic training:

[0025] (1) Select action a t =μ(s|θ μ );

[0026] (2) Interact with the environment to obtain the state and benefit S t+1 ,r t = env.step(a);

[0027] (3) Stored in memory R R = (S, a, r, S t+1 ,a t+1 );

[0028] (4) Sampling from memory R;

[0029] (5) Use the evaluation network to evaluate the movement;

[0030] (6) Minimize target loss Update the current Critic network Q ω (s,a);

[0031] (7) Using the formula:

[0032]

[0033] Calculate the sampled policy gradient to update the current Actor network μ θ (s);

[0034] (8) Update the target network.

[0035] By adopting the above technical solution, the deep reinforcement learning model is used to learn and simulate before satellite attitude control to reduce the error state of adjustment control.

[0036] Preferably, the valuation using the evaluation network adopts the following formula:

[0037] y i =r i +γQ ω -(s i+1 ,μ θ -(s i+1 ))

[0038] Valuing the movement.

[0039] The above technical solution is used to evaluate the two networks during use.

[0040] Preferably, the real-time sampling time interval of the micro-nano satellite attitude for the model training is Δt=0.01s.

[0041] The above technical solution is used to provide real-time sampling of micro-nano satellite attitudes for model training.

[0042] Preferably, the dynamic range of the PID parameters is [0,1], and the learning rate is 10 -3 , the discount factor is γ=0.9, episodes=200.

[0043] By adopting the above technical solution, a standard for cyclic training is provided for the deep reinforcement learning model by adjusting the PID parameters.

[0044] Compared with the prior art, the beneficial effects of the present invention are: the micro-nano satellite attitude control method based on deterministic policy gradient and PID:

[0045] Due to the complexity of the motion system of the controlled satellite and the perturbation of the environment, traditional PID control requires manual experience to adjust the PID parameters.

[0046] 1. The deterministic policy gradient algorithm can learn a better control strategy. Combined with the traditional PID, it can make more accurate adjustments to the complex attitude changes of micro-nano satellites, reduce attitude deviations, and improve the pointing accuracy of satellites in space. It can also adapt to different working environments and mission requirements in real time, and accurately control the attitude of micro-nano satellites in various situations to ensure the normal functioning of satellites.

[0047] After the deep reinforcement learning model is trained repeatedly, the satellite attitude adjustment is precisely controlled. The deterministic policy gradient algorithm can automatically adapt to the dynamic characteristics of the micro-nano satellite through continuous learning and optimization. There is no need to manually adjust the control parameters, which reduces the dependence on professionals and effectively reduces the energy consumed by the satellite to adjust the attitude, improving energy efficiency. This is of great significance for micro-nano satellites, which have limited energy.

[0048] 2. It combines mature PID control and advanced deterministic policy gradient algorithm, and innovates on the basis of existing technologies. PID control has a certain stability basis, and the introduction of deterministic policy gradient can further optimize the control performance, so that the micro-nano satellite can restore a stable attitude more quickly when subjected to external interference, such as space radiation, micro-meteorite impact, etc. When the satellite's mass distribution, inertial characteristics, etc. change, the control method can automatically adjust the strategy to maintain a good control effect, thereby improving the flexibility and adaptability of the satellite system, improving the reliability of the satellite, effectively reducing the oscillation of the system, making the satellite's attitude changes smoother, reducing the mechanical stress on the satellite's internal equipment, and extending the satellite's service life. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the micro-nano satellite attitude control process of the present invention;

[0050] Figure 2 This is a schematic diagram of the control flow of the PID adaptive parameter adjustment of the present invention;

[0051] Figure 3 This is a schematic diagram of the framework of the combination of deep learning and PID controller of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] See also Figure 1-Figure 3 ,The present invention provides a technical solution: a micro-nano satellite attitude control method based on deterministic policy gradient and PID;

[0054] Among them, the micro-nano satellite attitude control includes the following steps:

[0055] Step 1: Obtain the satellite's pitch θ, roll ψ, and yaw angle through sensitive sensors

[0056] Step 2: Use the satellite attitude dynamics model to convert the required control torque into attitude angle and angular velocity

[0057] Step 3: According to the attitude kinematics model, obtain the angle θ of the satellite in the current state t , ψ t , The satellite attitude quaternion error q is determined by the difference in the satellite attitude angle determined by the sensitive sensor e , fed back into the satellite attitude control model;

[0058] Step 4: By adding a PID controller to the controller, the initial gain K is given p , K d ;

[0059] Step 5: Add a deep reinforcement learning model to the PID controller and adaptively tune the PID controller gain K through reinforcement learning p , K i , K d , to achieve the optimal control of a certain attitude angle control, so that it can adjust the attitude angle faster and more steadily than the initial gain;

[0060] In conjunction with the accompanying drawings Figure 1 As shown in the figure, the PID controller works in conjunction with the deep reinforcement learning model, adding the closed-loop control of the DDPG algorithm based on the traditional PID.

[0061] The micro-nano satellite attitude is obtained based on real-time interactive data, and the satellite attitude dynamics equation is obtained based on the momentum theorem:

[0062]

[0063] Where T is the control torque of the satellite, H is the angular momentum of the satellite, ω is the angular velocity of the satellite, t is the time, and H = Iω, so ω = I -1 H;

[0064] Pitch θ, roll ψ, yaw angle according to satellite attitude rotation The angular velocity of x ,ω y ,ω z The data corresponds to the satellite attitude rotation equation:

[0065]

[0066] Among them, the Euler angle and quaternion conversion matrix are based on the formula:

[0067]

[0068] Calculate the quaternion Q of the target attitude angle t ;

[0069] According to the satellite attitude current state quaternion Quaternion Q of target attitude angle t , calculate the error attitude quaternion Q e ;

[0070]

[0071] The attitude of micro-nano satellite is calculated according to the formula Calculate the quaternion projection of the satellite's attitude in the stellar coordinate system b relative to the orbital coordinate system o

[0072] According to system requirements, design system control equations:

[0073]

[0074] The dynamic equation outputs the angular velocity ω of the satellite in the inertial coordinate system ib ;

[0075] After matrix operation on the attitude matrix, the following formula is used:

[0076]

[0077] It is the satellite rotation angle of Euler axis and quaternion representation;

[0078] According to the satellite attitude kinematic equation: We can get:

[0079]

[0080] According to the above, the posture conversion matrix is:

[0081]

[0082] Further simplified to:

[0083]

[0084] According to the formula in step 4 above, the Euler angle of the controlled micro-nano satellite target is solved as follows:

[0085]

[0086] The attitude angle value of the satellite in each step is obtained by the attitude sensitivity. Assuming that the satellite attitude angle measured at time is:

[0087]

[0088] Combining the attitude angle of the satellite's current step length in the controlled state mentioned above, the satellite attitude angle error matrix can be obtained by subtracting the two equations:

[0089]

[0090] From the PID controller model, the angle error PID control model is:

[0091]

[0092] Among them, K p , K i and K d are proportional gain, integral gain and differential gain respectively, e k is the difference between the current posture and the target posture, is the rate of change of attitude difference;

[0093] Starting from a certain state, according to its state transfer matrix,<S,A,Ρ,r,γ> where S is a set of finite states, A is a set of actions, γ is a discount factor, and r(s,a) is a reward function. The reward at this time depends on both state s and action a. When the reward function depends only on state s, it degenerates to r(s).

[0094] P(S'|s,a) is the state transition function, which represents the probability of reaching a state after executing an action in the state;

[0095] The above Markov reward G t The formula is:

[0096]

[0097] Where R t represents the reward obtained at time t,

[0098] The agent's strategy is a mapping from state to action. The deterministic strategy is defined as the strategy function:

[0099] a=π(s;θ)

[0100] State value function V π In the case of strategy π, the expected return when starting from the state is expressed as:

[0101] V π =E π [G t |S t =s]

[0102] Action value function setting Q π(S,a) represents the value of taking action a in state s under a strategy π, and the expected return when following strategy π thereafter, expressed as:

[0103] Q π (s,a)=E π [G t |S t =s,A t =a]

[0104] From the Bellman expectation equation:

[0105]

[0106] To express the relationship between state and action value function

[0107] When there is any state s k When π (s t ) is the largest, this strategy is the optimal strategy, recorded as:

[0108] V * (s k )=maxV π (s k )

[0109] The control signal is generated by the control strategy μ. Let a k =μ(s k ), so the optimal action value function that meets the control strategy can be expressed as:

[0110] μ(s)=argmax a Q(s,a)

[0111] At this time, the Bellman equation for the optimal action value is expressed as:

[0112] Q * (s k ,a k )=E[r(s k ,a k )+rQ * (s k+1 ,a k+1 )]

[0113] Initialize the Critic network Q with random network parameters ω and θ respectively ω (s,a) and Actor network μ θ (s),

[0114] According to the state value function, the differentiation is:

[0115]

[0116] Therefore, the deterministic policy gradient theorem is derived from the above formula:

[0117]

[0118] According to each step size, update θ;

[0119] With the establishment of two networks: Critic network Q ω (s,a) and Actor network μ θ (s);

[0120] The angular velocity of the three flywheels is distributed as ω=[ω1ω2ω3], and the satellite’s error attitude quaternion Q e Stored as a state variable S;

[0121] PID parameter K p , K i , K d Stored as action a, and the satellite's attitude error quaternion Q e Discount it to a constant according to the discount factor, and stipulate that Q e The smaller the reward coefficient, the higher the storage;

[0122] The error attitude quaternion Q after the next action e(t+1) Store as the next state S t+1 ;

[0123] The next PID parameter K p , K i , K d As an action t+1 store;

[0124] By repeating the above process starting from establishing two networks, until the PID parameter K p , K i , K d As an action t+1 The stored state action is recorded as R = (S, a, r, S t+1 ,a t+1 );

[0125] Set the maximum number of training times and input the initial satellite attitude S at time t0;

[0126] The deep reinforcement learning model performs the following training cycle:

[0127] (1) Select action a t =μ(s|θ μ );

[0128] (2) Interact with the environment to obtain the state and benefit S t+1 ,rt = env.step(a);

[0129] (3) Stored in memory R R = (S, a, r, S t+1 ,a t+1 );

[0130] (4) Sampling from memory R;

[0131] (5) Use the evaluation network to evaluate the movement;

[0132] (6) Minimize target loss Update the current Critic network Q ω (s,a);

[0133] (7) Using the formula:

[0134]

[0135] Calculate the sampled policy gradient to update the current Actor network μ θ (s);

[0136] (8) Update the target network.

[0137] The valuation using the evaluation network adopts the following formula:

[0138] y i =r i +γQ ω -(s i+1 ,μ θ -(s i+1 ))

[0139] Valuing the movement;

[0140] The real-time sampling time interval of the micro-nano satellite attitude for model training is Δt=0.01s, the dynamic range of PID parameters is [0,1], and the learning rate is 10 -3 , the discount factor is γ = 0.9, episodes = 200, save the intermediate training results, select the network parameters with the best return value, load the final network parameters, design the micro-nano satellite attitude control PID parameters of reinforcement learning control, and optimize the PID parameters.

[0141] Working principle: When using the micro-nano satellite attitude control method based on deterministic policy gradient and PID, firstly, according to the sampling of real-time satellite attitude data, the acquired data is put into the micro-nano satellite attitude control training model, and the data enters the Critic network Q in the algorithm. ω (s,a) and Actor network μ θ(s) training, the network parameters with the best return value are selected, the final network parameters are loaded, and a PID controller is added to the controller to give the initial gain K p , K d , add a deep reinforcement learning model to the PID controller, and adaptively adjust the PID controller gain K through reinforcement learning p , K i , K d , to achieve the optimal control of a certain attitude angle control, so that it can adjust the attitude angle faster and more steadily than the initial gain. The model training is designed to be the PID parameters of the micro-nano satellite attitude control controlled by reinforcement learning, and the PID parameters are optimized for the micro-nano satellite attitude control.

[0142] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A micro-nano satellite attitude control method based on deterministic policy gradient and PID, characterized in that: The following steps are involved: Step 1: Obtain the satellite's pitch θ, roll ψ, and yaw angle through sensitive sensors Step 2: Use the satellite attitude dynamics model to convert the required control torque into attitude angle and angular velocity Step 3: According to the attitude kinematics model, obtain the angle θ of the satellite in the current state t , ψ t , The satellite attitude quaternion error q is determined by the difference in the satellite attitude angle determined by the sensitive sensor e , fed back into the satellite attitude control model; Step 4: By adding a PID controller to the controller, the initial gain K is given p , K d ; Step 5: Add a deep reinforcement learning model to the PID controller and adaptively tune the PID controller gain K through reinforcement learning p , K i , K d , to achieve optimized control of a certain attitude angle, so that it can adjust the attitude angle faster and more steadily than the initial gain.

2. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: The micro-nano satellite attitude is based on the formula Calculate the quaternion projection of the satellite's attitude in the stellar coordinate system b relative to the orbital coordinate system o 3. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: Said is the current state quaternion of the satellite attitude, and the quaternion Q of the target attitude angle t , calculate the error attitude quaternion Q e .

4. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: The dynamic equation outputs the angular velocity ω of the satellite in the inertial coordinate system ib .

5. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: After matrix operations are performed on the posture matrix, the following formula is used: It is the Euler axis satellite rotation angle and quaternion representation.

6. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: The PID parameter K p , K i , K d Stored as action a, and the satellite's attitude error quaternion Q e Discount it to a constant according to the discount factor, and stipulate that Q e The smaller the reward coefficient, the higher the storage.

7. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 1, characterized in that: The deep reinforcement learning model in step 5 is trained in the following cycle: (1) Select action a t =μ(s|θ μ ); (2) Interact with the environment to obtain the state and benefit S t+1 ,r t = env.step(a); (3) Stored in memory R R = (S, a, r, S t+1 ,a t+1 ); (4) Sampling from memory R; (5) Use the evaluation network to evaluate the movement; (6) Minimize target loss Update the current Critic network Q ω (s,a); (7) Using the formula: Calculate the sampled policy gradient to update the current Actor network μ θ (s); (8) Update the target network.

8. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 7, characterized in that: The valuation using the evaluation network adopts the following formula: y i =r i +γQ ω -(s i+1 ,μ θ -(s i+1 )) Valuing the movement.

9. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 7, characterized in that: The real-time sampling time interval of the micro-nano satellite attitude for model training is Δt=0.01s.

10. The micro-nano satellite attitude control method based on deterministic policy gradient and PID according to claim 7, characterized in that: The dynamic range of the PID parameters is [0,1], and the learning rate is 10 -3 , the discount factor is γ=0.9, episodes=200.

Images