A PID soft reinforcement learning spacecraft attitude control method and computer-readable medium

Abstract

The present invention proposes a PID soft reinforcement learning spacecraft attitude control method and computer-readable medium. The present invention constructs an improved SAC algorithm network, generates the control action of the spacecraft at the current moment through the improved SAC algorithm network, calculates the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method, and obtains the state data of the spacecraft attitude at the next moment; calculates the reward value at the current moment through an improved reward function model; and updates the policy network and value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment. The present invention shortens the learning time of the SAC algorithm, and even when the parameters change, the algorithm can still maintain good control performance. In practical applications, the spacecraft can achieve attitude stability in a relatively short time.

Description

Technical Field

[0001] The present invention belongs to the technical field of spacecraft attitude control, and in particular relates to a PID soft reinforcement learning spacecraft attitude control method and a computer-readable medium. Background Art

[0002] Spacecraft attitude maneuver control is of great significance in fields such as space exploration, Earth observation, and communications. The attitude control system needs to be able to accurately and quickly adjust the attitude of the spacecraft to ensure the smooth progress of the mission. In past studies, methods based on proportional-integral-derivative (PID) controllers have been widely used in spacecraft attitude control. However, the performance of the PID controller is limited by the system parameter settings, and these parameters are often difficult to obtain accurately in practical applications. The soft reinforcement learning (SAC) algorithm is a model-based adaptive control method that can be used to solve problems in continuous action space. However, directly applying the SAC algorithm to spacecraft attitude control may result in poor control performance because the SAC algorithm usually requires a long learning process. Summary of the Invention

[0003] In order to solve the problems of poor performance and long training time that may be encountered when directly applying the SAC algorithm, the present invention proposes a PID soft reinforcement learning spacecraft attitude control method and a computer-readable medium.

[0004] The technical solution of the method of the present invention is a PID soft reinforcement learning spacecraft attitude control method, comprising the following steps:

[0005] Step 1: Construct an improved SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment.

[0006] Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model;

[0007] Step 3: The target network updates the policy network and value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment;

[0008] Preferably, the improved SAC algorithm network in step 1 includes: a policy network, an action generation module, a value network, and a target network;

[0009] The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows:

[0010] a t,1

[0011] t∈[1,K]

[0012] Among them, a t,1 represents the strategic control action of the spacecraft at time t, and K represents the number of time moments;

[0013] The state data of the spacecraft attitude at the current moment includes:

[0014] The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment;

[0015] The action generation module generates the PID control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, a t,2 The specific formula is as follows:

[0016]

[0017] Among them, K p , K i , K d Respectively represent the proportional gain of the PID controller, the integral gain of the PID controller and the differential gain of the PID controller, q e represents the error quaternion vector, represents the derivative vector of the error quaternion vector, and ω represents the angular velocity vector.

[0018] The value network calculates the state data of the spacecraft attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the current moment;

[0019] The value network calculates the state data of the spacecraft attitude at the current moment and the PID control action of the spacecraft at the current moment to obtain the value of the PID control action of the spacecraft at the current moment;

[0020] The value network selects the control action corresponding to the higher value between the value of the strategy control action of the spacecraft at the previous moment and the value of the PID control action of the spacecraft at the previous moment as the control action of the spacecraft at the current moment;

[0021] The target network is used to update the strategy network and the value network;

[0022] Preferably, the state data of the spacecraft attitude at the current moment is obtained in step 2 by calculating the state data of the spacecraft attitude at the previous moment according to the Runge-Kutta method.

[0023] The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows:

[0024]

[0025] Among them, K a represents the action coefficient, a represents the control action, ||·||2 represents the 2-norm, q e,new Represents the error quaternion vector at the current moment, θ a represents the angle threshold, q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, and ELSE means except the above conditions.

[0026] The present invention also provides a computer-readable medium, which stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the steps of the PID soft reinforcement learning spacecraft attitude control method are executed.

[0027] Compared with the prior art, the present invention is beneficial in that:

[0028] This invention combines the stability of a PID controller with the adaptability of a SAC algorithm. By introducing a PID controller as a guiding strategy, it provides an initial control strategy for the SAC algorithm, thereby shortening the learning time of the SAC algorithm. Furthermore, this invention can perform online learning and adjustments in practical applications, making it more adaptable to complex tasks and uncertain environments.

[0029] By introducing a PID controller as a guiding strategy, the SAC algorithm can converge to the optimal strategy more quickly. This means that in practical applications, spacecraft can complete attitude maneuver control in a shorter time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 : A flow chart of a method according to an embodiment of the present invention;

[0031] Figure 2 : Comparison diagram of the intelligent agent training process of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0034] The following combination Figure 1-2 The specific embodiment of the present invention is a PID soft reinforcement learning spacecraft attitude control method, and the specific process is as follows:

[0035] Step 1: Construct an improved SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment.

[0036] The improved SAC algorithm network described in step 1 includes: a policy network, an action generation module, a value network, and a target network;

[0037] The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows:

[0038] a t,1

[0039] t∈[1,K]

[0040] Among them, a t,1 represents the strategic control action of the spacecraft at time t, and K represents the number of time moments;

[0041] The state data of the spacecraft attitude at the current moment includes:

[0042] The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment;

[0043] The action generation module generates the PID control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, a t,2 The specific formula is as follows:

[0044]

[0045] in, Represents the proportional gain of the PID controller, K i =0 represents the integral gain of the PID controller, represents the differential gain of the PID controller, q e represents the error quaternion vector, represents the derivative vector of the error quaternion vector, and ω represents the angular velocity vector.

[0046] The value network calculates the state data of the spacecraft attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the current moment;

[0047] The value network calculates the state data of the spacecraft attitude at the current moment and the PID control action of the spacecraft at the current moment to obtain the value of the PID control action of the spacecraft at the current moment;

[0048] The value network selects a control action corresponding to a higher value between the value of the strategy control action of the spacecraft at the current moment and the value of the PID control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment;

[0049] The target network is used to update the strategy network and the value network;

[0050] Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model;

[0051] The state data of the spacecraft attitude at the current moment is obtained in step 2 by calculating the state data of the spacecraft attitude at the previous moment using the Runge-Kutta method.

[0052] The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows:

[0053]

[0054] Among them, K a represents the action coefficient, which is 0.2 in this example; a represents the control action; ||·||2 represents the 2-norm calculation; q e,new Represents the error quaternion vector at the current moment; θ arepresents the angle threshold, which is 0.99962 in this example; q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, ELSE means except the above conditions;

[0055] Step 3: The target network updates the policy network and value network respectively according to the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment.

[0056] Figure 2 The simulation results verify the control method of the present invention. Under the same conditions, the training speed of this method is increased by nearly 150 episodes compared with the SAC control method.

[0057] And as shown in the following table, with the same initial conditions such as q0=[0,1,0,0] T ,ω=[0,0,0] T When the following three methods are simulated, the time comparison of converging to the target quaternion and target angular velocity is as follows:

[0058] Table 1: Convergence time comparison table

[0059] PD controller PID soft reinforcement learning control method SAC control method 10s 6s 7s

[0060] A specific embodiment of the present invention also provides a computer-readable medium.

[0061] The computer readable medium is a server workstation;

[0062] The server workstation stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the PID soft reinforcement learning spacecraft attitude control method of an embodiment of the present invention.

[0063] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0064] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A PID soft reinforcement learning spacecraft attitude control method, characterized in that: The following steps are involved: Step 1: Construct an improved SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment. Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model; Step 3: The target network updates the policy network and value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment; The step 2 of obtaining the state data of the spacecraft attitude at the current moment is obtained by calculating the state data of the spacecraft attitude at the previous moment using the Runge-Kutta method; The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows: Among them, K a represents the action coefficient, a represents the control action, ||·||2 represents the 2-norm, q e,new Represents the error quaternion vector at the current moment, θ a represents the angle threshold, q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, and ELSE means except the above conditions.

2. The PID soft reinforcement learning spacecraft attitude control method according to claim 1, characterized in that: The improved SAC algorithm network described in step 1 includes: a policy network, an action generation module, a value network, and a target network.

3. The PID soft reinforcement learning spacecraft attitude control method according to claim 2, characterized in that: The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows: a t,1 t∈[1,K] Among them, a t,1 represents the strategic control action of the spacecraft at time t, and K represents the number of time moments; The state data of the spacecraft attitude at the current moment includes: The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment.

4. The PID soft reinforcement learning spacecraft attitude control method according to claim 3, characterized in that: The action generation module generates the PID control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, a t,2 The specific formula is as follows: Among them, K p , K i , K d Respectively represent the proportional gain of the PID controller, the integral gain of the PID controller and the differential gain of the PID controller, q e represents the error quaternion vector, represents the derivative vector of the error quaternion vector, and ω represents the angular velocity vector.

5. The PID soft reinforcement learning spacecraft attitude control method according to claim 4, characterized in that: The value network calculates the state data of the spacecraft attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the current moment; The value network calculates the state data of the spacecraft attitude at the current moment and the PID control action of the spacecraft at the current moment to obtain the value of the PID control action of the spacecraft at the current moment; The value network selects the control action with the higher value between the value of the strategy control action of the spacecraft at the previous moment and the value of the PID control action of the spacecraft at the previous moment as the control action of the spacecraft at the current moment; the target network is used to update the strategy network and the value network.

Images