A TD3 soft reinforcement learning spacecraft attitude control method and computer-readable medium

Abstract

The present invention proposes a TD3 soft reinforcement learning spacecraft attitude control method and computer-readable medium. The present invention constructs an improved TD3-SAC algorithm network, generates the control action of the spacecraft at the current moment through the improved TD3-SAC algorithm network, calculates the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method, and obtains the state data of the spacecraft attitude at the next moment; calculates the reward value at the current moment through an improved reward function model; and the target network updates the policy network and the improved value network according to the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment. The present invention learns through a shared buffer, which saves computing resources while also achieving more efficient learning and faster convergence speed, thereby improving the performance of spacecraft attitude control.

Description

Technical Field

[0001] The present invention belongs to the technical field of spacecraft attitude control, and in particular relates to a TD3 soft reinforcement learning spacecraft attitude control method and a computer-readable medium. Background Art

[0002] Spacecraft attitude control is a key technology in aerospace engineering. It involves managing and adjusting a spacecraft's attitude during orbital maneuvers to meet predetermined attitude and angular velocity requirements. Spacecraft attitude control is crucial for achieving precise orbital maneuvers, performing effective communications, ensuring efficient energy utilization, and meeting other mission requirements. A spacecraft attitude control system primarily consists of three components: sensors, actuators, and control algorithms. Sensors monitor spacecraft attitude information, such as angle and angular velocity, in real time. Actuators execute control commands, such as those controlling reaction wheels, magnetometers, or thrusters. Control algorithms generate control commands based on the attitude information provided by sensors to achieve the desired attitude target. Traditional spacecraft attitude control methods primarily include proportional-integral-derivative (PID) controllers, fuzzy controllers, and adaptive controllers. These methods offer advantages in stability and robustness, but their performance can be limited when faced with complex environments and uncertainties. With the development of advanced technologies such as deep reinforcement learning (DRL), a growing number of researchers are exploring how to leverage these techniques to improve the performance of spacecraft attitude control.

[0003] DRL is a technology that combines deep learning and reinforcement learning, enabling end-to-end learning and decision-making in complex environments. In recent years, DRL has achieved remarkable success in many fields, particularly demonstrating its superiority in continuous control tasks. Applying DRL to spacecraft attitude control tasks can fully leverage its online learning capabilities, achieving more efficient and flexible performance. By introducing advanced DRL algorithms such as TD3 and SAC, spacecraft attitude control systems can achieve higher adaptability and performance in complex and dynamic environments.

[0004] Existing methods such as TD3 and SAC algorithms usually have problems such as poor training effect, slow training speed, poor stability, and low convergence accuracy when applied to spacecraft attitude control. Summary of the Invention

[0005] In order to solve the problems of poor performance and long training time that may be encountered when directly applying the SAC and TD3 algorithms, the present invention proposes a TD3 soft reinforcement learning spacecraft attitude control method and computer-readable medium.

[0006] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] A TD3 soft reinforcement learning spacecraft attitude control method includes the following steps:

[0008] Step 1: Construct an improved TD3-SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved TD3-SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved TD3-SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment.

[0009] Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model;

[0010] Step 3: The target network updates the policy network and the improved value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment;

[0011] Preferably, the improved TD3-SAC algorithm network in step 1 includes: a policy network, an action generation module, an improved value network, and a target network;

[0012] The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows:

[0013] a t,1

[0014] t∈[1,K]

[0015] Among them, a t,1 represents the strategic control action of the spacecraft at time t, and K represents the number of time moments;

[0016] The state data of the spacecraft attitude at the current moment includes:

[0017] The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment;

[0018] The action generation module is composed of a TD3 algorithm network, and generates the TD3 control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, which is defined as follows:

[0019] a t,2

[0020] t∈[1,K]

[0021] Among them, a t,2 represents the TD3 control action of the spacecraft at time t, and K represents the number of moments;

[0022] The improved value network calculates the state data of the spacecraft's attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the previous moment. The specific calculation process is as follows:

[0023]

[0024] in, represents the value network of SAC, s t Indicates the current state data, a t,1 Indicates the strategic control action of the spacecraft at the current moment.

[0025] The improved value network calculates the state data of the spacecraft attitude at the current moment and the TD3 control action of the spacecraft at the current moment to obtain the value of the TD3 control action of the spacecraft at the previous moment. The specific calculation process is as follows:

[0026]

[0027] in, represents the value network of SAC, s t Indicates the current state data, a t,2 Indicates the TD3 control action of the spacecraft at the current moment.

[0028] The value network selects the control action with the higher value between the value of the strategy control action of the spacecraft at the current moment and the value of the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment according to the ε-greedy strategy, and randomly selects the strategy control action of the spacecraft at the current moment or the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment with a probability of 1-ε;

[0029] The target network is used to update the strategy network and the improved value network;

[0030] Preferably, the state data of the spacecraft attitude at the current moment is obtained in step 2 by calculating the state data of the spacecraft attitude at the previous moment according to the Runge-Kutta method.

[0031] The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows:

[0032]

[0033] Among them, Ka represents the action coefficient, a represents the control action, ||·||2 represents the 2-norm calculation, q e,new Represents the error quaternion vector at the current moment, θ a represents the angle threshold, q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, ELSE means except the above conditions;

[0034] The present invention also provides a computer-readable medium, which stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the steps of the TD3 soft reinforcement learning spacecraft attitude control method are executed.

[0035] Compared with the prior art, the present invention is beneficial in that:

[0036] The present invention combines the TD3 and SAC algorithms, and uses the ε-greedy strategy to efficiently combine the advantages of TD3 and SAC, fully utilizing the exploration and utilization characteristics of the two algorithms, selecting the more optimal action, and then learning through a shared buffer. While saving computing resources, it can also achieve more efficient learning and faster convergence speed, thereby improving the spacecraft attitude control performance and shortening the spacecraft attitude control stabilization time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 : A flow chart of a method according to an embodiment of the present invention;

[0038] Figure 2 : Comparison diagram of the intelligent agent training process of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0041] The following combination Figure 1-2The specific embodiment of the present invention is a TD3 soft reinforcement learning spacecraft attitude control method, and the specific process is as follows:

[0042] Step 1: Construct an improved TD3-SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved TD3-SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved TD3-SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment.

[0043] The improved TD3-SAC algorithm network described in step 1 includes: a policy network, an action generation module, an improved value network, and a target network;

[0044] The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows:

[0045] a t,1

[0046] t∈[1,K]

[0047] Among them, a t,1 represents the strategic control action of the spacecraft at time t, and K represents the number of time moments;

[0048] The state data of the spacecraft attitude at the current moment includes:

[0049] The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment;

[0050] The action generation module is composed of a TD3 algorithm network, and generates the TD3 control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, which is defined as follows:

[0051] a t,2

[0052] t∈[1,K]

[0053] Among them, a t,2 represents the TD3 control action of the spacecraft at time t, and K represents the number of moments;

[0054] The improved value network calculates the state data of the spacecraft's attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the previous moment. The specific calculation process is as follows:

[0055]

[0056] in, represents the value network of SAC, s t Indicates the current state data, a t,1 Indicates the strategic control action of the spacecraft at the current moment.

[0057] The improved value network calculates the state data of the spacecraft attitude at the current moment and the TD3 control action of the spacecraft at the current moment to obtain the value of the TD3 control action of the spacecraft at the previous moment. The specific calculation process is as follows:

[0058]

[0059] in, represents the value network of SAC, s t Indicates the current state data, a t,2 Indicates the TD3 control action of the spacecraft at the current moment.

[0060] The value network selects the control action with the higher value between the value of the strategy control action of the spacecraft at the current moment and the value of the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment according to the ε-greedy strategy, and randomly selects the strategy control action of the spacecraft at the current moment or the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment with a probability of 1-ε;

[0061] The target network is used to update the strategy network and the improved value network;

[0062] Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model;

[0063] The state data of the spacecraft attitude at the current moment is obtained in step 2 by calculating the state data of the spacecraft attitude at the previous moment using the Runge-Kutta method.

[0064] The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows:

[0065]

[0066] Among them, K a represents the action coefficient, which is 0.2 in this example; a represents the control action; ||·||2 represents the 2-norm calculation; q e,new Represents the error quaternion vector at the current moment; θ arepresents the angle threshold, which is 0.99962 in this example; q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, ELSE means except the above conditions;

[0067] Step 3: The target network updates the policy network and the improved value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment;

[0068] Figure 2 The simulation results verify the control method of the present invention. Under the same conditions, the training speed of this method is nearly 100 episodes faster than that of the SAC control method, and the reward value stabilizes at around 280 after reaching convergence. Compared with the TD3 and SAC methods, the convergence accuracy is higher and more stable.

[0069] And as shown in the following table, with the same initial conditions such as q0=[0,1,0,0] T ,ω=[0,0,0] T When the following three methods are simulated, the time comparison of convergence to the target quaternion and target angular velocity is shown in Table 1:

[0070] Table 1: Convergence time comparison table

[0071] PD controller TD3 soft reinforcement learning control method SAC control method 10s 5s 7s

[0072] A specific embodiment of the present invention also provides a computer-readable medium.

[0073] The computer readable medium is a server workstation;

[0074] The server workstation stores a computer program executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the TD3 soft reinforcement learning spacecraft attitude control method of an embodiment of the present invention.

[0075] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0076] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A TD3 soft reinforcement learning spacecraft attitude control method, characterized in that: The following steps are involved: Step 1: Construct an improved TD3-SAC algorithm network to obtain the state data of the spacecraft attitude at the current moment. Input the state data of the spacecraft attitude at the current moment into the improved TD3-SAC algorithm network. Generate the control action of the spacecraft at the current moment through the improved TD3-SAC algorithm network. Calculate the state data of the spacecraft attitude at the current moment and the control action of the spacecraft at the current moment through the Runge-Kutta numerical analysis method to obtain the state data of the spacecraft attitude at the next moment. Step 2: Obtain the state data of the spacecraft's attitude at the current moment and the control action of the spacecraft at the current moment, and calculate the reward value at the current moment through the improved reward function model; Step 3: The target network updates the policy network and the improved value network based on the state data of the spacecraft attitude at the current moment, the control action of the spacecraft at the current moment, the state data of the spacecraft attitude at the next moment, and the reward value at the current moment; The improved TD3-SAC algorithm network described in step 1 includes: a policy network, an action generation module, an improved value network, and a target network; The strategy network generates a strategy control action for the spacecraft at the current moment based on the state data of the spacecraft attitude at the current moment. The strategy control action for the spacecraft at the current moment is defined as follows: a t,1 t∈[1,K] Among them, a t,1 represents the strategic control action of the spacecraft at the current moment, and K represents the number of moments; The state data of the spacecraft attitude at the current moment includes: The error data of the spacecraft attitude at the current moment, the derivative of the error data of the spacecraft attitude at the current moment, and the angular velocity of the spacecraft at the current moment; The action generation module is composed of a TD3 algorithm network, and generates the TD3 control action of the spacecraft at the current moment according to the state data of the spacecraft attitude at the current moment, which is defined as follows: a t,2 t∈[1,K] Among them, a t,2 represents the TD3 control action of the spacecraft at the current moment, and K represents the number of moments; The improved value network calculates the state data of the spacecraft's attitude at the current moment and the strategic control action of the spacecraft at the current moment to obtain the value of the strategic control action of the spacecraft at the previous moment. The specific calculation process is as follows: in, represents the value network of SAC, s t Indicates the current state data, a t,1 Indicates the strategic control action of the spacecraft at the current moment; The improved value network calculates the state data of the spacecraft attitude at the current moment and the TD3 control action of the spacecraft at the current moment to obtain the value of the TD3 control action of the spacecraft at the previous moment. The specific calculation process is as follows: in, represents the value network of SAC, s t Indicates the current state data, a t,2 Indicates the TD3 control action of the spacecraft at the current moment; The value network selects the control action with the higher value between the value of the strategy control action of the spacecraft at the current moment and the value of the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment according to the ε-greedy strategy, and randomly selects the strategy control action of the spacecraft at the current moment or the TD3 control action of the spacecraft at the current moment as the control action of the spacecraft at the current moment with a probability of 1-ε; The target network is used to update the strategy network and the improved value network; The step 2 of obtaining the state data of the spacecraft attitude at the current moment is obtained by calculating the state data of the spacecraft attitude at the previous moment using the Runge-Kutta method; The reward value at the current moment is calculated by the improved reward function model described in step 2, as follows: Among them, K a represents the action coefficient, a represents the control action, ||·||2 represents the 2-norm calculation, q e,new Represents the error quaternion vector at the current moment, θ a represents the angle threshold, q e Indicates the error quaternion of the previous moment, IF means if the condition is met, ELIF means if another condition is met, and ELSE means except the above conditions.

Images