A continuous low-thrust satellite orbit control method based on two-layer SAC

Abstract

The present invention discloses a continuous low-thrust satellite orbit control method based on two-layer SAC, which relates to the field of satellite orbit control technology. The present invention connects the SAC-1 network model and the SAC-2 network model in a cascade manner during the satellite orbit change process, and takes the vicinity of the target orbit and the final target position as the satellite low-thrust orbit change target, respectively, effectively solving the problem of low-thrust orbit transfer and fixed-point integrated control, and making up for the deficiency of large orbit terminal state error caused by one orbit change; using the two direction angles α and β to represent the low thrust applied to the satellite, further improves the dynamic modeling of the satellite transfer process; for the same orbital elements, different weights are set in different cascade networks, and the SAC-2 network model uses the output of the SAC-1 network model as input, thereby improving the satellite orbit change accuracy; a satellite state potential function-assisted reward function is designed, which is conducive to improving the training efficiency of the satellite orbit change control strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite orbit control, and in particular to a continuous low-thrust satellite orbit control method based on a two-layer SAC. Background Art

[0002] All-electric propulsion satellites are one of the important development directions of the new generation of high-orbit satellites. The essence of electric propulsion satellite orbit control is continuous low-thrust satellite orbit control. The traditional chemical fuel pulse propulsion method is difficult to meet the increasingly complex needs of space missions due to its high impulse but high fuel consumption and limited control accuracy. The continuous low-thrust system, with its advantages of low thrust, high specific impulse and low fuel consumption, has become an important means to achieve high-precision orbit control and long-term residence. Under the condition of using existing technology, the maneuvering of satellites from low earth orbit to geostationary orbit usually involves two orbit control stages: orbit transfer and fixed point. How to carry out integrated control of orbit transfer and fixed point, so as to further reduce flight time and fuel consumption, is a technical problem that needs to be solved urgently in low-thrust orbit control.

[0003] Most of the existing continuous low-thrust satellite orbit control methods study the orbit control problems of each stage of orbit transfer and fixed-point separately, and the control objectives and constraints are different, which makes the low-thrust orbit control process cumbersome; or in the process of controlling the satellite maneuver, the inclination and altitude are changed at the same time to maneuver to the target orbit, and the target orbit is transferred to the target orbit at one time, which causes large errors, while the thrust yaw angle does not change much, making it difficult to achieve the accuracy of integrated control of low-thrust satellite orbit transfer and fixed-point. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a continuous low-thrust satellite orbit control method based on two-layer SAC, which solves the problem that the prior art only studies the orbit control methods of the orbit transfer and fixed-point stages separately, resulting in a cumbersome control process or simultaneously changing the inclination and altitude to maneuver to the target orbit, and transferring to the target orbit at one time, thereby causing a large error problem, making it difficult to achieve integrated control of the low-thrust satellite orbit transfer and fixed-point.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A continuous low-thrust satellite orbit control method based on a two-layer SAC is provided, which comprises the following steps:

[0007] S1. Determine the orbit transfer problem of continuous low-thrust satellites;

[0008] S2. Based on the track transfer problem, a corresponding Markov decision model is established;

[0009] S3, using the SAC-1 network model to process the Markov decision model in step S2, applying continuous maneuvers to the continuous low-thrust satellite and pushing it to the vicinity of the target orbit;

[0010] S4. Input the output of the SAC-1 network model into the SAC-2 network model, apply continuous maneuvers to the continuous low-thrust satellite, and push it to the target orbit.

[0011] Furthermore, the formula of the Markov decision model MDP in step S2 is:

[0012] MDP={S,A,P sa ,R,γ}

[0013] S={s 1 ,s 2 ,…,s m}

[0014] s q ={h q ,h xq ,h yq ,e xq ,e yq ,φ q}

[0015] A={a 1 ,a 2 ,…,a m}

[0016] a q ={α q ,β q}

[0017] R=φ(s t+1 )-φ(s t )-τ-5ξ+100d

[0018] Among them, S represents the state space, A represents the action space, and P sa represents the state transfer function, R represents the reward function, γ represents the discount factor, {s 1 ,s 2 ,…,s m}、s q Respectively represent the 1st to mth states and the qth state, {a 1 ,a 2 ,…,a m}、a q Respectively represent the 1st to mth actions, the i-th action, α q , β qThey represent the angle between the projection of the thrust unit vector in the orbital plane and the radial direction in the qth state and action, and the angle between the thrust unit vector in the qth state and action and the orbital plane, respectively. q 、h xq 、h yq They represent the angular momentum of the continuous low-thrust satellite in the qth state and action and the components of the angular momentum in the two coordinate axes, e xq 、e yq They represent the components of the eccentricity vector in the xy-axis direction in the qth state and action, respectively, i represents the true anomaly of the continuous low-thrust satellite under the qth state and action, φ(s t ),φ(s t+1 ) represent the state s respectively t 、s t+1 The corresponding state potential function, τ, ξ, and d represent constants.

[0019] Furthermore, the SAC-1 network model and the SAC-2 network model are connected in a cascade manner, and both include a serially connected Actor strategy network and a Critic value network; wherein the Critic value network includes a Critic current value network and a Critic target value network.

[0020] Furthermore, the training process of the SAC-1 network model in step S3 includes the following steps:

[0021] S3-1. Initialize the network parameters θ and learning rate α of the Actor strategy network 1 、The network parameter φ of the Critic current value network 1 and φ 2 , Network parameters of Critic target value network and And the Critic value network learning rate α 2 ;

[0022] S3-2, set the time step n, and initialize the experience replay pool capacity B;

[0023] S3-3. Use the Actor strategy network to interact with the environment and obtain the state s corresponding to the current moment t 、Action a t ;

[0024] S3-4, based on the current state s t , through the continuous small thrust satellite to perform action a t , and get the corresponding reward R t ;

[0025] S3-5, determine whether the running angle of the continuous low-thrust satellite reaches the time step n; if so, proceed to step S3-6; otherwise, proceed to step S3-9;

[0026] S3-6, calculating the semi-major axis a, eccentricity e and orbital inclination i of the continuous low-thrust satellite at the current moment;

[0027] S3-7, based on the semi-major axis a, eccentricity e and orbital inclination i at the current moment, update the time step n;

[0028] S3-8, storing the state, action and reward as a sample sequence into the experience replay pool;

[0029] S3-9, repeating steps S3-3 to S3-8 until N groups of sample sequences are obtained;

[0030] S3-10, calculate the output of the Critic target value network;

[0031] S3-11. Calculate the loss value L corresponding to the Actor strategy network, Critic current value network, and Critic target value network θ ,

[0032] S3-12, based on the loss value L θ , Update the network parameters of the Actor strategy network and the Critic current value network;

[0033] S3-13, based on the updated Critic current value network, the network parameters of the Critic target value network are updated using soft update;

[0034] S3-14, repeat steps S3-1 to S3-13 until the preset number of training times is reached.

[0035] Furthermore, the calculation formula of the semi-major axis a is:

[0036]

[0037] The calculation formula of eccentricity e is:

[0038]

[0039] The calculation formula of orbital inclination i is:

[0040]

[0041] Among them, h t Represents the current state s t and action a tThe angular momentum of the continuous low-thrust satellite under the condition of xt 、e yt Represent the current state s t and action a t The component of the lower eccentricity vector in the xy-axis direction, h xt 、h yt Represent the current state s t and action a t The components of the angular momentum in the two coordinate axes.

[0042] Further, step S3-7 includes the following steps:

[0043] S3-7-1, determine whether the semi-major axis a, eccentricity e and orbital inclination i at the current moment meet the first judgment criterion; if so, set the time step n to 0.1 and proceed to step 3-8; otherwise, proceed to step S3-7-2; wherein the first judgment criterion is:

[0044]

[0045] S3-7-2, determine whether the semi-major axis a, eccentricity e and orbital inclination i at the current moment meet the second judgment criterion; if so, set the time step n to 1; otherwise, set the time step n to 10; proceed to step S3-8; wherein the second judgment criterion is:

[0046] a∈{a tar ±2100km}

[0047] e≤0.01

[0048] i≤0.1

[0049] Among them, a tar Represents the minimum semi-major axis.

[0050] Furthermore, the formula of step S3-10 is:

[0051]

[0052] Among them, s' represents the state, a' θ represents the action, y(r,s',d) represents the output of the Critic target value network, min represents the minimum function, Represents state s' and action a' θ The action value function under θ |s) represents state s and action a θ The policy function under the condition, log(·) represents the logarithmic function with a constant as the base, α represents the control coefficient, and a′ θ~π(·|s') represents action a′ θ is the action taken by the policy function π in state s';

[0053] The formula for step S3-11 is:

[0054]

[0055] in, represents the loss value of the kth value network. When k is 1, the value network is the Critic current value network. When k is 2, the value network is the Critic target value network. ∑(·) represents the summation function, |·| represents the absolute value, and π θ (a θ |s) represents state s and action a θ The policy function is Represents the Critic value network in state s and action a θ is the total action-value function under Represents the action-value function of the k-th value network in state s and action a.

[0056] Furthermore, the weights of the semi-major axis a, eccentricity e and orbital inclination i of the SAC-2 network model in the reward function in step S4 are three times the weights of the semi-major axis a, eccentricity e and orbital inclination i of the SAC-1 network model in the reward function, and the rest of the process adopts the same method as step S3.

[0057] Furthermore, step S5 is also included, and the corresponding process is:

[0058] Based on the target orbit standard, it is determined whether the continuous low-thrust satellite has reached the target orbit. If it has not reached the target orbit, steps S3 and S4 are repeated; wherein the target orbit standard is:

[0059]

[0060] Among them, a tar 、a tol They represent the minimum semi-major axis and the maximum semi-major axis respectively, and e tol represents the maximum eccentricity, i tol Indicates the maximum orbital inclination.

[0061] The beneficial effects of the present invention are:

[0062] 1. The present invention connects the SAC-1 network model and the SAC-2 network model in a cascade manner during the satellite orbit change process, and takes the vicinity of the target orbit and the final target position as the satellite low-thrust orbit change targets respectively, which effectively solves the problem of low-thrust orbit transfer and fixed-point integrated control, and makes up for the deficiency of a large error in the orbit end state caused by a single orbit change;

[0063] 2. The present invention uses the two direction angles α and β to represent the small thrust applied to the satellite, further improving the dynamic modeling of the satellite transfer process;

[0064] 3. For the same orbital elements, the present invention sets different weights in different cascade networks. At the same time, the SAC-2 network model takes the output of the SAC-1 network model as input, thereby improving the accuracy of the satellite orbit change process. A reward function assisted by a satellite state potential function is designed, which is conducive to improving the training efficiency of the satellite orbit change control strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a method flow chart;

[0066] Figure 2 Schematic diagram of the SAC-1 training process;

[0067] Figure 3 This is the reward value map for SAC-1 training;

[0068] Figure 4 This is the reward value map for SAC-2 training;

[0069] Figure 5 This is the semi-major axis variation diagram during the satellite operation process;

[0070] Figure 6 This is the eccentricity variation diagram during the satellite operation;

[0071] Figure 7 This is the orbital inclination diagram of the satellite during its operation. DETAILED DESCRIPTION

[0072] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0073] like Figure 1 As shown, a continuous low-thrust satellite orbit control method based on two-layer SAC includes the following steps:

[0074] S1. Determine the orbit transfer problem of continuous low-thrust satellites;

[0075] Assuming the satellite uses solar electric propulsion, the thrust F of the propulsion can be expressed as:

[0076]

[0077] Where λ represents the effective value of thrust, P represents the power provided by the solar cell to the electric propulsion, g 0 is the gravitational acceleration on the Earth’s surface, I sp Indicates the specific impulse of the propulsion system. In the dynamic model of the satellite orbit coordinate system, except for the solar eclipse part of the orbit, the satellite thruster maintains a constant thrust value throughout the orbit transfer process. Because during the solar eclipse, the solar cell cannot provide energy, so the thrust value is set to 0.

[0078] S2. Based on the track transfer problem, a corresponding Markov decision model is established;

[0079] The formula of the Markov decision model MDP in step S2 is:

[0080] MDP={S,A,P sa ,R,γ}

[0081] S={s 1 ,s 2 ,…,s m}

[0082] s q ={h q ,h xq ,h yq ,e xq ,e yq ,φ q}

[0083] A={a 1 ,a 2 ,…,a m}

[0084] a q ={α q ,β q}

[0085] R=φ(s t +1)-φ(s t )-τ-5ξ+100d

[0086] Among them, S represents the state space, A represents the action space, and P sa represents the state transfer function, R represents the reward function, γ represents the discount factor, {s 1 ,s 2 ,…,s m}、s q Respectively represent the 1st to mth states and the qth state, {a 1 ,a 2 ,…,a m}、a qRespectively represent the 1st to mth actions, the i-th action, α q , β q They represent the angle between the projection of the thrust unit vector in the orbital plane and the radial direction in the qth state and action, and the angle between the thrust unit vector in the qth state and action and the orbital plane, respectively. q 、h xq 、h yq They represent the angular momentum of the continuous low-thrust satellite in the qth state and action and the components of the angular momentum in the two coordinate axes, e xq 、e yq They represent the components of the eccentricity vector in the xy-axis direction in the qth state and action, respectively, i represents the true anomaly of the continuous low-thrust satellite under the qth state and action, φ(s t ),φ(s t+1 ) represent the state s respectively t 、s t+1 The corresponding state potential function, τ, ξ, and d represent constants.

[0087] In order to achieve the transfer of the satellite to the target orbit in the shortest time and avoid the satellite exceeding the boundary conditions in the process, the reward function setting is the key to the entire reinforcement learning. The most important thing in the satellite orbit transfer process is to determine the gap between the satellite's real-time orbit information and the target orbit information. Therefore, this part of the reward function measures the impact of the transfer from the current state to the next state on reaching the target state by calculating the state potential function φ(s')-φ(s). The corresponding first reward function reward 1 The formula is:

[0088]

[0089] reward 1 =φ(s t+1 )-φ(s t )

[0090] Among them, w 1 、w 2 、w 3 represents the weight parameter, s tar It indicates that the orbit is in the state of minimum boundary, and e represents a natural constant.

[0091] The satellite needs to consider the time issue during the orbit transfer process. If the orbit transfer time lasts too long, the satellite needs to be penalized. The corresponding second reward function is reward 2 The formula is:

[0092] reward 2 =-τ

[0093] If the satellite transfer time is too long, the value of τ is 0.005.

[0094] When the satellite is in the later stage of orbit transfer, it is necessary to consider the boundary problem and judge whether the current state exceeds the boundary condition. The corresponding third reward function is 3 The formula is:

[0095] reward 3 =-5ξ

[0096] When the satellite crosses the boundary condition during the orbit transfer process, the value of ξ is 1, otherwise it is 0. In the process of reaching the vicinity of the target orbit using the SAC-1 network model, the boundary condition is the content of step S3-7; in the process of reaching the final target position using the SAC-2 network model, the boundary condition is:

[0097]

[0098] The satellite needs to consider the time issue during the orbit transfer process. If the orbit transfer time lasts too long, the satellite needs to be penalized. The corresponding fourth reward function is reward 4 The formula is:

[0099] reward 4 =100d

[0100] When the satellite successfully reaches the target orbit, the value of d is set to 1, otherwise it is set to zero.

[0101] The SAC-1 network model and the SAC-2 network model are connected in a cascade manner, and both include a serially connected Actor strategy network and a Critic value network; among them, the Critic value network includes a Critic current value network and a Critic target value network.

[0102] S3, using the SAC-1 network model to process the Markov decision model in step S2, applying continuous maneuvers to the continuous low-thrust satellite, and pushing it to the vicinity of the target orbit;

[0103] like Figure 2 As shown, the training process of the SAC-1 network model in step S3 includes the following steps:

[0104] S3-1. Initialize the network parameters θ and learning rate α of the Actor strategy network 1 、The network parameter φ of the Critic current value network 1 and φ 2 , Network parameters of Critic target value network and And the Critic value network learning rate α2 ;

[0105] S3-2, set the time step n, and initialize the experience replay pool capacity B;

[0106] The time step of the SAC algorithm is determined based on the true anomaly φ of the spacecraft. The value range of φ is 0 to 360°. n is recorded as a time step, and the corresponding unit is degree. The satellite takes action every n degrees. The satellite will gradually approach the target orbit in the later stage of the orbit transfer mission. If n decreases as the mission progresses, the satellite can take more frequent actions in the later stage, which is conducive to the precise control of the satellite, but it will also increase the mission time. In order to balance the optimality of accuracy and mission time, a variable n value strategy is designed, that is, the size of n is reduced as the orbit transfer mission progresses.

[0107] S3-3. Use the Actor strategy network to interact with the environment and obtain the state s corresponding to the current moment t 、Action a t ;

[0108] S3-4, based on the current state s t , through the continuous small thrust satellite to perform action a t , and get the corresponding reward R t ;

[0109] S3-5, determine whether the running angle of the continuous low-thrust satellite reaches the time step n; if so, proceed to step S3-6; otherwise, proceed to step S3-9;

[0110] S3-6, checking the semi-major axis a, eccentricity e and orbital inclination i at each time step to determine whether the satellite has reached the expected position and calculating the semi-major axis a, eccentricity e and orbital inclination i of the continuous low-thrust satellite at the current moment;

[0111] The calculation formula of the semi-major axis a is:

[0112]

[0113] The calculation formula of eccentricity e is:

[0114]

[0115] The calculation formula of orbital inclination i is:

[0116]

[0117] Among them, h t Represents the current state s t and action a t The angular momentum of the continuous low-thrust satellite under the condition ofxt 、e yt Represent the current state s t and action a t The component of the lower eccentricity vector in the xy-axis direction, h xt 、h yt Represent the current state s t and action a t The components of the angular momentum in the two coordinate axes.

[0118] S3-7, based on the semi-major axis a, eccentricity e and orbital inclination i at the current moment, update the time step n;

[0119] Step S3-7 includes the following steps:

[0120] S3-7-1, determine whether the semi-major axis a, eccentricity e and orbital inclination i at the current moment meet the first judgment criterion; if so, set the time step n to 0.1 and proceed to step 3-8; otherwise, proceed to step S3-7-2; wherein the first judgment criterion is:

[0121]

[0122] S3-7-2, determine whether the semi-major axis a, eccentricity e and orbital inclination i at the current moment meet the second judgment criterion; if so, set the time step n to 1; otherwise, set the time step n to 10; proceed to step S3-8; wherein the second judgment criterion is:

[0123] a∈{a tar ±2100km}

[0124] e≤0.01

[0125] i≤0.1

[0126] Among them, a tar Represents the minimum semi-major axis.

[0127] S3-8, storing the state, action and reward as a sample sequence into the experience replay pool;

[0128] S3-9, repeating steps S3-3 to S3-8 until N groups of sample sequences are obtained;

[0129] S3-10, calculate the output of the Critic target value network;

[0130] The formula for step S3-10 is:

[0131]

[0132] Among them, s' represents the state, a' θrepresents the action, y(r,s',d) represents the output of the Critic target value network, min represents the minimum function, Represents state s' and action a' θ The action value function under θ |s) represents state s and action a′ θ The policy function under the condition, log(·) represents the logarithmic function with a constant as the base, α represents the control coefficient, and a′ θ ~π(·|s') represents action a′ θ is the action taken by the policy function π in state s';

[0133] S3-11. Calculate the loss value L corresponding to the Actor strategy network, Critic current value network, and Critic target value network θ ,

[0134] The formula for step S3-11 is:

[0135]

[0136] in, represents the loss value of the kth value network. When k is 1, the value network is the Critic current value network. When k is 2, the value network is the Critic target value network. ∑(·) represents the summation function, |·| represents the absolute value, and π θ (a θ |s) represents state s and action a θ The policy function is Represents the Critic value network in state s and action a θ is the total action-value function under Represents the action-value function of the k-th value network in state s and action a.

[0137] S3-12, based on loss value L θ Update the network parameters of the Critic current value network and the Actor strategy network, which can be summarized as follows:

[0138]

[0139] Among them, α 1 , α 2 Represents the learning rate of the Actor strategy network and the Critic network, Indicates strategy.

[0140] S3-13, based on the updated Critic current value network, the network parameters of the Critic target value network are updated using soft update;

[0141] S3-14, repeat steps S3-1 to S3-13 until the preset number of training times is reached.

[0142] S4. Input the output of the SAC-1 network model into the SAC-2 network model, apply continuous maneuvers to the continuous low-thrust satellite, and push it to the target orbit.

[0143] The weights of the semi-major axis a, eccentricity e and orbital inclination i of the SAC-2 network model in the reward function in step S4 are three times the weights of the semi-major axis a, eccentricity e and orbital inclination i of the SAC-1 network model in the reward function, and the rest of the process adopts the same method as step S3.

[0144] It also includes step S5, and the corresponding process is:

[0145] Based on the target orbit standard, it is determined whether the continuous low-thrust satellite has reached the target orbit. If it has not reached the target orbit, steps S3 and S4 are repeated; wherein the target orbit standard is:

[0146]

[0147] Among them, a tar 、a tol They represent the minimum semi-major axis and the maximum semi-major axis respectively, and e tol represents the maximum eccentricity, i tol Indicates the maximum orbital inclination.

[0148] In one embodiment of the present invention, the method is used to perform experimental simulation of satellite operation. Figure 3 and Figure 4 As shown in the figure, in the process of using the SAC-1 network to propel the satellite to the vicinity of the target orbit, the reward value of the SAC-1 network converges to about 2700 at about 6400 steps, indicating that the satellite has reached the vicinity of the target orbit; in the process of using the SAC-2 network to propel the satellite from the vicinity of the target orbit to the final position target, the reward value of SAC-2 converges to about 2000 after 12000 steps, indicating that the satellite has reached the target orbit with a higher accuracy rate. Figure 5 , Figure 6 and Figure 7 As shown in the figure, the semi-major axis, eccentricity and orbit inclination change slightly. This is because SAC-2 uses the training results of SAC-1 as the initial parameters, which are close to the final target orbit value. The semi-major axis fluctuates within the allowable error range and finally converges to the target value. The orbit inclination converges to the allowable error range. From the above results, it can be seen that the proposed cascade SAC orbit control algorithm can effectively output the orbit transfer strategy to transfer the satellite to the target orbit and meet the high-precision requirements.

[0149] In summary, the present invention connects the SAC-1 network model and the SAC-2 network model in a cascade manner during the satellite orbit change process, and takes the vicinity of the target orbit and the final target position as the satellite low-thrust orbit change targets respectively, which effectively solves the problem of low-thrust orbit transfer and fixed-point integrated control, and makes up for the deficiency of large orbit terminal state error caused by one orbit change; the two direction angles α and β are used to represent the low thrust applied to the satellite, which further improves the dynamic modeling of the satellite transfer process; for the same orbital elements, different weights are set in different cascade networks, and the SAC-2 network model takes the output of the SAC-1 network model as input, which improves the accuracy of the satellite orbit change process; a reward function assisted by a satellite state potential function is designed, which is conducive to improving the training efficiency of the satellite orbit change control strategy.

Claims

1. A continuous low-thrust satellite orbit control method based on two-layer SAC, characterized by: The following steps are involved: S1. Determine the orbit transfer problem of continuous low-thrust satellites; S2. Based on the track transfer problem, a corresponding Markov decision model is established; S3, using the SAC-1 network model to process the Markov decision model in step S2, applying continuous maneuvers to the continuous low-thrust satellite, and pushing it to the vicinity of the target orbit; S4, input the output of the SAC-1 network model into the SAC-2 network model, apply continuous maneuvers to the continuous low-thrust satellite, and push it to the target orbit; The SAC-1 network model and the SAC-2 network model are connected in a cascade manner, and both include a serially connected Actor strategy network and a Critic value network; wherein the Critic value network includes a Critic current value network and a Critic target value network; The training process of the SAC-1 network model in step S3 includes the following steps: S3-1. Initialize the network parameters of the Actor strategy network , learning rate 、Network parameters of the Critic current value network and , Network parameters of Critic target value network and And the Critic value network learning rate ; S3-2. Set the time step , and initialize the experience replay pool capacity ; S3-3. Use the Actor strategy network to interact with the environment and obtain the state corresponding to the current moment ,action ; S3-4. Based on the current state , through the continuous small thrust satellite to perform actions , get the corresponding reward ; S3-5. Determine whether the running angle of the continuous low-thrust satellite reaches the time step ; If yes, go to step S3-6; otherwise, go to step S3-9; S3-6. Calculate the semi-major axis of the continuous low-thrust satellite at the current moment , eccentricity and orbital inclination ; S3-7. Semi-major axis based on the current moment , eccentricity and orbital inclination , update the time step ; S3-8, storing the state, action and reward as a sample sequence into the experience replay pool; S3-9, repeating steps S3-3 to S3-8 until N groups of sample sequences are obtained; S3-10, calculate the output of the Critic target value network; S3-11. Calculate the loss values ​​corresponding to the Actor strategy network, Critic current value network, and Critic target value network , , ; S3-12, based on loss value , , , update the network parameters of the Actor strategy network and the Critic current value network; S3-13, based on the updated Critic current value network, the network parameters of the Critic target value network are updated using soft update; S3-14, repeat steps S3-1 to S3-13 until the preset number of training times is reached.

2. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: The Markov decision model in step S2 The formula is: in, represents the state space, represents the action space, represents the state transition function, represents the reward function, represents the discount factor, , Respectively represent the 1st to the state, status, , Respectively represent the 1st to the Action, Actions, , Respectively represent The angle between the projection of the thrust unit vector in the orbital plane and the radial direction under the first state and action, The angle between the thrust unit vector and the orbital plane under each state and action, , , Respectively represent The magnitude of the angular momentum and the components of the angular momentum in the two coordinate axes of a continuous low-thrust satellite in a state and action. , Respectively represent The component of the eccentricity vector in the xy axis direction under each state and action, Indicates The true anomaly of a continuous low-thrust satellite under various states and actions, , Respectively indicate the status , The corresponding state potential function is, , , Represents a constant.

3. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: The semi-major axis The calculation formula is: The eccentricity The calculation formula is: The orbital inclination The calculation formula is: in, Indicates the current state and actions The angular momentum of the continuous small thrust satellite under represents the Earth's gravitational constant, , Respectively represent the current state and actions The component of the lower eccentricity vector in the xy-axis direction, , Respectively represent the current state and actions The components of the angular momentum in the two coordinate axes.

4. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: The step S3-7 comprises the following steps: S3-7-1. Determine the semi-major axis at the current moment , eccentricity and orbital inclination Whether the first judgment criterion is met; if so, the time step Set to 0.1 and go to step 3-8; otherwise go to step S3-7-2; wherein the first judgment criterion is: ； S3-7-2. Determine the semi-major axis at the current moment , eccentricity and orbital inclination Whether the second judgment criterion is met; if so, the time step Set to 1; otherwise, the time step Set to 10; proceed to step S3-8; wherein the second judgment criterion is: in, Represents the minimum semimajor axis.

5. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: The formula of step S3-10 is: in, Indicates the status, Indicates action, Represents the output of the Critic target value network, represents the minimum function, Indicates status and actions The action value function under Indicates status and actions The policy function is represents the logarithmic function with a constant base of 2, represents the control coefficient, Indicates action Is subject to the state The policy function under the actions performed; The formula of step S3-11 is: in, Indicates The loss value of the network is When it is 1, the value network is the Critic current value network. When it is 2, the value network is the Critic target value network; represents the summation function, represents the absolute value, Indicates status and actions The policy function is Indicates that the Critic value network is in state and actions is the total action-value function under Indicates The network is in state and actions is the action-value function under .

6. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: The semi-major axis of the SAC-2 network model in step S4 , eccentricity and orbital inclination The weight in the reward function is the semi-major axis of the SAC-1 network model , eccentricity and orbital inclination The weight in the reward function is tripled, and the rest of the process adopts the same method as step S3.

7. The continuous low-thrust satellite orbit control method based on two-layer SAC according to claim 1 is characterized in that: It also includes step S5, and the corresponding process is: Based on the target orbit standard, it is determined whether the continuous low-thrust satellite has reached the target orbit. If it has not reached the target orbit, steps S3 and S4 are repeated; wherein the target orbit standard is: in, , denote the minimum semi-major axis and the maximum semi-major axis respectively. represents the maximum eccentricity, Indicates the maximum orbital inclination.

Images