Satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm

Abstract

This invention discloses a satellite orbit control method based on a continuous-time proximal policy optimization (PPO) reinforcement learning algorithm, comprising the following steps: Step 1, determining the satellite's actions (acceleration) and execution time based on the PPO algorithm; Step 2, achieving satellite orbit control based on the satellite's actions and execution time obtained in Step 1. This invention, by combining an advanced PPO algorithm with a continuous-time control framework, provides a high-precision, stable, and energy-efficient satellite orbit control solution, exhibiting significant technological advancement and broad application prospects, providing strong technical support for future space exploration and satellite operation.

Description

Technical Field

[0001] This invention relates to the field of satellite orbit control methods, specifically a satellite orbit control method based on a continuous-time near-end policy optimization reinforcement learning algorithm. Background Technology

[0002] In the field of satellite orbit control, precise decision-making and control are crucial for the effective execution of various tasks. Traditional control methods mainly rely on predetermined trajectories and pre-programmed commands, which often lack sufficient flexibility in handling unexpected dynamic changes or responding to emergencies. Furthermore, traditional methods are typically based on discrete-time decision models, meaning the decision interval is fixed. This can lead to an inability to respond promptly to rapid environmental changes in highly dynamic orbital environments, thus affecting the effectiveness and stability of control.

[0003] As satellite missions become increasingly complex, the demand for real-time and adaptive control strategies in orbital pursuit environments is growing. These missions require control systems to analyze the current state in real time, make dynamic control decisions, and optimize orbital operations to adapt to constantly changing environmental conditions. Against this backdrop, reinforcement learning, as an artificial intelligence approach that learns optimal strategies through interaction with the environment, has shown great potential. However, most current reinforcement learning methods are based on discrete-time steps, with a fixed time interval between decision-making and execution. This can easily lead to a mismatch between decision-making and execution in orbital control missions that require rapid and continuous decision-making, thereby reducing the effectiveness and safety of operations. Summary of the Invention

[0004] This invention provides a satellite orbit control method based on a continuous-time proximal policy optimization (PPO) reinforcement learning algorithm, aiming to achieve efficient, stable, and precise control of satellite orbits through advanced reinforcement learning techniques. This invention overcomes the limitations of traditional discrete-time control methods by utilizing a continuous-time frame and a PPO algorithm, demonstrating superior control performance in dynamic and complex orbital environments.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A satellite orbit control method based on continuous-time near-end policy optimization reinforcement learning algorithm includes the following steps:

[0007] Step 1: Optimize the reinforcement learning algorithm based on continuous-time proximal policy to determine the satellite's actions (acceleration) and the execution time of the actions. The process is as follows:

[0008] An intelligent agent for the satellite is established, defining its position state space and action space. The position state space includes the position state vector of the satellite agent in a continuous time frame, and the action space includes the actions taken by the satellite agent in a continuous time frame, i.e., accelerations. A reward function is also established for the satellite agent.

[0009] A policy network is constructed and trained using historical satellite trajectory data. The trained policy network then outputs recommended actions and their execution times to the satellite agent.

[0010] After receiving the recommended action and execution time output by the trained policy network, the satellite agent evaluates the received recommended action and execution time through a reward function, thereby determining the optimal action, i.e., acceleration, and optimal execution time, which are then used as the actions taken by the satellite, i.e., acceleration and execution time.

[0011] Step 2: Based on the actions taken by the satellite and the execution time obtained in Step 1, achieve satellite orbit control.

[0012] Furthermore, in step 1, the reward function of the satellite agent... Feedback to the satellite intelligent agent based on execution time The magnitude, cost, and state deviation of the acceleration 'a' during the execution of the action (acceleration 'a'), and the reward function. As shown in the following formula:

[0013] ,

[0014] in: Indicates that the satellite intelligent agent is based on execution time. After performing the action, i.e., acceleration 'a', the position state vector 's' and the target position state vector 's' Deviation between; and These are the cost coefficient and delay cost coefficient for the satellite intelligent agent to perform the action, i.e., acceleration 'a'. It is the magnitude of the action, i.e., the acceleration 'a', which reflects the cost of performing the acceleration. Indicates action, i.e., acceleration. Execution time.

[0015] Furthermore, in step 1, the policy network employs a deep neural network. Deep neural networks, serving as policy networks, consist of multiple stacked fully connected layers, with the output of each fully connected layer serving as the input to the next fully connected layer; among them, This represents the weights and bias parameters of the deep neural network, where s is the satellite's current position and state vector. It is the recommended action, or acceleration, output by the policy network. The recommended action output by the policy network is acceleration. Execution time.

[0016] Furthermore, deep neural networks The activation function for each fully connected layer is the ReLU function.

[0017] Furthermore, deep neural networks That is, during the training of the policy network, the position and state vectors and velocity vectors from historical satellite trajectory data are used as the deep neural network. The input is provided by a deep neural network. After calculating the multi-layer fully connected layers sequentially, the recommended action, i.e., acceleration, is obtained. And the execution of the recommended action, i.e., acceleration. Execution time .

[0018] Furthermore, the deep neural network The objective function during training is based on the advantage function. Construct the objective function for pruning The objective function of pruning As shown in the formula below:

[0019] ,

[0020] in: This indicates the policy network, or deep neural network, after parameter updates. The old policy network before parameter updates, i.e., the deep neural network. The output probability ratio of the recommended action selection; This indicates the currently updated network parameters, while This represents the network parameters of the previous old strategy;

[0021] These are the clipping parameters used to limit the range of variation in the probability ratio;

[0022] Used for cutting probability ratio To limit the magnitude of policy updates, if Exceeding the range Then Clipping to this range avoids excessive or unstable policy updates;

[0023] This represents the position, state, and action in the old policy network. Expectation of the distribution under the given conditions;

[0024] Represents the dominance function, dominance function The calculation formula is as follows:

[0025] ,

[0026] in: It is the satellite performing actions, i.e., acceleration. In position The action value function under the given conditions, and It is a value function, representing location. Expected returns;

[0027] During training, the advantage function It is approximately obtained by combining empirical data and using the generalized dominance estimation method. The dominance estimate is adjusted by weighting the time-series difference residuals at different time steps, thereby approximating the dominance function. The calculation formula is as follows:

[0028] ,

[0029] in: A t express Advantage function at time The approximate solution result of the generalized dominance estimation method is as follows: Difference estimation of time-of-flight acceleration value relative to location value;

[0030] ,in Indicates the satellite at a historical moment The reward obtained after performing a certain action; It is the position in the current policy network. The value estimate, expressed in the state The expected return; The next position state The value estimate, It is a discount factor used to adjust the current value of future rewards;

[0031] It is a smoothing parameter used to adjust the trade-off between the bias and variance of the dominance estimate;

[0032] Minimize the objective function of pruning during training Therefore, based on the objective function of pruning Gradient ascent method for deep neural networks Parameters in Update.

[0033] Furthermore, after step 2, the advantage function obtained by the generalized advantage estimation method is adjusted based on the performance changes of the policy network. Discount factor at time and the objective function of pruning. Clipping parameters in This allows for further optimization of the policy network.

[0034] Furthermore, in step 2, a satellite state update equation is established. Then, by combining the 8th-order Runge-Kutta method and the actions and execution times obtained in step 1, the satellite state update equation is solved, and the satellite orbit is controlled based on the solution results.

[0035] Furthermore, in step 2, an integral form of the satellite state update equation is established. Then, the 8th-order Runge-Kutta method is used to approximate the integral part of the integral form of the satellite state update equation based on the actions and execution times obtained in step 1, thereby solving the satellite state update equation.

[0036] This invention proposes a satellite orbit control method based on a continuous-time proximal policy optimization reinforcement learning algorithm, thereby achieving satellite orbit control. The continuous-time proximal policy optimization reinforcement learning algorithm employed in this invention fully considers the characteristics of the continuous-time environment and uses numerical integration techniques to achieve continuous prediction and updating of the state, ensuring real-time correspondence and high synchronization between the decision and the environmental state.

[0037] To overcome the limitations of discrete-time reinforcement learning models in dynamic environments, this invention employs a continuous-time proximal policy optimization reinforcement learning algorithm. This allows the algorithm to process time and state in a continuous manner, rather than relying on predefined time intervals. The introduction of the continuous-time frame enables the reinforcement learning algorithm to more accurately simulate and predict satellite behavior in complex orbital environments.

[0038] Furthermore, the introduction of a planned acceleration mode further enhances the algorithm's application capabilities. Through this mode, the invention can not only determine the optimal acceleration selection but also calculate and schedule the optimal acceleration execution time, effectively eliminating the uncertainty and risks caused by acceleration execution delays.

[0039] This invention addresses the problem of excessive oscillations that may occur during policy updates in highly volatile environments by applying a proximal policy optimization algorithm to this continuous-time and planned acceleration framework and by truncating the policy gradient. This improvement to the proximal policy optimization algorithm ensures the stability of the learning process and the reliability of the policy, making it more suitable for applications such as satellite orbit control where safety and accuracy are extremely critical.

[0040] This invention addresses the dynamic and complex problems encountered in satellite orbit control by combining a continuous-time reinforcement learning framework with a planned execution acceleration mode. This approach improves the real-time performance and accuracy of operations, as well as the overall efficiency and responsiveness of the system. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method according to an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the embodiments will be described in detail below with reference to the accompanying drawings and examples. This will allow for a full understanding of how the present invention uses technical means to solve technical problems and achieve corresponding technical effects, and to facilitate its implementation. The embodiments of the present invention and the various features within them can be combined with each other without conflict, and all resulting technical solutions are within the protection scope of the present invention.

[0043] Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion.

[0045] like Figure 1 As shown in the figure, this embodiment discloses a satellite orbit control method based on a continuous-time near-end policy optimization reinforcement learning algorithm, including the following steps:

[0046] Step 1: Optimize the reinforcement learning algorithm based on continuous-time near-end policy to determine the satellite's actions (acceleration) and the execution time of the actions. The specific process is explained below:

[0047] (1) For satellite orbit control, this embodiment establishes a satellite intelligent agent, which defines the satellite intelligent agent's position state space and action space. The position state space includes the satellite intelligent agent's position state vectors in a continuous time frame. ,in x , y , z This represents the position and state vector of the satellite intelligent agent along the x, y, and z axes in a three-dimensional spatial coordinate system. It provides a complete dynamic description of the satellite intelligent agent in three-dimensional space, providing a basis for the selection of actions, i.e. acceleration, in the decision-making process.

[0048] Action space includes the actions, i.e., accelerations, taken by the satellite intelligence within a continuous time frame. Let the position of the satellite intelligence agent at time t be s(t) and its velocity be v(t). The action, i.e., the acceleration, performed by the satellite intelligence agent at time t is... Execution of action, i.e., acceleration The execution time is Then the satellite intelligent agent executes at time t according to the execution time. Execution of action is acceleration Then, the satellite agent's velocity vector v(t) changes from time t to t+ The velocity at time t+ And cause the satellite agent to change its position s(t) from time t to t+ Position at time s(t+) ).

[0049] In this embodiment, a reward function is also established for the satellite agent. The reward function of satellite agents As shown in the following formula:

[0050] ,

[0051] in: Indicates that the satellite intelligent agent is based on execution time. After performing the action, i.e., acceleration 'a', the position state vector 's' and the target position state vector 's' Deviation between; and These are the cost coefficient and delay cost coefficient for the satellite intelligent agent to perform the action, i.e., acceleration 'a'. It is the magnitude of the action, i.e., the acceleration 'a', which reflects the cost of performing the acceleration. Indicates action, i.e., acceleration. Execution time.

[0052] The reward function Feedback to the satellite intelligent agent based on execution time The magnitude of acceleration 'a', cost, and state deviation during the execution of the action (acceleration 'a') are determined by this reward function. The aim is to guide satellite agents to optimize the cost and execution time of acceleration usage while minimizing positional deviations, thereby achieving efficient orbit control.

[0053] (2) In this embodiment, a policy network is constructed to generate recommended actions, i.e. recommended accelerations, and the execution time of recommended actions for satellite agents.

[0054] The policy network uses a deep neural network. ,in This represents the weights and bias parameters of the deep neural network, where s is the satellite's current position and state vector. It is the recommended action output by the policy network, i.e., the recommendation acceleration. The recommended action output by the policy network is called the recommendation acceleration. Execution time.

[0055] Deep neural networks, serving as policy networks, consist of multiple stacked fully connected layers (also known as dense layers). The output of each fully connected layer serves as the input to the next fully connected layer. The computation of each fully connected layer in a deep neural network can be represented as follows:

[0056] ,

[0057] in: It is the first The output of the fully connected layer, No. +1 output of the fully connected layer; and They are the first The weight matrix and bias vector of the fully connected layer; G() is the activation function used to introduce nonlinearity, enabling deep neural networks to learn and simulate more complex functional relationships.

[0058] In the deep neural network of this embodiment, the ReLU (Rectified Linear Unit) function is used as the activation function G() for each fully connected layer, resulting in... ,in x This is the input data for the ReLU function. The ReLU function outputs this input value if the input is greater than zero, otherwise it outputs zero.

[0059] In this embodiment, historical satellite trajectory data (the position, state, actions, and corresponding reward information of satellites at various historical moments) is used to analyze the deep neural network serving as the policy network. Training is performed using the position and state vectors from historical satellite trajectory data at various points in time. and velocity vector As a deep neural network The input, where v x , v y , v z These represent the satellite's speed. v The components of the x, y, and z axes in a three-dimensional coordinate system. Derived from a deep neural network. After calculating the recommended action, i.e., the recommended acceleration, through multiple fully connected layers, the recommended action is obtained. And to perform the recommended action, namely the recommended acceleration. Execution time Deep neural networks, which are trained to function as policy networks, are implemented. Iterative learning gradually optimizes the system. This training process involves calculating gradients and updating network parameters to improve prediction accuracy and policy effectiveness.

[0060] In this embodiment, based on the dominance function Construct the objective function for pruning The objective function of pruning As a deep neural network Objective function during training. Pruning-based objective function during training. To update the deep neural network parameters And through the objective function of pruning Controlling deep neural networks during training The magnitude of the update and prevention of performance crashes during the update process. The objective function of pruning. The expression is as follows:

[0061] ,

[0062] in: This indicates the policy network, or deep neural network, after parameter updates. The old policy network before parameter updates, i.e., the deep neural network. The output probability ratio of the recommended action selection; This indicates the currently updated network parameters, while This represents the network parameters of the previous old strategy. It is a key quantity in the algorithm for measuring the differences between the old and new policy networks. When the value is close to 1, it indicates that the new policy network is very similar to the old policy network; when... A deviation from 1 indicates a significant difference between the new policy network's preference for the current recommended action and that of the old policy network.

[0063] This is the clipping parameter, used to limit the range of variation of the probability ratio. The initial clipping parameter can be set to 0.1.

[0064] Used for cutting probability ratio To limit the magnitude of policy updates, if Exceeding the range Then Trim to this range to avoid excessive or unstable policy updates.

[0065] This represents the position, state, and action in the old policy network. The expectation is calculated based on the distribution under the old policy network, that is, the expectation of the objective function is calculated based on the position state-action data collected during the operation of the old policy network. When updating the policy network, the near-end policy optimization algorithm uses the trajectory data obtained by downsampling from the old policy network, and then evaluates and optimizes the new policy network based on these data.

[0066] Represents the dominance function, dominance function It measures the acceleration of the action taken. and execution time At that time, in position The additional value relative to the average policy, the objective function of pruning. The study refines policy learning by calculating the advantage function, thereby achieving a more efficient and precise control policy. Advantage function This is a key metric in the continuous-time proximal policy optimization reinforcement learning algorithm of this embodiment when updating the policy network. It is used to guide the policy network towards a more favorable recommendation acceleration, i.e., acceleration update. Advantage function The calculation formula is as follows:

[0067] ,

[0068] in: It is the satellite performing actions, i.e., acceleration. In position The action value function under the given conditions, and It is a value function, representing location. The expected return.

[0069] In this embodiment, during training, the advantage function... It will not be directly derived from the definition. Instead of precise calculation, the action value function is approximated by combining empirical data and using the generalized advantage estimation method. The value function is typically calculated by combining the value obtained from sampling with an estimate of the value of the future location state. By independent value network An estimation is provided. To reduce variance and improve the accuracy of the dominance estimate, this embodiment employs a generalized dominance estimation method, which adjusts the dominance estimate by weighting the time-series difference residuals at different time steps, thereby approximating the dominance function. The calculation formula is as follows:

[0070] ,

[0071] in: A t express Advantage function at time The approximate solution result of the generalized dominance estimation method is as follows: Difference estimation of time-of-flight acceleration value relative to location value;

[0072] ,in Indicates the satellite at a historical moment The reward obtained after performing a certain action; It is the position in the current policy network. The value estimate, expressed in the state The expected return; The next position state The value estimate, It is a discount factor used to adjust the current value of future rewards.

[0073] It is a smoothing parameter used to adjust the trade-off between the bias and variance of the dominance estimate;

[0074] Advantage function Once calculated, the policy network, i.e., the deep neural network, is... This can be achieved through the objective function of pruning. The network parameters are updated iteratively. During each learning cycle, the deep neural network... Its parameters are adjusted using data obtained from the trajectory. Optimize the objective function of pruning This ensures the gradual improvement of the policy network while preventing excessive updates.

[0075] This embodiment is based on the objective function of pruning. The gradient ascent method is used to train the policy network, i.e., the deep neural network, during the training process. Parameters in The update is performed as shown in the following formula:

[0076] ,

[0077] in: It is the learning rate; It is the gradient of the objective function with respect to the policy network parameters; This represents the new parameters of the policy network after the current time step. This represents the original parameters of the policy network before the current time step update.

[0078] Through this objective function calculation and policy network update mechanism based on the advantage function pruning, the continuous-time proximal policy optimization reinforcement learning algorithm in this embodiment can effectively drive the policy toward optimization while maintaining learning stability.

[0079] In this embodiment, the process of training the policy network using historical satellite trajectory data is as follows:

[0080] First, based on historical satellite trajectory data, generate a dataset containing... Position and state at any moment ,action Execution time ,award and the next state The training samples.

[0081] Then, the training samples are input into the policy network, and the policy network parameters are iteratively optimized. By minimizing the objective function of pruning This improves the performance of the policy network in orbit control tasks. After each iteration, the trained policy network is used to adjust the current state. and speed Output recommended actions, i.e., acceleration, to the satellite intelligent agent. and execution time .

[0082] In this embodiment, a policy network trained using historical satellite trajectory data can provide high-precision acceleration and execution time decisions in satellite orbit control. Specifically, it can determine the time from the current time t to the execution time. The continuous-time integral form between the two allows the policy network to more accurately describe the behavior and changes of the satellite, and the control signals can be adjusted more smoothly, avoiding the jumps and discontinuities that may occur in discrete time.

[0083] (3) In this embodiment, the satellite agent receives the recommended action, i.e., the recommended acceleration, output by the trained policy network. and execution time Then, through the reward function Assess the performance of recommended actions, i.e., acceleration. The effects include the magnitude of the action, cost, and state deviation. Based on the reward function. The feedback from the policy network outputs multiple recommended actions and execution times to determine the optimal action, i.e., acceleration, and optimal execution time, which is the optimal solution that achieves the best balance between efficiency, cost, and state deviation.

[0084] Therefore, in this embodiment, based on the aforementioned continuous-time proximal strategy optimization reinforcement learning algorithm, the optimal acceleration at time t is ultimately determined by the satellite agent. and optimal execution time (opt) represents the action taken by the satellite at time t, namely acceleration and the execution time.

[0085] Step 2: Based on the actions taken by the satellite and the execution time obtained in Step 1, achieve satellite orbit control.

[0086] Specifically, in this embodiment, a satellite state update equation is established. Then, combining the 8th-order Runge-Kutta method and the actions and their execution times obtained in step 1, the satellite state update equation is solved. Based on the solution results, satellite orbit control is achieved. The process is as follows:

[0087] (2.1) Establish the first-order differential dynamic equations of the satellite state, including the first-order differential dynamic equations of the position vector and the first-order differential dynamic equations of the velocity vector.

[0088] In this embodiment, to accurately simulate the satellite's orbital dynamics, a dynamic equation for the satellite's motion state in its orbit is first established. The satellite's motion state can be represented by its position vector. and velocity vector To fully describe, in Represents the satellite's spatial position coordinates relative to the Earth's center. The velocity components are represented by x, y, and z, which are the x-axis, y-axis, and z-axis coordinates of the satellite's spatial position, respectively. , , They are respectively x , y , z The first derivative. Furthermore, considering the effects of perturbations such as the J2 effect, these states of the satellite can be described by the following second-order dynamic differential equation:

[0089] ,

[0090] in, The resultant force representing all external forces, especially the primary gravitational term. ; Position vector The first derivative; Position vector The second derivative or velocity vector The first derivative; It is the gravitational constant; Earth mass; l It is the distance from the satellite to the Earth's center.

[0091] The above second-order differential dynamic equation is transformed into an equivalent first-order differential dynamic equation by introducing a velocity vector. As a new variable, the first-order differential dynamic equation is shown below:

[0092] ,

[0093] ,

[0094] in, t For time; velocity vector The first derivative.

[0095] (2.2) Based on the above first-order differential dynamic equations, a continuous time frame is adopted to establish the satellite state update equations in integral form, including the position state update equations and the velocity state update equations.

[0096] The current state of a satellite is determined by its position vector ( ) and velocity vector ( The system is structured as follows. To ensure the continuity and accuracy of state changes, this embodiment establishes an integral form of the satellite state update equations, including position state update equations and velocity state update equations, as shown in the following formulas:

[0097] ,

[0098] ,

[0099] in: Indicates the satellite's time speed; This represents the action, or acceleration, exerted by the satellite at time t; Indicates the satellite's time Position status; Indicates the satellite's time Speed ​​state; Indicates the updated satellite position status; This indicates the updated satellite velocity status.

[0100] This embodiment transforms the first-order differential dynamic equation into this integral form, fully considering the current time point. To the next time point The dynamic changes provide a continuous time-to-time status update mechanism for the satellite.

[0101] (2.3) The 8th-order Runge-Kutta numerical integration method (i.e. RK8 method) is adopted to approximate the integral part of the satellite state update equation in integral form by calculating the 8th-order slope, thereby solving for the updated satellite position and velocity state.

[0102] In this embodiment, the 8th-order Runge-Kutta numerical integration method is used to calculate the 8th-order position slope. and 8th order velocity slope This approximates the integral form of the satellite state update equation in step 2, where... ki s Indicates the first position state update. Slope of position, k i v The first time when updating the speed state The slope of the first velocity.

[0103] In the specific calculation process, the first Step position slope k i s By satellite in time velocity vector The decision is made, and the calculation formula is as follows:

[0104] ,

[0105] In the formula: The first is determined by the 8th-order Runge-Kutta numerical integration method. i The time coefficient of the order;

[0106] velocity vector It is determined by the satellite's velocity v at time t. t The actions performed by the satellite, namely acceleration. The calculated result, i.e. = v( t )+ The acceleration is the action performed by the satellite at time t. This refers to the action at time t obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm. = ; To perform the action The time step is the execution time of the action at time t, obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm. = (opt).

[0107] No. First velocity slope k i v The satellite's position is calculated using the dynamic equation f(). Location Satellite in time speed The decision is made, and the calculation formula is as follows:

[0108] ( ),

[0109] In the formula: Indicates the time step, i.e. That is, the execution time of the action at time t obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm. = (opt);

[0110] , These are the 8th-order Runge-Kutta numerical integration methods, which are determined by the 8th-order Runge-Kutta numerical integration method. i The time coefficient of the order;

[0111] Location It is through numerical integration methods in time The position vector obtained at time t, i.e. ;in, It is in time The velocity vector at a given moment can be obtained from the known velocity slope. Perform approximate calculations; To perform the action The time step is the execution time of the action at time t, obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm. = (opt);

[0112] speed It is determined by the satellite's velocity v at time t. t The actions performed by the satellite, namely acceleration. Calculated, i.e. = v( t )+ The action performed by the satellite at time t is called acceleration. The action at time t obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm is... = ; To perform the action The time step is the execution time of the action at time t, obtained in step 1 through the continuous-time proximal policy optimization reinforcement learning algorithm. = (opt).

[0113] Finally, by assigning weights to all 8th order slopes... Weighted summation is used to accurately approximate the integral part of the satellite state update equation in integral form. These slopes reflect the changing trend of the state at different time steps. The updated satellite position and velocity states are obtained by solving for them, as shown in the following equation:

[0114] ,

[0115] In the formula: These are weighting coefficients used to ensure the reasonableness of contributions at different stages and the accuracy of calculation results.

[0116] Using the aforementioned 8th-order Runge-Kutta method, the integral part of the satellite state update equation in integral form is approximated with high precision, thus realizing the accurate tracking of satellite position. and speed State at time step Precise updates are then made to ensure the accuracy and stability of orbit control.

[0117] After the calculation of each time step is completed, the time variable is updated, and the above process is continuously looped in the RK8 method until the predetermined simulation termination time is reached.

[0118] In this embodiment, the optimal action (acceleration) and optimal execution time obtained from the continuous-time proximal policy optimization reinforcement learning algorithm in step 1 are applied to the satellite. Based on the optimal action (acceleration) and optimal execution time, and combined with the 8th-order Runge-Kutta method, the satellite calculates the 8th-order position slope. and 8th order velocity slope This is used to approximate the integral form of the satellite state update equation, thereby solving for the updated satellite position and velocity states, and enabling control of the satellite orbit.

[0119] In this embodiment, after satellite orbit control is performed at each time step, the dominance function obtained by solving the generalized dominance estimation method is adjusted based on the performance changes of the policy network. Discount factor at time and the objective function of pruning. Clipping parameters in This allows for further optimization of the policy network. Details are as follows:

[0120] (A) Discount Factor The discount factor is primarily responsible for controlling the emphasis placed on future rewards in near-term policy optimization algorithms, thereby influencing the long-term goal positioning of the policy. Appropriately adjusting the discount factor can help the policy better reflect the emphasis on long-term goals or immediate effects. Discount Factor The adjustment formula is as follows:

[0121] ,

[0122] in: This represents the adjusted discount factor; Indicates the discount factor currently in use; It is a scaling factor adjusted based on policy network performance feedback, used to fine-tune the discount factor to optimize the weighting of long-term rewards; This is the lower limit of the discount factor, ensuring that the discount factor will not fall below this value; This is the upper limit for the discount factor, ensuring that the discount factor does not exceed this value. and Ensure adjustments are within a reasonable range to avoid extreme values ​​affecting the stability of the learning process.

[0123] (B) Cutting parameters In proximal policy optimization algorithms, pruning parameters plays a crucial role in limiting the policy update step size. The key to optimization lies in preventing excessive oscillations during the learning process and ensuring the smoothness and reliability of policy updates. The formula for adjusting the pruning parameters is as follows:

[0124] ,

[0125] in: These are the adjusted cutting parameters; These are the currently used cropping parameters; It adjusts the sensitivity factor, which determines how performance changes affect the adjustment of the cutting range; This represents the observed changes in policy network performance; and These are the lower and upper limits of the trimming parameters, ensuring that the trimming parameters will not fall below or exceed these values. and A reasonable range of variation for the trimming parameters was defined to ensure stability and effectiveness during the adjustment process.

[0126] Adjusting discount factor and cutting parameters The algorithm balances the relationship between long-term goals and short-term feedback in the policy network, ensuring that the continuous-time proximal policy optimization reinforcement learning algorithm remains efficient and stable under constantly changing task conditions. This is achieved by dynamically adjusting the discount factor. and cutting parameters This enables the continuous-time proximal policy optimization reinforcement learning algorithm in this embodiment to effectively drive the policy toward optimization while maintaining learning stability, ensuring efficient and stable execution of satellite orbit control under various mission conditions.

[0127] This embodiment demonstrates significant advancements and numerous advantages based on the application of a continuous-time-frame proximal policy optimization reinforcement learning algorithm. By limiting the magnitude of each policy update, the algorithm ensures the stability and convergence of the policy update process. This is particularly important in satellite orbit control, as orbit adjustments require highly precise and stable control policies to prevent excessive adjustments from causing system instability or orbital deviations. Simultaneously, the algorithm exhibits high sample efficiency, enabling it to learn effective control policies within a limited number of interactions, reducing reliance on large amounts of training data. This is crucial for satellite orbit control, given its high-dimensional state and action spaces and the high cost of sample acquisition.

[0128] Furthermore, satellite orbit control requires fine-tuning within a continuous action space, such as the magnitude and direction of thrust. This embodiment utilizes a continuous-time-frame near-end policy optimization reinforcement learning algorithm, which excels at handling continuous action spaces and can generate smooth and precise control commands to meet the needs of orbit adjustments. The satellite orbit control environment is typically highly uncertain and dynamically changing, including external disturbances and fuel consumption. Near-end policy optimization algorithms exhibit good robustness, maintaining stable performance in the face of environmental changes and disturbances, and continuously learning to adapt to new environmental conditions.

[0129] This embodiment demonstrates several advantages of a continuous-time frame near-end policy optimization reinforcement learning algorithm in satellite orbit control. First, the continuous-time frame more closely resembles real-world modeling, enabling a more accurate description of the behavior and changes of continuous dynamic systems such as satellites. Compared to discrete-time frames, the continuous-time architecture allows for decisions and updates at any given moment, unrestricted by fixed time steps, which is particularly important for tasks requiring high-precision time control. Simultaneously, control signals in continuous-time can be adjusted more smoothly, avoiding the jumps and discontinuities that may occur in discrete-time models, thus improving the stability and reliability of the control strategy.

[0130] In terms of mathematical foundations, this embodiment utilizes a continuous-time frame-based near-end policy optimization reinforcement learning algorithm, which better integrates differential equations and dynamic systems theory from classical control theory, leveraging existing mathematical tools and theories to improve algorithm performance and stability. Continuous-time models often possess superior properties in theoretical analysis, such as smoothness and differentiability, facilitating the design of more efficient learning algorithms and optimization methods. Furthermore, the continuous-time framework enables adaptive sampling, selecting sampling time points based on the system's dynamic changes, avoiding unnecessary computations when system changes are slow, thus utilizing computational resources more effectively. Continuous-time reinforcement learning also enhances the model's generalization ability, better adapting to changes at different time scales and improving the model's adaptability and flexibility in different environments and tasks. Policies can be finely adjusted at any time point, making the learned policies more flexible and adaptable. In addition, the continuous-time framework can more naturally integrate with the laws and phenomena of the physical world, such as dynamic modeling and energy conservation, improving the physical feasibility and effectiveness of the control strategy.

[0131] This embodiment applies the continuous-time frame near-end policy optimization reinforcement learning algorithm to satellite orbit control. It not only overcomes the shortcomings of traditional control methods and existing reinforcement learning methods in dynamic environments, but also achieves higher precision and smoother orbit control through its stability, efficiency, continuity, and robustness. These advantages make the continuous-time near-end policy optimization algorithm an advanced technology in the field of satellite orbit control, capable of meeting the stringent requirements of modern satellite missions for real-time performance, adaptability, and efficiency.

[0132] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0133] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A satellite orbit control method based on a continuous-time near-end policy optimization reinforcement learning algorithm, characterized in that, Includes the following steps: Step 1: Optimize the reinforcement learning algorithm based on continuous-time proximal policy to determine the satellite's actions (acceleration) and the execution time of the actions. The process is as follows: An intelligent agent for the satellite is established, defining its position state space and action space. The position state space includes the position state vector of the satellite agent in a continuous time frame, and the action space includes the actions taken by the satellite agent in a continuous time frame, i.e., accelerations. A reward function is also established for the satellite agent. A policy network is constructed and trained using historical satellite trajectory data. The trained policy network then outputs recommended actions and their execution times to the satellite agent. After receiving the recommended action and execution time output by the trained policy network, the satellite agent evaluates the received recommended action and execution time through the reward function, thereby determining the optimal action, i.e. acceleration, and the optimal execution time, which are then used as the actions taken by the satellite, i.e. acceleration and execution time. Step 2: Based on the actions taken by the satellite and the execution time obtained in Step 1, achieve satellite orbit control.

2. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 1, characterized in that, In step 1, the reward function of the satellite agent Feedback to the satellite intelligent agent based on execution time The magnitude, cost, and state deviation of the acceleration 'a' during the execution of the action (acceleration 'a'), and the reward function. As shown in the following formula: ， in: This indicates that the satellite intelligent agent is based on execution time. After performing the action, i.e., acceleration 'a', the position state vector 's' and the target position state vector 's' Deviation between; and These are the cost coefficient and delay cost coefficient for the satellite intelligent agent to perform the action, i.e., acceleration 'a'. It is the magnitude of the action, i.e., the acceleration 'a', which reflects the cost of performing the acceleration. Indicates action, i.e., acceleration. Execution time.

3. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 1, characterized in that, In step 1, the policy network employs a deep neural network. Deep neural networks, serving as policy networks, consist of multiple stacked fully connected layers, with the output of each fully connected layer serving as the input to the next fully connected layer; among them, This represents the weights and bias parameters of the deep neural network, where s is the satellite's current position and state vector. It is the recommended action, or acceleration, output by the policy network. The recommended action output by the policy network is acceleration. Execution time.

4. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 3, characterized in that, Deep Neural Networks The activation function for each fully connected layer is the ReLU function.

5. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 3, characterized in that, Deep Neural Networks That is, during the training of the policy network, the position and state vectors and velocity vectors from historical satellite trajectory data are used as the deep neural network. The input is provided by a deep neural network. After calculating the recommended action, i.e., acceleration, from the multiple fully connected layers, the recommended action is obtained. And the execution of the recommended action, i.e., acceleration. Execution time .

6. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 3, characterized in that, The deep neural network The objective function during training is based on the advantage function. Construct the objective function for pruning The objective function of pruning As shown in the formula below: ， in: This indicates the policy network, or deep neural network, after parameter updates. The old policy network before parameter updates, i.e., the deep neural network. The output probability ratio of the recommended action selection; This indicates the currently updated network parameters, while This represents the network parameters of the previous old strategy; These are the clipping parameters used to limit the range of variation in the probability ratio; Used for cutting probability ratio To limit the magnitude of policy updates, if Beyond the range Then Clipping to this range avoids excessive or unstable policy updates; This represents the position, state, and action in the old policy network. Expectation of the distribution under the given conditions; Represents the dominance function, dominance function The calculation formula is as follows: ， in: It is the satellite performing actions, i.e., acceleration. In position The action value function under the given conditions, and It is a value function, representing location. Expected returns; During training, the advantage function It is obtained approximately by combining empirical data and using the generalized dominance estimation method. The dominance estimate is adjusted by weighting the time-series difference residuals at different time steps, thereby approximating the dominance function. The calculation formula is as follows: ， in: A t express Advantage function at time The approximate solution result of the generalized dominance estimation method is as follows: Difference estimation of time-of-flight acceleration value relative to location value; ,in Indicates the satellite at a historical moment The reward obtained after performing a certain action; It is the position in the current policy network. The value estimate, expressed in the state The expected return; The next position state The value estimate, It is a discount factor used to adjust the current value of future rewards; It is a smoothing parameter used to adjust the trade-off between the bias and variance of the dominance estimate; Minimize the objective function of pruning during training Therefore, based on the objective function of pruning Gradient ascent method for deep neural networks Parameters in Update.

7. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 6, characterized in that, After step 2, the advantage function obtained by the generalized advantage estimation method is adjusted based on the performance changes of the policy network. Discount factor at time and the objective function of pruning. Clipping parameters in This allows for further optimization of the policy network.

8. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to any one of claims 1-7, characterized in that, In step 2, the satellite state update equation is established. Then, the satellite state update equation is solved by combining the 8th-order Runge-Kutta method and the actions and execution times obtained in step 1. The satellite orbit is controlled based on the solution results.

9. The satellite orbit control method based on continuous-time proximal policy optimization reinforcement learning algorithm according to claim 8, characterized in that, In step 2, an integral form of the satellite state update equation is established. Then, the 8th-order Runge-Kutta method is used to approximate the integral part of the integral form of the satellite state update equation based on the actions and execution times obtained in step 1, thereby solving the satellite state update equation.

Images