Spacecraft attitude and orbit integrated game control method and device based on PPO

Abstract

The application provides a spacecraft attitude and orbit integrated game control method and device based on PPO, which comprises establishing a spacecraft attitude and orbit integrated model, modeling the interaction process of a tracking spacecraft and a target spacecraft as a non-cooperative zero-sum game; setting a multi-agent deep reinforcement learning framework, wherein the tracking spacecraft and the target spacecraft are respectively configured with independent policy networks and value networks; training in a double-agent PPO mode, generating control instructions of orbit thrust and attitude moment through the policy network, and evaluating state value through the value network; and controlling the behavior of the spacecraft according to the obtained policy. The application effectively improves the autonomous decision-making ability and cooperative control performance of the spacecraft in complex tasks, and is suitable for high-precision space tasks such as interception of non-cooperative targets, autonomous trajectory planning and attitude control.

Description

Technical Field

[0001] This invention belongs to the field of spacecraft control and artificial intelligence technology, specifically relating to a spacecraft attitude and orbit integrated game control technology based on the Proximal Policy Optimization (PPO) algorithm, used for non-cooperative target interception, autonomous trajectory planning and attitude adjustment tasks. Background Technology

[0002] During spacecraft interception, the spacecraft's sensors need to maintain continuous observation of the target to provide accurate relative position and velocity information. Due to the maneuverability and uncertainty of non-cooperative targets, the spacecraft must not only maintain orbital adjustments to approach the target but also make real-time attitude adjustments based on changes in the target's attitude. At this point, the spacecraft's attitude and orbital control must be optimized synchronously. In spacecraft game control, many studies have only considered the orbital level, such as orbital interception and pursuit escape. These methods neglect the crucial role of attitude control in practical missions, leading to control strategies that are difficult to execute efficiently in complex tasks. In complex mission environments, the coupling between orbital control and attitude control cannot be ignored. Due to the strong coupling between attitude and orbital motion, if only orbital control is considered, the spacecraft's motion may not achieve high precision and efficiency. Therefore, there is an urgent need to design an integrated spacecraft attitude and orbit game control scheme. Summary of the Invention

[0003] This invention aims to address the problems of attitude and orbit control separation, lack of coupled control, and one-sided game model in existing technologies. It proposes an integrated attitude and orbit game control method for spacecraft interception missions, which takes into account both orbit and attitude constraints, and optimizes the strategy through the PPO algorithm to improve the spacecraft's autonomous decision-making ability and mission completion efficiency.

[0004] The method of this invention provides a spacecraft attitude and orbit integrated game-theoretic control method based on PPO, comprising the following processes: Establish an integrated attitude and orbit model for spacecraft, and model the interaction process between the tracking spacecraft and the target spacecraft as a non-cooperative zero-sum game; A multi-agent deep reinforcement learning framework is set up, in which the tracking spacecraft and the target spacecraft are configured with independent policy networks and value networks, respectively. The system is trained using a dual-agent PPO approach. The policy network generates control commands for orbital thrust and attitude torque, and the value network evaluates the state value. The behavior of the spacecraft is controlled and guided based on the strategies learned during training.

[0005] Furthermore, the modeling process for the non-cooperative zero-sum game includes defining the cost function for the tracking spacecraft and the cost function for the target spacecraft. The tracking spacecraft and the target spacecraft optimize their strategies by minimizing their respective cost functions, ultimately forming a Nash equilibrium strategy. Furthermore, the policy network uses a Gaussian policy to generate control instructions for a continuous action space.

[0006] Moreover, the PPO is a near-end policy optimization algorithm that uses a probability ratio shearing mechanism to limit the policy update magnitude.

[0007] Furthermore, the proximal policy optimization algorithm employs a generalized advantage estimation method to calculate the advantage function.

[0008] Moreover, the training process employs a multi-scale reward function, including pose error reward, fuel consumption reward, terminal pose reward, and time reward.

[0009] Moreover, the terminal pose reward is triggered when the attitude error and position error between the tracking spacecraft and the target spacecraft are both less than a predetermined threshold.

[0010] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the spacecraft attitude and orbit integrated game control method based on PPO as described above.

[0011] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the spacecraft attitude and orbit integrated game control method based on PPO as described above.

[0012] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the spacecraft attitude and orbit integrated game control method based on PPO as described above.

[0013] The beneficial effects of this invention are: (1) This invention unifies the modeling of the spacecraft's attitude control and orbit control in the Lie group SE(3) space, avoiding the attitude-orbit coupling error caused by model separation in traditional methods, and improving the consistency and accuracy of the control model. (2) In terms of game modeling, the interaction process between the tracking spacecraft and the target spacecraft is constructed as a non-cooperative zero-sum game model, making the strategy solution process more in line with the real adversarial scenario and able to dynamically adapt to the maneuvering behavior of the target spacecraft. (3) In the reinforcement learning training process, by introducing a proximal policy optimization algorithm based on the policy shearing mechanism and combining it with a multi-scale reward function, the problems of local optima, convergence difficulties and inaccurate task completion evaluation in the high-dimensional action space are effectively solved, especially in terms of pose error and fuel consumption control. The introduction of the PPO algorithm significantly improves the training stability through the policy shearing mechanism, and realizes the smoothness of action generation and the diversity of policy exploration in the high-dimensional continuous action space. Finally, when facing a target spacecraft with complex maneuvering characteristics, the tracking spacecraft can complete trajectory tracking and attitude alignment more quickly, and the control output response is more stable and accurate, meeting the actual needs of high-dynamic mission environments. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the spacecraft attitude and orbit integrated game control process based on PPO provided in an embodiment of the present invention; Figure 2 This is a training logic diagram of the PPO algorithm according to an embodiment of the present invention; Figure 3 This is a diagram showing the deep reinforcement learning training results of an embodiment of the present invention; Figure 4 This is a graph showing the change in attitude error over time according to an embodiment of the present invention; Figure 5 This is a graph showing the relative attitude change over time according to an embodiment of the present invention; Figure 6 This is a graph showing the change in position error over time according to an embodiment of the present invention; Figure 7 This is a graph showing the relative distance changing over time according to an embodiment of the present invention; Figure 8 This is a trajectory maneuver curve diagram of an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0016] To address the problems existing in the prior art, this invention proposes an attitude-orbit integrated game-theoretic control method for spacecraft interception missions, taking into account both orbital and attitude constraints, and using the PPO algorithm to achieve policy optimization, thereby improving the spacecraft's autonomous decision-making ability and mission completion efficiency. This method first establishes a six-degree-of-freedom relative kinematics and dynamics coupled model of the spacecraft based on the Lie group SE(3), achieving unified modeling of attitude and orbit control. Subsequently, the interaction between the tracking spacecraft and the target spacecraft is modeled as a non-cooperative zero-sum game process, and a Markov decision model is constructed. Based on a deep reinforcement learning framework, an Actor-Critic structure policy network and value network are designed, and Gaussian policies are used to generate control commands. During training, the PPO algorithm is used to achieve policy optimization, and a multi-scale reward mechanism is introduced to improve interception accuracy, efficiency, and robustness. This invention effectively improves the autonomous decision-making ability and cooperative control performance of spacecraft in complex missions, and is suitable for high-precision aerospace missions such as interception of non-cooperative targets, autonomous trajectory planning, and attitude control.

[0017] See Figure 1 The spacecraft attitude and orbit integrated game-theoretic control method based on PPO provided in this embodiment of the invention includes the following steps: Step S1: Based on the Lie group SE(3), establish a six-degree-of-freedom relative kinematics and dynamics coupling model of the spacecraft, and model the attitude control and orbit control in a unified manner. It has global singularity and mathematical consistency, and ensures that the control strategy has good adaptability to the attitude-orbit coupling relationship.

[0018] In the process of modeling the relative motion of spacecraft, this invention introduces the Lie group SE(3) to describe the integrated attitude and orbit relative model of spacecraft.

[0019] The preferred implementation of step S2 further includes the following sub-steps: Step S11: Establish a spacecraft attitude and orbit integrated model based on Lie group SE(3).

[0020] Spacecraft pose in Lie group SE(3) It can be described as (1) in It is orthogonal matrix, , is the rotation matrix of the spacecraft's body coordinate system relative to the inertial coordinate system, used to describe the spacecraft's rotation in three-dimensional space. It is The column vectors represent the position of the spacecraft's body coordinate system relative to the inertial coordinate system, and are used to describe the spacecraft's translation in three-dimensional space. It is a three-dimensional rotation group. It is a three-dimensional special Euclidean group. It is a 1×3 zero matrix.

[0021] General attitude velocity of spacecraft It can be described as (2) in Let be the angular velocity of the spacecraft in its body coordinate system. Let be the linear velocity of the spacecraft in its body coordinate system. Generalized attitude velocity The six-dimensional space in which it exists. The secondary form of generalized velocity. for (3) Where the superscript × denotes the antisymmetric matrix operator, (3) is a special Euclidean group SE (3) The corresponding Lie algebra.

[0022] To establish an integrated attitude and orbit model of a spacecraft, several adjoint operators need to be defined to describe the changes in the spacecraft's position and velocity.

[0023] The adjoint transformation matrix of a rigid body in space can be expressed as: (4) In the formula, express antisymmetric transformation, It is a 1×3 zero matrix. Vector antisymmetric transformation to (5) in, , , These are the three components of the three-dimensional vector x.

[0024] The adjoint operator matrix of rigid body velocity It can be represented as (6) yes isomorphic space, yes and The semi-direct product, , It is a three-dimensional real vector space. anti-adjoint operator It can be obtained through the dual operation of Lie algebras. (7) in, for The transpose of .

[0025] Based on the mapping relationship above, the attitude and orbit kinematic equations of the spacecraft in the inertial coordinate system can be obtained. (8) in, It is a position The time derivative of is the physical meaning of the instantaneous motion state of the aircraft on the SE(3) group.

[0026] An integrated attitude and orbit dynamic model of the spacecraft in the inertial coordinate system can be obtained. (9) in It is the inertial tensor matrix composed of the spacecraft's mass and moment of inertia. For the spacecraft's rotational inertia, For the mass of the spacecraft. For the gravitational gradient moment and gravity of the spacecraft, Interference experienced by the spacecraft For the control input of the spacecraft. This represents the gravitational gradient moment acting on the spacecraft. It is the control torque of the spacecraft. The disturbance torque experienced by the spacecraft For the gravitational force acting on the spacecraft, Indicates the control force of a spacecraft. This indicates the interference force experienced by the spacecraft.

[0027] Then the kinematic and dynamic equations of the spacecraft's attitude and orbit integration can be obtained as follows: (10) Step S12: Establish a six-degree-of-freedom relative kinematics and dynamics coupling model of the spacecraft based on the Lie group SE(3).

[0028] Let the desired attitude of the target spacecraft be... Tracking the spacecraft's own attitude Tracking attitude and orbit tracking errors between the spacecraft and the target spacecraft. It can be described as (11) (12) in, Let be the rotation error matrix of the two spacecraft. Here is the position error matrix for the two spacecraft. The transpose of the target spacecraft's attitude matrix. The location of the target spacecraft. The tracking error is described using the exponential coordinates on the Lie group SE(3). (13) in This represents the tracking error in the attitude section. This indicates the position tracking error. and Satisfy the following formula (14) (15) (16) in, For rotation angle, The trace of the matrix, It is a 3×3 identity matrix. This is an extended form of the Rodriguez rotation formula.

[0029] Speed ​​tracking error for (17) in, This is the transpose of the angular velocity error. For the transpose of the speed error, For Li Qun SE (3) is an accompanying representation. For the desired generalized speed.

[0030] Using exponential coordinates based on the Lie group SE(3) To describe the attitude tracking error of a spacecraft, the kinematic equation for the attitude error can be expressed as: (18) In the formula (19) (20) (twenty one) in, Exponential coordinates based on the Lie group SE(3) Time derivative, This is the velocity transformation matrix. For generalized velocity error, Used to handle rotational motion, Used to handle the coupling effect of rotation and translation. It is an identity matrix.

[0031] The relative attitude-orbit integrated dynamic equation based on the Lie group SE(3) can be described as follows: (twenty two) The kinematic and dynamic model of spacecraft attitude and orbit integrated error is as follows (twenty three) Step S2 involves modeling the interaction between the tracking spacecraft and the target spacecraft as a non-cooperative zero-sum game process, and defining the state space, action space, state transition function, and reward function based on this, thus abstracting the game task into a Markov decision process (MDP).

[0032] The non-cooperative zero-sum game is a game of complete information, with the tracking spacecraft and the target spacecraft acting as players. The objective is to minimize their respective cost functions, ultimately leading to a Nash equilibrium solution. The attitude and orbit integration game is a non-cooperative, complete-information zero-sum differential game problem, and a mathematical model of the strategies of both parties can be established.

[0033] The preferred implementation of step S2 further includes the following sub-steps: Step S21: Based on the established spacecraft attitude and orbit integration model, construct a spacecraft attitude and orbit integration game model. The basic elements of game theory include the following six aspects: participants, states, strategies, state equations, cost functions, and Nash equilibrium. In the spacecraft attitude and orbit integration game, participants include the tracking spacecraft and the target spacecraft. States include the attitude, angular velocity, position, and velocity of the tracking and target spacecraft. Strategies can change the states of the participants and can be represented by an N-dimensional vector. It is subject to specific constraints within the bounded set of all available strategies. These represent the components of the control strategy. The state equation of a game can be described by differential equations. (twenty four) in, For time, Let t be the state at time t. The derivative of the state. For dynamic equation functions, To control the policy components, The system state at time 0. This represents the initial state of the system.

[0034] Each participant has its own cost function and aims to minimize it, i.e. (25) in Representing the Cost function of each participant It is the terminal state cost function, describing the system at the final moment. status Punishment or reward. It is a process state cost function, describing the process state from the initial time step. At the terminal time The cumulative cost.

[0035] In the spacecraft attitude and orbit integration game, the tracking spacecraft learns the optimal strategy to catch up with the target spacecraft. Given the relationship between the tracking and target spacecraft, the attitude and orbit integration game is a non-cooperative, zero-sum differential game problem with complete information. The mathematical model of the strategies of both sides is described as follows: (26) in It is the cost function for tracking spacecraft. and It tracks the spacecraft's state vector. It is a control strategy for tracking spacecraft; It is the cost function of the target spacecraft. and It is the state vector of the target spacecraft. It is the control strategy for the target spacecraft; It is the optimal cost for tracking spacecraft. It is the optimal cost for the target spacecraft.

[0036] In this game, the tracking and target spacecraft optimize their strategies based on a cost function, aiming to maximize or minimize their respective costs. Since each player's strategy influences the final game outcome, both sides must optimize their strategies to achieve the optimal result during the game. Therefore, assuming both sides are rational, the final strategies will form a Nash equilibrium condition, i.e. (27) in, It is the minimum cost for tracking spacecraft under Nash equilibrium. It is the cost function for tracking spacecraft when both strategies depend on each other. It is the minimum cost of the target spacecraft under Nash equilibrium. It is the cost function of the target spacecraft when the strategies of both parties depend on each other.

[0037] Step S22, the integrated attitude and orbit game of the spacecraft can be modeled as a Markov process. It consists of five elements.

[0038] (28) in Represents the state space of the intelligent agent. Is the intelligent agent at time step The state; Represents the action space. Is the intelligent agent at time step The chosen action; Represents the state transition probability. Represents the state transition probability, policy It is in state Select action The probability of; Represents the reward function; This represents the discount factor.

[0039] Given the current state And the actions currently taken by the intelligent agent. Then the next state The probability of occurrence can be derived from the state transition probability matrix. express (29) Based on the current state and actions Corresponding reward function The state transition probability matrix can be obtained as follows: .

[0040] Adding a reward function to the Markov process and discount factor This allows us to obtain the Markov reward process, specifically a certain state. Rewards This refers to the expected reward obtained upon transitioning to that state. (30) in, Expected reward based on conditions. The reward obtainable from a state is persistent; the sum of the decay of all rewards is called the reward. This represents the weighted sum of the current immediate reward and all persistent rewards. The rate of return for a given state decreases as you move further into the future; this is known as the discount factor. The smaller, the more (31) The state value function measures the state value of a function. The expected reward that can be obtained by following a certain strategy. It measures the quality of a state from a global perspective, from the state... Depart, according to strategy The long-term benefit of taking an action is essentially a probabilistic-weighted average of the values ​​of all possible actions in the current state. (State value function) The representation is as follows (32) in, Indicates cumulative return Expectations Indicates will Split into instant rewards and future discount returns , Represent the value of the next state. Alternative This forms a recursive relationship.

[0041] The action-value function is the expected cumulative reward after performing a specific action in a given state; it directly relates to the value of the state-action pair. By comparing different actions... The action value function allows the agent to dynamically select the optimal action, driving the strategy to iterate towards higher returns. The representation is as follows (33) in, Indicates the value of an action as derived from The expected cumulative return upon departure Break it down into immediate rewards for the current action. and future discount returns , Represent the value of the next state-action pair. Alternative This forms a recursive process.

[0042] Step S3: Combine the attitude-track integrated model with deep reinforcement learning to design a control strategy based on the PPO algorithm. Using a deep reinforcement learning framework, the policy network and value network of the attitude-track integrated model are constructed through an Actor-Critic structure to generate control commands for trajectory thrust and attitude torque, optimizing the control strategy in a high-dimensional action space.

[0043] This invention further proposes a reward function with a multi-scale design, specifically including pose error reward, fuel consumption reward, terminal pose reward, and time reward. The PPO algorithm uses a shearing objective function for policy optimization, introducing a policy probability ratio and a dominance function to improve training stability. The dominance function is calculated using the generalized dominance estimation (GAE) method. The control strategy employs a Gaussian strategy, generating continuous control commands to optimize both orbital thrust and attitude torque.

[0044] The implementation example designs a deep reinforcement learning architecture, using an Actor-Critic structure to construct the policy network and value network, and using a Gaussian policy to generate control commands for orbital thrust and attitude torque.

[0045] This invention proposes a multi-agent game theory architecture, comprising two main agents: a tracking spacecraft and a target spacecraft. Each agent consists of an independent policy network and a value network. The policy network outputs actions, while the value network evaluates the long-term expected reward in the current state. The entire training framework employs a dual-agent PPO training method, with the training logic as follows: Figure 2 .

[0046] The entire training process begins with initializing environmental parameters, setting both training and simulation steps to zero. Then, in the main loop, the AI ​​receives the current state from the environment, generates control inputs through the Actor network, updates the environmental state, and stores this interaction data (state / input / reward) in the experience pool. Two loop conditions are set in the process: one is a simulation step loop, which continues looping when the simulation step count j is less than the set maximum value Jmax; the other is a training step count loop, which determines whether to continue training based on conditions until the termination condition (reaching the corresponding maximum value Imax) is met, ending the training. The system adopts an Actor-Critic structure, constructing a policy network and a value network, and using a Gaussian policy to generate control commands for orbital thrust and attitude torque.

[0047] The core objective of reinforcement learning is to optimize strategies. Maximize cumulative rewards Policy gradient methods improve decision-making ability by directly optimizing the policy, but excessively large updates can lead to instability. To address this issue, the PPO algorithm introduces a mechanism to limit the update magnitude, thereby enhancing training stability while ensuring policy optimization.

[0048] Exploration strategies for continuous action spaces typically employ Gaussian strategies. A Gaussian strategy for an N-dimensional action can be represented as follows: (34) In the formula Indicates the state given s Next, take action a The probability is composed of the product of the Gaussian distributions of each action component. Indicates the state s Below, in the action space, the first i The mean of each action component, Indicates the state s Below, in the action space, the first i The variance of each action component. and It is a parameter vector. Action components. It can be represented as (35) In the formula, , The noise variable is randomly sampled from the standard normal distribution. It follows a standard normal distribution.

[0049] The goal of the PPO algorithm is to optimize the following expectations. (36) in, The objective function to be maximized is... In order to be in the current strategy Under these conditions, the expected value of the cumulative return R.

[0050] The PPO algorithm incorporates probability ratios during the optimization process. Used to measure the differences between the old and new strategies: (37) in These are the parameters of the current strategy. Is the new strategy in the state? Select action The probability, Is the old strategy in the state? Select action The probability of.

[0051] To limit the scope of policy updates, PPO introduces a clipping objective function. The core of the PPO algorithm is to update the policy by optimizing this clipping objective function.

[0052] (38) in Indicates time step Expectations In time step The estimated advantage function value, It is a hyperparameter. It is a clipping function used to... The value is limited to Within the range; Used to compare the advantage weights of the original policy update Update weights with the pruned strategy.

[0053] The PPO algorithm uses a dominance function to measure the merit of taking a certain action in a given state. The formula for the dominance function is: (39) in It is a discount factor. It is the attenuation coefficient of the Generalized Advantage Estimation (GAE). Reward step size for the future. It is a time step The TD (Temporal Difference) error is calculated using the following formula: (40) in, For instant rewards, For the state value function, The value function for the next state.

[0054] The PPO algorithm updates the value function while optimizing the strategy. The value function is used to estimate the value of each state. The expression for the value function is: (41) in It is a value function prediction of the current state. It is the objective value function. The expression for a value function, Let be the expected return at time step t. The objective value function is calculated using the current policy and the value function, representing the expected return of the current state. The formula for the objective value function is: (42) Where T is the time step at the end of the round, and t is the current time step. For the first i Instant rewards for each step This represents the value of the target state value function. For the objective value function, This represents the final state.

[0055] To encourage strategy exploration, an entropy regularization term is introduced.

[0056] (43) in It is the entropy of the strategy.

[0057] PPO combines policy loss, value function loss, and entropy regularization term to form the total loss function.

[0058] (44) in, These are the weighting coefficients for the value function loss. These are the weighting coefficients for entropy regularization. Based on the PPO algorithm, this invention proposes a reward function for tracking spacecraft, which consists of four parts: (1) pose error reward; (2) fuel consumption reward; (3) terminal pose reward; and (4) time reward.

[0059] The tracking spacecraft is considered to have successfully caught up with the target spacecraft when the attitude relationship between the two spacecraft meets the following conditions. (45) in This represents the attitude error between the tracking spacecraft and the target spacecraft. To track the critical range of the attitude of the spacecraft and the target spacecraft, This indicates the positional error between the tracking spacecraft and the target spacecraft. To track the critical range between the position of the spacecraft and the target spacecraft.

[0060] To reward pose error and guide the spacecraft as close to the target pose as possible, the pose error penalty mechanism provides a negative reward based on the difference between the current pose and the target pose. The expression for the pose reward function is as follows: (46) in, For positional reward, This is the attitude error reward coefficient. This is the position error reward coefficient.

[0061] To optimize spacecraft fuel usage, a reward mechanism was designed to reduce excessive fuel consumption, providing a negative reward for fuel usage. The expression for the fuel reward function is as follows: (47) in, As a fuel reward, For fuel incentive coefficient, To track spacecraft control inputs.

[0062] The terminal pose reward refers to the degree of closeness between the pose achieved by the tracking spacecraft at the end of the mission and the pose of the target spacecraft. If the pose errors of both the tracking spacecraft and the target spacecraft are within a given range, a large positive reward is given to the tracking spacecraft. Specifically, the terminal pose reward is triggered when both the attitude error and position error between the tracking and target spacecraft are less than a predetermined threshold. The expression for the terminal pose reward is as follows: (48) in, For terminal pose reward, This represents the terminal pose reward coefficient.

[0063] The time reward is related to the speed at which the mission is completed, i.e., the time required for the spacecraft to reach its objective. Each step provides a negative reward, incentivizing the spacecraft to complete the mission faster. Time Reward The expression is (49) In the formula, This is the time reward coefficient. The number of time steps used. For time step, n The number of steps required for the tracking spacecraft to catch up with the target spacecraft; if it fails to catch up, then... n This is the maximum step size set.

[0064] The total reward function for tracking spacecraft is (50) In the reward function, All are positive real numbers.

[0065] Based on the above embodiments, in order to verify the effectiveness of the attitude-track integrated game control method based on the PPO algorithm established in this invention, the algorithm was trained and visualized using the Python platform after setting the initial pose and maneuver constraints.

[0066] (1) Parameter settings By introducing a multi-scale reward mechanism, the reward function considers pose error reward, fuel consumption reward, terminal pose reward, and time reward. The expression of the reward function is as follows: (51) Assuming the attitude-orbit integrated game occurs near geosynchronous orbit, considering time constraints, thrust and thrust moment constraints, and the conditions for successful catch-up. Simulation constraints are shown in Table 1.

[0067]

[0068] The initial states between the two spacecraft are shown in Table 2. The initial state is relative to the reference coordinate system, including the initial position, initial rotation matrix, initial velocity, and initial angular velocity.

[0069]

[0070] Maximum number of training steps The maximum number of simulation steps is The training parameters of the PPO algorithm are shown in Table 3. The network structure uses a two-layer fully connected neural network, with 256 neurons in the first hidden layer and 128 neurons in the second hidden layer, and the activation function is ReLU. During training, 2048 samples are sampled from the experience pool each time. The PPO algorithm divides these samples into multiple mini-batches, each containing 64 samples. Mini-batch training can effectively reduce the variance during training, allowing the model to be updated more stably in each training step.

[0071]

[0072] (2) Results Analysis A deep neural network is used to approximate the policy network and value network. TensorFlow is used to train the two agents in the environment, normalization is used to handle the states, and Matplotlib is used to visualize the training rewards. The training results are as follows: Figure 3 .

[0073] In the training results graph, the blue line represents the actual reward for each episode, and the orange line represents the average reward. It can be seen that in the early stages of training, the agent performed poorly, with reward values ​​as low as -5 × 10⁻⁵. 7 Around 100 episodes. After that, the learning curve plateaus, with an average reward of -1 × 10⁻⁶. 7 The algorithm has essentially converged after fluctuating between positive and negative values. The rapid increase in the initial stage reflects the algorithm's effective capture of environmental dynamics and rapid policy adjustments. The continuous fluctuations during the plateau phase indicate that the agent is constantly trying new policies to find possible improvements while maintaining the learned effective behaviors. The final convergence to a stable negative reward value indicates that the agent has found a near-optimal solution under the current environment and network architecture, demonstrating that the PPO algorithm has learned an effective policy. Although there is still room for improvement, it can now stably execute relatively efficient policies.

[0074] Simultaneously training both the tracking spacecraft and the target spacecraft, and analyzing their attitude and position errors as key indicators to evaluate the control effectiveness and performance during the mission. The attitude and position errors of both spacecraft are as follows: Figures 4-7 Through training, the tracking spacecraft engages in a game of strategy with the target spacecraft in the environment. The tracking spacecraft gradually reduces these errors, thereby approaching the target spacecraft and ultimately achieving a successful interception.

[0075] As can be seen from the attitude error, the angle errors in all three directions show a gradually decreasing trend, indicating that the agent adjusted the spacecraft's attitude during the training process, which gradually reduced the attitude error and reached the set attitude error threshold of 5° at 595 seconds.

[0076] As can be seen from the position error curve, the position errors in all three directions gradually decreased over time. The relative distance curve shows that, through training, the relative distance between the two spacecraft continuously shortened, eventually approaching the set threshold of 50 meters, reaching 27.9 meters at 595 seconds, successfully intercepting the target. In reinforcement learning, the agent gradually learns the optimal strategy through extensive training. Because the task's reward function tends to reduce distance, the agent chose the simplest and most direct path to minimize distance, hence the decreasing trend in the relative distance between the two spacecraft.

[0077] The trajectory maneuvers of both spacecraft are as follows Figure 8 As can be seen from the trajectories of the two spacecraft, over time, the tracking spacecraft maneuvered and adjusted its trajectory, eventually approaching the target spacecraft and successfully intercepting it.

[0078] Simulation results show that, after training, the tracking spacecraft can successfully intercept the target spacecraft. Through 2000 training steps, the tracking spacecraft gradually optimized its trajectory in multiple simulations, successfully shortening the distance to the target spacecraft and precisely adjusting its attitude and velocity. During training, the agent continuously learns how to adjust its speed, direction, and attitude angle to achieve the optimal interception path.

[0079] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0080] The following embodiments describe the electronic device provided by the present invention. The electronic device described below can be referred to in correspondence with the spacecraft attitude and orbit integrated game control method based on PPO described above.

[0081] The electronic device may include a processor, a communications interface, memory, and a communication bus. The processor, communications interface, and memory communicate with each other via the communication bus. The processor can call logical instructions from the memory to execute a PPO-based spacecraft attitude and orbit integrated game control method, which mainly includes the software processing portion described above.

[0082] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the software processing part of the spacecraft attitude and orbit integrated game control method based on PPO provided by the above methods.

[0084] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the software processing portion of the PPO-based spacecraft attitude and orbit integrated game control method provided by the above methods.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spacecraft attitude and orbit integrated game-theoretic control method based on PPO, characterized in that, Includes the following processes, An integrated attitude and orbit model for spacecraft is established, and the interaction process between the tracking spacecraft and the target spacecraft is modeled as a non-cooperative zero-sum game. The integrated attitude and orbit model is based on the six-degree-of-freedom relative kinematics and dynamics coupling model of spacecraft established by the Lie group SE(3), and the attitude control and orbit control are modeled in a unified manner. The integrated kinematic equations and integrated dynamic model of the attitude and orbit of the spacecraft are established. The non-cooperative zero-sum game modeling includes defining the cost function of the tracking spacecraft and the cost function of the target spacecraft. Both parties optimize their strategies by minimizing their respective cost functions, and finally form a Nash equilibrium strategy. A multi-agent deep reinforcement learning framework is set up, in which the tracking spacecraft and the target spacecraft are configured with independent policy networks and value networks, respectively. Training is performed using a dual-agent PPO approach. The policy network generates control commands for orbital thrust and attitude torque, while the value network evaluates state values. The training process employs a multi-scale reward function, including pose error reward, fuel consumption reward, terminal pose reward, and time reward. This indicates the attitude error between the tracking spacecraft and the target spacecraft. To track the critical range of the attitude of the spacecraft and the target spacecraft, This indicates the positional error between the tracking spacecraft and the target spacecraft. To track the critical range between the positions of the spacecraft and the target spacecraft; Pose error reward: in, For positional reward, This is the attitude error reward coefficient. This is the reward coefficient for position error; Fuel Consumption Rewards: in, As a fuel reward, For fuel incentive coefficient, To track spacecraft control inputs; Terminal pose reward: in, For terminal pose reward, This represents the terminal pose reward coefficient. Time reward: in, This is the time reward coefficient. The number of time steps used. For time step, n The number of steps required for the tracking spacecraft to catch up with the target spacecraft; if it fails to catch up, then... n The maximum step size is set; the spacecraft behavior is controlled and guided according to the trained strategy.

2. The spacecraft attitude and orbit integrated game-theoretic control method based on PPO according to claim 1, characterized in that: The policy network uses a Gaussian policy to generate control instructions for a continuous action space.

3. The spacecraft attitude and orbit integrated game-theoretic control method based on PPO according to claim 1, characterized in that: The PPO is a near-end policy optimization algorithm that uses a probability ratio shearing mechanism to limit the policy update magnitude.

4. The spacecraft attitude and orbit integrated game control method based on PPO according to claim 3, characterized in that: The near-end strategy optimization algorithm uses the generalized advantage estimation method to calculate the advantage function.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the spacecraft attitude and orbit integrated game control method as described in any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the spacecraft attitude and orbit integrated game control method as described in any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the spacecraft attitude and orbit integrated game control method as described in any one of claims 1 to 4.

Images