Spacecraft pursuit control method based on IPPO algorithm
Through the deep reinforcement learning method based on IPPO algorithm, a spacecraft chase control model is built, which solves the problems of complex calculations and poor adaptability of traditional control methods, and achieves more efficient and reliable spacecraft chase control, adapting to complex environments and autonomous motion goals.
Patent Information
- Application Number
- CN202411133950.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The traditional spacecraft pursuit control method is complex in calculations, difficult to adapt to complex environments and autonomous moving targets, and difficult to effectively control in unknown environments, and does not consider the performance limitations of spacecraft onboard computers.
Using deep reinforcement learning method based on IPPO algorithm, the spacecraft chase control model is constructed, the control problem is described as Markov's decision-making process, through multi-layer optimization strategy network training, and the reward function is used to update the neural network to achieve the gradual step of the spacecraft and the target. Game training, output control instructions.
It improves the spacecraft control accuracy and reliability, overcomes the problem of low training efficiency caused by large state space, enhances the spacecraft's autonomous control capabilities and generalization performance, and adapts to complex environments and autonomous motion goals.
Smart Images

Figure CN118655904B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to multi-spacecraft control allocation, and in particular to a spacecraft pursuit control method based on an IPPO algorithm. Background Art
[0002] When completing close approach to a target, modern spacecraft mostly adopts a motion model based on the CW equation. The CW equation is a continuous differential equation that can describe the continuous change of the spacecraft's motion over time. This enables the CW equation to provide real-time updates of the spacecraft's state, which is helpful for real-time navigation, control, and mission planning. According to the analytical solution form of the CW equation, reasonable assumptions are made within an acceptable range to ignore the extremely small variables in the corresponding scenario, making the mathematical form of spacecraft control intuitive, clear, and logical, simplifying the complex equations of spacecraft motion control and reducing the amount of calculation, while enhancing the control and feedback experience of ground staff on the spacecraft. The pursuit, tracking, and approach of spacecraft usually adopt the multi-pulse rendezvous method and the parallel approach method. These methods use the analytical form of the spacecraft motion model to perform quantitative calculations when the initial state is known to obtain the size of the corresponding control quantity.
[0003] However, these traditional control technologies require large amounts of computation, are complex to solve, have poor adaptability, require the initial state to be known, and the target does not have the ability to avoid autonomously. It is difficult to track targets with certain motion capabilities and difficult to play games in unknown environments. The performance limitations of spacecraft onboard computers are not taken into account. Traditional control systems usually rely on mathematical models and rules, which may be difficult to capture all changes and situations of complex systems. Artificial intelligence technology can enable control systems to better adapt to various environments and changes through learning and adaptation. Many modern systems have nonlinear and complex characteristics, and traditional control methods are difficult to accurately model and control them. Artificial intelligence technologies such as deep learning and reinforcement learning can solve these problems to some extent and achieve precise control of complex systems. At the same time, artificial intelligence technology is also widely used in the field of control for real-time decision-making and optimization problems. In short, the application of artificial intelligence technology is a necessary trend in the design and manufacture of modern spacecraft, which can provide more stable and reliable guarantees for the performance, safety and autonomous control capabilities of spacecraft. However, as the number of spacecraft increases and the target has a certain avoidance capability, how to apply reasonable control instructions to spacecraft has become a new problem that needs to be solved. Summary of the invention
[0004] Purpose of the invention: In view of the above shortcomings, the present invention provides a spacecraft pursuit control method based on the IPPO algorithm with better control effect.
[0005] Technical solution: To solve the above problems, the present invention adopts a spacecraft pursuit control method based on the IPPO algorithm, which includes the following steps:
[0006] (1) Establish a spacecraft control model based on deep reinforcement learning and describe the problem of controlling the spacecraft to pursue the target as a Markov decision process;
[0007] (2) According to the IPPO algorithm, a multi-spacecraft layer-by-layer optimization strategy network is built and trained; after the spacecraft interacts with the environment, it obtains rewards for each step through the reward function, and the neural network uses these rewards to perform corresponding updates through the IPPO algorithm; specifically, it includes:
[0008] (2.1) Based on the initial target state, the PPO algorithm is used to train a single spacecraft neural network model in a set environment;
[0009] (2.2) Based on the trained single spacecraft neural network model, the PPO algorithm is used to train the target neural network model in the set environment;
[0010] (2.3) Based on the trained single spacecraft neural network model and target neural network model, the PPO algorithm is used to perform game training on the single spacecraft and the target in the set environment;
[0011] (2.4) Based on the single spacecraft neural network model and the target neural network model trained in step (2.3), multiple spacecraft are trained in a set environment based on the IPPO algorithm;
[0012] (3) Multiple spacecraft use the trained strategy network to output control instructions based on their own observation information of the environment, thereby achieving multi-spacecraft pursuit target control.
[0013] Furthermore, the spacecraft control model based on deep reinforcement learning in step (1) is:
[0014] ;
[0015] in, is the current state of the spacecraft, Spacecraft takes action After the update status, Update the matrix for the state; action is the spacecraft pulse control quantity;
[0016] The expression of the state update matrix is:
[0017] ;
[0018] in, is the reference orbital angular velocity; is the simulation step size.
[0019] Furthermore, the problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the reward function of the spacecraft constructed includes: terminal reward, distance reward, and fuel reward;
[0020] The terminal reward of the spacecraft is: if the spacecraft approaches the target within the preset range, it is considered that the mission is completed and a positive numerical reward is given to the spacecraft. If the spacecraft reaches the maximum number of simulation steps but has not approached the target within the preset range, the terminal reward is 0.
[0021] The distance reward of the spacecraft is: a negative reward related to the distance between the spacecraft and the target. The smaller the distance between the spacecraft and the target, the smaller the absolute value of the distance reward.
[0022] The fuel reward of a spacecraft is: a negative function related to the absolute value of the spacecraft's pulse control quantity.
[0023] Furthermore, the problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the reward function of the constructed target includes: terminal reward, distance reward, and fuel reward;
[0024] The terminal reward of the target is: if the spacecraft reaches the maximum number of simulation steps but has not approached the target to within the preset range, the target is considered to have completed the avoidance task and a positive numerical reward is given to the target; if the spacecraft approaches the target to within the preset range, a negative numerical reward is given to the target;
[0025] The distance reward of the target is: a positive reward related to the distance between the target and the spacecraft. The greater the distance between the target and the spacecraft, the greater the absolute value of the distance reward.
[0026] The target fuel reward is a negative function related to the absolute value of the target pulse control amount.
[0027] Furthermore, the problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the state space constructed is the relative distance between the spacecraft and the target in the coordinate system and the relative speed between the spacecraft and the target, which is expressed as:
[0028] ;
[0029] in, for The relative distance between the spacecraft and the target on the coordinate axis, for The relative distance between the spacecraft and the target on the coordinate axis, for The relative distance between the spacecraft and the target on the coordinate axis; for The relative speed between the spacecraft and the target on the coordinate axis, for The relative speed between the spacecraft and the target on the coordinate axis, for The relative speed between the spacecraft and the target on the coordinate axis.
[0030] Furthermore, the problem of controlling a spacecraft to chase a target is described as the action space constructed in the Markov decision process. is the pulse control quantity of the three axes of the spacecraft in the coordinate system, and its expression is:
[0031] ;
[0032] in, For spacecraft The pulse control amount on the coordinate axis, For spacecraft The pulse control amount on the coordinate axis, For spacecraft Pulse control quantity on the coordinate axis.
[0033] Furthermore, the problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the constructed policy network actor is set to a three-layer neural network with an input layer structure of [6,128]; a hidden layer of [128,128]; and an output layer of [128,3]. The policy network actor input is the six-dimensional state of the spacecraft, and the output is a three-dimensional pulse control quantity.
[0034] Furthermore, the problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the constructed state value network critic is set to a three-layer neural network with an input layer structure of [6,128]; a hidden layer of [128,128]; and an output layer of [128,1]. The state value network critic input is the six-dimensional state of the spacecraft, and the output is the value evaluation of the state.
[0035] Furthermore, the activation function used in describing the problem of controlling the spacecraft to chase the target as a Markov decision process is the tanh function.
[0036] Beneficial effects: Compared with the prior art, the significant advantage of the present invention is that it overcomes the shortcomings of traditional control calculation complexity and difficulty in chasing targets with autonomous motion capabilities through neural network intelligent control. Through layer-by-layer optimization training, the convergence speed of the algorithm is improved, and the problem of low training efficiency caused by the excessive state space when the spacecraft is moving is overcome. Taking into account the autonomous motion of the target and the control amount constraints of the spacecraft, the difficulty of training is increased from a simple scenario, which improves the performance and generalization ability of the spacecraft neural network, improves the control accuracy of the spacecraft, and has higher reliability. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flow chart of the spacecraft pursuit control method of the present invention. DETAILED DESCRIPTION
[0038] like Figure 1 As shown, a spacecraft pursuit control method based on the IPPO algorithm in this embodiment first converts the controlled spacecraft system into a control model that conforms to Markov decision-making, and constructs the state space, action space, reward function, strategy network actor, state value network critic and reward function of the spacecraft pursuit game based on deep reinforcement learning, thereby establishing a spacecraft game environment; then training starts with a single spacecraft neural network, and then related target intelligent agents are designed based on the trained single spacecraft; then a game is conducted to improve the performance of the spacecraft and target neural networks, and finally, layer-by-layer optimization training of multiple spacecraft and targets is performed based on the IPPO algorithm, thereby completing the multi-spacecraft layer-by-layer optimization game strategy based on the IPPO algorithm.
[0039] The specific steps include:
[0040] Step 1: Convert the scenario of a spacecraft chasing a target into a Markov decision process and establish a spacecraft control model based on deep reinforcement learning.
[0041] In order to describe the motion of spacecraft and targets in space, it is necessary to establish a corresponding coordinate system and a relative coordinate system for description, with a virtual reference to the center of mass of the spacecraft. O The OX axis points outward along the radius of the earth, the OY axis is perpendicular to the OX axis and points to the front of the spacecraft. The OZ axis is determined by the right-hand rule, thus establishing a reference satellite moving coordinate system. The relative motion between the virtual reference spacecraft and the spacecraft can be expressed by the following formula in this coordinate system:
[0042] ;
[0043] in, , , is the three-axis coordinate size of the spacecraft in the previously specified coordinate system, is the force exerted on the spacecraft by different axes in the corresponding coordinate system, is the Earth's gravitational constant, is the orbit radius of the virtual reference satellite, is the reference satellite orbital angular velocity.
[0044] Model linearization is a method for approximating a nonlinear system to a linear system within a certain range. This approximation helps to simplify the complexity of system analysis and control design. The following assumptions are made in this embodiment:
[0045] The Earth and the spacecraft are considered as two mutually attractive mass points;
[0046] The Earth is a spherical body with uniform mass, and its gravitational force on the spacecraft is directed toward the center of the Earth. No other perturbations are considered.
[0047] The orbital eccentricity of the virtual reference satellite and the corresponding spacecraft is 0 or very small, which can be regarded as a circular orbit;
[0048] The distance between the spacecraft and the reference satellite of the virtual spacecraft is much smaller than the orbital radius of its operation.
[0049] Under the above assumptions, since the spacecraft is usually propelled by pulses, the external force is not considered, and the motion equation of the spacecraft relative to the virtual spacecraft is linearized and simplified. The simplified equation is as follows:
[0050] ;
[0051] This equation is called the CW equation; take the initial time , let the initial state be:
[0052] ;
[0053] ;
[0054] This equation can be solved to its analytical form:
[0055] ;
[0056] According to the analytical equation, the coordinates of the spacecraft relative to the reference virtual spacecraft in the set coordinate system can be obtained, and the control input of the spacecraft is set to pulse control, that is:
[0057] ;
[0058] It is the three-axis pulse-controlled velocity vector of the spacecraft in the coordinate system. This set of variables directly affects the speed of the spacecraft, and the motion control of the spacecraft is completed by controlling the speed.
[0059] The state of the spacecraft in the set coordinate system is , is a 6*1 dimensional vector, which includes the three-axis coordinates and velocity of the spacecraft.
[0060] The corresponding specific target state is , is a 6*1 dimensional vector, which includes the three-axis coordinates and speed of a specific target.
[0061] According to the above formula, the state update matrix of the relevant agent can be obtained, and the expression is as follows:
[0062] ;
[0063] in, yes The dimensional control input is the spacecraft pulse control quantity; yes dimensional state variables; is the transfer matrix related to the initial state of the spacecraft; Represents the simulation step length. This matrix is used to update the position and degree of the spacecraft in each time step. is the reference orbital angular velocity; After the state transfer dimensional state variables. In summary, the spacecraft model is: .
[0064] According to the above formula, the spacecraft chasing game scenario can be converted into a Markov decision process. The spacecraft is an intelligent agent and its state is , take actions according to the designed strategy Afterwards, you can Perform state conversion to obtain corresponding reward returns.
[0065] Step 2: Design the state space, action space, reward function, policy network actor, state value network critic and reward function of the spacecraft pursuit game based on deep reinforcement learning.
[0066] Reinforcement learning is a method of learning through the interaction between an agent and its environment. The key elements include state, action, strategy, and reward. It does not require a large amount of input data, but focuses on optimizing behavior through trial and error, guiding the agent to learn to achieve its goals based on the rewards obtained. Deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and is particularly suitable for tasks in complex environments where the state and action spaces are large or continuous. The Actor-Critic algorithm combines the advantages of strategy and value functions, improves the processing efficiency of action space continuity and strategy randomness, and updates the network by selecting actions by actors and evaluating the value of actions by critics to achieve the optimal strategy.
[0067] For a spacecraft, its equivalent state can be defined as the relative distance between the spacecraft and the target in the coordinate system and the relative speed between the spacecraft and the target, which can be expressed as:
[0068] ;
[0069] Among them, all relative states constitute the state space of this scenario, where represents the relative distance between the spacecraft and the target on the corresponding coordinate axis, Represents the relative speed between the spacecraft and the target on the corresponding coordinate axis.
[0070] For a spacecraft, its motion can be defined as the pulse control quantity of the spacecraft in the three axes of the coordinate system, which is expressed as All actions constitute the action space in this scenario.
[0071] Since the state space of the spacecraft pursuit game environment is large, in order to avoid sparse rewards causing difficulty in training convergence, the reward function needs to be designed accordingly.
[0072] The spacecraft reward function is set to consist of three parts: terminal reward, distance reward, and fuel reward.
[0073] Among them, the terminal reward is when the spacecraft approaches the target within a certain range. At this time, it is considered that the mission is completed and a large positive reward is given to the spacecraft. If the spacecraft reaches the maximum number of simulation steps but is not close to the target within a certain range, this part of the reward function is 0, and its expression is:
[0074] ;
[0075] in, For spacecraft terminal rewards, is the distance between the spacecraft and the target, is the number of simulation steps in this round, The maximum number of steps set for a round, in this embodiment, the distance range for judging proximity to the target is set to 500, and the reward value is 300.
[0076] The distance reward is a negative reward related to the distance between the spacecraft and the target. The smaller the distance between the spacecraft and the target, the smaller the absolute value of this part of the reward. This part of the reward function is used to guide the spacecraft to approach the target. Its expression is:
[0077] ;
[0078] in, Reward for spacecraft distance.
[0079] The fuel reward is set as a negative function related to the absolute value of the pulse control amount. This part of the reward is used to guide the spacecraft to reduce fuel consumption while completing the mission. The reward function expression is:
[0080] ;
[0081] in, Bonus for spacecraft fuel.
[0082] The total reward function of the spacecraft is expressed as:
[0083] ;
[0084] in, , , is the correlation coefficient.
[0085] The reward function of the target is set similarly to the reward function of the spacecraft, which consists of three parts: target terminal reward, target distance reward, and target fuel reward.
[0086] The target terminal reward is when the spacecraft reaches the maximum number of simulation steps and the target is away from the spacecraft by a certain range. At this time, the target is considered to have completed the avoidance and a positive large value reward is given to the target. If the spacecraft is close to the target by a certain range, this part of the reward function is a negative large value, and its expression is:
[0087] ;
[0088] in, For the target terminal reward, is the distance between the spacecraft and the target, is the number of simulation steps in this round, Set the maximum number of steps for a round.
[0089] The target distance reward is a positive reward related to the distance between the target and the spacecraft. The greater the distance between the spacecraft and the target, the greater the reward. This part of the reward function is used to guide the target away from the spacecraft. Its expression is:
[0090] ;
[0091] in, Reward for target distance.
[0092] The target fuel reward is set as a negative function related to the absolute value of the pulse control amount. This part of the reward is used to guide the target to reduce fuel consumption while completing the task. The reward function expression is:
[0093] ;
[0094] in, Fuel rewards for goals.
[0095] The target total reward function expression is:
[0096] ;
[0097] in, , , is the correlation coefficient.
[0098] The policy network actor is set as a three-layer neural network, where the input layer structure is [6,128]; the hidden layer is [128,128]; and the output layer is [128,3]. That is, the policy network actor input is the six-dimensional state of the spacecraft, and the output is the three-dimensional impulse control quantity.
[0099] The state value network critic is set as a three-layer neural network, where the input layer structure is [6,128]; the hidden layer is [128,128]; and the output layer is [128,1]. That is, the state value network critic input is the six-dimensional state of the spacecraft, and the output is the value evaluation of the state.
[0100] The activation functions used in this section are all tanh functions.
[0101] Step 3: Multi-spacecraft layer-by-layer optimization game strategy training based on IPPO algorithm:
[0102] Since the spacecraft game environment has the characteristics of large state space, weak agent movement ability, and sparse rewards, it is necessary to adopt layer-by-layer optimization for training. The main reason is that layer-by-layer optimization can enable the agent to start training from a simple environment, from easy to difficult, which can effectively improve the training efficiency and the performance and generalization ability of the agent in complex environments. Through staged training and optimization, the agent can gradually adapt and optimize its behavior strategy to cope with different scenarios. This method not only helps to improve the stability of the agent during training, but also enhances its generalization ability in unknown environments, ensuring that it can make effective decisions and actions in a variety of situations. At the same time, layer-by-layer optimization also helps to optimize the learning algorithm used, and by gradually adjusting and optimizing parameters, the efficiency and performance of the algorithm in practical applications can be improved.
[0103] The PPO algorithm belongs to the reinforcement learning algorithm. It has the advantages of easy implementation, high computational efficiency, stable update, and wide application range in reinforcement learning training. The PPO algorithm belongs to a single-agent algorithm. According to the PPO algorithm, the corresponding IPPO algorithm for multi-agents can be expanded. The IPPO algorithm belongs to a decentralized algorithm. In this embodiment, each spacecraft and target are individually assigned a neural network based on the actor-critic framework, and each neural network is independently trained by the IPPO algorithm. The IPPO algorithm allows each agent to learn its strategy independently without the information of other agents. Each agent updates its own policy parameters based on its own experience. This independence makes the algorithm more scalable and can be applied to large-scale multi-agent systems. The core of the IPPO algorithm is the proximal policy optimization PPO method. PPO improves the stability of the algorithm by introducing proximal constraints in the optimization process. In IPPO, each agent uses the PPO algorithm to update the policy parameters based on its own experience without the need for global information or communication with other agents. Through the IPPO algorithm, each agent can locally optimize its own strategy while the multi-agent system can achieve collaborative learning.
[0104] After the spacecraft agent interacts with the environment, it obtains rewards for each step through the corresponding reward function, and the neural network uses these rewards to perform corresponding updates through the IPPO algorithm;
[0105] ;
[0106] in, Represents the policy loss function of the PPO algorithm; The parameters of the policy network represented by Represents the time step t Expected operation on represents the advantage function; represents the hyperparameter used to clip the probability ratio.
[0107] ;
[0108] in, Indicates that at time step t The probability ratio between the new strategy and the old strategy; Indicates in status New strategy selection action probability; Indicates in status Select action for the old strategy probability.
[0109] ;
[0110] in, Indicates that at time step t Discounted rewards for Monte Carlo estimation on ; Indicates that at time step t Previous status The state value function of .
[0111] The research object used in this example is a spacecraft operating at an orbital altitude of 18982km and an average angular velocity of the orbit. The control cycle of each step is 10 seconds, the maximum number of simulation steps is 300 steps, the number of spacecraft is 2, the spacecraft single pulse control range, and the target single pulse control range.
[0112] The initial state of the spacecraft is as follows: , , .
[0113] The initial state of the target is as follows: , , , , , .
[0114] To improve the generalization performance of the neural network, the initial value of the target is randomly selected within the range shown.
[0115] Optimize the game strategy training process layer by layer:
[0116] Spacecraft initialization training: First, create a single spacecraft in the environment where the spacecraft chases the target. The spacecraft is an intelligent agent controlled by the neural network. In addition, create a target, which does not have autonomous motion capabilities in this step. The spacecraft and the target are randomly generated within a certain range. The initial state speed of the spacecraft is 0, and the target speed is randomly initialized. The PPO algorithm is used for training in the set environment. After the training is completed, the spacecraft neural network model is saved. This stage trains 3 million steps. Through this step, a spacecraft intelligent agent with a certain degree of intelligence is initially trained.
[0117] Target training: The spacecraft loads the trained neural network model. In this step, the target has a certain self-motion ability. The spacecraft and the target are randomly generated within a certain range. The initial state speed of the spacecraft is 0, and the target speed is randomly initialized. Initialize the target state and its neural network, and use the PPO algorithm to train in the set environment. After the training is completed, save the target neural network model. This stage trains 3 million steps. The spacecraft is not trained in this step. Through this step, a target intelligent agent with a certain degree of intelligence is initially trained.
[0118] Spacecraft and target neural network model testing: Based on the previously trained neural network model, corresponding test verification games are carried out to improve the accuracy of the neural network output actions.
[0119] Game training between spacecraft and target: The spacecraft and target load the pre-saved neural network model respectively, and use the PPO algorithm to conduct game training in the set environment. After both sides train to the pre-set maximum simulation round, the model is saved. This stage trains 6 million steps. This step is used to improve the performance and generalization ability of the previously trained neural network.
[0120] Spacecraft and target neural network model testing: The spacecraft trained by the game and the spacecraft not trained by the game were tested with the target trained by the game respectively. This test verified that the game improved the performance and generalization ability of the spacecraft neural network.
[0121] Training of multiple spacecraft and targets: Spacecraft 1, spacecraft 2 and the target are loaded with pre-saved models respectively. Multiple spacecraft are trained in a set environment based on the IPPO algorithm. After training to the pre-set maximum number of simulation rounds, the neural network model of each spacecraft is saved for 6 million steps of training.
[0122] Multi-spacecraft neural network model test: Each spacecraft conducts corresponding target shooting tests based on the previously trained neural network model to verify that the multi-spacecraft layer-by-layer optimization game strategy of the IPPO algorithm can complete the tracking and approach of targets with certain avoidance capabilities.
[0123] Step 4: Multiple spacecraft use the trained strategy network to output control instructions based on their own observation information of the environment to achieve multi-spacecraft pursuit target control.
Claims
1. A spacecraft pursuit control method based on IPPO algorithm, characterized in that: The following steps are involved: (1) Establish a spacecraft control model based on deep reinforcement learning and describe the problem of controlling the spacecraft to pursue the target as a Markov decision process; The spacecraft control model based on deep reinforcement learning is: ; in, is the current state of the spacecraft, Spacecraft takes Updated status after the action, Update the matrix for the state; action is the spacecraft pulse control quantity; The expression of the state update matrix is: ; in, is the reference orbital angular velocity; is the simulation step length; The problem of controlling the spacecraft to chase the target is described as a Markov decision process. The state space constructed is the relative distance between the spacecraft and the target in the coordinate system and the relative speed between the spacecraft and the target. Its expression is: ; in, for The relative distance between the spacecraft and the target on the coordinate axis, for The relative distance between the spacecraft and the target on the coordinate axis, for The relative distance between the spacecraft and the target on the coordinate axis; for The relative speed between the spacecraft and the target on the coordinate axis, for The relative speed between the spacecraft and the target on the coordinate axis, for The relative speed between the spacecraft and the target on the coordinate axis; The problem of controlling a spacecraft to pursue a target is described as a Markov decision process, where the action space is constructed is the pulse control quantity of the three axes of the spacecraft in the coordinate system, and its expression is: ; in, For spacecraft The pulse control amount on the coordinate axis, For spacecraft The pulse control amount on the coordinate axis, For spacecraft Pulse control quantity on the coordinate axis; The problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the constructed policy network actor is set as a three-layer neural network, where the input layer structure is [6,128]; the hidden layer is [128,128]; the output layer is [128,3], the policy network actor input is the six-dimensional state of the spacecraft, and the output is the three-dimensional impulse control amount; The problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the constructed state value network critic is set as a three-layer neural network, where the input layer structure is [6,128]; the hidden layer is [128,128]; the output layer is [128,1], and the state value network critic input is the six-dimensional state of the spacecraft, and the output is the value evaluation of the state; (2) According to the IPPO algorithm, a multi-spacecraft layer-by-layer optimization strategy network is built and trained; after the spacecraft interacts with the environment, it obtains rewards for each step through the reward function, and the neural network uses these rewards to perform corresponding updates through the IPPO algorithm; specifically, it includes: (2.1) Based on the initial target state, the PPO algorithm is used to train a single spacecraft neural network model in a set environment; (2.2) Based on the trained single spacecraft neural network model, the PPO algorithm is used to train the target neural network model in the set environment; (2.3) Based on the trained single spacecraft neural network model and target neural network model, the PPO algorithm is used to perform game training on the single spacecraft and the target in the set environment; (2.4) Based on the single spacecraft neural network model and the target neural network model trained in step (2.3), multiple spacecraft are trained in a set environment based on the IPPO algorithm; (3) Multiple spacecraft use the trained strategy network to output control instructions based on their own observation information of the environment, thereby achieving multi-spacecraft pursuit target control.
2. The spacecraft pursuit control method according to claim 1, characterized in that: The problem of controlling a spacecraft to chase a target is described as a Markov decision process, and the reward function of the spacecraft constructed includes: terminal reward, distance reward, and fuel reward; The terminal reward of the spacecraft is: if the spacecraft approaches the target within the preset range, it is considered that the mission is completed and a positive numerical reward is given to the spacecraft. If the spacecraft reaches the maximum number of simulation steps but has not approached the target within the preset range, the terminal reward is 0. The distance reward of the spacecraft is: a negative reward related to the distance between the spacecraft and the target. The smaller the distance between the spacecraft and the target, the smaller the absolute value of the distance reward. The fuel reward of a spacecraft is: a negative function related to the absolute value of the spacecraft's pulse control quantity.
3. The spacecraft pursuit control method according to claim 2, characterized in that: The problem of controlling the spacecraft to chase the target is described as a Markov decision process, and the reward function of the target constructed includes: terminal reward, distance reward, and fuel reward; The terminal reward of the target is: if the spacecraft reaches the maximum number of simulation steps but has not approached the target to within the preset range, the target is considered to have completed the avoidance task and a positive numerical reward is given to the target; if the spacecraft approaches the target to within the preset range, a negative numerical reward is given to the target; The distance reward of the target is: a positive reward related to the distance between the target and the spacecraft. The greater the distance between the target and the spacecraft, the greater the absolute value of the distance reward. The target fuel reward is a negative function related to the absolute value of the target pulse control amount.
4. The spacecraft pursuit control method according to claim 3, characterized in that: The problem of controlling a spacecraft to chase a target is described as a Markov decision process, and the activation function used is the tanh function.
5. The spacecraft pursuit control method according to claim 3, characterized in that: Policy loss function of the PPO algorithm for: ; in, The parameters of the policy network represented by Represents the time step t Expected operation on Indicates that at time step t The probability ratio between the new strategy and the old strategy; represents the advantage function; represents the hyperparameter used to clip the probability ratio, Represents the truncation function.
