Spacecraft with active defense game system and method thereof

Abstract

The application discloses a spacecraft with an active defense game system and a method thereof, and the system comprises a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model; the active defense game training module is composed of a first full connection neural network and a second full connection neural network; wherein the motion control model is pulse orbit data of each spacecraft in tracking a star, defending the star and escaping the star in a LVLH coordinate system; the "pursuit-defense-escape" game model is used for constructing game targets of the tracking star, the defending star and the escaping star; the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to a multi-agent reinforcement learning algorithm; and the optimized "pursuit-defense-escape" game model outputs a defense game strategy online according to a three-party game scene of the spacecraft; and the application can realize an active defense strategy of a defense spacecraft, and effectively improves the safety performance of the spacecraft in orbit.

Description

TECHNICAL FIELD

[0001] The application relates to the field of spacecraft game, in particular to a spacecraft with an active defense game system and a method thereof. BACKGROUND

[0002] With the rapid development of global aerospace industry, the space environment is becoming increasingly complex, the number of on-orbit spacecraft is increasing exponentially, the capability is greatly improved, and the intelligent level of spacecraft of various countries is also continuously improved. On the basis of traditional guidance, navigation and control technology, intelligent perception, intelligent decision and autonomous control are gradually developed. Meanwhile, a large number of on-orbit spacecraft cannot cope with real-time and high-precision tasks by relying on ground control mode. In addition, the current control system mainly faces deterministic task scenarios, and the intelligent level is weak, and lacks autonomous decision-making ability and generalization ability in a strong uncertain game situation, so it is necessary to study the intelligent game strategy of the spacecraft.

[0003] Therefore, by analyzing the "pursuit-defense-escape" three-party game situation, a spacecraft active defense game method based on multi-agent reinforcement learning is designed to realize the active defense strategy of the defense party, effectively improve the safety performance of the spacecraft in orbit, and protect the spacecraft from being damaged by the enemy spacecraft. SUMMARY

[0004] The purpose of the application is to provide a spacecraft with an active defense game system and a method thereof, which solves the bottleneck of low calculation efficiency and difficulty in solving multiple constraints of the existing spacecraft "pursuit-defense-escape" game problem based on differential game method; the application mainly realizes offline training by constructing a spacecraft relative kinematics model, constructing a "pursuit-defense-escape" three-party game model, designing and establishing a spacecraft "pursuit-defense-escape" active defense game training system, and using a neural network to realize online decision-making; it is proved by simulation cases that the method can realize the active defense strategy of the defense party.

[0005] The application is implemented by adopting the following technical solutions (the technical solutions are supplemented after being determined):

[0006] 1. A spacecraft with an active defense game system, characterized in that the active defense game system comprises a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model; the active defense game training module is composed of a first fully connected neural network and a second fully connected neural network; wherein:

[0007] The motion control model is the impulse orbit data of each spacecraft in tracking stars, defense stars and escape stars in the LVLH coordinate system, that is:

[0008]

[0009] wherein: are position and velocity vectors,△v i = [△v i,x ,△v i,y ] T is the velocity increment vector; t0is the initial time of the mission, t i is the i-th time of the maneuver; φ(t, t i ) and φ v (t, t i ) are the state transition matrices of the CW equation;

[0010] The "pursuit-defense-escape" game model constructs the game objectives of the pursuit satellite, the defense satellite and the escape satellite through the following formula:

[0011]

[0012] wherein: u i＝P,E,D is the control strategy of the three spacecrafts, is the velocity increment, t f is the terminal time, r DP is the relative distance between the defense satellite and the pursuit satellite, r PE is the relative distance between the pursuit satellite and the escape satellite,△r max is the specified minimum distance, and are the single maneuver constraint and the total maneuver constraint, respectively,

[0013] The active defense game training module optimizes the "pursuit-defense-escape" game model offline according to the multi-agent reinforcement learning algorithm;

[0014] The optimized "pursuit-defense-escape" game model outputs the defense game strategy online according to the three-party game scenario of the spacecrafts.

[0015] 2. The spacecraft with an active defense game system according to claim 1, wherein the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to the multi-agent reinforcement learning algorithm, and the optimization process comprises:

[0016] 301. Initialize the parameter experience pool space of the first fully connected neural network and the second fully connected neural network respectively, and each group of data of the initialized experience pool is:

[0017] D t = {s P,t , s E,t , s D,t , s P,t+1 , s E,t+1 , s D,t+1 , aP,t a E,t a D,t r P,t r E,t r D,t}

[0018] wherein: s P,t s E,t s D,t is the current state, s P,t+1 s E,t+1 s D,t+1 is the new state, a P,t a E,t a D,t is the current maneuver value, r P,t r E,t r D,t is the current reward value;

[0019] 302, initialize the states of the three spacecraft in each simulation round, and input the initial states of the three spacecraft into corresponding first and second fully connected neural networks; wherein: the first fully connected neural network outputs a maneuver value according to the input states of the three spacecraft, and the second fully connected neural network outputs an evaluation value according to the overall input states of the three spacecraft;

[0020] 303, input the maneuver value output by the first fully connected neural network into a motion control model to obtain the impulse orbit data of each spacecraft in the LVLH coordinate system for tracking the star, the defense star and the escape star;

[0021] 304, input the impulse orbit data of each tracking star, defense star and escape star into the following reward function model to obtain the optimal strategy from state to action for each tracking star, defense star and escape star; wherein:

[0022] Distance reward r L : the calculation formula of the distance reward is:

[0023]

[0024] wherein, α l , α g1 and α g2 are reward coefficients; at time t, the relative distance between the tracking star and the escape star is defined as △L PE (t) = ||r P (t)-r E (t)||2, and the relative distance between the defense star and the tracking star is defined as △L DP (t) = ||r D (t)-r P (t)||2;

[0025] Time reward r t :

[0026]

[0027] wherein: p represents the time reward value; for the pursuit star and the defense star, if the task is not completed, a fixed negative reward is given at each decision point, while the escape star will obtain a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the pulse PED task;

[0028] Fuel consumption reward r △v :

[0029]

[0030] Terminal reward r Done , including:

[0031] (d) the terminal reward function of the defense star successfully capturing the escape star, that is:

[0032]

[0033] (e) the terminal reward function of the defense star successfully intercepting the defense star, that is:

[0034]

[0035] (f) the terminal reward function of the game time exceeding the maximum time, that is:

[0036]

[0037] The final total reward is:

[0038]

[0039] 305. Store the optimal strategy of each pursuit star, defense star and escape star from state to action in the experience pool;

[0040] 306. Determine whether the optimal strategy of each pursuit star, defense star and escape star from state to action reaches the terminal state, if not, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the states of the three spacecraft need to be reinitialized;

[0041] 307. Determine the number of experience pools, if the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302;

[0042] 308、In training, the first fully connected neural network and the second fully connected neural network are updated by using gradient descent method; the experience pool is emptied after the first fully connected neural network and the second fully connected neural network are updated;

[0043] 309、The system determines whether the maximum training round is reached, if yes, the training is stopped, otherwise, the training is continued, and the step 301 is returned.

[0044] 4. The spacecraft with an active defense game system according to claim 1, wherein the process of updating the first fully connected neural network and the second fully connected neural network by using the MAPPO algorithm in the step 307 comprises:

[0045] The first fully connected neural network is updated by using the following formula:

[0046]

[0047] Wherein, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the advantage of the current action a t over the strategy π θ ;

[0048] The second fully connected neural network is updated by using the following formula:

[0049]

[0050] Wherein: L critic (θ) represents the difference between the real value function and the estimated value corresponding to the state s t , since the value function V t (s π ) corresponding to s t is unknown, θ can be represented as a function of the neural network weight parameter θ, i.e. For a certain trajectory, the real value function at the state s t can be estimated by the following formula:

[0051]

[0052] 5. The spacecraft with an active defense game system according to claim 1, wherein the active defense game training module comprises the following steps of:

[0053] The first fully connected neural network adopts an Actor network, the network input variable of which is the relative position and velocity of the spacecraft and other two spacecrafts and the energy loss of itself, has three hidden layers, the hidden layer node is 256, the output is the spacecraft velocity increment, adopts the ReLU function as the network activation function, and the neural network is trained using the Adam optimizer;

[0054] The second fully connected neural network adopts a Critic network, the network input variable of which is the observation variable of the three spacecrafts as a whole; the node number of the output layer is an evaluation value, which is used to judge the pros and cons of the current state; the number of layers and nodes of the hidden layer can be defined by itself, three layers of hidden layers are designed here, the node number of each layer is 256, the ReLU function is adopted as the network activation function, and the neural network is trained using the Adam optimizer.

[0055] 6. A spacecraft active defense game building method, characterized in that it comprises the following steps:

[0056] S1. Each spacecraft tracks the pulse orbits of the star, defense star and escape star in the LVLH coordinate system to establish a motion control model:

[0057] S2. A spacecraft "pursuit-defense-escape" game model is built according to the motion trajectories of the pursuit star, defense star and escape star in the motion control model;

[0058] S3. An active defense game training module is built according to the multi-agent reinforcement learning algorithm; wherein:

[0059] 301. The parameter experience pool space of the first fully connected neural network and the second fully connected neural network is initialized respectively, each group of data of the initialized experience pool is:

[0060] D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t}

[0061] Wherein: s P,t ,s E,t ,s D,t is the current state, s P,t+1 ,s E,t+1 ,s D,t+1 is the new state, a P,t ,a E,t ,aD,t Current mobility value, r P,t ,r E,t ,r D,t Current reward value;

[0062] 302. In each simulation round, the states of the three spacecraft are initialized, and the initial states of the three spacecraft are input into the corresponding first fully connected neural network and second fully connected neural network; wherein: the first fully connected neural network outputs the maneuver value based on the input states of the three spacecraft, and the second fully connected neural network outputs the evaluation value based on the overall input states of the three spacecraft;

[0063] 303. Based on the maneuver value output by the first fully connected neural network, input the motion control model to obtain the pulse orbit data of the tracking star, defense star, and escape star for each spacecraft in the LVLH coordinate system;

[0064] 304. Based on the pulse orbit data of each tracking star, defense star, and escape star, input the following reward function model to obtain the optimal strategy from state to action for each tracking star, defense star, and escape star; where:

[0065] Distance reward r L The formula for calculating distance reward is:

[0066]

[0067] Where, α l α g1 and α g2 It is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as ΔL. PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as ΔL DP (t)=||r D (t)-r P (t)||2;

[0068] Time reward r t :

[0069]

[0070] Where: ρ represents the time reward value; for tracking and defense stars, if the mission is not completed, a fixed negative reward (penalty) will be given at each decision point, while escape stars will receive a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the pulse PED mission;

[0071] Fuel consumption reward r △v :

[0072]

[0073] terminal reward r Done :

[0074] (a) the terminal reward function of the defense star successfully capturing the escape star, that is,

[0075]

[0076] (b) the terminal reward function of the defense star successfully intercepting the defense star, that is,

[0077]

[0078] (c) the terminal reward function of the game time exceeding the maximum time, that is,

[0079]

[0080] The final total reward is:

[0081]

[0082] 305. Store the optimal strategy of each tracking star, defense star and escape star from state to action in the experience pool;

[0083] 306. Determine whether the optimal strategy of each tracking star, defense star and escape star from state to action reaches the terminal state, if not, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the state of the three spacecrafts needs to be reinitialized;

[0084] 307. Determine the number of experience pools, if the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302; wherein:

[0085] The first fully connected neural network is updated by the following formula:

[0086]

[0087] Wherein, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the advantage of the current action a t compared to the strategy π θ ;

[0088] The first fully connected neural network is updated by the following formula:

[0089]

[0090] Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t (Unknown) θ can be expressed as a function of the neural network weight parameters θ, i.e. For a given trajectory, state s t The true value function at a given location can be estimated using the following formula:

[0091]

[0092] 308. During training, gradient descent is used to update the first fully connected neural network and the second fully connected neural network; after updating the first and second fully connected neural networks, the experience pool is cleared.

[0093] 309. The system determines whether the maximum number of training rounds has been reached. If it has, training stops; otherwise, training continues and returns to step 301.

[0094] S4. The optimized three-way game model of "pursuit-defense-escape" is obtained by offline optimization of the "pursuit-defense-escape" game model through the spacecraft active defense game training module;

[0095] S5. Apply the optimized "pursuit-defense-escape" three-way game model to the online spacecraft three-way game scenario and output the defensive game strategy.

[0096] Beneficial effects

[0097] 1. High computational efficiency: Existing differential game-based methods for solving the spacecraft "pursuit-defense-escape" game problem require solving complex differential equations, resulting in a very large computational load and making it difficult to meet real-time requirements. This invention, however, achieves offline training and online decision-making by constructing a spacecraft relative kinematics model, a three-way "pursuit-defense-escape" game model, and utilizing multi-agent reinforcement learning. This method eliminates the need to solve complex differential equations, significantly improving computational efficiency and meeting real-time requirements.

[0098] 2. Capable of solving problems with multiple constraints: This invention is designed to solve problems with multiple constraints. Existing methods often can only solve single-objective optimization problems and cannot consider the constraints of multiple objectives, thus failing to obtain a globally optimal solution. However, this invention, through the design of a multi-agent reinforcement learning algorithm, can simultaneously consider the constraints of multiple objectives, thereby obtaining a globally optimal solution.

[0099] 3. The problem of implementing the active defense strategy of the defense party can be solved: the existing technology can only passively deal with the attack of the attack party when facing the spacecraft "pursuit-defense-escape" game problem, and it is difficult to implement the active defense strategy of the defense party. The spacecraft "pursuit-defense-escape" active defense game training system is designed and established, so that the defense party can continuously learn and optimize its strategy in the training process, thereby realizing the active defense strategy.

[0100] 4. Strong adaptability: the training system of the application can be trained in different scenes, has strong adaptability, and can cope with various complex "pursuit-defense-escape" game problems.

[0101] 5. Easy to implement: the training system of the application adopts a fully connected neural network model, which is easy to implement and does not require complex settings and adjustments, greatly reducing the difficulty and cost of use.

[0102] In summary, compared with the prior art, the application has the advantages of high computational efficiency, ability to solve multiple constraints, ability to implement the active defense strategy of the defense party, strong adaptability, and easy implementation, and has high practical value and broad application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0103] Figure 1 is a spacecraft framework with an active defense game system.

[0104] Figure 2 is a spacecraft "pursuit-defense-escape" active defense game training flowchart.

[0105] Figure 3 is an escape star training reward curve diagram of the application.

[0106] Figure 4 is a defense star training reward curve diagram of the application.

[0107] Figure 5 is a tracking star training reward curve diagram of the application.

[0108] Figure 6 is an escape star training success rate curve diagram of the application.

[0109] Figure 7 is a defense star training success rate curve diagram of the application.

[0110] Figure 8 is a tracking star training success rate curve diagram of the application.

[0111] Figure 9 is a three-party game trajectory diagram of the application. DETAILED DESCRIPTION

[0112] The application is further described below with reference to the drawings.

[0113] The spacecraft with the active defense game system comprises a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model.

[0114] The motion control model is pulse orbit data of each spacecraft in tracking stars, defense stars and escape stars in the LVLH coordinate system.

[0115] The "pursuit-defense-escape" game model is constructed by the following formula to construct the game target of the tracking stars, the defense stars and the escape stars.

[0116] The active defense game training module is offline optimized according to the multi-agent reinforcement learning algorithm.

[0117] The optimized "pursuit-defense-escape" game model outputs the defense game strategy according to the three-party game scene of the spacecraft.

[0118] The application provides a spacecraft active defense game method based on multi-agent reinforcement learning, comprising the following steps:

[0119] S1, a motion control model is established for each spacecraft in tracking stars, defense stars and escape stars in the LVLH coordinate system.

[0120] S2, a spacecraft "pursuit-defense-escape" game model is constructed according to the motion trajectories of the tracking stars, the defense stars and the escape stars in the motion control model.

[0121] S3, an active defense game training module is constructed according to the multi-agent reinforcement learning algorithm.

[0122] S4, the "pursuit-defense-escape" game model is offline optimized by the active defense game training module of the spacecraft to obtain an optimized "pursuit-defense-escape" three-party game model.

[0123] S5, the optimized "pursuit-defense-escape" three-party game model is applied to an online spacecraft three-party game scene to output a defense game strategy.

[0124] The application mainly comprises two stages: offline training and online decision-making. A large amount of different simulation data is generated through a simulation environment to train the MAPPO algorithm offline, and finally a trained neural network model is obtained. The trained neural network model can be used to realize online autonomous game. Wherein:

[0125] The relative kinematics model of the spacecraft has the specific process that:

[0126] The spacecraft "chase-defense-escape" three-party game problem is mainly the game problem of the chase star, the defense star and the escape star, and the defense game strategy for the defense party aims to ensure that the escape star is not captured by the chase star. The method takes the initial coordinates of the escape star as the origin, and establishes a reference coordinate system based on the local vertical local horizontal (LVLH) coordinate system. The CW equation is established based on the LVLH coordinate system, and then the pulse orbit maneuver control model of the chase star, the defense star and the escape star in the LVLH coordinate system is

[0127]

[0128] wherein: is the position and velocity vector, △v i = [△v i,x ,△v i,y ] T is the velocity increment vector; t0 is the initial time of the task, t i is the i-th maneuver time. φ (t, t i ) and φ v (t, t i ) are the state transition matrices of the CW equation, and the expressions are as follows:

[0129]

[0130] represents the initial state of the spacecraft at t i , represents the state of the spacecraft after the maneuver at t i . Therefore, the state transition of the spacecraft at t i can be expressed as:

[0131]

[0132] Therefore, the state of the next maneuver time can be determined by the following formula:

[0133]

[0134] The specific process of the "chase-defense-escape" three-party game model in the S2 step is as follows:

[0135] The states of the chase star, the defense star and the escape star at t i are defined as and At each △t, the three spacecrafts will simultaneously implement pulse control, that is, generate respective velocity increments and The specific expressions are as follows:

[0136]

[0137] In the actual "chase-avoid-escape" game problem, due to the limitations of energy, task time and other factors, the whole game process will not continue forever, so for the "chase-avoid-escape" game problem, there is a maximum end time t f :

[0138] t f ≤t max (9)

[0139] The maneuverability of a spacecraft is defined as the amount of velocity change that can be produced by a single impulse control. In actual flight missions, the single-impulse velocity increments of the tracking star, the defense star and the escape star in the x and y directions have the following constraints:

[0140]

[0141] The maximum total impulse velocity increment that a spacecraft can achieve is used to measure its fuel reserves. This indicator reflects the maximum total speed change that a spacecraft can achieve through continuous pulse control during a flight mission. Therefore, the constraints on the fuel reserves of the tracking star, the defense star and the escape star can be expressed as follows:

[0142]

[0143] For the "chase-avoid-escape" game problem oriented to active defense, the main goal is to ensure that the escape star is not tracked by the pursuit star, so the terminal goal set of the "chase-avoid-escape" game problem is defined as follows:

[0144]

[0145] Where△r max is the maximum relative distance when the pursuit is successful, r DP (t) and r PE (t) are the coordinate vectors of the tracking star and the defense star, and the tracking star and the escape star, respectively.

[0146] When the state of the defense star and the escape star first reaches the terminal condition, the "chase-avoid-escape" game task is successful. Therefore, the definition equation of the end time is as follows:

[0147]

[0148] In the impulse "chase-avoid-escape" game problem, the goal of the tracking star is to capture the escapee in the shortest possible time under the constraint conditions. That is, the tracking star finds the impulse control sequence on to minimize t f . Therefore, the objective function of the pursuer can be expressed as:

[0149]

[0150] where

[0151] For the active defense-oriented "chase-defense-escape" game problem, the defense star should actively approach the pursuit side when it discovers the pursuit star to destroy or weaken the ability of the pursuit side. Then the objective function of the defense star can be expressed as:

[0152]

[0153] In contrast, the objective function of the escape star is to increase its survival time as much as possible under the constraint condition, and then the objective function of the escape star can be expressed as:

[0154]

[0155] After in-depth analysis of the game objectives of the pursuit star, the defense star and the escape star, the key factors such as the principle of orbit dynamics, the mechanism of generating thrust, the constraint of spacecraft maneuvering performance, the limited energy and the predetermined mission completion time are fully integrated. On this basis, a comprehensive and multi-constrained impulse "chase-defense-escape" game model is proposed, and its calculation formula is:

[0156]

[0157] Step three is to establish a spacecraft "chase-defense-escape" active defense game training system based on multi-agent reinforcement learning algorithm, and its specific process is:

[0158] The most representative in the field of reinforcement learning is the actor-critic (Actor-Critic) algorithm architecture. The proximal policy optimization (PPO) algorithm is widely used because of its easy implementation and superior performance. Therefore, the MAPPO algorithm based on the PPO algorithm is used as the spacecraft game algorithm.

[0159] The purpose of the spacecraft "chase-defense-escape" active defense game training system is for the three spacecrafts to select the optimal action under the current state, and through continuous decision-making to make the spacecrafts be able to game successfully. This process meets the characteristics of random sequential decision-making. The "chase-defense-escape" game problem belongs to the problem of multi-agent sequential decision-making, that is, the multi-agent Markov decision process, which mainly includes the state set, the action set, the state transition equation, the reward function and the discount factor:

[0160] <S P ,S E ,S D ,A P ,A E ,A D ,R P ,RE ,R D ,P P ,P E ,P D ,γ P ,γ E ,γ D > (18)

[0161] where S P , S E and S D represent the environment state perceived by the pursuer, the evader and the defender, respectively; A P , A E and A D denote the action space; the reward functions R P , R E and R D measure the reward value of the pursuer, the evader and the defender, respectively. The strategies P P , P E and P D correspond to the game strategies of the three parties, respectively, while the discount factors γ P , γ E and γ D reflect the discounted value of future rewards. The goal of the pursuer, the evader and the defender is to find an optimal strategy mapping from state to action, policy, that maximizes their cumulative future rewards:

[0162]

[0163] To maximize the reward, a reasonable reward function needs to be designed. The reward of the three spacecrafts is designed by the invention to consist of distance reward, time reward, fuel consumption reward and terminal reward.

[0164] Distance reward r L : At time t, the relative distance between the pursuer and the evader is defined as △L PE (t) = ||r P (t) - r E (t) ||2, and the relative distance between the defender and the pursuer is defined as △L DP (t) = ||r D (t) - r P (t) ||2. The goal of the pursuer is to minimize the distance from the evader, while the goal of the evader is to maximize this distance. Similarly, the goal of the defender is to shorten the distance from the pursuer. Therefore, the calculation formula of the distance reward is:

[0165]

[0166] where α l , α g1and a g2 is the reward coefficient.

[0167] Time reward r t : The goal of the chaser is to catch the escapee as soon as possible, the goal of the defender is to catch the chaser as soon as possible, and the goal of the escapee is to prolong the terminal time as much as possible. The time reward formula is:

[0168]

[0169] where p represents the time reward value. For the chaser and the defender, if the task is not completed, a fixed negative reward (penalty) will be given at each decision point, while the escapee will obtain a positive reward at each decision point. Regardless of success or failure, "Done" marks the end of the impulse PED task.

[0170] Fuel consumption reward r △v : Although each spacecraft has its specific task, it should still try to minimize fuel consumption while achieving the goal. The fuel consumption reward is defined as follows:

[0171]

[0172] Terminal reward r Done : In the impulse PED problem, each spacecraft corresponds to different terminal rewards. The terminal conditions can be divided into three cases: (a) the defender successfully captures the escapee; (b) the defender successfully intercepts the defender; (c) the game time exceeds the maximum time. The definition of the terminal reward is as follows:

[0173]

[0174] The final total reward is:

[0175]

[0176] The entire decision update process of the training system can be composed by the above constructed multi-agent MDP model.

[0177] The spacecraft "chaser-defender-escapee" active defense game training system uses the MAPPO algorithm as the training algorithm, as shown in Figure 1 , MAPPO adopts a centralized training decentralized execution framework. In the decision-making process, the Actor network of each spacecraft takes the current observed environment state (such as S P , S E or S D ) as input and outputs the corresponding action. For spacecraft i, it is denoted as , where θ i represents the parameters of the Actor network.

[0178] The flowchart of the spacecraft "pursuit-defense-escape" active defense game training system is as follows: Figure 2 As shown, in the initial stage of training, the parameters of the Actor network and Critic network for each spacecraft are first initialized, and the experience pool space is initialized, where each set of data in the experience pool...

[0179] D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t} represents the current state s P,t ,s E,t ,s D,t New state s P,t+1 ,s E,t+1 ,s D,t+1 Current mobility value a P,t ,a E,t ,a D,t and the current reward value r P,t ,r E,t ,r D,t ;

[0180] In each simulation round, the initial states of the three spacecraft are first initialized, and these states are then input into the Actor network and Critic network corresponding to each spacecraft. The Actor network outputs maneuver values ​​based on the input states, and the Critic network outputs evaluation values ​​based on the overall input states of the three spacecraft.

[0181] The system inputs the maneuver values ​​output by the Actor network into the relative kinematics equations to obtain a new state, calculates the reward function value, and stores the data in the experience pool.

[0182] The system further determines whether the new state has reached the terminal state. If it has not reached the terminal state, the Actor network and Critic network continue to interact with the environment; if it has reached the terminal state, the states of the three spacecraft need to be reinitialized.

[0183] The system determines the size of the experience pool. If the size of the experience pool is reached, the Actor network and Critic network are updated using the MAPPO algorithm; otherwise, the system continues to collect data.

[0184] The MAPPO algorithm updates the parameters of the Actor network and Critic network as follows:

[0185] The method defines the Actor network loss function as L actor (θ), and the specific expression is:

[0186]

[0187] Wherein, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the advantage of the current action a t over the strategy π θ .

[0188] L critic (θ) represents the difference between the real value function and the estimated value corresponding to the state s t , which is used to update the Critic network parameters. Since s t corresponding value function V π (s t ) is unknown, it is generally estimated by a neural network, which can be represented as a function of neural network weight parameters θ, i.e. For a certain trajectory, the real value function at state s t can be estimated by the following formula:

[0189]

[0190] Then the loss function of the value function can be represented as:

[0191]

[0192] During training, the Actor and Critic networks are updated using gradient descent method;

[0193] After updating the Actor and Critic networks, the experience pool is emptied;

[0194] The system determines whether the maximum training round has been reached. If it has, the training is stopped, otherwise the training continues.

[0195] Both the Actor network and the Critic network use a fully connected neural network model: for each Actor network, a fully connected neural network is designed, with the network input variables being the relative positions and velocities of the spacecraft and the other two spacecraft, as well as the energy loss of the spacecraft itself. It has three hidden layers with 256 hidden layer nodes, and the output is the spacecraft velocity increment. The ReLU function is used as the network activation function, and the Adam optimizer is used to train the neural network.

[0196] For each Critic network, a fully connected neural network is designed, and the input variables of the network are the observation variables of the three spacecrafts; the number of nodes in the output layer is an evaluation value, which is used to judge the pros and cons of the current state. The number of layers and nodes in the hidden layer can be defined by oneself, and three hidden layers are designed here, and the number of nodes in each layer is 256, and the ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network.

[0197] In order to facilitate the description of the training process of step S4, the following simulation scenario is designed:

[0198] The simulation game scenario is set near the geosynchronous orbit, and the orbital radius is 36000 kilometers, and the game period is 36000 seconds. The origin of the reference coordinate system is selected on the geosynchronous orbit. According to the specific PED task requirements, the simulation experiment will fully consider the terminal time, fuel reserve, maneuverability and pulse interval and other constraint conditions. At the same time, at the initial moment of simulation, the state of the tracking star and the defense star is randomly generated. The win-lose judgment condition of PED game is that if the escape star is never caught by the tracking star, the defense party wins, otherwise fails.

[0199] The simulation program is built using Python and PyTorch, and is trained on a machine equipped with Intel i7-9700F@3.00GHz and Nvidia 3080Ti.

[0200] The following examples are used to verify the beneficial effects of the present application:

[0201] 1) Experimental environment

[0202] The simulation experiment environment described in step S4 is adopted

[0203] 2) Analysis of experimental results

[0204] The reward curve of Figures 3-5 and the success rate curve of Figures 6-8 obtained by step four training of the present application. As shown in the figure, since the maneuvering ability of the tracking star is relatively strong, the tracking star success rate and reward value rise quickly in the early training period, but as the training continues, the defense party gradually learns the defense strategy, mainly reflected in the defense star can actively attack the tracking star, and the tracking test defense party success rate is 92%.

[0205] In order to evaluate the game strategy of the algorithm, a simulation case is randomly generated. Figure 9 The game trajectory of the three spacecrafts can be seen that the tracking star approaches the escape star at a very fast speed, but the defense star can accurately intercept the tracking star, proving the effectiveness of the present application.

[0206] Due to the advancement of the present application, it can be widely applied in the fields of spacecraft navigation and control, intelligent decision support system, multi-agent reinforcement learning, etc. In the field of spacecraft navigation and control, the present application provides a spacecraft "chase-avoid-escape" active defense game method based on multi-agent reinforcement learning, which can realize autonomous navigation and control of spacecraft, and improve the autonomy and safety of spacecraft. Especially in complex space environment, the method of the present application can effectively deal with multiple threats, realize active defense of spacecraft, and improve the success probability of space mission. In the field of intelligent decision support system, the present application provides a decision method based on multi-agent reinforcement learning, which can be used to solve complex decision problems. Especially in the environment of multi-agent interaction, the method of the present application can realize autonomous decision of agent, and improve the efficiency and accuracy of decision. In the field of multi-agent reinforcement learning, the present application provides a new multi-agent reinforcement learning method, which can be used to study the interaction and cooperation between multiple agents. Especially for complex game problems with multiple constraints, the method of the present application can provide an effective solution, and improve the solving efficiency and quality. In summary, the present application has broad application prospects and large market demand, and is expected to play an important role in the fields of spacecraft navigation and control, intelligent decision support system, multi-agent reinforcement learning, etc.

[0207] The above specific embodiments are used to explain and illustrate the present application, rather than limit the present application. Any modifications and changes made to the present application within the spirit and protection scope of the claims fall within the protection scope of the present application.

Claims

1. A spacecraft with an active defense game-theoretic system, characterized in that, The active defense game system includes a motion control model, a "chase-defense-escape" game model, an active defense game training module, and an optimized "chase-defense-escape" game model; the active defense game training module consists of a first fully connected neural network and a second fully connected neural network; wherein: The motion control model consists of the pulse orbit data of each spacecraft in the LVLH coordinate system for tracking, defense, and escape satellites; that is: in: It is the position and velocity vector, Δv i =[△v i,x ,△v i,y ] T This is the velocity increment vector; t0 is the mission initial time, t i It is the time of the i-th maneuver; φ(t,t) i ) and φ v (t,t i ) is the state transition matrix of the CW equation; The "chase-defense-escape" game model constructs the game objectives of the tracking star, defense star, and escape star using the following formula: Where: u i＝P,E,D Control strategies for the three spacecraft For the velocity increment, t f For the terminal time, r DP r represents the relative distance between the defense star and the tracking star. PE Δr represents the relative distance between the tracking star and the escape star. max To specify the minimum distance, and These are constraints on single maneuver and total maneuver, respectively. The active defense game training module optimizes the "chase-defense-escape" game model offline using a multi-agent reinforcement learning algorithm; The optimized "pursuit-defense-escape" game model outputs defensive game strategies online based on the three-way game scenario of the spacecraft; The active defense game training module optimizes the "chase-defense-escape" game model offline using a multi-agent reinforcement learning algorithm, including:

301. Initialize the experience pool space of the first fully connected neural network and the second fully connected neural network respectively. Each set of data in the initialization experience pool is as follows: D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t }; Where: s P,t ,s E,t ,s D,t As the current state, s P,t+1 ,s E,t+1 ,s D,t+1 New state, a P,t ,a E,t ,a D,t Current mobility value, r P,t ,r E,t ,r D,t Current reward value; 302. In each simulation round, the states of the three spacecraft are initialized, and the initial states of the three spacecraft are input into the corresponding first fully connected neural network and second fully connected neural network; wherein: the first fully connected neural network outputs the maneuver value based on the input states of the three spacecraft, and the second fully connected neural network outputs the evaluation value based on the overall input states of the three spacecraft; 303. Based on the maneuver value output by the first fully connected neural network, input the motion control model to obtain the pulse orbit data of the tracking star, defense star, and escape star for each spacecraft in the LVLH coordinate system; 304. Based on the pulse orbit data of each tracking star, defense star, and escape star, input the following reward function model to obtain the optimal strategy from state to action for each tracking star, defense star, and escape star; where: Distance reward r L The formula for calculating distance reward is: Where, α l α g1 and α g2 It is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as ΔL. PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as ΔL DP (t)=||r D (t)-r P (t)||2; Time reward r t : Where: ρ represents the time reward value; for tracking and defense stars, a fixed negative reward will be given at each decision point if the mission is not completed, while escape stars will receive a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the pulse PED mission; Fuel consumption reward r △v : Terminal reward r Done ,include: (a) The terminal reward function for the defensive star successfully capturing the escaped star, namely: (b) The terminal reward function for a successful interception of a defensive satellite, i.e.: (c) The terminal reward function for games where the game time exceeds the maximum time, i.e.: The final total reward is:

305. Store the optimal strategy from state to action for each tracking star, defense star, and escape star in the experience pool; 306. Determine whether the optimal strategy from state to action for each tracking star, defense star, and escape star has reached the terminal state. If the terminal state has not been reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state has been reached, the states of the three spacecraft need to be reinitialized.

307. Determine the number of data in the experience pool. If the number of data in the experience pool has been reached, update the first fully connected neural network and the second fully connected neural network using the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302.

308. During training, gradient descent is used to update the first fully connected neural network and the second fully connected neural network; after updating the first and second fully connected neural networks, the experience pool is cleared.

309. The system determines whether the maximum number of training rounds has been reached. If it has, training stops; otherwise, training continues and returns to step 301. The process of updating the first fully connected neural network and the second fully connected neural network using the MAPPO algorithm in step 307 includes: The first fully connected neural network is updated using the following formula: Where, r t (θ) represents the probability ratio between the old and new strategies. Let be the dominance function, which represents the current action 'a'. t Compared to strategy π θ Advantages; The second fully connected neural network is updated using the following formula: Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t (Unknown) θ represents a function of the neural network weight parameters θ, i.e. For a given trajectory, state s t The true value function at a given location can be estimated using the following formula: The active defense game training module optimizes the "chase-defense-escape" game model offline using a multi-agent reinforcement learning algorithm, including: The first fully connected neural network adopts an Actor network. Its network input variables are the relative position and velocity of the spacecraft with the other two spacecraft and its own energy loss. It has three hidden layers with 256 hidden layer nodes. The output is the spacecraft velocity increment. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network. The second fully connected neural network adopts the Critic network, whose input variables are the observation variables of the three spacecraft as a whole; the number of nodes in the output layer is an evaluation value, which is used to judge the quality of the current state; the number of hidden layers and nodes can be defined by the user. Here, three hidden layers are designed, each with 256 nodes. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network.

2. The system processing method for active defense game theory of spacecraft according to claim 1 includes the following steps: S1. Establish a motion control model for each spacecraft in the pulse orbits of the tracking star, defense star, and escape star in the LVLH coordinate system: S2. Construct a spacecraft "pursuit-defense-escape" game model based on the motion trajectories of the tracking star, defense star, and escape star in the motion control model; S3. Construct an active defense game training module based on a multi-agent reinforcement learning algorithm; where:

301. Initialize the experience pool space of the first fully connected neural network and the second fully connected neural network respectively. Each set of data in the initialization experience pool is as follows: D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t } Where: s P,t ,s E,t ,s D,t As the current state, s P,t+1 ,s E,t+1 ,s D,t+1 New state, a P,t ,a E,t ,a D,t Current mobility value, r P,t ,r E,t ,r D,t Current reward value; 302. In each simulation round, the states of the three spacecraft are initialized, and the initial states of the three spacecraft are input into the corresponding first fully connected neural network and second fully connected neural network; wherein: the first fully connected neural network outputs the maneuver value based on the input states of the three spacecraft, and the second fully connected neural network outputs the evaluation value based on the overall input states of the three spacecraft; 303. Based on the maneuver value output by the first fully connected neural network, input the motion control model to obtain the pulse orbit data of the tracking star, defense star, and escape star for each spacecraft in the LVLH coordinate system; 304. Based on the pulse orbit data of each tracking star, defense star, and escape star, input the following reward function model to obtain the optimal strategy from state to action for each tracking star, defense star, and escape star; where: Distance reward r L The formula for calculating distance reward is: Where, α l α g1 and α g2 It is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as ΔL. PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as ΔL DP (t)=||r D (t)-r P (t)||2; Time reward r t : Where: ρ represents the time reward value; for tracking and defense stars, if the mission is not completed, a fixed negative reward (penalty) will be given at each decision point, while escape stars will receive a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the pulse PED mission; Fuel consumption reward r △v : Terminal reward r Done : (a) The terminal reward function for the defensive star successfully capturing the escaped star, namely: (b) The terminal reward function for a successful interception of a defensive satellite, i.e.: (c) The terminal reward function for games where the game time exceeds the maximum time, i.e.: The final total reward is:

305. Store the optimal strategy from state to action for each tracking star, defense star, and escape star in the experience pool; 306. Determine whether the optimal strategy from state to action for each tracking star, defense star, and escape star has reached the terminal state. If the terminal state has not been reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state has been reached, the states of the three spacecraft need to be reinitialized.

307. Determine the number of data points in the experience pool. If the number of data points has been reached, update the first fully connected neural network and the second fully connected neural network using the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302. The process of updating the first fully connected neural network and the second fully connected neural network using the MAPPO algorithm is as follows: The first fully connected neural network is updated using the following formula: Where, r t (θ) represents the probability ratio between the old and new strategies. Let be the dominance function, which represents the current action 'a'. t Compared to strategy π θ Advantages; The first fully connected neural network is updated using the following formula: Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t (Unknown) θ represents a function of the neural network weight parameters θ, i.e. For a given trajectory, state s t The true value function at a given location can be estimated using the following formula:

308. During training, gradient descent is used to update the first fully connected neural network and the second fully connected neural network; after updating the first and second fully connected neural networks, the experience pool is cleared.

309. The system determines whether the maximum number of training rounds has been reached. If it has, training stops; otherwise, training continues and returns to step 301. S4. The optimized three-way game model of "pursuit-defense-escape" is obtained by offline optimization of the "pursuit-defense-escape" game model through the spacecraft active defense game training module; S5. Apply the optimized "pursuit-defense-escape" three-way game model to the online spacecraft three-way game scenario and output the defensive game strategy.

Images