Spacecraft with active defense game system and method thereof

Abstract

The invention discloses a spacecraft with an active defense game system and a method thereof. The system comprises a motion control model, a'chasing-preventing-escaping 'game model, an active defense game training module and an optimized'chasing-preventing-escaping' game model. The active defense game training module is composed of a first full-connection neural network and a second full-connection neural network; wherein the motion control model is pulse orbit data of a tracking star, a defense star and an escape star of each spacecraft in an LVLH coordinate system; the tracking-prevention-escape game model is used for constructing respective game targets of a tracking star, a defense star and an escape star; the active defense game training module optimizes the'chasing-preventing-escaping 'game model offline according to a multi-agent reinforcement learning algorithm; the optimized'chasing-preventing-escaping 'game model outputs a defense game strategy on line according to a three-party game scene of the spacecraft; according to the method, an active defense strategy of a defense party spacecraft can be realized, and the on-orbit safety performance of the spacecraft is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft games, and in particular to a spacecraft with an active defense game system and a method thereof. Background Art

[0002] With the rapid development of the global aerospace industry, the space environment is becoming increasingly complex, the number of spacecraft on orbit has increased exponentially and their capabilities have been greatly improved, and the intelligence level of spacecraft in various countries has also been continuously improved. On the basis of traditional guidance, navigation and control technology, it is gradually moving towards intelligent perception, intelligent decision-making and autonomous control. At the same time, a large number of spacecraft on orbit are unable to cope with tasks with high real-time and high decision-making accuracy by relying solely on the ground command mode. In addition, the current control system is mainly oriented to deterministic task scenarios, with a weak level of intelligence, and lacks autonomous decision-making ability and generalization ability in a game situation with strong uncertainty, so it is necessary to study the intelligent game strategy of spacecraft.

[0003] Therefore, by analyzing the "pursuit-defense-escape" three-party game situation, designing a spacecraft active defense game method based on multi-agent reinforcement learning can realize the defender's active defense strategy, effectively improve the safety performance of the spacecraft on orbit, and ensure that the spacecraft is not damaged by enemy spacecraft. Summary of the invention

[0004] The purpose of the present invention is to provide a spacecraft with an active defense game system and a method thereof. The present invention solves the bottleneck of low computational efficiency and difficulty in solving multiple constraints in the existing spacecraft "pursuit-defense-escape" game problem based on differential game mode; the present invention mainly realizes offline training by constructing a spacecraft relative kinematic model, constructing a "pursuit-defense-escape" three-party game model, and designing and establishing a spacecraft "pursuit-defense-escape" active defense game training system, and uses a neural network to realize online decision-making; through simulation case verification, it is proved that the method can realize the active defense strategy of the defender.

[0005] The present invention is implemented by the following technical solutions (to be added after the technical solutions are determined):

[0006] 1. A spacecraft with an active defense game system, characterized in that the active defense game system includes a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model; the active defense game training module is composed of a first fully connected neural network and a second fully connected neural network; wherein:

[0007] The motion control model is the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system; that is:

[0008]

[0009] in: are the position and velocity vectors, △v i =[△v i,x ,△v i,y ] T is the velocity increment vector; t0 is the mission initial time, t i is the i-th maneuver time; φ(t,t i ) and φ v (t,t i ) is the state transition matrix of the CW equation;

[0010] The "pursuit-defense-escape" game model constructs the game objectives of the pursuit star, defense star and escape star respectively through the following formula:

[0011]

[0012] Where: u i＝P,E,D are the control strategies for the three spacecraft, is the velocity increment, t f is the terminal moment, r DP is the relative distance between the defense star and the tracking star, r PE is the relative distance between the tracking star and the escape star, △r max To specify the minimum distance, and are the single maneuver constraint and the total maneuver constraint, respectively.

[0013] The active defense game training module optimizes the “pursuit-defense-escape” game model offline according to the multi-agent reinforcement learning algorithm;

[0014] The optimized "pursuit-defense-escape" game model outputs a defense game strategy online according to a three-party game scenario of a spacecraft.

[0015] 2. A spacecraft with an active defense game system according to claim 1, characterized in that the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to a multi-agent reinforcement learning algorithm, including:

[0016] 301. Initialize the parameter experience pool space of the first fully connected neural network and the second fully connected neural network respectively, wherein each set of data of the initialized experience pool is:

[0017] D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,aP,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t}

[0018] Where: s P,t ,s E,t ,s D,t is the current state, s P,t+1 ,s E,t+1 ,s D,t+1 The new state, a P,t ,a E,t ,a D,t Current maneuver value, r P,t ,r E,t ,r D,t Current reward value;

[0019] 302. Initialize the states of the three spacecraft in each simulation round, and input the initial states of the three spacecraft into the corresponding first fully connected neural network and the second fully connected neural network; wherein: the first fully connected neural network outputs a maneuver value according to the input states of the three spacecraft, and the second fully connected neural network outputs an evaluation value according to the overall input states of the three spacecraft;

[0020] 303. Inputting the motion control model with the maneuver value output by the first fully connected neural network to obtain the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system;

[0021] 304. According to the pulse orbit data of each tracking star, defense star and escape star, the following reward function model is input to obtain the optimal strategy from state to action for each tracking star, defense star and escape star; wherein:

[0022] Distance Reward L : The calculation formula for distance reward is:

[0023]

[0024] Among them, α l , α g1 and α g2 is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as △L PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as △L DP (t)=||r D (t)-r P (t)||2;

[0025] Time Reward t :

[0026]

[0027] Where: ρ represents the time reward value; for tracking stars and defending stars, if the task is not completed, a fixed negative reward will be given at each decision point, while escape stars will receive a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the PED task;

[0028] Fuel Consumption Reward △v :

[0029]

[0030] Terminal Rewards Done ,include:

[0031] (d) The terminal reward function of the defending star when it successfully captures the escaped star, namely:

[0032]

[0033] (e) The terminal reward function of a defense star that successfully intercepts a defense star is:

[0034]

[0035] (f) The terminal reward function when the game time exceeds the maximum time, that is:

[0036]

[0037] The final total reward is:

[0038]

[0039] 305. The optimal strategy from status to action for each tracking star, defense star and escape star is stored in the experience pool;

[0040] 306. Determine whether the optimal strategy from state to action of each tracking star, defense star and escape star reaches the terminal state. If the terminal state is not reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the states of the three spacecraft need to be reinitialized;

[0041] 307. Determine the number of experience pools. If the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302.

[0042] 308. During training, the first fully connected neural network and the second fully connected neural network are updated using a gradient descent method; and the experience pool is cleared after the first fully connected neural network and the second fully connected neural network are updated;

[0043] 309. The system determines whether the maximum number of training rounds has been reached. If so, the system stops training; otherwise, the system continues training and returns to step 301.

[0044] 4. A spacecraft with an active defense game system according to claim 1, characterized in that the process of updating the first fully connected neural network and the second fully connected neural network by using the MAPPO algorithm in step 307 comprises:

[0045] The first fully connected neural network is updated using the following formula:

[0046]

[0047] Among them, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the current action a t Compared with strategy π θ Advantages:

[0048] The second fully connected neural network is updated using the following formula:

[0049]

[0050] Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t ) is unknown, θ can be expressed as a function of the neural network weight parameter θ, that is, For a certain trajectory, the state s t The true value function at can be estimated by the following formula:

[0051]

[0052] 5. A spacecraft with an active defense game system according to claim 1, characterized in that the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to a multi-agent reinforcement learning algorithm, including:

[0053] The first fully connected neural network adopts an Actor network, whose network input variables are the relative position and speed of the spacecraft and the other two spacecrafts and its own energy loss, has three hidden layers, the number of hidden layer nodes is 256, and the output is the spacecraft speed increment. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network;

[0054] The second fully connected neural network adopts the Critic network, whose network input variables are the observation variables of the three spacecraft as a whole; the number of nodes in the output layer is an evaluation value, which is used to judge the quality of the current state; the number of layers and nodes in the hidden layer can be defined by oneself. Three hidden layers are designed here, with 256 nodes in each layer. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network.

[0055] 6. A method for constructing a spacecraft active defense game, characterized in that it includes the following steps:

[0056] S1. The pulse orbits of each spacecraft tracking star, defense star and escape star in the LVLH coordinate system are used to establish the motion control model:

[0057] S2. Construct a spacecraft "pursuit-defense-escape" game model based on the motion trajectories of the pursuit star, defense star and escape star in the motion control model;

[0058] S3. Construct an active defense game training module based on the multi-agent reinforcement learning algorithm; where:

[0059] 301. Initialize the parameter experience pool space of the first fully connected neural network and the second fully connected neural network respectively, wherein each set of data of the initialized experience pool is:

[0060] D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t}

[0061] Where: s P,t ,s E,t ,s D,t is the current state, s P,t+1 ,s E,t+1 ,s D,t+1 The new state, a P,t ,a E,t ,aD,t Current maneuver value, r P,t ,r E,t ,r D,t Current reward value;

[0062] 302. Initialize the states of the three spacecraft in each simulation round, and input the initial states of the three spacecraft into the corresponding first fully connected neural network and the second fully connected neural network; wherein: the first fully connected neural network outputs a maneuver value according to the input states of the three spacecraft, and the second fully connected neural network outputs an evaluation value according to the overall input states of the three spacecraft;

[0063] 303. Inputting the motion control model with the maneuver value output by the first fully connected neural network to obtain the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system;

[0064] 304. According to the pulse orbit data of each tracking star, defense star and escape star, the following reward function model is input to obtain the optimal strategy from state to action for each tracking star, defense star and escape star; wherein:

[0065] Distance Reward L : The calculation formula for distance reward is:

[0066]

[0067] Among them, α l , α g1 and α g2 is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as △L PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as △L DP (t)=||r D (t)-r P (t)||2;

[0068] Time Reward t :

[0069]

[0070] Where: ρ represents the time reward value; for tracking stars and defending stars, if the task is not completed, a fixed negative reward (penalty) will be given at each decision point, while escape stars will receive positive rewards at each decision point; regardless of success or failure, "Done" marks the end of the PED task;

[0071] Fuel Consumption Reward △v :

[0072]

[0073] Terminal Rewards Done :

[0074] (a) The terminal reward function of the defending star when it successfully captures the escaped star, namely:

[0075]

[0076] (b) The terminal reward function of a defense star that successfully intercepts a defense star is:

[0077]

[0078] (c) The terminal reward function when the game time exceeds the maximum time, that is:

[0079]

[0080] The final total reward is:

[0081]

[0082] 305. The optimal strategy from status to action for each tracking star, defense star and escape star is stored in the experience pool;

[0083] 306. Determine whether the optimal strategy from state to action of each tracking star, defense star and escape star reaches the terminal state. If the terminal state is not reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the states of the three spacecraft need to be reinitialized;

[0084] 307. Determine the number of experience pools. If the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302; wherein: the process of updating the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm:

[0085] The first fully connected neural network is updated using the following formula:

[0086]

[0087] Among them, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the current action a t Compared with strategy π θ Advantages:

[0088] The first fully connected neural network is updated using the following formula:

[0089]

[0090] Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t ) is unknown, θ can be expressed as a function of the neural network weight parameter θ, that is, For a certain trajectory, the state s t The true value function at can be estimated by the following formula:

[0091]

[0092] 308. During training, the first fully connected neural network and the second fully connected neural network are updated using a gradient descent method; and the experience pool is cleared after the first fully connected neural network and the second fully connected neural network are updated;

[0093] 309. The system determines whether the maximum number of training rounds has been reached. If so, the system stops training. Otherwise, the system continues training and returns to step 301.

[0094] S4. Offline optimization of the "pursuit-defense-escape" game model through the spacecraft active defense game training module obtains an optimized "pursuit-defense-escape" three-party game model;

[0095] S5. Apply the optimized “pursuit-defense-escape” three-party game model to the online spacecraft three-party game scenario to output the defense game strategy.

[0096] Beneficial Effects

[0097] 1. High computational efficiency: The existing differential game-based method to solve the spacecraft "pursuit-defense-escape" game problem requires solving complex differential equations, which is very computationally intensive and difficult to meet real-time requirements. However, the present invention achieves offline training and online decision-making by constructing a spacecraft relative kinematics model, a "pursuit-defense-escape" three-party game model, and a multi-agent reinforcement learning method. This method does not require solving complex differential equations, greatly improves computational efficiency, and can meet real-time requirements.

[0098] 2. Ability to solve multiple constraints: The present invention is designed to solve multiple constraints: Existing methods can only solve single-objective optimization problems, and it is difficult to consider the constraints of multiple objectives, so it is impossible to obtain a global optimal solution. However, the present invention can consider the constraints of multiple objectives at the same time by designing a multi-agent reinforcement learning algorithm, so as to obtain a global optimal solution.

[0099] 3. The problem of being able to implement the active defense strategy of the defender: When facing the spacecraft "pursuit-defense-escape" game problem, the existing technology can only passively respond to the attack of the attacker, and it is difficult to implement the active defense strategy of the defender. However, the present invention, by designing and establishing a spacecraft "pursuit-defense-escape" active defense game training system, can enable the defender to continuously learn and optimize its own strategy during the training process, thereby realizing the active defense strategy.

[0100] 4. Strong adaptability: The training system of the present invention can be trained in different scenarios, has strong adaptability, and can cope with various complex "pursuit-defense-escape" game problems.

[0101] 5. Easy to implement: The training system of the present invention adopts a fully connected neural network model, which is easy to implement and does not require complicated settings and adjustments, greatly reducing the difficulty and cost of use.

[0102] In general, compared with the prior art, the present invention has the advantages of high computational efficiency, ability to solve multiple constraints, ability to implement the defender's active defense strategy, strong adaptability and easy implementation, and has high practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is a framework diagram of a spacecraft with an active defense game system according to the present invention.

[0104] Figure 2 It is a flow chart of the spacecraft "pursuit-defense-escape" active defense game training of the present invention.

[0105] Figure 3 It is the escape star training reward curve diagram of the present invention.

[0106] Figure 4 It is a defense star training reward curve diagram of the present invention.

[0107] Figure 5 It is a star tracking training reward curve diagram of the present invention.

[0108] Figure 6 It is a curve diagram of the success rate of escape star training of the present invention.

[0109] Figure 7 It is a curve diagram of the success rate of defense star training of the present invention.

[0110] Figure 8 It is a curve diagram of the success rate of star tracking training of the present invention.

[0111] Fig. 9 It is a three-party game trajectory diagram of the present invention. DETAILED DESCRIPTION

[0112] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0113] The present invention provides a spacecraft with an active defense game system, wherein the active defense game system comprises a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model; the active defense game training module is composed of a first fully connected neural network and a second fully connected neural network; wherein:

[0114] The motion control model is the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system;

[0115] The "pursuit-defense-escape" game model constructs the game objectives of the pursuit star, defense star and escape star respectively through the following formula;

[0116] The active defense game training module optimizes the “pursuit-defense-escape” game model offline according to the multi-agent reinforcement learning algorithm;

[0117] The optimized "pursuit-defense-escape" game model outputs a defense game strategy online according to a three-party game scenario of a spacecraft.

[0118] The present invention provides a spacecraft active defense game method based on multi-agent reinforcement learning, comprising the following steps:

[0119] S1. Establish motion control models for each spacecraft in the LVLH coordinate system for the pulse orbits of the tracking star, the defense star, and the escape star;

[0120] S2. Construct a spacecraft "pursuit-defense-escape" game model based on the motion trajectories of the pursuit star, defense star and escape star in the motion control model;

[0121] S3, construct active defense game training module based on multi-agent reinforcement learning algorithm;

[0122] S4. Offline optimization of the "pursuit-defense-escape" game model through the spacecraft active defense game training module obtains an optimized "pursuit-defense-escape" three-party game model;

[0123] S5. Apply the optimized “pursuit-defense-escape” three-party game model to the online spacecraft three-party game scenario to output the defense game strategy.

[0124] The present invention is mainly divided into two stages: offline training and online decision-making. A large amount of different simulation data is generated through a simulation environment to train the MAPPO algorithm offline, and finally a trained neural network model is obtained. The trained neural network model can be used to realize online autonomous game. Among them:

[0125] The relative kinematics model of the spacecraft has the following specific process:

[0126] The three-party game problem of "pursuit-defense-escape" of spacecraft is mainly a game problem among the tracking star, the defending star and the escape star, and the defense game strategy for the defender is to ensure that the escape star is not captured by the tracking star. This method takes the initial coordinates of the escape star as the origin and establishes a reference coordinate system based on the local vertical local horizontal (LVLH) coordinate system. The CW equation is easy to establish based on the LVLH coordinate system, so the pulse orbit maneuver control model of the tracking star, the defending star and the escape star in the LVLH coordinate system is:

[0127]

[0128] in: are the position and velocity vectors, △v i =[△v i,x ,△v i,y ] T is the velocity increment vector; t0 is the mission initial time, t i is the i-th maneuver time. φ(t,t i ) and φ v (t,t i ) is the state transition matrix of the CW equation, which is expressed as follows:

[0129]

[0130] Indicates that at t i The initial state of the spacecraft at time t, Indicates that at t i The state of the spacecraft after the maneuver. Therefore, the spacecraft is i The state transition at time can be expressed as:

[0131]

[0132] Therefore, the state of the next maneuver time can be determined by the following formula:

[0133]

[0134] The specific process of the "pursuit-defense-escape" three-party game model in step S2 is as follows:

[0135] Tracking stars, defense stars and escape stars are in t i The state is defined as and At each △t, the three spacecraft will simultaneously implement pulse control, that is, generate their own velocity increments and The specific expression is:

[0136]

[0137] In the actual "pursuit-defense-escape" game problem, due to the limitations of energy, task time and other factors, the entire game process will not last forever. Therefore, for the "pursuit-defense-escape" game problem, there is a maximum end time t f :

[0138] t f ≤t max (9)

[0139] The maneuverability of a spacecraft is defined as the velocity change that can be produced by a single pulse control. In actual flight missions, the single pulse velocity increments of the tracking satellite, defense satellite, and escape satellite in the x-axis and y-axis directions are subject to the following constraints:

[0140]

[0141] The maximum total pulse velocity increment that a spacecraft can achieve is used to measure its fuel reserve. This metric reflects the maximum total velocity change that can be achieved by continuous pulse control during the mission. Therefore, the limits on the fuel reserves of the pursuit, defense, and escape satellites can be expressed as follows:

[0142]

[0143] For the active defense-oriented "pursuit-defense-escape" game problem, the main goal is to ensure that the escaped star is not tracked by the pursuer star. Therefore, the terminal goal set of the "pursuit-defense-escape" game problem is defined as follows:

[0144]

[0145] Where △r max is the maximum relative distance for successful pursuit, r DP (t) and r PE (t) are the coordinate vectors of the tracking star and the defending star, the tracking star and the escape star, respectively.

[0146] When the states of the defense star and the escape star reach the terminal condition for the first time, the "pursuit-defense-escape" game task is successful. Therefore, the definition equation of the terminal time is as follows:

[0147]

[0148] In the pulse "pursuit-prevention-escape" game problem, the goal of the tracking star is to capture the evader in the shortest possible time under the constraints. Find the pulse control sequence on the f Therefore, the objective function of the pursuer can be expressed as:

[0149]

[0150] in

[0151] For the "pursuit-defense-escape" game problem for active defense, the defending star should actively approach the pursuer when discovering the tracking star to destroy or weaken the pursuer's ability. Then the objective function of the defending star can be expressed as:

[0152]

[0153] In contrast, the objective function of the escape star is to increase its survival time as much as possible under the constraints, so the objective function of the escape star can be expressed as:

[0154]

[0155] After deeply analyzing the game objectives of the tracking star, the defending star, and the escaping star, a comprehensive, multi-constrained pulse "pursuit-defense-escape" game model is proposed, and the calculation formula is as follows:

[0156]

[0157] The step three is to establish a spacecraft "pursuit-defense-escape" active defense game training system based on a multi-agent reinforcement learning algorithm, and the specific process is as follows:

[0158] The most representative algorithm in the field of reinforcement learning is the Actor-Critic algorithm architecture. The Proximal Policy Optimization (PPO) algorithm is widely used due to its advantages such as easy implementation and better performance. Therefore, this method uses the MAPPO algorithm, which is extended from the PPO algorithm, as the spacecraft game algorithm.

[0159] The purpose of the spacecraft "pursuit-defense-escape" active defense game training system is to select the optimal action for the three spacecraft in the current state, and make the spacecraft succeed in the game through continuous decision-making. This process meets the characteristics of random sequential decision-making. The "pursuit-defense-escape" game problem belongs to the multi-agent sequential decision-making problem, that is, the multi-agent Markov decision process, which mainly includes the state set, action set, state transfer equation, reward function and discount factor:

[0160] P ,S E ,S D ,A P ,A E ,A D ,R P ,R​E ,R D ,P P ,P E ,P D ,γ P ,γ E ,γ D > (18)

[0161] Where: S P , S E and S D Respectively represent the environmental states perceived by the tracking star, escape star, and defense star; A P , A E and A D Represents the action space; reward function R P , R E and R D Weigh the rewards of tracking stars, escaping stars, and defending stars respectively. Strategy P P , P E and P D Corresponding to the game strategies of the three parties, and the discount factor γ P , γ E and γ D reflects the discounted value of future rewards. The goal of the tracking star, escape star, and defending star is to find an optimal policy mapping strategy from state to action to maximize their cumulative future returns:

[0162]

[0163] In order to maximize the reward, a reasonable reward function needs to be designed. The rewards of the three spacecraft designed by the present invention are composed of distance reward, time reward, fuel consumption reward and terminal reward.

[0164] Distance Reward L : At time t, the relative distance between the pursuer and the escaper is defined as $△L PE (t)=||r P (t)-r E (t)||2$, the relative distance between the defending star and the tracking star is defined as △L DP (t)=||r D (t)-r P (t)||2. The goal of the chasing star is to minimize the distance to the escaping star, while the goal of the escaping star is to maximize this distance. Similarly, the goal of the defending star is to shorten the distance to the chasing star. Therefore, the distance reward is calculated as:

[0165]

[0166] Among them, α l , α g1and α g2 is the reward coefficient.

[0167] Time Reward t : The goal of the tracking star is to catch up with the escaping star as quickly as possible, the goal of the defending star is to catch up with the tracking star quickly, and the escaping star tries to extend the finish time as much as possible. The time reward formula is:

[0168]

[0169] Where ρ represents the time reward value. For tracking stars and defending stars, if the task is not completed, a fixed negative reward (penalty) will be given at each decision point, while escape stars will receive a positive reward at each decision point. Regardless of success or failure, "Done" marks the end of the PED task.

[0170] Fuel Consumption Reward △v :While each spacecraft has its own specific mission, it should still minimize fuel consumption while achieving its goals. The fuel consumption bonus is defined as follows:

[0171]

[0172] Terminal Rewards Done : In the PED problem, each spacecraft corresponds to a different terminal reward. The terminal conditions can be divided into three cases: (a) the defending star successfully captures the escaped star; (b) the defending star successfully intercepts the defending star; (c) the game time exceeds the maximum time. The terminal reward is defined as follows:

[0173]

[0174] The final total reward is:

[0175]

[0176] The multi-agent MDP model constructed above can constitute the entire decision-making update process of the training system.

[0177] The spacecraft "pursuit-defense-escape" active defense game training system uses the MAPPO algorithm as the training algorithm, such as Figure 1 As shown in Figure 2, MAPPO adopts a centralized training and decentralized execution framework. In the decision-making process, each spacecraft’s Actor network considers its currently observed environmental state (such as S P , S E or S D ) as input and output the corresponding action. For spacecraft i, use Indicates that, where θ i Represents the parameters of the Actor network.

[0178] The flow chart of the spacecraft "pursuit-defense-escape" active defense game training system is as follows: Figure 2 As shown, at the beginning of training, the Actor network and Critic network parameters of each spacecraft are initialized, and the experience pool space is initialized, where each set of data in the experience pool

[0179] D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t} is the current state s P,t ,s E,t ,s D,t , the new state s P,t+1 ,s E,t+1 ,s D,t+1 , current maneuver value a P,t ,a E,t ,a D,t And the current reward value r P,t ,r E,t ,r D,t ;

[0180] In each simulation round, the initial states of the three spacecraft are first initialized, and the states are input to the Actor network and Critic network corresponding to each spacecraft. The Actor network outputs the maneuver value according to the input state, and the Critic network outputs the evaluation value according to the overall input state of the three spacecraft;

[0181] The system brings the maneuver value output by the Actor network into the relative kinematics equation to obtain a new state, calculates the reward function value, and stores the data in the experience pool.

[0182] The system further determines whether the new state reaches the terminal state. If it does not reach the terminal state, the Actor network and the Critic network continue to interact with the environment; if it reaches the terminal state, the states of the three spacecraft need to be reinitialized.

[0183] The system determines the number of experience pools. If the number of experience pools is reached, the Actor network and the Critic network are updated through the MAPPO algorithm. Otherwise, the system continues to collect data.

[0184] The MAPPO algorithm updates the parameters of the Actor network and the Critic network as follows:

[0185] This method defines the Actor network loss function as L actor (θ), its specific expression is:

[0186]

[0187] Among them, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the current action a t Compared with strategy π θ Advantages.

[0188] L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value is used to update the Critic network parameters. t The corresponding value function V π (s t ) is unknown, and a neural network is generally used to estimate it, which can be expressed as a function of the neural network weight parameter θ, that is, For a certain trajectory, the state s t The true value function at can be estimated by the following formula:

[0189]

[0190] Then the loss function of the value function can be expressed as:

[0191]

[0192] During training, the Actor and Critic networks are updated using the gradient descent method;

[0193] Clear the experience pool after updating the Actor and Critic networks;

[0194] The system determines whether the maximum number of training rounds has been reached. If so, training stops; otherwise, training continues.

[0195] The Actor network and the Critic network both adopt a fully connected neural network model: for each Actor network, a fully connected neural network is designed. The network input variables are the relative position and speed of the spacecraft and the other two spacecraft and its own energy loss. It has three hidden layers with 256 hidden layer nodes. The output is the spacecraft speed increment. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network.

[0196] For each Critic network, a fully connected neural network is designed. The network input variables are the observation variables of the three spacecraft as a whole. The number of nodes in the output layer is an evaluation value, which is used to judge the quality of the current state. The number of layers and nodes in the hidden layer can be defined by yourself. Here, three hidden layers are designed, with 256 nodes in each layer. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network.

[0197] To facilitate the description of the training process in step S4, the following simulation scenario is designed:

[0198] The simulation game scenario is set near the geosynchronous orbit, with an orbital radius of 36,000 kilometers and a game period of 36,000 seconds. The origin of the reference coordinate system is selected in the geosynchronous orbit. The simulation experiment will fully consider the constraints such as terminal time, fuel reserves, maneuverability and pulse interval according to the specific PED mission requirements. At the same time, at the initial moment of the simulation, the states of the tracking satellite and the defending satellite are randomly generated. The winning and losing criteria of the PED game are that if the escaped satellite is never caught by the tracking satellite, the defending party wins, otherwise it fails.

[0199] The simulations were built using Python and PyTorch and trained on machines equipped with an Intel i7-9700F@3.00GHz and an Nvidia 3080Ti.

[0200] The following examples are used to verify the beneficial effects of the present invention:

[0201] 1) Experimental environment

[0202] Use the simulation experiment environment described in step S4

[0203] 2) Experimental results analysis

[0204] Through the training of step 4 of the present invention, Figure 3 to Figure 5 The reward curve and Figure 6 to Figure 8 As shown in the figure, due to the strong mobility of the tracking star, the tracking star success rate and reward value increased rapidly in the early stage of training. However, with continuous training, the defender gradually learned the defense strategy, which is mainly reflected in the fact that the defending star can actively attack the tracking star. The success rate of the defender in the tracking test is 92%.

[0205] In order to evaluate the game strategy of the algorithm, a simulation case is randomly generated. Fig. 9 The game trajectories of the three spacecraft show that the tracking star approaches the escape star at a very fast speed, but the defending star can accurately intercept the tracking star, which proves the effectiveness of the present invention.

[0206] Due to the advanced nature of the present invention, it can be widely used in the fields of spacecraft navigation and control, intelligent decision support system, multi-agent reinforcement learning, etc. In the field of spacecraft navigation and control, the present invention provides a spacecraft "pursuit-defense-escape" active defense game method based on multi-agent reinforcement learning, which can realize autonomous navigation and control of spacecraft and improve the autonomy and safety of spacecraft. Especially in complex space environments, the method of the present invention can effectively respond to various threats, realize active defense of spacecraft, and improve the probability of success of space missions. In the field of intelligent decision support systems, the present invention provides a decision-making method based on multi-agent reinforcement learning, which can be used to solve complex decision-making problems, especially in an environment where multiple agents interact with each other, the method of the present invention can realize autonomous decision-making of agents and improve the efficiency and accuracy of decision-making. In the field of multi-agent reinforcement learning, the present invention provides a new multi-agent reinforcement learning method, which can be used to study the interaction and cooperation between multiple agents, especially for complex game problems with multiple constraints, the method of the present invention can provide an effective solution to improve the efficiency and quality of solution. In general, the technical application prospects of the present invention are broad and the market demand is large, and it is expected to play an important role in the fields of spacecraft navigation and control, intelligent decision support systems, and multi-agent reinforcement learning.

[0207] The above specific implementation modes are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A spacecraft with an active defense game system, characterized in that: The active defense game system includes a motion control model, a "pursuit-defense-escape" game model, an active defense game training module and an optimized "pursuit-defense-escape" game model; the active defense game training module is composed of a first fully connected neural network and a second fully connected neural network; wherein: The motion control model is the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system; that is: in: are the position and velocity vectors, △v i =[△v i,x ,△v i,y ] T is the velocity increment vector; t0 is the mission initial time, t i is the i-th maneuver time; φ(t,t i ) and φ v (t,t i ) is the state transition matrix of the CW equation; The "pursuit-defense-escape" game model constructs the game objectives of the pursuit star, defense star and escape star respectively through the following formula: Where: u i＝P,E,D are the control strategies for the three spacecraft, is the velocity increment, t f is the terminal moment, r DP is the relative distance between the defense star and the tracking star, r PE is the relative distance between the tracking star and the escape star, △r max To specify the minimum distance, and are the single maneuver constraint and the total maneuver constraint, respectively. The active defense game training module optimizes the "pursuit-defense-escape" game model offline according to the multi-agent reinforcement learning algorithm; The optimized "pursuit-defense-escape" game model outputs a defense game strategy online according to a three-party game scenario of a spacecraft. Furthermore, the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to the multi-agent reinforcement learning algorithm, including:

301. Initialize the parameter experience pool space of the first fully connected neural network and the second fully connected neural network respectively, wherein each set of data of the initialized experience pool is: D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t } Where: s P,t ,s E,t ,s D,t is the current state, s P,t+1 ,s E,t+1 ,s D,t+1 The new state, a P,t ,a E,t ,a D,t Current maneuver value, r P,t ,r E,t ,r D,t Current reward value; 302. Initialize the states of the three spacecraft in each simulation round, and input the initial states of the three spacecraft into the corresponding first fully connected neural network and the second fully connected neural network; wherein: the first fully connected neural network outputs a maneuver value according to the input states of the three spacecraft, and the second fully connected neural network outputs an evaluation value according to the overall input states of the three spacecraft; 303. Inputting the motion control model with the maneuver value output by the first fully connected neural network to obtain the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system; 304. According to the pulse orbit data of each tracking star, defense star and escape star, the following reward function model is input to obtain the optimal strategy from state to action for each tracking star, defense star and escape star; wherein: Distance Reward L : The calculation formula for distance reward is: Among them, α l , α g1 and α g2 is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as △L PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as △L DP (t)=||r D (t)-r P (t)||2; Time Reward t : Where: ρ represents the time reward value; for tracking stars and defending stars, if the task is not completed, a fixed negative reward will be given at each decision point, while escape stars will receive a positive reward at each decision point; regardless of success or failure, "Done" marks the end of the PED task; Fuel Consumption Reward △v : Terminal Rewards Done ,include: (a) The terminal reward function of the defending star when it successfully captures the escaped star, namely: (b) The terminal reward function of a defense star that successfully intercepts a defense star is: (c) The terminal reward function when the game time exceeds the maximum time, that is: The final total reward is:

305. The optimal strategy from status to action for each tracking star, defense star and escape star is stored in the experience pool; 306. Determine whether the optimal strategy from state to action of each tracking star, defense star and escape star reaches the terminal state. If the terminal state is not reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the states of the three spacecraft need to be reinitialized; 307. Determine the number of experience pools. If the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302.

308. During training, the first fully connected neural network and the second fully connected neural network are updated using a gradient descent method; and the experience pool is cleared after the first fully connected neural network and the second fully connected neural network are updated; 309. The system determines whether the maximum number of training rounds has been reached. If so, the system stops training; otherwise, the system continues training and returns to step 301. Furthermore, the process of updating the first fully connected neural network and the second fully connected neural network by using the MAPPO algorithm in step 307 includes: The first fully connected neural network is updated using the following formula: Among them, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the current action a t Compared with strategy π θ Advantages: The second fully connected neural network is updated using the following formula: Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t ) is unknown, θ can be expressed as a function of the neural network weight parameter θ, that is, For a certain trajectory, the state s t The true value function at can be estimated by the following formula: Furthermore, the active defense game training module optimizes the "pursuit-defense-escape" game model offline according to the multi-agent reinforcement learning algorithm, including: The first fully connected neural network adopts an Actor network, whose network input variables are the relative position and speed of the spacecraft and the other two spacecrafts and its own energy loss, has three hidden layers, the number of hidden layer nodes is 256, and the output is the spacecraft speed increment. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network; The second fully connected neural network adopts the Critic network, whose network input variables are the observation variables of the three spacecraft as a whole; the number of nodes in the output layer is an evaluation value, which is used to judge the quality of the current state; the number of layers and nodes in the hidden layer can be defined by oneself. Three hidden layers are designed here, with 256 nodes in each layer. The ReLU function is used as the activation function of the network, and the Adam optimizer is used to train the neural network. The present invention can also adopt the following technical solutions: A method for constructing a spacecraft active defense game includes the following steps: S1. The pulse orbits of each spacecraft tracking star, defense star and escape star in the LVLH coordinate system are used to establish the motion control model: S2. Construct a spacecraft "pursuit-defense-escape" game model based on the motion trajectories of the pursuit star, defense star and escape star in the motion control model; S3. Construct an active defense game training module based on the multi-agent reinforcement learning algorithm; where:

301. Initialize the parameter experience pool space of the first fully connected neural network and the second fully connected neural network respectively, wherein each set of data of the initialized experience pool is: D t ={s P,t ,s E,t ,s D,t ,s P,t+1 ,s E,t+1 ,s D,t+1 ,a P,t ,a E,t ,a D,t ,r P,t ,r E,t ,r D,t } Where: s P,t ,s E,t ,s D,t is the current state, s P,t+1 ,s E,t+1 ,s D,t+1 The new state, a P,t ,a E,t ,a D,t Current maneuver value, r P,t ,r E,t ,r D,t Current reward value; 302. Initialize the states of the three spacecraft in each simulation round, and input the initial states of the three spacecraft into the corresponding first fully connected neural network and the second fully connected neural network; wherein: the first fully connected neural network outputs a maneuver value according to the input states of the three spacecraft, and the second fully connected neural network outputs an evaluation value according to the overall input states of the three spacecraft; 303. Inputting the motion control model with the maneuver value output by the first fully connected neural network to obtain the pulse orbit data of the tracking star, the defense star and the escape star of each spacecraft in the LVLH coordinate system; 304. According to the pulse orbit data of each tracking star, defense star and escape star, the following reward function model is input to obtain the optimal strategy from state to action for each tracking star, defense star and escape star; wherein: Distance Reward L : The calculation formula for distance reward is: Among them, α l , α g1 and α g2 is the reward coefficient; at time t, the relative distance between the tracking star and the escape star is defined as △L PE (t)=||r P (t)-r E (t)||2, the relative distance between the defense star and the tracking star is defined as △L DP (t)=||r D (t)-r P (t)||2; Time Reward t : Where: ρ represents the time reward value; for tracking stars and defending stars, if the task is not completed, a fixed negative reward (penalty) will be given at each decision point, while escape stars will receive positive rewards at each decision point; regardless of success or failure, "Done" marks the end of the PED task; Fuel Consumption Reward △v : Terminal Rewards Done : (a) The terminal reward function of the defending star when it successfully captures the escaped star, namely: (b) The terminal reward function of a defense star that successfully intercepts a defense star is: (c) The terminal reward function when the game time exceeds the maximum time, that is: The final total reward is:

305. The optimal strategy from status to action for each tracking star, defense star and escape star is stored in the experience pool; 306. Determine whether the optimal strategy from state to action of each tracking star, defense star and escape star reaches the terminal state. If the terminal state is not reached, the first fully connected neural network and the second fully connected neural network continue to interact with the environment; return to step 303; if the terminal state is reached, the states of the three spacecraft need to be reinitialized; 307. Determine the number of experience pools. If the number of experience pools is reached, update the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm; otherwise, the system continues to collect data and returns to step 302; wherein: the process of updating the first fully connected neural network and the second fully connected neural network through the MAPPO algorithm: The first fully connected neural network is updated using the following formula: Among them, r t (θ) is the probability ratio of the new and old strategies, is the advantage function, which represents the current action a t Compared with strategy π θ Advantages: The first fully connected neural network is updated using the following formula: Where: L critic (θ) represents the state s t The difference between the corresponding true value function and the estimated value, due to s t The corresponding value function V π (s t ) is unknown, θ can be expressed as a function of the neural network weight parameter θ, that is, For a certain trajectory, the state s t The true value function at can be estimated by the following formula:

308. During training, the first fully connected neural network and the second fully connected neural network are updated using a gradient descent method; and the experience pool is cleared after the first fully connected neural network and the second fully connected neural network are updated; 309. The system determines whether the maximum number of training rounds has been reached. If so, the system stops training. Otherwise, the system continues training and returns to step 301. S4. Offline optimization of the "pursuit-defense-escape" game model through the spacecraft active defense game training module obtains an optimized "pursuit-defense-escape" three-party game model; S5. Apply the optimized "pursuit-defense-escape" three-party game model to the online spacecraft three-party game scenario to output the defense game strategy.

Images