Spacecraft swarm game method and system based on generative adversarial reinforcement learning

Abstract

The present invention belongs to the field of aerospace technology, and specifically discloses a spacecraft cluster game method and system based on generative adversarial inverse reinforcement learning, the method comprising: establishing a relative motion dynamics model of the pursuit and escape parties, and then establishing a game relationship model of the pursuit and escape parties based on a two-person zero-sum game model; based on the game relationship model of the pursuit and escape parties, using a differential game method to solve the Nash equilibrium solution of the pursuit and escape game, and generate an expert database; designing a generative adversarial inverse reinforcement learning algorithm based on the expert database; using a generative adversarial inverse reinforcement learning algorithm to solve the game relationship between the pursuit and escape parties, and obtain the optimal game strategy. The present invention combines the optimal control method with the artificial intelligence method to solve the approach strategy of the virtual master star of the spacecraft cluster to the non-cooperative target, solving the problems of the artificial intelligence method being difficult to converge, difficult to set rewards, and poor effect when solving orbital game problems, and finally achieving efficient and accurate approach to the target.

Description

Technical Field

[0001] The present invention belongs to the field of aerospace technology, and specifically relates to a spacecraft cluster game method and system based on generative adversarial reinforcement learning. Background Art

[0002] With the rapid development of space science and technology, the number of spacecraft launches and failures is increasing, and more and more space debris is floating in space, which not only occupies precious orbital resources, but also may collide with normally operating spacecraft, posing a serious threat to their safety. The use of orbital game technology to deal with these potential threats has important economic and military value. The miniaturization and clustering of spacecraft are important directions for the development of future aerospace technology. How to give full play to the number advantages and teamwork capabilities of clusters in orbital games will be more in line with future development trends. The characteristic of a spacecraft cluster is to maintain loose flight within a certain range. When performing a mission, each member spacecraft needs to revolve around the virtual main spacecraft and track the expected trajectory to achieve close approach to the target. Therefore, in the process of orbital game, how to quickly, efficiently and intelligently formulate the optimal game strategy for the virtual main spacecraft has become an important part of determining the outcome of the game.

[0003] There are two main methods to solve this problem. One is traditional methods such as differential games. Although such methods have rigorous theoretical derivations, they rely on many assumptions and have the disadvantages of complex solutions and limited scenarios. The other is artificial intelligence methods such as reinforcement learning. Such methods treat both spacecraft and targets as intelligent agents, put them in a game environment, train the policy network through continuous confrontation, and eventually converge to the optimal policy network. Such methods have the disadvantages of difficult reward function setting, poor generalization, poor interpretability, and weak robustness, and usually cannot guarantee the convergence of the algorithm. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of difficulty in convergence, difficulty in reward setting and poor effect of existing artificial intelligence methods in solving orbital game problems. A spacecraft cluster game method and system based on generative adversarial reinforcement learning is proposed.

[0005] The technical solution of the present invention is: a spacecraft cluster game method based on generative adversarial reinforcement learning, comprising the following steps:

[0006] S1. Establish a relative motion dynamics model between the pursuers and fugitives, and then model the game relationship between the pursuers and fugitives based on game theory to obtain a game relationship model between the pursuers and fugitives;

[0007] S2. Based on the game relationship model of the two parties involved in the pursuit, a Nash equilibrium solution of the game relationship between the two parties involved in the pursuit is solved by using a differential game method to generate an expert database;

[0008] S3. Designing and generating an adversarial reinforcement learning algorithm based on the expert database;

[0009] S4. Use the generative adversarial reinforcement learning algorithm to solve the game relationship between the pursuers and fugitives, and obtain the optimal game strategy for the spacecraft cluster.

[0010] The beneficial effects of the present invention are:

[0011] The present invention combines the optimal control method with the artificial intelligence method to solve the approach strategy of the virtual master of the spacecraft cluster to the non-cooperative target. First, the game relationship problem between the pursuit and escape parties is solved based on the differential game method, and the optimal strategies under different situations are collected to generate an expert data set, which provides expert guidance for generating adversarial reinforcement learning algorithms, thereby greatly accelerating the learning speed of the spacecraft, and the spacecraft can learn a better strategy than that in the expert database, and finally achieve efficient and accurate approach to the target.

[0012] Preferably, the relative motion dynamics model of the chasing and escaping parties in step S1 is expressed as:

[0013]

[0014] in, express The independent variable in direction, express The independent variable in direction, express The first derivative of express The second-order derivative of express The first derivative of express The second-order derivative of represents the cosine function, is the orbital eccentricity, represents the true anomaly angle, express The independent variable in direction, express The second-order derivative of , and Indicates the spacecraft exerts , and Actively controlled acceleration in three directions.

[0015] Preferably, the game relationship model between the two parties in step S1 adopts a two-person zero-sum game model; the state function of the game relationship model between the two parties is:

[0016]

[0017] in, The state variable representing the game relationship between the two parties, express The first derivative of Indicates time, and They represent the strategies of the two parties in pursuit and escape, represents the initial state, Indicates the initial time The state of the game relationship between the two parties in pursuit and escape. represents the state transition function;

[0018] The payment function of the game relationship model between the two parties is:

[0019]

[0020] in, represents the payment function, Indicates about and The quadratic function, It represents the state information set of both parties of the pursuit. represents the time series of the control exerted by the pursuit and fugitive parties, represents the terminal cost function;

[0021] Find the solution to the game relationship between the two parties, that is, to make the game between the two parties take the optimal solution in the game relationship model between the two parties, which can be expressed as:

[0022]

[0023] in, and They represent the optimal solutions for the two parties in pursuit and escape, It is a saddle point, indicating that both the pursuer and the fugitive are in the optimal state in the game relationship between them.

[0024] Preferably, the step S2 of solving the Nash equilibrium solution of the game relationship between the pursuit and escape parties, i.e. solving the saddle point, comprises the following steps:

[0025] Construct the Hamiltonian function of the game relationship between the two parties;

[0026] Based on the Hamiltonian function, the existence condition of the Nash equilibrium solution is obtained;

[0027] Construct a two-point boundary value problem based on the existence conditions of Nash equilibrium solutions, state functions and initial state information;

[0028] Solve the two-point boundary value problem and obtain the Nash equilibrium solution.

[0029] Preferably, the Hamiltonian function is:

[0030]

[0031] in, represents the Hamiltonian function, represents the Lagrangian operator, represents a co-variable, Represents co-variable The transpose of represents the state transition function, represents the control force of the first spacecraft, Indicates The control force of a spacecraft;

[0032] The existence conditions of the Nash equilibrium solution include the co-state equation, the optimal control equation and the transversal condition;

[0033] The co-state equation is:

[0034]

[0035] in, represents the first derivative of the covariate variable;

[0036] The optimal control equation is:

[0037] ;

[0038] The cross-sectional condition is:

[0039]

[0040] in, express The covariate variables at time, represents the performance indicator function, represents the 3D Lagrange multiplier vector, represents the 3D Lagrange multiplier vector The transpose of Represents the terminal states and constraints that are satisfied at the time.

[0041] Preferably, the generative adversarial inverse reinforcement learning algorithm in step S3 is designed based on a generative adversarial network; the generative adversarial network includes a generator and a discriminator;

[0042] The generator is used to generate sampled trajectories related to expert trajectories in the expert database;

[0043] The discriminator is used to receive the sampled trajectory and the expert trajectory in the expert database, and identify the sampled trajectory mixed in the expert trajectory.

[0044] Preferably, the step S3 specifically comprises the following steps:

[0045] Generate sample trajectories through the generator ;

[0046] The expert trajectory in the expert database and sampling trajectory Perform mixed sampling and input it into the discriminator for discrimination to obtain the output of the discriminator;

[0047] Process the output of the discriminator and calculate the reward value of the sampled trajectory state action pair;

[0048] The generative adversarial network is iteratively updated using the reward value until the discriminator cannot determine whether the sampled trajectory comes from the expert trajectory, thereby obtaining a generative adversarial inverse reinforcement learning algorithm.

[0049] Preferably, the loss function of the discriminator during training is:

[0050]

[0051] in, represents the loss function of the discriminator; Indicates status Next, take action The value output by the discriminator, represents the logarithmic function with the natural base e as the base, It indicates that the discriminator judges that the trajectory state comes from the expected value of the expert trajectory, It indicates that the discriminator judges that the trajectory state comes from the expected value of the sampled trajectory;

[0052] The calculation formula of the reward value of the sampled trajectory state action pair is:

[0053]

[0054] in, Indicates status Next, take action The reward value, Indicates status Next, take action The value output by the discriminator is Represents the logarithmic function with the natural base e as the base.

[0055] The beneficial effects of the above preferred solution are:

[0056] The above-mentioned preferred scheme constructs a Hamiltonian function, solves the two-point boundary value problem to obtain the Nash equilibrium solution of the two game players, and constructs a generative adversarial inverse reinforcement learning algorithm through the continuous interaction of the generator and the discriminator, and uses the discriminator to calculate the reward value of the sampled trajectory state-action pair. It not only overcomes the shortcomings of complex differential game solutions and limited scenarios, but also solves the problem that the reward function of the deep reinforcement learning method is difficult to set, and effectively improves the weak robustness of generative adversarial inverse reinforcement learning in dealing with spacecraft cluster games, and can efficiently and accurately obtain the optimal strategy.

[0057] In a second aspect, a spacecraft cluster game system based on generative adversarial inverse reinforcement learning is provided, the system comprising a processor, the processor being used to execute the spacecraft cluster game method based on generative adversarial inverse reinforcement learning as described in the first aspect.

[0058] In a third aspect, a computer-readable storage medium stores computer instructions. In response to a computer reading the computer instructions in the storage medium, the computer executes the spacecraft cluster game method based on generative adversarial reinforcement learning as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Shown is a flowchart of a spacecraft cluster game method based on generative adversarial reinforcement learning.

[0060] Figure 2 Shown is a diagram showing the relationship between differential games and generative adversarial reinforcement learning.

[0061] Figure 3 Shown is a diagram of the training process structure of generative adversarial reinforcement learning. DETAILED DESCRIPTION

[0062] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0063] Embodiment 1:

[0064] like Figure 1 As shown in the figure, a spacecraft cluster game method based on generative adversarial reinforcement learning is proposed. The game objects are space debris, invalid spacecraft and other non-cooperative targets. The game fully considers the situation that the intention, strategy and maneuverability of non-cooperative targets are unclear, and the main spacecraft can efficiently approach the target. The specific steps include:

[0065] S1. Establish a relative motion dynamics model between the pursuers and fugitives, and then model the game relationship between the pursuers and fugitives based on game theory to obtain a game relationship model between the pursuers and fugitives;

[0066] S2. Based on the game relationship model of the two parties, the Nash equilibrium solution of the game relationship between the two parties is solved by using the differential game method, the maneuver strategy sequence of the two parties is obtained, and an expert database is generated;

[0067] S3. Designing and generating an adversarial reinforcement learning algorithm based on the expert database;

[0068] S4. Use the generative adversarial reinforcement learning algorithm to solve the game relationship between the pursuers and fugitives, and obtain the optimal game strategy for the spacecraft cluster.

[0069] In this embodiment, the relative motion dynamics model of the chasing and escaping parties in step S1 is expressed as:

[0070]

[0071] in, express The independent variable in direction, express The independent variable in direction, express The first derivative of express The second-order derivative of express The first derivative of express The second-order derivative of represents the cosine function, is the orbital eccentricity, represents the true anomaly angle, express The independent variable in direction, express The second-order derivative of , and Indicates the spacecraft exerts , and Actively controlled acceleration in three directions.

[0072] In this embodiment, the game relationship model between the two parties in step S1 adopts a two-person zero-sum game model; the state function of the game relationship model between the two parties is:

[0073]

[0074] in, The state variable representing the game relationship between the two parties, express The first derivative of Indicates time, and They represent the strategies of the two parties in pursuit and escape, represents the initial state, Indicates the initial time The state of the game relationship between the two parties in pursuit and escape. represents the state transition function;

[0075] The payment function of the game relationship model between the two parties is:

[0076]

[0077] in, represents the payment function, Indicates about and The quadratic function, It represents the state information set of both parties of the pursuit. represents the time series of the control exerted by the pursuit and fugitive parties, represents the terminal cost function;

[0078] The game relationship between the two parties of the spacecraft cluster pursuit, namely the orbital game, is a competitive and conflicting game problem. The interests of the two parties are fundamentally contradictory, and the game result of the two parties is a zero-sum relationship. To solve the game relationship between the two parties, even if the two parties take the optimal solution, it can be expressed as:

[0079]

[0080] in, and They represent the optimal solutions for both parties. When , the game strategies of both parties can be optimal at the same time, called Saddle point. In a broad sense, a saddle point can represent a critical value on a smooth surface, which is neither a maximum nor a minimum. In differential games, it means that both parties are in the optimal state. If one party is not here, both parties will not be able to reach the optimal state at the same time.

[0081] In this embodiment, the step S2 of solving the Nash equilibrium solution of the game relationship between the pursuit and escape parties is to solve the saddle point; the solution of the Nash equilibrium solution includes the following steps:

[0082] The Hamiltonian function of the game relationship between the two parties of pursuit and escape is constructed according to the calculus of variations and the Pontryagin maximum principle. The Hamiltonian function is:

[0083]

[0084] in, represents the Hamiltonian function, represents the Lagrangian operator, represents a co-variable, Represents co-variable The transpose of represents the state transition function, represents the control force of the first spacecraft, Indicates The control force of a spacecraft;

[0085] According to the maximum-minimum principle, taking the maximum or minimum value of the payoff function can be transformed into taking the maximum or minimum value of the Hamiltonian function. Therefore, based on the Hamiltonian function, the existence condition of the Nash equilibrium solution is obtained; the existence condition of the Nash equilibrium solution includes the co-state equation, the optimal control equation and the transversal condition;

[0086] The co-state equation is:

[0087]

[0088] in, represents the first derivative of the covariate variable;

[0089] The optimal control equation is:

[0090] ;

[0091] The cross-sectional condition is:

[0092]

[0093] in, express The covariate variables at time, represents the performance indicator function, represents the 3D Lagrange multiplier vector, represents the 3D Lagrange multiplier vector The transpose of Represents the constraints that are satisfied at the terminal state and time;

[0094] A two-point boundary value problem is constructed based on the existence conditions of the Nash equilibrium solution, the state function and the initial state information; and the two-point boundary value problem is solved to obtain the Nash equilibrium solution.

[0095] In this embodiment, the generative adversarial inverse reinforcement learning algorithm in step S3 is designed based on a generative adversarial network; the generative adversarial network includes a generator and a discriminator; the generator is used to generate sampling trajectories related to expert trajectories in an expert database; the discriminator is used to receive the sampling trajectories and the expert trajectories in the expert database, and identify the sampling trajectories mixed in the expert trajectories.

[0096] The generator and the discriminator supervise each other during the training process and update iteratively. The final effect is that the sampling trajectory generated by the generator is similar to the expert trajectory in the expert database, and the discriminator has difficulty distinguishing the difference between the expert trajectory and the sampling trajectory. Figure 2 As shown in Figure 1. Generative adversarial inverse reinforcement learning does not have an explicit reward function and requires an external discriminator network to provide reward values ​​for training. Therefore, when training generative adversarial inverse reinforcement learning, after receiving the reward values ​​of the trajectory state action pairs given by the discriminator network, the training of the generator network completely follows the algorithm flow of reinforcement learning to update the network.

[0097] like Figure 3 As shown, step S3 specifically includes the following steps: generating a sampling trajectory through a generator ; The expert trajectory in the expert database and sampling trajectory Mixed sampling is performed and input into the discriminator for discrimination to obtain the output of the discriminator; the output of the discriminator is processed to calculate the reward value of the sampled trajectory state-action pair; the generative adversarial network is iteratively updated using the reward value until the discriminator cannot determine whether the sampled trajectory comes from the expert trajectory, thereby obtaining a generative adversarial inverse reinforcement learning algorithm.

[0098] In this embodiment, the loss of the discriminator during training is the average logarithmic probability assigned to the correct classification. This loss is evaluated on the mixed trajectory of the real sample and the generator output. The loss function is:

[0099]

[0100] in, represents the loss function of the discriminator; Indicates status Next, take action The value output by the discriminator, represents the logarithmic function with the natural base e as the base, It indicates that the discriminator judges that the trajectory state comes from the expected value of the expert trajectory, It indicates that the discriminator judges that the trajectory state comes from the expected value of the sampled trajectory;

[0101] The task of the discriminator is to distinguish whether the state-action pair of the trajectory comes from the expert trajectory. Therefore, for the state-action pair input from the expert trajectory, its output should be as close to 1 as possible, and for the trajectory input from the generator, its output should be as close to 0 as possible. That is, the discriminator can be regarded as a binary classification unit. As the training time increases, if the training proceeds normally, the data generated by the generator will continue to approach the expert data. The calculation formula for the reward value of the sampled trajectory state-action pair is:

[0102]

[0103] in, Indicates status Next, take action The reward value, Indicates status Next, take action The value output by the discriminator is Represents the logarithmic function with the natural base e as the base.

[0104] Embodiment 2:

[0105] On the basis of Example 1, this embodiment provides a spacecraft cluster game system based on generative adversarial inverse reinforcement learning, which is used to configure and execute a spacecraft cluster game method based on generative adversarial inverse reinforcement learning in Example 1.

[0106] In this embodiment, the system may be an electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, and the processor executes the program to implement part or all of the steps of the spacecraft cluster game method based on generative adversarial inverse reinforcement learning as described in Example 1.

[0107] In this embodiment, the electronic device may include: a processor, a memory, a bus and a communication interface. The processor, the communication interface and the memory are connected through a bus. The memory stores a computer program that can be run on the processor. When the processor runs the computer program, it executes part or all of the steps of the spacecraft cluster game method based on generative adversarial inverse reinforcement learning provided in the aforementioned embodiment 1 of the present application.

[0108] The system in the embodiment of the present invention can also be a computer-readable storage medium, which stores a computer program. When the computer program is executed, some or all steps of the spacecraft cluster game method based on generative adversarial reinforcement learning as described in Example 1 are implemented.

[0109] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0114] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A spacecraft cluster game method based on generative adversarial reinforcement learning, characterized in that: The following steps are involved: S1. Establish a relative motion dynamics model between the pursuers and fugitives, and then model the game relationship between the pursuers and fugitives based on game theory to obtain a game relationship model between the pursuers and fugitives; S2. Based on the game relationship model of the two parties involved in the pursuit, a Nash equilibrium solution of the game relationship between the two parties involved in the pursuit is solved by using a differential game method to generate an expert database; S3. Designing and generating an adversarial reinforcement learning algorithm based on the expert database; S4. Using the generative adversarial reinforcement learning algorithm to solve the game relationship between the pursuit and escape parties, and obtain the optimal game strategy of the spacecraft cluster; The relative motion dynamics model of the two parties in step S1 is expressed as: in, Represents a three-dimensional Cartesian coordinate system The independent variable in direction, Represents a three-dimensional Cartesian coordinate system The independent variable in direction, express The first derivative of express The second-order derivative of express The first derivative of express The second-order derivative of represents the cosine function, is the orbital eccentricity, represents the true anomaly angle, Represents a three-dimensional Cartesian coordinate system The independent variable in direction, express The second-order derivative of , and Indicates the spacecraft exerts , and Actively controlled acceleration in three directions; The generative adversarial inverse reinforcement learning algorithm in step S3 is designed based on a generative adversarial network; the generative adversarial network includes a generator and a discriminator; The generator is used to generate sample trajectories related to expert trajectories in the expert database; The discriminator is used to receive the sampled trajectory and the expert trajectory in the expert database, and identify the sampled trajectory mixed in the expert trajectory.

2. The spacecraft cluster game method based on generative adversarial reinforcement learning according to claim 1 is characterized in that: The game relationship model between the two parties in step S1 adopts a two-person zero-sum game model; the state function of the game relationship model between the two parties is: in, The state variable representing the game relationship between the two parties, express The first derivative of Indicates time, and They represent the strategies of the two parties in pursuit and escape, represents the initial state, Indicates the initial time The state of the game relationship between the two parties in pursuit and escape. represents the state transition function; The payment function of the game relationship model between the two parties is: in, represents the payment function, Indicates about and The quadratic function, It represents the state information set of both parties of the pursuit. represents the time series of the control exerted by the pursuit and fugitive parties, represents the terminal cost function; Find the solution to the game relationship between the two parties, that is, to make the game between the two parties take the optimal solution in the game relationship model between the two parties, which can be expressed as: in, and They represent the optimal solutions for the two parties in pursuit and escape, It is a saddle point, indicating that both the pursuer and the fugitive are in the optimal state in the game relationship between them.

3. The spacecraft cluster game method based on generative adversarial reinforcement learning according to claim 2 is characterized in that: The step S2 is to solve the Nash equilibrium solution of the game relationship between the two parties of the pursuit, that is, to solve the saddle point; the solution of the Nash equilibrium solution includes the following steps: Construct the Hamiltonian function of the game relationship between the two parties; Based on the Hamiltonian function, the existence condition of the Nash equilibrium solution is obtained; Construct a two-point boundary value problem based on the existence conditions of Nash equilibrium solutions, state functions and initial state information; Solve the two-point boundary value problem and obtain the Nash equilibrium solution.

4. The spacecraft cluster game method based on generative adversarial reinforcement learning according to claim 3 is characterized in that: The Hamiltonian function is: in, represents the Hamiltonian function, represents the Lagrangian operator, represents a co-variable, Represents co-variable The transpose of represents the state transition function, represents the control force of the first spacecraft, Indicates The control force of a spacecraft; The existence conditions of the Nash equilibrium solution include the co-state equation, the optimal control equation and the transversal condition; The co-state equation is: in, represents the first derivative of the covariate variable; The optimal control equation is: ； The cross-sectional condition is: in, express The covariate variables at time, represents the performance indicator function, represents the 3D Lagrange multiplier vector, represents the 3D Lagrange multiplier vector The transpose of Represents the terminal states and constraints that are satisfied at the time.

5. The spacecraft cluster game method based on generative adversarial reinforcement learning according to claim 1 is characterized in that: The step S3 specifically comprises the following steps: Generate sample trajectories through the generator ; The expert trajectory in the expert database and sampling trajectory Perform mixed sampling and input it into the discriminator for discrimination to obtain the output of the discriminator; Process the output of the discriminator and calculate the reward value of the sampled trajectory state action pair; The generative adversarial network is iteratively updated using the reward value until the discriminator cannot determine whether the sampled trajectory comes from the expert trajectory, thereby obtaining a generative adversarial inverse reinforcement learning algorithm.

6. The spacecraft cluster game method based on generative adversarial reinforcement learning according to claim 5 is characterized in that: The loss function of the discriminator during training is: in, represents the loss function of the discriminator; Indicates status Next, take action The value output by the discriminator, represents the logarithmic function with the natural base e as the base, It indicates that the discriminator judges that the trajectory state comes from the expected value of the expert trajectory, It indicates that the discriminator judges that the trajectory state comes from the expected value of the sampled trajectory; The calculation formula of the reward value of the sampled trajectory state action pair is: in, Indicates status Next, take action The reward value, Indicates status Next, take action The value output by the discriminator is Represents the logarithmic function with the natural base e as the base.

7. A spacecraft cluster game system based on generative adversarial reinforcement learning, characterized in that: It includes a processor, which is used to execute the spacecraft cluster game method based on generative adversarial inverse reinforcement learning as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions. In response to the computer reading the computer instructions in the storage medium, the computer executes the spacecraft cluster game method based on generative adversarial reinforcement learning as described in any one of claims 1 to 6.

Images