A game optimization method, device, equipment and medium based on spacecraft

Abstract

A game optimization method, device, equipment and medium based on spacecraft, including obtaining the current state quantity of the tracker and the current state quantity of the escaper in a preset pursuit and escape game scenario, wherein the state quantity includes position and speed; inputting the current state quantity of the tracker into the tracker agent trained by the agent training method to obtain the current action quantity of the tracker, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle; according to the current action quantity of the tracker and the current action quantity of the escaper, the tracker and the escaper are controlled to play the pursuit and escape game. The technical problem that game optimization is usually designed for non-game unilateral training optimization scenarios, which can only allow the agent to learn to play games, but cannot achieve the optimal game effect, is solved.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft control technology, and in particular to a game optimization method, device, equipment and medium based on a spacecraft. Background Art

[0002] All kinds of game problems in continuous space, such as air combat, missile interception, and spacecraft games, can be modeled as differential game problems. The core of the problem is to solve the optimal strategy of both parties in the game - Nash equilibrium. The traditional differential game solution is to complete the optimal strategy solution by building a model, deriving the optimal conditions, and solving mathematical equations, but there are problems such as complex process, limited solution ability, and large model constraints. The intelligent game method combines game theory ideas with deep reinforcement learning methods to show greater advantages. Game theory provides deep reinforcement learning methods with the concept of understanding and the principle of equilibrium. Deep reinforcement learning can bring computational advantages in solving game solutions. The goal of intelligent game is to allow two opposing intelligent agents to learn better strategies in the game only through data collection and training, without solving complex mathematical equations. However, most artificial intelligence algorithms are essentially designed for non-game unilateral training optimization scenarios. If this algorithm is applied to both parties in the game at the same time, it can only allow the intelligent agent to learn the game, but cannot achieve the optimal game effect. The strategies of both parties will continue to change due to changes in the opponent's strategy, and it is difficult to guarantee that they will converge to the Nash equilibrium in the end. Summary of the invention

[0003] The main technical problem solved by the present invention is that game optimization is usually designed for non-game unilateral training optimization scenarios, which can only enable the intelligent agent to learn to play games, but cannot achieve the optimal game effect.

[0004] According to the first aspect, an embodiment provides a game optimization method based on a spacecraft, comprising:

[0005] Obtain the current state of the tracker and the current state of the escaper in the preset pursuit-escape game scenario, where the state includes position and speed;

[0006] Inputting the current state quantity of the tracker into the tracker agent trained by the agent training method to obtain the current action quantity of the tracker, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle;

[0007] Controlling the tracker and the escaper to play a chase-and-escape game according to the current movement amount of the tracker and the current movement amount of the escaper;

[0008] Wherein, the agent training method comprises:

[0009] Constructing a Markov game model and an experience pool; wherein the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent during the game, and the experience pool includes experience data corresponding to the state quantity, action quantity, reference function and reference factor;

[0010] Construct a learner, and construct a main network and a target network based on the learner; wherein the learner includes a value neural network and a strategy neural network corresponding to each of the intelligent agents in the game process, the main network is constructed according to the parameters of the value neural network and the strategy neural network, the main network includes a main value network and a main strategy network, the target network is determined according to the network structure and network parameters of the main network, and the target network includes a target value network and a target strategy network;

[0011] Constructing a pair of intelligent agents to be trained, using the Markov game model as a training environment, and obtaining target experience data of the pair of intelligent agents to be trained after interacting with the environment in the training environment; wherein the pair of intelligent agents to be trained includes a tracker intelligent agent and an escaper intelligent agent that play a pursuit-escape game;

[0012] The target experience data is stored in the experience pool, the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network;

[0013] Calculate and obtain the deterministic symplectic policy gradients corresponding to the plurality of pieces of experience data, and update the parameters of the policy neural network in the learner according to the deterministic symplectic policy gradients, the main policy network, and the target policy network;

[0014] Until the updated value neural network and the strategy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and the strategy neural network that meet the preset convergence requirements are used as the network parameters of the to-be-trained intelligent agent pair to obtain a trained intelligent agent pair.

[0015] In some embodiments, the updating of parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network includes:

[0016] Calculating the gradient of the value loss function with respect to the network parameters in the value neural network;

[0017] Determining the update amplitude of the network parameters according to the gradient, the learning rate of the value neural network and the gradient descent method;

[0018] The parameter direction of the main value network is used as the update direction, and the parameters of the target value network are updated in combination with the update amplitude. The parameters of the value neural network in the learner are updated according to the parameter update process of the target value network.

[0019] In some embodiments, the calculating to obtain the deterministic symplectic policy gradient corresponding to the plurality of empirical data includes:

[0020] Constructing a Hessian matrix corresponding to the policy network parameters of the value neural network and the policy neural network;

[0021] A deterministic symplectic policy gradient is generated according to the expansion corresponding to the multiple second-order derivatives of the value neural network and the Hessian matrix.

[0022] In some embodiments, obtaining target experience data of the to-be-trained agent after interacting with the environment in the training environment includes:

[0023] Obtaining the state quantity, reward value and discount factor of the to-be-trained agent in the training environment;

[0024] Calculate the corresponding action amount according to the strategy neural network in the intelligent agent to be trained and the state amount in the training environment;

[0025] Determining a state quantity at a next moment corresponding to the state quantity in the training environment according to a state transfer function in the training environment;

[0026] The state quantity in the training environment, the reward value, the discount factor, the action quantity, and the state quantity at the next moment corresponding to the state quantity in the training environment are summarized as target experience data.

[0027] In some embodiments, the step of calculating the corresponding action amount based on the policy neural network in the agent to be trained and the state amount in the training environment includes:

[0028] ,

[0029] in, Indicates the amount of action, represents the policy neural network, represents the state quantity in the training environment, represents the standard deviation, represents the standard normal distribution.

[0030] In some embodiments, the step of constructing a main network and a target network according to the learner includes:

[0031] Initializing the parameters of the value neural network in the learner and the parameters of the strategy neural network corresponding to each of the intelligent agents respectively, and constructing the main network according to the initialized parameters;

[0032] A target network is constructed based on the network structure and parameters of the main network and a preset update rate.

[0033] In some embodiments, the reference function includes a state transfer function and a reward function, and the reference factor includes a discount factor.

[0034] According to the second aspect, an embodiment provides a game optimization device based on a spacecraft, comprising:

[0035] A data preparation module is used to obtain the current state quantity of the tracker and the current state quantity of the escaper in a preset pursuit-escape game scenario, where the state quantity includes position and speed;

[0036] An action amount determination module is used to input the current state amount of the tracker into the tracker agent trained by the agent training method to obtain the current action amount of the tracker, and input the current state amount of the escaper into the escaper agent trained by the agent training method to obtain the current action amount of the escaper; wherein the action amount includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle;

[0037] A pursuit-escape game module, used to control the tracker and the escaper to play a pursuit-escape game according to the current action amount of the tracker and the current action amount of the escaper;

[0038] An agent training module is used to construct a Markov game model and an experience pool; wherein the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent during the game, and the experience pool includes experience data corresponding to the state quantity, the action quantity, the reference function and the reference factor; construct a learner, and construct a main network and a target network based on the learner; wherein the learner includes a value neural network and a strategy neural network corresponding to each of the agents during the game, the main network is constructed according to the parameters of the value neural network and the strategy neural network, the main network includes a main value network and a main strategy network, the target network is determined according to the network structure and network parameters of the main network, and the target network includes a target value network and a target strategy network; construct a pair of agents to be trained, use the Markov game model as a training environment, and obtain the target experience of the pair of agents to be trained after interacting with the environment in the training environment data; wherein the pair of agents to be trained includes a tracker agent and an escaper agent that play a pursuit-escape game; the target experience data is stored in the experience pool, the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network; the deterministic symplectic policy gradient corresponding to the multiple experience data is calculated, and the parameters of the policy neural network in the learner are updated according to the deterministic symplectic policy gradient, the main policy network and the target policy network; until the updated value neural network and the policy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and the policy neural network that meet the preset convergence requirements are used as the network parameters of the pair of agents to be trained to obtain a trained pair of agents.

[0039] According to the third aspect, an embodiment provides a game optimization device based on a spacecraft, including:

[0040] Memory, used to store programs;

[0041] A processor is used to implement a spacecraft-based game optimization method by executing the program stored in the memory.

[0042] According to the fourth aspect, an embodiment provides a computer-readable storage medium, characterized in that a program is stored on the medium, and the program can be executed by a processor to implement a spacecraft-based game optimization method.

[0043] According to the spacecraft-based game optimization method, device, equipment and computer-readable storage medium of the above-mentioned embodiment, since the trained tracker agent and the trained escaper agent are trained by the agent training method, the current state quantity of the tracker is input into the tracker agent trained by the agent training method to obtain the current action quantity of the tracker, and the current state quantity of the escaper is input into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper. The agent training method designs a learning structure of value sharing and strategy independence for both parties in the game, and incorporates the idea of ​​deterministic symplectic policy gradient, which improves the convergence ability and accuracy of agent training, so that the tracker and the escaper can be accurately controlled to play the pursuit and escape game according to the current action quantity of the tracker and the current action quantity of the escaper. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a game optimization method based on a spacecraft according to an embodiment of the present application;

[0045] Figure 2 is a flow chart of an agent training method in an embodiment;

[0046] Figure 3 A flowchart of constructing a main network and a target network according to a learner in one embodiment;

[0047] Figure 4 A flowchart of obtaining target experience data of each intelligent agent to be trained after interacting with the environment in the training environment in one embodiment;

[0048] Figure 5 A flowchart of updating parameters of a value neural network in a learner according to a value loss function, a main value network, and a target value network corresponding to a value distribution in an embodiment;

[0049] Figure 6 A flowchart of calculating a deterministic symplectic policy gradient corresponding to a plurality of empirical data in an embodiment;

[0050] Figure 7a and Figure 7b D4P2G algorithm, D4SPG algorithm and DDPG algorithm in one embodiment respectively control the direction angle of the spacecraft pulse and The training results;

[0051] Figure 8 The final converged solution obtained by using the D4P2G algorithm and the DDPG algorithm in one embodiment;

[0052] Figure 9a and Figure 9bA reference diagram for testing the training effect of a Nash equilibrium controller under different initial example configurations in an embodiment;

[0053] Fig.10 The present invention is a schematic diagram of the structure of a game optimization device based on a spacecraft according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0055] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0056] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0057] Please refer to Figure 1 In an embodiment of the present invention, a game optimization method based on a spacecraft is proposed, including steps S10 to S30, which are described in detail below.

[0058] Step S10: Obtain the current state quantity of the tracker and the current state quantity of the escaper in the preset pursuit-escape game scenario, wherein the state quantity includes position and speed.

[0059] Step S20: Input the current state quantity of the tracker into the tracker agent trained by the agent training method to obtain the current motion quantity of the tracker, and input the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current motion quantity of the escaper; wherein the motion quantity includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle.

[0060] Please refer to Figure 2 In some embodiments, the agent training method includes steps S21 to S26, which are described in detail below.

[0061] Step S21: Constructing a Markov game model and experience pool.

[0062] In some embodiments, the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent during the game process, wherein the reference function includes a state transfer function and a reward function, and the reference factor includes a discount factor.

[0063] In some embodiments, the description node of the selected orbital coordinate system k Moments, Trackers and Escapers n- The dimensional state vectors are and , then the state quantity in the Markov game model can be expressed by the state vectors of the tracker and the escaper as: .

[0064] In some embodiments, the action amount in the Markov game model can be expressed as the following formula:

[0065] ,

[0066] in, and represents the amount of action in the Markov game model, To control the amount of Represents a tracker, Indicates an escape device.

[0067] In some embodiments, in a dynamic system described by a second-order differential equation, the control quantity in the differential game is and is the acceleration vector, which can be directly used as the action quantity in the Markov game model.

[0068] In some embodiments, the action amount in the Markov game model can also be expressed as , in-plane direction angle and the out-of-plane orientation angle It is expressed as the following formula:

[0069] ,

[0070] In some embodiments, the state transfer function in the reference function can be expressed as the following formula:

[0071] ,

[0072] in, Indicated in The state of the time node, Indicated in k The amount of movement of the tracker at the time node, Indicated in k The amount of action of the escaper at the time node.

[0073] In some embodiments, the state transfer function can be converted by integrating the state differential equation in the differential game, that is, from one discrete time node to another node. The state differential equation can be expressed as the following formula:

[0074] ,

[0075] in, The ordinary differential equation representing the differential game, Represents the state transition function.

[0076] In some embodiments, the reward function in the reference function can be expressed as the following formula:

[0077] ,

[0078] ,

[0079] in, express The payoff function for the time period trajectory, Indicates the total number of time nodes, Indicates a time node. Indicates time node The terminal function at Indicates time node The Lagrangian function at Indicates time node The rewards of Indicates time node The discount factor at .

[0080] In some embodiments, the reference factor includes a discount factor, which is a The multiplier between them, where the cumulative discounted reward function is the same as the payment function of the differential game, and the discount factor is 1.

[0081] In some embodiments, the experience pool is used to store experience data corresponding to state quantities, action quantities, reference functions, and reference factors.

[0082] Step S22: construct a learner, and construct a main network and a target network based on the learner.

[0083] In some embodiments, the learner includes a value neural network And the strategy neural network corresponding to each agent in the game process, where the game process includes the tracker agent and the escaper agent, then the strategy neural network corresponding to the tracker agent is , the strategy neural network corresponding to the escaper agent is .

[0084] Please refer to Figure 3 In some embodiments, step S22 constructs a main network and a target network according to the learner, including steps S221 to S222, which are described in detail below.

[0085] Step S221: Initialize the parameters of the value neural network in the learner and the parameters of the strategy neural network corresponding to each agent respectively, and build the main network according to the initialized parameters.

[0086] In some embodiments, the parameters of the value neural network in the learner and the parameters of the policy neural network corresponding to each agent are initialized as , and , according to the parameters after initialization , and Build the main network, which includes the main value network and the main strategy network.

[0087] Step S222: constructing a target network based on the network structure and parameters of the main network and a preset update rate.

[0088] In some embodiments, the target network is determined according to the network structure and network parameters of the main network. The target network is constructed with reference to the network structure of the main network, and the network parameters of the main network are copied to obtain the network parameters of the target network. The update rate of the target network is slower than that of the main network. The target network includes a target value network and a target strategy network. The network parameters of the target network include , , .

[0089] Step S23: construct a pair of intelligent agents to be trained, use the Markov game model as the training environment, and obtain target experience data after the pair of intelligent agents to be trained interact with the environment in the training environment.

[0090] In some embodiments, the agent pair to be trained includes a tracker agent and a evader agent that play a pursuit-escape game.

[0091] Please refer to Figure 4 In some embodiments, step S23 obtains target experience data of each agent to be trained after interacting with the environment in the training environment, including steps S231 to S234, which are described in detail below.

[0092] Step S231: Obtain the state quantity, reward value and discount factor of the agent to be trained in the training environment.

[0093] Step S232: Calculate the corresponding action amount based on the policy neural network in the agent to be trained and the state quantity in the training environment.

[0094] In some embodiments, the corresponding action amount is calculated based on the policy neural network in the agent to be trained and the state amount in the training environment, including:

[0095] ,

[0096] in, Indicates the amount of action, represents the policy neural network, represents the state quantity in the training environment, represents the standard deviation, represents the standard normal distribution.

[0097] Step S233: Determine the state quantity at the next moment corresponding to the state quantity in the training environment according to the state transfer function in the training environment.

[0098] Step S234: Summarize the state quantity, reward value, discount factor, action quantity in the training environment and the state quantity at the next moment corresponding to the state quantity in the training environment as target experience data.

[0099] Step S24: Store the target experience data in the experience pool, the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function, main value network and target value network corresponding to the value distribution.

[0100] In some embodiments, before training the tracker agent and the escaper agent for the pursuit-escape game, the training parameters are initialized, that is, the batch size is initialized. , N The step returns the length, the number of pursuer and escaper agent pairs, the exploration constant, the learning rate of the value neural network, and the learning rate of the policy neural network.

[0101] In some embodiments, the experience pool stores target experience data and pre-stored experience data, and the data in the experience pool is continuously updated based on the experience data recorded when the agent interacts with the environment. M When batches of data are collected, the value distribution corresponding to the empirical data is calculated according to the value distribution function. The formula of the value distribution function is as follows:

[0102] ,

[0103] in, represents the future value distribution predicted based on empirical data, Indicates the total number of returned lengths. represents the value neural network in the learner, represents the target value network in the target network, Indicates The state quantity of the step return length, represents the network parameters of the policy neural network corresponding to the tracker agent, represents the network parameters of the policy neural network corresponding to the escaper agent, express N Step length n The node's reward value, represents the discount factor, represents the Bellman operator.

[0104] In some embodiments, a cross entropy loss function is defined to calculate the difference between the current value distribution and the old value distribution corresponding to the empirical data. The cross entropy loss function is as follows:

[0105] ,

[0106] in, represents the cross entropy loss function, represents the Bellman operator, and also represents the future value distribution predicted based on empirical data, represents the value neural network in the learner, represents the state quantity in the Markov game model, Indicated in k The amount of movement of the tracker at the time node, Indicated in k The amount of action of the escaper at the time node.

[0107] Please refer to Figure 5 In some embodiments, step S24 updates the parameters of the value neural network in the learner according to the value loss function, the main value network and the target value network corresponding to the value distribution, including steps S241 to S243, which are described in detail below.

[0108] Step S241: Calculate the gradient of the value loss function with respect to the network parameters in the value neural network.

[0109] Step S242: Determine the update amplitude of the network parameters based on the gradient, the learning rate of the value neural network and the gradient descent method.

[0110] Step S243: Taking the parameter direction of the main value network as the update direction, the parameters of the target value network are updated in combination with the update amplitude, and the parameters of the value neural network in the learner are updated according to the parameter update process of the target value network.

[0111] In some embodiments, the parameter update of the target value network is implemented by the following formula:

[0112] ,

[0113] in, represents the updated network parameters, Indicates the current network parameters. represents the learning rate of the value neural network, Represents the update amplitude determined by the gradient.

[0114] In some embodiments, the target value network is updated slowly using the parameter direction of the main value network as the update direction:

[0115] ,

[0116] in, represents the network parameters of the target value network in the target network, represents the exploration constant.

[0117] Step S25: Calculate the deterministic symplectic policy gradient corresponding to the multiple pieces of experience data, and update the parameters of the policy neural network in the learner according to the deterministic symplectic policy gradient, the main policy network and the target policy network.

[0118] Please refer to Figure 6 In some embodiments, step S25 calculates the deterministic symplectic policy gradient corresponding to multiple pieces of experience data, including steps S251 to S252, which are described in detail below.

[0119] Step S251: construct a Hessian matrix corresponding to the policy network parameters of the value neural network and the policy neural network.

[0120] In some embodiments, the Hessian matrix corresponding to the policy network parameters of the value neural network and the policy neural network is shown in the following formula:

[0121] ,

[0122] ,

[0123] ,

[0124] ,

[0125] ,

[0126] in, Q represents the expected value of the value function, , They represent the parameters of the policy neural network, represents the policy neural network corresponding to the tracker agent, represents the policy neural network corresponding to the escaper agent, and Represents multiple second-order derivatives of a value neural network.

[0127] Step S252: Generate a deterministic symplectic policy gradient based on the expansion corresponding to multiple second-order derivatives of the value neural network and the Hessian matrix.

[0128] In some embodiments, the deterministic symplectic policy gradient may refer to the following formula:

[0129] ,

[0130] ,

[0131] in, represents the deterministic symplectic policy gradient of the tracker agent, represents the deterministic symplectic policy gradient of the escaper agent, and Represents multiple second-order derivatives of a value neural network.

[0132] In some embodiments, the policy neural network in the learner is parameterized according to the deterministic symplectic policy gradient, the main policy network, and the target policy network, and the parameter update is performed with reference to the following formula:

[0133] ,

[0134] in, represents the updated parameters, Indicates the current network parameters. represents the learning rate of the policy neural network, represents the deterministic symplectic policy gradient, P represents the tracker agent, represents an escaper agent.

[0135] In some embodiments, the policy neural network in the learner is slowly updated along the parameter direction of the main policy network, and the following formula can be referred to:

[0136] ,

[0137] ,

[0138] in, , represents the parameters of the policy neural network, and represents the network parameters of the target network, represents the exploration constant.

[0139] Step S26: until the updated value neural network and strategy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and strategy neural network that meet the preset convergence requirements are used as the network parameters of the intelligent agent pair to be trained to obtain a trained intelligent agent pair.

[0140] In some embodiments, when a preset convergence requirement is met or the training exceeds a preset fixed number of rounds, the training ends and a trained agent pair is obtained.

[0141] Step S30: Control the tracker and the evader to play a pursuit-escape game according to the current movement amount of the tracker and the current movement amount of the evader.

[0142] In some embodiments, when the tracker and the evader are controlled to play the pursuit-escape game according to the current action amount of the tracker and the current action amount of the evader, the optimal game effect can be achieved.

[0143] Please refer to Figure 7a and Figure 7b In the first embodiment, the six classical orbital numbers of the virtual spacecraft are , track the orbital position and velocity of the spacecraft relative to the virtual spacecraft in the LVLH coordinate system , the orbital position and velocity of the escaper relative to the virtual spacecraft During the game, both parties can apply a single pulse at the initial moment, the pulse amplitude is 10 m / s, the direction angle is freely selected, and the game time is 1000 s. According to the above agent training method, the following game results can be obtained. When the initial state of the spacecraft is fixed, Figure 7a and Figure 7b The D4P2G algorithm, D4SPG algorithm and DDPG algorithm are respectively given to calculate the pulse control direction angle of the spacecraft. and The training results are shown in Figure 2. It can be seen that both the D4P2G algorithm and the D4SPG algorithm can converge to the same stable solution, while the DDPG algorithm converges to another solution in terms of the in-plane direction angle, with a more obvious deviation. At the same time, by comparing the convergence speed of the algorithms, it is found that the convergence efficiency of the D4SPG algorithm is the highest, while the oscillation periods of the D4P2G algorithm and the DDPG algorithm are relatively long. In order to verify whether the stable solution of the training is a Nash equilibrium, the traditional method is used to solve the simultaneous decision-making pulse pursuit game problem. It can be calculated that there is a unique Nash equilibrium solution in this problem, and the pulse direction angle corresponding to this solution is:

[0144] ,

[0145] And reference Figure 8 The two algorithms in this embodiment finally converge to the following solution:

[0146] ,

[0147] Therefore, it can be known that this embodiment has found the Nash equilibrium of the game, and this Nash equilibrium is consistent with the theoretical Nash equilibrium result.

[0148] Please refer to Figure 9a and Figure 9b In the second embodiment, the basic steps are basically the same as those in the first embodiment, and the main difference is that the input initial scenario parameters are different: the six classical orbital elements of the virtual spacecraft are ; Track the orbit of the spacecraft relative to the virtual spacecraft in the LVLH coordinate system , the orbital position and velocity of the escaper relative to the virtual spacecraft The initial states of the two spacecraft are taken as the central reference, and the random deviations of the state quantities are set so that they are evenly distributed in the random space. Therefore, the initial states of the two spacecraft are .in, is a 6-dimensional random vector, and each element is evenly distributed between [-0.5, 0.5]. During the game, both parties can apply a single pulse at the initial moment, the pulse amplitude is 10 m / s, the direction angle is freely selected, and the game time is 1000 s. According to the above-mentioned agent training method, the following game results can be obtained. For a given initial state of the spacecraft, the method of this embodiment can converge to the Nash equilibrium with high accuracy. For the case where the initial state is randomly given in space, this embodiment is also trained and tested, and the results are compared with the theoretical Nash equilibrium solution solved by the traditional algorithm. Here, the trained neural network policy function is called a Nash equilibrium controller, because it can give the Nash equilibrium strategy in the current state in real time, and has the characteristics of robust control or feedback control. Figure 9a and Figure 9b The training effect of the Nash equilibrium controller is tested under different initial example configurations. The true value of the Nash equilibrium of each example is also solved using the traditional method as a reference for accuracy. Figure 9a and Figure 9b The results show that the Nash equilibrium controller trained by the D4PSG algorithm has a higher degree of overlap with the true value, and its accuracy and robustness are higher than the training results of the D4P2G algorithm. The D4P2G algorithm will have some individual cases with poor accuracy.

[0149] Please refer to Fig.10 The embodiment of the present application proposes a game optimization device based on a spacecraft, including a data preparation module 10, an action amount determination module 20, a pursuit and escape game module 30 and an intelligent agent training module 40, which are described in detail below.

[0150] The data preparation module 10 is used to obtain the current state quantity of the tracker and the current state quantity of the escaper in a preset pursuit-escape game scenario, wherein the state quantity includes position and speed.

[0151] The motion quantity determination module 20 is used to input the current state quantity of the tracker into the tracker agent trained by the agent training method to obtain the current motion quantity of the tracker, and input the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current motion quantity of the escaper; wherein the motion quantity includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle.

[0152] The pursuit-escape game module 30 is used to control the tracker and the escaper to play the pursuit-escape game according to the current action amount of the tracker and the current action amount of the escaper.

[0153] The agent training module 40 is used to construct a Markov game model and an experience pool; wherein the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent in the game process, and the experience pool includes experience data corresponding to the state quantity, action quantity, reference function and reference factor; construct a learner, and construct a main network and a target network based on the learner; wherein the learner includes a value neural network and a strategy neural network corresponding to each agent in the game process, the main network is constructed according to the parameters of the value neural network and the strategy neural network, the main network includes a main value network and a main strategy network, the target network is determined according to the network structure and network parameters of the main network, and the target network includes a target value network and a target strategy network; construct a pair of agents to be trained, use the Markov game model as the training environment, and obtain the target of the pair of agents to be trained after interacting with the environment in the training environment Experience data; wherein the agent pair to be trained includes a tracker agent and an escaper agent that play a pursuit-escape game; the target experience data is stored in the experience pool, and the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function, the main value network and the target value network corresponding to the value distribution; the deterministic symplectic policy gradient corresponding to the multiple experience data is calculated, and the parameters of the policy neural network in the learner are updated according to the deterministic symplectic policy gradient, the main policy network and the target policy network; until the updated value neural network and policy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and policy neural network that meet the preset convergence requirements are used as the network parameters of the agent pair to be trained, and a trained agent pair is obtained.

[0154] In some embodiments, the embodiments of the present application propose a spacecraft-based game optimization device, including: a memory for storing programs; and a processor for implementing a spacecraft-based game optimization method by executing the programs stored in the memory.

[0155] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0156] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A game optimization method based on spacecraft, characterized in that: include: Obtain the current state of the tracker and the current state of the escaper in the preset pursuit-escape game scenario, where the state includes position and speed; Inputting the current state quantity of the tracker into the tracker agent trained by the agent training method to obtain the current action quantity of the tracker, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle; Controlling the tracker and the escaper to play a chase-and-escape game according to the current movement amount of the tracker and the current movement amount of the escaper; Wherein, the agent training method comprises: Constructing a Markov game model and an experience pool; wherein the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent during the game, and the experience pool includes experience data corresponding to the state quantity, action quantity, reference function and reference factor; Construct a learner, and construct a main network and a target network based on the learner; wherein the learner includes a value neural network and a strategy neural network corresponding to each of the intelligent agents in the game process, the main network is constructed according to the parameters of the value neural network and the strategy neural network, the main network includes a main value network and a main strategy network, the target network is determined according to the network structure and network parameters of the main network, and the target network includes a target value network and a target strategy network; Constructing a pair of intelligent agents to be trained, using the Markov game model as a training environment, and obtaining target experience data of the pair of intelligent agents to be trained after interacting with the environment in the training environment; wherein the pair of intelligent agents to be trained includes a tracker intelligent agent and an escaper intelligent agent that play a pursuit-escape game; The target experience data is stored in the experience pool, the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network; Calculate and obtain the deterministic symplectic policy gradients corresponding to the plurality of pieces of experience data, and update the parameters of the policy neural network in the learner according to the deterministic symplectic policy gradients, the main policy network, and the target policy network; Until the updated value neural network and the strategy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and the strategy neural network that meet the preset convergence requirements are used as the network parameters of the to-be-trained intelligent agent pair to obtain a trained intelligent agent pair.

2. The game optimization method according to claim 1, characterized in that: The updating of parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network includes: Calculating the gradient of the value loss function with respect to the network parameters in the value neural network; Determining the update amplitude of the network parameters according to the gradient, the learning rate of the value neural network and the gradient descent method; The parameter direction of the main value network is used as the update direction, and the parameters of the target value network are updated in combination with the update amplitude. The parameters of the value neural network in the learner are updated according to the parameter update process of the target value network.

3. The game optimization method according to claim 1, characterized in that: The calculating and obtaining the deterministic symplectic policy gradient corresponding to the plurality of empirical data includes: Constructing a Hessian matrix corresponding to the policy network parameters of the value neural network and the policy neural network; A deterministic symplectic policy gradient is generated according to the expansion corresponding to the multiple second-order derivatives of the value neural network and the Hessian matrix.

4. The game optimization method according to claim 1, characterized in that: The step of obtaining target experience data of the intelligent agent to be trained after interacting with the environment in the training environment includes: Obtaining the state quantity, reward value and discount factor of the to-be-trained agent in the training environment; Calculate the corresponding action amount according to the strategy neural network in the intelligent agent to be trained and the state amount in the training environment; Determining a state quantity at a next moment corresponding to the state quantity in the training environment according to a state transfer function in the training environment; The state quantity in the training environment, the reward value, the discount factor, the action quantity, and the state quantity at the next moment corresponding to the state quantity in the training environment are summarized as target experience data.

5. The game optimization method according to claim 4, characterized in that: The step of calculating the corresponding action amount according to the policy neural network in the to-be-trained agent and the state amount in the training environment includes: ， in, Indicates the amount of action, represents the policy neural network, represents the state quantity in the training environment, represents the standard deviation, represents the standard normal distribution.

6. The game optimization method according to claim 1, characterized in that: The step of constructing a main network and a target network according to the learner comprises: Initializing the parameters of the value neural network in the learner and the parameters of the strategy neural network corresponding to each of the intelligent agents respectively, and constructing the main network according to the initialized parameters; A target network is constructed based on the network structure and parameters of the main network and a preset update rate.

7. The game optimization method according to claim 1, characterized in that: The reference function includes a state transfer function and a reward function, and the reference factor includes a discount factor.

8. A game optimization device based on a spacecraft, characterized in that: include: A data preparation module is used to obtain the current state quantity of the tracker and the current state quantity of the escaper in a preset pursuit-escape game scenario, where the state quantity includes position and speed; An action amount determination module is used to input the current state amount of the tracker into the tracker agent trained by the agent training method to obtain the current action amount of the tracker, and input the current state amount of the escaper into the escaper agent trained by the agent training method to obtain the current action amount of the escaper; wherein the action amount includes acceleration, pulse amplitude, in-plane direction angle and out-of-plane direction angle; A pursuit-escape game module, used to control the tracker and the escaper to play a pursuit-escape game according to the current action amount of the tracker and the current action amount of the escaper; An agent training module is used to construct a Markov game model and an experience pool; wherein the Markov game model includes the state quantity, action quantity, reference function and reference factor of the agent during the game, and the experience pool includes experience data corresponding to the state quantity, the action quantity, the reference function and the reference factor; construct a learner, and construct a main network and a target network based on the learner; wherein the learner includes a value neural network and a strategy neural network corresponding to each of the agents during the game, the main network is constructed according to the parameters of the value neural network and the strategy neural network, the main network includes a main value network and a main strategy network, the target network is determined according to the network structure and network parameters of the main network, and the target network includes a target value network and a target strategy network; construct a pair of agents to be trained, use the Markov game model as a training environment, and obtain the target experience of the pair of agents to be trained after interacting with the environment in the training environment data; wherein the pair of agents to be trained includes a tracker agent and an escaper agent that play a pursuit-escape game; the target experience data is stored in the experience pool, the learner samples multiple experience data from the experience pool, calculates the value distribution corresponding to the multiple experience data, and updates the parameters of the value neural network in the learner according to the value loss function corresponding to the value distribution, the main value network and the target value network; the deterministic symplectic policy gradient corresponding to the multiple experience data is calculated, and the parameters of the policy neural network in the learner are updated according to the deterministic symplectic policy gradient, the main policy network and the target policy network; until the updated value neural network and the policy neural network in the learner meet the preset convergence requirements, the network parameters of the updated value neural network and the policy neural network that meet the preset convergence requirements are used as the network parameters of the pair of agents to be trained to obtain a trained pair of agents.

9. A game optimization device based on a spacecraft, characterized in that: include: Memory, used to store programs; A processor, configured to implement the game optimization method according to any one of claims 1 to 7 by executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the game optimization method as described in any one of claims 1-7.

Images