Control method, device and equipment for cooperative game of multiple spacecrafts and medium

Abstract

The invention relates to a multi-spacecraft cooperative game control method, device, equipment and medium, and the method comprises the steps: inputting the current state quantities of a plurality of trackers having a cooperative relationship in a multi-spacecraft cooperative pursuit scene into a plurality of tracker agents trained by employing an agent training method, so as to obtain the current motion quantities of the plurality of trackers, inputting the current state quantity of the escaper into an escaper intelligent agent trained by using an intelligent agent training method to obtain the current action quantity of the escaper; and controlling the plurality of trackers to perform a cooperative game according to the current action quantities of the plurality of trackers and the current action quantity of the escaper, so that the plurality of trackers in the cooperative game perform pursuit capture on the escaper, and the cooperative pursuit game is ended until any one of the plurality of trackers completes capture of the escaper within the game duration. The technical problem that the cooperative game effect among multiple spacecrafts is not good when the multi-spacecraft orbit game problem is solved.

Description

Technical Field

[0001] The present invention relates to the field of spacecraft control technology, and in particular to a control method, device, equipment and medium for multi-spacecraft cooperative game. Background Art

[0002] With the recent development of satellite formation and cluster technology, multi-spacecraft cooperative control technology has received increasing attention. Multi-satellite cluster control and use can significantly enhance space surveillance, approach and manipulation capabilities. In the context of orbital games, the use of multiple spacecraft to participate in the game can also improve overall capabilities through numerical advantages, such as cooperatively capturing a target with stronger maneuverability when the maneuverability is weak, or cooperatively capturing a target in a shorter time or with a higher success rate when the maneuverability is strong. However, the cooperative control of multi-spacecraft games involves both cooperative and confrontational relationships. It is difficult to use traditional differential game methods to complete the modeling and design of the correlation of objective functions, information interoperability, and cooperative constraints among spacecraft, and the solution efficiency is also low. Artificial intelligence methods do not need to solve complex mathematical equations. Only through data collection and training, the intelligent agent can have the ability to learn and improve, and has strong flexibility. However, when existing artificial intelligence methods are applied to multi-spacecraft cooperative orbital games, there are still problems such as poor adaptability to the orbital dynamics environment, low efficiency in strategy application, and poor training effect. Summary of the invention

[0003] The main technical problem that the present invention solves is that when facing the problem of multi-spacecraft orbital game, when artificial intelligence methods are applied to the cooperative orbital game of multiple spacecraft, there are problems such as poor adaptability to the orbital dynamics environment, low efficiency in strategy application, and poor training effect, which leads to poor effect of cooperative game between multiple spacecraft.

[0004] According to the first aspect, an embodiment provides a control method for a multi-spacecraft cooperative game, comprising:

[0005] Acquire the state quantities of multiple trackers and escapers in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities include position and speed; the multi-spacecraft cooperative pursuit scenario includes tracking the escaper by using the multiple trackers in a cooperative relationship within a preset game duration, and the escaper will escape when being tracked;

[0006] Inputting the current state quantities of the multiple trackers into the multiple tracker agents trained by the agent training method to obtain the current motion quantities of the multiple trackers, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current motion quantity of the escaper; wherein the motion quantity includes control acceleration;

[0007] Controlling the multiple trackers to perform cooperative game according to the current action amounts of the multiple trackers and the current action amounts of the escaper, so that the multiple trackers in the cooperative game pursue and capture the escaper, until any tracker among the multiple trackers completes the capture of the escaper within the game duration, and then the cooperative pursuit game ends;

[0008] Wherein, the agent training method comprises:

[0009] Constructing a Markov game model, a tracker experience pool, and an escaper experience pool; wherein the Markov game model includes a state quantity, an action quantity, a state transfer function, and a reward function in a cooperative pursuit game scenario; the state quantity is constructed by local observation quantities between multiple tracker agents to be trained and escaper agents to be trained;

[0010] Constructing a group of tracker agents to be trained and a group of escaper agents to be trained, and constructing a tracker learner and an escaper learner; wherein the group of tracker agents to be trained includes a plurality of tracker agents to be trained; the tracker learner includes a plurality of tracker value neural networks and strategy neural networks; the escaper learner includes a value neural network and a strategy neural network of an escaper;

[0011] Using the Markov game model as a training environment, obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, updating the tracker experience pool using the tracking experience data after interacting with the environment, and obtaining escape experience data of the escaper agents to be trained after interacting with the environment in the training environment, updating the escaper experience pool using the escape experience data after interacting with the environment;

[0012] Calculate the value distribution and gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data collected from the updated tracker experience pool and the local observations, and calculate the value distribution and gradient corresponding to the escaper agent to be trained based on the multiple escape experience data collected from the updated escaper experience pool;

[0013] Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements, and the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

[0014] In some embodiments, calculating the gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data collected from the updated tracker experience pool and the local observations includes:

[0015]

[0016] in, represents the gradient corresponding to the group of tracker agents to be trained, represents the network parameters of the strategy neural network corresponding to the jth tracker agent to be trained in the group of tracker agents to be trained, J P represents the payment function of the group of tracker agents to be trained, P j represents the jth tracker agent to be trained in the group of tracker agents to be trained, represents the expectation when the state takes a fixed strategy distribution, Represents the jth tracker agent strategy neural network to be trained About the network parameters θ Pj The gradient of represents the gradient of the value neural network with respect to the action strategy of the jth tracker agent to be trained, represents the value neural network of the group of tracker agents to be trained, N represents the number of the tracker agents to be trained, represents the local observation of the jth tracker agent to be trained in the group of tracker agents to be trained, s represents the state quantity, represents the action amount of the Nth tracker agent to be trained in the group of tracker agents to be trained, represents the action amount of the jth tracker agent to be trained in the group of tracker agents to be trained, Represents the policy neural network of the jth tracker agent to be trained in the group of tracker agents to be trained.

[0017] In some embodiments, the reward function of the Markov game model includes a process reward function and a terminal reward function;

[0018] The process reward function is:

[0019]

[0020] in, represents the process reward of the tracker agent, represents the process reward of the escaper agent, d k Indicates the minimum distance of the current node, d k-1 represents the minimum distance to the previous node, K represents the terminal node, and d0 represents the distance at the initial moment;

[0021] The terminal reward function is:

[0022]

[0023] in, represents the terminal reward of the tracker agent, represents the terminal reward of the escaper agent, and C represents the reward value that meets the preset threshold.

[0024] In some embodiments, a main network is constructed according to the parameters of the value neural networks and the policy neural networks of the multiple tracking parties in the tracker learner, and the parameters of the value neural network and the policy neural network of the escaper in the escaper learner; wherein the main network includes the main value network and the main policy network of the tracking party, and the main value network and the main policy network of the escaper;

[0025] A target network is constructed according to the network structure and network parameters of the main network; wherein the target network includes the target value network and target strategy network of the tracking party, and the target value network and target strategy network of the escaping party.

[0026] In some embodiments, updating the parameters of the tracker learner based on the value distribution and gradient corresponding to the group of tracker agents to be trained includes:

[0027] Calculate a corresponding value loss function according to the value distribution corresponding to the group of tracker agents to be trained, and update parameters of the value neural networks of multiple trackers in the tracker learner according to the value loss function, the main value network of the tracker, and the target value network of the tracker;

[0028] The policy neural networks of multiple trackers in the tracker learner are parameter updated according to the gradients corresponding to the group of tracker agents to be trained, the main policy network of the tracker, and the target policy network of the tracker.

[0029] In some embodiments, the local observations between the plurality of tracker agents to be trained and the escaper agents to be trained include:

[0030]

[0031] in, represents the local observation between the jth tracker agent to be trained and the escaper agent to be trained at time node k, represents the state of the jth tracker agent to be trained at time node k, represents the state quantity of the escaper agent to be trained at time node k, and LVLH represents the relative orbital coordinate system of the virtual spacecraft.

[0032] In some embodiments, obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment includes:

[0033] Obtain the state quantity, reward value and discount factor of the tracking agent group to be trained in the training environment;

[0034] Calculate the corresponding action amount according to the strategy neural network in the group of tracking agents to be trained and the state amount in the training environment;

[0035] Determining a state quantity at a next moment corresponding to the state quantity in the training environment according to a state transfer function in the training environment;

[0036] The state quantity in the training environment, the reward value, the discount factor, the action quantity, and the state quantity at the next moment corresponding to the state quantity in the training environment are summarized as tracking experience data after interaction with the environment.

[0037] According to the second aspect, an embodiment provides a control device for a multi-spacecraft cooperative game, comprising:

[0038] A state quantity acquisition module is used to acquire the state quantities of multiple trackers with a cooperative relationship and the state quantity of the escaper in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities include position and speed; the multi-spacecraft cooperative pursuit scenario includes tracking the escaper by using the multiple trackers with a cooperative relationship within a preset game time, and the escaper will escape when being tracked;

[0039] An action quantity generating module, used for inputting the current state quantities of the multiple trackers into the multiple tracker agents trained by the agent training method to obtain the current action quantities of the multiple trackers, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes control acceleration;

[0040] A cooperative pursuit game module, used for controlling the multiple trackers to play a cooperative game according to the current action amounts of the multiple trackers and the current action amounts of the escaper, so that the multiple trackers in the cooperative game pursue and capture the escaper, until any tracker among the multiple trackers completes the capture of the escaper within the game duration, and then the cooperative pursuit game ends;

[0041] The agent training module is used to build an agent training method, including:

[0042] Constructing a Markov game model, a tracker experience pool, and an escaper experience pool; wherein the Markov game model includes a state quantity, an action quantity, a state transfer function, and a reward function in a cooperative pursuit game scenario; the state quantity is constructed by local observation quantities between multiple tracker agents to be trained and escaper agents to be trained;

[0043] Constructing a group of tracker agents to be trained and a group of escaper agents to be trained, and constructing a tracker learner and an escaper learner; wherein the group of tracker agents to be trained includes a plurality of tracker agents to be trained; the tracker learner includes a plurality of tracker value neural networks and strategy neural networks; the escaper learner includes a value neural network and a strategy neural network of an escaper;

[0044] Using the Markov game model as a training environment, obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, updating the tracker experience pool using the tracking experience data after interacting with the environment, and obtaining escape experience data of the escaper agents to be trained after interacting with the environment in the training environment, updating the escaper experience pool using the escape experience data after interacting with the environment;

[0045] Calculate the value distribution and gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data collected from the updated tracker experience pool and the local observations, and calculate the value distribution and gradient corresponding to the escaper agent to be trained based on the multiple escape experience data collected from the updated escaper experience pool;

[0046] Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements, and the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

[0047] According to the third aspect, an embodiment provides a control device for a multi-spacecraft cooperative game, including:

[0048] Memory, used to store programs;

[0049] The processor is used to implement the control method by executing the program stored in the memory.

[0050] According to a fourth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement a control method.

[0051] According to the control method, device, equipment and computer-readable storage medium of the multi-spacecraft cooperative game of the above-mentioned embodiment, a group of tracker agents to be trained and a group of escaper agents to be trained are constructed during the training process, and multiple trained tracker agents and escaper agents are obtained by using the agent training method, wherein each spacecraft can complete cooperative control only by relying on local observations, thereby improving the control efficiency of multi-spacecraft decision-making. A Markov game model constructed according to the state quantity, action quantity, state transfer function and reward function in the cooperative pursuit and escape game scenario is used as the training environment, thus adapting to the orbital dynamics environment. The trained multiple tracker agents with cooperative game capabilities and escaper agents with multi-target escape capabilities can better control multiple trackers to perform cooperative games, so that the multiple trackers in the cooperative game can pursue and capture the escaper. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a control flow chart of the multi-spacecraft cooperative game according to an embodiment of the present application;

[0053] Figure 2 A flowchart of an agent training method according to an embodiment;

[0054] Figure 3A flowchart of obtaining tracking experience data of a group of tracker agents to be trained after interacting with the environment in a training environment in one embodiment;

[0055] Figure 4 A flowchart of updating parameters of a tracker learner based on the value distribution and gradient corresponding to the group of tracker agents to be trained in one embodiment;

[0056] Figure 5 A comparison chart of the capture success rates of different tracking strategies and escape strategies in a random initial state space in an embodiment;

[0057] Figure 6a and Figure 6b It is a trajectory diagram of an embodiment in which both trackers use the LQ-P strategy and both trackers use the DRL-P strategy;

[0058] Figure 7a and Figure 7b In one embodiment, in a random space, the miss amount results of the trackers using the LQ-P strategy and the miss amount results of the trackers using the DRL-P strategy;

[0059] Figure 8 A control structure diagram of a multi-spacecraft cooperative game according to an embodiment;

[0060] Fig. 9 The present invention is a schematic diagram of the structure of a control device for a multi-spacecraft cooperative game according to an embodiment. DETAILED DESCRIPTION

[0061] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0062] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0063] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0064] Please refer to Figure 1 In an embodiment of the present invention, a control method for a multi-spacecraft cooperative game is proposed, including steps S1 to S3, which are described in detail below.

[0065] Step S1: Obtain the state quantities of multiple trackers and escapers that have a cooperative relationship in a multi-spacecraft cooperative pursuit scenario.

[0066] In some embodiments, a multi-spacecraft cooperative pursuit scenario includes tracking a fugitive using multiple trackers in a cooperative relationship within a preset game duration, and at the same time, a scenario in which the fugitive attempts to escape when being tracked by multiple trackers.

[0067] In some embodiments, four trackers in a cooperative relationship are used to cooperate to capture an escaping escapee, the state quantities of the multiple trackers include the positions and speeds of the multiple trackers, and the state quantity of the escapee includes the position and speed of the escapee.

[0068] For example, the escaper initially runs in an orbit with an orbital element number of C r =[6778.137km,0,0°,0°,0°,0°,0°, with the initial point as the reference point, the relative orbital position and velocity of the escaper are [0,0,0,0,0,0] T The four trackers initially move in a coplanar natural fly-by configuration relative to the escaper. The radius of the fly-by and the phase angle of the tracker in the fly-by ellipse are randomly given in the space of a∈[2km,40km] and f0∈[0,2π] respectively.

[0069] In some embodiments, the coplanar natural fly-around configuration motion can be expressed as:

[0070]

[0071] in, and Indicates tracker P j location, and Indicates tracker P j The speed of the reference orbit is represented by a, a represents the length of the major semi-axis of the ellipse formed by the track of the pursuer relative to the escaper, b represents the length of the minor semi-axis of the ellipse formed by the track of the pursuer relative to the escaper, ω represents the average angular velocity of the reference orbit, Indicates tracker P j The initial phase of the pursuer is t, and t represents the time value when the pursuer flies around the escaper naturally.

[0072] Step S2: respectively input the current state quantities of the multiple trackers into the multiple tracker agents trained by the agent training method to obtain the current action quantities of the multiple trackers, and input the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper.

[0073] In some embodiments, the current motion amounts of the plurality of trackers and the current motion amounts of the escaper include controlling acceleration.

[0074] Step S3: Control multiple trackers to play a cooperative game according to their current action amounts and the escapee's current action amounts, so that the multiple trackers in the cooperative game pursue and capture the escapee, until any tracker among the multiple trackers completes the capture of the escapee within the game duration, and the cooperative pursuit game ends.

[0075] In some embodiments, the current motion amounts of the multiple trackers and the current motion amounts of the escaper include control acceleration. For example, the control acceleration magnitudes of the tracker and the escaper are both 0.04 m / s. 2 , the direction is freely chosen, the game duration is 1000s, and the capture radius is 1km. Within the given game duration, once any tracker among multiple trackers completes the capture of the escapee, and the tracker is considered to have tracked the escapee successfully, otherwise, the escapee is considered to have escaped successfully, and the cooperative pursuit game ends.

[0076] In some embodiments, multiple trackers and escapers respectively minimize or maximize the minimum distance between the two trajectories. Within a given game duration, when the distance between any tracker and the escaper is less than the capture radius, the capture is considered complete and the tracking party composed of multiple trackers wins. Otherwise, the escaper corresponding to the escaper wins. Therefore, the capture condition can be expressed as:

[0077]

[0078] Among them, t k represents the current time, t frepresents the end time of the game, N represents the number of tracker agent groups to be trained, k represents the time node, d h represents the minimum distance between all trackers and escapers from the initial moment to the current moment, P j represents the jth tracker among multiple trackers, E represents the escaper, Indicates tracker P j The coordinate value on the x-axis, Indicates the coordinate value of the escaper E on the x-axis, Indicates tracker P j The coordinate value on the y-axis, Indicates the coordinate value of the escaper E on the y-axis, Indicates tracker P j The coordinate value on the z-axis, represents the coordinate value of the escaper E on the z-axis, R c Indicates the capture radius.

[0079] Please refer to Figure 2 In some embodiments, the agent training method includes steps S21 to S25, which are described in detail below.

[0080] Step S21: Construct a Markov game model, a tracker experience pool, and a evader experience pool.

[0081] In some embodiments, the Markov game model includes state quantities, action quantities, state transfer functions, and reward functions in a cooperative pursuit game scenario.

[0082] In some embodiments, a virtual spacecraft relative orbit LVLH coordinate system is established, a virtual spacecraft near multiple spacecraft is used as a reference spacecraft, a local orbit coordinate system is established, the origin of the local orbit coordinate system is used as the center of mass of the reference spacecraft, and a right-handed coordinate system is formed with the ox axis and the oz axis. The ox axis is along the radial direction of the reference spacecraft, the oy axis is along the tangent direction of the reference spacecraft's motion trajectory, and the oz axis is along the normal direction of the reference spacecraft's orbital plane. The state of multiple spacecraft at each time node is described by the difference in relative state of the virtual spacecraft relative orbit LVLH coordinate system, which is expressed as:

[0083]

[0084] Among them, s k represents the state quantity at time node k, represents the state of tracker P1 at time node k, P1 represents the first tracker agent, x k E represents the state quantity of the escaper E at time node k, E represents the escaper agent, and N represents the number of tracker agents.

[0085] In some embodiments, represents the state of spacecraft i in the virtual system. Indicates relative position. represents the speed, and T represents the transposed sign. What we know about spacecraft i is that it includes multiple tracker agents P1, P2, ..., P N Any tracker agent in can also be an escaper agent E.

[0086] In some embodiments, the state quantity can be constructed by local observation quantities between multiple tracker agents to be trained and escaper agents to be trained, expressed as:

[0087]

[0088] Among them, s k represents the state quantity at time node k, represents the jth tracker agent P to be trained at time node k j The local observations between the escaper agent E to be trained, represents the jth tracker agent P to be trained at time node k j The state quantity, It represents the state of the escaper agent E to be trained at time node k, LVLH represents the virtual spacecraft relative orbital coordinate system, T represents the transposition sign, and N represents the number of tracker agents to be trained.

[0089] In some embodiments, the action amount in the Markov game model in the cooperative pursuit game scenario includes the control acceleration of each spacecraft, wherein the control acceleration of each spacecraft is a vector with a fixed magnitude but a free direction. In order to simplify the problem, in this embodiment, the thrust direction angles α∈[-π,π], β∈[-π / 2,π / 2] of the spacecraft can be used as the action amount output by the policy network:

[0090]

[0091] in, represents the action amount of spacecraft i at time node k, and They represent two different thrust direction angles of spacecraft i at time node k respectively.

[0092] In some embodiments, the amplitude of the action amount is u i =u i [cosβ i cosα i ,cosβ i sinα i ,sinβi ] T , where α i and β i represents two different thrust direction angles of spacecraft i, and T represents the transposed sign.

[0093] In some embodiments, the strategy function α~π(·|s) from state to action in the Markov game model replaces the guidance law of the closed-loop control of the spacecraft.

[0094] In some embodiments, regarding the state transfer function in the Markov game model, under continuous thrust control, the state transfer of each spacecraft needs to be obtained by numerically integrating the ordinary differential equation, and the time interval of each integration is the time between two maneuver decisions. Under the condition of the linear dynamic equation, it can be indirectly expressed as an integral function form of the state transfer matrix:

[0095]

[0096] in, represents the state of spacecraft i at the next time node k+1, represents the state of spacecraft i at time node k, represents the action amount of the escaper agent E at time node k, represents the action amount of the tracker agent P1 at time node k, N represents the number of tracker agents, Φ(t k+1 ,t k ) indicates that from t k to k+1 The state transfer matrix, t k Indicates the corresponding moment at time node k, t k+1 Indicates that at the corresponding moment of the next time node k+1, Φ v (t k+1 ,τ) is the state transfer matrix about speed, τ represents the preset parameter, and u(τ) represents the control acceleration.

[0097] In some embodiments, the state transition matrix from t0 to t can be expressed as:

[0098]

[0099] Among them, Φ(t,t0) represents the state transition matrix from t0 to t, Φ r (t,t0) represents the state transfer matrix about the position in Φ(t,t0), Φ v (t, t0) represents the state transfer matrix about speed in Φ(t, t0), τ represents the preset parameters, s=sinτ, c=cosτ, τ=ω(t-t0).

[0100] In some embodiments, the reward function of the Markov game model includes a process reward function and a terminal reward function, and the process reward function is:

[0101]

[0102] in, represents the process reward of the tracker agent, represents the process reward of the escaper agent, d k Indicates the minimum distance of the current node k, d k-1 represents the minimum distance to the previous node k-1, K represents the terminal node, and d0 represents the distance at the initial moment.

[0103] In some embodiments, the terminal reward function is:

[0104]

[0105] in, represents the terminal reward of the tracker agent, represents the terminal reward of the escaper agent, and C represents the reward value that meets the preset threshold.

[0106] In some embodiments, C represents a larger reward value given to the tracker agent when the terminal capture is completed, wherein the size of the reward value can be defined by oneself, but it must be much larger than the process reward. In the embodiment of the present application, C=10 5 .

[0107] In some embodiments, for the process reward function, if the current node is closer to the previous node, the reward of the tracker is positive. For the terminal reward function, the terminal reward of the spacecraft is determined based on whether the distance within the game time is within the capture radius. At the same time, assuming that the discount factor is 1, the reward functions of the tracker and the escaper are opposite, forming a zero-sum game.

[0108] In some embodiments, the tracker experience pool and the escaper experience pool contain experience data related to the tracker and experience data related to the escaper, respectively.

[0109] Step S22: Construct a group of tracker agents to be trained and a group of escaper agents to be trained, and construct a tracker learner and an escaper learner.

[0110] In some embodiments, a group of tracker agents to be trained is constructed, wherein the group of tracker agents to be trained includes multiple tracker agents to be trained, and each tracker agent to be trained is composed of multiple tracker agent strategy neural networks. The tracker agent strategy neural network can be expressed as N represents the number of tracker agents to be trained, P j represents the jth tracker agent to be trained, Represents the tracker agent P to be trained j The network parameters of the policy neural network, represents the local observation of the jth tracker agent to be trained at time node k. The network parameters of the policy neural network of the tracker agent to be trained are copied from the network parameters of the policy neural network of the tracker learner.

[0111] In some embodiments, a to-be-trained escaper agent is constructed, and each to-be-trained escaper agent is composed of a single escaper agent strategy neural network, wherein the escaper agent strategy neural network can be expressed as θ E Represents the network parameters of the policy neural network of the escaper agent E to be trained. The network parameters of the escaper agent’s policy neural network are copied from the network parameters of the escaper learner’s policy neural network.

[0112] In some embodiments, a tracker learner and an escaper learner are constructed, wherein the tracker learner includes a value neural network and a strategy neural network of multiple trackers, and the strategy neural networks of the multiple trackers are The value neural network is The corresponding network parameters in the tracker learner are initialized as

[0113] In some embodiments, the escaper learner includes a value neural network of the escapee and policy neural network The corresponding network parameters in the escaper learner are initialized as θ E ,φ E .

[0114] In some embodiments, the main network is constructed based on the network parameters of the value neural networks and the network parameters of the policy neural networks of the multiple tracking parties in the tracker learner, and the network parameters of the value neural network and the network parameters of the policy neural network of the escaper in the escaper learner. φ P and θ E ,φ E Build the main network.

[0115] In some embodiments, the target network is constructed based on the network structure and network parameters of the main network, and the network parameters of the target network are copied from the main network. θ E' =θ E ,φ P' =φP ,φ E' =φ E .

[0116] In some embodiments, the main network includes the main value network and the main policy network of the tracking party, and the main value network and the main policy network of the escapee. The target network includes the target value network and the target policy network of the tracking party, and the target value network and the target policy network of the escapee.

[0117] In some embodiments, multiple groups of tracker agents to be trained and multiple escaper agents to be trained can be constructed, wherein one group of tracker agents to be trained corresponds to one escaper agent to be trained, thereby achieving parallel training of the agents.

[0118] Step S23: Using the Markov game model as the training environment, obtain the tracking experience data of the tracker agent group to be trained after interacting with the environment in the training environment, use the tracking experience data after interacting with the environment to update the tracker experience pool, and obtain the escape experience data of the escaper agent to be trained after interacting with the environment in the training environment, and use the escape experience data after interacting with the environment to update the escaper experience pool.

[0119] In some embodiments, training parameters are initialized before training, where the training parameters include the batch size B M , the number of trackers N, the number of groups of tracker agents to be trained A KP , the number of escaper agents to be trained A KE , exploration constant ò, value neural network learning rate l α , the learning rate of the policy neural network l β .

[0120] Please refer to Figure 3 In some embodiments, step S23 obtains tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, including steps S231 to S234, which are described in detail below.

[0121] Step S231: Obtain the state quantity, reward value and discount factor of the tracking agent group to be trained in the training environment.

[0122] Step S232: Calculate the corresponding action amount based on the policy neural network in the group of tracking agents to be trained and the state quantity in the training environment.

[0123] In some embodiments, the action amount is calculated by the following formula:

[0124]

[0125] in, Indicates the amount of action, represents the policy neural network in the group of tracking agents to be trained, represents the tracker agent P in the group of tracking agents to be trained in the training environment j is the local observation of , σ represents the standard deviation, and n(0,1) represents the standard normal distribution.

[0126] Step S233: Determine the state quantity at the next moment corresponding to the state quantity in the training environment according to the state transfer function in the training environment.

[0127] Step S234: The state quantity, reward value, discount factor, action quantity in the training environment and the state quantity at the next moment corresponding to the state quantity in the training environment are summarized as tracking experience data after interaction with the environment.

[0128] In some embodiments, the method for obtaining the escape experience data of the escaper agent to be trained after interacting with the environment in the training environment is consistent with the method for obtaining the tracking experience data of the tracker agent group to be trained after interacting with the environment in the training environment, except that the following formula is used to calculate the action amount:

[0129]

[0130] in, Indicates the amount of action, represents the policy neural network of the escaper agent to be trained, s k represents the state quantity in the training environment, σ represents the standard deviation, and N(0,1) represents the standard normal distribution.

[0131] In some embodiments, the tracker agent and the evader agent interact with the environment separately and periodically update the parameters of their corresponding policy networks.

[0132] Step S24: Calculate the value distribution and gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data and local observations collected from the updated tracker experience pool, and calculate the value distribution and gradient corresponding to the escaper agent to be trained based on the multiple escape experience data collected from the updated escaper experience pool.

[0133] In some embodiments, the value distribution corresponding to the group of tracker agents to be trained is calculated according to the following formula:

[0134]

[0135] Among them, Y P is the value distribution corresponding to the group of tracker agents to be trained, s k Represents the state quantity of node k at the current time, represents the action amount of escaper E at the current time node k, represents the action amount of tracker P1 at the current time node k, N represents the number of tracker agents to be trained in the group of tracker agents to be trained, represents the reward of the current time node k, γ represents the discount factor, represents the value neural network corresponding to the group of tracker agents to be trained, s k+1 represents the state quantity at the next time node k+1, represents the action amount of tracker P1 at the next time node k+1, Represents the value function of the next node.

[0136] In some embodiments, the state quantity and the action quantity in the above formula are calculated based on multiple tracking experience data collected from the updated tracker experience pool.

[0137] In some embodiments, the value distribution corresponding to the escaper agent to be trained is calculated according to the following formula:

[0138]

[0139] Among them, Y E represents the value distribution corresponding to the escaper agent to be trained, s k Represents the state quantity of node k at the current time, represents the action amount of escaper E at the current time node k, represents the action amount of tracker P1 at the current time node k, N represents the number of tracker agents to be trained in the group of tracker agents to be trained, represents the reward of the current time node k, γ represents the discount factor, represents the value distribution corresponding to the escaper agent to be trained, s k+1 represents the state quantity at the next time node k+1, represents the action amount of escaper E at the next time node k+1, Represents the value function of the next node.

[0140] In some embodiments, the gradient corresponding to the group of tracker agents to be trained is calculated based on multiple tracking experience data and local observations collected from the updated tracker experience pool, including:

[0141]

[0142] in, represents the gradient corresponding to the group of tracker agents to be trained, represents the network parameters of the policy neural network corresponding to the jth tracker agent to be trained in the group of tracker agents to be trained, J P represents the payment function of the group of tracker agents to be trained, P j represents the jth tracker agent to be trained in the group of tracker agents to be trained, represents the expectation when the state takes a fixed strategy distribution, Represents the jth tracker agent strategy neural network to be trained About the network parameters θ Pj The gradient of represents the gradient of the value neural network with respect to the action strategy of the jth tracker agent to be trained, represents the value neural network of the group of tracker agents to be trained, N represents the number of tracker agents to be trained, represents the local observation of the jth tracker agent to be trained in the group of tracker agents to be trained, s represents the state quantity, represents the action amount of the Nth tracker agent to be trained in the group of tracker agents to be trained, represents the action amount of the jth tracker agent to be trained in the group of tracker agents to be trained, Represents the policy neural network of the jth tracker agent to be trained in the group of tracker agents to be trained.

[0143] In some embodiments, the gradient corresponding to the escaper agent to be trained is calculated based on multiple escape experience data collected from the updated escaper experience pool, including:

[0144]

[0145] in, represents the gradient corresponding to the escaper agent to be trained, θ E represents the network parameters of the strategy neural network corresponding to the escaper to be trained, s represents the state quantity, a E Represents the action amount of the escaper agent to be trained.

[0146] Step S25: Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements, and the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

[0147] Please refer to Figure 4 In some embodiments, step S25 updates the parameters of the tracker learner based on the value distribution and gradient corresponding to the group of tracker agents to be trained, including steps S251 to S252, which are described in detail below.

[0148] Step S251: Calculate the corresponding value loss function according to the value distribution corresponding to the group of tracker agents to be trained, and update the parameters of the value neural networks of multiple trackers in the tracker learner according to the value loss function, the main value network of the tracker and the target value network of the tracker.

[0149] In some embodiments, a cross entropy loss function is defined to calculate the difference between the current latest value distribution and the old value distribution, that is, to calculate the value loss function of the group of tracker agents to be trained. The calculation formula of the value loss function is as follows:

[0150]

[0151] Among them, Loss(φ P ) represents the value loss function of the group of tracker agents to be trained, Y P is the value distribution corresponding to the group of tracker agents to be trained, represents the old value distribution of the group of tracker agents to be trained, φ P It represents the network parameters of the main value network of the tracking party, and P represents the tracking party, that is, the tracker.

[0152] In some embodiments, for the update of the value neural network of multiple trackers in the tracker learner, the above-mentioned value loss function should be minimized, so the gradient of the value loss function with respect to the network parameters of the value neural network is first solved, and the update direction of the network parameters can be obtained by the gradient descent method, and the target value network of the tracker is slowly updated along the parameter direction of the main value network of the tracker, so as to realize the parameter update of the value neural network of multiple trackers in the tracker learner. The update process is expressed by the formula as follows:

[0153]

[0154] φ P' =(1-∈)φ P' +∈φ P ,for∈<<1

[0155] in, represents the network parameters of the updated value neural network, Indicates the current network parameters, l α represents the learning rate of the tracking value neural network, represents the update amplitude determined by the gradient, φ P represents the network parameters of the main value network of the tracking party, φ P' represents the network parameters of the target value network of the tracking party, and ∈ represents the exploration constant.

[0156] Step S252: Update the parameters of the strategy neural networks of multiple trackers in the tracker learner according to the gradients corresponding to the group of tracker agents to be trained, the main strategy network of the tracker, and the target strategy network of the tracker.

[0157] In some embodiments, the policy neural networks of multiple trackers in the tracker learner are updated according to the gradient corresponding to the group of tracker agents to be trained, the main policy network of the tracker, and the target policy network of the tracker. The target policy neural network of the tracker is slowly updated along the parameter direction of the main policy network of the tracker to achieve parameter update of the policy neural networks of multiple trackers in the tracker learner. The updating process is expressed by the following formula:

[0158]

[0159] θ P' =(1-∈)θ P' +∈θ P

[0160] in, represents the network parameters of the updated policy neural network, Indicates the current network parameters, l βrepresents the learning rate of the tracking party’s policy neural network, represents the gradient of the group of tracker agents to be trained, represents the network parameters of the policy neural network of tracker P1 in the group of tracker agents to be trained, N represents the number of tracker agents to be trained, θ P represents the parameters of the policy neural network, θ P' represents the network parameters of the target policy neural network, and ∈ represents the exploration constant.

[0161] In some embodiments, the parameters of the escaper learner are updated according to the value distribution and gradient corresponding to the escaper agent to be trained, and the parameter updating process is consistent with the above-mentioned parameter updating process for the group of tracker agents to be trained.

[0162] In some embodiments, the value loss function of the escaper agent to be trained is calculated as follows:

[0163]

[0164] Among them, Loss(φ E ) represents the value loss function of the escaper agent group to be trained, Y E is the value distribution corresponding to the escaper agent to be trained, represents the old value distribution of the escaper agent to be trained, φ E It represents the network parameters of the main value network of the escapee, and E represents the tracking party, i.e., the escaper.

[0165] In some embodiments, the update of the value network parameters of the escaper agent to be trained is:

[0166]

[0167] φ E' =(1-∈)φ E' +∈φ E ,for∈<<1

[0168] in, represents the network parameters of the updated value neural network, Indicates the current network parameters, l α represents the learning rate of the tracking value neural network, represents the update amplitude determined by the gradient, represents the network parameters of the main value network of the tracking party, φ E' represents the network parameters of the target value network of the tracking party, and ∈ represents the exploration constant.

[0169] In some embodiments, the parameters of the strategy neural network of the escapee in the escaper learner are updated, and the updating process is expressed by the following formula:

[0170]

[0171] θ E' =(1-∈)θ E' +∈θ E

[0172] in, represents the network parameters of the updated policy neural network, Indicates the current network parameters. represents the learning rate of the escaper’s strategy neural network, represents the gradient of the escaper agent to be trained, θ E represents the parameters of the policy neural network, θ E' represents the network parameters of the target policy neural network, and ∈ represents the exploration constant.

[0173] Please refer to Figure 5 In some embodiments, in the random initial state space, the capture success rates of different tracking strategies and escape strategies are different. Deep reinforcement learning is used to train both the tracker and the escaper at the same time. After multiple trainings, a tracking strategy based on deep reinforcement learning (DRL-P) and an escape strategy based on deep reinforcement learning (DRL-E) are obtained. In order to test the final pursuit and escape capabilities of the intelligent agent, not only game confrontations between intelligent agents are carried out, but also game confrontations between intelligent agents and spacecraft using traditional strategies are carried out. Among them, the traditional tracking strategies in the confrontation include linear quadratic pursuit (LQ-P) strategy and vector guidance (VG) strategy, and the traditional escape strategies in the confrontation include linear quadratic escape strategy (LQ-E) and constant thrust direction (CTD) escape strategy. Figure 5 The results of the confrontation between these strategies are given in , and the capture success rate is statistically analyzed by Monte Carlo shooting. Figure 5 It can be found that no matter which strategy the tracker adopts, the escape ability of the escape agent trained by reinforcement learning exceeds that of the two traditional escape agents. For the tracking agent, when the escape agent adopts deep reinforcement learning or constant thrust strategy, its capture success rate is the highest, but when the escape agent adopts linear quadratic escape strategy, its tracking effect is slightly inferior to the other two tracking strategies.

[0174] Please refer to Figure 6a and Figure 6b ,exist Figure 6a In the tracker, the LQ-P strategy is used. Figure 6b In , the trackers all use the DRL-P strategy. Figure 6a As shown in , under the non-cooperative LQ-P strategy, all trackers independently track the escaper, and their trajectories almost intersect at one point. In the end, all trackers failed to capture the escaper. However, under the DRL-P strategy, not all trackers track the escaper. Figure 6b As shown, P2 tracks the escaper's current position, but P1 tracks the escaper's future trajectory, which limits its current escape direction. Ultimately, the escaper is captured by P2.

[0175] Please refer to Figure 7a and Figure 7b In some embodiments, Figure 7a Indicates the miss-target results of the tracker using the LQ-P strategy in random space. Figure 7b Indicates the miss distance results when the tracker uses the DRL-P strategy in random space. In the miss distance results, the color distribution essentially represents the difficulty of capturing the escaper at different initial positions (horizontal and vertical coordinates) (the color represents the final miss distance, the darker the color, the closer the distance). Figure 7a and 7b From the comparison, we can see that the strategy trained by deep reinforcement learning is significantly better than the traditional LQ strategy. However, no matter which strategy is used, the larger the radius of the initial encirclement ellipse, the greater the miss.

[0176] Please refer to Figure 8 The embodiment of the present application provides a control structure diagram of a multi-spacecraft cooperative game, which applies a decentralized decision-making neural network structure. Each spacecraft can complete cooperative control only by relying on local observations, thereby improving the control efficiency of multi-spacecraft decision-making. The tracking agent with cooperative game capabilities and the escape agent with multi-target escape capabilities are trained, and the effect is better than the traditional tracking and escape strategies.

[0177] Please refer to Fig. 9 The embodiment of the present application provides a control device for a multi-spacecraft cooperative game, including a state quantity acquisition module 10, an action quantity generation module 20, a cooperative pursuit and escape game module 30 and an intelligent agent training module 40, which are described in detail below.

[0178] The state quantity acquisition module 10 is used to obtain the state quantities of multiple trackers with a cooperative relationship and the state quantities of the escaper in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities include position and speed; the multi-spacecraft cooperative pursuit scenario includes tracking the escaper using the multiple trackers with a cooperative relationship within a preset game time, and the escaper will escape when being tracked.

[0179] The action quantity generation module 20 is used to input the current state quantities of multiple trackers into multiple tracker agents trained using the agent training method to obtain the current action quantities of the multiple trackers, and input the current state quantity of the escaper into the escaper agent trained using the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes control acceleration.

[0180] The cooperative pursuit game module 30 is used to control multiple trackers to play a cooperative game according to the current action volume of the multiple trackers and the current action volume of the escapee, so that the multiple trackers in the cooperative game pursue and capture the escapee until any tracker among the multiple trackers completes the capture of the escapee within the game duration, and then the cooperative pursuit game ends.

[0181] The agent training module 40 is used to construct an agent training method, including:

[0182] Construct a Markov game model, a tracker experience pool, and an escaper experience pool; the Markov game model includes state quantities, action quantities, state transfer functions, and reward functions in a cooperative pursuit game scenario; the state quantities are constructed through local observation quantities between multiple tracker agents to be trained and escaper agents to be trained.

[0183] Construct a group of tracker agents to be trained and a group of escaper agents to be trained, and construct a tracker learner and an escaper learner; wherein, the group of tracker agents to be trained includes multiple tracker agents to be trained; the tracker learner includes multiple tracker value neural networks and strategy neural networks; the escaper learner includes the escaper value neural network and strategy neural network.

[0184] Using the Markov game model as the training environment, the tracking experience data of the tracker agent group to be trained after interacting with the environment in the training environment is obtained, and the tracking experience data after interacting with the environment is used to update the tracker experience pool. The escape experience data of the escaper agent to be trained after interacting with the environment in the training environment is obtained, and the escape experience data after interacting with the environment is used to update the escaper experience pool.

[0185] Based on the multiple tracking experience data and local observations collected from the updated tracker experience pool, the value distribution and gradient corresponding to the tracker agent group to be trained are calculated. Based on the multiple escape experience data collected from the updated escaper experience pool, the value distribution and gradient corresponding to the escaper agent to be trained are calculated.

[0186] Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements. Then, the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

[0187] In some embodiments, the embodiments of the present application provide a control device for a multi-spacecraft cooperative game, including: a memory for storing programs; and a processor for implementing a control method by executing the programs stored in the memory.

[0188] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0189] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A control method for multi-spacecraft cooperative game, characterized in that: include: Acquire the state quantities of multiple trackers and escapers in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities include position and speed; the multi-spacecraft cooperative pursuit scenario includes tracking the escaper by using the multiple trackers in a cooperative relationship within a preset game duration, and the escaper will escape when being tracked; Inputting the current state quantities of the multiple trackers into the multiple tracker agents trained by the agent training method to obtain the current motion quantities of the multiple trackers, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current motion quantity of the escaper; wherein the motion quantity includes control acceleration; Controlling the multiple trackers to perform cooperative game according to the current action amounts of the multiple trackers and the current action amounts of the escaper, so that the multiple trackers in the cooperative game pursue and capture the escaper, until any tracker among the multiple trackers completes the capture of the escaper within the game duration, and then the cooperative pursuit game ends; Wherein, the agent training method comprises: Constructing a Markov game model, a tracker experience pool, and an escaper experience pool; wherein the Markov game model includes a state quantity, an action quantity, a state transfer function, and a reward function in a cooperative pursuit game scenario; the state quantity is constructed by local observation quantities between multiple tracker agents to be trained and escaper agents to be trained; Constructing a group of tracker agents to be trained and a group of escaper agents to be trained, and constructing a tracker learner and an escaper learner; wherein the group of tracker agents to be trained includes a plurality of tracker agents to be trained; the tracker learner includes a plurality of tracker value neural networks and strategy neural networks; the escaper learner includes a value neural network and a strategy neural network of an escaper; Using the Markov game model as a training environment, obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, updating the tracker experience pool using the tracking experience data after interacting with the environment, and obtaining escape experience data of the escaper agents to be trained after interacting with the environment in the training environment, updating the escaper experience pool using the escape experience data after interacting with the environment; Calculate the value distribution and gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data collected from the updated tracker experience pool and the local observations, and calculate the value distribution and gradient corresponding to the escaper agent to be trained based on the multiple escape experience data collected from the updated escaper experience pool; Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements, and the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

2. The control method according to claim 1, characterized in that: Calculating the gradient corresponding to the group of tracker agents to be trained according to the multiple tracking experience data collected from the updated tracker experience pool and the local observations, including: in, represents the gradient corresponding to the group of tracker agents to be trained, represents the network parameters of the strategy neural network corresponding to the jth tracker agent to be trained in the group of tracker agents to be trained, J P represents the payment function of the group of tracker agents to be trained, P j represents the jth tracker agent to be trained in the group of tracker agents to be trained, represents the expectation when the state takes a fixed strategy distribution, Represents the jth tracker agent strategy neural network to be trained About the network parameters θ Pj The gradient of represents the gradient of the value neural network with respect to the action strategy of the jth tracker agent to be trained, represents the value neural network of the group of tracker agents to be trained, N represents the number of the tracker agents to be trained, represents the local observation of the jth tracker agent to be trained in the group of tracker agents to be trained, s represents the state quantity, represents the action amount of the Nth tracker agent to be trained in the group of tracker agents to be trained, represents the action amount of the jth tracker agent to be trained in the group of tracker agents to be trained, Represents the policy neural network of the jth tracker agent to be trained in the group of tracker agents to be trained.

3. The control method according to claim 1, characterized in that: The reward function of the Markov game model includes a process reward function and a terminal reward function; The process reward function is: in, represents the process reward of the tracker agent, represents the process reward of the escaper agent, d k Indicates the minimum distance of the current node, d k-1 represents the minimum distance to the previous node, k represents the terminal node, and d0 represents the distance at the initial moment; The terminal reward function is: in, represents the terminal reward of the tracker agent, represents the terminal reward of the escaper agent, and C represents the reward value that meets the preset threshold.

4. The control method according to claim 1, characterized in that: Constructing a main network according to the parameters of the value neural networks and the policy neural networks of the multiple tracking parties in the tracker learner, and the parameters of the value neural network and the policy neural network of the escaper in the escaper learner; wherein the main network includes the main value network and the main policy network of the tracking party, and the main value network and the main policy network of the escaper; A target network is constructed according to the network structure and network parameters of the main network; wherein the target network includes the target value network and target strategy network of the tracking party, and the target value network and target strategy network of the escaping party.

5. The control method according to claim 4, characterized in that: The tracker learner is updated with parameters based on the value distribution and gradient corresponding to the group of tracker agents to be trained, including: Calculate a corresponding value loss function according to the value distribution corresponding to the group of tracker agents to be trained, and update parameters of the value neural networks of multiple trackers in the tracker learner according to the value loss function, the main value network of the tracker, and the target value network of the tracker; The policy neural networks of multiple trackers in the tracker learner are parameter updated according to the gradients corresponding to the group of tracker agents to be trained, the main policy network of the tracker, and the target policy network of the tracker.

6. The control method according to claim 1, characterized in that: The local observations between the plurality of to-be-trained tracker agents and to-be-trained escaper agents include: in, represents the local observation between the jth tracker agent to be trained and the escaper agent to be trained at time node k, represents the state of the jth tracker agent to be trained at time node k, represents the state quantity of the escaper agent to be trained at time node k, and LVLH represents the relative orbital coordinate system of the virtual spacecraft.

7. The control method according to claim 1, characterized in that: The step of obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment includes: Obtain the state quantity, reward value and discount factor of the tracking agent group to be trained in the training environment; Calculate the corresponding action amount according to the strategy neural network in the group of tracking agents to be trained and the state amount in the training environment; Determining a state quantity at a next moment corresponding to the state quantity in the training environment according to a state transfer function in the training environment; The state quantity in the training environment, the reward value, the discount factor, the action quantity, and the state quantity at the next moment corresponding to the state quantity in the training environment are summarized as tracking experience data after interaction with the environment.

8. A control device for a multi-spacecraft cooperative game, characterized in that: include: A state quantity acquisition module is used to acquire the state quantities of multiple trackers with a cooperative relationship and the state quantity of the escaper in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities include position and speed; the multi-spacecraft cooperative pursuit scenario includes tracking the escaper by using the multiple trackers with a cooperative relationship within a preset game time, and the escaper will escape when being tracked; An action quantity generating module, used for inputting the current state quantities of the multiple trackers into the multiple tracker agents trained by the agent training method to obtain the current action quantities of the multiple trackers, and inputting the current state quantity of the escaper into the escaper agent trained by the agent training method to obtain the current action quantity of the escaper; wherein the action quantity includes control acceleration; A cooperative pursuit game module, used for controlling the multiple trackers to play a cooperative game according to the current action amounts of the multiple trackers and the current action amounts of the escaper, so that the multiple trackers in the cooperative game pursue and capture the escaper, until any tracker among the multiple trackers completes the capture of the escaper within the game duration, and then the cooperative pursuit game ends; The agent training module is used to build an agent training method, including: Constructing a Markov game model, a tracker experience pool, and an escaper experience pool; wherein the Markov game model includes a state quantity, an action quantity, a state transfer function, and a reward function in a cooperative pursuit game scenario; the state quantity is constructed by local observation quantities between multiple tracker agents to be trained and escaper agents to be trained; Constructing a group of tracker agents to be trained and a group of escaper agents to be trained, and constructing a tracker learner and an escaper learner; wherein the group of tracker agents to be trained includes a plurality of tracker agents to be trained; the tracker learner includes a plurality of tracker value neural networks and strategy neural networks; the escaper learner includes a value neural network and a strategy neural network of an escaper; Using the Markov game model as a training environment, obtaining tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, updating the tracker experience pool using the tracking experience data after interacting with the environment, and obtaining escape experience data of the escaper agents to be trained after interacting with the environment in the training environment, updating the escaper experience pool using the escape experience data after interacting with the environment; Calculate the value distribution and gradient corresponding to the group of tracker agents to be trained based on the multiple tracking experience data collected from the updated tracker experience pool and the local observations, and calculate the value distribution and gradient corresponding to the escaper agent to be trained based on the multiple escape experience data collected from the updated escaper experience pool; Based on the value distribution and gradient corresponding to the tracker agent group to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively, until the value neural network and the policy neural network in the updated tracker learner and the escaper learner meet the preset convergence requirements, and the network parameters of the value neural network and the policy neural network in the updated tracker learner that meet the preset convergence requirements are used as the network parameters of the tracker agent group to be trained, and the network parameters of the value neural network and the policy neural network in the updated escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, so as to obtain the trained tracker agent group and escaper agent.

9. A control device for a multi-spacecraft cooperative game, characterized in that: include: Memory, used to store programs; A processor, configured to implement the control method according to any one of claims 1 to 7 by executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the control method according to any one of claims 1 to 7.

Images