A control method, device, equipment, and medium for multi-spacecraft cooperative game.

Abstract

A multi-spacecraft cooperative game control method, device, equipment and medium, including inputting the current state quantity of a plurality of trackers having a cooperative relationship in a multi-spacecraft cooperative pursuit and escape scene into a plurality of tracker agents trained by an agent training method respectively to obtain the current action quantity of the plurality of trackers, inputting the current state quantity of an escape agent into an escape agent trained by an agent training method to obtain the current action quantity of the escape agent; controlling the plurality of trackers to cooperate in game according to the current action quantity of the plurality of trackers and the current action quantity of the escape agent, so that the plurality of trackers in cooperative game capture the escape agent, and the cooperative pursuit and escape game ends until any tracker in the plurality of trackers completes the capture of the escape agent within the game duration. The technical problem of poor cooperative game effect between multi-spacecrafts when facing multi-spacecraft orbital game problem is solved.

Description

Technical Field

[0001] This invention relates to the field of spacecraft control technology, specifically to a control method, device, equipment, and medium for multi-spacecraft cooperative game. Background Technology

[0002] With the recent development of satellite formation and swarm technologies, multi-spacecraft cooperative control technology has gained increasing attention. Multi-satellite swarm control and application can significantly enhance space surveillance, approach, and manipulation capabilities. In the context of orbital game theory, utilizing multiple spacecraft in the game can also improve overall capabilities through numerical advantage. For example, a spacecraft with weaker maneuverability can cooperate to capture a more maneuverable target, or a spacecraft with stronger maneuverability can cooperate to capture a target in a shorter time or with a higher success rate. However, cooperative control in multi-spacecraft game theory involves both cooperative and adversarial relationships. Using traditional differential game theory methods to model and design the correlation of objective functions among spacecraft, information exchange, and cooperative constraints is difficult and inefficient. Artificial intelligence methods do not require solving complex mathematical equations; they can enable agents to learn and improve through data collection and training, offering high flexibility. However, existing artificial intelligence methods applied to multi-spacecraft cooperative orbital game theory still suffer from problems such as poor adaptability to orbital dynamics environments, low strategy application efficiency, and poor training effects. Summary of the Invention

[0003] The main technical problem this invention addresses is that when applying artificial intelligence methods to multi-spacecraft cooperative orbital game problems, there are issues such as poor adaptability to orbital dynamics environments, low efficiency in strategy application, and poor training effects, resulting in poor performance in cooperative games among multiple spacecraft.

[0004] According to the first aspect, one embodiment provides a control method for multi-spacecraft cooperative game, comprising:

[0005] The state variables of multiple trackers with a cooperative relationship and the state variables of the escape vehicle are obtained in a multi-spacecraft cooperative pursuit and escape scenario; wherein, the state variables include position and velocity; the multi-spacecraft cooperative pursuit and escape scenario includes using the multiple trackers with a cooperative relationship to track the escape vehicle within a preset game time, and the escape vehicle will escape when being tracked.

[0006] The current state values ​​of the multiple trackers are respectively input into multiple tracker agents trained using the agent training method to obtain the current action values ​​of the multiple trackers; the current state value of the escaper is input into the escaper agent trained using the agent training method to obtain the current action value of the escaper; wherein, the action value includes control acceleration.

[0007] Based on the current action amount of the multiple trackers and the current action amount of the escapee, the multiple trackers are controlled to cooperate in a game, so that the multiple trackers in the cooperative game chase and capture the escapee until any tracker among the multiple trackers completes the capture of the escapee within the game time, then the cooperative chase and capture game ends.

[0008] The agent training method includes:

[0009] A Markov game model, a tracker experience pool, and an escaper experience pool are constructed. The Markov game model includes state variables, action variables, state transition functions, and reward functions in a cooperative pursuit-escape game scenario. The state variables are constructed through local observations between multiple tracker agents to be trained and escaper agents to be trained.

[0010] Construct a group of tracker agents to be trained and an escaper agent to be trained, and construct tracker learners and escaper learners; wherein, the group of tracker agents to be trained includes multiple tracker agents to be trained; the tracker learners include multiple value neural networks and policy neural networks of the trackers; the escaper learners include the value neural network and policy neural network of the escapers.

[0011] Using the Markov game model as the training environment, the tracking experience data of the group of trackers to be trained after interacting with the environment in the training environment is obtained, and the tracker experience pool is updated using the tracking experience data after interacting with the environment. The escape experience data of the escape agent to be trained after interacting with the environment in the training environment is obtained, and the escape experience pool is updated using the escape experience data after interacting with the environment.

[0012] The value distribution and gradient of the group of tracker agents to be trained are calculated based on multiple tracking experience data collected from the updated tracker experience pool and the local observations. The value distribution and gradient of the escape agent to be trained are calculated based on multiple escape experience data collected from the updated escaper experience pool.

[0013] Based on the value distribution and gradient corresponding to the group of tracker agents to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively until the updated value neural network and policy neural network in the tracker learner and the escaper learner meet the preset convergence requirements. Then, the network parameters of the updated value neural network and policy neural network in the tracker learner that meet the preset convergence requirements are used as the network parameters of the group of tracker agents to be trained, and the network parameters of the updated value neural network and policy neural network in the escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, thus obtaining the trained group of tracker agents and escaper agents.

[0014] In some embodiments, the gradient corresponding to the group of tracker agents to be trained is calculated based on multiple tracking experience data collected from the updated tracker experience pool and the local observations, including:

[0015]

[0016] in, This represents the gradient corresponding to the group of tracker agents to be trained. J represents the network parameters of the policy neural network corresponding to the j-th tracker agent in the group of tracker agents to be trained. P P represents the payment function for the group of tracker agents to be trained. j This represents the j-th tracker agent in the group of tracker agents to be trained. This represents the expectation when the state follows a fixed policy distribution. This represents the policy neural network of the j-th tracker agent to be trained. Regarding network parameter θ Pj gradient, This represents the gradient of the value neural network with respect to the action policy of the j-th tracker agent to be trained. The value neural network represents the group of tracker agents to be trained, where N represents the number of tracker agents to be trained. Let s represent the local observations of the j-th tracker agent in the group of tracker agents to be trained, and s represent the state variables. This represents the action amount of the Nth tracker agent in the group of tracker agents to be trained. This represents the action amount of the j-th tracker agent in the group of tracker agents to be trained. Let represent the policy neural network of the j-th tracker agent in the group of tracker agents to be trained.

[0017] In some embodiments, the reward function of the Markov game model includes a process reward function and a terminal reward function;

[0018] The reward function for the process is:

[0019]

[0020] in, This represents the process reward for the tracker agent. d represents the process reward of the escape agent. k d represents the minimum distance to the current node. k-1 d0 represents the minimum distance to the previous node, K represents the terminal node, and d0 represents the distance at the initial time.

[0021] The terminal reward function is:

[0022]

[0023] in, The terminal reward represents the tracker agent's reward. C represents the terminal reward for the escape agent, and C represents the reward value for meeting a preset threshold.

[0024] In some embodiments, a main network is constructed based on the parameters of the value neural networks and policy neural networks of multiple trackers in the tracker learner, and the parameters of the value neural networks and policy neural networks of the escapers in the escaper learner; wherein the main network includes the main value network and main policy network of the trackers, and the main value network and main policy network of the escapers.

[0025] The target network is constructed based on the network structure and network parameters of the main network; wherein, the target network includes the target value network and target strategy network of the tracking party, and the target value network and target strategy network of the escaping party.

[0026] In some embodiments, the parameters of the tracker learner are updated based on the value distribution and gradient corresponding to the group of tracker agents to be trained, including:

[0027] The corresponding value loss function is calculated based on the value distribution of the group of tracker agents to be trained, and the parameters of the value neural networks of multiple trackers in the tracker learners are updated based on the value loss function, the main value network of the tracker, and the target value network of the tracker.

[0028] The parameters of the policy neural networks of multiple trackers in the tracker learner are updated based on the gradient corresponding to the group of tracker agents to be trained, the main policy network of the tracker agent, and the target policy network of the tracker agent.

[0029] In some embodiments, the local observations between the plurality of trainable tracker agents and trainable escaper agents include:

[0030]

[0031] in, This represents the local observation between the j-th trainable tracker agent and the trainable escape agent at time node k. This represents the state of the j-th tracker agent to be trained at time node k. LVLH represents the state variables of the escape agent to be trained at time node k, and LVLH represents the virtual spacecraft relative to the orbital coordinate system.

[0032] In some embodiments, obtaining the tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment includes:

[0033] Obtain the state variables, reward values, and discount factors of the group of tracking agents to be trained in the training environment;

[0034] The corresponding action quantities are calculated based on the policy neural network in the group of tracking agents to be trained and the state quantities in the training environment.

[0035] The state quantity at the next moment corresponding to the state quantity in the training environment is determined based on the state transition function in the training environment.

[0036] The state variables in the training environment, the reward value, the discount factor, the action quantity, and the state variables at the next moment corresponding to the state variables in the training environment are summarized into tracking experience data after interaction with the environment.

[0037] According to the second aspect, one embodiment provides a control device for multi-spacecraft cooperative game, comprising:

[0038] The state quantity acquisition module is used to acquire the state quantities of multiple trackers with cooperative relationships and the state quantities of the escape vehicle in a multi-spacecraft cooperative pursuit and escape scenario; wherein, the state quantities include position and velocity; the multi-spacecraft cooperative pursuit and escape scenario includes using the multiple trackers with cooperative relationships to track the escape vehicle within a preset game time, and the escape vehicle will escape when being tracked.

[0039] The motion quantity generation module is used to input the current state quantities of the multiple trackers into multiple tracker agents trained using the agent training method to obtain the current motion quantities of the multiple trackers, and to input the current state quantity of the escaper into the escaper agent trained using the agent training method to obtain the current motion quantity of the escaper; wherein, the motion quantity includes control acceleration.

[0040] The cooperative pursuit and escape game module is used to control the multiple trackers to engage in cooperative game based on the current action amount of the multiple trackers and the current action amount of the escapee, so that the multiple trackers in the cooperative game pursue and capture the escapee until any tracker among the multiple trackers completes the capture of the escapee within the game time, then the cooperative pursuit and escape game ends.

[0041] The agent training module is used to construct agent training methods, including:

[0042] A Markov game model, a tracker experience pool, and an escaper experience pool are constructed. The Markov game model includes state variables, action variables, state transition functions, and reward functions in a cooperative pursuit-escape game scenario. The state variables are constructed through local observations between multiple tracker agents to be trained and escaper agents to be trained.

[0043] Construct a group of tracker agents to be trained and an escaper agent to be trained, and construct tracker learners and escaper learners; wherein, the group of tracker agents to be trained includes multiple tracker agents to be trained; the tracker learners include multiple value neural networks and policy neural networks of the trackers; the escaper learners include the value neural network and policy neural network of the escapers.

[0044] Using the Markov game model as the training environment, the tracking experience data of the group of trackers to be trained after interacting with the environment in the training environment is obtained, and the tracker experience pool is updated using the tracking experience data after interacting with the environment. The escape experience data of the escape agent to be trained after interacting with the environment in the training environment is obtained, and the escape experience pool is updated using the escape experience data after interacting with the environment.

[0045] The value distribution and gradient of the group of tracker agents to be trained are calculated based on multiple tracking experience data collected from the updated tracker experience pool and the local observations. The value distribution and gradient of the escape agent to be trained are calculated based on multiple escape experience data collected from the updated escaper experience pool.

[0046] Based on the value distribution and gradient corresponding to the group of tracker agents to be trained, and the value distribution and gradient corresponding to the escaper agent to be trained, the parameters of the tracker learner and the escaper learner are updated respectively until the updated value neural network and policy neural network in the tracker learner and the escaper learner meet the preset convergence requirements. Then, the network parameters of the updated value neural network and policy neural network in the tracker learner that meet the preset convergence requirements are used as the network parameters of the group of tracker agents to be trained, and the network parameters of the updated value neural network and policy neural network in the escaper learner that meet the preset convergence requirements are used as the network parameters of the escaper agent to be trained, thus obtaining the trained group of tracker agents and escaper agents.

[0047] According to a third aspect, one embodiment provides a control device for multi-spacecraft cooperative game, comprising:

[0048] Memory, used to store programs;

[0049] A processor for implementing a control method by executing a program stored in the memory.

[0050] According to a fourth aspect, one embodiment provides a computer-readable storage medium storing a program that can be executed by a processor to implement a control method.

[0051] According to the control method, apparatus, device, and computer-readable storage medium for multi-spacecraft cooperative game theory described in the above embodiments, a group of tracker agents to be trained and an escaper agent to be trained are constructed during the training process. Multiple trained tracker agents and escaper agents are obtained using an agent training method. Each spacecraft can complete cooperative control relying only on local observations, improving the control efficiency of multi-spacecraft decision-making. A Markov game model constructed based on the state variables, action variables, state transition functions, and reward functions in the cooperative pursuit-escape game scenario is used as the training environment, thus adapting to the orbital dynamics environment. The trained multiple tracker agents with cooperative game theory capabilities and escaper agents with multi-target escape capabilities can better control multiple trackers to engage in cooperative game theory, enabling the multiple trackers in the cooperative game to pursue and capture the escaper. Attached Figure Description

[0052] Figure 1 This is a control flowchart for a multi-spacecraft cooperative game in an embodiment of this application.

[0053] Figure 2 A flowchart of an embodiment of an agent training method;

[0054] Figure 3This is a flowchart of one embodiment for obtaining tracking experience data of a group of tracker agents to be trained after interacting with the environment in the training environment;

[0055] Figure 4 This is a flowchart of updating the parameters of a tracker learner based on the value distribution and gradient corresponding to a group of tracker agents to be trained, in one embodiment.

[0056] Figure 5 This is a comparison chart of capture success rates of different tracking and escape strategies in a random initial state space in one embodiment.

[0057] Figure 6a and Figure 6b The image shows a trajectory diagram in one embodiment where all trackers use the LQ-P strategy and all trackers use the DRL-P strategy.

[0058] Figure 7a and Figure 7b In one embodiment, the off-target results of trackers using the LQ-P strategy and the off-target results of trackers using the DRL-P strategy are presented in a random space.

[0059] Figure 8 This is a block diagram of the control structure for a multi-spacecraft cooperative game in one embodiment;

[0060] Figure 9 This is a schematic diagram of the structure of a control device for multi-spacecraft cooperative game in one embodiment. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0062] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0063] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0064] Please refer to Figure 1 This invention proposes a control method for cooperative game among multiple spacecraft, including steps S1 to S3, which are described in detail below.

[0065] Step S1: Obtain the state variables of multiple trackers and escapers that have a cooperative relationship in a multi-spacecraft cooperative pursuit scenario.

[0066] In some embodiments, the multi-spacecraft cooperative pursuit scenario includes a scenario in which multiple trackers with a cooperative relationship track the escape vehicle within a preset game duration, and the escape vehicle will escape while being tracked by multiple trackers.

[0067] In some embodiments, four trackers with a cooperative relationship are used to capture an escapee that is attempting to escape. The state variables of the multiple trackers include the positions and velocities of the multiple trackers, and the state variables of the escapee include the positions and velocities of the escapee.

[0068] For example, the escape vehicle initially operates on an orbit with the element C. r =[6778.137km,0,0°,0°,0°,0°,On a circular orbit, with the initial point as the reference point, the relative orbital position and velocity of the escape vehicle are [0,0,0,0,0,0]. T The four trackers initially move relative to the escaper in a coplanar, natural orbital configuration. The orbital radius and the phase angle of the tracker within the orbital ellipse are randomly given in the spaces a∈[2km,40km] and f0∈[0,2π].

[0069] In some embodiments, the coplanar natural flight configurational motion can be represented as:

[0070]

[0071] in, and Indicates tracker P j Location, and Indicates tracker P j The velocity is given by ω, where 'a' represents the length of the major semi-axis of the ellipse formed by the tracker relative to the escaper's trajectory, 'b' represents the length of the minor semi-axis of the ellipse formed by the tracker relative to the escaper's trajectory, and ω represents the average angular velocity of the reference orbit. Indicates tracker P j The initial phase, t represents the time value when the tracker flies naturally around the escape vehicle.

[0072] Step S2: Input the current state values ​​of multiple trackers into multiple tracker agents trained using the agent training method to obtain the current action values ​​of multiple trackers. Input the current state value of the escaper into the escaper agent trained using the agent training method to obtain the current action value of the escaper.

[0073] In some embodiments, the current motion of multiple trackers and the current motion of the escaper include control acceleration.

[0074] Step S3: Based on the current action amount of multiple trackers and the current action amount of the escapee, control multiple trackers to cooperate in a game, so that multiple trackers in the cooperative game chase and capture the escapee until any tracker among the multiple trackers completes the capture of the escapee within the game time, then the cooperative chase and escape game ends.

[0075] In some embodiments, the current motion magnitudes of the multiple trackers and the escaper include control acceleration; for example, the control acceleration magnitudes of both the trackers and the escaper are 0.04 m / s². 2 The direction can be freely chosen, the game duration is 1000s, and the capture radius is 1km. Within the given game duration, once any tracker among the multiple trackers completes the capture of the escapee, it is considered that the tracker has successfully tracked it; otherwise, it is considered that the escapee has successfully escaped, and the cooperative pursuit game ends.

[0076] In some embodiments, multiple trackers and escapers respectively minimize or maximize the minimum distance between their trajectories. Within a given game duration, if there exists a moment when the distance between any tracker and the escaper is less than the capture radius, the capture is considered complete, and the trackers win. Otherwise, the escapers win. Therefore, the capture condition can be expressed as:

[0077]

[0078] Among them, t k Indicates the current time, t fThe time at which the game ends is indicated, N represents the number of tracker agents in the group to be trained, k represents the time point, and d represents the time of the game's end. h P represents the minimum distance of all trackers and escapers from the initial time to the current time. j Let J represent the j-th tracker out of a set of trackers, and E represent the escape device. Indicates tracker P j The coordinate value on the x-axis, This represents the x-coordinate of the escape device E. Indicates tracker P j The coordinate value on the y-axis, This represents the y-coordinate of the escape device E. Indicates tracker P j The coordinate value on the z-axis, R represents the z-coordinate of the escape device E. c Indicates the capture radius.

[0079] Please refer to Figure 2 In some embodiments, the agent training method includes steps S21 to S25, which are described in detail below.

[0080] Step S21: Construct the Markov game model, the tracker experience pool, and the escaper experience pool.

[0081] In some embodiments, the Markov game model includes state variables, action variables, state transition functions, and reward functions in a cooperative pursuit game scenario.

[0082] In some embodiments, a virtual spacecraft relative to its orbital LVLH coordinate system is established. A virtual spacecraft near the multiple spacecraft is used as a reference spacecraft. A local orbital coordinate system is established, with its origin as the centroid of the reference spacecraft, forming a right-handed coordinate system with the ox and oz axes. The ox axis is along the radial direction of the reference spacecraft, the oy axis is along the tangent of the reference spacecraft's trajectory, and the oz axis is along the normal to the reference spacecraft's orbital plane. The state of the multiple spacecraft at each time point is described by the difference in relative state between the virtual spacecraft and its orbital LVLH coordinate system, expressed as:

[0083]

[0084] Among them, s k This represents the state quantity at time node k. This represents the state of tracker P1 at time node k, where P1 represents the first tracker agent, x. k E This represents the state of the escaper E at time node k, where E represents the escaper agent and N represents the number of tracker agents.

[0085] In some embodiments, This represents the state variables of spacecraft i in the virtual system. Indicates relative position, Let T denote velocity, and T denote the transpose sign. Regarding spacecraft i, we know that it includes multiple tracker agents P1, P2, ..., P... N Any tracker agent in the system can also be an escape agent E.

[0086] In some embodiments, the state variables can be constructed from local observations between multiple trainable tracker agents and trainable escaper agents, and are represented as follows:

[0087]

[0088] Among them, s k This represents the state quantity at time node k. This represents the j-th tracker agent P to be trained at time node k. j Local observations between the escape agent E and the agent to be trained. This indicates that at time node k, the j-th tracker agent P to be trained... j The state variables, LVLH represents the state variables of the escape agent E to be trained at time node k, T represents the transpose symbol, and N represents the number of tracker agents to be trained.

[0089] In some embodiments, the action quantities in the Markov game model under the cooperative pursuit game scenario include the control acceleration of each spacecraft. Each spacecraft's control acceleration is a vector with a fixed magnitude but free direction. To simplify the problem, in this embodiment, the thrust direction angles α∈[-π,π] and β∈[-π / 2,π / 2] of the spacecraft can be used as the action quantities output by the policy network.

[0090]

[0091] in, This represents the amount of motion of spacecraft i at time node k. and These represent two different thrust direction angles of spacecraft i at time node k.

[0092] In some embodiments, the amplitude of the motion is u i =u i [cosβ i cosα i ,cosβ i sinα i sinβi ] T , where α i and β i The two thrust direction angles of spacecraft i are represented by T, which represents the transpose symbol.

[0093] In some embodiments, the policy function α~π(·|s) from state to action in the Markov game model replaces the guidance law of the spacecraft's closed-loop control.

[0094] In some embodiments, regarding the state transition function in the Markov game model, under continuous thrust control, the state transition of each spacecraft needs to be obtained by numerical integration of the ordinary differential equation. The time interval for each integration is the time between two maneuver decisions. Under the condition of linear dynamic equations, it can be indirectly expressed as an integral function of the state transition matrix:

[0095]

[0096] in, This represents the state of spacecraft i at the next time node k+1. This represents the state of spacecraft i at time node k. This represents the amount of action of the escape agent E at time node k. Φ(t) represents the action of tracker agent P1 at time node k, N represents the number of tracker agents, and Φ(t) represents the action of tracker agent P1 at time node k. k+1 ,t k ) indicates from t k to t k+1 The state transition matrix, t k This indicates that at the corresponding time point k, t k+1 This indicates that at the corresponding time point k+1 in the next time node, Φ v (t k+1 ,τ) is the state transition matrix with respect to velocity, τ represents the preset parameter, and u(τ) represents the control acceleration.

[0097] In some embodiments, the state transition matrix from t0 to t can be represented as:

[0098]

[0099] Where Φ(t,t0) represents the state transition matrix from t0 to t, Φ r (t,t0) represents the state transition matrix with respect to position in Φ(t,t0), Φ v (t,t0) represents the state transition matrix with respect to velocity in Φ(t,t0), τ represents the preset parameters, s=sinτ, c=cosτ, τ=ω(t-t0).

[0100] In some embodiments, the reward function of the Markov game model includes a process reward function and a terminal reward function, wherein the process reward function is:

[0101]

[0102] in, This represents the process reward for the tracker agent. d represents the process reward of the escape agent. k d represents the minimum distance to the current node k. k-1 d0 represents the minimum distance from the previous node k-1, where K represents the terminal node and d0 represents the distance at the initial time.

[0103] In some embodiments, the terminal reward function is:

[0104]

[0105] in, The terminal reward represents the tracker agent's reward. C represents the terminal reward for the escape agent, and C represents the reward value for meeting a preset threshold.

[0106] In some embodiments, C represents a large reward value given to the tracker agent when the terminal completes capture. The size of this reward value can be defined by the user, but it must be much larger than the process reward. In the embodiments of this application, C = 10. 5 .

[0107] In some embodiments, for the process reward function, the tracker's reward is positive if the current node is closer to the previous node. For the terminal reward function, the spacecraft's terminal reward is determined based on whether the distance within the game time enters the capture radius. Meanwhile, assuming a discount factor of 1, the reward functions of the tracker and the escape vehicle are opposite, thus forming a zero-sum game.

[0108] In some embodiments, the tracker experience pool and the escaper experience pool contain experience data related to the tracker and experience data related to the escaper, respectively.

[0109] Step S22: Construct a group of tracker agents to be trained and an escaper agent to be trained, and construct tracker learners and escaper learners.

[0110] In some embodiments, a group of tracker agents to be trained is constructed, comprising multiple tracker agents to be trained, each of which is composed of multiple tracker agent policy neural networks. The tracker agent policy neural network can be represented as follows: N represents the number of tracker agents to be trained, P j Let j represent the j-th tracker agent to be trained. This represents the tracker agent P to be trained. j The network parameters of the strategy neural network, This represents the local observations of the j-th tracker agent to be trained at time node k. The network parameters of the policy neural network of the tracker agent to be trained are copied from the network parameters of the policy neural network of the tracker learner.

[0111] In some embodiments, an escape agent to be trained is constructed, and each escape agent to be trained consists of a single escape agent policy neural network, wherein the escape agent policy neural network can be represented as follows: θ E This represents the network parameters of the policy neural network of the escape agent E to be trained. The network parameters of the escape agent's policy neural network are copied from the network parameters of the escape learner's policy neural network.

[0112] In some embodiments, a tracker learner and an escaper learner are constructed, wherein the tracker learner includes multiple tracker value neural networks and policy neural networks, and the policy neural networks of the multiple trackers are... Value neural networks are The corresponding network parameters in the tracker learners are initialized as follows:

[0113] In some embodiments, the escaper learner includes the escaper's value neural network. and policy neural networks The network parameters corresponding to the escaper learners are initialized to θ. E ,φ E .

[0114] In some embodiments, the main network is constructed based on the network parameters of the value neural networks and policy neural networks of multiple trackers in the tracker learners, and the network parameters of the value neural networks and policy neural networks of the escapers in the escaper learners. This can also be understood as constructing the main network based on the initialized parameters. φ P and θ E ,φ E Build the main network.

[0115] In some embodiments, a target network is constructed based on the network structure and parameters of the main network. The network parameters of the target network are copied from the main network. θ E' =θ E , φ P' =φP , φ E' =φ E .

[0116] In some embodiments, the main network includes the main value network and main policy network of the tracking party, and the main value network and main policy network of the escaping party. The target network includes the target value network and target policy network of the tracking party, and the target value network and target policy network of the escaping party.

[0117] In some embodiments, multiple groups of tracker agents to be trained and multiple escape agents to be trained can be constructed, wherein one group of tracker agents to be trained corresponds to one escape agent to be trained, thereby enabling parallel training of the agents.

[0118] Step S23: Using the Markov game model as the training environment, obtain the tracking experience data of the group of trackers to be trained after interacting with the environment in the training environment, update the tracker experience pool using the tracking experience data after interacting with the environment, and obtain the escape experience data of the escape agent to be trained after interacting with the environment in the training environment, update the escaper experience pool using the escape experience data after interacting with the environment.

[0119] In some embodiments, training parameters are initialized before training, including the batch size B. M The number of trackers N, and the number of tracker agents A in the group to be trained. KP The number A of escape agent agents to be trained KE Exploring constant ò, value neural network learning rate l α The learning rate of the policy neural network is l β .

[0120] Please refer to Figure 3 In some embodiments, step S23 acquires tracking experience data of the group of tracker agents to be trained after interacting with the environment in the training environment, including steps S231 to S234, which are described in detail below.

[0121] Step S231: Obtain the state variables, reward values, and discount factors of the group of tracking agents to be trained in the training environment.

[0122] Step S232: Calculate the corresponding action quantities based on the policy neural network in the group of tracking agents to be trained and the state quantities in the training environment.

[0123] In some embodiments, the amount of motion is calculated using the following formula:

[0124]

[0125] in, Indicates the amount of action. This represents the policy neural network in the group of tracking agents to be trained. This represents the tracker agent P in the group of tracking agents to be trained in the training environment. j The local observations are σ, which represents the standard deviation, and n(0,1) represents the standard normal distribution.

[0126] Step S233: Determine the state quantity at the next moment corresponding to the state quantity in the training environment based on the state transition function in the training environment.

[0127] Step S234: Summarize the state variables, reward values, discount factors, action variables, and the state variables of the next time step corresponding to the state variables in the training environment into tracking experience data after interacting with the environment.

[0128] In some embodiments, the method for obtaining escape experience data of the escape agent to be trained after interacting with the environment in the training environment is the same as the method for obtaining tracking experience data of the group of trackers to be trained after interacting with the environment in the training environment. The difference lies in the following formula used when calculating the motion quantity:

[0129]

[0130] in, Indicates the amount of action. s represents the policy neural network of the escape agent to be trained. k Let σ represent the state variables in the training environment, σ represent the standard deviation, and N(0,1) represent the standard normal distribution.

[0131] In some embodiments, the tracker agent and the escape agent interact with the environment and periodically update their respective policy network parameters.

[0132] Step S24: Calculate the value distribution and gradient of the group of tracker agents to be trained based on multiple tracking experience data and local observations collected from the updated tracker experience pool, and calculate the value distribution and gradient of the escape agent to be trained based on multiple escape experience data collected from the updated escaper experience pool.

[0133] In some embodiments, the value distribution corresponding to the group of tracker agents to be trained is calculated according to the following formula:

[0134]

[0135] Among them, Y P Let s be the value distribution corresponding to the group of tracker agents to be trained. k This represents the state quantity at the current time node k. This represents the amount of action of escaper E at the current time point k. This represents the action amount of tracker P1 at the current time node k, and N represents the number of tracker agents to be trained in the group of tracker agents to be trained. γ represents the reward at the current time point k, and γ represents the discount factor. s represents the value neural network corresponding to the group of tracker agents to be trained. k+1 This represents the state quantity at the next time node k+1. This represents the amount of action of tracker P1 at the next time node k+1. This represents the value function of the next node.

[0136] In some embodiments, the state and motion quantities in the above formulas are calculated based on multiple tracking experience data collected from the updated tracker experience pool.

[0137] In some embodiments, the value distribution corresponding to the escape agent to be trained is calculated according to the following formula:

[0138]

[0139] Among them, Y E s represents the value distribution corresponding to the escape agent to be trained. k This represents the state quantity at the current time node k. This represents the amount of action of escaper E at the current time point k. This represents the action amount of tracker P1 at the current time node k, and N represents the number of tracker agents to be trained in the group of tracker agents to be trained. γ represents the reward at the current time point k, and γ represents the discount factor. s represents the value distribution corresponding to the escape agent to be trained. k+1 This represents the state quantity at the next time node k+1. This represents the amount of action of escaper E at the next time node k+1. This represents the value function of the next node.

[0140] In some embodiments, the gradient corresponding to the group of tracker agents to be trained is calculated based on multiple tracking experience data and local observations collected from the updated tracker experience pool, including:

[0141]

[0142] in, This represents the gradient corresponding to the group of tracker agents to be trained. J represents the network parameters of the policy neural network corresponding to the j-th tracker agent in the group of tracker agents to be trained. P P represents the payment function for the group of tracker agents to be trained. j This represents the j-th tracker agent in the group of tracker agents to be trained. This represents the expectation when the state follows a fixed policy distribution. This represents the policy neural network of the j-th tracker agent to be trained. Regarding network parameter θ Pj gradient, This represents the gradient of the value neural network with respect to the action policy of the j-th tracker agent to be trained. Let N represent the value neural network of the group of tracker agents to be trained. Let s represent the local observations of the j-th tracker agent in the group of tracker agents to be trained, and let s represent the state variables. This represents the action amount of the Nth tracker agent in the group of tracker agents to be trained. This represents the action amount of the j-th tracker agent in the group of tracker agents to be trained. Let represent the policy neural network of the j-th tracker agent in the group of tracker agents to be trained.

[0143] In some embodiments, the gradient corresponding to the escape agent to be trained is calculated based on multiple escape experience data collected from the updated escape agent experience pool, including:

[0144]

[0145] in, θ represents the gradient corresponding to the escape agent to be trained. E The parameters of the policy neural network corresponding to the escaper to be trained are denoted by s, which represents the state variables, and a represents the network parameters of the escaper to be trained. E This represents the amount of action of the escape agent to be trained.

[0146] Step S25: Based on the value distribution and gradient of the tracker agent group to be trained, and the value distribution and gradient of the escape agent to be trained, update the parameters of the tracker learner and the escape agent learner respectively, until the value neural network and policy neural network in the updated tracker learner and escape agent learner meet the preset convergence requirements. Then, use the network parameters of the updated value neural network and policy neural network in the tracker learner that meet the preset convergence requirements as the network parameters of the tracker agent group to be trained, and use the network parameters of the updated value neural network and policy neural network in the escape agent learner that meet the preset convergence requirements as the network parameters of the escape agent to be trained, thus obtaining the trained tracker agent group and escape agent.

[0147] Please refer to Figure 4 In some embodiments, step S25 updates the parameters of the tracker learner based on the value distribution and gradient corresponding to the group of tracker agents to be trained, including steps S251 to S252, which are described in detail below.

[0148] Step S251: Calculate the corresponding value loss function based on the value distribution of the group of tracker agents to be trained, and update the parameters of the value neural networks of multiple trackers in the tracker learners based on the value loss function, the main value network of the tracker, and the target value network of the tracker.

[0149] In some embodiments, a cross-entropy loss function is defined to calculate the difference between the latest value distribution and the old value distribution, that is, to calculate the value loss function of the group of tracker agents to be trained. The formula for calculating the value loss function is as follows:

[0150]

[0151] Among them, Loss(φ P Y represents the value loss function of the group of tracker agents to be trained. P The value distribution corresponding to the group of tracker agents to be trained. φ represents the old value distribution of the group of tracker agents to be trained. P The network parameters of the main value network of the tracker are represented by P, where P represents the tracker, or tracker.

[0152] In some embodiments, for updating the value neural networks of multiple trackers in the tracker learner, the value loss function should be minimized. Therefore, the gradient of the value loss function with respect to the network parameters of the value neural network is first calculated. The update direction of the network parameters can be obtained through gradient descent. Then, the target value network of the tracker is slowly updated along the parameter direction of the main value network of the tracker, thereby updating the parameters of the value neural networks of multiple trackers in the tracker learner. The update process is expressed by the following formula:

[0153]

[0154] φ P' =(1-∈)φ P' +∈φ P for∈<<1

[0155] in, This represents the updated network parameters of the value neural network. Indicates the current network parameters, l α The learning rate of the neural network representing the value of the tracking party. This represents the update magnitude determined based on the gradient, φ. P φ represents the network parameters of the main value network of the tracking party. P' The network parameters of the target value network of the tracking party are represented, and ∈ represents the exploration constant.

[0156] Step S252: Update the parameters of the policy neural networks of multiple trackers in the tracker learner according to the gradients corresponding to the group of tracker agents to be trained, the main policy network of the tracker, and the target policy network of the tracker.

[0157] In some embodiments, the parameters of the policy neural networks of multiple trackers in the tracker learner are updated according to the gradient corresponding to the group of tracker agents to be trained, the main policy network of the tracker, and the target policy network of the tracker. The target policy neural network of the tracker is slowly updated along the parameter direction of the main policy network of the tracker, so as to realize the parameter update of the policy neural networks of multiple trackers in the tracker learner. The update process is expressed by the following formula:

[0158]

[0159] θ P' =(1-∈)θ P' +∈θ P

[0160] in, This represents the network parameters of the updated policy neural network. Indicates the current network parameters, l βThis represents the learning rate of the tracking neural network's policy. This represents the gradient of the group of tracker agents to be trained. θ represents the network parameters of the policy neural network of tracker P1 in the group of tracker agents to be trained, N represents the number of tracker agents to be trained, and θ represents the network parameters of the policy neural network of tracker P1 in the group of tracker agents to be trained. P θ represents the parameters of the policy neural network. P' ∈ represents the network parameters of the target policy neural network, and ∈ represents the exploration constant.

[0161] In some embodiments, the parameters of the escape learner are updated according to the value distribution and gradient corresponding to the escape agent to be trained. The parameter update process is consistent with the parameter update process of the group of tracker agents to be trained described above.

[0162] In some embodiments, the value loss function of the escape agent to be trained is calculated using the following formula:

[0163]

[0164] Among them, Loss(φ E Y represents the value loss function of the swarm of escape agent agents to be trained. E The value distribution corresponding to the escape agent to be trained. φ represents the old value distribution of the escape agent to be trained. E The network parameters of the main value network of the escapee are represented by E, which represents the pursuer, i.e., the escapee.

[0165] In some embodiments, updates to the value network parameters of the escape agent to be trained include:

[0166]

[0167] φ E' =(1-∈)φ E' +∈φ E for∈<<1

[0168] in, This represents the updated network parameters of the value neural network. Indicates the current network parameters, l α The learning rate of the neural network representing the value of the tracking party. This represents the update magnitude determined based on the gradient. φ represents the network parameters of the main value network of the tracking party. E' The network parameters of the target value network of the tracking party are represented, and ∈ represents the exploration constant.

[0169] In some embodiments, the parameters of the policy neural network of the escaper learner are updated, and the update process is expressed by the following formula:

[0170]

[0171] θ E' =(1-∈)θ E' +∈θ E

[0172] in, This represents the network parameters of the updated policy neural network. This indicates the current network parameters. The learning rate of the neural network represents the policy of the escaped party. θ represents the gradient of the escape agent to be trained. E θ represents the parameters of the policy neural network. E' ∈ represents the network parameters of the target policy neural network, and ∈ represents the exploration constant.

[0173] Please refer to Figure 5 In some embodiments, the capture success rates of different tracking and escape strategies differ in a random initial state space. Deep reinforcement learning is used to train both the tracker and the escaper simultaneously, resulting in a deep reinforcement learning-based tracking strategy (DRL-P) and a deep reinforcement learning-based escape strategy (DRL-E) after multiple training iterations. To test the agent's final pursuit and escape capabilities, game-based adversarial scenarios are conducted not only between agents but also between the agent and a spacecraft using traditional strategies. The traditional tracking strategies in the adversarial scenarios include Linear-quadratic pursuit (LQ-P) and Vector guidance (VG) strategies, while the traditional escape strategies include Linear-quadratic evasion (LQ-E) and Constant thrust direction (CTD) escape strategies. Figure 5 The results of these strategies working against each other are presented, and the capture success rate is statistically analyzed using Monte Carlo shooting. From Figure 5 It can be observed that regardless of the strategy employed by the tracker, the escape capability of the escape agent trained by reinforcement learning surpasses that of the two traditional escape mechanisms. For the tracking agent, the capture success rate is highest when the escape mechanism employs deep reinforcement learning or a constant thrust strategy; however, the tracking performance is slightly inferior to the other two tracking strategies when the escape mechanism uses a linear quadratic escape strategy.

[0174] Please refer to Figure 6a and Figure 6b ,exist Figure 6a In this context, all trackers use the LQ-P strategy. Figure 6b In this context, all trackers use the DRL-P strategy. For example... Figure 6a As shown, under the non-cooperative LQ-P strategy, all trackers independently pursue the escaper, and their trajectories almost intersect at a single point. Ultimately, none of the trackers manage to capture the escaper. However, under the DRL-P strategy, not all trackers pursue the escaper. Figure 6b As shown, P2 tracks the escaper toward its current position, but P1 tracks the escaper toward its future trajectory, which restricts its current escape direction. Ultimately, the escaper is captured by P2.

[0175] Please refer to Figure 7a and Figure 7b In some embodiments, Figure 7a This indicates that in the random space, the trackers all use the off-target results of the LQ-P strategy. Figure 7b This represents the miss distance results for the tracker using the DRL-P strategy in random space. In the miss distance results, the color distribution essentially represents the difficulty of capturing the escapee at different initial positions (horizontal and vertical axes) (the color represents the final miss distance; the darker the color, the closer the distance). From... Figure 7a and 7b The comparison shows that deep reinforcement learning training strategies are significantly more effective than traditional LQ strategies. However, regardless of the strategy used, the larger the radius of the initial enclosing ellipse, the greater the miss rate.

[0176] Please refer to Figure 8 This application provides a control structure block diagram for multi-spacecraft cooperative game theory, which applies a decentralized decision-making neural network structure. Each spacecraft can complete cooperative control by relying only on local observations, improving the control efficiency of multi-spacecraft decision-making. The trained tracker agent with cooperative game theory capabilities and escape agent with multi-target escape capabilities outperform traditional tracking and escape strategies.

[0177] Please refer to Figure 9 This application provides a control device for multi-spacecraft cooperative game, including a state quantity acquisition module 10, an action quantity generation module 20, a cooperative pursuit and escape game module 30, and an agent training module 40, which are described in detail below.

[0178] The state quantity acquisition module 10 is used to acquire the state quantities of multiple trackers with cooperative relationships and the state quantities of the escaper in a multi-spacecraft cooperative pursuit and escape scenario; wherein, the state quantities include position and velocity; the multi-spacecraft cooperative pursuit and escape scenario includes using the multiple trackers with cooperative relationships to track the escaper within a preset game time, and the escaper will escape when being tracked.

[0179] The motion quantity generation module 20 is used to input the current state quantities of multiple trackers into multiple tracker agents trained using the agent training method to obtain the current motion quantities of multiple trackers, and to input the current state quantities of the escaper into the escaper agent trained using the agent training method to obtain the current motion quantities of the escaper; wherein, the motion quantities include control acceleration.

[0180] The cooperative pursuit and escape game module 30 is used to control multiple trackers to engage in cooperative game based on the current action amount of multiple trackers and the current action amount of the escapee. This allows multiple trackers in the cooperative game to pursue and capture the escapee until any tracker among the multiple trackers completes the capture of the escapee within the game duration, at which point the cooperative pursuit and escape game ends.

[0181] Agent training module 40, used to construct agent training methods, includes:

[0182] A Markov game model, a tracker experience pool, and an escaper experience pool are constructed. The Markov game model includes state variables, action variables, state transition functions, and reward functions in a cooperative pursuit-escape game scenario. The state variables are constructed through local observations between multiple tracker agents to be trained and escaper agents to be trained.

[0183] Construct a group of tracker agents to be trained and an escape agent to be trained, and construct tracker learners and escape learners; wherein, the group of tracker agents to be trained includes multiple tracker agents to be trained; the tracker learners include multiple value neural networks and policy neural networks of trackers; the escape learners include the value neural network and policy neural network of escapers.

[0184] Using a Markov game model as the training environment, we obtain tracking experience data of the group of trackers to be trained after interacting with the environment in the training environment. We update the tracker experience pool using the tracking experience data after interacting with the environment. We also obtain escape experience data of the escape agent to be trained after interacting with the environment in the training environment. We update the escape agent experience pool using the escape experience data after interacting with the environment.

[0185] The value distribution and gradient of the group of tracker agents to be trained are calculated based on multiple tracking experience data and local observations collected from the updated tracker experience pool. The value distribution and gradient of the escape agent to be trained are calculated based on multiple escape experience data collected from the updated escaper experience pool.

[0186] Based on the value distribution and gradient of the group of tracker agents to be trained, and the value distribution and gradient of the escape agent to be trained, the parameters of the tracker learner and the escape agent learner are updated respectively until the value neural network and policy neural network in the updated tracker learner and escape agent learner meet the preset convergence requirements. The network parameters of the updated value neural network and policy neural network in the tracker learner that meet the preset convergence requirements are used as the network parameters of the group of tracker agents to be trained, and the network parameters of the updated value neural network and policy neural network in the escape agent learner that meet the preset convergence requirements are used as the network parameters of the escape agent to be trained, thus obtaining the trained group of tracker agents and escape agent.

[0187] In some embodiments, this application provides a control device for multi-spacecraft cooperative game, including: a memory for storing programs; and a processor for implementing a control method by executing the programs stored in the memory.

[0188] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0189] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for controlling multi-spacecraft cooperative game, characterized in that, The method comprises the following steps: acquiring state quantities of a plurality of chasers having a cooperative relationship and a state quantity of an evader in a multi-spacecraft cooperative pursuit scenario; wherein the state quantities comprise positions and velocities; the multi-spacecraft cooperative pursuit scenario comprises tracking the evader by the plurality of chasers having a cooperative relationship within a preset game duration, while the evader escapes when being tracked; the plurality of chasers having a cooperative relationship move in a coplanar natural fly-around configuration, and a fly-around radius and a phase angle of a chaser in the fly-around ellipse are randomly given in a space of a∈[2km, 40km], f0∈[0, 2π]; inputting the current state quantities of the plurality of chasers into a plurality of chaser agents trained by an agent training method to obtain current action quantities of the plurality of chasers, and inputting the current state quantity of the evader into an evader agent trained by the agent training method to obtain a current action quantity of the evader; wherein the action quantities comprise control accelerations; controlling the plurality of chasers to perform cooperative game according to the current action quantities of the plurality of chasers and the current action quantity of the evader, so that the plurality of chasers perform pursuit and capture on the evader in the cooperative game, and the cooperative pursuit game ends when any chaser of the plurality of chasers completes the capture of the evader within the game duration; wherein the agent training method comprises: constructing a Markov game model, a chaser experience pool and an evader experience pool; wherein the Markov game model comprises state quantities, action quantities, a state transition function and a reward function in a cooperative pursuit game scenario; the state quantities are constructed by local observations between a plurality of to-be-trained chaser agents and a to-be-trained evader agent; constructing a to-be-trained chaser agent group and a to-be-trained evader agent, and constructing a chaser learner and an evader learner; wherein the to-be-trained chaser agent group comprises a plurality of to-be-trained chaser agents; the chaser learner comprises a plurality of chaser value neural networks and policy neural networks; the evader learner comprises an evader value neural network and a policy neural network; wherein network parameters of a to-be-trained chaser agent policy neural network are copied from network parameters of a chaser learner policy neural network, and one to-be-trained chaser agent group corresponds to one to-be-trained evader agent; using the Markov game model as a training environment, acquiring tracking experience data of the to-be-trained chaser agent group after interacting with the environment in the training environment, updating the chaser experience pool by using the tracking experience data after interacting with the environment, and acquiring escape experience data of the to-be-trained evader agent after interacting with the environment in the training environment, and updating the evader experience pool by using the escape experience data after interacting with the environment. According to the plurality of tracking experience data collected from the updated tracker experience pool and the local observation, the value distribution and the gradient corresponding to the group of to-be-trained tracker agents are calculated, and according to the plurality of escape experience data collected from the updated escape experience pool, the value distribution and the gradient corresponding to the to-be-trained escape agent are calculated; Based on the value distribution and the gradient corresponding to the group of to-be-trained tracker agents and the value distribution and the gradient corresponding to the to-be-trained escape agent, the parameter of the tracker learner and the escape learner is updated respectively until the value neural network and the policy neural network in the updated tracker learner and the escape learner meet the preset convergence requirement, and then the network parameters of the value neural network and the policy neural network in the updated tracker learner meeting the preset convergence requirement are taken as the network parameters of the group of to-be-trained tracker agents, and the network parameters of the value neural network and the policy neural network in the updated escape learner meeting the preset convergence requirement are taken as the network parameters of the to-be-trained escape agent, so as to obtain the trained group of tracker agents and the escape agent.

2. The control method according to claim 1, characterized by, According to the plurality of tracking experience data collected from the updated tracker experience pool and the local observation, the value distribution and the gradient corresponding to the group of to-be-trained tracker agents are calculated, and according to the plurality of escape experience data collected from the updated escape experience pool, the value distribution and the gradient corresponding to the to-be-trained escape agent are calculated; wherein, denotes the gradient corresponding to the group of to-be-trained tracker agents, denotes the network parameters of the policy neural network corresponding to the jth to-be-trained tracker agent in the group of to-be-trained tracker agents, J P denotes the payment function of the group of to-be-trained tracker agents, P j denotes the jth to-be-trained tracker agent in the group of to-be-trained tracker agents, denotes the expectation when the state takes the fixed policy distribution, denotes the jth to-be-trained tracker agent policy neural network with respect to the network parameters θ Pj gradient, denotes the gradient of the value neural network with respect to the jth to-be-trained tracker agent action policy, denotes the value neural network of the group of to-be-trained tracker agents, N denotes the number of to-be-trained tracker agents, denotes the local observation of the jth to-be-trained tracker agent in the group of to-be-trained tracker agents, s denotes the state quantity, denotes the action of the Nth to-be-trained tracker agent in the group of to-be-trained tracker agents, denotes the action of the jth to-be-trained tracker agent in the group of to-be-trained tracker agents, denotes the policy neural network of the jth to-be-trained tracker agent in the group of to-be-trained tracker agents.

3. The control method according to claim 1, characterized by, The reward function of the Markov game model includes a process reward function and a terminal reward function; The process reward function is: wherein, represents the process reward of the tracker agent, represents the process reward of the evader agent, k represents the minimum distance of the current node, k-1 represents the minimum distance of the previous node, K represents the terminal node, and d0 represents the distance at the initial time. The terminal reward function is: wherein, denotes the terminal reward of the tracker agent, denotes the terminal reward of the evader agent, and C denotes a reward value satisfying a preset threshold.

4. The control method according to claim 1, characterized by, According to the parameters of the value neural network and the policy neural network of the plurality of tracking parties in the tracker learner and the parameters of the value neural network and the policy neural network of the escape party in the escape learner, a main network is constructed; wherein the main network includes a main value network and a main policy network of the tracking party, and a main value network and a main policy network of the escape party; According to the network structure and the network parameters of the main network, a target network is constructed; wherein the target network includes a target value network and a target policy network of the tracking party, and a target value network and a target policy network of the escape party.

5. The control method according to claim 4, characterized by, Based on the value distribution and the gradient corresponding to the group of to-be-trained tracker agents, the parameter of the tracker learner is updated, including: According to the value distribution corresponding to the group of to-be-trained tracker agents, a corresponding value loss function is calculated, and according to the value loss function, the main value network of the tracking party and the target value network of the tracking party, the parameter of the value neural network of the plurality of tracking parties in the tracker learner is updated; According to the gradient corresponding to the group of to-be-trained tracker agents, the main policy network of the tracking party and the target policy network of the tracking party, the parameter of the policy neural network of the plurality of tracking parties in the tracker learner is updated.

6. The control method according to claim 1, characterized by, The local observation between the plurality of to-be-trained tracker agents and the to-be-trained escape agent includes: wherein, represents the local observation between the jth trained chaser agent and the trained evader agent at time node k, represents the state quantity of the jth trained chaser agent at time node k, represents the state quantity of the trained evader agent at time node k, LVLH represents the virtual spacecraft relative orbit coordinate system.

7. The control method according to claim 1, characterized by, The acquisition of the tracking experience data of the group of to-be-trained tracker agents after interacting with the environment in the training environment includes: Obtaining state quantities of the group of to-be-trained tracker agents in the training environment, reward values, and discount factors; According to the policy neural network in the group of to-be-trained tracker agents and the state quantities in the training environment, corresponding action quantities are calculated; According to the state transition function in the training environment, the state quantities in the training environment are determined corresponding to the state quantities at the next time; The state quantities in the training environment, the reward values, the discount factors, the action quantities, and the state quantities in the training environment corresponding to the state quantities at the next time are summarized as tracking experience data after interacting with the environment.

8. A control device of a multi-spacecraft cooperative game, characterized by, Comprise: The state quantity acquisition module is used for obtaining the state quantities of a plurality of trackers having a cooperative relationship and the state quantity of an escape agent in a multi-spacecraft cooperative pursuit and escape scene; wherein the state quantity comprises position and velocity; the multi-spacecraft cooperative pursuit and escape scene comprises tracking the escape agent by the plurality of trackers having a cooperative relationship within a preset game duration, while being tracked, the escape agent will escape, the plurality of trackers having a cooperative relationship move in a coplanar natural flyby configuration, the radius of flyby and the phase angle of the tracker in the flyby ellipse are randomly given in the space of a∈[2km, 40km], f0∈[0, 2π]; The action quantity generation module is used for inputting the current state quantities of the plurality of trackers into a plurality of tracker agents trained by an agent training method to obtain the current action quantities of the plurality of trackers, and inputting the current state quantity of the escape agent into an escape agent trained by an agent training method to obtain the current action quantity of the escape agent; wherein the action quantity comprises control acceleration; The cooperative pursuit and escape game module is used for controlling the plurality of trackers to perform cooperative game according to the current action quantities of the plurality of trackers and the current action quantity of the escape agent, so that the plurality of trackers in cooperative game capture the escape agent, until any tracker in the plurality of trackers completes the capture of the escape agent within the game duration, then the cooperative pursuit and escape game ends; The agent training module is used for constructing an agent training method, comprising: Constructing a Markov game model, a tracker experience pool, and an escape agent experience pool; wherein the Markov game model comprises state quantities, action quantities, state transition functions, and reward functions in a cooperative pursuit and escape game scene; the state quantities are constructed by local observations between a plurality of to-be-trained tracker agents and a to-be-trained escape agent; Construct a group of to-be-trained tracker agents and a to-be-trained evader agent, and construct a tracker learner and an evader learner; wherein the group of to-be-trained tracker agents includes a plurality of to-be-trained tracker agents; the tracker learner includes value neural networks and policy neural networks of a plurality of pursuers; the evader learner includes value neural networks and policy neural networks of an evader; wherein the network parameters of the policy neural networks of the to-be-trained tracker agents are copied from the network parameters of the policy neural networks of the tracker learner, and one group of to-be-trained tracker agents corresponds to one to-be-trained evader; Use the Markov game model as a training environment, obtain tracking experience data of the group of to-be-trained tracker agents after interacting with the environment in the training environment, update the tracker experience pool using the tracking experience data after interacting with the environment, and obtain evading experience data of the to-be-trained evader after interacting with the environment in the training environment, update the evader experience pool using the evading experience data after interacting with the environment; According to the plurality of tracking experience data collected from the updated tracker experience pool and the local observation, calculate the value distribution and gradient corresponding to the group of to-be-trained tracker agents, and according to the plurality of evading experience data collected from the updated evader experience pool, calculate the value distribution and gradient corresponding to the to-be-trained evader; Based on the value distribution and gradient corresponding to the group of to-be-trained tracker agents and the value distribution and gradient corresponding to the to-be-trained evader, respectively update the parameters of the tracker learner and the evader learner until the value neural networks and policy neural networks in the updated tracker learner and evader learner meet the preset convergence requirement, then use the network parameters of the value neural networks and policy neural networks in the updated tracker learner that meet the preset convergence requirement as the network parameters of the group of to-be-trained tracker agents, and use the network parameters of the value neural networks and policy neural networks in the updated evader learner that meet the preset convergence requirement as the network parameters of the to-be-trained evader, to obtain a group of trained tracker agents and an evader agent.

9. A control device of a multi-spacecraft cooperative game, characterized by, Comprise: a memory for storing a program; a processor for executing the program stored in the memory to implement the control method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The medium has a program stored thereon, and the program can be executed by a processor to implement the control method of any one of claims 1-7.

Images