Clustered spacecraft cooperative game control method based on CTDE architecture
By using the CTDE architecture and MAPPO algorithm, the problems of environmental non-stationarity and reward sparsity in multi-agent spacecraft cooperative tracking and adversarial game are solved, forming an encirclement configuration that satisfies engineering constraints, adapts to high-dynamic adversarial situations, and achieves stable cooperative control of spacecraft.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIVERSITY SHENZHEN
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies for multi-agent spacecraft cooperative tracking and adversarial game tasks suffer from problems such as non-stationarity of the multi-agent environment, sparse rewards, lack of encirclement configurations, difficulty in incorporating hard engineering constraints, and insufficient high-maneuverability evasion capabilities, which lead to difficulties in strategy convergence and training failure.
A collaborative game control method for swarm spacecraft based on the CTDE architecture is adopted. By constructing a global state space and a local observation space, a composite multidimensional reward function and a phased course learning mechanism are designed. Combined with the MAPPO algorithm, stable convergence, dense reward signals and engineering constraints are integrated to form a swarm encirclement configuration that can adapt to high dynamic adversarial situations.
It achieves stable convergence in a multi-agent environment, improves learning efficiency, forms an encirclement configuration, satisfies engineering constraints, adapts to highly dynamic adversarial situations, overcomes the bottleneck of limited communication resources, and improves the robustness of the strategy.
Smart Images
Figure CN122284432A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of aerospace control engineering and multi-agent artificial intelligence technology, specifically to a collaborative game control method for clustered spacecraft based on the CTDE architecture. Background Technology
[0002] With the rapid development of micro- and nano-satellite technology and space-distributed architecture, traditional large, single spacecraft are gradually evolving into "spacecraft clusters" composed of multiple low-cost, highly redundant small spacecraft. In future space situational awareness, on-orbit servicing, space debris removal, and space offensive and defensive missions, the collaborative execution of high-dynamic tracking and capture missions by clustered spacecraft has become a core technological requirement. Especially when facing "non-cooperative targets" with autonomous evasion maneuvers and unknown strategies, this mission essentially constitutes a typical high-dimensional, nonlinear, many-to-one dynamic pursuit-escape game problem.
[0003] In this scenario, spacecraft clusters need to share local situational information through inter-satellite links. Under extremely stringent engineering constraints (such as thrust physical limits, fuel consumption optimization, and collision avoidance safety distances within the cluster), they can autonomously plan their flight trajectories and spontaneously form a spatial encirclement and synchronous suppression posture against enemy targets.
[0004] In recent years, the mainstream methods for solving this problem mainly include traditional optimal control (such as linear quadratic regulators (LQR) and model predictive control (MPC)) and differential game theory. Although these methods have good theoretical completeness, they have revealed significant bottlenecks in high-dimensional dynamic spaces involving multiple agents. To overcome the limitations of traditional analytical methods, deep reinforcement learning, with its powerful nonlinear fitting and data-driven sequential decision-making capabilities in unknown environments, has begun to be introduced into the field of spacecraft orbit control.
[0005] Despite the enormous potential of deep reinforcement learning in the field of game theory, existing technologies still face a series of significant theoretical and engineering bottlenecks when directly deploying it in multi-spacecraft cooperative tracking and adversarial game tasks, as follows: 1. The non-stationarity problem in multi-agent environments is extremely prominent. In the framework of independent reinforcement learning algorithms (such as IQL and IPPO), each spacecraft updates its policy synchronously during training, causing the environment transition probability perceived by each individual to drift continuously with the changes in the behavior of teammates, making it difficult to stabilize the state distribution. This leads to the policy being difficult to converge, and even the phenomenon of training divergence or getting trapped in local optima.
[0006] 2. In large-scale space environments, reward signals are extremely sparse. Due to the vast scale of space and the stringent conditions required for target acquisition, such as extremely small relative distances and relative velocities, successful events are almost impossible to trigger during the random exploration phase. Traditional methods often employ single-distance penalties or terminal reward mechanisms, which cannot provide continuous and effective gradient feedback, resulting in low learning efficiency.
[0007] 3. Existing multi-agent tracking methods often lack mathematical characterization of "cluster encirclement" configurations, usually degenerating into simple point-to-point greedy tracking, which can easily lead to spacecraft clustering, trajectory overlap, or even internal collisions, making it difficult to reflect the advantages of the coordinated encirclement formation.
[0008] 4. At the engineering level, existing algorithms typically assume continuous and unlimited thrust. However, in actual on-orbit missions, thrust has dead zones and saturation limits, fuel resources are extremely limited, and collisions must be strictly avoided within the swarm. Model-free reinforcement learning methods struggle to smoothly incorporate these inviolable hard constraints during policy optimization.
[0009] 5. When facing adversarial targets with high maneuverability and evasion capabilities, swarm strategies that have not been trained in a tiered manner often fail repeatedly in the early stages of exploration, leading to degradation of training signals and severely affecting the model's convergence and generalization capabilities. Summary of the Invention
[0010] To overcome the shortcomings of existing technologies, the present invention aims to provide a collaborative game control method for clustered spacecraft based on the CTDE architecture. This method can achieve stable convergence in a multi-agent environment, provide dense reward signals to alleviate the reward sparsity problem, form a cluster encirclement configuration, smoothly integrate hard engineering constraints, and improve the robustness of the strategy under high-dynamic adversarial conditions through phased course learning.
[0011] To achieve the objective of this invention, the following solution is adopted: A collaborative game control method for clustered spacecraft based on CTDE architecture includes the following steps: Step S1: Establish an LVLH local orbit coordinate system based on the target spacecraft or virtual reference star, construct a cluster relative nonlinear dynamic model containing multiple tracking spacecraft, and obtain the local observation space definition of each tracking spacecraft and the global state space definition containing the global state information of all spacecraft. Step S2: Based on the global state space, construct a collaborative configuration index evaluation system for quantitatively evaluating the encirclement situation formed by the cluster of spacecraft around the target spacecraft, and obtain configuration evaluation indexes for guiding collaborative encirclement configurations. Step S3: Model the many-to-one chase game as a partially observable Markov game. Based on the global state space, the local observation space, and the configuration evaluation index, define the joint action space and reward function structure to obtain the environmental framework of the Markov game. Step S4: Within the framework of the Markov game, construct a composite multidimensional reward function as a specific implementation of the reward function structure to obtain a dense reward signal for policy optimization; Step S5: Based on the CTDE architecture, the MAPPO algorithm is adopted. The Markov game environment framework and the dense reward signal are used as the training environment. In the centralized training phase, the global value network is updated using the global state space. In the distributed execution phase, each tracking spacecraft uses its local observation space to output the thrust command belonging to the joint action space through the policy network, so as to obtain the training converged cooperative game control strategy. Step S6: The cooperative game control strategy is trained using a phased course learning mechanism to obtain a robust cooperative control strategy that can adapt to the high-maneuverability avoidance behavior of non-cooperative targets.
[0012] Furthermore, in step S2, the collaborative configuration index evaluation system includes a cluster distribution uniformity index and a cluster proximity index. The cluster distribution uniformity index is constructed by calculating the standard deviation of the distance between the cluster spacecraft and the target spacecraft and the distance between the cluster virtual centroid and the target spacecraft. It is used to suppress excessive dispersion or aggregation of the cluster spacecraft and guide the cluster centroid to press towards the target center. The cluster proximity index is constructed by calculating the sum of the absolute distances of all cluster spacecraft to the target and the distance of the closest tracking spacecraft in the cluster to the target. It is used to balance the overall proximity of the cluster with the local rapid proximity.
[0013] Furthermore, in step S4, the composite multidimensional reward function includes a terminal task reward and penalty item, a cluster security collision avoidance penalty item, a collaborative process guidance reward item based on the configuration evaluation index, a fuel optimal constraint penalty item, and a time efficiency penalty item.
[0014] Furthermore, the cooperative process-guided reward term transforms the cluster distribution uniformity index and the cluster proximity index into a dense reward signal, which is used to continuously optimize the strategy when no capture event occurs; the fuel optimal constraint penalty term introduces a control cost penalty to encourage orbital maneuvers with minimum fuel consumption.
[0015] Furthermore, the terminal task reward and punishment items include: A successful capture reward is given when the relative distance between any tracking spacecraft and the target spacecraft is less than the capture determination radius and the relative speed is less than the capture speed threshold. A failure penalty is imposed when the relative distance between any tracking spacecraft and the target spacecraft exceeds the escape radius or reaches the maximum time step. The cluster safety collision avoidance penalty is triggered when the distance between any two tracked spacecraft is less than the safety collision avoidance threshold, terminating the current round and applying a hard penalty.
[0016] Further, in step S5, the CTDE architecture specifically includes: During the centralized training phase, a shared global value network is constructed, which takes the global state space as input and outputs the estimated value of the state-action value function to evaluate the expected benefits of joint actions. In the distributed execution phase, each tracking spacecraft deploys an independent policy network. Taking the local observation space as input, the policy network directly maps and outputs three-dimensional thrust commands, achieving decentralized collaborative decision-making.
[0017] Furthermore, in step S5, the MAPPO algorithm calculates the advantage function through generalized advantage estimation, updates the policy network by replacing the target loss function with pruning, and updates the global value network by minimizing the mean square error to limit the policy update step size.
[0018] Furthermore, in step S6, the phased course learning mechanism includes: The first phase is the basic close encirclement phase, which limits the initial relative distance between the cluster of spacecraft and the target spacecraft, and sets the target spacecraft to be non-maneuverable; The second phase is the mid-range maneuver adaptation phase, which expands the initial relative distance and gives the target spacecraft a heuristic escape maneuver. The third stage is a high-order zero-sum game stage across the entire domain, which extends the initial relative distance to the entire mission airspace and switches the target spacecraft to an intelligent escape model trained with deep reinforcement learning.
[0019] Furthermore, in step S1, the cluster relative nonlinear dynamics model uses the orbital angular velocity and angular acceleration of the LVLH local orbital coordinate system corresponding to the reference as parameters to describe the three-dimensional relative motion of the tracking spacecraft in the LVLH local orbital coordinate system, and includes thruster physical limiting and saturation constraints.
[0020] Furthermore, the reward function of the target spacecraft is set to be in a zero-sum game form with the composite multidimensional reward function of the tracking spacecraft cluster, so that the target spacecraft spontaneously evolves an intelligent avoidance strategy during the training process.
[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention solves the problem of non-stationarity in multi-agent environments, achieving stable convergence. Employing a CTDE architecture, this invention utilizes the global state space to update the global value network during the centralized training phase, eliminating the environment transition probability drift problem caused by policy synchronization updates during independent learning by each tracking spacecraft. This fundamentally overcomes the shortcomings of traditional independent reinforcement learning algorithms, such as training divergence or getting trapped in local optima. In the distributed execution phase, each tracking spacecraft outputs thrust commands through the policy network using only its local observation space, achieving decentralized collaborative decision-making.
[0022] 2. This invention alleviates the reward sparsity problem in large-scale spaces and improves learning efficiency. This invention constructs a composite multidimensional reward function, fusing multiple reward and punishment signals into a dense reward signal. In particular, based on configuration evaluation metrics, it transforms the encirclement situation into continuous and effective gradient feedback, solving the problem of low learning efficiency caused by stringent capture conditions and the difficulty in triggering successful events through random exploration in traditional methods.
[0023] 3. This invention forms a cluster encirclement configuration, avoiding point-to-point greedy tracking. This invention constructs a collaborative configuration index evaluation system to quantitatively evaluate the encirclement posture formed by cluster spacecraft around the target spacecraft, mathematizing the tactical intent of the cluster encirclement. This enables the strategies obtained from subsequent training to spontaneously form a spatial encirclement posture, effectively avoiding the problems of spacecraft aggregation and trajectory overlap in traditional tracking methods.
[0024] 4. This invention smoothly incorporates hard engineering constraints, ensuring on-orbit safety and energy efficiency. It introduces physical thrust limit constraints in dynamic modeling and fuel consumption penalties in the composite multidimensional reward function, integrating engineering constraints into the strategy optimization process. This ensures that the trained control strategy meets thrust limitations and fuel optimization requirements, guaranteeing the spacecraft's on-orbit safety.
[0025] 5. This invention adapts to highly dynamic adversarial targets and enhances strategy robustness. It employs a phased learning mechanism to progressively train cooperative game control strategies, enabling smooth transitions from basic conditions to highly dynamic adversarial scenarios. This overcomes the problem of training signal degradation when directly facing highly maneuverable targets, ultimately resulting in a robust cooperative control strategy capable of adapting to the highly maneuverable evasion behavior of non-cooperative targets.
[0026] 6. This invention overcomes the engineering bottleneck of limited communication resources and achieves decentralized collaboration. Through the distributed execution phase of the CTDE architecture, this invention enables each tracking spacecraft to output thrust commands through the policy network relying only on local observation space, eliminating the dependence on global, all-time, high-bandwidth communication via inter-satellite links, which is in line with the engineering reality of communication delays and easy interruptions during spacecraft operation in orbit. Attached Figure Description
[0027] Figure 1This is a flowchart of a cluster spacecraft cooperative game control method based on CTDE architecture in an embodiment of the present invention; Figure 2 is a schematic diagram of the relative motion coordinate system of the cluster spacecraft dynamic game in an embodiment of the present invention; Figure 3 is a flowchart of the framework and control process of the MAPPO-CTDE algorithm in an embodiment of the present invention. Detailed Implementation
[0028] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.
[0029] like Figure 1-3 As shown, this embodiment of the invention provides a collaborative game control method for clustered spacecraft based on the CTDE architecture, including the following steps: Step S1: Establish an LVLH local orbital coordinate system based on the target spacecraft or virtual reference star, construct a cluster relative nonlinear dynamic model containing multiple tracking spacecraft, and obtain the local observation space definition of each tracking spacecraft and the global state space definition containing the global state information of all spacecraft.
[0030] In this embodiment, step S1 is the nonlinear relative dynamics modeling step under complex constraints, and the specific content is as follows: This invention establishes an LVLH (Local-Vertical-Local-Horizontal) local orbital coordinate system based on the nominal orbit of the target spacecraft or virtual reference satellite. The origin of the coordinate system is located at the reference centroid, and the three axes are respectively along the orbital radial direction (…). ), trace ( ) and the orbital plane normal ( ) Construction. Definition includes A cluster of tracking spacecraft, with the orbital angular velocity of a reference spacecraft being ω. angular acceleration is The first in the cluster The three-dimensional relative nonlinear dynamic equations of a spacecraft in the LVLH coordinate system are described as follows: in, The gravitational constant of Earth; The geocentric distance is used as a reference. For the first The Earth's center distance of each spacecraft; For the first The control acceleration commands for a cluster of spacecraft. In practical engineering, the thrust of the spacecraft must meet strict physical limits and saturation constraints, namely: For highly maneuverable escape target spacecraft It also follows the above dynamic model, and its thrust limit is usually set to... The objective of this invention is to address situations where the target may possess a thrust advantage ( Even in the worst-case scenario, collaborative capture is still achieved.
[0031] Step S2: Based on the global state space, construct a collaborative configuration index evaluation system for quantitatively assessing the encirclement posture formed by the cluster of spacecraft around the target spacecraft, and obtain configuration evaluation indexes for guiding collaborative encirclement configurations.
[0032] In this embodiment, step S2 is the step of constructing the cluster collaborative configuration and situation indicator evaluation system: To break away from the clustering phenomenon caused by traditional point-to-point tracking, this invention proposes a spatial configuration evaluation index system as the basis for state shaping and reward shaping in reinforcement learning.
[0033] 1. Cluster distribution uniformity index (UniformityEvaluation): This metric is used to quantitatively assess whether the encirclement posture formed by the tracking swarm around the target is balanced. First, the current time is calculated. Cluster virtual centroid coordinates .
[0034] Definition of the first The distance from each spacecraft to the virtual center of mass is The standard deviation of the distance between the swarm spacecraft and the target is: Define the distance between the cluster's virtual centroid and the target spacecraft as Based on this, a distribution uniformity evaluation function is constructed: Among them, weight parameters The first factor inhibits excessive dispersion or aggregation of the cluster, promoting a relatively uniform topology among its members; the second factor guides the cluster's overall centroid to continuously press towards the target center. A higher value of this factor indicates a better encirclement strategy.
[0035] 2. Cluster proximity metric (Approach Evaluation): definition The sum of the absolute distances from all cluster spacecraft to the target at any given time is Define the distance to the target spacecraft in the cluster as... .
[0036] The proximity evaluation function is defined as follows: in, . Used to regulate the overall tendency of the cluster to shrink towards the target. This is used to enhance the rapid approach behavior of the nearest sub-star to the target. The two are weighted to achieve a balance between the overall proximity of the cluster and the rapid local proximity.
[0037] Step S3: Model the many-to-one pursuit game as a partially observable Markov game. Based on the global state space, the local observation space, and the configuration evaluation index, define the joint action space and reward function structure to obtain the environmental framework of the Markov game.
[0038] In this embodiment, step S3 is the definition step of the reinforcement learning framework based on Markov games, and the specific content is as follows: The many-to-one pursuit game is modeled as a partially observable Markov game (POMG), and tuples are defined. : 1. Global state space It contains global information such as the three-dimensional position and velocity of all spacecraft, as well as the system's relative configuration parameters, and is only visible to the Critic network during the ground-based intensive training phase.
[0039] 2. Local observation space Considering the latency and bandwidth limitations of inter-satellite communication, the first Local observation of a spacecraft It only includes its own relative state, the relative state estimate of the target, and the state of teammates within the adjacent communication topology.
[0040] 3. Combined Action Space : This refers to the continuous motion space, i.e., the three-axis thrust acceleration vector. Range normalized to It is then mapped to the actual thrust range through environmental mapping.
[0041] 4. Transition probability : This represents the probability distribution of the system transitioning to the next state given the current state and joint actions, and is determined by the discretized form of the nonlinear orbital dynamics equations.
[0042] 5. Reward Function : No. Reward function for each spacecraft Defined as a composite function that includes terminal rewards and penalties, configuration guidance, safety collision avoidance, and fuel constraints, used to guide strategy optimization.
[0043] 6. Discount Factor Used to balance the weights of current rewards and future long-term benefits, ensuring the convergence of infinite time-bound tasks.
[0044] Step S4: Within the framework of the Markov game, construct a composite multidimensional reward function as a specific implementation of the reward function structure to obtain a dense reward signal for policy optimization.
[0045] In this embodiment, step S4 is the construction step of the composite multidimensional reward function, the specific content of which is as follows: To address the problem of convergence failure caused by extremely sparse rewards in vast spaces, this invention designs a composite reward function that combines dense and sparse elements and includes hard constraint penalties. .
[0046] 1. Terminal task rewards and penalties ( The capture condition is defined as the existence of at least one tracker. Its relative distance from the target And relative speed If the capture is successful, If the target escapes (relative distance exceeds the limit). (or reaching the maximum time step) If it is still not captured, then .
[0047] 2. Cluster security collision avoidance penalty ( The distance between any two tracking spacecraft is less than the safe collision avoidance threshold. At that time, a one-time hard penalty is triggered and the round is terminated: 3. Collaborative process guidance rewards ( Transforming evaluation metrics into dense reward signals allows the guiding strategy to continuously optimize even when no capture events occur. 4. Fuel Optimization Constraint Penalty ( Introducing control costs into rewards encourages "small maneuvers, big benefits," extending the on-orbit lifespan of spacecraft. 5. Time efficiency penalty ( A slight penalty is imposed for each time step of survival (e.g.) This encourages the cluster to complete the capture as quickly as possible.
[0048] The target spacecraft's reward function is set as a zero-sum game, the exact opposite of the tracking cluster. Through this zero-sum competitive reward design, the target spacecraft will spontaneously evolve a highly intelligent avoidance strategy.
[0049] Step S5: Based on the CTDE architecture, the MAPPO algorithm is adopted. The Markov game environment framework and the dense reward signal are used as the training environment. In the centralized training phase, the global value network is updated using the global state space. In the distributed execution phase, each tracking spacecraft uses its local observation space to output thrust commands belonging to the joint action space through the policy network, thereby obtaining a cooperative game control strategy with training convergence.
[0050] In this embodiment, step S5 is the CTDE control architecture and policy update step based on MAPPO, and the specific content is as follows: This invention employs the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm and combines it with the CTDE paradigm for architecture design: 1. Centralized Training: In ground-based supercomputing centers or high-performance simulation nodes, a shared global value network (Critic) is built due to the absence of communication limitations. The network receives the global real-time state. It accurately assesses the expected benefits of current joint actions, fundamentally eliminating the non-stationarity of multi-agent environments.
[0051] 2. Decentralized Execution: Each tracking spacecraft deploys an independent policy network (Actor). During in-orbit operation, the spacecraft only needs to transmit the local observations acquired by its own sensors and short-range inter-satellite links. By inputting into the Actor network, three-dimensional thrust commands can be output in real time, completely eliminating the dependence on high-bandwidth communication across the entire network at all times.
[0052] The MAPPO policy update formula is as follows: Calculate individual using generalized dominance estimation (GAE) Advantage function valuation : The pruning substitution objective loss function of the Actor network is: Importance sampling ratio , This is to truncate hyperparameters. This truncation mechanism strictly controls the step size of policy updates to prevent catastrophic policy degradation in the complex orbital dynamics space of the spacecraft.
[0053] The Critic network updates by minimizing the mean square error: Step S6: The cooperative game control strategy is trained using a phased course learning mechanism to obtain a robust cooperative control strategy that can adapt to the high-maneuverability avoidance behavior of non-cooperative targets.
[0054] In this embodiment, step S6 is a step of the phased course learning and training mechanism, the specific content of which is as follows: To address the "curse of dimensionality" and "exploration difficulties" that are difficult to directly converge in highly maneuverable adversarial environments, this invention pioneers a three-stage orbital game curriculum learning mechanism: 1. First Phase (Basic Close Encirclement): The initial relative distance between the two sides is limited to a very small encirclement (e.g., 100m), while the target spacecraft is forced not to perform any maneuvers. This phase allows the swarm to quickly learn how to converge toward the target center and maintain its configuration under collision avoidance constraints.
[0055] 2. Second Phase (Medium-Range Maneuver Adaptation): The initial distance distribution is relaxed to a medium range (e.g., 500m), and the target spacecraft begins to employ random or simple heuristic escape maneuvers. In this phase, the tracker learns to identify the enemy's maneuvering intentions and develops the ability to predict interception points, building upon existing basic strategies.
[0056] 3. Third Stage (High-Order Zero-Sum Game Across the Entire Domain): The initial distance is expanded to the entire mission airspace (e.g., 4000m), and the target spacecraft is switched to an intelligent escape actor model also trained by DRL. At this point, the system enters a pure multi-agent adversarial game, and the model trains a globally optimal robust cooperative strategy that adapts to any non-cooperative extreme maneuvering in endless adversarial battles.
[0057] The collaborative game control method for spacecraft swarms based on the CTDE architecture in this invention constructs a comprehensive index system capable of quantitatively evaluating the equilibrium of the swarm's encirclement situation, guiding spacecraft to form a multi-faceted encirclement topology. During the training phase, the CTDE architecture combined with the MAPPO algorithm utilizes global situational information to stabilize the Critic network, achieving efficient and stable centralized training. In the on-orbit execution phase, decentralized collaborative decision-making relies solely on local observations, effectively overcoming inter-satellite communication bandwidth limitations. Simultaneously, a composite reward function integrating terminal rewards and penalties, configuration guidance, fuel consumption penalties, and collision avoidance hard constraints is designed to alleviate reward sparsity and achieve a unification of engineering constraints and strategy optimization. Furthermore, the collaborative game control method for spacecraft swarms based on the CTDE architecture in this invention introduces a phased learning mechanism for space orbital games. Through a step-by-step progression of task difficulty, it achieves a smooth migration of strategies from low-dynamic tracking to high-dynamic adversarial approaches, significantly improving the algorithm's convergence speed and robustness against unknown maneuvering strategies.
[0058] The following describes the collaborative game control method for swarm spacecraft based on the CTDE architecture, according to an embodiment of the present invention, using a high-performance computer and a spacecraft orbital dynamics simulation platform. The specific details are as follows: 1. System modeling and simulation environment initialization: (1) Construct a dynamic simulation model based on the Earth's gravitational field and the LVLH relative coordinate system. Set the system's running time step. Maximum duration of a single round .
[0059] (2) Configure physical constraint parameters: Set the maximum maneuvering acceleration of the tracking spacecraft and the target spacecraft. and (The enemy has a mobility advantage). Set the capture detection radius. Safe collision avoidance radius escape radius if mission fails .
[0060] (3) Initialize the neural network parameters. Establish a multilayer perceptron (MLP) with 3 hidden layers and 256 neurons per layer as the Critic network, using ReLU as the activation function. Establish an Actor network with the same structure for each spacecraft, using the Tanh activation function for its output layer.
[0061] 2. First stage of course learning (basic encirclement training): (1) Initialization round: In three-dimensional space, the target's initial position is at the origin, the velocity is zero and no thrust maneuver is performed.
[0062] (2) Randomly distribute the four tracking cluster spacecraft in the spherical shell space at a distance of [100m, 200m] from the target.
[0063] (3) The spacecraft interacts with the environment by sampling actions through the Actor network and collects a trajectory that includes status, observation, action and reward.
[0064] (4) Calculate the composite reward including configuration evaluation and fuel consumption, and calculate the GAE advantage function.
[0065] (5) Update the Actor network using the PPO pruning formula and update the global Critic network using the mean square error.
[0066] (6) Repeat this process. When the average capture rate of the last 100 rounds is greater than 90%, the first stage of training is deemed mature. The current network weights are frozen and saved, and the next stage is entered.
[0067] 3. Second stage of course learning (mid-range maneuver adaptation): (1) Load the network weights saved in the first stage as the starting point.
[0068] (2) Expand the initial generation airspace: The initial state of the target spacecraft is random, and the tracking spacecraft is randomly generated within a range of [200m, 1000m] from the target.
[0069] (3) Activate the basic maneuverability of the target spacecraft: give the target a set of heuristic orbital avoidance logic, and always output maximum thrust in the opposite direction of the virtual centroid connection of the tracking cluster.
[0070] (4) During the pursuit, the cluster gradually learns to predict the escape direction through the time penalty and distance penalty in the compound reward and spontaneously forms a division of labor.
[0071] (5) When the capture rate rises again and stabilizes above 85%, the final stage begins.
[0072] 4. The third stage of course learning (full-domain adversarial game): (1) The initial range is expanded to the complete game adversarial space.
[0073] (2) Activate bilateral learning mode: not only train the cluster's MAPPO, but also give the target spacecraft an independent adversarial strategy network, and the two sides compete under the zero-sum game mechanism.
[0074] (3) At this stage, due to the CTDE architecture and the strategy foundation laid in the previous courses, the cluster Actor network can keenly capture highly nonlinear situational changes and finally converge to obtain a robust cooperative tracking strategy with Nash equilibrium characteristics.
[0075] The collaborative game control method for clustered spacecraft based on CTDE architecture in this invention has the following advantages: 1. This invention avoids the "curse of dimensionality" by eliminating the need for an exact analytical solution model. Traditional Hamilton-Jacobi-Isaacs equations based on differential game theory suffer from the curse of dimensionality when the number of participating spacecraft is large. This invention utilizes a data-driven MAPPO neural network to fit the value function, completely eliminating the dependence on explicit analytical solutions and enabling its expansion to game tasks involving dozens or even larger spacecraft swarms.
[0076] 2. This invention overcomes the engineering bottleneck of limited communication resources. Existing multi-agent algorithms require real-time joint decision-making based on global state. The CTDE architecture of this invention only uses global information during the centralized training phase on the ground; during the actual operation phase in space orbit, each spacecraft can achieve spontaneous coordination through the Actor network based solely on local observations, effectively meeting the engineering realities of high latency and easy interruption of inter-satellite links.
[0077] 3. This invention solves the convergence problem of "sparse reward" in large-scale space missions. Traditional tracking methods typically only award rewards upon successful capture, resulting in extremely sparse reward signals in the vast space environment, making strategy optimization difficult. This invention constructs a configuration index based on the system distance standard deviation and centroid proximity, transforming sparse terminal rewards into continuous and effective dense guidance signals. This not only significantly accelerates the algorithm's convergence speed but also naturally fosters multi-directional encirclement and siege tactics, avoiding the phenomenon of spacecraft clustering and blindly pursuing each other.
[0078] 4. This invention effectively addresses highly dynamic adversarial conditions and avoids policy degradation. Utilizing its unique three-stage progressive learning framework, it solves the problem of policy non-convergence or getting trapped in local optima when conventional deep reinforcement learning algorithms directly face highly maneuverable targets. Smooth policy transfer ensures that the model can progressively master advanced adversarial techniques, significantly improving policy robustness.
[0079] 5. This invention balances engineering safety and energy efficiency. By employing a composite multidimensional reward function and a policy pruning mechanism, this invention fully considers the saturation dead zone of thrusters and the limited fuel capacity of micro / nano spacecraft, achieving low-energy orbital maneuvers. Simultaneously, a strict collision avoidance hard penalty function forces the neural network to strictly adhere to the safety envelope during multiple iterations, ensuring the safety of space assets.
[0080] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A collaborative game control method for clustered spacecraft based on CTDE architecture, characterized in that, Includes the following steps: Step S1: Establish an LVLH local orbit coordinate system based on the target spacecraft or virtual reference star, construct a cluster relative nonlinear dynamic model containing multiple tracking spacecraft, and obtain the local observation space definition of each tracking spacecraft and the global state space definition containing the global state information of all spacecraft. Step S2: Based on the global state space, construct a collaborative configuration index evaluation system for quantitatively evaluating the encirclement situation formed by the cluster of spacecraft around the target spacecraft, and obtain configuration evaluation indexes for guiding collaborative encirclement configurations. Step S3: Model the many-to-one chase game as a partially observable Markov game. Based on the global state space, the local observation space, and the configuration evaluation index, define the joint action space and reward function structure to obtain the environmental framework of the Markov game. Step S4: Within the framework of the Markov game, construct a composite multidimensional reward function as a specific implementation of the reward function structure to obtain a dense reward signal for policy optimization; Step S5: Based on the CTDE architecture, the MAPPO algorithm is adopted. The Markov game environment framework and the dense reward signal are used as the training environment. In the centralized training phase, the global value network is updated using the global state space. In the distributed execution phase, each tracking spacecraft uses its local observation space to output the thrust command belonging to the joint action space through the policy network, so as to obtain the training converged cooperative game control strategy. Step S6: The cooperative game control strategy is trained using a phased course learning mechanism to obtain a robust cooperative control strategy that can adapt to the high-maneuverability avoidance behavior of non-cooperative targets.
2. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, In step S2, the collaborative configuration index evaluation system includes a cluster distribution uniformity index and a cluster proximity index. The cluster distribution uniformity index is constructed by calculating the standard deviation of the distance between the cluster spacecraft and the target spacecraft and the distance between the cluster virtual centroid and the target spacecraft. It is used to suppress excessive dispersion or aggregation of the cluster spacecraft and guide the cluster centroid to press towards the target center. The cluster proximity index is constructed by calculating the sum of the absolute distances of all cluster spacecraft to the target and the distance of the closest tracking spacecraft in the cluster to the target. It is used to balance the overall proximity of the cluster with the local rapid proximity.
3. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 2, characterized in that, In step S4, the composite multidimensional reward function includes a terminal task reward and penalty item, a cluster security collision avoidance penalty item, a collaborative process guidance reward item based on the configuration evaluation index, a fuel optimal constraint penalty item, and a time efficiency penalty item.
4. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 3, characterized in that, The cooperative process-guided reward term transforms the cluster distribution uniformity index and the cluster proximity index into a dense reward signal, which is used to continuously optimize the strategy when no capture event occurs; the fuel optimal constraint penalty term introduces a control cost penalty to encourage orbital maneuvers with minimum fuel consumption.
5. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 3, characterized in that, The terminal task reward and penalty items include: A successful capture reward is given when the relative distance between any tracking spacecraft and the target spacecraft is less than the capture determination radius and the relative speed is less than the capture speed threshold. A failure penalty is imposed when the relative distance between any tracking spacecraft and the target spacecraft exceeds the escape radius or reaches the maximum time step. The cluster safety collision avoidance penalty is triggered when the distance between any two tracked spacecraft is less than the safety collision avoidance threshold, terminating the current round and applying a hard penalty.
6. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, In step S5, the CTDE architecture specifically includes: During the centralized training phase, a shared global value network is constructed, which takes the global state space as input and outputs the estimated value of the state-action value function to evaluate the expected benefits of joint actions. In the distributed execution phase, each tracking spacecraft deploys an independent policy network. Taking the local observation space as input, the policy network directly maps and outputs three-dimensional thrust commands, achieving decentralized collaborative decision-making.
7. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, In step S5, the MAPPO algorithm calculates the advantage function through generalized advantage estimation, updates the policy network by replacing the target loss function with pruning, and updates the global value network by minimizing the mean square error to limit the policy update step size.
8. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, In step S6, the phased course learning mechanism includes: The first phase is the basic close encirclement phase, which limits the initial relative distance between the cluster of spacecraft and the target spacecraft, and sets the target spacecraft to be non-maneuverable; The second phase is the mid-range maneuver adaptation phase, which expands the initial relative distance and gives the target spacecraft a heuristic escape maneuver. The third stage is a high-order zero-sum game stage across the entire domain, which extends the initial relative distance to the entire mission airspace and switches the target spacecraft to an intelligent escape model trained with deep reinforcement learning.
9. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, In step S1, the cluster relative nonlinear dynamics model uses the orbital angular velocity and angular acceleration of the LVLH local orbit coordinate system corresponding to the reference as parameters to describe the three-dimensional relative motion of the tracking spacecraft in the LVLH local orbit coordinate system, and includes thrust physical limiting and saturation constraints.
10. The collaborative game control method for clustered spacecraft based on CTDE architecture according to claim 1, characterized in that, The reward function of the target spacecraft is set to be in a zero-sum game form with the composite multidimensional reward function of the tracking spacecraft cluster, so that the target spacecraft spontaneously evolves an intelligent avoidance strategy during the training process.