A heterogeneous constellation autonomous command and control system and method for deep reinforcement learning

Abstract

The application discloses a kind of heterogeneous star cluster autonomous command and control system and method of deep reinforcement learning, belong to aerospace technology field.System includes strategy network training layer, intelligent game decision layer and star cluster cooperation planning layer.Strategy network training layer pre-trains strategy network in ground environment.Intelligent game decision layer calls the tracking strategy network of training completion, generates tracking action according to the current state of target star and pursuit star.Star cluster cooperation planning layer includes global task allocation module and tactical configuration maintaining module, global task allocation module generates slave star task allocation scheme;Tactical configuration maintaining module controls slave star to keep tactical configuration around master star in pursuit initial stage, constructs virtual master star that approaches target star in terminal segment encircling stage, and controls slave star to move around virtual master star, realizes the orbit encirclement of target star;The application can make satellite realize autonomous planning, efficient cooperation and precision confrontation, to improve its task execution capability and practical value.

Description

Technical Field

[0001] This invention relates to the field of aerospace technology, and more specifically to a deep reinforcement learning-based autonomous command and control system and method for heterogeneous constellations. Background Technology

[0002] Currently, with the rapid development of aerospace technology, single spacecraft with fixed functions can no longer meet the increasingly complex needs of space missions. Therefore, satellite constellations or swarm systems, which consist of multiple satellites working in coordination, have emerged. With advantages such as wide-area coverage, high robustness, and flexible mission modes, satellite constellations or swarm systems have demonstrated significant application potential in fields such as communication, navigation, remote sensing, space situational awareness, and space security.

[0003] Most existing spacecraft game strategy algorithms are limited to one-on-one adversarial scenarios. These models simplify complex inter-swarm game problems into a chase or confrontation between two single entities. While convenient for theoretical analysis and modeling, they fail to meet the practical needs of multi-satellite coordination in complex space missions. In real-world swarm coordination missions, one or both sides typically consist of multiple satellites with varying functions, performance, and payloads. This heterogeneity is a core characteristic and advantage of swarm systems. However, one-on-one game models neglect this crucial characteristic and cannot address task allocation issues such as who executes the pursuit, who is responsible for surveillance, who performs interference, and who performs relay in swarm adversarial scenarios. An efficient swarm system typically needs to allocate tasks based on the mission situation and the individual satellite states, such as fuel, position, and payload capacity, before entering a game state to improve resource utilization efficiency. During the game, swarm members also need to maintain or dynamically adjust specific geometric configurations through coordinated control to ensure effective communication, observation, or maneuvering.

[0004] However, existing technologies typically treat the game process in isolation, severing the intrinsic connection between game decision-making, task allocation, and configuration maintenance. This makes it difficult for the strategy output to achieve system-level optimality and also fails to fully leverage the overall synergistic advantages of the constellation.

[0005] Furthermore, current research on spacecraft pursuit and escape game-theoretic decision-making largely employs differential game theory. This method, combining game theory and optimal control theory, can obtain the Nash equilibrium strategy for spacecraft orbital games with relatively high accuracy. However, this method still has the following limitations: First, in most cases, differential game theory does not have analytical solutions, and this is also true for orbital game problems, typically requiring significant computational resources for numerical solutions. Second, for orbital game problems under incomplete observation, this method usually requires additional methods such as filtering to estimate the missing information before using algorithms with complete information to solve the game strategy, further increasing the computational load. Existing onboard computers have limited computing power and cannot handle such complex online calculations, thus making it difficult to guarantee the real-time performance of the algorithm; for high-speed spacecraft, insufficient real-time performance significantly limits the practical application value of this type of method.

[0006] Therefore, how to achieve real-time autonomous pursuit and escape decision-making and cooperative configuration control of heterogeneous constellations, while taking into account constellation heterogeneity and resource optimization scheduling, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide a deep reinforcement learning-based heterogeneous constellation autonomous command and control system and method that overcomes or at least partially solves the above problems.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, embodiments of the present invention provide a heterogeneous constellation autonomous command and control system based on deep reinforcement learning, comprising a policy network training layer, a constellation collaborative planning layer, and an intelligent game decision-making layer, wherein the heterogeneous constellation comprises a master star and K slave stars, wherein: The policy network training layer is used to pre-train the policy network in a ground environment, and the policy network includes a tracking policy network and an escape policy network. The intelligent game decision-making layer includes a strategy generation module; the strategy generation module is used to call the tracking strategy network during on-orbit execution and generate the tracking actions of the tracking star towards the target star; The constellation collaborative planning layer includes a global task allocation module and a tactical configuration maintenance module. The global task allocation module is used to generate a satellite task allocation scheme based on the satellite capability parameters of the K satellites in the heterogeneous constellation and a global optimization algorithm. The satellite task allocation scheme includes the combination of satellites participating in the collaborative task and the task to be performed by each satellite, where K is an integer greater than 1. The tactical configuration maintenance module is used to control the satellites to maintain their tactical configuration based on the satellite task allocation scheme and the motion state of the pursuing satellite, and to generate obstacle avoidance control variables based on the escape strategy network when there is a collision risk for the satellites. In the initial pursuit phase, the pursuing star is the primary star, which tracks the target star according to the tracking action, and the secondary stars maintain a tactical configuration around the primary star. In the final encirclement phase, the primary star maintains a safe distance from the target star and constructs a virtual primary star as the virtual pursuit center. The pursuing star switches to the virtual primary star, which continues to approach the target star according to the tracking action. The secondary stars detach from the primary star and maintain a tactical configuration around the virtual primary star to achieve orbital encirclement of the target star.

[0010] In one embodiment, the policy network training layer specifically includes: A model pool construction module is used to construct M model pools, each model pool including at least a tracking strategy model pool, an escape strategy model pool, and a rule strategy model pool; the tracking strategy model pool includes multiple tracking strategy models, the escape strategy model pool includes multiple escape strategy models, and the rule strategy model pool includes multiple rule strategy models, where M is an integer greater than or equal to 3; The adversarial model selection module is used to determine the model to be trained, calculate the selection weight based on the historical average reward of the model to be trained and the candidate adversarial model, and select the candidate adversarial model from the model pool other than the model pool where the model to be trained is located based on the selection weight. The historical average reward is the average reward obtained by the model to be trained and the candidate adversarial model in the historical game process. The adversarial module is used to obtain a trained tracking strategy model or a trained escape strategy model by playing a game against the model to be trained and the candidate adversarial models, and to add the trained tracking strategy model or the trained escape strategy model to the corresponding strategy model pool as a trained strategy network.

[0011] In one embodiment, the policy network training layer further includes a reward function construction module, which specifically includes: The status acquisition unit is used to acquire the relative distance between the tracking star and the target star, the change in relative distance, and the acceleration corresponding to the execution of the tracking action; The tracking process reward function construction unit is used to construct a tracking process reward sub-function based on the predicted distance between the tracking satellite and the target satellite at the next moment when the tracking satellite does not perform the tracking action, the actual distance between the tracking satellite and the target satellite at the next moment after the tracking action is performed, the relative distance reduction value before and after the tracking action is performed, and the acceleration.

[0012] in, This is the predicted distance between the tracking star and the target star at the next moment when the tracking action is not performed. This refers to the actual distance between the tracking star and the target star at the next moment after the tracking action is performed. This indicates the relative distance between the tracking satellite and the target satellite before the tracking action is performed. This indicates the relative distance between the tracking satellite and the target satellite after the tracking action is performed again. This represents the relative distance reduction value after the tracking action is performed. For the acceleration, , , These are weight parameters; A terminal reward function construction unit is used to construct a terminal reward sub-function, wherein the terminal reward sub-function is:

[0013] in, To track terminal rewards, The time decay coefficient, The maximum number of training steps, The capture distance threshold; The comprehensive reward construction unit is used to merge the tracking process reward sub-function and the tracking terminal reward sub-function to construct a comprehensive reward function for training the tracking policy network.

[0014] In one embodiment, the policy network training layer further includes: The strategy model training module is used to iteratively train and generate the tracking strategy model and the escape strategy model based on the actor-critic architecture and the TD3 algorithm.

[0015] In one embodiment, the policy model training module is specifically used for: Initialize the network parameters of the actor network, the first critic network, and the second critic network, and initialize the corresponding actor target network, the first critic target network, and the second critic target network; The initialized actor network interacts with the simulation environment to obtain a data tuple, which includes the current state data, the action output under the current state data, the reward obtained, and the next state data after the action is executed. The network parameters of the initialized first critic network and the initialized second critic network are updated using the data tuples and the TD3 algorithm, and the network parameters of the initialized actor network are updated according to a preset update frequency. The actor target network, the first critic target network, and the second critic target network are updated using the updated actor network, the updated first critic network, and the updated second critic network. Based on the updated actor network, the updated first critic network, the updated second critic network, and their respective target networks, iterative training is performed to obtain policy models, which are the tracking policy model and the escape policy model.

[0016] In one embodiment, the global task allocation module is specifically used for: Based on the functional type and satellite capability parameters of each satellite, an effectiveness weight matrix is ​​constructed. The matrix elements in the effectiveness weight matrix are used to characterize the effectiveness of each satellite in performing the corresponding task type. Construct a 0-1 task allocation matrix to characterize the satellite task allocation scheme, wherein the 0-1 task allocation matrix must satisfy the constraints of the single task allocation of each satellite. Multiply the corresponding elements of the performance weight matrix and the 0-1 task allocation matrix and sum them to obtain the fitness evaluation index used to evaluate the rationality of the slave task allocation scheme. Based on the fitness evaluation index, the 0-1 task allocation matrix is ​​iteratively selected, crossovered, and mutated using a genetic algorithm to optimize it and output the slave task allocation scheme corresponding to the highest fitness evaluation index.

[0017] In one embodiment, the tactical configuration holding module is specifically used for: Based on the position of the current configuration reference center and the preset tactical configuration, the desired positions of each of the slave stars are determined; wherein, the current configuration reference center is the master star in the initial stage of pursuit and the virtual master star in the final encirclement stage, and the preset tactical configuration includes an elliptical encirclement configuration formed around the current configuration reference center; Obtain the observed position of each of the slave stars, and determine the position error of each of the slave stars based on the observed position and the desired position; Based on the position error, error rate of change, and PD control algorithm, a position control variable is generated to control the slave satellite to maintain its tactical configuration. The position control variable is:

[0018] in, For position control quantity, For proportional gain, For positional error, = , For the desired position, For the observation position, For differential gain, The rate of change of error; When the slave star is at risk of collision, obstacle avoidance control quantity is generated through the escape strategy network based on the relative state of the slave star and the object to be avoided; The position control quantity and the obstacle avoidance control quantity are superimposed to generate the integrated control command for the slave satellite. The integrated control command is as follows:

[0019] in, The integrated control command, For position control quantity, To control the amount of obstacle avoidance, and These are preset weighting coefficients.

[0020] Secondly, embodiments of the present invention provide a deep reinforcement learning-based heterogeneous constellation autonomous command and control method, applied to a deep reinforcement learning-based heterogeneous constellation autonomous command and control system as described in the first aspect, employing the following technical solution: The policy network training layer is pre-trained in a ground environment. The policy network includes a tracking policy network and an escape policy network. The global task allocation module of the constellation collaborative planning layer generates a satellite task allocation scheme based on the satellite capability parameters of K satellites in the heterogeneous constellation and a global optimization algorithm. The satellite task allocation scheme includes the combination of satellites participating in the collaborative task and the task to be performed by each satellite, where K is an integer greater than 1. In the initial stage of the pursuit, the main star is used as the pursuing star. The strategy generation module of the intelligent game decision layer calls the tracking strategy network and generates the tracking action of the pursuing star towards the target star. The tactical configuration maintenance module of the star cluster collaborative planning layer controls the slave star to maintain the tactical configuration around the main star. When entering the final encirclement phase, the tactical configuration maintenance module of the star cluster cooperative planning layer controls the primary star to maintain a safe distance from the target star, constructs a virtual primary star as a virtual pursuit center, switches the pursuit star to the virtual primary star, controls the virtual primary star to continue to approach the target star according to the tracking action, and controls the slave star to detach from the primary star and maintain a tactical configuration around the virtual primary star. When the target star is at risk of collision, the tactical configuration maintenance module of the constellation cooperative planning layer calls the escape strategy network to generate obstacle avoidance control quantities, and generates integrated control commands based on the obstacle avoidance control quantities and the position control quantities corresponding to configuration maintenance, so as to achieve orbital encirclement of the target star.

[0021] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: 1. This invention addresses the pursuit and escape missions of heterogeneous constellations by proposing a comprehensive collaborative control scheme covering ground training, on-orbit mission allocation, tracking decision-making, and configuration maintenance. Compared to existing methods that only address the pursuit and escape game for a single spacecraft, this invention can allocate tasks based on the functional types and capability parameters of different slave satellites, and combine the primary satellite's tracking decision-making with the slave satellites' configuration maintenance. This allows different satellites in a heterogeneous constellation to undertake corresponding tasks according to their respective capabilities and collaboratively execute the pursuit and escape encirclement process, thereby improving the resource utilization, collaborative control capabilities, and overall mission execution efficiency of the heterogeneous constellation.

[0022] 2. This invention designs a composite reward function that integrates relative distance change, task time, and fuel consumption, and uses this function to train the policy network. This enables the policy network to balance tracking efficiency, time window, and maneuver cost when generating tracking or escape actions, thereby improving the practical applicability of the strategy.

[0023] 3. This invention employs a group reinforcement learning training paradigm to train the policy network. By conducting adversarial training among the tracking policy model pool, the escape policy model pool, and the rule policy model pool, the policy network can engage in game training with opponents of different types and strengths, thereby improving the robustness and generalization ability of the policy network. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of a heterogeneous constellation autonomous command and control system based on deep reinforcement learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of model training based on reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the training process of a policy network based on a swarm reinforcement learning algorithm provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the execution process of the rule-based escape strategy provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the execution process of the rule-based pursuit strategy provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the actor network structure provided in an embodiment of the present invention; Figure 7This is a schematic diagram of the structure of the critic network provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the constellation task allocation process provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the overall process of the genetic algorithm provided in this embodiment of the invention; Figure 10 This is a schematic diagram of a pursuit and escape game scenario provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Currently, with the rapid development of aerospace technology, single spacecraft with fixed functions can no longer meet the increasingly complex needs of space missions. Therefore, satellite constellations or swarm systems, which consist of multiple satellites working in coordination, have emerged. With advantages such as wide-area coverage, high robustness, and flexible mission modes, satellite constellations or swarm systems have demonstrated significant application potential in fields such as communication, navigation, remote sensing, space situational awareness, and space security.

[0028] Most existing spacecraft game strategy algorithms are often limited to one-on-one adversarial scenarios. These models simplify complex inter-stellar game problems into a chase or confrontation between two single entities. While this facilitates theoretical analysis and modeling, it fails to meet the practical needs of multi-satellite collaboration in complex space missions. In real-world stellar collaboration missions, one or both sides typically consist of multiple satellites with varying functions, performance, and payloads; heterogeneity is the core characteristic and advantage of stellar systems. However, one-on-one game models neglect this crucial characteristic and cannot address issues such as which satellites participate in the mission, what tasks each satellite undertakes, and how satellites maintain tactical configurations with the primary or virtual primary satellite in group collaboration missions.

[0029] This invention addresses the problem of oversimplified pursuit-escape game models in existing technologies, which are ill-suited for heterogeneous satellite constellation collaborative missions. It designs a complete command and control process covering ground training, on-orbit mission allocation, tracking decision-making, and configuration maintenance. Specifically, the system pre-trains tracking and escape strategy networks on the ground. During the on-orbit execution phase, missions are first allocated to the slave satellites based on their functional types and capabilities, determining the combinations of slave satellites participating in the collaborative mission and their respective tasks. Subsequently, the master satellite invokes the tracking strategy network to track the target satellite, while the slave satellites maintain a preset tactical configuration around the master satellite. Upon entering the final encirclement phase, the master satellite maintains a safe distance, and the system constructs a virtual master satellite as a virtual pursuit center to continue approaching the target satellite. The slave satellites then switch to maintaining a tactical configuration around the virtual master satellite, thus forming an orbital encirclement of the target satellite.

[0030] In terms of game theory decision-making, this embodiment of the invention introduces deep reinforcement learning technology to train the pursuit-escape game strategy. This method transforms the complex online numerical solution process in differential games into a lightweight policy network forward inference process. Offline training is completed in advance on the ground, and tracking actions are quickly generated on the satellite based on real-time state information, thereby reducing online computational complexity and improving the real-time performance of tracking decisions. Regarding slave satellite collaborative control, this embodiment of the invention introduces a proportional-differential control algorithm for tactical configuration maintenance, and when there is a collision risk with a slave satellite, it invokes the escape policy network to generate obstacle avoidance control variables, enabling the slave satellite to avoid potential collisions while maintaining its configuration. Therefore, this embodiment of the invention can achieve integrated collaboration of slave satellite task allocation, master / virtual master satellite tracking decision-making, slave satellite configuration maintenance, and safe encirclement control, while considering the heterogeneity of the satellite constellation.

[0031] Example 1: This invention discloses a heterogeneous swarm autonomous command and control system based on deep reinforcement learning. The system includes a policy network training layer 101, an intelligent game-theoretic decision-making layer 102, and a swarm collaborative planning layer 103. Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of the policy network training layer 101, the intelligent game decision layer 102, and the constellation cooperative planning layer 103 provided in an embodiment of the present invention, wherein: The policy network training layer 101 is used to pre-train the policy network in a ground environment. The policy network includes a tracking policy network and an escape policy network.

[0032] Specifically, a deep neural network is pre-trained on the ground to map the state information in the pursuit-escape game into actions. Thus, during the on-orbit execution phase, the system can quickly generate tracking or escape actions through forward inference via the policy network, reducing the complexity of online solution.

[0033] In this embodiment of the invention, the policy network is trained using a reinforcement learning framework, which employs the TD3 algorithm. The algorithm flow is as follows: Figure 2 As shown, this is an actor-critic architecture. It reduces overestimation of action value and improves training stability in continuous action control tasks through dual-critic networks, target policy smoothing, and delayed policy updates. The upper branch inputs the current state into the actor network, which outputs the action without noise. The current state and output action are then fed into the first and second critic networks, which output Q1 and Q2 (the action values ​​output by the first and second critics, respectively). The first and second target TD errors are calculated based on Q1 and Q2, respectively, and the first and second critic networks are updated in reverse order with the goal of minimizing the TD errors. The lower branch outputs the next state at the next time step to the actor-target network, which outputs the action at the next time step with added noise. The next state and the action at the next time step are then input into the first and second critic-target networks. The minimum Q-value output by the two target critic networks is taken, and Q' is calculated using the joint reward function R and the minimum Q-value. Then Q' is fed back as the target value for calculating the TD error, that is, the Critic main network training is completed by minimizing the TD error.

[0034] Understandably, in traditional methods, training a pursuing agent requires a fixed escape opponent, which can easily lead to the pursuing agent only learning strategies specific to that opponent, resulting in insufficient generalization ability. However, this invention employs a swarm reinforcement learning training paradigm, maintaining a pool of pursuing policy models, an escape policy model pool, and a rule-based policy model pool.

[0035] During each training round, candidate adversarial models are selected from other model pools using the PFSP matching algorithm. This allows the model to be trained to play against opponents of different types and strengths, thereby avoiding policy overfitting or training imbalance and improving the robustness and generalization ability of the final policy model.

[0036] Furthermore, in this embodiment of the invention, the TD3 algorithm includes six networks: an actor network for outputting actions to be performed based on the current state; a first critic network (Critic1) and a second critic network (Critic2), both used to evaluate the value of the actions corresponding to the current state and the actions to be performed; and an actor target network, a first critic target network, and a second critic target network, all of which are delayed-update copies of their respective networks to improve the stability of target value calculation.

[0037] The training process of the policy network includes the following steps: initializing the TD3 algorithm, setting the reward function, training a single policy model, and building multiple model pools for adversarial game to achieve the training of the policy network.

[0038] The initialization of the TD3 algorithm specifically includes: Let the first critic network and the second critic network be respectively... , Actor network is Initialize network parameters to obtain , , Let the first critic's target network and the second critic's target network be respectively... , The actor's target network is The network parameters are then passed to their respective target networks. , , .

[0039] make After interacting with the environment for N rounds, N data tuples are obtained, and the data tuples are:

[0040] in, This is the current state data. The action to be executed by the Actor network in its current state. The reward obtained after performing the action. This provides the next state data after the action is performed. These N data tuples are added to the experience pool for training the policy network.

[0041] The specific process of training a single model includes: selecting m tuples from the experience pool, each tuple being represented as... .

[0042] The target action corresponding to the next state is calculated using an actor target network, and truncated noise is added to limit the range of actions taken. Within the range: ,

[0043] in, It is Gaussian noise. The next step in targeting actors' online presence. This is the target action after noise has been added, used for the next step of prediction.

[0044] The target value is then calculated using the first critic target network and the second critic target network. To reduce overestimation, the smaller value output by the two critic target networks can be selected as the target value.

[0045] in, For the target value, Let be the immediate reward corresponding to the i-th experience sample tuple. As a discount factor, For the first critic target network or the second critic target network.

[0046] The first critic network and the second critic network are updated according to the target value, and the loss function during the update process is:

[0047] in, For the critics network The loss value at time, where m is the number of sample batches sampled from the experience pool. For the first or second critic network, Let Q be the Q-value of the commentator network output for the i-th sample group.

[0048] The actor network is updated every 5 steps, and the loss function of the actor network is:

[0049] in, is the loss value, m is the number of sample batches sampled from the experience pool, and i is the sample index number; This represents the current state corresponding to the i-th sample group; For actors, network according to status The output action; For the first commentator network, This indicates the first commentator network's opinion on the state. And the action value evaluation value of the actions output by the actor network. It can be understood that by minimizing the loss function of the actor network and updating its parameters, the actor network can output actions that receive higher evaluation values.

[0050] Finally, a soft update is performed on the actor target network and the critic target network (i.e., the first critic target network and the second critic target network):

[0051]

[0052] in, This is the soft update coefficient. Let i be the network parameters of the i-th critic network. Let i be the network parameters of the target network for the i-th critic. For the network parameters of the actor network, These are the network parameters for the actor target network. It is understandable that through the above iterative training process, a tracking policy model or an escape policy model can be obtained, and the trained model can be added to the corresponding policy model pool.

[0053] In reinforcement learning, the reward function not only determines the convergence speed of the policy network but also affects the final training effect. This invention comprehensively considers the relative distance changes between the pursuing and escaping stars, tracking time, and fuel consumption, thus designing tracking and escape reward functions. Both the tracking and escape reward functions consist of a process reward and a terminal reward. This multi-objective design guides the reinforcement agent to autonomously weigh the relationship between pursuit speed, maneuver cost, and task time window during training, ultimately learning a balanced optimal policy that balances speed, efficiency, and energy consumption, greatly enhancing the policy's practical value.

[0054] The tracking reward function is as follows:

[0055] in, This is the state-space encoding at time t. Let t be the tracking action. Rewards for tracking the process, To track terminal rewards; The reward for the tracking process is:

[0056] in, This is the predicted distance between the tracking star and the target star at the next moment when the tracking star is not performing tracking actions; This refers to the actual distance between the tracking satellite and the target satellite at the next moment after the tracking maneuver is performed. Used to evaluate the distance improvement effect of the current tracking action compared to not performing the tracking action; This indicates the relative distance between the tracking satellite and the target satellite before the tracking action is performed; This represents the relative distance reduction value after the tracking action is performed; The acceleration norm corresponding to the current tracking action is used to characterize fuel consumption, and the larger the acceleration norm, the greater the corresponding fuel consumption. , , These are the weight parameters.

[0057] The reward for tracking the terminal is:

[0058] in, To track terminal rewards, The time decay coefficient, The maximum number of training steps, To capture the distance threshold, as Increase The reward gradually decreases, meaning the shorter the capture time, the greater the capture reward.

[0059] The escape reward is as follows:

[0060] in, This is the state-space encoding at time t. Let t be the escape action at time t. As a reward for the escape process, Rewards for escaping the terminal.

[0061] The reward for the escape process is:

[0062] in, As a reward for the escape process, Used to evaluate the distance increase effect of an escape star performing an escape maneuver compared to not performing an escape maneuver; This represents the increase in the relative distance between the escape star and the pursuing star after the escape maneuver is executed. The acceleration norm corresponding to the escape action reflects fuel consumption; the less fuel consumed, the greater the reward. , , These are the weighting parameters used to balance the weights of each component.

[0063] Furthermore, an escape terminal reward can be obtained, which is as follows:

[0064] in, This is a reward for escaping terminals, used to inflict penalties when the escaped planet is captured. This is the time decay factor; the later the escape star is captured, the smaller the corresponding penalty.

[0065] It is understood that the reward function in this embodiment of the invention comprehensively considers the relative distance change, task time and fuel consumption. Therefore, training the policy network through the above reward function can avoid the policy network from tracking or escaping quickly without regard to cost, so that the policy network can make a trade-off between speed and resource consumption, thereby improving the practicality and sustainable execution capability of the policy.

[0066] Furthermore, in adversarial environments with two agents, traditional one-to-one alternating training methods have inherent flaws. During training, a small improvement in one agent's strategy may be quickly adapted by the other, causing the policy levels of both agents to fall into a cycle, stagnate, or even result in one agent gaining a long-term advantage over the other, leading to training imbalance. Therefore, in this embodiment of the invention, a swarm reinforcement learning algorithm is used for training.

[0067] like Figure 3 The diagram illustrates the training process of a policy network based on a swarm reinforcement learning algorithm provided in this embodiment of the invention. It establishes a tracking policy model pool, an escape policy model pool, and a rule-based policy model pool. The pool is continuously supplemented and selected using the PFSP method to achieve matching. Then, the corresponding parameters are updated using the TD3 algorithm, ultimately completing the construction of the escape policy network and the tracking policy network. The specific process of training the policy network based on multiple model pools includes: dividing the trained models into three parts to construct a tracking policy model pool, an escape policy model pool, and a rule-based policy model pool. Each model pool includes multiple corresponding models. For example, the tracking policy model pool contains multiple tracking policy models obtained from different training stages, the escape policy model pool contains multiple escape policy models obtained from different training stages, and the rule-based policy model pool contains multiple tracking policy models or escape policy models generated based on preset rules.

[0068] The PFSP matching algorithm can be used to select adversarial models. Specifically, it involves calculating selection weights based on the historical average rewards of the model to be trained and the candidate adversarial models, making the model to be trained more inclined to choose opponents with training value for adversarial training.

[0069] The model's selection weights are described as follows:

[0070] in, The weights for selecting the i-th candidate adversarial model are... This represents the historical average reward obtained by the tracking policy model to be trained when it engages in adversarial training against the i-th candidate adversarial model. A smaller average reward indicates that the candidate adversarial model is more difficult for the training model to handle, the more challenging the model is, and the higher the training selection weights are required. This represents the historical average reward obtained when the tracking strategy model to be trained engages in adversarial competition with the k-th candidate adversarial model.

[0071] like Figure 4 The diagram shown illustrates the execution flow of a rule-based escape strategy provided in an embodiment of the present invention. The rule-based escape strategy can determine the next moment ( ) of the escape star based on the CW equations and the thrust limit of the escape star. The maximum achievable motion region (escape region) S is determined, and within this region, the escape target point A of the tracking star under uncontrolled conditions is calculated using the CW equation. Then, point B, the point farthest from A within the escape region, is identified as the target point, and the escape star's position in the next moment is determined. The thrust and thrust angle at each moment ensure the escape satellite reaches the target point in the next moment. Then, based on the escape target point, the required thrust magnitude and direction are calculated. The thrust magnitude and threshold of the thrusters are used to determine if amplitude limiting is necessary. If so, amplitude limiting is performed. Then, it is determined whether the maximum mission duration has been reached or the escape satellite has entered the capture area. Otherwise, it directly checks whether the maximum mission duration has been reached or whether the escape satellite has entered the capture area. When the maximum mission duration is reached or the escape satellite enters the capture area, the current round stops; otherwise, the calculation continues until the maximum mission duration is met or the escape satellite enters the capture area. The maximum mission duration is... .

[0072] like Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the execution flow of a rule-based pursuit strategy provided in an embodiment of the present invention. In the rule-based pursuit strategy, the CW equation is used to predict the behavior under uncontrolled conditions. The position of the escape star at the current moment is determined, and the predicted position is used as the expected position of the pursuer star. Then, the magnitude and direction of the thrust required for the pursuer star to reach the expected position at the current moment are calculated, and the amplitude is limited according to the thruster capability constraints. When the maximum mission duration is reached or the escape star enters the capture area, the current round is stopped; otherwise, the next strategy is executed.

[0073] Understandably, the PFSP matching algorithm and the selection of adversarial models using weighted selection during the matching process allow the model to face more challenging opponents, avoiding policy overfitting and training loops caused by long-term training with fixed opponents. Simultaneously, introducing a rule-based policy model pool provides a stable baseline opponent in the early stages of training and increases the diversity of adversarial examples. The policy model obtained through this training paradigm exhibits stronger robustness and generalization ability compared to traditional fixed-opponent training methods.

[0074] Furthermore, in this embodiment of the invention, the Actor network and the Actor target network are respectively the actor network and the actor target network, such as... Figure 6The diagram shown is a schematic of the actor network structure provided in an embodiment of the present invention, including two MLP network layers, a hidden activation layer ReLU, and a tanh activation layer. The multilayer perceptron MLP performs a linear weighted transformation on the input state to extract features. The ReLU activation function sets the negative numbers in the features to 0, introducing nonlinearity. The multilayer perceptron MLP performs linear weighting again and maps the features. The tanh activation layer compresses the features, that is, it limits the range of the action, and finally outputs the agent's action.

[0075] During training, the input to the actor network is the relative state code between the current pursuing star and the escaping star. The relative state code is as follows:

[0076] in, Puma's coordinates representing the relative positions of the tracking star and the target star. To encode the relative velocities of the pursuing and target stars, the relative state codes are input into the actor network. The actor network uses two fully connected neural networks and a tanh activation layer to analyze and make decisions, scaling the action space and outputting the next action. The next action can be represented as:

[0077] in, For thrust acceleration, For thrust angle, The pitch angle.

[0078] Both the Critic network and the Critic target network belong to the category of critic networks, such as... Figure 7 The diagram shown illustrates the critic network structure provided in this embodiment of the invention. It includes two MLP network layers and a ReLU hidden activation layer. The critic network scores the agent's actions output by the actor network. The input to the critic network is the current state code X and the next action A. The current state code X and the output action A are concatenated and input into the critic network. The MLP network layer in the critic network performs a linear weighting on the input to extract features; ReLU sets negative numbers to 0, introducing non-linearity. The MLP then performs another linear transformation, outputting an action score to evaluate the quality of the action.

[0079] The intelligent game decision layer 102 includes: a strategy generation module; the strategy generation module is used to obtain the trained tracking strategy network and generate the tracking actions of the tracking star towards the target star.

[0080] Specifically, when the virtual master star completes its pursuit of the target star or the virtual master star and the target star coincide in position, the slave star can achieve orbital encirclement of the target star, while the master star of the slave star will also monitor and command from a safe distance.

[0081] In the initial pursuit phase, the strategy generation module invokes the trained tracking strategy network to generate the next tracking action of the primary star based on the current states of the target star and the primary star, enabling the primary star to act as the pursuing star in pursuit of the target star. In the final encirclement phase, the primary star and the target star maintain a preset safe distance, and the system constructs a virtual primary star as a virtual pursuit center. The strategy generation module then invokes the tracking strategy network to generate the next tracking action of the virtual primary star based on the current states of the target star and the virtual primary star, allowing the virtual primary star to continue approaching the target star.

[0082] In this embodiment of the invention, the virtual master star is not a physical satellite, but a virtual reference point constructed during the final encirclement phase. The virtual master star is used to replace the physical master star in continuing the tracking process towards the target star, thereby avoiding the collision risk caused by the physical master star continuously approaching the target star; at the same time, the virtual master star serves as a reference center for maintaining the configuration of the slave star in the final phase, enabling the slave star to form an orbital encirclement of the target star around the virtual master star.

[0083] The constellation collaborative planning layer 103 includes a global task allocation module and a tactical configuration maintenance module. The global task allocation module is used to generate a slave satellite task allocation scheme based on the satellite capability parameters of K slave satellites in the heterogeneous constellation and a global optimization algorithm. The slave satellite task allocation scheme includes the combination of slave satellites participating in the collaborative task and the task to be performed by each slave satellite, where K is an integer greater than 1.

[0084] Specifically, in order to achieve the overall combat effectiveness of the constellation, a constellation collaborative planning layer is constructed. This layer consists of a global task allocation module and a tactical configuration maintenance module, which provides a basis for subsequent dynamic game decision-making and formation.

[0085] like Figure 8 As shown, Figure 8This is a schematic diagram of the satellite constellation task allocation process provided in an embodiment of the present invention. The satellite resource library includes at least a primary satellite and multiple types of secondary satellites, including directed energy attack satellites, kinetic energy attack satellites, cyber attack satellites, electromagnetic interference satellites, and optoelectronic interference satellites. Performance weight matrices and task allocation matrices are constructed for each. For example, in the performance weight matrix, the performance of a cyber attack satellite performing task 1 (deception) is 1, its performance performing task 2 (blocking) is 1, and its performance performing task 3 (destruction) is 0.1; the performance of an electromagnetic interference satellite performing task 1 (deception) is 0.1, and its performance performing task 2 (blocking) is 0.1. The effectiveness of the first satellite is 1, and its effectiveness when performing mission 3 (destruction) is 0.2; the effectiveness of the kinetic attack satellite when performing mission 1 (deception) is 0.1, its effectiveness when performing mission 2 (interception) is 0.8, and its effectiveness when performing mission 3 (destruction) is 0.3; the effectiveness of the electro-optical interference satellite when performing mission 1 (deception) is 0, its effectiveness when performing mission 2 (interception) is 0.2, and its effectiveness when performing mission 3 (destruction) is 1; the effectiveness of the directed energy attack satellite when performing mission 1 (deception) is 0, its effectiveness when performing mission 2 (interception) is 0.2, and its effectiveness when performing mission 3 (destruction) is 0.8.

[0086] In the task allocation matrix, the network attack satellite satisfies the constraint of deception for task objective 1, but does not satisfy the constraint of blocking for task objective 2, nor the constraint of destroying for task objective 3; the electromagnetic interference satellite does not satisfy the constraint of deception for task objective 1, nor the constraint of blocking for task objective 2, but satisfies the constraint of destroying for task objective 3; the kinetic energy attack satellite does not satisfy the constraint of deception for task objective 1, satisfies the constraint of blocking for task objective 2, but does not satisfy the constraint of destroying for task objective 3; the electro-optical interference satellite does not satisfy the constraint of deception for task objective 1, satisfies the constraint of blocking for task objective 2, but does not satisfy the constraint of destroying for task objective 3; the directed energy attack satellite satisfies the constraint of deception for task objective 1, but does not satisfy the constraint of blocking for task objective 2, nor the constraint of destroying for task objective 3.

[0087] Multiplying the elements of the two matrices respectively yields the following fitness values ​​for satellites: network attack satellites have a fitness of 1 for deception, 0 for blocking, and 0 for destruction; electromagnetic interference satellites have a fitness of 0 for deception, 0 for blocking, and 0.2 for destruction; kinetic attack satellites have a fitness of 0 for deception, 0.8 for blocking, and 0 for destruction; electro-optical interference satellites have a fitness of 0 for deception, 0.2 for blocking, and 0 for destruction; and directed energy attack satellites have a fitness of 0 for deception, 0 for blocking, and 0 for destruction. These values ​​are then summed to obtain an evaluation index of 2.2. A genetic algorithm is then used to iteratively select, crossover, and mutate the task allocation matrix until the satellite task allocation scheme corresponding to the highest fitness evaluation index is output.

[0088] The specific process for generating a satellite mission allocation scheme includes: first, constructing a satellite resource database to obtain the functional types and capability parameters of each satellite, including at least sensor type, maneuverability, communication capabilities, and payload characteristics. Then, constructing a performance weight matrix based on the effectiveness of each satellite in performing different mission types. The types of missions performed by the satellites include, but are not limited to, detection missions, electro-optical jamming missions, electromagnetic interference missions, cyber attack missions, or cooperative encirclement missions.

[0089] For different slave satellite task allocation schemes, a 0-1 task allocation matrix is ​​constructed. In this embodiment of the invention, the matrix elements in the 0-1 task allocation matrix are used to indicate whether the corresponding slave satellite is assigned the corresponding task. In this embodiment of the invention, the 0-1 task allocation matrix satisfies the constraint condition of single task allocation for each slave satellite, that is, there is at most one element with a value of 1 in the same row of the matrix.

[0090] Furthermore, the corresponding elements in the performance weight matrix and the 0-1 task allocation matrix are multiplied, and the results are summed to obtain the fitness evaluation index of the satellite task allocation scheme. In this embodiment of the invention, a larger value of the evaluation index indicates a better satellite task allocation scheme.

[0091] Then, a genetic algorithm is used to globally optimize the 0-1 task assignment matrix, such as... Figure 9The diagram shows the overall flowchart of the genetic algorithm provided in this embodiment of the invention. It follows the standard paradigm of initializing the population – iterative evolution – termination and outputting the optimal solution. Specifically, it initializes the population and determines whether the iteration termination condition is met. If met, iteration stops; otherwise, replication occurs, i.e., population updates are performed through crossover and mutation. Then, it again determines whether the termination condition is met. Specifically, in practical applications, multiple different 0-1 task allocation matrices are first generated as the initial population, and the fitness evaluation index corresponding to each 0-1 task allocation matrix is ​​calculated to determine whether the termination condition is met. If the termination condition is met, the slave task allocation scheme with the highest current fitness evaluation index is output. If the termination condition is not met, a new 0-1 task allocation matrix is ​​generated through selection, crossover, and mutation operations, and the population is updated. It is understood that in the crossover and mutation operations, the new 0-1 task allocation matrix must also maintain the constraints of single task allocation for each slave. Through multiple rounds of iterative optimization, the slave task allocation scheme with the highest fitness evaluation index is output.

[0092] The tactical configuration maintenance module is used to control the slave to maintain its tactical configuration according to the slave mission allocation scheme and the motion state of the pursuing star, and to generate obstacle avoidance control variables based on the escape strategy network when there is a collision risk to the slave.

[0093] Specifically, in this embodiment of the invention, the configuration reference center is different for different stages of the pursuit and escape game. In the initial stage, the configuration reference center is the master star, and the tactical configuration maintenance module controls the slave stars to move around the master star to maintain the preset tactical configuration; when entering the final encirclement stage, the master star maintains a safe distance from the target star, and a virtual master star that approaches the target star is dynamically constructed. At this time, the configuration reference center switches from the master star to the virtual master star, and the tactical configuration maintenance module controls the slave stars to leave the master star and move around the virtual master star to form an orbital encirclement of the target star with the virtual master star.

[0094] During the initial pursuit phase, the PD control algorithm is used to overcome the effects of maneuvers and external perturbations, ensuring that the slave stars in the heterogeneous star cluster maintain a preset tactical configuration around the host star. The preset tactical configuration can be an elliptical encirclement configuration around a configuration reference center. The observed and desired positions of the slave stars are obtained, and their position errors are determined based on these positions. The position error is:

[0095] in, For positional error, For the desired position, This is the observation location.

[0096] Then, a position control quantity is generated based on the proportional gain, differential gain, and error rate of change. In this embodiment of the invention, the position control quantity is represented as the thrust from the satellite. The position control quantity is:

[0097] in, For position control commands, For proportional gain, For positional error, For the desired position, For the observation position, For differential gain, The rate of change of error, As a proportional term, it can be adjusted according to the current position error, thereby helping the system respond quickly and reduce errors; As a differential term, the error change trend can be predicted based on the error change rate, thereby improving the stability of configuration maintenance while suppressing overshoot and oscillation of the system.

[0098] When a heterogeneous star cluster is in the final encirclement phase of a chase-and-escape game, the slave stars need to follow the virtual master star to orbit the target star in order to perform tasks such as close-range detection, interference, or capture. At this time, the master star maintains monitoring or command at a preset safe distance, while the virtual master star continues to approach the target star according to the tracking actions generated by the tracking strategy network. The configuration reference center of each slave star changes from the original master star to the virtual master star, that is, the slave stars maintain a tactical configuration around the virtual master star. As the virtual master star gradually approaches the target star, the slave stars form an encirclement around the target star.

[0099] At this point, the secondary star must not only maintain its configuration but also avoid collisions with the primary star, target star, or other secondary stars. Therefore, when a collision risk exists for the secondary star, the tactical configuration maintenance module generates obstacle avoidance control variables through an escape strategy network based on the relative state of the secondary star and the avoidance target. The avoidance target includes at least one of the primary star, target star, other secondary stars, or other obstacles in the space environment.

[0100] The specific process of generating obstacle avoidance control variables includes: obtaining the escape action output by the escape policy network and converting the escape action into an acceleration vector in a Cartesian coordinate system. The acceleration vector is:

[0101] Where 'a' represents the acceleration in the escape algorithm's control action. To control the thrust angle in the escape algorithm's action, To control the pitch angle in the escape algorithm's actions, It is the acceleration vector.

[0102] Next, obtain the mass of the satellite, and based on the satellite's mass and acceleration vector, generate obstacle avoidance control variables. The obstacle avoidance control variables are:

[0103] in, The variable is used for obstacle avoidance control, where m is the mass. It is the acceleration vector.

[0104] Furthermore, the obstacle avoidance control variables and position control variables are weighted and superimposed to obtain the integrated control command, which is as follows:

[0105] in, For position control quantity, To control the amount of obstacle avoidance, and These are preset weighting coefficients.

[0106] Understandably, through the above methods, the secondary stars can avoid potential collision targets while approaching and maintaining a preset tactical configuration. As the virtual primary star and the target star gradually approach and meet the preset overlap conditions or capture criteria, the secondary stars maintaining their configuration around the virtual primary star form orbital encirclements of the target star from different directions.

[0107] The embodiments of this invention take the scenario of heterogeneous star clusters chasing and surrounding as an example, such as... Figure 10 The diagram illustrates a pursuit and encirclement scenario involving a primary star, a virtual primary star, secondary stars, and a target star, as provided in an embodiment of the present invention. T1, T2, and T3 represent the motion states of the primary star, secondary stars, virtual primary star, and target star at three different times. At time T1, the three secondary stars revolve around the primary star, while the target star moves in its orbit. At time T2, a virtual primary star is generated, following the target star. Simultaneously, the secondary stars that were originally orbiting the primary star begin to orbit the virtual primary star, maintaining a safe distance from it. As the pursuit and encirclement game continues, the position of the virtual primary star coincides with that of the target star, at which point multiple secondary stars encircle the target star.

[0108] Specifically, upon receiving a pursuit mission instruction, a satellite mission allocation scheme is first generated using a genetic algorithm based on the functional type and satellite capability parameters of each satellite. This scheme determines the combination of satellites participating in the collaborative mission and the corresponding execution tasks for each satellite. The primary satellite, acting as a tracking and signal processing node, does not participate in the satellite mission allocation.

[0109] In the initial stage of the chase game, the primary star acts as the pursuing star. The intelligent game decision layer invokes the tracking strategy network to generate the primary star's next tracking action based on the current states of the target star and the primary star, enabling the primary star to track the target star. Simultaneously, the tactical configuration maintenance module generates position control variables based on the desired and observed positions of each slave star using the PD control algorithm, ensuring that multiple slave stars maintain an elliptical encirclement configuration around the primary star.

[0110] During the final encirclement phase, the primary star maintains a preset safe distance from the target star, and the system constructs a virtual primary star as a virtual pursuit center. The intelligent game decision layer invokes the tracking strategy network to generate the next tracking action of the virtual primary star based on the current states of the target star and the virtual primary star, causing the virtual primary star to continue approaching the target star. At this point, the configuration reference center of the slave star switches from the primary star to the virtual primary star, and the tactical configuration maintenance module controls the slave star to detach from the primary star and maintain a preset tactical configuration around the virtual primary star.

[0111] During the orbit of a follower star around a virtual master star, if there is a risk of collision with the master star, target star, other follower stars, or other obstacles in the space environment, the tactical configuration maintenance module calls the escape strategy network to generate obstacle avoidance control variables. The module then weights and superimposes the position control variables corresponding to configuration maintenance with the obstacle avoidance control variables to obtain the follower star's comprehensive control command. Thus, the follower star can avoid potential collisions while maintaining its tactical configuration.

[0112] As the virtual primary star gradually approaches the target star, the secondary stars maintaining a tactical configuration around the virtual primary star also gradually form an encirclement around the target star. When the virtual primary star and the target star meet the preset overlap conditions or capture criteria, multiple secondary stars complete the orbital encirclement of the target star, thereby achieving tasks such as close-range detection, interference, or capture.

[0113] It is understood that the embodiments of the present invention integrate the slave mission allocation, master star or virtual master star tracking decision, slave configuration maintenance and slave obstacle avoidance control into a complete heterogeneous star cluster collaborative control process, so that different slave stars in the heterogeneous star cluster can undertake corresponding tasks according to their own capabilities and cooperate with the master star or virtual master star to perform pursuit, escape and encirclement tasks, thereby improving the overall mission execution efficiency of the heterogeneous star cluster.

[0114] Example 2: Based on the same inventive concept, this embodiment of the invention also provides a method for autonomous command and control of heterogeneous constellations using deep reinforcement learning. Since these methods are similar to the autonomous command and control system for heterogeneous constellations using deep reinforcement learning in the aforementioned embodiment 1, the implementation of this method can refer to the implementation of the aforementioned system, and the repeated parts will not be described again.

[0115] The method specifically includes: The policy network training layer is pre-trained in a ground environment. The policy network includes a tracking policy network and an escape policy network. The global task allocation module of the constellation collaborative planning layer generates a satellite task allocation scheme based on the satellite capability parameters of K satellites in the heterogeneous constellation and the global optimization algorithm. The satellite task allocation scheme includes the combination of satellites participating in the collaborative task and the task to be performed by each satellite, where K is an integer greater than 1. In the initial stage of the pursuit, the main star is used as the pursuing star. The strategy generation module of the intelligent game decision layer calls the tracking strategy network and generates the tracking action of the pursuing star towards the target star. The tactical configuration maintenance module of the star cluster collaborative planning layer controls the follower stars to maintain the tactical configuration around the main star. When entering the final encirclement phase, the tactical configuration maintenance module of the constellation coordination planning layer controls the main star to maintain a safe distance from the target star, constructs a virtual main star as the virtual pursuit center, switches the pursuing star to the virtual main star, controls the virtual main star to continue to approach the target star according to the tracking action, and controls the slave star to detach from the main star and maintain the tactical configuration around the virtual main star. When there is a risk of collision with a target star, the tactical configuration maintenance module of the constellation collaborative planning layer calls the escape strategy network to generate obstacle avoidance control quantities, and generates integrated control commands based on the obstacle avoidance control quantities and the position control quantities corresponding to configuration maintenance, so as to achieve orbital encirclement of the target star.

[0116] Specifically, the policy network training layer reduces the online computational complexity of on-orbit pursuit and escape game decision-making by using a ground-pre-trained tracking policy network and an escape policy network.

[0117] The global task allocation module of the constellation collaborative planning layer uses a global optimization algorithm to allocate slave star combinations and slave star task roles.

[0118] The tactical configuration maintenance module of the intelligent game decision layer uses the primary star or virtual primary star as the pursuing star at different stages, and controls the slave stars to maintain the tactical configuration around the corresponding configuration reference center. At the same time, when there is a collision risk of the slave star, the tactical configuration maintenance module calls the escape strategy network to generate obstacle avoidance control quantity, and superimposes it with the position control quantity to obtain the comprehensive control command.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0120] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A heterogeneous constellation autonomous command and control system based on deep reinforcement learning, characterized in that, The system comprises a policy network training layer, an intelligent game decision-making layer, and a constellation collaborative planning layer. The heterogeneous constellation includes a primary star and K secondary stars, wherein: The policy network training layer is used to pre-train the policy network in a ground environment, and the policy network includes a tracking policy network and an escape policy network. The intelligent game decision-making layer includes a strategy generation module; the strategy generation module is used to call the tracking strategy network during on-orbit execution and generate the tracking actions of the tracking star towards the target star; The constellation collaborative planning layer includes a global task allocation module and a tactical configuration maintenance module. The global task allocation module is used to generate a satellite task allocation scheme based on the satellite capability parameters of the K satellites in the heterogeneous constellation and a global optimization algorithm. The satellite task allocation scheme includes the combination of satellites participating in the collaborative task and the task to be performed by each satellite, where K is an integer greater than 1. The tactical configuration maintenance module is used to control the satellites to maintain their tactical configuration based on the satellite task allocation scheme and the motion state of the pursuing satellite, and to generate obstacle avoidance control variables based on the escape strategy network when there is a collision risk for the satellites. In the initial pursuit phase, the pursuing star is the primary star, which tracks the target star according to the tracking action, and the secondary stars maintain a tactical configuration around the primary star. In the final encirclement phase, the primary star maintains a safe distance from the target star and constructs a virtual primary star as the virtual pursuit center. The pursuing star switches to the virtual primary star, which continues to approach the target star according to the tracking action. The secondary stars detach from the primary star and maintain a tactical configuration around the virtual primary star to achieve orbital encirclement of the target star.

2. The heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 1, characterized in that, The policy network training layer specifically includes: A model pool construction module is used to construct M model pools, each model pool including at least a tracking strategy model pool, an escape strategy model pool, and a rule strategy model pool; the tracking strategy model pool includes multiple tracking strategy models, the escape strategy model pool includes multiple escape strategy models, and the rule strategy model pool includes multiple rule strategy models, where M is an integer greater than or equal to 3; The adversarial model selection module is used to determine the model to be trained, calculate the selection weight based on the historical average reward of the model to be trained and the candidate adversarial model, and select the candidate adversarial model from the model pool other than the model pool where the model to be trained is located based on the selection weight. The historical average reward is the average reward obtained by the model to be trained and the candidate adversarial model in the historical game process. The adversarial module is used to obtain a trained tracking strategy model or a trained escape strategy model by playing a game against the model to be trained and the candidate adversarial models, and to add the trained tracking strategy model or the trained escape strategy model to the corresponding strategy model pool as a trained strategy network.

3. The heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 2, characterized in that, The policy network training layer further includes a reward function construction module, which specifically includes: The status acquisition unit is used to acquire the relative distance between the tracking star and the target star, the change in relative distance, and the acceleration corresponding to the execution of the tracking action; The tracking process reward function construction unit is used to construct a tracking process reward sub-function based on the predicted distance between the tracking satellite and the target satellite at the next moment when the tracking satellite does not perform the tracking action, the actual distance between the tracking satellite and the target satellite at the next moment after the tracking action is performed, the relative distance reduction value before and after the tracking action is performed, and the acceleration. in, This is the predicted distance between the tracking star and the target star at the next moment when the tracking action is not performed. This refers to the actual distance between the tracking star and the target star at the next moment after the tracking action is performed. This indicates the relative distance between the tracking satellite and the target satellite before the tracking action is performed. This indicates the relative distance between the tracking satellite and the target satellite after the tracking action is performed again. This represents the relative distance reduction value after the tracking action is performed. For the acceleration, , , These are weight parameters; A terminal reward function construction unit is used to construct a terminal reward sub-function, wherein the terminal reward sub-function is: in, To track terminal rewards, The time decay coefficient, The maximum number of training steps, The capture distance threshold; The comprehensive reward construction unit is used to merge the tracking process reward sub-function and the tracking terminal reward sub-function to construct a comprehensive reward function for training the tracking policy network.

4. The heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 2, characterized in that, The policy network training layer also includes: The strategy model training module is used to iteratively train and generate the tracking strategy model and the escape strategy model based on the actor-critic architecture and the TD3 algorithm.

5. A heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 4, characterized in that, The policy model training module is specifically used for: Initialize the network parameters of the actor network, the first critic network, and the second critic network, and initialize the corresponding actor target network, the first critic target network, and the second critic target network; The initialized actor network interacts with the simulation environment to obtain a data tuple, which includes the current state data, the action output under the current state data, the reward obtained, and the next state data after the action is executed. The network parameters of the initialized first critic network and the initialized second critic network are updated using the data tuples and the TD3 algorithm, and the network parameters of the initialized actor network are updated according to a preset update frequency. The actor target network, the first critic target network, and the second critic target network are updated using the updated actor network, the updated first critic network, and the updated second critic network. Based on the updated actor network, the updated first critic network, the updated second critic network, and their respective target networks, iterative training is performed to obtain policy models, which are the tracking policy model and the escape policy model.

6. The heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 1, characterized in that, The global task allocation module is specifically used for: Based on the functional type and satellite capability parameters of each satellite, an effectiveness weight matrix is ​​constructed. The matrix elements in the effectiveness weight matrix are used to characterize the effectiveness of each satellite in performing the corresponding task type. Construct a 0-1 task allocation matrix to characterize the satellite task allocation scheme, wherein the 0-1 task allocation matrix must satisfy the constraints of the single task allocation of each satellite. Multiply the corresponding elements of the performance weight matrix and the 0-1 task allocation matrix and sum them to obtain the fitness evaluation index used to evaluate the rationality of the slave task allocation scheme. Based on the fitness evaluation index, the 0-1 task allocation matrix is ​​iteratively selected, crossovered, and mutated using a genetic algorithm to optimize it and output the slave task allocation scheme corresponding to the highest fitness evaluation index.

7. The heterogeneous constellation autonomous command and control system based on deep reinforcement learning as described in claim 1, characterized in that, The tactical configuration maintenance module is specifically used for: Based on the position of the current configuration reference center and the preset tactical configuration, the desired positions of each of the slave stars are determined; wherein, the current configuration reference center is the master star in the initial stage of pursuit and the virtual master star in the final encirclement stage, and the preset tactical configuration includes an elliptical encirclement configuration formed around the current configuration reference center; Obtain the observed position of each of the slave stars, and determine the position error of each of the slave stars based on the observed position and the desired position; Based on the position error, error rate of change, and PD control algorithm, a position control variable is generated to control the slave satellite to maintain its tactical configuration. The position control variable is: in, For position control quantity, For proportional gain, For positional error, = , For the desired position, For the observation position, For differential gain, The rate of change of error; When the slave star is at risk of collision, obstacle avoidance control quantity is generated through the escape strategy network based on the relative state of the slave star and the object to be avoided; The position control quantity and the obstacle avoidance control quantity are superimposed to generate the integrated control command for the slave satellite. The integrated control command is as follows: in, The integrated control command, For position control quantity, To control the amount of obstacle avoidance, and These are preset weighting coefficients.

8. A deep reinforcement learning-based heterogeneous constellation autonomous command and control method, applied to a deep reinforcement learning-based heterogeneous constellation autonomous command and control system as described in any one of claims 1-7, characterized in that, include: The policy network training layer is pre-trained in a ground environment. The policy network includes a tracking policy network and an escape policy network. The global task allocation module of the constellation collaborative planning layer generates a satellite task allocation scheme based on the satellite capability parameters of K satellites in the heterogeneous constellation and a global optimization algorithm. The satellite task allocation scheme includes the combination of satellites participating in the collaborative task and the task to be performed by each satellite, where K is an integer greater than 1. In the initial stage of the pursuit, the main star is used as the pursuing star. The strategy generation module of the intelligent game decision layer calls the tracking strategy network and generates the tracking action of the pursuing star towards the target star. The tactical configuration maintenance module of the star cluster collaborative planning layer controls the slave star to maintain the tactical configuration around the main star. When entering the final encirclement phase, the tactical configuration maintenance module of the star cluster cooperative planning layer controls the primary star to maintain a safe distance from the target star, constructs a virtual primary star as a virtual pursuit center, switches the pursuit star to the virtual primary star, controls the virtual primary star to continue to approach the target star according to the tracking action, and controls the slave star to detach from the primary star and maintain a tactical configuration around the virtual primary star. When the target star is at risk of collision, the tactical configuration maintenance module of the constellation cooperative planning layer calls the escape strategy network to generate obstacle avoidance control quantities, and generates integrated control commands based on the obstacle avoidance control quantities and the position control quantities corresponding to configuration maintenance, so as to achieve orbital encirclement of the target star.

Images