Aircraft cluster multi-task scheduling system based on cooperative game and working method thereof

Abstract

The invention discloses an aircraft cluster multi-task scheduling system based on a cooperative game and a working method thereof. The system comprises a simulation environment module, a hierarchical strategy network module, a centralized value network module and a training and execution module. The simulation environment module is used for constructing a multi-agent air combat confrontation environment and generating states, rewards and interaction data required by training; the hierarchical strategy network comprises a shared space-time representation encoder f theta (.), a high-layer strategy network pi H (aHz) and a low-layer strategy network pi L (aLz; aH); the centralized value network is used for receiving global state information in a training stage, estimating the overall return of our agent cluster, calculating a dominant function, and realizing the optimization of the network by minimizing the value loss; and the training and execution module optimizes a centralized value network parameter and hierarchical strategy network parameters theta H and theta L by using a global state St in a centralized training stage, and outputs an air combat decision ai according to local observation independent decisions of each agent in a distributed execution stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent adversarial game and task scheduling, in particular to a spacecraft cluster multi-task scheduling system based on cooperative game and a working method thereof. BACKGROUND

[0002] In recent years, spacecraft clusters have been widely used in tasks such as cooperative interception, escort, and reconnaissance suppression in adversarial airspace. Compared with single-machine operation, clusters can form a synergistic advantage in a larger airspace and more complex situation, but the resulting multi-task scheduling problem is more challenging: task arrival is random, target threat levels and spatial distribution are constantly changing, engagement geometry and weapon use have strict constraints (range, ammunition, launch permission, kill radius, etc.), platform dynamics and safety intervals need to be continuously satisfied, and alliance coordination and redistribution also need to be performed on a rolling time axis. How to make the formation make reasonable choices among multiple tasks and maintain a high win rate under dynamic uncertainty and strong confrontation has become a research hotspot in the fields of swarm intelligence and adversarial game.

[0003] With the development of reinforcement learning in swarm intelligence, researchers have begun to use multi-agent reinforcement learning for adversarial coordination and task scheduling of spacecraft clusters and have achieved certain results on simulation platforms. Existing training paradigms mostly use centralized training and decentralized execution (CTDE), value decomposition (VDN / QMIX), or single-layer policy gradient (such as MAPPO, PPO) frameworks, drive learning through pre-designed reward functions, and interact for a long time under fixed networks and hyperparameters to obtain executable policies. However, in cooperative game-type adversarial scenarios, the above methods generally rely on artificially designed reward templates and weights to compromise "win rate, loss, distance, formation constraints," etc. The weights are difficult to reconcile and mismatch with the situation; reward coupling leads to a large search space, large gradient variance, low sample efficiency, and long learning time; the non-stationarity introduced by the opponent's strategy causes the experience distribution to drift continuously, and the training process is prone to oscillation or premature stagnation, making it uncertain to stabilize and reproduce high win rates.

[0004] The improved methods for the above problems mainly include: completing the end-to-end learning of "alliance formation / task assignment / maneuver control" at the same time by using a single-layer strategy, or using value decomposition to split the team return to individuals to alleviate the credit allocation, or introducing imitation / reverse learning to reduce the reward design pressure. However, the single-layer strategy strongly couples tactics and maneuver, and is prone to combinatorial explosion and unstable training in the joint action space; the value decomposition relies on the additivity or monotonicity assumption, and it is difficult to depict the alliance value and marginal contribution, and the decision of "who allies with whom to fight against what kind of target" lacks clear expression; the imitation / reverse strong method relies on high-quality expert trajectories and has high computational cost, and it is difficult to cover the opponent's variable tactics. More importantly, the existing methods generally lack explicit cooperative game modeling and hierarchical time scale strategy updating, and it is difficult to maintain the stability and real-time performance of training and deployment when facing the opponent's tactical switching, weapon / safety constraints and replanning requirements. Therefore, the existing training strategies still have problems such as insufficient computability, stability and fairness in the victory rate-oriented cooperative game confrontation, and further improvement is needed.

[0005] However, for the multi-task scheduling of the aircraft cluster in the confrontation of the airspace, the existing centralized optimization, single-layer reinforcement learning and static cooperative game method still has the following technical deficiencies: ① Lack of explicit modeling of opponent / alliance value, making it difficult to uniformly process the cooperative decision of "alliance formation-task assignment-individual maneuver" in the confrontation environment; ② Tactics allocation and maneuver control are often coupled in a single-layer strategy, resulting in a large search space, unstable training and poor adaptability to tactical switching; ③ Value / credit allocation often relies on additivity or experience weight, making it difficult to reflect marginal contribution and guide stable learning; ④ Under the conditions of combat constraints (range, ammunition, launch permission, safety interval) and rolling replanning, there is a lack of computable online updating mechanism, based on which the present application proposes a cooperative game-based multi-task scheduling method for aircraft clusters. SUMMARY

[0006] In view of the technical problems of high coupling degree of tactical decision and maneuver control, unstable training and difficult credit allocation in the traditional method, the present application provides a cooperative game-based multi-task scheduling system for aircraft clusters.

[0007] Another object of the present application is to provide a working method of the cooperative game-based multi-task scheduling system for aircraft clusters.

[0008] The technical scheme adopted to achieve the object of the present application is:

[0009] A cooperative game-based multi-task scheduling system for aircraft clusters, comprising a simulation environment module, a hierarchical strategy network module, a centralized value network module and a training and execution module;

[0010] The simulation environment module is used to build a multi-agent air combat confrontation environment, and generate the state, reward and interaction data required for training;

[0011] The hierarchical policy network comprises a shared spatio-temporal representation encoder f θ (·), a high-level policy network π H (a H |z) and a low-level policy network π L (a L |z; a H ) three modules, the shared spatio-temporal representation encoder f θ (·) maps the local observation of each agent i into an abstract feature vector The high-level policy network π H (a H |z) outputs a probability distribution of high-level tactical actions according to the feature vector, and the low-level policy network π L (a L |z; a H ) outputs a Gaussian distribution of continuous maneuver instructions when the ESCAPE is triggered.

[0012] The centralized value network is used to receive global state information in the training stage, estimate the overall return of the agent cluster, calculate the advantage function according to the estimated value of the overall return, and then guide the update of the policy network through the policy gradient and optimize the network itself through minimizing the value loss.

[0013] The training and execution module optimizes the parameters of the centralized value network and the hierarchical policy network in the centralized training stage by using the global state S t and the hierarchical policy network parameters θ and θ H and θ L , and each agent independently decides according to the local observation in the distributed execution stage, and outputs the air combat decision a i .

[0014] In the above technical solution, the interaction data generated by the simulation environment module includes the local observation and the global state S t , and the reward includes: a reward for encouraging approaching enemy agents and attacking, a behavior penalty for avoiding over-attack, escape and blind approach, a reward based on hitting and being hit, and a final reward based on the final victory or defeat.

[0015] Another aspect of the application is a working method of an aircraft cluster multi-task scheduling system based on cooperative game, comprising the following steps:

[0016] ​Step 1, build an agent cluster air combat confrontation simulation environment with equal number of friendly and enemy agents, model each agent i in the confrontation simulation environment as a multi-agent Markov game model, and update the state of each agent i using a discrete-time particle model According to the state of each agent i Define local observation And global state S t Set different rewards according to the environment

[0017] Step 2, centralized training of friendly agents: initialize hyperparameters and policy network, reset simulation environment and clear experience replay buffer, then perform interaction sampling loop between friendly agent cluster and simulation environment, agent executes current policy in simulation environment and stores generated trajectory data in buffer, after each interaction, simulation environment generates corresponding reward and feeds back to friendly agent for comprehensive guidance of friendly agent's tactical behavior, when reaching round termination condition or buffer data volume accumulates to preset step number, pause interaction and enter estimation and arrangement stage, use centralized value network Forward propagation of global state S t Get the estimated value of the expected cumulative return of the friendly agent cluster in the future state Calculate advantage function using generalized advantage estimation combining reward data collected by agent Based on advantage function Use proximal policy optimization algorithm to build high-level policy loss function And low-level policy loss function Use Huber loss as value network loss function The high-level policy loss function The loss function of low-level policy And value network loss function Get the total loss function L total And use Adam optimizer to minimize the total loss function L total Update centralized value network parameters And hierarchical policy network parameters θ H And θ L ; Update centralized value network parameters And hierarchical policy network parameters θ H And θ L After updating, if the total number of rounds is not reached, return to the environment reset step to start a new round of iteration; if the upper limit is reached, the training is completed

[0018] Step 3, after centralized training, enter distributed execution, each friendly agent i receives the updated hierarchical policy network parameters θ Hand θ L , the agent is based on its local observation output air combat decision a i , the air combat decision a i Divided into high-level decision and low-level decision Two levels, the high level Need to select a tactical mode at each decision step, the low level Only when Activated and involved in gradient update.

[0019] In the above technical solution, in step 1, the local observation By the state of the agent i itself And its relative state with the closest enemy agent j is spliced; the global state S t Spliced from the complete state information of enemy and our agents.

[0020] In the above technical solution, in step 2, the total loss function L total The formula is:

[0021]

[0022] In the formula, c1 and c2 are hyperparameters, H(π) is the entropy of the high-level policy network, The value network loss function, The loss function of low-level policy, The loss function of high-level policy.

[0023] In the above technical solution, the value network loss function The formula is:

[0024]

[0025] In the formula, B represents a training batch of data, R t The target return value is the future discount cumulative target return value corrected by generalized advantage estimation.

[0026] In the above technical solution, the loss function of high-level policy The formula is:

[0027]

[0028] In the formula, ρ H (θ H ) is the probability ratio of high-level policy at an action, ε is a hyperparameter, The empirical expectation, The advantage function;

[0029] In the technical solution, the loss function of the low-level strategy in step 2 is calculated according to the following formula:

[0030]

[0031] In the formula, ρ L (θ L ) is the probability ratio of the low-level strategy on an action.

[0032] In the technical solution, the advantage function

[0033]

[0034] In the formula, r t is the average reward of the team at this time, γ is a discount factor, l is the number of steps from the current time t, δ t+l is the time difference error of the future time t+l;

[0035] In the technical solution, the time difference error of the future time t+l is calculated according to the following formula:

[0036]

[0037] In the formula, r t is the average reward of the team at this time, is the central value network at time t+1, is the central value network at time t.

[0038] Compared with the prior art, the beneficial effects of the present application are:

[0039] ​​This invention employs a hierarchical action structure of "high-level tactical decision-making at every step + low-level event triggering only upon disengagement," and utilizes centralized training and distributed execution. The Execution (CTDE) framework introduces a centralized value network used only during the training phase. Combined with PPO target pruning, GAE advantage estimation, advantage standardization, Huber (Smooth-L1) value regression, and unified gradient norm pruning and mini-batch multi-round updates for the four parameter groups (encoder, high-level, low-level, and value), it directly reduces gradient variance and training oscillations caused by optimization at the same level for tactics and maneuvers. It also alleviates non-stationarity and credit allocation problems caused by adversaries, making the learning process more stable and sample utilization higher. Under engagement constraints such as range, ammunition quantity, launch permission, and safety intervals, it enables step-by-step rolling reassignment and maneuver fine-tuning, inevitably improving strategy feasibility and win rate. Simultaneously, due to parameter sharing and the absence of a value network at the execution end, the inference model is lighter, with lower communication and computational overhead, exhibiting good engineering deployability and robustness. It enables online alliance reorganization, task reassignment, and maneuver execution under engagement constraints, thereby improving training stability and scheduling performance related to win rate. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the training process of the present invention.

[0041] Figure 2 This is a schematic diagram of the hierarchical strategy and centralized training architecture of the present invention.

[0042] Figure 3 This is a comparison chart of the number of wins, draws, and losses between the present invention and the comparison algorithm.

[0043] Figure 4 This is a comparison chart of the average rewards of the present invention and the comparison algorithm. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] Example 1

[0046] A multi-task scheduling system for aircraft swarms based on cooperative game theory includes a simulation environment module, a hierarchical policy network module, a centralized value network module, and a training and execution module.

[0047] The simulation environment module is used to construct a multi-agent air combat environment and generate the state, reward and interaction data required for training.

[0048] The hierarchical policy network comprises a shared spatiotemporal representation encoder f θ (·), a high-level policy network π H (a H |z) and a low-level policy network π L (a L |z; a H ) three modules, the shared spatiotemporal representation encoder f θ (·) maps the local observation of each agent i into an abstract feature vector The high-level policy network π H (a H |z) outputs a probability distribution of high-level tactical actions according to the feature vector, and the low-level policy network π L (a L |z; a H ) outputs a Gaussian distribution of continuous maneuver instructions when the ESCAPE is triggered.

[0049] The centralized value network is used to receive global state information in the training phase, estimate the overall return of the agent cluster, and calculate the advantage function according to the estimated value of the overall return , and then guide the update of the policy network through the policy gradient, and realize the optimization of the network itself by minimizing the value loss.

[0050] The training and execution module optimizes the parameters of the centralized value network and the hierarchical policy network in the centralized training phase t θ and θ H and θ L , and each agent only makes decisions according to the local observation in the distributed execution phase, outputs the air combat decision a i , realizes the cooperative scheduling process of centralized training and distributed execution.

[0051] Embodiment 2

[0052] A working method of an aircraft cluster multi-task scheduling system based on cooperative game, comprising the following steps:

[0053] Step 1, build a high-fidelity multi-agent confrontation simulation environment and Markov game model:

[0054] In order to effectively train and verify the scheduling method proposed in the present application, a 3vs3 agent cluster air combat confrontation simulation environment is first constructed. The environment is modeled as an extended form of a discrete-time Markov decision process (MDP), namely a multi-agent Markov game model.

[0055] In the aforementioned agent-based Markov game model, both the player (red) and the enemy (blue) consist of three independent aircraft agents. Both sides are assumed to have equivalent capabilities in core operational parameters such as maneuverability, weapon loadout, detection range, and weapon range to ensure fairness and challenge in the adversarial scenario.

[0056] The state of each agent i at any time t Its position vector in three-dimensional space and velocity vector Common definition, that is Environmental dynamics follows a discrete-time particle model, with the state update rule as follows:

[0057]

[0058] In the formula, Δt is the preset simulation time step (for example, Δt = 0.2 seconds), which is used to ensure the real-time performance and accuracy of the simulation; Let be the three-dimensional acceleration control command applied to agent i at time t. Simultaneously, to conform to the physical limitations of a real intelligent agent, the agent's velocity is limited to a maximum velocity V. MAX (For example, V) MAX Within 200m / s.

[0059] In a distributed execution scenario, each friendly agent i cannot obtain the complete global battlefield situation. Therefore, this invention designs a local observation method for this purpose. This local observation The state of agent i itself It is formed by concatenating its relative state (relative position and relative velocity) with that of the nearest enemy agent j, i.e. This design of local observation provides the agent with the minimum necessary information for decision-making (its own state and the main sources of threat), while reducing the information dimensionality, which is beneficial for the learning of the policy network.

[0060] To perform accurate value evaluation during the centralized training phase, a global state S is defined. t It is composed of the complete state information of all intelligent agents on the battlefield. This global state is used only during centralized training and contains complete situational information, providing an omniscient perspective for the value network. It is key to solving the problems of multi-agent credit allocation and environmental non-stationarity.

[0061] Furthermore, in order to guide our intelligent agent i to learn the desired cooperative adversarial behavior, various rewards are set according to the environment. These different rewards comprehensively consider multiple aspects such as task success or failure, process behavior, and resource consumption, specifically including the following three aspects:

[0062] Encourage the reward of approaching the enemy agent and attacking: give a larger positive reward when destroying an enemy aircraft; give a larger negative reward when being destroyed.

[0063] Reward based on destruction and being destroyed and endgame reward based on final victory or defeat: when the global task is won (all enemies are destroyed), give all surviving agents an additional team victory reward. This is the most important learning signal, directly linked to the final victory rate.

[0064] Behavior shaping rewards: To solve the problem of sparse rewards and accelerate learning, behavior shaping rewards are introduced. Including distance reward: give a reward according to the change of the distance between agent i and the nearest enemy, give a positive reward when attacking, otherwise give a negative reward. This encourages agent i to take an aggressive offensive posture; At the same time, there is an attack intention reward: when agent i chooses the "attack" tactic, give a small immediate positive reward to encourage it to remain offensive.

[0065] Behavioral penalties: including time penalties: at each time step, give all surviving agents a small negative reward to encourage agents to complete the task as soon as possible and avoid passive avoidance; Avoidance penalty, i.e. avoidance and blind approach penalty: when the agent chooses the "avoidance" tactic, give a small negative reward to prevent it from using avoidance actions too much and giving up attack opportunities; Resource penalty: when the agent fires a missile but "misses" because the distance is too far, give a negative reward to encourage it to use valuable ammunition resources under favorable conditions.

[0066] Step 2, centralized training of our agents: refer to Figure 1 , initialize hyperparameters and policy network, reset simulation environment and empty experience replay buffer, then enter the interaction sampling loop between agent cluster and simulation environment, agent executes the current policy in the simulation environment and stores the generated trajectory data in the buffer, after each interaction, the simulation environment generates the corresponding reward and feeds it back to the agent, the reward received by the agent is the fundamental source of motivation for network parameter update, which is used to guide the tactical behavior of the agent, when the round termination condition is reached or the buffer data volume accumulates to the preset number of steps, pause the interaction, enter the estimation and arrangement stage, refer to Figure 2 , use the centralized value network to forward propagate the global state S t , get the estimated value V that represents the future expected cumulative return of the agent cluster in this state, combine the immediate reward returned by the simulation environment collected by the agent to calculate the team average reward r t, the generalized advantage estimation (GAE) method is used to calculate the advantage function of each time step GAE achieves a good balance between high bias of the time-difference (TD) error and high variance of the Monte Carlo estimation by introducing a smoothing parameter λ, the advantage function The calculation formula is as follows:

[0067]

[0068] In the formula, r t is the average reward of the team at this moment, γ is a discount factor, l is the number of steps pushed back from the current time t, δ t+l is the time difference error of the future time t+l, and indicates the goodness of the current prediction of the value network;

[0069] In order to further improve the training stability and avoid the interference of different magnitude reward signals on the learning process, the calculated advantage function is standardized on the whole batch of data, so as to conform to the standard normal distribution with mean 0 and variance 1.

[0070] The calculation formula of the time difference error of the future time t+l is as follows:

[0071]

[0072] In the formula, r t is the average reward of the team at this moment, is the central value network at t+1, is the central value network at t.

[0073] Based on the advantage function , the high-level policy loss function and the low-level policy loss function are constructed by using the proximal policy optimization (PPO) algorithm. The core of the PPO algorithm is to limit the step length of each update by a clipped surrogate objective function, so as to ensure that the new policy does not deviate too far from the old policy, thereby ensuring the stability of the learning process. The calculation formula of the high-level policy loss function is as follows:

[0074]

[0075] In the formula, ρ H (θ H ) is the probability ratio of the high-level policy on an action, ε is a small hyperparameter, is the empirical expectation, which represents the average value of all sample data collected in a training batch, is the advantage function; the loss function of the high-level policy depends on both the encoder parameters θ enc and the parameters θ of the high-level policy network itself H Since the final probability of the high-level action is calculated by first processing the observation data through the encoder and then outputting the result by the high-level policy network, when performing gradient backpropagation optimization, the network parameters of the two parts need to be updated at the same time.

[0076] the loss function of the low-level policy is exactly the same, but according to the event trigger mechanism, its loss is only calculated and accumulated when the high-level action at the corresponding time step is "ESCAPE". The loss function of the low-level policy is calculated as follows:

[0077]

[0078] where ρ L (θ L ) is the probability ratio of the low-level policy at an action.

[0079] The update target of the centralized value network is to minimize the error between its predicted value and the calculated return (usually the sum of GAE advantage and old value prediction). To enhance the robustness to outliers in the training data, the present application adopts Huber loss (also known as Smooth L1 Loss), which behaves as mean squared error (MSE) when the error is small and as mean absolute error (MAE) when the error is large, taking into account both convergence speed and stability. The value network loss function is calculated as follows:

[0080]

[0081] where B represents a training batch of data, and R t is the target return value.

[0082] The high-level policy loss function the loss function of the low-level policy and the value network loss function are weighted to obtain the total loss function L total , and the calculation formula of the total loss function L total is as follows:

[0083]

[0084] where c1 and c2 are hyperparameters, and H(π) is the entropy of the policy.

[0085] The exploration is encouraged by entropy regularization term, and multiple rounds of small batch random gradient descent update are performed after the completion of data collection in a batch. After calculating the gradient in each back propagation, the gradient norm clipping is uniformly performed on all network parameter groups (encoder, high-level policy, low-level policy, value network) to limit the L2 norm of the gradient within a threshold. This key step can effectively prevent gradient explosion and is an important technical guarantee to ensure stable convergence in complex adversarial environment. Adam optimizer is used to minimize the total loss function L total updating the centralized value network parameters and the hierarchical policy network parameters θ H and θ L updating the centralized value network parameters and the hierarchical policy network parameters θ H and θ L After updating, if the upper limit of the total number of rounds is not reached, return to the environment reset step to start a new round of iteration; if the upper limit is reached, the training is completed.

[0086] Step 3, refer to Figure 2 After centralized training is completed, distributed execution is entered, and each intelligent agent i of our side receives the updated hierarchical policy network parameters θ H and θ L The parameters of the centralized value network are not loaded and used in this stage, realizing the collaborative architecture of "centralized training and distributed execution", and the intelligent agent outputs the air combat decision a i based on its local observation . To solve the "combination explosion" and unstable training problem caused by the strong coupling of tactical allocation and maneuver control in traditional end-to-end learning method, an innovative hierarchical action space design is adopted in the embodiment, and the action a i of the intelligent agent i of our side is output in two layers . The air combat decision a i is divided into high-level decision and low-level decision two levels; for the high-level decision , the high-level policy is a discrete action space, and the intelligent agent i of our side needs to select a tactical mode at each decision step. The high-level decision includes:

[0087] Attack (ATTACK): The intelligent agent locks the nearest enemy target and executes the automatic driving logic to approach the target. In this mode, when the distance to the target is less than the preset effective firing range (for example, FIRE_RANGE = 50.0 meters) and the intelligent agent's missile inventory is greater than zero, the weapon launch instruction is automatically executed. This action aims to encourage the intelligent agent to actively seek battle results.

[0088] TRACK: The agent enters a cruising state that maintains the current speed and heading, which is used for battlefield surveillance, maintaining a favorable position, or conserving energy. The introduction of this action increases the flexibility of the strategy.

[0089] ESCAPE: When facing a direct threat, the agent chooses to perform a high-maneuvering escape action. The decision-making power of this tactic will be handed over to the low-level strategy network.

[0090] Low-level decision is a continuous action space that outputs a normalized three-dimensional acceleration vector a L ∈[-1,1] 3 . The low-level strategy adopts an event-triggered mechanism: only when the high-level strategy chooses the "ESCAPE" tactic, the low-level strategy network is activated and outputs the maneuvering instruction. In the "attack" and "track" modes, the low-level network does not participate in decision-making, and its parameters do not produce gradients at this time step. This design greatly simplifies the learning problem, allowing the high-level network to focus on tactical-level trade-offs, while the low-level network focuses on learning complex high-maneuvering escape skills, achieving effective decomposition of tasks.

[0091] Comparative Example 1

[0092] The comparison chart of the number of wins, draws and losses of the aircraft cluster multi-task scheduling method based on cooperative game of the present application and the mainstream methods DDPG, DQN and PPO in the prior art is referred to Figure 3 , from Figure 3 it can be seen that the number of wins of the aircraft cluster multi-task scheduling method based on cooperative game of the present application is more than 1700 times in 2000 games, and the win rate is obviously higher than that of the other mainstream methods.

[0093] Comparative Example 2

[0094] The comparison chart of the average reward of the aircraft cluster multi-task scheduling method based on cooperative game of the present application and the mainstream methods DDPG, DQN and PPO in the prior art is referred to Figure 4 , from Figure 4 it can be seen that the average reward value of the aircraft cluster multi-task scheduling method based on cooperative game of the present application is as high as 5.7, which is higher than that of other mainstream methods, indicating that the decision of the agent using the scheduling method of the present application is more effective.

[0095] The above is only the preferred embodiment of the present application, it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An aircraft cluster multi-task scheduling system based on cooperative game, characterized in that, The simulation environment module, the hierarchical policy network module, the centralized value network module, and the training and execution module are included. The simulation environment module is configured to construct a multi-agent air combat environment, and generate state, reward and interaction data required for training. The hierarchical policy network comprises a shared spatiotemporal representation encoder f θ (·), a high-level policy network π H (a H |z) and a low-level policy network π L (a L |z; a H ) three modules, the shared spatiotemporal representation encoder f θ (·) maps the local observation of each agent i into an abstract feature vector The high-level policy network π H (a H |z) outputs a probability distribution of high-level tactical actions according to the feature vector, and the low-level policy network π L (a L |z; a H ) outputs a Gaussian distribution of continuous maneuver instructions when the ESCAPE is triggered. The central value network is used for receiving global state information in a training stage, estimating a global return of the agent cluster, and calculating an advantage function according to the estimated value of the global return Further, the policy gradient guides the update of the policy network, and the value loss is minimized to optimize the network itself. The training and execution module utilizes the global state S in the centralized training phase t Optimizing the centralized value network parameters and the hierarchical policy network parameters θ H and θ L In the distributed execution phase, each agent makes decisions independently according to the local observation and outputs the air combat decision a i .

2. The cooperative game based aircraft fleet multitasking scheduling system of claim 1, wherein, The interaction data generated by the simulation environment module includes local observations and global state S t The rewards include: rewards for approaching enemy agents and attacking, penalties for avoiding over-attack, escape, and blind approach, rewards based on destruction and being destroyed, and end-game rewards based on final victory or defeat.

3. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 1, characterized in that, The method comprises the following steps: Step 1, constructing an agent cluster air combat confrontation simulation environment in which the number of our agents and enemy agents is equal, modeling each agent i in the confrontation simulation environment as a multi-agent Markov game model, and updating the state of each agent i using a discrete-time particle model According to the state of each agent i Define local observations And the global state S t Set different rewards according to the environment; Step 2, our agent centralized training: initialize hyperparameters and policy network, reset the simulation environment and empty the experience replay buffer, then enter the interaction sampling loop between our agent cluster and the simulation environment, the agent executes the current policy in the simulation environment and stores the generated trajectory data in the buffer, after each interaction the simulation environment generates the corresponding reward and feeds it back to our agent for comprehensive guidance of our agent's tactical behavior, when the round termination condition is reached or the buffer data volume accumulates to the preset number of steps, pause the interaction and enter the estimation and arrangement stage, use the centralized value network to perform forward propagation on the global state S t to obtain the estimated value representing the future expected cumulative reward of our agent cluster in this state Combine the reward data collected by the agent to calculate the advantage function using generalized advantage estimation Based on the advantage function Use the proximal policy optimization algorithm to construct the high-level policy loss function and the low-level policy loss function Use Huber loss as the value network loss function The high-level policy loss function The loss function of the low-level policy and the value network loss function are weighted to obtain the total loss function L total , and the Adam optimizer is used to minimize the total loss function L total Update the centralized value network parameters and the hierarchical policy network parameters θ H and θ L ; the centralized value network parameters and the hierarchical policy network parameters θ H and θ L After updating, if the total number of rounds has not been reached, return to the environment reset step to start a new round of iteration; if the upper limit is reached, the training is completed; Step 3: After centralized training is completed, distributed execution begins. Each agent i receives the updated hierarchical policy network parameters θ from Step 2. H and θ L The agent is based on its local observations Output air combat decision a i The air combat decision a i Divided into high-level decision-making and lower-level decision-making Two levels, the higher level At each decision step, a tactical mode needs to be selected, the lower level Only It is activated and participates in gradient updates.

4. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 3, characterized in that, In step 1, the local observation is concatenated with the relative state of the closest enemy agent j to the current distance; the global state S is concatenated from the full state information of enemy and own agents. t is concatenated from the full state information of enemy and own agents.

5. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 3, characterized in that, In step 2, the total loss function L total The calculation formula is: where c1 and c2 are hyperparameters, H(π) is the entropy of the high-level policy network, is the value network loss function, is the loss function for the low-level policy, is the loss function for the high-level policy.

6. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 5, characterized in that, The value network loss function The calculation formula is: where B represents the data of a training batch, R t is the future discounted cumulative target return value, and is the future discounted cumulative target return value corrected by the generalized advantage estimation.

7. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 5, characterized in that, The loss function of the high-level policy The calculation formula is: where ρ H (θ H ) is the probability ratio of high-level policy on an action, and ε is a hyperparameter, is the empirical expectation, is the advantage function; 8. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 5, characterized in that, In step 2, the loss function of the low-level policy The calculation formula is: where p L (θ L ) is the probability ratio of the low-level policy over an action.

9. The working method of the cooperative game based aircraft fleet multitasking scheduling system according to any one of claims 7 or 8, characterized in that, The merit function The calculation formula is: where r t is the average reward of the team at that moment, γ is the discount factor, l is the number of steps to move back from the current moment t, and δ t+l is the temporal difference error for the future moment t + l.

10. The working method of the aircraft fleet multi-mission scheduling system based on cooperative game according to claim 9, characterized in that, The calculation formula of the time difference error of the future time t+l is: where r t is the average reward of the team at that moment, is the central value network at time t+1, is the central value network at time t.

Images