A game confrontation method and device for an unmanned cluster based on dynamic meta-learning

Abstract

The application provides a game confrontation method and device for an unmanned cluster based on dynamic meta-learning, and belongs to the technical field of unmanned cluster decision control. The method comprises the following steps: acquiring a game situation index of the unmanned cluster in a strategy learning process; acquiring a fuzzy input variable corresponding to the game situation index; inputting the fuzzy input variable into a fuzzy controller to obtain an adjustment amount of a hyperparameter of an initial strategy generator; updating the hyperparameter based on the adjustment amount, and updating the initial strategy generator based on the updated hyperparameter to obtain a target strategy generator; and generating a game confrontation strategy of the unmanned cluster based on the target strategy generator. In the application, the fuzzy controller is used as a controller of a strategy reinforcement learning process, the hyperparameter of the strategy generator is dynamically adjusted based on online analysis of real-time game situation indexes, the online meta-learning ability based on experience is injected into the unmanned cluster, and the problems of strategy oscillation, convergence to a suboptimal solution, and even complete failure to converge caused by environmental non-stationarity are solved.

Description

Technical Field

[0001] This application relates to the field of unmanned swarm decision control technology, and in particular to a game-theoretic adversarial method and apparatus for unmanned swarms based on dynamic meta-learning. Background Technology

[0002] With the widespread application of unmanned swarm systems in military confrontation, urban security and other fields, high demands are placed on their autonomous collaborative decision-making capabilities, real-time response capabilities and swarm resilience in highly dynamic and uncertain confrontation environments.

[0003] To meet these needs, existing technologies typically employ multi-agent reinforcement learning (MARL) methods for game-based behavioral decision-making. These methods generally rely on the assumption of a stable environment or fixed opponent strategies to generate game-based adversarial strategies.

[0004] However, in real-world game scenarios, the strategies of both sides are not fixed but are constantly adjusted and optimized as the competition progresses. This causes the environment faced by either side to change dynamically. The non-stationarity of the environment caused by the synchronous learning of both sides makes traditional algorithms prone to problems such as strategy oscillation, convergence to suboptimal solutions, or even complete failure to converge. Summary of the Invention

[0005] This application provides a game adversarial method and device for unmanned swarms based on dynamic meta-learning, aiming to solve the problems that traditional game adversarial behavior decision-making methods are prone to policy oscillation, convergence to suboptimal solutions or even complete failure to converge.

[0006] Firstly, this application provides a game-theoretic adversarial method for unmanned swarms based on dynamic meta-learning, including: The game situation indicators of the player's unmanned cluster during the policy learning process are obtained, and the game situation indicators characterize the game adversarial performance of the player's unmanned cluster's initial policy generator. Obtain the fuzzy input variables corresponding to the game situation indicators; The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; The target strategy generator generates a game strategy for the player's unmanned cluster.

[0007] In one embodiment, generating the game-playing strategy for the friendly unmanned cluster based on the target strategy generator includes: For each unmanned agent in the self-owned unmanned cluster, a joint state representation of the unmanned agent is generated by utilizing the heterogeneous observation information of the neighboring agents around the unmanned agent. The joint state representation is input into the target policy generator of the unmanned agent to obtain the game adversarial strategy of the unmanned agent output by the target policy generator.

[0008] In one embodiment, generating a joint state representation of the unmanned intelligent agent using heterogeneous observation information from neighboring intelligent agents around the unmanned intelligent agent specifically includes: Based on the heterogeneous observation information of all neighboring intelligent agents around the unmanned intelligent agent, an individual observation representation of the unmanned intelligent agent is generated. Based on the individual observation representations, a sparse communication topology for the unmanned intelligent agent is constructed at each time step. Generate the group interaction representation of the unmanned agent at the time step based on the heterogeneous observation information of the neighboring agents in the sparse communication topology; By integrating the individual observation representation and the group interaction representation, a joint state representation of the unmanned intelligent agent at the time step is generated.

[0009] In one embodiment, generating an individual observation representation of the unmanned intelligent agent based on heterogeneous observation information from all neighboring intelligent agents around the unmanned intelligent agent includes: Determine the entity categories of the unmanned intelligent agent and all neighboring entities around the unmanned intelligent agent, the neighboring entities including the unmanned intelligent agent and objects; For each entity category, the heterogeneous observation information of neighboring entities within the entity category is aggregated to obtain an intra-category aggregated feature representation; A global weight for each entity category is generated based on the intra-class aggregated feature representation of all entity categories. The individual observation representation of the unmanned intelligent agent is determined based on the intra-class aggregated feature representation and global weights of all the entity categories.

[0010] In one embodiment, constructing the sparse communication topology of the unmanned intelligent agent at each time step based on the individual observation representation includes: Based on the historical interaction information of the communication links between the unmanned intelligent agent and the neighboring intelligent agents, a communication value score is generated for each communication link; Differential sampling is performed on the communication value score to obtain the binary communication mask between the unmanned intelligent agent and the neighboring intelligent agent; The retained neighbor agents are determined based on the value of the binary communication mask; The sparse communication topology is constructed based on the communication links between all the retained neighboring agents and the unmanned agent.

[0011] In one embodiment, the policy learning process includes: Select and load a strategy model for the adversary unmanned cluster from the strategy model library; Control the friendly unmanned cluster to conduct asymmetric adversarial training with the opponent's unmanned cluster, and update the target policy generator of all unmanned agents in the friendly unmanned cluster based on the game situation index during the asymmetric adversarial training process. When the preset update conditions are met, a new strategy model is constructed based on all target strategy generators of the self-owned unmanned cluster, the new strategy model is added to the strategy model library, and a new strategy model is selected and loaded for the opponent's unmanned cluster.

[0012] In one embodiment, the policy learning process further includes: Collect historical data pairs, which include the intrinsic parameters of the fuzzy rule base and the learning outcome evaluation indicators corresponding to the intrinsic parameters; According to a preset cycle, a Bayesian optimization process is performed based on the historical data to update the intrinsic parameters.

[0013] In one embodiment, obtaining game situation indicators of the friendly unmanned cluster during the policy learning process includes at least one of the following: The advantage deviation index of the unmanned swarm is determined based on the historical reward deviation of the unmanned swarm during the strategy learning process. The strategy stability index of the unmanned cluster is determined based on the parameter changes of the target strategy generator of the unmanned cluster. The novelty index of the opponent's unmanned swarm strategy is determined based on the prediction error of the opponent's unmanned swarm behavior.

[0014] Secondly, this application also provides a game-playing adversarial device for unmanned swarms based on dynamic meta-learning, including; The meta-control module is used to acquire game situation indicators of the friendly unmanned cluster during the strategy learning process. The game situation indicators characterize the game adversarial performance of the initial strategy generator of the friendly unmanned cluster. The module acquires the fuzzy input variables corresponding to the game situation indicators. The module inputs the fuzzy input variables into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller. The module updates the hyperparameters based on the adjustment amount, and updates the initial strategy generator based on the updated hyperparameters to obtain the target strategy generator. The strategy learning module is used to generate a game-playing strategy for the friendly unmanned cluster based on the target strategy generator.

[0015] In one embodiment, the game-playing adversarial device for unmanned swarms further includes: The environment interaction module is used to acquire heterogeneous observation information of neighboring intelligent agents around each unmanned intelligent agent in the self-owned unmanned cluster; The information processing module is used to generate a joint state representation of the unmanned intelligent agent based on the heterogeneous observation information; Accordingly, the strategy learning module is also used to input the joint state representation into the target policy generator of the unmanned agent to obtain the game adversarial strategy of the unmanned agent output by the target policy generator.

[0016] Thirdly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the aforementioned game adversarial methods for unmanned swarms based on dynamic meta-learning.

[0017] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned game-playing adversarial methods for unmanned swarms based on dynamic meta-learning.

[0018] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the aforementioned game adversarial methods for unmanned swarms based on dynamic meta-learning. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the game adversarial method for unmanned swarms based on dynamic meta-learning provided in this application; Figure 2 This is the second flowchart of the game adversarial method for unmanned swarms based on dynamic meta-learning provided in this application; Figure 3 This is a schematic diagram of the process provided in this application for generating an individual observation representation of the unmanned intelligent agent based on the heterogeneous observation information of all neighboring intelligent agents around the unmanned intelligent agent; Figure 4 This is a schematic diagram of the process of constructing the sparse communication topology of the unmanned intelligent agent at each time step based on the individual observation representation provided in this application; Figure 5 This is a flowchart illustrating the strategy learning process provided in this application; Figure 6 This is a schematic diagram of the structure of the game adversarial device for unmanned swarms based on dynamic meta-learning provided in this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0024] With the rapid development of intelligent, low-cost, and swarm technologies for unmanned platforms such as drones and unmanned vehicles, unmanned swarm systems are playing an increasingly crucial role in modern military confrontation, urban security, emergency response, and intelligent competition. Especially in close-range combat scenarios, both sides deploy heterogeneous swarms of drones and unmanned vehicles with different maneuverability, sensing capabilities, and mission functions to perform complex strategic tasks, including coordinated reconnaissance, dynamic interception, and area defense. These scenarios are characterized by high dynamism, strong adversarial capabilities, and high uncertainty: the opponent's intentions are difficult to predict, the battlefield environment (such as buildings and electromagnetic interference) is complex and changeable, and the range and accuracy of sensor observations are limited by physical conditions. These challenges place unprecedented demands on the autonomous collaborative decision-making capabilities, real-time responsiveness, and swarm resilience of unmanned swarms. In the civilian sector, practical applications such as intercepting unauthorized drone flights in cities / airports, multi-agent competitive games, and collaborative search and rescue in complex scenarios also urgently require the development of intelligent game-based combat systems for unmanned swarms. In the military field, the development of intelligent game-based combat systems for unmanned swarms has become a consensus.

[0025] Currently, mainstream research methods for multi-agent cooperation and adversarial problems in unmanned swarms focus on multi-agent reinforcement learning, game theory, and traditional distributed control. However, these methods still face challenges such as learning failure caused by the dynamic evolution of the opponent's strategy and environmental non-stationarity. Classical multi-agent reinforcement learning algorithms, such as Independent Q-Learning (IQL) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG), typically rely on the assumption of environmental stability or fixed opponent strategies for convergence. However, in intelligent game adversarial situations, the opponent, as a collection of agents with autonomous learning capabilities, continuously adjusts and optimizes its strategy during the adversarial process. This directly leads to dynamic changes in the environment faced by the player, meaning that both the state transition probability and the reward function evolve over time. This environmental non-stationarity caused by simultaneous learning by both parties makes traditional algorithms prone to policy oscillations, convergence to suboptimal solutions, or even complete failure to converge.

[0026] The following description, in conjunction with the accompanying drawings, describes the game-playing adversarial method and apparatus for unmanned swarms based on dynamic meta-learning provided in this application.

[0027] It should be noted that the game-playing adversarial method of unmanned swarms based on dynamic meta-learning provided in this application embodiment is implemented using a game-playing adversarial device of unmanned swarms based on dynamic meta-learning. The game-playing adversarial method of unmanned swarms based on dynamic meta-learning constructs an intelligent adversarial framework that can integrate situational awareness, heterogeneous collaboration, and dynamic learning, and has good scalability. This enables unmanned swarms to maintain a strategic advantage in highly uncertain dynamic adversarial situations, allowing them to adapt to dynamic opponents, integrate heterogeneous information, and continuously evolve in game-playing adversarial situations. This not only has profound academic theoretical significance, but also has broad prospects for national defense applications and potential for civilian transformation.

[0028] This application describes the game-playing adversarial method of unmanned swarms based on dynamic meta-learning as an example, using a game-playing adversarial device based on dynamic meta-learning as the execution subject.

[0029] The application scenario of this application is a game-like confrontation between two unmanned swarms (including one's own and the other's) in a limited three-dimensional space. Each unmanned swarm includes at least one of drones and unmanned vehicles. Each drone and each unmanned vehicle has an independent unmanned intelligent agent. Each unmanned intelligent agent includes a policy generator, which generates game-like confrontation strategies for the drones or unmanned vehicles in real time.

[0030] Specifically, each unmanned cluster has a policy learning platform. During the policy learning process, the hyperparameters of the policy generator are updated using a fuzzy controller, forming a feedback loop that enables the learning process to adapt to dynamic changes in the environment. This allows the game adversarial strategy generated by the policy generator to be updated based on the dynamic information of the game adversarial.

[0031] Figure 1 This is one of the flowcharts illustrating the game-theoretic adversarial method for unmanned swarms based on dynamic meta-learning provided in this application. Figure 1 As shown, the game-playing adversarial method for unmanned swarms based on dynamic meta-learning provided in this application includes: S110: Obtain the game situation indicators of the friendly unmanned cluster during the policy learning process. The game situation indicators characterize the game adversarial performance of the initial policy generator of the friendly unmanned cluster.

[0032] Please combine Figure 2 The fuzzy meta-learning unit is used, and the input variable of the fuzzy controller is the fuzzified result of the game situation indicators. The game situation indicators are quantitative metrics of the game adversarial performance of the friendly unmanned swarm when executing game adversarial strategies during the strategy learning process, such as the stability of the game adversarial strategy.

[0033] In one possible implementation, each input variable has five linguistic variables, such as {negative large, negative small, zero, positive small, positive large}, {very low, low, medium, high, very high}, etc.

[0034] S120: Obtain the fuzzy input variables corresponding to the game situation indicators.

[0035] This step is implemented using existing obfuscation techniques, and this application does not impose any restrictions on it.

[0036] S130: Input the fuzzy input variable into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller.

[0037] The output variable of the fuzzy controller is a relative adjustment of the hyperparameters of the policy generator, such as the learning rate adjustment. α and exploration rate adjustment .

[0038] In one possible implementation, each output variable has five linguistic variables, such as {maximum, increase, remain unchanged, decrease, minimum}.

[0039] The fuzzy rules can be refined and the language variables expanded according to user needs and actual scenarios, as shown in the following example: Rule 1: If input variable 1 is negative and large, input variable 2 is low and input variable 3 is high, then output variable 1 is maximum and output variable 2 is increase; Rule 2: If input variable 1 is positive and large, input variable 2 is high and input variable 3 is low, then output variable 1 is negative and small and output variable 2 is negative and large.

[0040] The maximum number of fuzzy rules in a fuzzy rule base is the product of the number of linguistic variables in all input variables and the number of linguistic variables in all output variables. For example, in a fuzzy controller with three input variables and two output variables, where each input and output variable has five linguistic variables, the maximum number of fuzzy rules in the fuzzy rule base is: .

[0041] In one possible implementation, this application employs a Mamdani-type fuzzy inference method, using the centroid method for defuzzification, and transforming the set of fuzzy rule outputs into precise hyperparameter adjustments: the learning rate adjustment. α and exploration rate adjustment .

[0042] S140: Update the hyperparameters based on the adjustment amount (see reference) Figure 2 The parameter dynamic adjustment unit updates the initial policy generator based on the updated hyperparameters to obtain the target policy generator.

[0043] Specifically, the amount adjusted based on the learning rate α and exploration rate adjustment Obtain the updated learning rate and exploration rate : (1); (2); in, This is a truncation function. and This is a scaling factor for the learning rate and exploration rate. and For learning rate The upper and lower limits, and For exploration rate The upper and lower limits, and These are the learning rate and exploration rate before the update.

[0044] Updated learning rate and exploration rate The target policy generator is obtained by using it as a hyperparameter of the policy generator.

[0045] S150: Generate the game strategy of the unmanned cluster based on the target strategy generator.

[0046] This application's embodiments formalize the game adversarial strategy generation process of the target policy generator into a partially observable Markov game problem. Here, let the number of unmanned agents in the user's unmanned swarm be . The number of unmanned intelligent agents in the opponent's unmanned swarm is The total number of unmanned intelligent agents is Regarding state space In the middle, global state It includes the internal states (positions) of all unmanned intelligent agents. ,speed Action space includes (position, health, etc.) and environmental conditions (obstacle locations, terrain information, etc.). Each unmanned intelligent agent It has its own space for movement For drones, Includes three-dimensional acceleration commands; for autonomous vehicles It includes two-dimensional plane velocity and steering commands. The coordinated actions of an unmanned swarm are denoted as... Regarding the observation space Each unmanned intelligent agent obtains local observations. This local observation is a noisy function of the global state. The observation spaces of drones and unmanned vehicles are heterogeneous in both dimension and content. Each unmanned intelligent agent... strategy It is based on local observation Select Action The probability distribution. State transition function. Indicating joint actions The environment from state Transferred to The probability of each unmanned intelligent agent. You will receive a scalar reward at each step. In competitive games, the rewards for both sides are usually designed to be mutually exclusive.

[0047] The core objective of this application is to find a set of strategies for at least one unmanned agent in a friendly unmanned swarm (e.g., a friendly unmanned swarm). (in, For unmanned intelligent agents (Game theory adversarial strategies), making the opponent's unmanned cluster strategy In situations where learning may change dynamically over time, the long-term expected cumulative discount reward of the user's unmanned cluster is maximized: (3); in, It is a moment Discount factor, Indicates time In state and actions Scalar reward below Representing one's own side.

[0048] This application's embodiments introduce a fuzzy controller as the controller for the policy reinforcement learning process. This controller does not directly output actions, but dynamically adjusts the hyperparameters of the underlying policy generator based on the reinforcement learning algorithm by analyzing real-time game situation indicators online. This injects experience-based online meta-learning capabilities into the unmanned swarm, enabling it to automatically adjust the learning process according to the game adversarial effect. Thus, when faced with sudden changes in the opponent's strategy, it can actively explore new strategies, achieving a leap from passive adaptation to active evolution. This solves the problems of policy oscillation, convergence to suboptimal solutions, or even complete failure to converge caused by environmental non-stationarity.

[0049] In one possible implementation, the game situation indicators of the friendly unmanned cluster during the policy learning process are obtained, including at least one of the following: a. Determine the advantage deviation index of the unmanned cluster based on the historical reward deviation of the unmanned cluster during the policy learning process.

[0050] Advantage deviation index This indicates that our unmanned swarm has recently... The difference between the average reward of each round and the historical average reward (i.e., historical reward deviation) is represented by the linguistic variables {negative large, negative small, zero, positive small, positive large}. Advantage deviation index. This is used to quantify the deviation of a player's recent adversarial performance from its long-term historical average performance. It utilizes the advantage deviation metric. The system can perceive the performance trend of its own strategy at the macro level. A positive advantage deviation index indicates a recent performance improvement, while a negative value indicates a performance decline.

[0051] Determine the advantage deviation index The process includes the following steps: First, during the policy learning process, the reward value obtained by the friendly unmanned swarm in each adversarial round or time step is continuously recorded. In a specific embodiment, this reward value can be designed as the difference between the number of remaining friendly agents and the number of remaining enemy agents, thereby comprehensively evaluating the swarm's survivability and enemy-killing ability.

[0052] Next, calculate the total historical average reward, which is the arithmetic mean of all reward values ​​from the start of training to the current moment. Simultaneously, calculate the recent average reward, which is the arithmetic mean of all reward values ​​over the most recent K rounds, where K is a preset time window length; for example, K can be set to 100.

[0053] Finally, subtract the recent average reward from the total historical average reward; the difference is the current advantage deviation index.

[0054] b. Determine the policy stability index of the unmanned cluster based on the parameter changes of the target policy generator of the unmanned cluster.

[0055] Strategy stability index This represents the variance of the network parameter matrix of the target policy generator for the unmanned cluster before and after the update, with the corresponding linguistic variables being {very low, low, medium, high, very high}. Policy stability index. This metric measures the drastic changes in the target policy generator of a friendly unmanned swarm between two consecutive updates, reflecting the current stable state of the learning process. A higher policy stability metric indicates that the policy has changed drastically, and the learning process may be in the exploratory or oscillating phase; a lower metric indicates that the policy is stabilizing, and the learning process is in the fine-tuning or convergence phase.

[0056] c. Determine the strategy novelty index of the adversary unmanned swarm based on the prediction error of the adversary's behavior.

[0057] In one possible implementation, an online-trained opponent behavior prediction model (e.g., a Transformer model) is used, which is continuously trained online during policy learning to adapt to the opponent's behavior patterns.

[0058] The current behavior of the adversary unmanned swarm is predicted based on the adversary's joint actions over the past H steps, and the prediction error is used as an indicator of the adversary's strategy novelty at the current moment. The corresponding linguistic variables are {very low, low, medium, high, very high}. The novelty index of the opponent's unmanned swarm strategy. A high level of adversary unmanned swarm strategy novelty index is used to quantify the predictability of the current behavior of an adversary unmanned swarm. This means that the opponent's behavior was unexpected and they may have adopted a new strategy.

[0059] This application embodiment utilizes the aforementioned various game situation indicators to comprehensively and quantitatively perceive the non-stationary game environment from multiple dimensions, including the player's performance trend, learning status, and changes in the opponent's strategy. This provides rich and accurate decision-making basis for subsequent adaptive learning control. In particular, this application embodiment uses the output of the opponent behavior prediction model as a key input to the fuzzy controller, enabling direct quantitative perception and feedback of changes in the opponent's behavior patterns, which helps to significantly improve the accuracy of adaptive learning control.

[0060] In one possible implementation, step S150, generating the game-playing strategy for the friendly unmanned cluster based on the target strategy generator, includes: S1510: For each unmanned agent in the self-owned unmanned cluster, generate a joint state representation of the unmanned agent using the heterogeneous observation information of the neighboring agents around the unmanned agent.

[0061] S1520: Input the joint state representation into the target policy generator of the unmanned agent to obtain the game adversarial strategy of the unmanned agent output by the target policy generator.

[0062] In collaborative combat among unmanned swarms (especially heterogeneous air-ground unmanned swarms), different types of intelligent agents (such as drones and unmanned vehicles) have fundamental differences in kinematic characteristics, perception capabilities, and communication modes, namely, heterogeneous differences.

[0063] In a specific implementation scenario, each unmanned intelligent agent, whether it is a drone or an unmanned vehicle, acquires its local, heterogeneous observation information at every moment. This heterogeneous observation information may include the unmanned intelligent agent's own state (such as position, speed, attitude, remaining health points, etc.) and information about surrounding environmental entities observed through its sensors (such as cameras, radar), such as the positions and types of other friendly intelligent agents, opposing intelligent agents, and static or dynamic obstacles, thus constituting heterogeneous observation information.

[0064] Before generating the game-playing strategy, the heterogeneous observation information of each unmanned agent is first processed. The raw, heterogeneous, and local observation information is transformed into a fixed-dimensional feature vector that comprehensively reflects the current individual situation and the group's cooperative relationship, namely, the joint state representation. This joint state representation includes the agent's own understanding of the environment and the cooperative information obtained through communication with teammates.

[0065] After generating the joint state representation, it is provided to the target policy generator. In one possible implementation, the target policy generator employs the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. The target policy generator performs centralized training based on the joint representation to generate actions for each unmanned agent. : Thus, the game strategy of each unmanned intelligent agent can be obtained. The generated actions include, for example, acceleration commands for a drone in three-dimensional space, or speed and steering commands for an unmanned vehicle in a two-dimensional plane.

[0066] In an unmanned swarm, all unmanned agents execute their respective actions in a distributed manner to jointly complete a game-like adversarial task. This employs a training paradigm of Centralized Training, Decentralized Execution (CTDE). Please refer to [reference needed]. Figure 2 The CTDE policy optimizer.

[0067] In this embodiment, the heterogeneous observation information obtained by the unmanned intelligent agent is processed before policy generation to obtain a joint state representation, ensuring that the input information of the target policy generator is comprehensive, thereby improving the output accuracy of the target policy generator.

[0068] In one possible implementation, please combine Figure 2 In S1510, the step of generating a joint state representation of the unmanned intelligent agent using heterogeneous observation information from neighboring intelligent agents around the unmanned intelligent agent specifically includes: S1511: Based on the heterogeneous observation information of all neighboring intelligent agents around the unmanned intelligent agent, generate an individual observation representation of the unmanned intelligent agent. .

[0069] At every moment Based on unmanned intelligent agents Communication radius Determine the set of all possible neighboring agents that it can communicate with. ,in and It is an unmanned intelligent agent and unmanned intelligent agents The location.

[0070] Individual observation representation It is the initial, semantically rich understanding that an unmanned intelligent agent forms of itself and all entities within its perception range (including friendly forces, adversaries, obstacles, etc.), and it is the personalized understanding of the local battlefield environment by an unmanned intelligent agent.

[0071] S1512: Construct the sparse communication topology of the unmanned intelligent agent at each time step based on the individual observation representation.

[0072] Specifically, an intelligent filtering method is used to retain only the communication links between the neighboring intelligent agents most valuable to the current task and the unmanned intelligent agent, achieving a pruning effect. Subsequently, a sparse communication connection graph, i.e., a sparse communication topology, is generated based on the retained communication links.

[0073] S1513: Generate the group interaction representation of the unmanned agent at the time step based on the heterogeneous observation information of neighboring agents in the sparse communication topology. This characterizes the unmanned intelligent agent's collaborative understanding of the group.

[0074] In one possible implementation, a graph attention network is used to aggregate heterogeneous observation information from neighboring agents in a sparse communication topology to obtain a group interaction representation. Specifically, unmanned agents... From the retained neighboring intelligent agents Receiving unmanned intelligent agents heterogeneous observation information Furthermore, feature aggregation is performed through an attention mechanism to obtain a representation of group interactions. : (4); in, It is a learnable parameter matrix. It is the attention coefficient.

[0075] Attention coefficient The calculation formula is as follows: (5); (6); in, It is an unmanned intelligent agent With unmanned intelligent agents Attention coefficient It is an unmanned intelligent agent With unmanned intelligent agents Attention coefficient It is a learnable parameter matrix. It is the transpose of the learnable parameter matrix.

[0076] S1514: Integrate the individual observation representation and the group interaction representation to generate the joint state representation of the unmanned agent at the time step. .

[0077] The joint state representation formed by combining the unmanned intelligent agent's personalized understanding of the local battlefield environment with its collaborative understanding of the group creates a complete decision-making basis for a target strategy generator.

[0078] In one specific embodiment, the individual observation representation and the group interaction representation are concatenated to obtain a joint state representation. : (7).

[0079] This application embodiment filters and fuses raw, heterogeneous observation information based on communication requirements to generate a joint state representation that includes both an individual's deep understanding of the environment and efficient group collaborative information, providing high-quality input for subsequent strategy decisions.

[0080] In air-to-ground heterogeneous unmanned swarms, UAVs and unmanned vehicles have fundamental differences in kinematic models, perception modes, communication loads and latency. Most existing multi-agent reinforcement learning (MARL) research focuses on homogeneous agents, making it difficult to effectively model and utilize this heterogeneity. Therefore, existing technologies usually simply splice heterogeneous observations together or use homogeneous networks for processing, which will lose key platform characteristic information and lead to low collaborative efficiency.

[0081] Based on the above considerations, in one possible implementation, such as Figure 3 As shown, in S1511, generating an individual observation representation of the unmanned intelligent agent based on the heterogeneous observation information of all neighboring intelligent agents around the unmanned intelligent agent includes: S310: Determine the entity categories of the unmanned intelligent agent and all neighboring entities around the unmanned intelligent agent, the neighboring entities including the unmanned intelligent agent and objects.

[0082] Please refer to Figure 2 The heterogeneous graph attention individual observation representation module first, for each unmanned intelligent agent Constructing a directed heterogeneous graph ,in Indicates unmanned intelligent agents The observed set of entity nodes, each entity node is associated with an original feature vector. , Represents entity nodes Its own state vector, including position, velocity, direction, and health. This represents a fully connected neural network. Represents the set of connecting edges. Indicates unmanned intelligent agents Able to observe physical nodes .

[0083] Then, based on the directed heterogeneous graph Define the set of entity categories in the environment ,in, Indicates that it is one's own drone. This indicates that your team has unmanned vehicles. Indicates the opponent's drone. Indicates the opponent's unmanned vehicle, Represents a static object. Represents a dynamic object.

[0084] S320: For each entity category, aggregate the heterogeneous observation information of neighboring entities within the entity category to obtain an intra-category aggregated feature representation.

[0085] For category This will include all entity nodes belonging to this category. The features are input into the graph attention network, and the unmanned intelligent agent is calculated respectively. With category Medium entity node Attention coefficient : (8); in, It is a class-related learnable linear matrix. It is the attention vector corresponding to the category. It is the transpose of the attention vector. This represents a vector concatenation operation. This represents the activation function. The attention coefficients of all entity nodes within the same category are normalized using the softmax function to obtain standardized attention weights. : (9); Attention weights based on all entity nodes Get Category Intra-class aggregation feature representation : (10); in, It is an unmanned intelligent agent With category Medium entity node Attention coefficient It is a non-linear activation function.

[0086] S330: Generate a global weight for each entity category based on the intra-class aggregated feature representation of all entity categories.

[0087] In one possible implementation, a gated fusion module is used to perform cross-category feature aggregation to obtain the global weight for each entity category. : (11); in For learnable parameters, Learnable parameters transpose, It is the hyperbolic tangent function.

[0088] S340: Determine the individual observation representation of the unmanned intelligent agent based on the intra-class aggregated feature representation and global weights of all the entity categories, i.e., the unmanned intelligent agent. Individual-level observational representation of surrounding entities : (12); This application employs a category-aware heterogeneous graph attention network to explicitly extract and fuse differentiated features for different entity types, generating semantically rich individual observation representations. This solves the problem of low collaborative efficiency caused by improper processing of heterogeneous information and significantly improves the collaborative combat capability of the entire heterogeneous unmanned swarm.

[0089] As cluster sizes expand from tens to hundreds or even thousands, the interactions between agents exhibit a combinatorial explosion. The observation space and joint action space of each agent expand dramatically, causing traditional centralized or fully decentralized algorithms to face the curse of dimensionality, making unmanned clusters prone to the "decision-making dilemma." In realistic combat scenarios, limited communication bandwidth, unstable links, and susceptibility to interference are commonplace. Traditional fully connected or fixed-topology communication models are neither practical nor sustainable, and would incur significant communication overhead and latency.

[0090] Based on the above considerations, in one possible implementation, such as Figure 4 As shown, in S1512, constructing the sparse communication topology of the unmanned intelligent agent at each time step based on the individual observation representation includes: S410: Based on the historical interaction information of the communication links between the unmanned intelligent agent and the neighboring intelligent agents, generate a communication value score for each communication link.

[0091] Please combine Figure 2 The group interaction relationship representation module, at every moment Based on unmanned intelligent agents Communication radius Determine the set of all possible communicating neighbor agents. .

[0092] First, for each neighboring intelligent agent intelligent agents and In recent The observation representations at each time step are input into a Bidirectional Long Short Periodic Memory (BiLSTM) network to obtain contextual feature vectors encoding historical interaction information. Then the context feature vector Input a multilayer perceptron classifier and output a scalar communication value score. The communication value score represents the value of communication with neighbors in the next moment. The probability of maintaining the communication link.

[0093] S420: Differentiate the communication value score to obtain the binary communication mask between the unmanned intelligent agent and the neighboring intelligent agent.

[0094] In one possible implementation, Gumbel-Softmax is used to score the communication value. Differentiable sampling is performed to obtain the binary communication prediction mask. .

[0095] S430: Determine the neighboring agents to be retained based on the value of the binary communication mask.

[0096] When the binary communication mask is 1, the neighboring agent is retained; when the binary communication mask is 0, the neighboring agent is eliminated, thus achieving pruning.

[0097] S440: Construct the sparse communication topology based on the communication links between all the remaining neighboring agents and the unmanned agent.

[0098] This application constructs a dynamic communication pruning mechanism based on sequence prediction and attention weights. First, it dynamically predicts communication needs based on historical interactions and generates binary communication masks using differentiable sampling. This intelligently prunes redundant or low-value communication links and dynamically constructs a sparse but efficient collaborative information exchange network. Under limited and dynamically changing communication resources, it enables efficient and on-demand exchange of key information between heterogeneous platforms, avoiding the problem of exponential growth in communication overhead and computational complexity as the cluster size increases under traditional fully connected or fixed topology communication methods.

[0099] In one possible implementation, such as Figure 5 As shown, the policy learning process includes: S510: Select and load a strategy model for the adversary unmanned cluster from the strategy model library.

[0100] The strategy model library is a collection used to store different versions and styles of opponent strategies. When initializing the strategy model library, it can include heuristic strategies based on expert rules, as well as historical versions of the player's own strategy models.

[0101] At the start of training or at the start of each training phase, the system selects a strategy model from the strategy model library according to preset rules (such as random selection or selection of an opponent of a specific strength according to the current training difficulty) and loads it into the opponent's unmanned cluster. This means that the unmanned cluster no longer faces a single opponent during training, but a dynamic and challenging set of opponents.

[0102] S520: Control the friendly unmanned cluster to conduct asymmetric adversarial training with the opponent's unmanned cluster, and update the target policy generator of all unmanned agents in the friendly unmanned cluster based on the game situation index during the asymmetric adversarial training (see S110-S150).

[0103] Once the strategy pattern is selected, the opponent's strategy is fixed, and all target strategy generators of the player's unmanned swarm will learn and update based on the adversarial results and reward signals. Specifically, during asymmetric adversarial training, the system continuously updates the learning hyperparameters of the player's unmanned swarm's target strategy generators based on the game situation indicators, enabling the player's swarm to learn and evolve efficiently in this asymmetric adversarial environment.

[0104] S530: When the preset update conditions are met, a new strategy model is constructed based on all target strategy generators of the self-owned unmanned cluster, the new strategy model is added to the strategy model library, and a new strategy model is selected and loaded for the opponent's unmanned cluster.

[0105] In one possible implementation, the preset update condition is: the average win rate of the friendly unmanned cluster against the opponent's unmanned cluster exceeds a preset threshold.

[0106] When the preset update conditions are met, the friendly unmanned cluster has fully grasped how to deal with the current opponent's unmanned cluster. At this point, the system will construct a new strategy model based on all the target strategy generators currently in the friendly unmanned cluster. For example, the network weight parameters of all target strategy generators can be packaged and saved as a single file as this new strategy model. Subsequently, this new strategy model is added to the strategy model library as a new strategy model to be selected in the future.

[0107] After the addition is completed, the system will return to step S510, select and load a new strategy model for the adversary unmanned cluster from the updated strategy model library, and start the next round of adversarial training.

[0108] In the training process of the target policy generator for all unmanned agents in the unmanned swarm, this application combines asymmetric self-game theory with curriculum learning to maintain a policy model library containing diverse policy models. Through a cyclical evolutionary process of training-graduation-entry into the library-re-challenge, and by designing a curriculum learning progress from simple to complex (based on the basic idea of ​​curriculum learning) and from fixed to adaptive, the policies of the unmanned agents are driven to evolve robustly under continuous challenges. This allows the learning and evolution of the entire system to be directly driven by the game process. This game-driven-curriculum evolution training paradigm greatly improves the pertinence and robustness of the game adversarial strategy.

[0109] In one possible implementation, the policy learning process further includes: S540: Collect historical data pairs, wherein the historical data pairs include the intrinsic parameters of the fuzzy rule base and the learning outcome evaluation indicators corresponding to the intrinsic parameters.

[0110] In one possible implementation, the intrinsic parameters of the fuzzy rule base include the relative position parameters of the center points corresponding to each input and output variable of the fuzzy controller. and width factor .

[0111] For input variables, this application employs a triangular membership function, with a dominant deviation degree. For example, the triangular membership function of each of the five linguistic variables {negative large, negative small, zero, positive small, positive large} is determined by three parameters: the left boundary Center point and right boundary ,satisfy The parameterization process is as follows: Center point parameterization: For the center point Given and To maintain order and facilitate optimization, the relative position parameters of the three center points are defined. ,but It can be defined as follows, thereby ensuring : ; in, and It is the first The and the first The relative position parameters of the center points.

[0112] Width parameterization: Assuming that the left and right spans (distances from the center to the boundary) of all triangle membership functions are equal, a width factor is defined. Then the first The triangle boundary of the language variables is: (13); in, It is the preset base width constant.

[0113] Therefore, for input variables Its membership function has adjustable parameters including the relative position parameter of the center point. and width factor .

[0114] For the output variable, to simplify Mamdani inference, this application uses a single-point fuzzy set for parameterization, with the learning rate as the adjustment factor. For example, the single-point values ​​corresponding to its five language variables are defined as follows: Given and median The relative position parameters of the center point can be obtained in the same way as the parameterization of input variables, and this ensures... The width parameterization method is the same as that for input variables.

[0115] Based on the center point relative position parameters and width factor of each input and output variable obtained above, the intrinsic parameter vector of the fuzzy controller can be obtained. .

[0116] The learning outcome evaluation metrics corresponding to the intrinsic parameters are used to characterize the comprehensive evaluation obtained by training a preset number of times based on the intrinsic parameters. For example, the average reward rate represents the cumulative reward of all training rounds; the final win rate represents the win rate of the friendly unmanned cluster's agent against the opponent during the training process; and the convergence speed represents the number of training rounds required to reach a certain preset performance level (such as 80% win rate).

[0117] S550: Perform a Bayesian optimization process based on the historical data according to a preset period to update the intrinsic parameters (please refer to...). Figure 2 The target policy generator is updated using a Bayesian parameter tuning unit and a fuzzy controller that updates the intrinsic parameters of the fuzzy rule base (see S110-S150).

[0118] During the training process of the intelligent agent in an unmanned swarm, each preset period (e.g.) (Each round of iterative training) performs a Bayesian optimization process once.

[0119] The Bayesian optimization process includes: a. Initialization: Set an initial value based on expert experience. ; b. Evaluation: During the training of the decision model, the main thread will evaluate the current... Substituting these values ​​into the fuzzy controller, and through training, we obtain the evaluation index of the learning outcome. ; c. Intrinsic Parameter Optimization: The Bayesian optimizer employs a Bayesian optimization algorithm (e.g., BayesOpt for Categorical and Mixed Search Spaces), based on historical data... Finding the optimal intrinsic parameter values .

[0120] This application introduces an asynchronous Bayesian optimization thread to periodically and globally tune the intrinsic parameters of the fuzzy controller, thereby dynamically adjusting the overall performance of the fuzzy controller.

[0121] In summary, this application theoretically provides an effective framework for solving multi-agent collaborative learning in non-stationary environments, promoting the cross-integration of machine intelligence with fields such as automatic control and fuzzy logic. In engineering terms, this application provides a core technology solution with high adaptability, high coordination, and high scalability for the generation of my country's future intelligent and networked unmanned combat systems, especially unmanned swarm autonomous combat capabilities, playing a crucial supporting role in enhancing new-type combat forces.

[0122] Based on the above, this application also provides a game-playing adversarial device for unmanned swarms based on dynamic meta-learning. The game-playing adversarial device for unmanned swarms based on dynamic meta-learning can be referred to in correspondence with the aforementioned game-playing adversarial method for unmanned swarms based on dynamic meta-learning.

[0123] As an example, such as Figure 6 As shown, the game-playing adversarial device for unmanned swarms based on dynamic meta-learning provided in this application includes: The meta-control module 610 is used to acquire game situation indicators of the friendly unmanned cluster during the strategy learning process. These game situation indicators characterize the game adversarial performance of the initial strategy generator of the friendly unmanned cluster. It acquires fuzzy input variables corresponding to the game situation indicators; inputs these fuzzy input variables into a fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; updates the hyperparameters based on the adjustment amount, and updates the initial strategy generator based on the updated hyperparameters to obtain the target strategy generator. The meta-control module 610 monitors the adversarial situation in real time, calculates game situation indicators, and dynamically adjusts key hyperparameters of strategy learning (such as learning rate and exploration coefficient) through fuzzy inference, forming a feedback loop that enables the learning process to adapt to dynamic changes in the environment.

[0124] The strategy learning module 620 is used to generate a game-playing strategy for the unmanned cluster based on the target strategy generator.

[0125] This application's embodiments introduce a fuzzy controller as the controller for the policy reinforcement learning process. This controller does not directly output actions, but dynamically adjusts the hyperparameters of the underlying policy generator based on the reinforcement learning algorithm by analyzing real-time game situation indicators online. This injects experience-based online meta-learning capabilities into the unmanned swarm, enabling it to automatically adjust the learning process according to the game adversarial effect. Thus, when faced with sudden changes in the opponent's strategy, it can actively explore new strategies, achieving a leap from passive adaptation to active evolution. This solves the problems of policy oscillation, convergence to suboptimal solutions, or even complete failure to converge caused by environmental non-stationarity.

[0126] In one possible implementation, the game-playing adversarial device also includes: The environment interaction module 630 is used to obtain heterogeneous observation information of neighboring intelligent agents around each unmanned intelligent agent in the self-owned unmanned cluster.

[0127] The information processing module 640 is used to generate a joint state representation of the unmanned intelligent agent based on the heterogeneous observation information.

[0128] Accordingly, the strategy learning module 620 is also used to input the joint state representation into the target strategy generator of the unmanned intelligent agent to obtain the game adversarial strategy of the unmanned intelligent agent output by the target strategy generator.

[0129] Specifically, the agents of both unmanned swarms perform actions in simulated or real environments, generating new state and local observation information. The environment interaction module 630 captures the state and local observation information of neighboring agents around its own unmanned agent, forming heterogeneous observation information. The information processing module 640 processes this heterogeneous observation information, fusing the state and local observation information of neighboring agents with communication and coordination information to obtain a joint state representation of each unmanned agent in the swarm. The policy learning module 620 then learns from this joint representation. The target policy generator is trained centrally using the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm to generate actions for each unmanned agent. : Thus, the game strategy of each unmanned intelligent agent can be obtained. All unmanned intelligent agents in the unmanned swarm execute their respective actions in a distributed manner, jointly completing the game-like adversarial task.

[0130] In this embodiment, the heterogeneous observation information obtained by the unmanned intelligent agent is processed before policy generation to obtain a joint state representation, ensuring that the input information of the target policy generator is comprehensive, thereby improving the output accuracy of the target policy generator.

[0131] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a game-playing adversarial method for unmanned swarms based on dynamic meta-learning. This method includes: The game situation indicators of the player's unmanned cluster during the policy learning process are obtained, and the game situation indicators characterize the game adversarial performance of the player's unmanned cluster's initial policy generator. Obtain the fuzzy input variables corresponding to the game situation indicators; The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; The target strategy generator generates a game strategy for the player's unmanned cluster.

[0132] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the game-playing adversarial method for unmanned swarms based on dynamic meta-learning provided in the above embodiments. The method includes: The game situation indicators of the player's unmanned cluster during the policy learning process are obtained, and the game situation indicators characterize the game adversarial performance of the player's unmanned cluster's initial policy generator. Obtain the fuzzy input variables corresponding to the game situation indicators; The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; The target strategy generator generates a game strategy for the player's unmanned cluster.

[0134] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the game-playing adversarial method for unmanned swarms based on dynamic meta-learning provided in the above embodiments, the method comprising: The game situation indicators of the player's unmanned cluster during the policy learning process are obtained, and the game situation indicators characterize the game adversarial performance of the player's unmanned cluster's initial policy generator. Obtain the fuzzy input variables corresponding to the game situation indicators; The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; The target strategy generator generates a game strategy for the player's unmanned cluster.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A game-theoretic adversarial method for unmanned swarms based on dynamic meta-learning, characterized in that, include: The game situation indicators of the player's unmanned cluster during the policy learning process are obtained, and the game situation indicators characterize the game adversarial performance of the player's unmanned cluster's initial policy generator. Obtain the fuzzy input variables corresponding to the game situation indicators; The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; Generate a game strategy for the friendly unmanned cluster based on the target strategy generator; Among them, obtaining game situation indicators of one's own unmanned cluster during the policy learning process includes at least one of the following: The advantage deviation index of the unmanned swarm is determined based on the historical reward deviation of the unmanned swarm during the strategy learning process. The policy stability index of the unmanned cluster is determined based on the parameter changes of the target policy generator of the unmanned cluster. The policy stability index represents the variance of the network parameter matrix of the target policy generator of the unmanned cluster before and after the update. The strategy novelty index of the adversary unmanned swarm is determined based on the prediction error of the adversary unmanned swarm's behavior, wherein the prediction error is the prediction error corresponding to the current behavior of the adversary unmanned swarm obtained based on the joint action prediction of the adversary unmanned swarm over the past H steps. The strategy learning process includes: Collect historical data pairs, which include the intrinsic parameters of the fuzzy rule base and the learning outcome evaluation indicators corresponding to the intrinsic parameters; According to a preset cycle, a Bayesian optimization process is performed based on the historical data to update the intrinsic parameters, which include the relative position parameters of the center point and the width factor corresponding to each input and output variable of the fuzzy controller.

2. The game-playing adversarial method for unmanned swarms based on dynamic meta-learning according to claim 1, characterized in that, The process of generating a game-playing strategy for the unmanned cluster based on the target strategy generator includes: For each unmanned agent in the self-owned unmanned cluster, a joint state representation of the unmanned agent is generated by utilizing the heterogeneous observation information of the neighboring agents around the unmanned agent. The joint state representation is input into the target policy generator of the unmanned agent to obtain the game adversarial strategy of the unmanned agent output by the target policy generator.

3. The game-playing adversarial method for unmanned swarms based on dynamic meta-learning according to claim 2, characterized in that, The step of generating a joint state representation of the unmanned intelligent agent by utilizing heterogeneous observation information from neighboring intelligent agents around the unmanned intelligent agent specifically includes: Based on the heterogeneous observation information of all neighboring intelligent agents around the unmanned intelligent agent, an individual observation representation of the unmanned intelligent agent is generated. Based on the individual observation representations, a sparse communication topology for the unmanned intelligent agent is constructed at each time step. Generate the group interaction representation of the unmanned agent at the time step based on the heterogeneous observation information of the neighboring agents in the sparse communication topology; By integrating the individual observation representation and the group interaction representation, a joint state representation of the unmanned intelligent agent at the time step is generated.

4. The game-playing adversarial method for unmanned swarms based on dynamic meta-learning according to claim 3, characterized in that, The process of generating an individual observation representation of the unmanned intelligent agent based on heterogeneous observation information from all neighboring intelligent agents around the unmanned intelligent agent includes: Determine the entity categories of the unmanned intelligent agent and all neighboring entities around the unmanned intelligent agent, the neighboring entities including the unmanned intelligent agent and objects; For each entity category, the heterogeneous observation information of neighboring entities within the entity category is aggregated to obtain an intra-category aggregated feature representation; A global weight for each entity category is generated based on the intra-class aggregated feature representation of all entity categories. The individual observation representation of the unmanned intelligent agent is determined based on the intra-class aggregated feature representation and global weights of all the entity categories.

5. The game-playing adversarial method for unmanned swarms based on dynamic meta-learning according to claim 3, characterized in that, The construction of the sparse communication topology of the unmanned intelligent agent at each time step based on the individual observation representation includes: Based on the historical interaction information of the communication links between the unmanned intelligent agent and the neighboring intelligent agents, a communication value score is generated for each communication link; Differential sampling is performed on the communication value score to obtain the binary communication mask between the unmanned intelligent agent and the neighboring intelligent agent; The retained neighbor agents are determined based on the value of the binary communication mask; The sparse communication topology is constructed based on the communication links between all the retained neighboring agents and the unmanned agent.

6. The game-playing adversarial method for unmanned swarms based on dynamic meta-learning according to claim 1, characterized in that, The strategy learning process includes: Select and load a strategy model for the adversary unmanned cluster from the strategy model library; Control the friendly unmanned cluster to conduct asymmetric adversarial training with the opponent's unmanned cluster, and update the target policy generator of all unmanned agents in the friendly unmanned cluster based on the game situation index during the asymmetric adversarial training process. When the preset update conditions are met, a new strategy model is constructed based on all target strategy generators of the self-owned unmanned cluster, the new strategy model is added to the strategy model library, and a new strategy model is selected and loaded for the opponent's unmanned cluster.

7. A game-playing adversarial device for unmanned swarms based on dynamic meta-learning, characterized in that, include; The meta-control module is used to obtain the game situation indicators of the friendly unmanned cluster during the strategy learning process. The game situation indicators characterize the game adversarial performance of the initial strategy generator of the friendly unmanned cluster; and to obtain the fuzzy input variables corresponding to the game situation indicators. The fuzzy input variables are input into the fuzzy controller to obtain the adjustment amount of the hyperparameters of the initial strategy generator output by the fuzzy controller; The hyperparameters are updated based on the adjustment amount, and the initial policy generator is updated based on the updated hyperparameters to obtain the target policy generator; The strategy learning module is used to generate a game-playing strategy for the friendly unmanned cluster based on the target strategy generator. Among them, obtaining game situation indicators of one's own unmanned cluster during the policy learning process includes at least one of the following: The advantage deviation index of the unmanned swarm is determined based on the historical reward deviation of the unmanned swarm during the strategy learning process. The policy stability index of the unmanned cluster is determined based on the parameter changes of the target policy generator of the unmanned cluster. The policy stability index represents the variance of the network parameter matrix of the target policy generator of the unmanned cluster before and after the update. The strategy novelty index of the adversary unmanned swarm is determined based on the prediction error of the adversary unmanned swarm's behavior, wherein the prediction error is the prediction error corresponding to the current behavior of the adversary unmanned swarm obtained based on the joint action prediction of the adversary unmanned swarm over the past H steps. The strategy learning process includes: Collect historical data pairs, which include the intrinsic parameters of the fuzzy rule base and the learning outcome evaluation indicators corresponding to the intrinsic parameters; According to a preset cycle, a Bayesian optimization process is performed based on the historical data to update the intrinsic parameters, which include the relative position parameters of the center point and the width factor corresponding to each input and output variable of the fuzzy controller.

8. The unmanned swarm game-playing adversarial device according to claim 7, characterized in that, Also includes: The environment interaction module is used to acquire heterogeneous observation information of neighboring intelligent agents around each unmanned intelligent agent in the self-owned unmanned cluster; The information processing module is used to generate a joint state representation of the unmanned intelligent agent based on the heterogeneous observation information; Accordingly, the strategy learning module is also used to input the joint state representation into the target policy generator of the unmanned agent to obtain the game adversarial strategy of the unmanned agent output by the target policy generator.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the game-playing adversarial method for unmanned clusters based on dynamic meta-learning as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by the processor, it implements the game-playing adversarial method for unmanned clusters based on dynamic meta-learning as described in any one of claims 1 to 6.

11. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the game-playing adversarial method for unmanned clusters based on dynamic meta-learning as described in any one of claims 1 to 6.

Images