Heterogeneous unmanned cluster game decision method and device

Abstract

The application provides a heterogeneous unmanned cluster game decision method and device, belonging to the technical field of data processing, comprising: obtaining local observation information of each first-camp agent in a heterogeneous unmanned cluster in a multi-agent interaction scene, and a historical observation sequence of each agent in the multi-agent interaction scene; based on the historical observation sequence, performing behavior intention prediction on each agent through an intention prediction network to obtain an intention prediction vector of each agent; splicing the local observation information and the intention prediction vector to construct a node feature of each agent, and performing hierarchical attention feature aggregation through a hierarchical attention network to obtain a decision feature vector; wherein the hierarchical attention feature aggregation comprises same-camp feature aggregation, cross-camp feature aggregation and global feature aggregation performed in sequence; based on the decision feature vector, generating an action strategy of each first-camp agent through a strategy network to control each first-camp agent to perform a corresponding action.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a heterogeneous unmanned cluster game decision-making method and apparatus. Background Technology

[0002] With the development of intelligent swarm warfare technology, heterogeneous unmanned swarms composed of different types of unmanned platforms have shown broad application potential in game-theoretic scenarios such as area denial and key point capture. In highly dynamic red-blue team battles and other complex multi-agent interaction scenarios, the situation changes rapidly, placing extremely high demands on the swarm's collaborative decision-making capabilities.

[0003] Currently, multi-agent reinforcement learning methods are widely used to solve collaborative decision-making problems in swarms. However, heterogeneous unmanned swarms typically include multiple platform types, and adversarial scenarios simultaneously involve highly dynamic intra-faction cooperation and cross-faction game interactions.

[0004] Existing intelligent decision-making methods struggle to accurately grasp the complex and ever-changing environmental states and the deep dynamic interaction patterns of multiple individuals when dealing with large-scale, complex heterogeneous multi-agent game environments. This results in generated collaborative strategies that often suffer from slow response, rigidity, and low collaborative efficiency, failing to meet the practical requirements of heterogeneous unmanned swarms for high adaptability and global collaborative performance in highly adversarial environments. Summary of the Invention

[0005] This invention provides a heterogeneous unmanned swarm game decision-making method and apparatus to address the shortcomings of existing collaborative strategies, such as slow response, rigidity, and low collaborative efficiency, which cannot meet the practical requirements of heterogeneous unmanned swarms for high adaptability and global collaborative performance in highly adversarial environments.

[0006] This invention provides a heterogeneous unmanned cluster game decision-making method, applied to the execution phase, including: The local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario is obtained, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes the first-faction intelligent agent and the second-faction intelligent agent, and the heterogeneous unmanned cluster is composed of the first-faction intelligent agent and includes intelligent agents of at least two different platform types. Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intentions of each agent to obtain the intention prediction vector of each agent. The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; Based on the decision feature vector, the action policy of each first-faction agent is generated through the policy network, and the first-faction agents are controlled to perform corresponding actions.

[0007] According to a heterogeneous unmanned swarm game decision-making method provided by the present invention, the intention prediction network includes a local feature encoder, a temporal fusion network, and an intention output network; the step of predicting the behavioral intentions of each agent through the intention prediction network includes: Each frame of observation information from the historical observation sequence of each agent is input into a local feature encoder corresponding to the platform type of the agent for encoding to obtain an observation feature vector sequence; wherein, agents of the same platform type share the network parameters of the local feature encoder, while agents of different platform types use local feature encoders with independent network parameters. The observed feature vector sequence is concatenated with the intention prediction vector from the previous time step and input into the temporal fusion network to obtain the hidden state of each agent. Based on the hidden state, the association weights between each agent are calculated, and the hidden state is weighted and aggregated to obtain the context vector. The hidden state is concatenated with the context vector, and the intent prediction vector is obtained through the intent output network.

[0008] According to the heterogeneous unmanned cluster game decision-making method provided by the present invention, the intention prediction network is pre-trained through an intention training phase including a composite loss function, wherein the composite loss function includes: First-faction intention prediction loss: The real actions of the first-faction agents are mapped to real intention vectors through an action intention encoder, and the mean square error between the intention prediction vector and the real intention vector is calculated. Second camp intention prediction loss: The intention prediction vector of the second camp agent is concatenated with its current observation information, and the next observation of the second camp agent is predicted through a forward dynamical network. The error between the predicted value and the true value is calculated. The composite loss function is a weighted sum of the intention prediction loss of the first camp and the intention prediction loss of the second camp.

[0009] According to the heterogeneous unmanned cluster game decision-making method provided by the present invention, the aggregation of same-faction features in the hierarchical attention network includes: The cooperation index is calculated based on the distance between the first-faction neighboring agents and the current agent, and a preset number of first-faction neighboring agents are selected based on the cooperation index. Calculate the same-faction attention coefficient between the current agent and the first-faction neighboring agents after filtering; wherein, multiple sets of independent same-faction attention weight parameters are pre-set, and the same-faction attention coefficient is calculated by calling the same-faction attention weight parameters corresponding to the platform type combination of the current agent and the first-faction neighboring agents. Based on the same-faction attention coefficient, the node features of the selected first-faction neighboring agents are weighted and aggregated to obtain the same-faction aggregated features.

[0010] According to the heterogeneous unmanned cluster game decision-making method provided by the present invention, the cross-faction feature aggregation in the hierarchical attention network includes: The threat index is calculated based on the distance between the second-faction agent and the current agent, and a preset number of second-faction agents are selected based on the threat index. Calculate the cross-faction attention coefficient between the same-faction aggregation features of the current agent and the node features of the filtered second-faction agents; wherein, multiple sets of independent cross-faction attention weight parameters are pre-set, and the cross-faction attention coefficient is calculated by calling the cross-faction attention weight parameters corresponding to the platform type combination of the current agent and the second-faction agents. Based on the cross-faction attention coefficient, the same-faction aggregation features of the current agent are weighted and aggregated based on a specific learnable parameter matrix to obtain cross-faction aggregation features; The global feature aggregation includes: concatenating the same-faction aggregated features and the cross-faction aggregated features and obtaining pre-aggregated features through a fusion network; weighting and aggregating the pre-aggregated features based on global attention coefficients to obtain global aggregated features; and concatenating the node features of the current agent, the same-faction aggregated features, the cross-faction aggregated features, and the global aggregated features to obtain the decision feature vector.

[0011] According to the heterogeneous unmanned cluster game decision-making method provided by the present invention, the policy network is pre-trained using a reinforcement learning algorithm, and an adaptive exploration strategy is adopted in the policy training phase pre-trained by the policy network, including: A policy entropy regularization term is introduced into the policy optimization objective function, and a learnable temperature control coefficient is set. The temperature control coefficient of each first-faction intelligent agent platform type is set as a learnable variable associated with the uncertainty of intent prediction output by the intent prediction network. The target entropy value is determined based on the dimension of the agent's action space; Based on the difference between the current policy entropy and the target entropy value, the temperature control coefficient is updated by gradient descent; wherein, when the current policy entropy is lower than the target entropy value or the uncertainty of intention prediction increases, the temperature control coefficient is increased, and when the current policy entropy is higher than the target entropy value or the certainty of intention prediction increases, the temperature control coefficient is decreased.

[0012] According to the heterogeneous unmanned cluster game decision-making method provided by the present invention, the intention prediction network, the hierarchical attention network and the policy network are obtained by training in a paradigm of centralized training and decentralized execution in a pre-existing centralized training phase. Among them, agents of the same platform type share the parameters of the policy network and the corresponding value network, while agents of different platform types use policy networks and corresponding value networks with independent parameters. During the centralized training phase, the value network utilizes global state information for optimization and updates to assist in the training of the policy network. During the execution phase, each first-faction agent generates an action strategy based solely on the local observation information and the historical observation sequence obtained through communication.

[0013] This invention also provides a heterogeneous unmanned cluster game decision-making device, applied in the execution phase, characterized in that it includes: The acquisition module is used to acquire local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes the first-faction intelligent agent and the second-faction intelligent agent, and the heterogeneous unmanned cluster is composed of the first-faction intelligent agents and includes intelligent agents of at least two different platform types. The prediction module is used to predict the behavioral intentions of each agent based on the historical observation sequence through an intention prediction network, and obtain the intention prediction vector of each agent. The construction module is used to concatenate the local observation information with the intention prediction vector to construct the node features of each agent, and to obtain the decision feature vector by performing hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequentially executed same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; The control module is used to generate action policies for each first-faction intelligent agent through a policy network based on the decision feature vector, and to control each first-faction intelligent agent to execute corresponding actions.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous unmanned cluster game decision-making method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the heterogeneous unmanned cluster game decision-making method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the heterogeneous unmanned cluster game decision-making method as described above.

[0017] This invention provides a heterogeneous unmanned swarm game decision-making method and apparatus. It uses an intent prediction network to predict the behavioral intentions of all agents in a scenario based on historical observation sequences. The obtained intent prediction vectors are then concatenated with local observation information to construct node features that integrate intent semantics. This allows each agent to make decisions not only based on the current situation but also to proactively consider the action trends of all parties. Furthermore, a hierarchical attention network is used to sequentially perform intra-faction feature aggregation, cross-faction feature aggregation, and global feature aggregation. This process extracts intra-faction collaborative relationship features and cross-faction adversarial relationship features from the node features that integrate intent information in a hierarchical and targeted manner, and then globally integrates them to obtain a high-quality decision feature vector that combines intent perception capabilities with multi-level relationship discrimination. This enables the policy network to generate more forward-looking and collaborative action strategies based on this decision feature vector, effectively improving the collaborative decision-making quality and overall adversarial effectiveness of heterogeneous unmanned swarms in dynamic game adversarial environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the heterogeneous unmanned cluster game decision-making method provided by the present invention; Figure 2 This is a technical architecture diagram of the heterogeneous unmanned cluster game decision-making method provided by the present invention; Figure 3 A schematic diagram of the heterogeneous unmanned cluster game decision-making device provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] The heterogeneous unmanned cluster game decision-making method provided in this embodiment of the invention can be executed by a computing device with data processing capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), embedded processor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC) or other hardware platform with computing capabilities, or a software system, control system or decision-making system deployed on the above hardware platform.

[0022] In one specific implementation, the executing entity is a decision processor deployed in the cluster command and control system. This decision processor is pre-loaded with trained neural network model parameters, used to generate action strategies for each agent in the heterogeneous unmanned cluster in real time during the execution phase. In the following description of the embodiments, this decision processor will be used as the executing entity.

[0023] Figure 1 This is a flowchart illustrating the heterogeneous unmanned cluster game decision-making method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step S110: Obtain local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes first-faction intelligent agents and second-faction intelligent agents, and the heterogeneous unmanned cluster is composed of first-faction intelligent agents and includes intelligent agents of at least two different platform types. In this step, the multi-agent interaction scenario refers to a scenario in which multiple agents with autonomous decision-making capabilities interact within the same environment. This scenario can be a military confrontation scenario, a competitive game scenario, a multi-robot cooperative confrontation scenario, or other application scenarios involving multi-agent interaction. In one specific implementation, the multi-agent interaction scenario is a heterogeneous unmanned swarm close-range combat game scenario, in which multiple unmanned platforms from two camps engage in combat within a certain area.

[0024] In multi-agent interaction scenarios, the agents are categorized into two groups: first-team agents and second-team agents. The first-team agents refer to the agents controlled by the method of this invention that need to learn decision-making strategies, i.e., the red team agents.

[0025] The second-party intelligent agent refers to the opposing intelligent agent in the environment that is fighting against the first party, namely the blue party intelligent agent. Its strategy is preset or unknown during the execution phase.

[0026] In one specific implementation, the first camp of intelligent agents is the red team intelligent agent (learner), and the second camp of intelligent agents is the blue team intelligent agent (environment).

[0027] The set of red (learning) agents is denoted as... ( (where the total number of red agents is denoted as ), and the set of blue (environment) agents is denoted as . ( (where the total number of agents on the blue team is ), and the set of all agents is . .

[0028] The heterogeneous unmanned swarm consists of agents from the first camp and includes agents from at least two different platform types. The term "heterogeneous" here refers to the differences in physical configuration, functional attributes, and mobility among the agents in the swarm.

[0029] Each agent has a type identifier. This represents the physical and functional attributes of the intelligent agent, namely... This can be represented as a tuple (platform type, role), such as (drone, reconnaissance), (unmanned vehicle, attack), (drone, support), etc. Users can customize the type of intelligent agent according to their needs. Different types of intelligent agents have different observation spaces. Action space And state transition dynamics.

[0030] Local observation information refers to the environmental perception information obtained by each first-tier intelligent agent at the current time step based on its own sensor capabilities.

[0031] Each agent At time step Obtain a local observation , It is a vector consisting of its own state (position, velocity, health), relative information of the agent within the observation radius (relative distance, relative orientation, agent type and health), and relative distance to obstacles in the environment.

[0032] Historical observation sequence refers to the sequence of observation information of each agent within a continuous time window before the current time step.

[0033] For any intelligent agent in the scene Its historical observation sequence fragment at the current time step t can be represented as ( (The historical length is used to determine the length of the historical data). For agents in the first tier, their historical observation sequences can be directly retrieved from local storage; for other agents... Historical observation sequence This information can be obtained through communication links within the cluster. Furthermore, the predicted intentions of other agents at the previous time step can also be obtained through these communication links. .

[0034] Step S120: Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intention of each agent to obtain the intention prediction vector of each agent.

[0035] In this step, the intent prediction network is a pre-trained neural network model used to infer the agent's behavioral intent.

[0036] In this invention, behavioral intent is defined as a low-dimensional continuous vector that encodes the tactical objective or action pattern that the agent plans to achieve in the near future. The intent prediction vector is the predicted output value of the intent prediction network for the behavioral intent of each agent, denoted as [vector]. .

[0037] For the red team's intelligent agent Its intention is to predict the network The input includes fragments of its own historical observation sequence. ( (History length), other intelligent agents obtained from the communication link Historical observation sequence The predicted intentions of other agents at the previous time step obtained from the communication link. Intent to predict network The output is the current time step. Predicted intentions of all agents .

[0038] Intent prediction networks can utilize recurrent neural networks, long short-term memory networks, gated recurrent units, or attention-based temporal models to capture temporal dependencies in historical observation sequences.

[0039] Intent prediction networks can also include inter-agent information exchange mechanisms to leverage the relationships between multiple agents to improve the accuracy of intent prediction. The intent prediction network is shared by all first-tier agents, enabling each agent to trigger and infer the global intent picture from its own perspective.

[0040] Step S130: The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation. In this step, each agent Features Encoded by its observation and the prediction of its own intention It is pieced together, that is .

[0041] Here, concatenation refers to cascading the feature vectors corresponding to local observation information and the intention prediction vectors in the feature dimension to form a new, higher-dimensional feature vector as the node feature.

[0042] After constructing node features, the node features of each agent are input into a hierarchical attention network for hierarchical attention feature aggregation. The hierarchical attention network aims to construct a dynamic graph structure for each first-team agent and use a hierarchical attention mechanism to extract the most critical features for the current decision from complex enemy-ally interactions.

[0043] Layered attention feature aggregation includes three levels of feature aggregation operations executed sequentially: same-faction feature aggregation, cross-faction feature aggregation, and global feature aggregation.

[0044] The goal of homologous feature aggregation is to enable agents in the first camp to promote cooperation by capturing features of friendly agents. At this level, for the current agent in the first camp, cooperation-related features are extracted from friendly neighboring agents in the same camp based on an attention mechanism to obtain homologous aggregated features.

[0045] Cross-faction feature aggregation is used to aggregate features from agents across different factions. At this level, for the current first-faction agent, adversarial-related features are extracted from the second-faction agent of the opposing faction based on an attention mechanism to obtain cross-faction aggregated features.

[0046] Global feature aggregation is used to further fuse intra-faction and cross-faction aggregated features at a higher level. At this level, global aggregated features are obtained through attention-based feature fusion operations.

[0047] By aggregating the above three layers of attention features, the final decision feature vector can be obtained:

[0048] in, The node characteristics of the current agent, It is a feature of aggregation within the same faction. It is a cross-faction aggregation feature. For global aggregation features, For the multilayer perceptron network used in the final feature fusion, For intelligent agents j At time step t The final decision feature vector.

[0049] The decision feature vector fully integrates the agent's own features, cooperative features within the same faction, adversarial features across factions, and global situational features, providing a high-quality feature representation for the subsequent generation of action strategies.

[0050] Step S140: Based on the decision feature vector, generate the action policy of each first-faction agent through the policy network, and control each first-faction agent to execute the corresponding action.

[0051] In this step, the policy network is a pre-trained neural network model used to map decision feature vectors to action policies.

[0052] The policy network is a multilayer perceptron. Agents of the same type share the same policy network parameters, and their actions are as follows: ; Policy networks use decision feature vectors As input, the network output is mapped to a probability distribution in the action space through the Softmax function, and each first-team agent selects the action to be executed at the current time step according to the probability distribution.

[0053] in, For intelligent agents i The parameterized strategy, where θ represents the parameters of the policy network. For intelligent agents i The action, To work with intelligent agents i The policy network (multilayer perceptron) corresponding to the platform type is a normalized exponential function, which is used to map the network output to a probability distribution on the action space.

[0054] Actions can include, but are not limited to: movement direction control, speed control, attack commands, reconnaissance commands, etc. The specific content of the actions is determined by the platform type and action space definition of the agent.

[0055] Within the reinforcement learning framework, the red team's intelligent agent The strategy is a parameterized probability distribution, denoted as ,in These are the parameters of the policy network. This refers to the internal memory state of the intelligent agent; the policy network outputs actions based on current observations and historical memories. (Blue agent) The strategy is preset and remains unchanged in the simulation.

[0056] This invention realizes the game-theoretic decision-making process of heterogeneous unmanned swarms during the execution phase. By using an intent prediction network to predict the behavioral intentions of each agent in the scenario, agents in the first camp can perceive the cooperative intentions of their teammates and the tactical intentions of their opponents, providing forward-looking high-level semantic information for decision-making. By concatenating local observation information with intent prediction vectors to construct node features, effective fusion of perceptual and intent information is achieved. Through a three-layer attention aggregation mechanism of hierarchical attention networks (same-camp, cross-camp, and global), the most relevant features to the current decision are extracted accurately and efficiently from complex enemy-ally interactions, effectively solving the problems of high relationship complexity and the difficulty of capturing both cooperative and adversarial relationships simultaneously in heterogeneous multi-agent scenarios. Finally, an action policy is generated through a policy network to control the agents in the heterogeneous unmanned swarm to execute cooperative actions, achieving efficient swarm game-theoretic decision-making.

[0057] Optionally, the intent prediction network includes a local feature encoder, a temporal fusion network, and an intent output network; the intent prediction network predicts the behavioral intent of each agent, including: Each frame of observation information from the historical observation sequence of each agent is input into a local feature encoder corresponding to the platform type of the agent for encoding to obtain an observation feature vector sequence; agents of the same platform type share the network parameters of the local feature encoder, while agents of different platform types use local feature encoders with independent network parameters. The observed feature vector sequence is concatenated with the intention prediction vector from the previous time step and input into the temporal fusion network to obtain the hidden state of each agent. Based on the hidden state, the association weights between each agent are calculated, and the hidden states are weighted and aggregated to obtain the context vector. The hidden state is concatenated with the context vector, and the intent prediction vector is obtained through the intent output network.

[0058] In this application, the local feature encoder is set according to the following rules: agents of the same platform type share the network parameters of the local feature encoder, while agents of different platform types use local feature encoders with independent network parameters.

[0059] Observations of each agent Through a type-specific multilayer perceptron MLP encoder Encode the observed feature vectors. Such encoders share parameters among agents of the same type.

[0060] For example, in a scenario involving both drones and autonomous vehicles, all drone-type agents share the same set of local feature encoder parameters, while all autonomous vehicle-type agents share a different set. This parameter-sharing strategy ensures both experience sharing and consistency of feature representations among agents of the same type, and maintains the differences in feature representations between agents of different types.

[0061] intelligent agents By performing the above encoding operation on the observation information of each frame (i.e., each time step) in the historical observation sequence, the intelligent agent can be obtained. Observation feature vector sequence

[0062] Temporal fusion networks, or temporal fusion layers, are used to fuse temporal feature information of agents and capture the temporal evolution patterns of agent behavior. Temporal fusion networks can be implemented using gated recurrent units (GRUs), long short-term memory networks (LSTMs), or other recurrent neural network structures.

[0063] For each intelligent agent , and its recent Frame observation features and intention prediction After concatenation, the input is a shared gated recurrent neural network. In the process, the hidden state that incorporates temporal information is obtained. .

[0064] Among them, for the current time step t Since the intent prediction vector for the current time step has not yet been generated, it is filled with the zero vector 0. Hidden state It is a temporal feature representation that integrates the agent's historical behavior patterns and intent information.

[0065] To predict the behavioral intent of a given agent, it is necessary to consider the behavioral information of other agents associated with that agent. Association weights are used to quantify the degree of association between different agents.

[0066] intelligent agent In order to predict The intention needs to be considered in relation to For other related intelligent agents, this invention calculates the following attention weights:

[0067] in It is an attention scoring function. , , It is a learnable parameter matrix. For time steps t intelligent agent k For intelligent agents j The association weight.

[0068] By aggregating contextual information, we can obtain:

[0069] Context vector Encoded with the target intelligent agent j The relevant global context information reflects the influence of other agents on the behavioral intentions of the target agent j. is a learnable parameter matrix.

[0070] target intelligent agent Its own sequence hidden state is combined with the context vector, through a Obtain the intention prediction value This predictor It is shared by all Red Team agents, enabling each Red Team agent to trigger from its own perspective and infer the overall intent picture.

[0071] In this application, the intent prediction network predicts the behavioral intent of each agent in a scene. The above implementation achieves differentiated encoding of observation information from heterogeneous agents through a type-specific local feature encoder, captures the temporal evolution of agent behavior through a temporal fusion network, realizes information interaction between agents through attention-based association weight calculation and context vector aggregation, and generates high-quality intent prediction vectors through an intent output network. This provides crucial high-level semantic information about teammate cooperation intent and opponent tactical intent for subsequent decision-making.

[0072] Optionally, the intent prediction network is pre-trained through an intent training phase that includes a composite loss function, which includes: First-faction intention prediction loss: Map the real actions of the first-faction agents to real intention vectors through the action intention encoder, and calculate the mean square error between the intention prediction vector and the real intention vector. Second camp intention prediction loss: The intention prediction vector of the second camp agent is concatenated with its current observation information, and the next moment observation of the second camp agent is predicted through the forward dynamics network. The error between the predicted value and the true value is calculated. The composite loss function is the weighted sum of the intention prediction loss of the first camp and the intention prediction loss of the second camp.

[0073] In this application, the intent prediction network is pre-trained through an intent training phase that includes a composite loss function. Because the information availability of the first and second camp agents in the scenario differs—the true policies of the first camp agents are known, while those of the second camp agents are unknown—this invention designs different loss calculation methods for the two types of agents and combines them into a composite loss function for training the intent prediction network. The composite loss function includes the first camp intent prediction loss and the second camp intent prediction loss.

[0074] Regarding the first camp's intention to predict losses: For the red team agent (teammate), the actual actions of the red team agent during the training phase As is known, this invention uses an MLP-based action intent encoder. Real action Mapped to intent vector The supervised loss intended for prediction is the predicted value. Compared with the true value Mean square error between ;in, It represents the square of the 2-norm of a vector.

[0075] Through this supervised learning approach, the intention prediction network can learn to map the historical behavioral patterns of the first-tier agents into accurate representations of behavioral intentions.

[0076] Regarding the second camp's intention to predict losses: For the blue-side agent (opponent), since the blue-side agent's true policy parameters (i.e., true intent) cannot be obtained, this invention employs a self-supervised learning approach, utilizing the predicted blue-side agent intent. The current state of the blue team is obtained through an MLP network that represents the forward dynamics. To predict the next moment's observations of the blue agent The loss function is the difference between the predicted observation and the actual observation at the next time step. Error between .

[0077] The design idea behind this loss is that if the intent prediction vector can accurately encode the behavioral intent of the second-party agent, then, combined with its current state information, it should be able to accurately predict its state at the next moment. Therefore, by minimizing the prediction error of the next moment's observation, the intent prediction network can indirectly learn a meaningful representation of the second-party agent's behavioral intent.

[0078] The composite loss function is the weighted sum of the intention prediction loss of the first camp and the intention prediction loss of the second camp.

[0079] The total loss function of the network intended for prediction is then: ,in and These are adjustable hyperparameters. They are used to balance the relative weights of the two losses. By adjusting the values ​​of these two hyperparameters, the emphasis of the intention prediction network on learning the intention prediction of both first-team and second-team agents can be controlled.

[0080] Through training with the aforementioned composite loss function, the intention prediction network can simultaneously learn to predict the cooperative intentions of friendly agents and the tactical intentions of opposing agents. Furthermore, different training strategies are adopted for the two types of agents, which are in accordance with their respective information conditions. This fully utilizes available supervisory information and self-supervised signals, enabling the intention prediction network to provide high-quality intention prediction results for subsequent game decisions.

[0081] Optionally, the aggregation of homologous features in a hierarchical attention network includes: The cooperation index is calculated based on the distance between the first-tier neighboring agents and the current agent, and a preset number of first-tier neighboring agents are selected based on the cooperation index. Calculate the same-faction attention coefficient between the current agent and the first-faction neighboring agents after filtering; wherein, multiple sets of independent same-faction attention weight parameters are pre-set, and the same-faction attention coefficient is calculated by calling the same-faction attention weight parameters corresponding to the platform type combination of the current agent and the first-faction neighboring agents. Based on the same-faction attention coefficient, the node features of the first-faction neighboring agents after screening are weighted and aggregated to obtain the same-faction aggregated features.

[0082] In this application, the first-faction neighboring agent refers to other agents that belong to the same faction (i.e., the first faction) as the current agent and are within the communication range of the current agent.

[0083] For the red team's intelligent agent Within its communication range Friendly forces within constitute a neighborhood set ,in and Representing intelligent agents respectively and The position vector.

[0084] The cooperation metric is a quantitative indicator used to measure the potential for cooperation between a current agent and its leading neighboring agents.

[0085] To prevent the amount of information requiring aggregation from increasing with the number of agents, this invention first employs a hard attention mechanism to filter agents with strong cooperative tendencies. The degree of cooperation between agents is defined as... Then you can All friendly agents according to Sort the sizes from largest to smallest, then select... The largest value Each agent is a set of aggregated features. This greatly simplifies the subsequent computational burden.

[0086] Preset quantity K This selection mechanism can be configured according to the specific needs of the scenario and available computing resources. It constitutes a hard attention mechanism, which, by limiting the number of neighboring agents participating in feature aggregation, effectively controls computational complexity while ensuring the quality of the aggregated information.

[0087] It should be noted that the calculation method for the cooperation index is not limited to the above-mentioned inverse distance form. Other measurement methods based on factors such as the consistency of angle and speed direction, and functional complementarity can also be used.

[0088] Calculate the same-camp attention coefficient between the current agent and each of the selected first-camp neighbor agents to quantify the information contribution of each neighbor agent to the current agent.

[0089] This invention fully considers the differences in relationships between agents of different platform types in heterogeneous clusters when calculating the attention coefficient of agents within the same camp.

[0090] Specifically, multiple independent sets of same-faction attention weight parameters are pre-set. Based on the platform type combination of the current agent and the first-faction neighboring agents, the same-faction attention weight parameters corresponding to the platform type combination are called to calculate the same-faction attention coefficient.

[0091] Computable intelligent agents With intelligent agents The attention coefficient between them, if and For agents of the same type, the attention coefficient is:

[0092] in , , Let be a learnable parameter matrix, if and For agents that are not of the same type, the attention coefficient is:

[0093] in , , This is the learnable parameter matrix. Attention coefficients are normalized using softmax.

[0094] After weighted aggregation, the aggregation characteristics of the same faction are obtained:

[0095] If in the above formula and If they belong to the same type ,otherwise By designing different learnable parameter matrices for agents from different factions, the relationships between agents can be distinguished more precisely.

[0096] The same-faction aggregation feature encodes the collaborative information that the current agent obtains from friendly neighboring agents, reflecting the collaborative situation between the current agent and its surrounding friendly forces.

[0097] In this application, the homologous feature aggregation in the hierarchical attention network achieves selective aggregation of information about friendly agents. Homologous feature aggregation effectively controls computational complexity through cooperation index filtering, and achieves fine-grained modeling of differentiated relationships between heterogeneous agents through multiple sets of independent attention weight parameters based on platform type combinations. This enables homologous feature aggregation to more accurately capture collaborative relationships within heterogeneous clusters.

[0098] Optionally, cross-faction feature aggregation in hierarchical attention networks includes: The threat index is calculated based on the distance between the second-faction agents and the current agent, and a preset number of second-faction agents are selected based on the threat index. Calculate the cross-faction attention coefficient between the same-faction aggregation features of the current agent and the node features of the filtered second-faction agents; wherein, multiple sets of independent cross-faction attention weight parameters are pre-set, and the cross-faction attention coefficient is calculated by calling the cross-faction attention weight parameters corresponding to the platform type combination of the current agent and the second-faction agents. Based on the cross-faction attention coefficient, the same-faction aggregated features of the current agent are weighted and aggregated based on a specific learnable parameter matrix to obtain cross-faction aggregated features; Global feature aggregation includes: concatenating the aggregated features of the same faction and the aggregated features of the cross faction and obtaining pre-aggregated features through a fusion network; weighting and aggregating the pre-aggregated features based on the global attention coefficient to obtain global aggregated features; and concatenating the node features, aggregated features of the same faction, aggregated features of the cross faction and the global aggregated features of the current agent to obtain the decision feature vector.

[0099] In this application, the threat level index is used to measure the degree of threat posed by a second-faction agent to a current first-faction agent. The higher the threat level index value, the greater the threat posed by the second-faction agent to the current agent, and the more likely it should be included in the calculation of cross-faction feature aggregation.

[0100] The threat level index is similar to the cooperation level index in the feature aggregation of the same faction. Both are calculated based on the spatial distance between agents. However, the agents they target belong to different factions, and the semantics they reflect are different. The cooperation level index reflects the cooperation potential between friendly parties, while the threat level index reflects the degree of threat from the enemy.

[0101] For the red team's intelligent agent Within its observation range The enemy within constitutes the neighbor set Similarly, a hard attention mechanism is first employed to filter out adversaries with a high threat level. The threat level of an adversary agent is defined as... Then you can All enemy agents according to Sort the sizes from largest to smallest, then select... The largest value Each enemy agent is a set of features aggregated. .

[0102] It should be noted that the threat level index is not limited to the reciprocal of distance; it can also be comprehensively evaluated by combining information such as the type of enemy agent, weapon capabilities, and direction of movement.

[0103] In this step, it is important to note that in cross-faction feature aggregation, the query input used to calculate the attention coefficient is the aggregated features of the same faction for the current agent. (Instead of the original node features), the key input is the filtered node features of the second-faction agents. This design allows cross-faction feature aggregation to be built upon the aggregation of features within the same faction.

[0104] Computable intelligent agents With intelligent agents The attention coefficient between them, if and For agents of the same type, the attention coefficient is:

[0105] in , , For learnable parameter matrix, Represents intelligent agents against enemy intelligent agents splicing together the observed features and the intended prediction, T This indicates transpose. If... and For agents that are not of the same type, the attention coefficient is:

[0106] in , , This is the learnable parameter matrix. Attention coefficients are normalized using softmax.

[0107] Similar to the calculation of the same-faction attention coefficient, the calculation of the cross-faction attention coefficient also adopts multiple independent cross-faction attention weight parameters based on the combination of platform types, and calls the corresponding parameters according to the combination of platform types of the current agent and the second-faction agent.

[0108] After weighted aggregation, the cross-faction aggregation characteristics are obtained:

[0109] If in the above formula and If they belong to the same type ,otherwise By designing different learnable parameter matrices for agents from different factions, the relationships between agents can be distinguished more precisely.

[0110] In this application, the cross-faction aggregation feature encoding encodes the comprehensive assessment results of the current agent on the enemy threat information based on the friendly cooperative situation, reflecting the cross-faction confrontation situation currently faced by the agent.

[0111] In this application, each red agent can be obtained through the above two-layer attention calculation. aggregation features This feature is used for attention-based feature fusion among the red team agents. Calculate the attention coefficient:

[0112] in , , The learnable parameter matrix, after normalization, yields:

[0113] Global features can be calculated: ; in It is a learnable parameter matrix.

[0114] By aggregating the above three layers of attention features, the final decision feature vector can be obtained: ; In this application, the decision feature vector integrates information from the following four levels: the current agent's own node features. Features of same-faction aggregation Reflects information on friendly cooperation; cross-faction aggregation features Reflecting enemy confrontation information; global aggregation features It reflects the overall team situation information.

[0115] Through the aforementioned cross-faction feature aggregation and global feature aggregation, the hierarchical attention network implements a three-layer attention aggregation mechanism—entity-faction-global—which can accurately and efficiently extract multi-level features most relevant to the current decision from complex multi-agent interactions, providing sufficient information support for high-quality decision-making by the policy network. Simultaneously, by setting independent learnable parameter matrices for different platform type combinations in each layer of attention computation, it effectively and meticulously models the differentiated relationships between heterogeneous agents.

[0116] Optionally, the policy network is pre-trained using a reinforcement learning algorithm, and an adaptive exploration policy is employed during the policy training phase that the policy network undergoes, including: Introduce a policy entropy regularization term into the policy optimization objective function and set a learnable temperature control coefficient. Set the temperature control coefficient of each first-tier intelligent agent platform type as a learnable variable associated with the uncertainty of intent prediction output by the intent prediction network. The target entropy value is determined based on the dimension of the agent's action space; Based on the difference between the current policy entropy and the target entropy, the temperature control coefficient is updated through gradient descent. Specifically, when the current policy entropy is lower than the target entropy or the uncertainty of intention prediction increases, the temperature control coefficient is increased; when the current policy entropy is higher than the target entropy or the certainty of intention prediction increases, the temperature control coefficient is decreased.

[0117] In this application, the policy network is pre-trained using a reinforcement learning algorithm. This invention uses the MAPPO algorithm for model training. During the policy training phase pre-trained by the policy network, an adaptive exploration strategy is employed to address the imbalance between exploration and exploitation in adversarial environments.

[0118] The objective function for optimizing neural network parameters in the MAPPO algorithm can be defined as:

[0119] in To accumulate rewards, For joint strategy, For temperature control coefficient, Entropy is introduced to enhance the diversity of agent actions, thereby improving the exploration of optimal strategies and avoiding getting trapped in local optima. However, in the traditional MAPPO algorithm, it is used for the temperature coefficient. Its fixed nature makes it unable to adapt to highly dynamic and non-stationary environments.

[0120] In this invention, the temperature control coefficient is set as a learnable variable, rather than a fixed hyperparameter as in traditional methods. The temperature control coefficient for each first-tier agent platform type is set as a learnable variable associated with the uncertainty of intent prediction output by the intent prediction network. Intent prediction uncertainty can be measured by the output variance of the intent prediction network, historical statistics of prediction errors, or other metrics. Agents of different platform types can set their own independent temperature control coefficients to adapt to the different exploration needs faced by different types of agents.

[0121] This invention designs a learnable temperature control coefficient. To solve the above problem, a target entropy value is set. This is typically a negative action space dimension. For example, if an autonomous vehicle has five actions (accelerating forward, accelerating backward, turning left, turning right, and maintaining its motion), then... It can be set to -5. Temperature control coefficient. The loss function for self-learning is: ; in, For experience replay pool, express The observation state vectors of all agents at any given time. express Action vectors of all agents at any given time.

[0122] The design principle of this loss function is: when the current policy entropy is lower than the target entropy value, the gradient direction of the loss function makes the temperature control coefficient... Increase, thereby enhancing the incentive to explore; when the current policy entropy is higher than the target entropy value, the gradient direction of the loss function makes the temperature control coefficient... This reduces the risk of policy convergence and the use of existing experience.

[0123] Furthermore, the update of the temperature control coefficient is also related to the uncertainty of intention prediction. When the uncertainty of intention prediction increases (i.e., the adversary's intention is difficult to predict), the temperature control coefficient is increased to encourage the agent to try more different strategies to obtain more information and enhance robustness in uncertain environments; when the certainty of intention prediction increases (i.e., the adversary's intention is relatively clear), the temperature control coefficient is decreased to make the agent more inclined to use known effective strategies for targeted responses.

[0124] The temperature control coefficient is iteratively updated using the gradient descent algorithm on the aforementioned loss function to achieve adaptive adjustment of the temperature control coefficient.

[0125] By employing the aforementioned adaptive exploration strategy, this invention sets the temperature control coefficient as a learnable variable and correlates it with the uncertainty of intention prediction, thereby achieving adaptive adjustment of the exploration intensity. This allows the policy network to automatically adjust the exploration intensity based on the current policy exploration state and the degree of environmental uncertainty during training. This mechanism effectively solves the problem that traditional fixed-parameter exploration strategies cannot adapt to different training stages and the needs of facing different opponent strategies, improving the stability of adversarial training and the quality of the final strategy, and avoiding local optima caused by insufficient exploration or training non-convergence caused by over-exploration.

[0126] Optionally, the intention prediction network, the hierarchical attention network, and the policy network are trained using a paradigm of centralized training and distributed execution in a pre-experienced centralized training phase. Among them, agents of the same platform type share the parameters of the policy network and the corresponding value network, while agents of different platform types use policy networks and corresponding value networks with independent parameters. During the centralized training phase, the value network utilizes global state information for optimization and updates to assist in the training of the policy network. During the execution phase, each first-tier agent generates action strategies based solely on local observation information and historical observation sequences obtained through communication.

[0127] In this application, the intention prediction network, the hierarchical attention network, and the policy network are trained using a paradigm of Centralized Training with Decentralized Execution (CTDE) during a pre-existing centralized training phase.

[0128] The centralized training and distributed execution paradigm is a classic training paradigm in multi-agent reinforcement learning. Its core idea is that during the training phase, global state information and information from other agents can be used to assist each agent in policy optimization, while during the execution phase, each agent makes independent decisions based only on the local information it can obtain.

[0129] During the centralized training phase, the parameter sharing strategy of the policy network and the corresponding value network follows these rules: agents of the same platform type share the parameters of the policy network and the corresponding value network, while agents of different platform types use policy networks and corresponding value networks with independent parameters.

[0130] Value networks are auxiliary networks in reinforcement learning algorithms used to estimate state values ​​or state-action values. This parameter-sharing strategy allows agents of the same type to share learning experiences to improve sample efficiency, while maintaining the differences in strategies between agents of different types to adapt to their respective capabilities.

[0131] The value network can be implemented using a multilayer perceptron (MLP) architecture, which is trained in conjunction with the corresponding policy network. During the centralized training phase, the value network utilizes global state information for optimization updates to assist the policy network's training. Global state information includes complete state information for all agents in the scene, such as their positions, velocities, and health. Because the value network can access global state information during training, it can more accurately evaluate the value of the current state, thus providing more precise gradient signals for the policy network's optimization.

[0132] During the execution phase, each agent in the first camp generates its action policy based solely on local observation information and historical observation sequences obtained through communication. That is, in actual deployment, each agent no longer needs global state information; it only uses local observation information acquired by its own sensors and historical observation sequences shared with other friendly agents via cluster communication links as input to generate its own action policy through a pre-trained intent prediction network, hierarchical attention network, and policy network. This distributed execution approach allows each agent to operate independently even with limited or partially interrupted communication, exhibiting strong robustness and scalability.

[0133] It should be noted that the value network is no longer used during the execution phase, because the value network is only used to assist policy optimization during the training phase and does not participate in the action generation process during the execution phase.

[0134] The training process can employ multi-agent proximal policy optimization algorithms or other multi-agent reinforcement learning algorithms. During training, the first group of agents engages in adversarial training against a second group of agents with different built-in policies.

[0135] The second-tier agents can employ pre-defined rule-based strategies, random strategies, or strategies trained through self-play. By training against opponents with different strategies, the policy network can learn game-making strategies with a certain degree of generalization ability.

[0136] Through the aforementioned paradigm of centralized training and decentralized execution, this invention achieves the goal of fully utilizing global information to improve policy quality during the training phase and relying solely on local information to achieve decentralized decision-making during the execution phase. The parameter-sharing mechanism for agents of the same platform type improves the efficiency of training samples and the consistency of policies, while the independent parameter mechanism for agents of different platform types ensures the differentiation and specialization of policies for heterogeneous agents. This enables heterogeneous unmanned swarms to efficiently learn well-defined, coordinated cooperative game strategies.

[0137] Figure 2 This is a technical architecture diagram of the heterogeneous unmanned cluster game decision-making method provided by the present invention. (See diagram below.) Figure 2 As shown, the decision-making model of this invention consists of four core modules connected in series: an agent behavior intention prediction module, a hierarchical graph attention feature extraction module, a policy network module, and a learnable parameter adaptive optimization module. The entire system adopts a paradigm of centralized training and decentralized execution.

[0138] Taking the decision-making process of the current leading intelligent agents as an example, combined with Figure 2 The technical architecture and data flow process are as follows: Step 1 (Information Acquisition): At the current time step t, the agents in the first camp acquire local observation information through their own sensors, including their own position, velocity, survival status, and the relative distance, relative orientation, and type identification of friendly and enemy agents within the observation range. Simultaneously, they acquire the historical observation sequences of each agent over the most recent L time steps through the cluster communication link.

[0139] Step Two (Behavioral Intent Prediction): For example... Figure 2 As shown in the upper agent behavior intention prediction module, the historical observation sequence of each agent is processed through four stages: observation encoding, temporal fusion, intention reasoning, and intention prediction, and finally the intention prediction vector of each agent in the scene is obtained.

[0140] Step 3 (Hierarchical Feature Aggregation): For example... Figure 2 As shown in the layered graph attention feature extraction module in the middle, after concatenating the observation code and the intent prediction vector to construct node features, it sequentially goes through three layers of attention aggregation: same-faction feature aggregation, cross-faction feature aggregation, and global feature aggregation. Finally, the multi-layered features are concatenated to obtain the decision feature vector.

[0141] Step 4 (Strategy Generation and Action Execution): For example... Figure 2 As shown in the lower policy network module, the decision feature vector is input into the policy network corresponding to the platform type, and the action policy is output, which is used to select and execute the action at the current time step.

[0142] Training optimization aspects: such as Figure 2As shown in the learnable parameter adaptive optimization module on the right, during the training phase, the reinforcement learning algorithm combines the learnable temperature control coefficient and self-learning loss optimization to iteratively optimize the policy network. The temperature control coefficient is associated with the uncertainty of intention prediction, achieving an adaptive balance between exploration and exploitation.

[0143] The heterogeneous unmanned cluster game decision-making device provided by the present invention is described below. The heterogeneous unmanned cluster game decision-making device described below and the heterogeneous unmanned cluster game decision-making method described above can be referred to in correspondence.

[0144] Figure 3 This is a schematic diagram of the heterogeneous unmanned cluster game decision-making device provided by the present invention, as shown below. Figure 3 As shown, it includes: The acquisition module 310 is used to acquire local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes first-faction intelligent agents and second-faction intelligent agents, and the heterogeneous unmanned cluster is composed of first-faction intelligent agents and includes intelligent agents of at least two different platform types. The prediction module 320 is used to predict the behavioral intentions of each agent based on historical observation sequences through an intention prediction network, and obtain the intention prediction vector of each agent. The construction module 330 is used to concatenate local observation information with the intention prediction vector to construct the node features of each agent, and to obtain the decision feature vector by performing hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; The control module 340 is used to generate action policies for each first-faction intelligent agent based on the decision feature vector through the policy network, and control each first-faction intelligent agent to execute the corresponding action.

[0145] In this application, an intent prediction network is used to predict the behavioral intent of all agents in a scene based on historical observation sequences. The obtained intent prediction vector is then concatenated with local observation information to construct node features that integrate intent semantics. This allows each agent to make decisions not only based on the current situation but also to proactively consider the action trends of all parties. Furthermore, a hierarchical attention network is used to sequentially perform feature aggregation within the same faction, feature aggregation across factions, and global feature aggregation. This allows for the hierarchical and targeted extraction of intra-faction collaborative relationship features and cross-faction adversarial relationship features from the node features that integrate intent information. These features are then globally integrated to obtain a high-quality decision feature vector that combines intent perception capability with multi-level relationship discrimination. This enables the policy network to generate more forward-looking and collaborative action strategies based on this decision feature vector, effectively improving the collaborative decision-making quality and overall adversarial effectiveness of heterogeneous unmanned swarms in dynamic game adversarial environments.

[0146] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a heterogeneous unmanned swarm game decision-making method. The method includes: acquiring local observation information of each first-faction intelligent agent in the heterogeneous unmanned swarm in a multi-agent interaction scenario, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein each intelligent agent in the multi-agent interaction scenario includes first-faction intelligent agents and second-faction intelligent agents, and the heterogeneous unmanned swarm is composed of first-faction intelligent agents and includes intelligent agents of at least two different platform types; Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intentions of each agent to obtain the intention prediction vector of each agent. The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; Based on the decision feature vector, the action policy of each first-faction agent is generated through the policy network, and the first-faction agents are controlled to perform corresponding actions.

[0147] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the heterogeneous unmanned swarm game decision-making method provided by the above methods. The method includes: acquiring local observation information of each first-faction agent in a multi-agent interaction scenario, and historical observation sequences of each agent in the multi-agent interaction scenario; wherein, each agent in the multi-agent interaction scenario includes first-faction agents and second-faction agents, and the heterogeneous unmanned swarm is composed of first-faction agents and includes agents of at least two different platform types. Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intentions of each agent to obtain the intention prediction vector of each agent. The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; Based on the decision feature vector, the action policy of each first-faction agent is generated through the policy network, and the first-faction agents are controlled to perform corresponding actions.

[0149] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the heterogeneous unmanned swarm game decision-making method provided by the above methods. The method includes: acquiring local observation information of each first-faction intelligent agent in a multi-agent interaction scenario, and historical observation sequences of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes first-faction intelligent agents and second-faction intelligent agents, and the heterogeneous unmanned swarm is composed of first-faction intelligent agents and includes at least two different platform types of intelligent agents; Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intentions of each agent to obtain the intention prediction vector of each agent. The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; Based on the decision feature vector, the action policy of each first-faction agent is generated through the policy network, and the first-faction agents are controlled to perform corresponding actions.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heterogeneous unmanned cluster game decision-making method, applied to the execution phase, characterized in that, include: The local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario is obtained, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes the first-faction intelligent agent and the second-faction intelligent agent, and the heterogeneous unmanned cluster is composed of the first-faction intelligent agent and includes intelligent agents of at least two different platform types. Based on the historical observation sequence, the intention prediction network is used to predict the behavioral intentions of each agent to obtain the intention prediction vector of each agent. The local observation information is concatenated with the intention prediction vector to construct the node features of each agent, and the decision feature vector is obtained by hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequential execution of same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; Based on the decision feature vector, the action policy of each first-faction agent is generated through the policy network, and the first-faction agents are controlled to perform corresponding actions. The policy network is pre-trained using a reinforcement learning algorithm, and an adaptive exploration strategy is employed during the policy training phase that the policy network undergoes, including: A policy entropy regularization term is introduced into the policy optimization objective function, and a learnable temperature control coefficient is set. The temperature control coefficient of each first-faction intelligent agent platform type is set as a learnable variable associated with the uncertainty of intent prediction output by the intent prediction network. The target entropy value is determined based on the dimension of the agent's action space; Based on the difference between the current policy entropy and the target entropy value, the temperature control coefficient is updated by gradient descent; wherein, when the current policy entropy is lower than the target entropy value or the uncertainty of intention prediction increases, the temperature control coefficient is increased, and when the current policy entropy is higher than the target entropy value or the certainty of intention prediction increases, the temperature control coefficient is decreased.

2. The heterogeneous unmanned cluster game decision-making method according to claim 1, characterized in that, The intent prediction network includes a local feature encoder, a temporal fusion network, and an intent output network; The step of predicting the behavioral intentions of each agent through an intention prediction network includes: Each frame of observation information from the historical observation sequence of each agent is input into a local feature encoder corresponding to the platform type of the agent for encoding to obtain an observation feature vector sequence; wherein, agents of the same platform type share the network parameters of the local feature encoder, while agents of different platform types use local feature encoders with independent network parameters. The observed feature vector sequence is concatenated with the intention prediction vector from the previous time step and input into the temporal fusion network to obtain the hidden state of each agent. Based on the hidden state, the association weights between each agent are calculated, and the hidden state is weighted and aggregated to obtain the context vector. The hidden state is concatenated with the context vector, and the intent prediction vector is obtained through the intent output network.

3. The heterogeneous unmanned cluster game decision-making method according to claim 2, characterized in that, The intent prediction network is pre-trained through an intent training phase that includes a composite loss function, which comprises: First-faction intention prediction loss: The real actions of the first-faction agents are mapped to real intention vectors through an action intention encoder, and the mean square error between the intention prediction vector and the real intention vector is calculated. Second camp intention prediction loss: The intention prediction vector of the second camp agent is concatenated with its current observation information, and the next observation of the second camp agent is predicted through the forward dynamics network. The error between the predicted value and the true value is calculated. The composite loss function is a weighted sum of the intention prediction loss of the first camp and the intention prediction loss of the second camp.

4. The heterogeneous unmanned cluster game decision-making method according to claim 1, characterized in that, The aggregation of features from the same camp in the hierarchical attention network includes: The cooperation index is calculated based on the distance between the first-faction neighboring agents and the current agent, and a preset number of first-faction neighboring agents are selected based on the cooperation index. Calculate the same-faction attention coefficient between the current agent and the first-faction neighboring agents after filtering; wherein, multiple sets of independent same-faction attention weight parameters are pre-set, and the same-faction attention coefficient is calculated by calling the same-faction attention weight parameters corresponding to the platform type combination of the current agent and the first-faction neighboring agents. Based on the same-faction attention coefficient, the node features of the selected first-faction neighboring agents are weighted and aggregated to obtain the same-faction aggregated features.

5. The heterogeneous unmanned cluster game decision-making method according to claim 4, characterized in that, Cross-faction feature aggregation in the hierarchical attention network includes: The threat index is calculated based on the distance between the second-faction agent and the current agent, and a preset number of second-faction agents are selected based on the threat index. Calculate the cross-faction attention coefficient between the same-faction aggregation features of the current agent and the node features of the filtered second-faction agents; wherein, multiple sets of independent cross-faction attention weight parameters are pre-set, and the cross-faction attention coefficient is calculated by calling the cross-faction attention weight parameters corresponding to the platform type combination of the current agent and the second-faction agents. Based on the cross-faction attention coefficient, the same-faction aggregation features of the current agent are weighted and aggregated based on a specific learnable parameter matrix to obtain cross-faction aggregation features; The global feature aggregation includes: concatenating the same-faction aggregated features and the cross-faction aggregated features and obtaining pre-aggregated features through a fusion network; weighting and aggregating the pre-aggregated features based on global attention coefficients to obtain global aggregated features; and concatenating the node features of the current agent, the same-faction aggregated features, the cross-faction aggregated features, and the global aggregated features to obtain the decision feature vector.

6. The heterogeneous unmanned cluster game decision-making method according to claim 1, characterized in that, The intent prediction network, the hierarchical attention network, and the policy network were trained using a paradigm of centralized training and distributed execution during a pre-existing centralized training phase. Among them, agents of the same platform type share the parameters of the policy network and the corresponding value network, while agents of different platform types use policy networks and corresponding value networks with independent parameters. During the centralized training phase, the value network utilizes global state information for optimization and updates to assist in the training of the policy network. During the execution phase, each first-faction agent generates an action strategy based solely on the local observation information and the historical observation sequence obtained through communication.

7. A heterogeneous unmanned cluster game decision-making device, applied in the execution phase, characterized in that, include: The acquisition module is used to acquire local observation information of each first-faction intelligent agent in the heterogeneous unmanned cluster in the multi-agent interaction scenario, as well as the historical observation sequence of each intelligent agent in the multi-agent interaction scenario; wherein, each intelligent agent in the multi-agent interaction scenario includes the first-faction intelligent agent and the second-faction intelligent agent, and the heterogeneous unmanned cluster is composed of the first-faction intelligent agents and includes intelligent agents of at least two different platform types. The prediction module is used to predict the behavioral intentions of each agent based on the historical observation sequence through an intention prediction network, and obtain the intention prediction vector of each agent. The construction module is used to concatenate the local observation information with the intention prediction vector to construct the node features of each agent, and to obtain the decision feature vector by performing hierarchical attention feature aggregation through a hierarchical attention network; wherein, the hierarchical attention feature aggregation includes sequentially executed same-faction feature aggregation, cross-faction feature aggregation and global feature aggregation; The control module is used to generate action policies for each first-faction intelligent agent through a policy network based on the decision feature vector, and control each first-faction intelligent agent to execute corresponding actions. The policy network is pre-trained using a reinforcement learning algorithm, and an adaptive exploration strategy is employed during the policy training phase that the policy network undergoes, including: A policy entropy regularization term is introduced into the policy optimization objective function, and a learnable temperature control coefficient is set. The temperature control coefficient of each first-faction intelligent agent platform type is set as a learnable variable associated with the uncertainty of intent prediction output by the intent prediction network. The target entropy value is determined based on the dimension of the agent's action space; Based on the difference between the current policy entropy and the target entropy value, the temperature control coefficient is updated by gradient descent; wherein, when the current policy entropy is lower than the target entropy value or the uncertainty of intention prediction increases, the temperature control coefficient is increased, and when the current policy entropy is higher than the target entropy value or the certainty of intention prediction increases, the temperature control coefficient is decreased.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the heterogeneous unmanned cluster game decision-making method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the heterogeneous unmanned cluster game decision-making method as described in any one of claims 1 to 6.

Images