Heterogeneous agent training method and system for land battle confrontation scene and storage medium

Abstract

The invention discloses a heterogeneous agent collaborative strategy training method based on a large-scale distributed hybrid unmanned and manned land battle confrontation scene. The method comprises the following steps: step a, constructing a land battle confrontation scene, a red battle agent and a blue battle agent through a VBS simulation platform; b, respectively setting a state space, an action space and a reward function of the red-party combat intelligent agent and the blue-party combat intelligent agent; c, constructing a strategy network and a value network, and inserting a cross-agent attention mechanism into the strategy network and / or the value network; and step d, through large-scale distributed training, parallel operation of a multi-instance simulation environment, strategy network fusion cross-agent information output action, value network evaluation of joint action value, training to enable the model to converge, and through a reinforcement learning algorithm, outputting an action sequence or a simulation effect to analyze a cooperation strategy of a red-party combat agent. And the cooperative combat capability of the heterogeneous agent system in a land battle confrontation scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system and computer-readable storage medium for training heterogeneous intelligent agents in land combat scenarios. Background Art

[0002] First, in the face of a highly heterogeneous intelligent agent system, the training of collaborative strategies must fully consider the differences between the agents and the complexity of their interactions. The differences between different agents in terms of movement speed, firepower configuration, reconnaissance range, etc. require that the training algorithm can accurately capture and optimize the synergy brought about by these differences in order to maximize the overall combat effectiveness. However, this multi-dimensional collaborative optimization problem involves a large number of parameter adjustments and strategy iterations, making the training process extremely complex and time-consuming.

[0003] Secondly, the limitations of traditional stand-alone training methods are increasingly prominent when facing large-scale, highly complex intelligent agent systems. Single-machine training is limited by computing resources, memory, and video memory capacity, and it is difficult to support large-scale model training and efficient data processing. With the in-depth application of deep learning technology, the surge in the number of model parameters and the explosive growth in the amount of training data have made single-machine training face severe challenges in terms of efficiency and scalability. In addition, single-machine training lacks sufficient flexibility to deal with the complex interactions between different intelligent agents, and it is difficult to effectively simulate the dynamic changes in the real battlefield environment.

[0004] In view of the above background, an innovative training method is urgently needed to overcome the limitations of traditional single-machine training and enhance the collaborative combat capabilities of heterogeneous intelligent systems in land combat confrontation scenarios. Summary of the invention

[0005] In order to solve the problem of coordination between heterogeneous agents and the difficulty of single-machine training in complex scenarios, the present invention aims to propose a method for training heterogeneous agent coordination strategies in large-scale distributed hybrid unmanned and manned land combat scenarios. This method introduces a distributed computing framework and fully utilizes the parallel processing capabilities of multiple computing nodes to achieve efficient decomposition and coordinated execution of training tasks; at the same time, combined with advanced deep learning optimization algorithms, the coordination strategies of heterogeneous agents are accurately trained and optimized, in order to achieve efficient and accurate coordinated operations in complex and changing battlefield environments.

[0006] To achieve the above object, the present invention provides a heterogeneous agent collaborative strategy training method based on a large-scale distributed hybrid unmanned and manned land combat confrontation scenario, which comprises the following steps: Step a. Construct a land combat confrontation scenario, a red combat agent, and a blue combat agent through the VBS simulation platform; wherein the red combat agent includes more than one manned agent and more than one unmanned agent, and the blue combat agent includes more than one manned agent and / or more than one unmanned agent; Step b. respectively setting the state space, action space and reward function of the red combat agent and the blue combat agent; wherein the state space defines the set of environmental information that the agent can perceive at any time, the action space defines the set of all possible and effective actions that the agent can take at any time, and the reward function gives rewards or penalties according to the actions of the agent in the current state and the results produced; Step c. construct a strategy network and a value network, wherein the strategy network fuses the information of each combat agent through a cross-agent attention mechanism and outputs the action distribution of the combat agent; the value network evaluates the value of the joint action in the current state; and inserts a cross-agent attention mechanism into the strategy network and / or the value network; Step d. Through large-scale distributed training, multi-instance simulation environments are run in parallel. The policy network fuses cross-agent information to output actions, and the value network evaluates the value of joint actions. The training makes the model converge, and the reinforcement learning algorithm outputs action sequences or simulation effects to analyze the collaborative strategy of the red side's combat agents.

[0007] Preferably, among the red side combat intelligent agents, manned intelligent agents include tanks and combat vehicles, and unmanned intelligent agents include unmanned vehicles and drones; the blue side combat intelligent agents include tanks, helicopters and infantry squads; and the land combat confrontation scene adopts an urban scene.

[0008] Preferably, the state space includes local observations and global observations, and training information is screened out from all situations provided by the VBS simulation platform as the local observations and the global observations; the action space is set according to the simulation platform entity model and the action design of the entity model; the reward function is set according to the victory and defeat results and the confrontation process design; the reward function includes sparse rewards for victory and defeat, and rewards for the process of attacking and being attacked; wherein, the sparse rewards for victory and defeat are inversely proportional to the remaining troops and the duration of the confrontation.

[0009] Preferably, the policy network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer to obtain basic feature representations; the local observations include position, speed, weapon ammunition status, and quantity; Embedding layer: The local observation of each agent is converted into an embedding vector through the embedding layer; Cross-agent attention layer: Introduces a cross-agent attention mechanism. This layer accepts the embedded representations of all agents as input, learns the interaction relationship between different agents through the attention mechanism, and outputs the weighted context information of each agent. Hidden layer: includes multiple layers of fully connected networks, each layer is followed by a ReLU activation function; Output layer: Use LSTM to process time series information, output the action distribution of the agent, and output the probability of each action through the softmax layer.

[0010] Preferably, the value network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer; Embedding layer: The local observation of each agent is converted into an embedding vector through the embedding layer; Cross-agent attention layer: Introducing a cross-agent attention mechanism to learn the interaction relationship between different agents; Hidden layer: includes multiple layers of fully connected networks, each layer is followed by a ReLU activation function; Output layer: includes one neuron that outputs a scalar value representing the value of the joint action in the current state.

[0011] Preferably, the cross-agent attention mechanism is inserted as follows: a. Insert after the hidden layer: The policy network or the value network comprises a plurality of hidden layers, and the cross-agent attention layer is inserted after the penultimate hidden layer, which receives the hidden layer outputs of the agents as inputs and outputs weighted feature representations, which are then passed to a final output layer; or, b. Insert after the input layer: The cross-agent attention layer is added after the input layer, which receives the original state of the agent as input and outputs a weighted state representation, which is then passed to the subsequent hidden layer for further processing.

[0012] Preferably, the cross-agent attention mechanism comprises the following steps: c1. Information Collection: Collect each agent’s local observation space; c2. Attention weight calculation: For the information collected from other agents, each agent will calculate an attention weight, which indicates the degree of attention paid by the current agent to the information of other agents; c3. State fusion: Using the calculated attention weights, the current agent weightedly fuses the information from other agents; c4. Input the fused input representation into the policy network and value network respectively for action selection and value evaluation.

[0013] Preferably, the step d further comprises: d1. Build a distributed training cluster and distributed training framework to distribute training tasks to multiple computing nodes for parallel processing; d2. Assign the Red and Blue combat agents to different simulation instances, each of which contains the complete Red and Blue agents and their interactive environment; d3. In each simulation instance, the agent performs actions according to the current strategy, interacts with the environment and collects states, actions, and rewards; and regularly updates and synchronizes states, actions, and rewards between instances; d4. Perform multiple iterations of training until the preset convergence condition is reached and then stop training; d5. Use the trained policy network to generate the action sequence of the Red Team combat agent in the simulation environment; d6. Use the trained value network to evaluate the coordination strategy of the Red Team’s combat agents; d7. Collect training data and conduct quantitative analysis of the Red Team’s combat agent’s collaborative strategy based on pre-set evaluation criteria.

[0014] In addition, to achieve the above-mentioned objectives, the present invention also provides a heterogeneous intelligent agent collaborative strategy training system based on a large-scale distributed hybrid unmanned and manned land combat confrontation scenario, the system comprising a memory, a processor, and a heterogeneous intelligent agent collaborative strategy training program stored on the memory and executable on the processor, the heterogeneous intelligent agent collaborative strategy training program, when executed by the processor, implements the steps of the heterogeneous intelligent agent collaborative strategy training method as described in any one of the above items.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, characterized in that a heterogeneous intelligent agent collaborative strategy training program is stored on the computer-readable storage medium, and when the heterogeneous intelligent agent collaborative strategy training program is executed by the processor, the steps of the heterogeneous intelligent agent collaborative strategy training method as described in any one of the above items are implemented.

[0016] The beneficial effects of the present invention are: (1) In the training method of the present invention, the policy network integrates and fuses information from different combat agents through a cross-agent attention mechanism, and outputs the action distribution of each agent in the current state. This mechanism promotes information sharing and collaboration between agents, enabling them to make better decisions based on a global perspective. At the same time, the value network is responsible for evaluating the overall value of the joint actions of all agents in the current state, providing an important reference for the optimization of the policy network.

[0017] (2) By combining deep learning algorithms and cross-agent attention mechanisms, the present invention can accurately train and optimize the collaborative strategies between heterogeneous agents (including manned and unmanned agents). This mechanism effectively improves the perception, decision-making and collaborative combat capabilities of agents in complex and changeable battlefield environments, making collaborative strategies more efficient and accurate.

[0018] (3) By constructing a variety of red and blue combat agents and setting up complex state spaces, action spaces, and reward functions, the present invention can simulate an environment close to that of a real battlefield, which helps the agents to continuously adapt to and learn various battlefield situations during training and improve their ability to respond to unknown or unexpected situations.

[0019] (4) The present invention adopts distributed training, which can make full use of computing resources and improve training efficiency by distributing training tasks to multiple computing nodes for parallel processing. At the same time, distributed training also has good scalability and flexibility. The number and configuration of computing nodes can be dynamically adjusted according to actual needs to adapt to training scenarios of different scales and complexities.

[0020] (5) In the training of heterogeneous agents, distributed training technology can effectively promote collaborative learning and information sharing among different agents, thereby improving the overall training effect. In addition, distributed training can also reduce the risk of falling into local optimal solutions by exploring different strategy spaces in parallel, thereby improving the global optimization capability of training.

[0021] (6) The present invention makes full use of the parallelism, scalability and flexibility of distributed training technology to build an efficient, realistic and customizable training environment. In this environment, the Red Army's multi-heterogeneous intelligent agents can conduct collaborative combat training in a highly simulated battlefield environment, and improve combat effectiveness and adaptability through continuous trial and error and optimization. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be clearly and completely described in combination with the specific technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] In land combat scenarios, heterogeneous agents (such as different types of tanks, drones, etc.) need to work closely together to complete combat missions. However, since different agents have different capabilities, perception ranges, and action speeds, how to achieve effective coordination between them has become a major problem, mainly for the following reasons: 1) Capability differences: Different agents have differences in firepower, mobility, defense, reconnaissance capabilities, etc., which requires the collaborative algorithm to fully consider these differences and reasonably allocate tasks and resources; 2) Decision conflict: In the collaborative process, different agents may make conflicting decisions based on their own local information, which requires an effective coordination mechanism to resolve.

[0024] The present invention provides a heterogeneous agent collaborative strategy training method based on a large-scale distributed hybrid unmanned and manned land combat confrontation scenario, which comprises the following steps: Step a. Construct a land combat confrontation scenario, a red combat agent, and a blue combat agent through the VBS simulation platform; wherein the red combat agent includes more than one manned agent and more than one unmanned agent, and the blue combat agent includes more than one manned agent and / or more than one unmanned agent; Step b. respectively setting the state space, action space and reward function of the red combat agent and the blue combat agent; wherein the state space defines the set of environmental information that the agent can perceive at any time, the action space defines the set of all possible and effective actions that the agent can take at any time, and the reward function gives rewards or penalties according to the actions of the agent in the current state and the results produced; Step c. Construct a policy network and a value network. The policy network fuses the information of each combat agent through a cross-agent attention mechanism and outputs the action distribution of the combat agent. The value network evaluates the value of the joint action in the current state. The cross-agent attention mechanism is inserted into the policy network and / or the value network. Step d. Through large-scale distributed training, multi-instance simulation environments are run in parallel. The policy network fuses cross-agent information to output actions, and the value network evaluates the value of joint actions. The training makes the model converge, and the reinforcement learning algorithm outputs action sequences or simulation effects to analyze the collaborative strategy of the red side's combat agents.

[0025] In this embodiment, among the Red side combat agents, manned agents include tanks and chariots, and unmanned agents include unmanned vehicles and drones; chariots are mainly rear support, good at long-range attacks and reconnaissance; tanks are good at close-range attacks; unmanned vehicles are good at concealed engagement; drones are mainly used for reconnaissance. The Blue side combat agents include four tanks, a helicopter and two infantry squads; by simulating the coordinated combat between manned agents and unmanned agents, as well as the confrontation between different combat units of the Red and Blue sides, the present invention can effectively improve the coordinated combat capabilities of agents in complex battlefield environments.

[0026] The land battle confrontation scenario adopts an urban scenario, which is mainly used to train the Red Party's heterogeneous intelligent agents to find a collaborative strategy between entities in urban combat to defeat the Blue Party. The introduction of urban scenarios makes the training closer to actual combat needs and helps the intelligent agents adapt to different combat environments and changes in battlefield situations.

[0027] In this embodiment, the state space includes local observations and global observations, and training information is screened out from all situations provided by the VBS simulation platform as the local observations and the global observations; the action space is set according to the simulation platform entity model and the action design of the entity model; the reward function is set according to the victory and defeat results and the confrontation process design; the reward function includes sparse rewards for victory and defeat, and rewards for the process of attacking and being attacked; wherein, the sparse rewards for victory and defeat are inversely proportional to the remaining troops and the duration of confrontation.

[0028] Among them, local observation includes information such as position, speed, weapon and ammunition status, quantity, etc. Local observation allows each agent to obtain only part of the information about its surrounding environment, such as the position and status of nearby friendly and enemy forces, as well as data from its own sensors. This design requires the agent to consider the local situation when making decisions, which increases the difficulty and authenticity of decision-making, and also prompts more information exchange and collaboration between agents.

[0029] Compared with local observation, global observation provides comprehensive information about the entire battlefield environment. Although complete global information may be difficult to obtain in actual combat, providing such information during the training phase helps the agent understand the entire battlefield situation and make more global decisions. At the same time, global observation also promotes information sharing and collaborative operations between agents.

[0030] The action space is set according to the action design of the simulation platform entity model and its entity model, and the action of each agent is defined based on its actual combat capability. For example, a tank may include actions such as moving, shooting, and turning, while a drone may include actions such as flying, hovering, and reconnaissance. In this embodiment, the combat agent of the red party entity can maneuver, strike, and reconnaissance. In addition, when striking, the target to be struck and the weapons and ammunition to be used must be given. The maneuvering direction of the drone is discretized, and only four directions of east, west, south, and north are selected, and the reconnaissance height selects 10 commonly used heights as the drone's action. The dimension of the action space of the red party entity in this scene is a total of 1006 dimensions.

[0031] The sparse reward for victory or defeat is a reward or penalty given according to the final result of the confrontation. Since the result of victory or defeat is often binary (win or lose), this kind of reward is sparse. In this embodiment, the sparse reward for victory or defeat is inversely proportional to the remaining troops and the duration of the confrontation, that is, the fewer the remaining troops and the longer the confrontation, the less the reward (or the greater the penalty).

[0032] The process rewards for attacking and being attacked are given according to the specific performance of the agent in the confrontation process, such as successfully attacking the enemy, avoiding being attacked by the enemy, etc. The introduction of process rewards makes the reward function more detailed and comprehensive, and can more accurately evaluate the combat effectiveness and coordination ability of the agent.

[0033] In this embodiment, the policy network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer to obtain basic feature representations; the local observations include position, speed, weapon ammunition status, and quantity; Embedding layer: The local observation of each agent is converted into an embedding vector through an embedding layer (such as a fully connected network as an embedding layer). The purpose is to convert the high-dimensional original input into a low-dimensional representation suitable for neural network processing; this conversion not only reduces the dimension of the data, but also extracts the key features in the observed data, providing a more compact and effective information representation for subsequent processing. The shared nature of the embedding vector also promotes information exchange and sharing between different agents; Cross-agent attention layer: Introduces a cross-agent attention mechanism. This layer accepts the embedded representations of all agents as input, learns the interaction relationship between different agents through the attention mechanism, and outputs the weighted context information of each agent. Hidden layer: includes multiple layers of fully connected networks, each followed by a ReLU activation function; this enables the network to learn complex nonlinear relationships and perform deep abstraction and representation of input data, thus enhancing the network’s expressiveness and generalization capabilities; Output layer: Use LSTM (Long Short-Term Memory Network) to process time series information, output the action distribution of the agent, and output the probability of each action through the softmax layer.

[0034] LSTM is a special recurrent neural network (RNN) that can effectively capture long-term dependencies in sequence data. In collaborative combat scenarios, the decisions of agents are often affected by previous decisions and states, so the processing of time series information is crucial to generating accurate action distribution. After the output of LSTM is processed by the softmax layer, the probability distribution of each action can be obtained, thereby guiding the agent to make decisions.

[0035] In this embodiment, the value network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer; Embedding layer: The local observation of each agent is converted into an embedding vector through the embedding layer; Cross-agent attention layer: Introducing a cross-agent attention mechanism to learn the interaction relationship between different agents; Hidden layer: includes multiple layers of fully connected networks, each layer is followed by a ReLU activation function; Output layer: It includes a neuron that outputs a scalar value, which indicates the value of the joint action in the current state. The scalar value is a direct measure of the quality of the joint action, and it will be used as a feedback signal to guide the training of the policy network. By optimizing this scalar value, the policy network can gradually learn more effective and coordinated combat strategies.

[0036] The embedding process of the embedding layer of the policy network or the value network adopts any of the following schemes: Solution 1. Feature-based independent embedding: Design independent embedding layers for each feature (such as latitude and longitude, altitude, direction, etc.); Each embedding layer selects the appropriate embedding dimension based on the characteristics of the feature (such as range and discreteness); All embedded feature vectors (including embedding vectors of latitude and longitude, altitude, direction, speed, weapon ammunition quantity and status, etc.) are concatenated to form the final embedded representation.

[0037] Solution 2. Unified Embedding Layer: Use a unified embedding layer to process all features instead of designing a separate embedding layer for each feature; Convert all features (whether continuous or discrete) to fixed-length vectors (e.g., by normalization, one-hot encoding, etc.), and then concatenate them into a long vector; Finally, the long vector concatenated from the above two solutions is used as the input of the unified embedding layer to output a low-dimensional continuous embedding vector.

[0038] Among them, the feature embedding methods include: embedding based on sine and cosine functions, embedding based on interval division, normalization of continuous features (such as longitude and latitude, altitude, speed), one-hot encoding of discrete features (such as direction, survival status, quantity and status of weapons and ammunition), etc.

[0039] The value network can quantify the value of joint actions under different collaborative strategies in evaluating the action value of collaborative strategies. By comparing the output values ​​of the value network under different strategies, it can be intuitively seen which collaborative strategy can maximize the overall benefits; based on the evaluation results of the value network, decision makers can formulate collaborative strategies more scientifically; by analyzing the output of the value network, it is also possible to discover possible problems or bottlenecks in the collaborative strategy. For example, if the value of a joint action is always low, it may mean that the execution of the action is difficult or the coordination with other actions is not tacit.

[0040] Moreover, during the model training process, the value network provides important feedback signals to the policy network by evaluating the value of the joint action in the current state. This feedback signal is a key component in the reinforcement learning algorithm. It tells the policy network which actions are good (i.e., can bring high returns or low losses) and which are bad. By comparing the actual rewards obtained with the predicted values ​​of the value network, the loss function can be calculated, and the parameters of the policy network can be adjusted accordingly to make its output more conducive to obtaining high-value joint actions. In addition, the existence of the value network helps to accelerate the convergence of the training process. Since it can provide an accurate estimate of the value of the joint action in the current state, the policy network can learn which action combinations are effective more quickly, thereby reducing the time of blind exploration.

[0041] In this embodiment, the cross-agent attention mechanism is inserted as follows: a. Insert after the hidden layer: The policy network or the value network includes multiple hidden layers, and the cross-agent attention layer is inserted after the penultimate hidden layer. The layer receives the hidden layer output of the agent as input and outputs a weighted feature representation, which is then passed to the final output layer. Inserting the cross-agent attention mechanism after the hidden layer of the network allows the network to perform preliminary feature extraction and abstraction on the input state, and then integrate information between different agents through the cross-agent attention mechanism. This method helps to apply cross-agent attention on higher-level feature representations, which may be more helpful in capturing complex interaction patterns.

[0042] Or, b. insert after the input layer: The cross-agent attention layer is added after the input layer, which receives the original state of the agent as input and outputs a weighted state representation, which is then passed to the subsequent hidden layer for further processing; inserting the cross-agent attention mechanism directly after the input layer can integrate information between agents earlier, helping the network learn the dependencies between agents from the beginning. This approach may be suitable for situations where the state space is small or the interaction between agents is very critical.

[0043] In this embodiment, the cross-agent attention mechanism includes the following steps: c1. Information Collection: Collect the local observation space of each agent; in a multi-agent system, each agent has its own local observation space, including position, speed, number of weapons, etc. c2. Attention weight calculation: For the information collected from other agents, each agent will calculate an attention weight, which indicates the degree of attention paid by the current agent to the information of other agents; c3. State fusion: Using the calculated attention weights, the current agent weightedly integrates the information from other agents to form a more comprehensive understanding of the current environment; c4. Input the fused input representation into the policy network and value network respectively for action selection and value evaluation.

[0044] The calculation of attention weights is usually based on multiple factors such as the relative positions between agents, state differences, historical interaction records, etc. Through the attention mechanism, agents can automatically identify other agents that have the greatest impact on their decisions in the current state and pay more attention to them.

[0045] The calculation method of attention weight includes: a. Dot-Product Attention: This is one of the simplest ways to calculate attention. The attention score is obtained by calculating the dot product of the query and the key, and then it is usually normalized by the softmax function to obtain the attention weight.

[0046] b. Additive Attention: Different from dot-product attention, additive attention uses an additional neural network to calculate the similarity between the query and the key. This neural network can be a fully connected layer or a multi-layer perceptron (MLP).

[0047] c. Multi-Head Attention: In order to improve the representation ability and stability of the model, the multi-head attention mechanism splits the query, key, and value into multiple heads, then performs attention calculations independently on each head, and finally concatenates the results.

[0048] After obtaining the attention weights, the current agent will use these weights to perform weighted fusion on the information from other agents. The information of each agent is given different importance according to its corresponding attention weight. Through state fusion, the current agent can generate a comprehensive representation that contains the key information of other agents, which will serve as input for subsequent decision-making and value evaluation.

[0049] The cross-agent attention mechanism of the present invention is a method of allocating attention resources among multiple agents, which allows agents to consider the states, intentions, and actions of other agents in the decision-making process. This mechanism can enhance the information exchange and collaboration capabilities between agents, thereby achieving more effective collaboration in complex environments. Specifically, the collaboration between agents is enhanced through the following aspects: 1) Enhance information perception capabilities: The cross-agent attention mechanism allows each agent to pay attention not only to its own state and environmental information when making decisions, but also to the state and behavior of other agents. In this way, the agent can more comprehensively understand the current environment and the requirements of collaborative tasks, and thus make more accurate and reasonable decisions.

[0050] 2) Promote intention sharing and understanding: Through the cross-agent attention mechanism, agents are able to identify and analyze the intentions and plans of other agents. This sharing and understanding of intentions helps promote collaboration between agents. Agents can adjust their behaviors based on the intentions of other agents to achieve common goals.

[0051] 3) Improve the robustness of collaborative decision-making: Faced with complex and ever-changing environments and tasks, a single agent may not be able to make the best decision. The cross-agent attention mechanism allows agents to negotiate and compromise to reach a consistent decision. This collaborative decision-making approach can improve the robustness and adaptability of the system, enabling it to better cope with various challenges and uncertainties.

[0052] In this embodiment, the step d further comprises: d1. Build a distributed training cluster and distributed training framework to distribute training tasks to multiple computing nodes for parallel processing; d2. Assign the Red and Blue combat agents to different simulation instances, each of which contains the complete Red and Blue agents and their interactive environment; d3. In each simulation instance, the agent performs actions according to the current strategy, interacts with the environment and collects states, actions, and rewards; and regularly updates and synchronizes states, actions, and rewards between instances; d4. Perform multiple iterations of training until the preset convergence condition is reached and then stop training; d5. Use the trained policy network to generate the action sequence of the Red Team combat agent in the simulation environment; d6. Use the trained value network to evaluate the coordination strategy of the Red Team’s combat agents; d7. Collect training data and conduct quantitative analysis of the Red Team’s combat agent’s collaborative strategy based on pre-set evaluation criteria.

[0053] The present invention adopts distributed training, which has significant advantages over single-machine training, especially in terms of computing resources, convergence speed, and efficient collaboration, including: 1) Improve the efficiency of computing resource utilization: Distributed training greatly expands the available computing resources by distributing training tasks to multiple computers or multiple GPUs. In distributed training, computing resources can be used more efficiently.

[0054] 2) Accelerate the training process: Distributed training can perform gradient calculations or parameter updates on multiple computing nodes at the same time, greatly improving the convergence speed of the entire training process. This means that in the same amount of time, distributed training can complete more training iterations, thereby approaching the optimal solution faster.

[0055] 3) Efficient collaboration: The distributed training system supports real-time communication and information exchange between agents. This helps agents obtain the status, decisions, and results of other agents in a timely manner, thereby making more accurate and efficient collaborative decisions. Through real-time communication, agents can work more closely together to complete complex tasks.

[0056] Through large-scale distributed training, when the model converges, the collaborative strategy learned by the intelligent agent can be analyzed through the action sequence given by the algorithm; or it can be summarized and analyzed through the effects of the simulation platform front end.

[0057] The present invention trains heterogeneous agents in land combat scenarios and draws the conclusion that agents can learn collaborative strategies to ensure mission success, and in some cases learn strategies beyond human experience. In addition, the trained red entity has a high winning rate against the blue regular entity, indicating that the red heterogeneous agent has certain advantages.

[0058] In addition, to achieve the above-mentioned objectives, the present invention also provides a heterogeneous intelligent agent collaborative strategy training system based on a large-scale distributed hybrid unmanned and manned land combat confrontation scenario, the system comprising a memory, a processor, and a heterogeneous intelligent agent collaborative strategy training program stored on the memory and executable on the processor, the heterogeneous intelligent agent collaborative strategy training program, when executed by the processor, implements the steps of the heterogeneous intelligent agent collaborative strategy training method as described in any one of the above items.

[0059] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the system, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor and the input unit with access to the memory.

[0060] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, characterized in that a heterogeneous agent collaborative strategy training program is stored on the computer-readable storage medium, and when the heterogeneous agent collaborative strategy training program is executed by the processor, the steps of the heterogeneous agent collaborative strategy training method as described in any one of the above items are implemented. The computer-readable storage medium can be a computer-readable storage medium contained in the memory in the above-mentioned embodiment; or it can be a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores at least one instruction, and the instruction is loaded by the processor and executes the heterogeneous agent collaborative strategy training method. The computer-readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0061] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system embodiment and the storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0062] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0063] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for training heterogeneous intelligent agent collaborative strategies in a large-scale distributed hybrid unmanned and manned land combat confrontation scenario, characterized in that: The following steps are involved: Step a. Construct a land combat confrontation scenario, a red combat agent, and a blue combat agent through the VBS simulation platform; wherein the red combat agent includes more than one manned agent and more than one unmanned agent, and the blue combat agent includes more than one manned agent and / or more than one unmanned agent; Step b. respectively setting the state space, action space and reward function of the red combat agent and the blue combat agent; wherein the state space defines the set of environmental information that the agent can perceive at any time, the action space defines the set of all possible and effective actions that the agent can take at any time, and the reward function gives rewards or penalties according to the actions of the agent in the current state and the results produced; Step c. construct a strategy network and a value network, wherein the strategy network fuses the information of each combat agent through a cross-agent attention mechanism and outputs the action distribution of the combat agent; the value network evaluates the value of the joint action in the current state; and inserts a cross-agent attention mechanism into the strategy network and / or the value network; Step d. Through large-scale distributed training, multi-instance simulation environments are run in parallel. The policy network fuses cross-agent information to output actions, and the value network evaluates the value of joint actions. The training makes the model converge, and the reinforcement learning algorithm outputs action sequences or simulation effects to analyze the collaborative strategy of the red side's combat agents.

2. The method according to claim 1, characterized in that: Among the red side combat intelligent agents, manned intelligent agents include tanks and combat vehicles, and unmanned intelligent agents include unmanned vehicles and drones; the blue side combat intelligent agents include tanks, helicopters and infantry squads; the land combat confrontation scene adopts an urban scene.

3. The method according to claim 1, characterized in that: The state space includes local observations and global observations, and training information is screened out from all situations provided by the VBS simulation platform as the local observations and the global observations; the action space is set according to the simulation platform entity model and the action design of the entity model; the reward function is set according to the victory and defeat results and the confrontation process design; the reward function includes sparse rewards for victory and defeat, and rewards for the process of attacking and being attacked; wherein, the sparse rewards for victory and defeat are inversely proportional to the remaining troops and the duration of confrontation.

4. The method according to claim 1, characterized in that: The policy network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer to obtain basic feature representations; the local observations include position, speed, weapon ammunition status, and quantity; Embedding layer: The local observation of each agent is converted into an embedding vector through the embedding layer; Cross-agent attention layer: Introduces a cross-agent attention mechanism. This layer accepts the embedded representations of all agents as input, learns the interaction relationship between different agents through the attention mechanism, and outputs the weighted context information of each agent. Hidden layer: includes multiple layers of fully connected networks, each layer is followed by a ReLU activation function; Output layer: Use LSTM to process time series information, output the action distribution of the agent, and output the probability of each action through the softmax layer.

5. The method according to claim 4, characterized in that The value network includes: Input layer: The global state and each agent’s local observations are input into a shared embedding layer; Embedding layer: The local observation of each agent is converted into an embedding vector through the embedding layer; Cross-agent attention layer: Introducing a cross-agent attention mechanism to learn the interaction relationship between different agents; Hidden layer: includes multiple layers of fully connected networks, each layer is followed by a ReLU activation function; Output layer: includes one neuron that outputs a scalar value representing the value of the joint action in the current state.

6. The method according to claim 5, characterized in that The cross-agent attention mechanism is inserted as follows: a. Insert after the hidden layer: The policy network or the value network comprises a plurality of hidden layers, and the cross-agent attention layer is inserted after the penultimate hidden layer, the cross-agent attention layer receives the hidden layer output of the agent as input, and outputs a weighted feature representation, which is then passed to a final output layer; or, b. Insert after the input layer: The cross-agent attention layer is added after the input layer, which receives the original state of the agent as input and outputs a weighted state representation, which is then passed to the subsequent hidden layer for further processing.

7. The method according to any one of claims 1 to 6, characterized in that: The cross-agent attention mechanism includes the following steps: c1. Information Collection: Collect each agent’s local observation space; c2. Attention weight calculation: For the information collected from other agents, each agent will calculate an attention weight, which indicates the degree of attention paid by the current agent to the information of other agents; c3. State fusion: Using the calculated attention weights, the current agent weightedly fuses the information from other agents; c4. Input the fused input representation into the policy network and value network respectively for action selection and value evaluation.

8. The method according to any one of claims 1 to 6, characterized in that: The step d further comprises: d1. Build a distributed training cluster and distributed training framework to distribute training tasks to multiple computing nodes for parallel processing; d2. Assign the Red and Blue combat agents to different simulation instances, each of which contains the complete Red and Blue agents and their interactive environment; d3. In each simulation instance, the agent performs actions according to the current strategy, interacts with the environment and collects states, actions, and rewards; and regularly updates and synchronizes states, actions, and rewards between instances; d4. Perform multiple iterations of training until the preset convergence condition is reached and then stop training; d5. Use the trained policy network to generate the action sequence of the Red Team combat agent in the simulation environment; d6. Use the trained value network to evaluate the coordination strategy of the Red Team’s combat agents; d7. Collect training data and conduct quantitative analysis of the Red Team’s combat agent’s collaborative strategy based on pre-set evaluation criteria.

9. A heterogeneous intelligent agent collaborative strategy training system based on large-scale distributed hybrid unmanned and manned land combat confrontation scenarios, characterized in that: The system includes a memory, a processor, and a heterogeneous agent collaborative strategy training program stored in the memory and executable on the processor. When the heterogeneous agent collaborative strategy training program is executed by the processor, the steps of the heterogeneous agent collaborative strategy training method as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a heterogeneous agent collaborative strategy training program, which, when executed by a processor, implements the steps of the heterogeneous agent collaborative strategy training method as described in any one of claims 1 to 8.