A method for constructing a collaborative combat decision-making intelligent agent for ground unmanned equipment

Abstract

The present invention relates to a method for constructing a collaborative combat decision-making intelligent agent for ground unmanned equipment. To address the dimensionality curse problem in the solution space of complex collaborative combat tasks, a two-layer decision-making architecture based on data-driven and knowledge-driven approaches is constructed. This architecture effectively compresses the DRL solution space and improves exploration efficiency, while also enabling intelligent decision-making for tasks of different granularities throughout the entire combat process. To address the generation of combat command action strategies at the "task layer," a collaborative combat decision-making intelligent agent network model is constructed to achieve unified coordinated control of combat actions, from effective representation of complex state inputs. A distributed training framework is constructed to accelerate sampling and training efficiency, enabling high-throughput asynchronous parallel training of the collaborative combat decision-making intelligent agent network model. Furthermore, a combat decision-making intelligent agent learning algorithm based on deep reinforcement learning is established and solved, effectively improving the learning efficiency and ultimate performance of the intelligent agent network model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a method for constructing a ground unmanned equipment collaborative combat decision-making intelligent body. Background Art

[0002] Coordinated ground unmanned combat refers to a combat method in which two or more ground unmanned systems, possibly with the support of drones, coordinate and cooperate to complete a specific ground military mission in an information-based, unmanned, networked, and swarm-based combat environment. This includes coordinated maneuvers, coordinated reconnaissance, and coordinated strikes. To enhance the self-organizing and effective coordination capabilities of unmanned systems facing complex tasks, as well as the success rate of combat missions, numerous researchers have conducted in-depth research on decision-making methods that can help improve coordinated unmanned combat. This approach addresses various threats in highly competitive environments and aims to leverage decision-making advantages to defeat adversaries and achieve victory in future battlefields. Compared to the autonomous combat decisions of a single unmanned system, coordinated combat by multiple unmanned systems features a vast decision-making space, incomplete information, diverse equipment types, and dynamic and intense confrontation.

[0003] Currently, mainstream artificial intelligence (AI)-based autonomous collaborative combat decision-making methods fall into two main categories: knowledge-driven and data-driven. Knowledge-driven methods typically employ methods such as behavior trees, finite state machines, and goal-oriented action planning, leveraging existing experience, rules, models, and algorithms to model behavioral strategies. These methods offer good interpretability, reusability, and high algorithmic execution efficiency. However, when faced with complex large-scale collaborative tasks, these methods struggle to fully cover the vast solution space of swarm collaborative behaviors and are unable to achieve continuous learning and iterative evolution of behavioral strategies. Data-driven methods, which often employ methods such as artificial neural networks or deep reinforcement learning (DRL), can achieve extensive coverage and exploration of the solution space and iterative optimization of strategies without requiring precise modeling. However, these methods exhibit typical "black box" characteristics and require extensive data for training. Summary of the Invention

[0004] The present invention aims to provide a method for constructing a ground unmanned equipment collaborative combat decision-making intelligent body to address the deficiencies in the existing technology. The technical problem to be solved by the present invention is achieved through the following technical solutions.

[0005] A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent comprises the following steps:

[0006] S1: Build a two-tier decision-making architecture based on data-driven and knowledge-driven decision-making to achieve intelligent decision-making at two different granularities: the mission layer and the execution layer during combat;

[0007] S2, based on the decision-making process of Markov process, builds a network model of collaborative combat decision-making intelligent agents;

[0008] S3, by building a distributed training framework, enables high-throughput asynchronous parallel training of collaborative combat decision-making agent network models;

[0009] S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy;

[0010] S5, conduct simulation tests on the simulation deduction platform to verify the feasibility of the technical route.

[0011] Preferably, in S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data, and the macro-combat command action strategy includes maneuvering and attacking; the execution layer is used to respond to the macro-combat command action strategy issued by the task layer, and the response includes adjusting artillery, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and obstacle avoidance.

[0012] Preferably, in S2, the method for constructing the collaborative combat decision-making agent network model includes:

[0013] S21, define the state space, action space and establish the reward function;

[0014] S22, encodes the multi-feature information based on the state space and action space input to obtain a partially observable game state encoding vector;

[0015] S23, deriving multi-head compound actions through partially observable game state encoding vectors;

[0016] S24 decodes multiple compound actions to arrive at the final combat decision.

[0017] Preferably, in S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information of all different types of own ground unmanned equipment and drones, and the enemy entity information includes information of tanks, infantry fighting vehicles, soldiers, and drones; the action space is a collection of all actions that can be performed by different unmanned combat equipment during collaborative combat, and the action space is defined by composite actions.

[0018] Preferably, in S21, the reward function is a continuity function based on the process reward, and the process reward r of the agent at each time step is t P Divided into target discovery and loss, loss of manpower value and equipment damage, and changes in own position

[0019] Preferably, in S22, by using a multi-head attention mechanism, a multi-layer perceptron, an embedding layer, and a fully connected layer, feature extraction and fusion are performed on the global scalar information that can reflect the entire battlefield situation, the action performed by the agent in the previous step, and the attribute feature information reflecting each entity, thereby obtaining a partially observable game state encoding vector, which specifically includes the following steps:

[0020] S221, for both friendly and enemy combat units, use the entity feature vectors in the same group as the query, key, and value of the multi-head self-attention module respectively;

[0021] S222, then use the entity feature vector within the group as the query and value of the multi-head cross attention module, and use the query of another group of multi-head self-attention modules as the key of this group of multi-head cross attention modules to generate inter-group entity encoding features;

[0022] S223, repeat the process from S221 to S222 twice, so that the different dimensions of the feature vectors are fully crossed, and the final intra-group coding features and inter-group coding features of the two groups of units are generated;

[0023] S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain the final entity coding features of the friendly and enemy sides;

[0024] S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length. After stacking with the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and other numerical features and Embedding features, a partially observable game state encoding vector is formed.

[0025] Preferably, in S23, an inference module is constructed based on the Residual LSTM network, and a part of the considerable game state encoding vector is transformed by the FC layer with the ReLU activation function and sent to the Residual LSTM network to obtain a hidden layer encoding vector that takes into account the historical time series information, so as to extract higher-level abstract semantic features and establish long-term decision-making correlations, thereby obtaining more reasonable multi-head compound actions.

[0026] Preferably, in S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture, and the distributed training framework includes a sampler, a sample cache, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine and calls the data in the sample cache for training.

[0027] Preferably, in S3, the method for training using the distributed training framework includes the following steps:

[0028] S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and in parallel generate a large number of trajectory samples through their respective actors and the simulation environment, and store them in the sample buffer;

[0029] S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model and sends the updated network parameters to the parameter server;

[0030] S33, when the round of training ends, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronizes and updates the network parameters of the Actor in the sampler;

[0031] S34, repeat the above process until the entire training is completed.

[0032] Preferably, in S4, the off-policy data samples are reused through the SARD-PPO algorithm, and the stability of the algorithm is ensured through the "element-level" double-end pruning mechanism and the adaptive adjustment reuse of the entire policy trajectory.

[0033] The present invention provides a method for constructing a collaborative combat decision-making intelligent agent for ground unmanned equipment. This method addresses the problem of dimensionality curse in the solution space of complex collaborative combat tasks by constructing a two-layer decision-making architecture based on data-driven and knowledge-driven methods. While effectively compressing the DRL solution space and improving exploration efficiency, it also realizes intelligent decision-making for tasks of different granularities throughout the entire combat process. In response to the problem of generating combat command action strategies at the "task layer", a collaborative combat decision-making intelligent agent network model is constructed to achieve unified coordinated control of combat actions from effective representation of complex state inputs. A distributed training framework is constructed to accelerate sampling and training efficiency, thereby realizing high-throughput asynchronous parallel training of the collaborative combat decision-making intelligent agent network model. By establishing and solving a combat decision-making intelligent agent learning algorithm based on deep reinforcement learning, the learning efficiency and final performance of the intelligent agent network model are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the network model framework of the ground unmanned equipment collaborative combat decision-making intelligent agent in the present invention;

[0035] Figure 2 Schematic diagram of the network structure of Residual LSTM in the present invention;

[0036] Figure 3 A schematic diagram of the process of completing a complete training process for the sampling training framework;

[0037] Figure 4 The figure is a flow chart of sampling by a sampler. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent, the improvement of which is that it includes the following steps:

[0040] S1: Build a two-tier decision-making architecture based on data-driven and knowledge-driven decision-making to achieve intelligent decision-making at two different granularities: the mission layer and the execution layer during combat;

[0041] S2, based on the decision-making process of Markov process, builds a network model of collaborative combat decision-making intelligent agents;

[0042] S3, by building a distributed training framework, enables high-throughput asynchronous parallel training of collaborative combat decision-making agent network models;

[0043] S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy;

[0044] S5, conduct simulation tests on the simulation deduction platform to verify the feasibility of the technical route.

[0045] Furthermore, in S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data, and the macro-combat command action strategy includes maneuvering and attacking; the execution layer is used to respond to the macro-combat command action strategy issued by the task layer, and the response includes artillery adjustment, ammunition type selection, firing, acceleration, deceleration, steering, local path planning, and obstacle avoidance.

[0046] In this embodiment, the ground unmanned combat equipment is a complex control system, and its maneuvering and attack behaviors involve continuous, high-dimensional action spaces. For example, maneuvering involves continuous action spaces such as the equipment's throttle and movement direction angle, while attacking involves continuous action spaces such as artillery aiming angles, as well as discrete action spaces such as whether to fire. Furthermore, coordinated ground unmanned combat is a typical multi-factor, extremely complex cooperative task. In a coordinated combat scenario, taking 12 combat units on each side as an example, if fine-grained dynamic basic actions such as artillery aiming angle control and whether to fire are abstracted into macroscopic, coarse-grained attack behaviors, the decision space for attack behaviors is 1312 ≈ 2 * 1013. If continuous actions such as artillery aiming angle control are directly considered, the decision space will grow exponentially and become difficult to estimate. Therefore, for complex systems and complex tasks, especially when large-scale complex system groups face complex tasks such as coordination, as the group size and problem complexity increase, existing end-to-end learning models or rule-based empirical knowledge will find it difficult to fully cover the extremely large solution space of the entire complex group's actions. In this context, a reasonable task-oriented intelligent decision-making architecture becomes particularly important.

[0047] Coordinated operations for ground unmanned equipment primarily rely on battlefield situational awareness, making assessments of the battlefield situation, and outputting operational strategies for each combat unit. Specifically, for each combat unit, operational actions are the macroscopic manifestation of its dynamic fundamental motions. This embodiment proposes a top-down, two-tier intelligent decision-making architecture for coordinated operations for ground unmanned equipment. Unmanned equipment actions are divided into different levels based on the granularity of the actions. The decision-making and control authority of the lower levels is derived from the commands output by the upper levels, thereby reducing the complexity of problem solving. Specifically, the coordinated operations of ground unmanned equipment are divided into two levels: the "task layer" and the "execution layer." The "task layer" primarily describes macroscopic mission requirements and generates macroscopic operational command strategies, such as maneuver and strike, that reflect the coordinated operations of combat units based on battlefield situation data. The "execution layer" responds to the commands from the "task layer." Unmanned equipment with autonomous operational response capabilities implements these macroscopic commands from the "task layer" through dynamic fundamental motions. These primarily represent individual equipment actions such as gun maneuvering, ammunition type selection, firing, acceleration, deceleration, steering, local path planning, and obstacle avoidance.

[0048] Given that data-driven methods such as DRL can achieve large-scale exploration of the solution space and optimized iteration of strategies without precise modeling, and knowledge-driven methods such as mechanism models, priors or rules often have better performance for deterministic tasks, this embodiment uses a DRL data-driven method and a knowledge-driven method to make intelligent decisions on actions of two different granularities, the "task layer" and the "execution layer" during the entire combat process when coordinating operations with ground unmanned equipment.

[0049] As we all know, the complexity of the DRL algorithm solution space is determined by both the state space and the action space. Without considering the state space, its continuous action space is likely to form an infinite solution space. At the same time, it is quite difficult for the DRL algorithm to learn macro actions consisting of a series of basic actions by relying solely on random exploration in the absence of a targeted reward function design. Even if it is learned, it is difficult to reproduce and apply it continuously and stably. Using a top-down two-layer intelligent decision-making architecture, we directly use a knowledge-driven approach to model how to maneuver and how to attack. While realizing the discretized representation of the DRL algorithm action space, we also include macro actions with advanced skill combinations, which can effectively compress the dimensions of the solution space, improve exploration efficiency and final performance.

[0050] Furthermore, in S2, the method for constructing the collaborative combat decision-making agent network model includes:

[0051] S21, define the state space, action space and establish the reward function;

[0052] S22, encodes the multi-feature information based on the state space and action space input to obtain a partially observable game state encoding vector;

[0053] S23, deriving multi-head compound actions through partially observable game state encoding vectors;

[0054] S24 decodes multiple compound actions to arrive at the final combat decision.

[0055] In this embodiment, the purpose of ground unmanned equipment coordinated combat is to minimize losses to the enemy while eliminating as many enemy targets as possible through the coordinated cooperation of unmanned equipment. This embodiment defines the ground unmanned equipment coordinated combat process as a sequential decision problem based on the Markov decision process (MDP). The MDP is usually defined as a 5-tuple M =<S,A,P,R,γ> , where: S and A represent a finite set of states and a finite set of actions, respectively, s t ∈S and a t ∈A represents the state and action performed at the moment; P and R are the state transfer function and reward function, respectively, which model the uncertainty of the action and the environment. Indicates that in the current state s t Next, perform action a t Next state s t+1 The probability of transitioning to s', r t =R(s t ,a t ) is in state s t Take action a t The immediate reward obtained; γ∈[0,1] is the discount factor, which reflects the value ratio of future rewards at the current moment.

[0056] In the process of coordinated combat, unmanned equipment continuously observes the battlefield environment and relies on communication to form joint situational awareness, obtains the state s, and generates the optimal coordinated maneuver or coordinated attack and other composite actions a according to the combat command action strategy π(a|s), so that it can change the current battlefield situation while completing the confrontation with the entities participating in the game. This is repeated, through multiple steps of decision-making, until the final combat mission is completed. In the process of interacting with the battlefield simulation environment, the combat intelligent agent maximizes the cumulative discounted return through trial and error learning. Through the formal description of the collaborative combat process, a one-to-one correspondence between the battlefield situation set, collaborative action set, battlefield situation transfer, game confrontation action feedback, and collaborative combat objectives in the collaborative combat confrontation environment and the various elements in the MDP is achieved.

[0057] Furthermore, in S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information of all different types of own ground unmanned equipment and drones, and the enemy entity information includes information of tanks, infantry fighting vehicles, soldiers, and drones; the action space is a collection of all actions that can be performed by different unmanned combat equipment during collaborative combat, and the action space is defined by composite actions.

[0058] In this embodiment, the state space in the MDP problem represents the environmental information perceived by the agent and its dynamic changes. This is the prerequisite for the DRL algorithm to generate strategies and evaluate cumulative rewards. In the coordinated combat of ground unmanned equipment, the state space mainly reflects elements such as the position coordinates, status, and environmental information of the combat entity.

[0059] In view of the different combat units, unit attributes, and state attributes of both sides, this embodiment uses scalar information and entity information to define the state space attributes of the combat process. Scalar information primarily includes statistical information such as combat duration and remaining force value. Entity information, consisting of both friendly and enemy components, represents the current actual state of each combat unit on the battlefield. Friendly entity information includes information on all different types of friendly ground unmanned equipment and drones. Since rear-line combat command vehicles do not directly engage in combat, they are not included in friendly entity information. Enemy entity information includes information on tanks, infantry fighting vehicles, soldiers, and drones. Due to the fog of war, both combatants can only obtain partial entity information of the enemy's detected targets and cannot understand the situational information of undetected areas or entities. A detailed description of the state space information is shown in Table 1.

[0060] Table 1 Description of the state space of ground unmanned equipment collaborative combat

[0061]

[0062] The action space refers to the set of all possible actions that different unmanned combat equipment can perform during collaborative operations. The decision-making action outputs for ground unmanned equipment collaborative operations target "task-level" macro-combat command actions that reflect the coordinated reconnaissance, coordinated maneuvering, and coordinated strikes of unmanned equipment. To help intelligent agents fully explore various possibilities in the environment, avoid inaccessible "state blind spots," and ensure good accessibility to the optimal solution area, this paper describes the action space using a composite action approach. This ensures that the intelligent agent has the ability to explore the complete task solution space while reducing the difficulty of exploration, accelerating the convergence of the DRL algorithm and improving the final algorithm performance.

[0063] Each composite action output by the combat decision-making agent is discrete and highly structured. First, the action type is selected. The primary actions for assault unmanned vehicles are maneuver and strike, while those for drones are route maneuver patrol reconnaissance and area maneuver patrol reconnaissance. Second, any subset of friendly combat units is selected to issue the action to a target, which can be a specific location in the battlefield environment or an enemy target. Finally, these sub-actions are combined into action commands that can be received by the unmanned combat equipment, achieving coordinated action at the "task level." The action commands specify parameters describing the action, such as the reconnaissance area range (Range), maximum speed (Max_speed), and weapon (Weapon). Because our decision-making architecture outputs action decisions at equal intervals, it does not determine when the next action will be taken. Table 2 describes the specific action space.

[0064] Table 2 Description of the ground unmanned equipment collaborative combat action space

[0065]

[0066] Furthermore, in S21, the reward function is a continuity function based on the process reward, and the process reward r of the agent at each time step is t P It is divided into target discovery and loss, loss of troop value and equipment damage, and change of own position. The process reward is r t P The calculation formula is:

[0067]

[0068] in, A local reward for each new enemy combat unit discovered, which is positively correlated with the target's value;

[0069] The penalty for losing a discovered target due to maneuvering or other actions is negatively correlated with the target's value. If the enemy target is completely destroyed, there is no penalty for losing the target.

[0070] For each enemy target hit and damaged, a reward of enemy force loss is obtained, which is positively correlated with the target's damage status and target value;

[0071] The additional damage bonus obtained when the target is completely destroyed;

[0072] This is the penalty incurred when an enemy target hits and causes damage during combat, which is negatively correlated with the degree of damage and the value of the equipment itself.

[0073] This is the additional damage penalty you receive when your side is completely destroyed.

[0074] k a,t 、k a,t-1 They represent the degree of damage to the combat forces at time steps t and t-1 respectively, and m is the number of equipment on the side that has not been completely destroyed;

[0075] The calculation formula is:

[0076] Among them, l a_max is the effective strike distance of each combat equipment, [x g ,y g ] is the target location of the enemy position set in advance based on reliable intelligence, [x j,t ,y j,t ] is the position of each combat unit at time step t.

[0077] In this embodiment, rewards serve as a key factor in guiding reinforcement learning training and strategy optimization iterations, directly affecting whether the training of the intelligent agent can converge, the convergence direction, the training efficiency, and the final performance. Combat missions are typical scenarios where only victory or defeat is the final reward. The final reward is a description of the final result of the coordinated combat, which truly reflects the goal of the ground unmanned equipment coordinated combat. However, this reward is extremely sparse and delayed, and will only appear when and only when the combat victory or defeat determination conditions are met. Its general form is shown in formula (1), that is, a large positive reward r is given when the mission goal is achieved. win , a large penalty is imposed on failure -r win , there is no reward during the completion process. If this reward is directly used as a feedback guidance signal for training, it will be difficult for the agent to directly obtain reward feedback through simple actions or action sequences. This is not conducive to the formation of local knowledge, and it is difficult to provide guidance for exploration direction and learn effective strategies.

[0078]

[0079] To overcome the end game reward r T To solve the problem of sparse rewards and delays caused by the decomposition of mission objectives, this embodiment adopts a reward reshaping method based on mission objective decomposition. According to the characteristics of combat mission objectives and different types of combat equipment, the final main mission objectives are divided into several process sub-goals. Appropriate positive rewards and penalties are allocated according to their respective contribution and direction in the combat main mission objectives, thereby forming a denser process reward r PRealize the real final reward r T reshape, guiding the intelligent agent to explore more efficiently and accelerate strategy convergence, so that it can learn some effective action strategies and tactics as quickly as possible.

[0080] According to the characteristics of each unmanned equipment in the collaborative combat mission, the process reward r of the agent at each time step is t P The rewards or penalties are divided into multiple sub-goals such as target discovery and loss, loss of troop value and equipment damage, and position change. For the target discovery and loss sub-goals, in order to encourage effective reconnaissance by drones and ground unmanned equipment, each time a new enemy combat unit is discovered, a local reward positively correlated with the value of the target can be obtained. At the same time, if the target is lost due to maneuvering or other actions, a penalty will be imposed that is negatively correlated with the value of the target. When the enemy target is completely destroyed, there is no penalty for the loss of the target. Ground unmanned equipment is mainly responsible for the strike mission. In order to encourage its effective strike behavior, each time it hits an enemy target and causes damage, it can obtain an enemy force loss reward that is positively correlated with the target's damage status and target value. At the same time, in order to encourage the complete destruction of the target, when the target is completely destroyed, an additional damage reward will be obtained. Similarly, in order to minimize the combat losses of one's own side, if an enemy target is hit and damaged during combat, a penalty will be incurred that is negatively correlated with the degree of damage and the value of the equipment itself. When completely destroyed, additional damage penalty will be incurred. In addition, in order to guide the combat agent to learn a better action decision strategy faster, a continuity function is designed to guide the unmanned combat equipment to reach the pre-established enemy position area. Its expression is shown in formula (2), where l a_max It is expressed as the effective strike range of each combat equipment (for UAVs, it is the effective reconnaissance range), [x g ,y g ] is the target location of the enemy position set in advance based on reliable intelligence, [x j,t ,y j,t ] is the position of each combat unit at time step t, and m is the number of equipment that has not been completely destroyed. In summary, the process reward r of the combat agent at each time step is t P It can be shown by formula (3), where k a,t 、k a,t-1 They represent the degree of damage to the combat forces at time steps t and t-1 respectively.

[0081]

[0082]

[0083] in, Rewards and penalties for target discovery and loss, Rewards for enemy force loss and total destruction. Penalty for loss of value of own troops and complete destruction, Rewards and penalties for equipment position changes.

[0084] Furthermore, in S22, by using a multi-head attention mechanism, a multi-layer perceptron, an embedding layer, and a fully connected layer, the global scalar information that can reflect the entire battlefield situation, the actions performed by the agent in the previous step, and the attribute feature information reflecting each entity are extracted and integrated, thereby obtaining a partially observable game state encoding vector. Specifically, the following steps are included:

[0085] S221, for both friendly and enemy combat units, use the entity feature vectors in the same group as the query, key, and value of the multi-head self-attention module respectively;

[0086] S222, then use the entity feature vector within the group as the query and value of the multi-head cross attention module, and use the query of another group of multi-head self-attention modules as the key of this group of multi-head cross attention modules to generate inter-group entity encoding features;

[0087] S223, repeat the process from S221 to S222 twice, so that the different dimensions of the feature vectors are fully crossed, and the final intra-group coding features and inter-group coding features of the two groups of units are generated;

[0088] S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain the final entity coding features of the friendly and enemy sides;

[0089] S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length. After stacking with the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and other numerical features and Embedding features, a partially observable game state encoding vector is formed.

[0090] In this embodiment, when the decision-making problem of coordinated combat of unmanned ground equipment is regarded as a deep reinforcement learning problem, there will be the following challenges in the process of building the decision-making model: 1) Multi-feature information input. The input of the network model includes not only the attribute information of each entity and the global scalar information reflecting the battlefield situation, but also the action performed by the agent in the previous step and some masks reflecting the battlefield environment and entity logic. 2) Incomplete information game. The limited perception capabilities of the combat equipment's own sensors and the existence of the fog of war result in the agent's environment being partially observable, and the decision sequence in the entire combat process is long. 3) Complex action and decision space. The decision output of each step is a highly structured action coupled with multiple sub-actions such as behavior action type selection, execution unit selection, and target selection.

[0091] In order to deal with the complexity and challenges in the construction of the above-mentioned combat decision-making agent neural network model, and at the same time be able to better control all combat units in the unmanned combat team and avoid the problem of inconsistent coordination between different types of unmanned equipment, this embodiment constructs a large single "commander" type ground unmanned equipment collaborative combat decision-making agent network model similar to AlphaStar to achieve the effective representation of state characteristics and unified coordinated control of combat actions. The ground unmanned equipment collaborative combat decision-making agent network model framework is referenced. Figure 1 shown.

[0092] The effective encoding and representation of input information is the key to achieving intelligent agent decision-making and subsequent strategy optimization. For complex tasks with high state space dimensions such as collaborative combat, the limited representation capabilities of shallow networks make it difficult to achieve effective feature extraction. Therefore, it is necessary to design a deeper and more complex representation network in combination with state space and task characteristics. This embodiment mainly distinguishes between fixed-length and variable-length input feature information, and uses a multi-head attention mechanism, a multi-layer perceptron (MLP), an embedding layer, and a fully connected (FC) layer to extract and fuse global scalar information that can reflect the entire battlefield situation, the actions performed by the previous intelligent agent, and the attribute feature information reflecting each entity. The specific implementation details are as follows.

[0093] When encoding the attribute information of the entity, unlike AlphaStar which uniformly encodes all entity information, this embodiment encodes the two groups of combat units, one for oneself and one for the enemy, independently. For each group of combat units, the entity feature vectors within the same group are used as the query, key, and value of the multi-head self-attention module to generate intra-group entity coding features; then the entity feature vectors within the group are used as the query and value of the multi-head cross-attention module, and the query of another group of multi-head self-attention modules is used as the key of the multi-head cross-attention module of this group to generate inter-group entity coding features. The whole process is repeated twice to allow the different dimensions of the feature vectors to fully cross and generate the final intra-group coding features and inter-group coding features of the two groups of units. The intra-group coding features and inter-group coding features of each group are spliced ​​together to obtain the final entity coding features of the two groups, the enemy and the enemy.

[0094] The final entity encoding features of both the enemy and friendly groups serve as attention keys for unit or target selection, and are compressed into a fixed-length entity feature vector. This is then stacked with numerical features such as the encoding vector of global scalar information and the encoding vector of the previous action, as well as embedding features, to form a partially observable encoding vector of the game state. Statistical global scalar numerical features such as troop strength and combat time are concatenated and directly fed into the MLP for representation. Categorical features such as the agent's previous action are first fed into the embedding layer, where the sparse one-hot encoding vectors are converted into dense embedding vectors before being concatenated.

[0095] The main reason for compressing the final entity encoding features is that although non-essential features have been eliminated when designing the input state space of DRL, some redundant features may still exist, which establishes a false correlation between some useless information and the model output, which easily causes overfitting of the policy network model. This embodiment uses the attention pooling operation to compress it, thereby removing redundant information and reducing the learning difficulty of DRL. Compared with average pooling and maximum pooling, although attention pooling requires an increase in the number of network layers and the amount of calculation, it can obtain the importance of different elements in the features through learning, thereby capturing the key information in the features. While achieving more refined feature extraction and combination than average pooling, it also avoids the information loss caused by the maximum pooling operation discarding all minor information and directly using the maximum value of the elements in the feature as the output, thereby improving the accuracy and generalization of the model.

[0096] Furthermore, in S23, an inference module is constructed based on the Residual LSTM network. Part of the considerable game state encoding vector is transformed by the FC layer with the ReLU activation function and then sent to the Residual LSTM network to obtain the hidden layer encoding vector that takes into account the historical time series information. In this way, higher-level abstract semantic features are extracted and long-term decision-making correlations are established, thereby obtaining more reasonable multi-head compound actions.

[0097] In this embodiment, the actual confrontation process of ground unmanned equipment coordinated combat is different from a simple game scene. First, it often takes a long time to conduct battlefield operations and layout to form a favorable battlefield situation. Therefore, the time span is long and the number of decision steps is large. Second, the combat decision-making intelligent agent is not a simple stress-response decision-making system. It needs to consider the strategies to be adopted in time and space according to the combat objectives and the observation information obtained. For example, what kind of attack and maneuver strategies should be adopted in the early, middle and late stages of the confrontation, how to use the terrain advantages, etc. Therefore, the decision output has a strong time dependence. At the same time, due to the limited perception capability of the equipment, the information in the coordinated combat process is incomplete and uncertain, and there is a "fog of war". Therefore, it is necessary to make decisions under the condition of some considerable incomplete information.

[0098] In order to address the decision output issues of strong temporal dependency and incomplete information during collaborative operations, as well as the vanishing gradient problem of neural networks caused by a large number of decision steps, this embodiment uses the Residual LSTM network to construct an inference module. After transforming some of the significant game state encoding vectors through the FC layer with the ReLU activation function, it is fed into the Residual LSTM network to obtain the hidden layer encoding vector that takes into account historical temporal information. This allows for the extraction of higher-level abstract semantic features and the establishment of long-term decision relevance. The network structure and calculation formula of the Residual LSTM are as follows: Figure 2 As shown in equations (4)-(10), where l represents the index of each layer of LSTM, Represent the input gate, forget gate and output gate of this layer respectively. Represent the hidden layer output and internal memory unit state at time t-1, is the hidden layer output from the l-1th layer (when l=1, it is the transformed game state encoding vector), the projection matrix To reduce r t l dimension.

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] The main idea of ​​Residual LSTM is to combine LSTM units with residual networks, capture the dynamic characteristics of the time series by running LSTM units between residual blocks, and add an LSTM layer after the last block to fully capture the context of the entire sequence. Specifically, at time t, the Residual LSTM network uses the current information input at each layer and the LSTM network. And the hidden layer state of the previous moment As input, the internal memory unit C is updated by selectively discarding old information and receiving new information through the forget gate f and input gate i, ensuring the effective transmission of information in time sequence. The only difference is that the Residual LSTM provides a spatial shortcut path between layers through the residual network, which transfers the hidden layer state of the previous layer to the next layer. and the state of the memory unit in this layer Used together with the output gate for hidden layer states This avoids the vanishing gradient problem between LSTM layers.

[0107] In S24, to address the logical relationships and decision output issues in a highly structured multi-head compound action space, this embodiment draws on the autoregressive action structure in the AlphaStar network architecture. The output of each subsequent action head is conditioned on the output of the previous action head, thereby transforming an N-dimensional action decision problem to be performed in a given state into a sequence of N one-dimensional actions for processing. Intuitively, this is logical, as different action types require different combat units to implement and act on different enemy targets or locations. Given that AlphaStar's autoregressive embedding layer, which uses an addition operator to connect the outputs of all action heads, has limited expressive power, this embodiment employs an autoregressive network structure that first concatenates the encoding vector that considers historical information with the encoding vector of the previous action, followed by a fully connected layer, to provide more robust expressive power. A pointer network and an attention network are used to select combat units and targets. The output of an MLP layer with a residual structure in the action head's autoregressive network is used as a query. The final entity encoding features of the two groups of units are used as attention keys to output the sampling probability distributions for combat units and targets, respectively. The outputs of the remaining action heads are sampled using an MLP layer with a residual structure, followed by a Softmax layer. Because the pointer network calculates the weight of each element in the input sequence through an attention mechanism and directly uses the elements of the input sequence for output, the output of the combat unit action head does not need to be connected to a Softmax layer.

[0108] Of particular note, this embodiment does not use spatial map information to select target locations. Instead, it divides the entire battlespace into N×M coarse-grained regions and defines each numbered coarse-grained region as a global position intention label. Therefore, the output of the target position action head at each time step is the global position intention label. Compared with AlphaStar's method of using spatial information, convolutional neural networks, and deconvolutional neural networks to determine target locations, this method effectively reduces the policy decision space and the size of the combat agent's policy network structure. Furthermore, to guide the rationality of action outputs during the reinforcement learning exploration process and improve training efficiency, this embodiment uses action masks based on expert prior knowledge to mask unreasonable action head outputs. For example: 1) In the early stages of combat (advance phase), enemy targets are undetected and out of effective range, and friendly combat units focus on mobile reconnaissance, so strike actions are masked. 2) Strike actions by combat units that do not meet the strike conditions (i.e., within the strike cooldown period, with no weapons or ammunition, out of effective range, or unmanned aerial vehicles) are masked. 3) Ground combat equipment such as unmanned vehicles do not have the ability to pass through or reach certain areas (water areas) in the battlespace.

[0109] Furthermore, in S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture, and the distributed training framework includes a sampler, a sample cache, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine and calls the data in the sample cache for training.

[0110] In order to effectively utilize limited computing resources to train the neural network of combat decision-making agents, this embodiment designs a large-scale distributed sampling training framework based on the Actor-Learner architecture to achieve high-throughput asynchronous parallel training, improve data efficiency and computing performance, and effectively accelerate training efficiency. Specifically, the sampling training framework consists of components such as samplers, sample buffers, learners, parameter servers, and controllers, and runs on 512 CPU cores and 8 GeForce RTX 3080 GPUs (with a total of 96GB of video memory) packaged in Dockers. Among them, the sampler is deployed on the CPU machine, allowing the actor to interact with a single combat simulation environment running in the sampler to generate sample trajectories; the learner is deployed on the GPU machine and consumes data in the sample buffer to train network parameters. Figure 3 It shows how the sampling training framework completes a complete training process.

[0111] Furthermore, in S3, the method for training using the distributed training framework includes the following steps:

[0112] S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and in parallel generate a large number of trajectory samples through their respective actors and the simulation environment, and store them in the sample buffer;

[0113] S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model and sends the updated network parameters to the parameter server;

[0114] S33, when the round of training ends, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronizes and updates the network parameters of the Actor in the sampler;

[0115] S34, repeat the above process until the entire training is completed.

[0116] In the above process, unlike the synchronous training method, each sampler performs sampling according to its own progress, that is, after the environment with fast sampling is finished, the sampled data is first stored in the sample buffer and the next round of sampling starts immediately, without waiting for other samplers, and the Actor in the sampler is only responsible for sampling and is not used to calculate gradients, thus greatly improving the sampling efficiency. However, this sampling training framework also causes the target strategy in the learner to be updated earlier than the behavior strategy in the sampler, resulting in a policy lag between the behavior strategy and the target strategy. In addition, the following logic is followed during the sampling phase: 1) Obtain the current state data through the environment adapter; 2) Convert the action to be executed into instructions that can be received by the simulation environment through the action converter; 3) Calculate the reward for each decision step through the reward calculator; 4) Process the state data, action, and reward into sample data through the sample processor. The specific simulation environment scheduling process is as follows: Figure 4 shown.

[0117] Furthermore, in S4, the SARD-PPO algorithm is used to reuse off-policy data samples, and the stability of the algorithm is ensured through the "element-level" dual-end pruning mechanism and the adaptive adjustment and reuse of the entire policy trajectory.

[0118] In this embodiment, to enable the combat decision-making agent to generate a more effective collaborative combat strategy, a SARD-PPO algorithm that introduces policy entropy is proposed to solve the Markov decision model, thereby obtaining a reasonable ground unmanned equipment collaborative combat strategy. Because this embodiment uses asynchronous sampling and a large-scale distributed approach for training—that is, new trajectory samples are generated by the sampler, while the learner asynchronously updates the model parameters using old trajectory samples in the sample buffer—the combat decision-making agent learning problem is essentially an off-policy reinforcement learning problem. Especially in a large-scale off-policy training environment, the sample trajectories of the previous policy and the current policy may differ significantly. Directly using an on-policy reinforcement learning algorithm to learn from the old trajectory samples in the sample buffer can easily cause the policy model to fluctuate significantly, or even fail to converge.

[0119] The PPO algorithm is an easy-to-implement, efficient and popular on-policy reinforcement learning algorithm that only requires first-order optimization and has been proven to be effective in solving a series of challenging large-scale problems. The PPO algorithm uses the Actor-Critic framework, where the Actor network is used to generate the policy model π. θ (a|s), its strategy replacement objective optimization function is:

[0120]

[0121] Among them, π and πc denote the future strategy and the current strategy respectively, τ denotes the strategy sample trajectory, A(s t ,a t ) represents the estimated advantage function, r t (θ)=π(a t |s t ) / π c (a t |s t ) represents the probability ratio between the new and old strategies, and ε is the clipping factor. The Critic network is used to estimate the value function V ω (s t ), and uses the traditional TD-error method to update its network parameters.

[0122] In order to effectively utilize off-policy data samples caused by policy lag to update the policy while ensuring training stability, this embodiment adopts a more principled pruning mechanism, the SARD-PPO algorithm, to reuse off-policy data samples. The stability of the algorithm is ensured by the "element-level" double-end pruning mechanism and the adaptive adjustment and reuse of the entire policy trajectory. Its policy replacement objective function is:

[0123]

[0124]

[0125] Among them, π c-l Represents the distance from the current policy π c The lth previous strategy, b represents the lower bound parameter of double-ended pruning, Meanwhile, SARD-PPO uses the previous strategy π c-l Deviation from the current policy π over the entire trajectory c Expectations Whether the adaptive adjustment uses the previous policy π c-l The generated old samples are used to update the current strategy.

[0126]

[0127] In addition, in order to address the problem of exploration difficulties caused by the complexity of the land battlefield and the sparseness of rewards during the strategy optimization process, this embodiment introduces a strategy entropy-encouraged exploration mechanism to enhance the agent's exploration ability. The calculation method of strategy entropy is defined as:

[0128]

[0129] Then the policy substitution objective optimization function of the SARD-PPO algorithm with added policy entropy is:

[0130]

[0131] To reduce the number of parameters in the entire network model of the combat decision-making agent, the combat decision-making agent's value function network shares the network structure and weights before the policy network's Residual LSTM output. It uses the hidden state information of the final LSTM layer as input, and after transformation through an MLP layer, predicts the state value function. To reduce the variance of the value function estimate, this embodiment uses all information in the game state, encoding the hidden observation information and providing it as input to the value function network.

[0132] Using V-trace method with truncated importance sampling to achieve The multi-step advantage correction estimate is obtained by minimizing and target value The mean square error is used to update the value function network parameter ω of the current strategy:

[0133]

[0134]

[0135] in, ρ t and c t Importance sampling weights for truncation.

[0136] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0137] It should be noted that the terms used herein are intended only to describe specific embodiments and are not intended to limit the exemplary embodiments described herein. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0138] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0139] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0140] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be subsequently positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0141] In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless the context dictates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0142] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent, characterized by: The steps include: S1: Build a two-tier decision-making architecture based on data-driven and knowledge-driven approaches to achieve intelligent decision-making at both the mission and execution levels during combat. S2, based on the decision-making process of the Markov process, constructs a collaborative combat decision-making agent network model; the construction method of the collaborative combat decision-making agent network model includes: S21, define the state space, action space and establish a reward function; the reward function is a continuity function based on the process reward, and the process reward of the agent at each time step is It is divided into target discovery and loss, loss of troop value and equipment damage, change of own position, and process rewards. The calculation formula is: ， in, A local reward for each new enemy combat unit discovered, which is positively correlated with the target's value; The penalty for losing a discovered target due to maneuvering or other actions is negatively correlated with the target's value. If the enemy target is completely destroyed, there is no penalty for losing the target. For hitting an enemy target and causing damage at the current moment, you will receive a reward of enemy force loss value that is positively correlated with the target's damage status and target value at the current moment; The additional damage bonus obtained when the target is completely destroyed; This is the penalty incurred when an enemy target hits and causes damage to your side during combat, which is negatively correlated with the damage level and the equipment's own value. This is the additional damage penalty you receive when your side is completely destroyed. The number of your own equipment that has not been completely destroyed; The calculation formula is: , in, is the effective strike distance of each combat equipment, The target location of the enemy position is pre-set based on reliable intelligence. For each combat unit The own position at the time step; S22, encoding multiple feature information based on state space and action space input to obtain a partially observable game state encoding vector; S23, deriving multi-head compound actions through partially observable game state encoding vectors; S24 decodes multiple complex actions to arrive at the final combat decision; S3, by building a distributed training framework, enables high-throughput asynchronous parallel training of collaborative combat decision-making agent network models; S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy; S5, conduct simulation tests on the simulation deduction platform to verify the feasibility of the technical route.

2. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data. The macro-combat command action strategy includes maneuvering and attacking. The execution layer is used to respond to the macro-combat command action strategy issued by the task layer. The response includes adjusting artillery, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and obstacle avoidance.

3. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information on all different types of own ground unmanned equipment and drones, and the enemy entity information includes information on tanks, infantry fighting vehicles, soldiers, and drones; the action space is the set of all actions performed by different unmanned combat equipment during collaborative combat, and the action space is defined by composite actions.

4. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S22, a multi-head attention mechanism, a multi-layer perceptron, an embedding layer, and a fully connected layer are used to extract and fuse the global scalar information that reflects the entire battlefield situation, the actions performed by the agent in the previous step, and the attribute feature information reflecting each entity, thereby obtaining a partially significant game state encoding vector. Specifically, the following steps are included: S221, for both friendly and enemy combat units, use the entity feature vectors in the same group as the query, key, and value of the multi-head self-attention module respectively; S222, then use the entity feature vector within the group as the query and value of the multi-head cross attention module, and use the query of another group of multi-head self-attention modules as the key of the multi-head cross attention module to generate inter-group entity encoding features; S223, repeat the process from S221 to S222 twice, so as to fully cross the different dimensions of the feature vectors and generate the final intra-group coding features and inter-group coding features of the two groups of units; S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain the final entity coding features of the friendly and enemy sides; S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length. After stacking with the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and the Embedding feature, a partially observable game state encoding vector is formed.

5. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S23, an inference module is constructed based on the Residual LSTM network. Part of the considerable game state encoding vector is transformed by the FC layer with the ReLU activation function and sent to the Residual LSTM network to obtain the hidden layer encoding vector that takes into account the historical time series information. In this way, higher-level abstract semantic features are extracted and long-term decision-making correlations are established, thereby obtaining more reasonable multi-head compound actions.

6. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture. The distributed training framework includes a sampler, a sample cache, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine and calls the data in the sample cache for training.

7. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 6, characterized in that: In S3, the method for training using the distributed training framework includes the following steps: S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and in parallel generate a large number of trajectory samples through their respective actors and the simulation environment, and store them in the sample buffer; S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model and sends the updated network parameters to the parameter server; S33, when this round of training ends, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronizes and updates the network parameters of the Actor in the sampler; S34, repeat the above process until the entire training is completed.

8. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S4, the SARD-PPO algorithm is used to reuse off-policy data samples, and the stability of the algorithm is ensured through the element-level double-end pruning mechanism and the adaptive adjustment reuse of the entire policy trajectory.

Images