Construction method of cooperative combat decision-making agent for ground unmanned equipment

Abstract

The invention relates to a construction method of a cooperative combat decision-making agent for ground unmanned equipment, and aims to solve the problem of space dimension disaster of a complex cooperative combat task. The DRL solution space is effectively compressed, the exploration efficiency is improved, and meanwhile intelligent decision making of tasks with different granularities in the whole combat process is achieved; aiming at the problem of generating a'task layer 'combat command action strategy, a cooperative combat decision-making agent network model is constructed to realize the unified coordination control from the effective representation of complex state input to the combat action; by constructing a distributed training framework, the sampling and training efficiency is improved, and high-throughput asynchronous parallel training of a cooperative combat decision-making agent network model is realized; by establishing a combat decision agent learning algorithm based on deep reinforcement learning and solving the combat decision agent learning algorithm, the learning efficiency and final performance of an agent network model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a method for constructing a ground unmanned equipment collaborative combat decision-making intelligent body. Background Art

[0002] Cooperative combat of ground unmanned equipment refers to a combat method in which two or more ground unmanned equipment cooperate and collaborate with each other with the support of drones to complete a specific ground military mission in an information-based, unmanned, networked and clustered confrontation environment, including coordinated maneuvers, coordinated reconnaissance and coordinated strikes. In order to improve the self-organizing ability, effective coordination ability and success rate of combat missions of unmanned equipment in the face of complex tasks, many researchers have conducted in-depth research on decision-making methods that help improve the cooperative combat of unmanned equipment, in order to cope with various threats in a game-intensive confrontation environment, and strive to use the advantages of decision-making to defeat the opponent and win the war in the future battlefield environment. Compared with the autonomous combat decision-making of a single unmanned equipment, the cooperative combat of multiple unmanned equipment has the outstanding characteristics of huge decision-making space, incomplete information, diverse equipment types, and dynamic strong confrontation.

[0003] At present, the mainstream autonomous collaborative combat decision-making methods based on artificial intelligence (AI) are mainly divided into two categories: knowledge-driven and data-driven. Knowledge-driven methods usually use methods such as behavior trees, finite state machines, and goal-oriented action plans to model behavioral strategies using existing experience, rules, or models and algorithms. They have good interpretability, reusability, and high algorithm execution efficiency. However, when facing large-scale group complex collaborative tasks, such methods are difficult to achieve complete coverage of the huge solution space of cluster collaborative behavior, and cannot achieve continuous learning and iterative evolution of behavioral strategies. Data-driven methods mostly use methods such as artificial neural networks or deep reinforcement learning (DRL), which can achieve large-scale coverage and exploration of the solution space and optimization and iteration of strategies without precise modeling. However, this method has typical "black box" characteristics and requires a large amount of data for training. Summary of the invention

[0004] The present invention aims to provide a method for constructing a ground unmanned equipment collaborative combat decision-making intelligent body to solve the deficiencies in the prior art. The technical problem to be solved by the present invention is achieved through the following technical solutions.

[0005] A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent comprises the following steps:

[0006] S1, build a two-layer decision-making architecture based on data-driven and knowledge-driven, and realize intelligent decision-making of two different granularity tasks in the mission layer and execution layer during the combat process;

[0007] S2, based on the decision-making process of Markov process, builds a network model of collaborative combat decision-making agents;

[0008] S3, by building a distributed training framework, realizes high-throughput asynchronous parallel training of the collaborative combat decision-making agent network model;

[0009] S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy;

[0010] S5, conduct simulation tests on the simulation platform to verify the feasibility of the technical route.

[0011] Preferably, in S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data, and the macro-combat command action strategy includes maneuvering and attacking; the execution layer is used to respond to the macro-combat command action strategy issued by the task layer, and the response includes adjusting artillery, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and avoiding obstacles.

[0012] Preferably, in S2, the method for constructing the collaborative combat decision-making agent network model includes:

[0013] S21, define the state space, action space and establish the reward function;

[0014] S22, encoding the multi-feature information based on the state space and action space input to obtain a partially observable game state encoding vector;

[0015] S23, deriving multi-head compound actions through partially observable game state encoding vectors;

[0016] S24, decodes multiple compound actions to arrive at the final combat decision.

[0017] Preferably, in S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information of all different types of own ground unmanned equipment and drones, the enemy entity information includes information of tanks, infantry fighting vehicles, soldiers, and drones; the action space is a collection of all actions that can be performed by different unmanned combat equipment during coordinated combat, and the action space is defined by composite actions.

[0018] Preferably, in S21, the reward function is a continuity function based on the process reward, and the process reward r of the agent at each time step is t P It is divided into target discovery and loss, loss of manpower value and equipment damage, and change of own position.

[0019] Preferably, in S22, by using a multi-head attention mechanism, a multi-layer perceptron, an embedding layer and a fully connected layer, feature extraction and fusion are performed on the global scalar information that can reflect the entire battlefield situation, the action performed by the agent in the previous step, and the attribute feature information reflecting each entity, so as to obtain a partially observable game state encoding vector, which specifically includes the following steps:

[0020] S221, for the combat units of the own side and the enemy, the entity feature vectors in the same group are used as the query, key and value of the multi-head self-attention module respectively;

[0021] S222, then use the entity feature vector in the group as the query and value of the multi-head cross attention module, use the query of another group of multi-head self-attention modules as the key of the group of multi-head cross attention modules, and generate inter-group entity encoding features;

[0022] S223, repeat the process from S221 to S222 twice, so that the different dimensions of the feature vectors are fully crossed, and the final intra-group coding features and inter-group coding features of the two groups of units are generated;

[0023] S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain the final entity coding features of the own side and the enemy side;

[0024] S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length, which is stacked with numerical features such as the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and the Embedding features to form a partially observable game state encoding vector.

[0025] Preferably, in S23, an inference module is constructed based on the Residual LSTM network, and a part of the considerable game state encoding vector is transformed by the FC layer with the ReLU activation function and then sent to the Residual LSTM network to obtain a hidden layer encoding vector that takes into account the historical time series information, so as to extract higher-level abstract semantic features and establish long-term decision-making correlations, thereby obtaining more reasonable multi-head compound actions.

[0026] Preferably, in S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture, and the distributed training framework includes a sampler, a sample buffer, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine, and calls the data in the sample buffer for training.

[0027] Preferably, in S3, the method for training by the distributed training framework comprises the following steps:

[0028] S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and parallelly generate a large number of trajectory samples through their respective actors and continuously interact with the simulation environment, and store them in the sample buffer;

[0029] S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model, and sends the updated network parameters to the parameter server;

[0030] S33, when the round of training is terminated, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronously updates the network parameters of the Actor in the sampler;

[0031] S34, repeat the above process until the entire training is completed.

[0032] Preferably, in S4, off-policy data samples are reused through the SARD-PPO algorithm, and the stability of the algorithm is ensured through the "element-level" double-end pruning mechanism and the adaptive adjustment reuse of the entire policy trajectory.

[0033] The present invention provides a method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent. Aiming at the problem of dimensionality curse in the solution space of complex collaborative combat tasks, the present invention constructs a two-layer decision-making architecture based on data-driven and knowledge-driven, effectively compresses the DRL solution space to improve the exploration efficiency, and realizes intelligent decision-making for tasks of different granularities in the entire combat process. Aiming at the problem of generating combat command action strategies at the "task layer", the present invention constructs a collaborative combat decision-making intelligent agent network model to achieve unified coordinated control from effective representation of complex state inputs to combat actions. By constructing a distributed training framework, the sampling and training efficiency is accelerated, and high-throughput asynchronous parallel training of the collaborative combat decision-making intelligent agent network model is realized. By establishing a combat decision-making intelligent agent learning algorithm based on deep reinforcement learning and solving it, the learning efficiency and final performance of the intelligent agent network model are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the network model framework of the ground unmanned equipment collaborative combat decision-making intelligent agent in the present invention;

[0035] Figure 2 Schematic diagram of the network structure of Residual LSTM in the present invention;

[0036] Figure 3 A schematic diagram of the process of completing a complete training process for the sampling training framework;

[0037] Figure 4 The figure is a flow chart of sampling by a sampler. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent, the improvement of which is that it includes the following steps:

[0040] S1, build a two-layer decision-making architecture based on data-driven and knowledge-driven, and realize intelligent decision-making of two different granularity tasks in the mission layer and execution layer during the combat process;

[0041] S2, based on the decision-making process of Markov process, builds a network model of collaborative combat decision-making agents;

[0042] S3, by building a distributed training framework, realizes high-throughput asynchronous parallel training of the collaborative combat decision-making agent network model;

[0043] S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy;

[0044] S5, conduct simulation tests on the simulation platform to verify the feasibility of the technical route.

[0045] Furthermore, in S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data, and the macro-combat command action strategy includes maneuvering and attacking; the execution layer is used to respond to the macro-combat command action strategy issued by the task layer, and the response includes adjusting artillery, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and obstacle avoidance.

[0046] In this embodiment, the ground unmanned combat equipment is a complex control system, and its maneuvering and attacking behaviors involve continuous high-dimensional action spaces. For example, maneuvering involves continuous action spaces such as equipment throttle and moving direction angle, and attacking involves continuous action spaces such as artillery aiming angle and discrete action spaces such as whether to fire. At the same time, the coordinated combat of ground unmanned equipment is a typical multi-factor, extremely complex cooperative task. In the process of coordinated combat, taking 12 combat units on each side as an example, the fine-grained dynamic basic actions such as the control of the artillery aiming angle and whether to fire are abstracted into macroscopic coarse-grained attack behaviors, then the decision space of the attack behavior is 1312≈2*1013. If continuous actions such as artillery aiming angle control are directly considered, the decision space will grow exponentially and be difficult to estimate. Therefore, for complex systems and complex tasks, especially when large-scale complex system groups face complex tasks such as collaboration, as the group size and problem complexity increase, the existing end-to-end learning model or rule experience knowledge will be difficult to fully cover the large solution space of the entire complex group action. At this time, a reasonable task-oriented intelligent decision architecture will be particularly important.

[0047] The coordinated combat of ground unmanned equipment is mainly based on the battlefield environment situation, making judgments on the battlefield situation and outputting the combat action strategy of each combat unit. Specifically for each combat unit, the combat action is the macro-manifestation of its basic dynamic action. This embodiment proposes a top-down two-layer intelligent decision-making architecture for the coordinated combat of ground unmanned equipment. The actions of unmanned equipment are divided into different levels according to the coarseness of the action granularity. The decision-making and control rights of the lower layer come from the command output of its upper layer, thereby reducing the complexity of problem solving. Specifically, the actions of the coordinated combat of ground unmanned equipment are divided into two levels: "task layer" and "execution layer". The "task layer" mainly describes the macro-task requirements, and generates macro-combat command action strategies such as maneuvers and strikes that can reflect the cooperation of combat units based on battlefield situation data; the "execution layer" is a response to the "task layer" instructions. Unmanned equipment with autonomous action response capabilities realizes the "task layer" macro-instructions through basic dynamic actions, which are mainly manifested in single equipment actions such as adjusting guns, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and avoiding obstacles.

[0048] Given that data-driven methods such as DRL can achieve large-scale exploration of the solution space and optimization iteration of strategies without precise modeling, and knowledge-driven methods such as mechanism models, priors or rules often have better performance for deterministic tasks, this embodiment uses a DRL data-driven method and a knowledge-driven method to make intelligent decisions on actions of two different granularities, the "task layer" and the "execution layer" in the entire combat process when coordinating operations with ground unmanned equipment.

[0049] As we all know, the complexity of the DRL algorithm solution space is determined by the state space and the action space. Without considering the state space, its continuous action space is likely to form an infinite solution space. At the same time, it is quite difficult for the DRL algorithm to learn macro actions composed of a series of basic actions completely by random exploration in the absence of targeted reward function design. Even if it is learned, it is difficult to reproduce and apply it continuously and stably. Using a top-down two-layer intelligent decision-making architecture, we directly use a knowledge-driven approach to model how to maneuver and how to attack. While realizing the discretized representation of the DRL algorithm action space, we also include macro actions with advanced skill combinations, which can effectively compress the dimension of the solution space and improve exploration efficiency and final performance.

[0050] Furthermore, in S2, the method for constructing the collaborative combat decision-making agent network model includes:

[0051] S21, define the state space, action space and establish the reward function;

[0052] S22, encoding the multi-feature information based on the state space and action space input to obtain a partially observable game state encoding vector;

[0053] S23, deriving multi-head compound actions through partially observable game state encoding vectors;

[0054] S24, decodes multiple compound actions to arrive at the final combat decision.

[0055] In this embodiment, the purpose of ground unmanned equipment coordinated combat is to eliminate enemy targets as much as possible while minimizing the loss of one's own side through the coordinated cooperation of unmanned equipment. In this embodiment, the ground unmanned equipment coordinated combat process is defined as a sequential decision problem based on Markov decision process (MDP). MDP is usually defined as a 5-tuple M=<S,A,P,R,γ> , where: S and A represent a finite set of states and a finite set of actions, respectively, s t ∈S and a t ∈A represents the state and action performed at the moment; P and R are the state transfer function and reward function, respectively, which model the uncertainty of the action and the environment. Indicates that in the current state s t Next, perform action a t Next state s t+1 The probability of transitioning to s', r t =R(s t ,a t ) is in state s t Take action a t The immediate reward obtained; γ∈[0,1] is the discount factor, which reflects the value ratio of future rewards at the current moment.

[0056] In the process of coordinated operations, unmanned equipment continuously observes the battlefield environment and relies on communication to form joint situational awareness, obtains the state s, and generates the optimal coordinated maneuver or coordinated strike and other composite actions a according to the combat command action strategy π(a|s), so that it can change the current battlefield situation while completing the confrontation with the entities participating in the game. This is repeated, through multiple steps of decision-making, until the final combat mission is completed. In the process of interacting with the battlefield simulation environment, the combat agent maximizes the cumulative discounted return through trial and error learning. The combat command strategy converges to the optimal state. Through the formal description of the collaborative combat process, the one-to-one correspondence between the battlefield situation set, collaborative action set, battlefield situation transfer, game confrontation action feedback and collaborative combat objectives in the collaborative combat confrontation environment and the elements in the MDP is achieved.

[0057] Furthermore, in S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information of all different types of own ground unmanned equipment and drones, the enemy entity information includes information of tanks, infantry fighting vehicles, soldiers, and drones; the action space is a collection of all actions that can be performed by different unmanned combat equipment during collaborative combat, and the action space is defined by composite actions.

[0058] In this embodiment, in the MDP problem, the state space represents the environmental information perceived by the agent and its dynamic changes, which is the premise for the DRL algorithm to generate strategies and evaluate cumulative returns. In the process of ground unmanned equipment coordinated combat, the state space is mainly reflected in the position coordinates, state and environmental information of the combat entity.

[0059] In view of the different combat units, different unit attributes and different state attributes of the enemy and our side, this embodiment uses scalar information and entity information to define the state space attributes of the combat process. Scalar information mainly includes statistical information such as combat duration and remaining force value. Entity information includes two parts, one for our side and one for the enemy, which represents the current actual state of each combat unit on the battlefield. Our entity information includes information on all different types of ground unmanned equipment and drones of our side. Since the rear combat command vehicle does not directly engage in combat, it is not included in our entity information; enemy entity information includes information on tanks, infantry fighting vehicles, soldiers, and drones. Due to the existence of the fog of war, the combat parties can only obtain partial entity information of the other party's targets detected by each party, and cannot understand the situation information of undetected areas or entities. The specific state space information description is shown in Table 1.

[0060] Table 1 Description of the state space of ground unmanned equipment collaborative combat

[0061]

[0062] Action space refers to the set of all actions that different unmanned combat equipment can perform during the collaborative combat process. The decision-making action output of ground unmanned equipment collaborative combat is aimed at the "task layer" macro-combat command actions that can reflect the collaborative reconnaissance, coordinated maneuver and coordinated strike of unmanned equipment. In order to help the intelligent agent fully explore various possibilities in the environment, so as to avoid the existence of an unreachable "state blind spot" and ensure the good accessibility of the area where the optimal solution is located, this paper uses a composite action to describe the action space, so as to ensure that the intelligent agent has the ability to explore the complete task solution space while reducing the difficulty of exploration, accelerating the convergence of the DRL algorithm and improving the final algorithm performance.

[0063] Each composite action output by the combat decision agent is discrete and highly structured: first, the action type is selected, among which the main actions of the assault unmanned vehicle are maneuver and strike, and the main actions of the drone are route mobile patrol reconnaissance and regional mobile patrol reconnaissance; second, any subset of the own combat units is selected to issue the action to the target, where the target can be a certain location in the battlefield environment or a certain target of the enemy; finally, these sub-actions are combined into action instructions that can be received by unmanned combat equipment to realize the collaborative action of the "task layer". Among them, the relevant parameters describing the action, such as the reconnaissance area range (Range), maximum speed (Max_speed), and weapon (Weapon), are specified in the action instructions. Since the decision architecture of this paper uses equal time intervals to output decision actions, it does not make decisions on when to take the next action. The specific action space description is shown in Table 2.

[0064] Table 2 Description of ground unmanned equipment collaborative combat action space

[0065]

[0066] Furthermore, in S21, the reward function is a continuity function based on the process reward, and the process reward r of the agent at each time step is t P It is divided into target discovery and loss, loss of troop value and equipment damage, and change of own position. The process reward is r t P The calculation formula is:

[0067]

[0068] in, A local reward for each new enemy combat unit discovered, which is positively correlated with the target's value;

[0069] The penalty for losing a discovered target due to maneuver or other actions is negatively correlated with the value of the target. If the enemy target is completely destroyed, there is no penalty for the loss of the target.

[0070] For each enemy target hit and damaged, a reward of enemy force loss value is obtained, which is positively correlated with the damage status and target value of the target;

[0071] The additional damage bonus is obtained when the target is completely destroyed;

[0072] This is the penalty that is negatively correlated with the degree of damage and the value of the equipment itself when the enemy targets hit and cause damage to the equipment during combat.

[0073] The additional damage penalty is obtained when the team is completely destroyed.

[0074] k a,t , k a,t-1 They represent the degree of damage to the combat forces at time steps t and t-1 respectively, and m is the number of equipment that has not been completely destroyed on the side;

[0075] The calculation formula is:

[0076] Among them, l a_max is the effective strike distance of each combat equipment, [x g ,y g ] is the target location of the enemy position pre-set based on reliable intelligence, [x j,t ,y j,t ] is the position of each combat unit at time step t.

[0077] In this embodiment, the reward, as a key factor in guiding reinforcement learning training and strategy optimization iteration, directly affects whether the training of the intelligent agent can converge, the convergence direction, the training efficiency and the final performance. The combat mission is a typical scenario in which only victory or defeat is the final reward. The final reward is a description of the final result of the coordinated combat, which truly shows the goal of the coordinated combat of ground unmanned equipment. However, this reward is extremely sparse and delayed. It will only appear when and only when the conditions for determining the victory or defeat of the combat are met. Its general form is shown in formula (1), that is, a large positive reward r is given when the mission goal is achieved. win , a large penalty is imposed on failure -r win , there is no reward in the completion process. If the reward is directly used as a feedback guidance signal for training, it will be difficult for the agent to directly obtain reward feedback through simple actions or action sequences, which is not conducive to the formation of local knowledge, and it is difficult to provide guidance for exploration direction and learn effective strategies.

[0078]

[0079] To overcome the final reward r T In order to solve the problem of sparse rewards and delays caused by the decomposition of mission objectives, this embodiment adopts a reward reshaping method based on mission objective decomposition. According to the characteristics of combat mission objectives and different types of combat equipment, the final main mission objectives are divided into several process sub-objectives, and appropriate positive rewards and penalties are allocated according to their respective contribution and direction in the combat main mission objectives, thereby forming a denser process reward r PRealize the final reward r T reshape, guiding the intelligent agent to explore more efficiently and accelerate strategy convergence, so that it can learn some effective action strategies and tactics as quickly as possible.

[0080] According to the characteristics of each unmanned equipment in the collaborative combat mission, the process reward r of the agent at each time step is t P It is divided into multiple sub-goal rewards or penalties, such as target discovery and loss, loss of force value and equipment damage, and change of own position. For the target discovery and loss sub-goals, in order to encourage effective reconnaissance by drones and ground unmanned equipment, each time a new enemy combat unit is discovered, a local reward positively correlated with the value of the target can be obtained. At the same time, if the discovered target is lost due to maneuvering or other actions, a penalty will be imposed that is negatively correlated with the value of the target. When the enemy target is completely destroyed, there is no penalty for the loss of the target. Ground unmanned equipment is mainly responsible for strike missions. To encourage its effective strike behavior, each time an enemy target is hit and damaged, a reward of enemy force loss value that is positively correlated with the target damage status and target value can be obtained. At the same time, in order to encourage the complete destruction of the target, when the target is completely destroyed, an additional damage reward will be obtained. Similarly, in order to minimize the combat losses of one's own side, if the enemy targets are hit and damaged during the combat, a penalty will be incurred that is negatively correlated with the degree of damage and the value of the equipment itself. When completely destroyed, an additional damage penalty will be incurred In addition, in order to guide the combat agent to learn a better action decision strategy faster, a continuity function is designed to guide the unmanned combat equipment to reach the pre-established enemy position area. Its expression is shown in formula (2), where l a_max It is expressed as the effective strike distance of each combat equipment (for UAVs, it is the effective reconnaissance distance), [x g ,y g ] is the target location of the enemy position pre-set based on reliable intelligence, [x j,t ,y j,t ] is the position of each combat unit at time step t, and m is the number of equipment that has not been completely destroyed. In summary, the process reward r of the combat agent at each time step t P It can be shown by formula (3), where k a,t , k a,t-1 They represent the degree of damage to the combat forces at time steps t and t-1 respectively.

[0081]

[0082]

[0083] in, Rewards and penalties for target discovery and loss, Rewards for loss of enemy force value and complete destruction, Penalty for loss of value and complete destruction of one's own troops, Rewards and penalties for equipment position changes.

[0084] Furthermore, in S22, by using a multi-head attention mechanism, a multi-layer perceptron, an embedding layer, and a fully connected layer, the global scalar information that can reflect the entire battlefield situation, the actions performed by the previous agent, and the attribute feature information reflecting each entity are extracted and fused, thereby obtaining a partially observable game state encoding vector, which specifically includes the following steps:

[0085] S221, for the combat units of the own side and the enemy, the entity feature vectors in the same group are used as the query, key and value of the multi-head self-attention module respectively;

[0086] S222, then use the entity feature vector in the group as the query and value of the multi-head cross attention module, use the query of another group of multi-head self-attention modules as the key of the group of multi-head cross attention modules, and generate inter-group entity encoding features;

[0087] S223, repeat the process from S221 to S222 twice, so that the different dimensions of the feature vectors are fully crossed, and the final intra-group coding features and inter-group coding features of the two groups of units are generated;

[0088] S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain the final entity coding features of the own side and the enemy side;

[0089] S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length, which is stacked with numerical features such as the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and the Embedding features to form a partially observable game state encoding vector.

[0090] In this embodiment, when the ground unmanned equipment collaborative combat decision-making problem is regarded as a deep reinforcement learning problem, there will be the following challenges in the process of building the decision model: 1) Multi-feature information input. The input of the network model includes not only the attribute information of each entity and the global scalar information reflecting the battlefield situation, but also the action performed by the agent in the previous step and some masks reflecting the battlefield environment and entity logic. 2) Incomplete information game. The limited perception capabilities of the combat equipment's own sensors and the existence of the fog of war make the environment of the agent partially observable, and the decision sequence in the entire combat process is long. 3) Complex action and decision space. The decision output of each step is a highly structured action coupled with multiple sub-actions such as behavior action type selection, execution unit selection, and target selection.

[0091] In order to cope with the complexity and challenges in the construction of the above-mentioned combat decision-making agent neural network model, and at the same time be able to better control all combat units in the unmanned combat team and avoid the problem of inconsistent coordination between different types of unmanned equipment, this embodiment constructs a large single "commander" type ground unmanned equipment collaborative combat decision-making agent network model similar to AlphaStar to achieve the unified coordinated control from the effective representation of state characteristics to combat actions. The framework of the ground unmanned equipment collaborative combat decision-making agent network model is referenced Figure 1 shown.

[0092] The effective encoding and characterization of input information is the key to achieving intelligent agent decision-making and subsequent strategy optimization. For complex tasks with high state space dimensions such as collaborative combat, the limited characterization capabilities of shallow networks make it difficult to achieve effective feature extraction. Therefore, it is necessary to design a deeper and more complex representation network in combination with state space and task characteristics. This embodiment mainly distinguishes between fixed-length and variable-length input feature information, and uses a multi-head attention mechanism, a multilayer perceptron (MLP), an embedding layer, and a fully connected (FC) layer to extract and fuse global scalar information that can reflect the entire battlefield situation, the actions performed by the previous intelligent agent, and the attribute feature information reflecting each entity. The specific implementation details are as follows.

[0093] When encoding the attribute information of the entity, unlike AlphaStar which uniformly encodes all entity information, this embodiment independently encodes the two groups of combat units, one for oneself and one for the enemy. For each group of combat units, the entity feature vectors within the same group are used as the query, key, and value of the multi-head self-attention module to generate the intra-group entity coding features; then the entity feature vectors within the group are used as the query and value of the multi-head cross-attention module, and the query of another group of multi-head self-attention modules is used as the key of the multi-head cross-attention module of this group to generate the inter-group entity coding features. The whole process is repeated twice to allow the different dimensions of the feature vectors to fully cross and generate the final intra-group coding features and inter-group coding features of the two groups of units. The intra-group coding features and inter-group coding features of each group are spliced ​​to obtain the final entity coding features of the enemy and our groups.

[0094] The final entity encoding features of the enemy and our own groups are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length, which is stacked with the encoding vector of global scalar information, the encoding vector of the action executed in the previous step, and other numerical features and Embedding features to form a partially considerable game state encoding vector. Among them, statistical global scalar numerical features such as military strength and combat time are directly sent to MLP for representation after splicing, while categorical features such as the action performed by the agent in the previous step are first sent to the Embedding layer, and the sparse one-hot encoding vector is converted into a dense Embedding vector before splicing.

[0095] The main reason for compressing the final entity encoding features is that although non-essential features have been eliminated when designing the input state space of DRL, some redundant features may still exist, which establishes a false association between some useless information and the model output, which can easily cause overfitting of the policy network model. This embodiment uses the attention pooling operation to compress it, thereby removing redundant information and reducing the learning difficulty of DRL. Compared with average pooling and maximum pooling, although attention pooling requires an increase in the number of network layers and the amount of calculation, it can obtain the importance of different elements in the features through learning, thereby capturing the key information in the features. While achieving more refined feature extraction and combination than average pooling, it also avoids the maximum pooling operation that discards all minor information and directly uses the maximum value of the elements in the feature as the output. The information loss caused by the output improves the accuracy and generalization of the model.

[0096] Furthermore, in S23, an inference module is constructed based on the Residual LSTM network, and some considerable game state encoding vectors are transformed by the FC layer with ReLU activation function and sent to the Residual LSTM network to obtain the hidden layer encoding vector that considers the historical time series information, so as to extract higher-level abstract semantic features and establish long-term decision-making correlations, thereby obtaining more reasonable multi-head compound actions.

[0097] In this embodiment, the actual confrontation process of ground unmanned equipment coordinated combat is different from a simple game scene. First, it often takes a long time to conduct battlefield operations and layout to form a favorable battlefield situation. Therefore, the time span is long and the number of decision steps is large. Secondly, the combat decision-making agent is not a simple stress-response decision-making system. It needs to consider the strategies to be adopted in time and space according to the combat objectives and the observation information obtained. For example, what kind of attack and maneuver strategies should be adopted in the early, middle and late stages of the confrontation, how to use the terrain advantages, etc. Therefore, the decision output has a strong time dependence. At the same time, due to the limited perception ability of the equipment, the information in the coordinated combat process is incomplete and uncertain, and there is a "fog of war". Therefore, it is necessary to make decisions under the condition of partially considerable incomplete information.

[0098] In order to solve the problem of decision output with strong temporal dependence and incomplete information in the process of coordinated operations, as well as the problem of gradient vanishing of neural network caused by many decision steps, this embodiment uses the Residual LSTM network to construct the reasoning module, and sends the partially significant game state encoding vector into the Residual LSTM network after being transformed by the FC layer with the ReLU activation function, so as to obtain the hidden layer encoding vector that considers the historical temporal information, so as to extract higher-level abstract semantic features and establish long-term decision correlation. The network structure and calculation formula of the Residual LSTM are as follows: Figure 2 As shown in equations (4)-(10), l represents the index of each layer of LSTM. Respectively represent the input gate, forget gate and output gate of this layer, Respectively represent the hidden layer output and internal memory unit state at time t-1, is the hidden layer output from the l-1th layer (when l=1, it is the transformed game state encoding vector), the projection matrix To reduce r t l Dimension.

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] The main idea of ​​Residual LSTM is to combine LSTM units with residual networks, capture the dynamic characteristics of time series by running LSTM units between residual blocks, and add an LSTM layer after the last block to fully capture the context of the entire sequence. Specifically, at time t, the Residual LSTM network uses the current information input at each layer and the LSTM network And the hidden layer state of the previous moment As input, the internal memory unit C is updated by selectively discarding old information and receiving new information using the forget gate f and the input gate i, ensuring the effective transmission of information in time sequence. The only difference is that the Residual LSTM provides a spatial shortcut between layers through the residual network, which transfers the hidden state of the previous layer to the next layer. The state of the memory unit in this layer Used together with the output gate for the hidden layer state This avoids the vanishing gradient problem between LSTM layers.

[0107] In S24, in order to solve the logical relationship and decision output problem of the highly structured multi-head compound action space, this embodiment draws on the autoregressive action structure in the AlphaStar network structure, so that the output of each subsequent action head is conditioned on the output of the previous action head, thereby converting an N-dimensional action decision problem to be done in a certain state into N 1-dimensional action sequences for processing. Intuitively speaking, this is in line with the logical relationship, because different action types require different combat units to implement and act on different enemy targets or positions. In view of the limited expressive power of the autoregressive embedding layer using the addition operator in AlphaStar to connect the outputs of all action heads, this embodiment adopts an autoregressive network structure that first concatenates the encoding vector considering historical information with the encoding vector of the previous action and then connects the fully connected layer to provide more sufficient expressive power. The pointer network and the attention network are used to select combat units and targets, and the output of an MLP layer with a residual structure of the action head autoregressive network is used as a query, and the final entity encoding features of the two groups of units are used as attention keys to output the sampling probability distribution of combat units and targets respectively. The outputs of the remaining action heads are obtained by sampling the MLP layer with a residual structure in series with the Softmax layer. Among them, since the pointer network calculates the weight of each element in the input sequence through the attention mechanism and directly uses the elements of the input sequence for output, the output of the combat unit action head does not need to be connected to the Softmax layer.

[0108] It is particularly noteworthy that this embodiment does not use map spatial information to select the target position, but divides the entire combat space into N×M coarse-grained areas, and defines each numbered coarse-grained area as a global position intention label. Therefore, the output of the target position action head at each time step is the global position intention label. Compared with the method of AlphaStar using spatial information, convolutional neural network and deconvolutional neural network to determine the target position, this method effectively reduces the decision space of the strategy and the size of the combat agent strategy network structure. At the same time, in order to guide the rationality of action output in the process of reinforcement learning exploration and improve training efficiency, this embodiment uses an action mask based on expert prior knowledge to shield unreasonable action head outputs, for example: 1) In the early stage of combat (advance stage), the enemy target is not discovered and is not within the effective range, and the combat unit of the party is mainly mobile reconnaissance, and the strike action is shielded; 2) The strike action of the combat unit that does not have the conditions for strike (in the strike cooling time, zero weapons and ammunition, non-effective range, drone) is shielded; 3) Ground combat equipment such as unmanned vehicles do not have the ability to pass and reach certain areas (waters) in the combat space.

[0109] Furthermore, in S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture, and the distributed training framework includes a sampler, a sample cache, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine and calls the data in the sample cache for training.

[0110] In order to effectively utilize limited computing resources to train combat decision-making intelligent agent neural networks, this embodiment designs a large-scale distributed sampling training framework based on the Actor-Learner architecture to achieve high-throughput asynchronous parallel training, improve data efficiency and computing performance, and effectively speed up training efficiency. Specifically, the sampling training framework consists of components such as samplers, sample buffers, learners, parameter servers, and controllers, and runs on 512 CPU cores and 8 GeForce RTX 3080 GPUs (with a total of 96GB of video memory) packaged in Dockers. Among them, the sampler is deployed on a CPU machine, allowing the actor to interact with a single combat simulation environment running in the sampler to generate sample trajectories; the learner is deployed on a GPU machine, consuming data in the sample buffer to train network parameters. Figure 3 It shows how the sampling training framework completes a complete training process.

[0111] Furthermore, in S3, the method for training by the distributed training framework includes the following steps:

[0112] S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and parallelly generate a large number of trajectory samples through their respective actors and continuously interact with the simulation environment, and store them in the sample buffer;

[0113] S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model, and sends the updated network parameters to the parameter server;

[0114] S33, when the round of training is terminated, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronously updates the network parameters of the Actor in the sampler;

[0115] S34, repeat the above process until the entire training is completed.

[0116] In the above process, unlike the synchronous training method, each sampler performs sampling according to its own progress, that is, after the environment with fast sampling ends, the sampled data is first stored in the sample buffer and the next round of sampling begins immediately, without waiting for other samplers, and the Actor in the sampler is only responsible for sampling and is not used to calculate gradients, thus greatly improving the sampling efficiency. However, this sampling training framework also causes the target strategy in the learner to be updated earlier than the behavior strategy in the sampler, resulting in a policy lag between the behavior strategy and the target strategy. In addition, the following logic is followed during the sampling phase: 1) Obtain the current state data through the environment adapter; 2) Convert the action to be executed into instructions that can be received by the simulation environment through the action converter; 3) Calculate the reward for each decision step through the reward calculator; 4) Process the state data, action, and reward into sample data through the sample processor. The specific simulation environment scheduling process is as follows: Figure 4 shown.

[0117] Furthermore, in S4, the SARD-PPO algorithm is used to reuse off-policy data samples, and the stability of the algorithm is ensured by the "element-level" double-end pruning mechanism and the adaptive adjustment reuse of the entire policy trajectory.

[0118] In this embodiment, in order to enable the combat decision-making agent to generate a more effective collaborative combat strategy, this embodiment proposes a SARD-PPO algorithm that introduces policy entropy to solve the Markov decision model, so as to obtain a reasonable ground unmanned equipment collaborative combat strategy. Since this embodiment adopts asynchronous sampling and large-scale distributed methods for training, that is, new trajectory samples are generated by the sampler, and the learner uses the old trajectory samples in the sample buffer to asynchronously update the model parameters, the combat decision-making agent learning problem is essentially an off-policy reinforcement learning problem. Especially in a large-scale off-policy training environment, there may be a large difference between the sample trajectories of the previous strategy and the current strategy. If the on-policy reinforcement learning algorithm is directly used to learn the old trajectory samples in the sample buffer, it is easy to cause the policy model to fluctuate greatly, or even fail to converge.

[0119] The PPO algorithm is an easy-to-implement, efficient and popular on-policy reinforcement learning algorithm that only requires first-order optimization and has been proven to be effective in solving a series of challenging large-scale problems. The PPO algorithm uses the Actor-Critic framework, where the Actor network is used to generate the policy model π θ (a|s), its strategy substitution objective optimization function is:

[0120]

[0121] Among them, π and πc denote the future strategy and the current strategy respectively, τ denotes the strategy sample trajectory, A(s t ,a t ) represents the estimated advantage function, r t (θ)=π(a t |s t ) / π c (a t |s t ) represents the probability ratio between the new and old strategies, and ε is the clipping factor. The Critic network is used to estimate the value function V ω (s t ), and uses the traditional TD-error method to update its network parameters.

[0122] In order to effectively utilize the off-policy data samples caused by policy lag to update the policy while ensuring the stability of training, this embodiment adopts a more principled pruning mechanism SARD-PPO algorithm to reuse off-policy data samples, and ensures the stability of the algorithm through the "element-level" double-end pruning mechanism and the adaptive adjustment and reuse of the entire policy trajectory. Its policy replacement objective function is:

[0123]

[0124]

[0125] Among them, π c-l Represents the distance from the current strategy π c The lth previous strategy of , b represents the lower bound parameter of double-ended pruning, At the same time, SARD-PPO uses the previous strategy π c-l Deviation from the current policy π over the entire trajectory c Expectations Whether the adaptive adjustment uses the previous strategy π c-l The generated old samples are used to update the current strategy.

[0126]

[0127] In addition, in order to solve the problem of exploration difficulties caused by the complexity of the land battlefield and the sparseness of rewards during the strategy optimization process, this embodiment introduces a mechanism to encourage exploration of strategy entropy to enhance the exploration ability of the agent. The calculation method of strategy entropy is defined as:

[0128]

[0129] Then the strategy substitution objective optimization function of the SARD-PPO algorithm with added strategy entropy is:

[0130]

[0131] In order to reduce the number of parameters of the entire network model of the combat decision-making agent, the value function network of the combat decision-making agent shares the network structure and weights before the output of the strategy network Residual LSTM, and uses the hidden layer state information of the last layer of LSTM as input, and predicts the state value function after transformation through an MLP layer. In this process, in order to reduce the variance of the value function estimation, this embodiment uses all the information in the game state, encodes the hidden observation information and uses it together as the input of the value function network.

[0132] Using the V-trace method with truncated importance sampling The multi-step advantage correction estimate is obtained by minimizing With target value The mean square error is used to update the value function network parameter ω of the current strategy:

[0133]

[0134]

[0135] in, ρ t and c t Importance sampling weights for truncation.

[0136] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0137] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments described in the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0138] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0139] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0140] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0141] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent, characterized in that: The steps include: S1, build a two-layer decision-making architecture based on data-driven and knowledge-driven, and realize intelligent decision-making of two different granularity tasks in the mission layer and execution layer during the combat process; S2, based on the decision-making process of Markov process, builds a network model of collaborative combat decision-making agents; S3, by building a distributed training framework, realizes high-throughput asynchronous parallel training of the collaborative combat decision-making agent network model; S4, establish a combat decision-making agent learning algorithm based on deep reinforcement learning, and continuously optimize the collaborative combat decision-making agent network model through large-scale off-policy reinforcement training to obtain the optimal collaborative combat strategy; S5, conduct simulation experiments on the simulation platform to verify the feasibility of the technical route.

2. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S1, the task layer is used to generate a macro-combat command action strategy that can reflect the cooperation of combat units based on battlefield situation data, and the macro-combat command action strategy includes maneuvering and attacking; the execution layer is used to respond to the macro-combat command action strategy issued by the task layer, and the response includes adjusting artillery, selecting ammunition types, firing, accelerating, decelerating, turning, local path planning, and obstacle avoidance.

3. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S2, the construction method of the collaborative combat decision-making agent network model includes: S21, define the state space, action space and establish the reward function; S22, encoding the multi-feature information based on the state space and action space input to obtain a partially observable game state encoding vector; S23, deriving multi-head compound actions through partially observable game state encoding vectors; S24, decodes multiple compound actions to arrive at the final combat decision.

4. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 3, characterized in that: In S21, the state space is defined by scalar information and entity information, the scalar information includes statistical information, the statistical information includes combat duration and remaining force value, the entity information includes own entity information and enemy entity information, the own entity information includes information of all different types of own ground unmanned equipment and drones, the enemy entity information includes information of tanks, infantry fighting vehicles, soldiers, and drones; the action space is a collection of all actions that can be performed by different unmanned combat equipment during coordinated combat, and the action space is defined by composite actions.

5. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 3, characterized in that: In S21, the reward function is a continuity function based on the process reward. The process reward r of the agent at each time step is t P It is divided into target discovery and loss, loss of troop value and equipment damage, and change of own position.

6. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 3, characterized in that: In S22, by using a multi-head attention mechanism, a multi-layer perceptron, an embedding layer, and a fully connected layer, the global scalar information that can reflect the entire battlefield situation, the actions performed by the previous agent, and the attribute feature information reflecting each entity are extracted and fused to obtain a partially observable game state encoding vector, which specifically includes the following steps: S221, for the combat units of the own side and the enemy, the entity feature vectors in the same group are used as the query, key and value of the multi-head self-attention module respectively; S222, then use the entity feature vector in the group as the query and value of the multi-head cross attention module, use the query of another group of multi-head self-attention modules as the key of the group of multi-head cross attention modules, and generate inter-group entity encoding features; S223, repeat the process from S221 to S222 twice, so that the different dimensions of the feature vectors are fully crossed, and the final intra-group coding features and inter-group coding features of the two groups of units are generated; S224, concatenating the intra-group coding features and inter-group coding features of each group to obtain final entity coding features of the own side and the enemy side; S225, the final entity encoding features of the own side and the enemy side are used as attention keys for unit selection or target selection, and are compressed into an entity feature vector of fixed length, which is stacked with numerical features such as the encoding vector of the global scalar information, the encoding vector of the action executed in the previous step, and the Embedding features to form a partially observable game state encoding vector.

7. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 3, characterized in that: In S23, an inference module is constructed based on the Residual LSTM network. Part of the considerable game state encoding vector is transformed by the FC layer with ReLU activation function and then sent to the Residual LSTM network to obtain the hidden layer encoding vector that takes into account the historical time series information, so as to extract higher-level abstract semantic features and establish long-term decision-making correlations, thereby obtaining more reasonable multi-head compound actions.

8. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S3, the distributed training framework is a large-scale distributed training framework based on the Actor-Learner architecture, and the distributed training framework includes a sampler, a sample buffer, a learner, a parameter server and a controller. The sampler is deployed on a CPU machine, and the actor interacts with a single combat simulation environment running in the sampler to generate sample trajectories. The learner is deployed on a GPU machine and calls the data in the sample buffer for training.

9. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 8, characterized in that: In S3, the method for training by the distributed training framework includes the following steps: S31, the controller sends a sampling instruction to the sampler, and multiple samplers asynchronously and parallelly generate a large number of trajectory samples through their respective actors and continuously interact with the simulation environment, and store them in the sample buffer; S32, when there are enough training samples in the sample buffer, the controller sends a training instruction to the learner, and the learner calls a batch of sample data in the buffer to perform gradient calculation to train the collaborative combat decision-making agent network model, and sends the updated network parameters to the parameter server; S33, when the round of training is terminated, the controller sends a parameter synchronization instruction to the parameter server, and the parameter server synchronously updates the network parameters of the Actor in the sampler; S34, repeat the above process until the entire training is completed.

10. The method for constructing a ground unmanned equipment collaborative combat decision-making intelligent agent according to claim 1, characterized in that: In S4, the SARD-PPO algorithm is used to reuse off-policy data samples, and the stability of the algorithm is ensured by the "element-level" double-end pruning mechanism and the adaptive adjustment reuse of the entire policy trajectory.

Images