An army unmanned combat cluster confrontation command method and device based on hierarchical multi-agent reinforcement learning, equipment and storage medium

Abstract

The application provides an army unmanned combat cluster confrontation command method and device based on hierarchical multi-agent reinforcement learning, a storage medium and equipment, which replaces the traditional hard command by generating a tactical target vector describing the combat intention through a continuous agent network of global situation data, realizes the decoupling of intention and execution, and converts the vector into an internal reward signal and a strategy bias vector in a target explanation network in a platoon agent network to guide autonomous cooperative decision-making, and outputs specific tactical actions according to the single unit agent network; meanwhile, the uplink attention network is used to compress the abstract information of each single unit state and feed back to the continuous level to form a decision-making closed loop; each level network is trained by using a hierarchical curriculum learning method from bottom to top, convergence difficulties of joint training are avoided, and finally the cross-level cooperative efficiency is optimized through global fine-tuning.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, device, equipment and storage medium for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning. Background Technology

[0002] In modern army unmanned combat swarm scenarios, command and control systems need to achieve hierarchical decomposition and coordinated execution from macro-tactical intentions to micro-tactical actions within a multi-level command structure of "company-platoon-individual unit". The core technical challenge in this scenario is how to achieve efficient and robust cross-level collaborative command while maintaining the flexibility of autonomous decision-making among intelligent agents at different levels.

[0003] Specifically, this problem manifests as a triple dilemma. First, regarding hierarchical interaction, if upper-level agents directly issue specific action commands or precise coordinates to lower-level agents, the lower-level agents lose their ability to autonomously adjust their execution methods based on the local situation, leading to a deep binding of strategies between upper and lower levels, making the entire system rigid and difficult to expand. Conversely, if the lower level is given complete autonomy, the tactical intentions of the upper level cannot be effectively communicated and executed, making cluster collaboration impossible. Second, regarding training, the joint strategy search space of multi-level, multi-agent systems grows exponentially with the number of levels and agents. Direct end-to-end joint training easily leads to algorithms failing to converge or converging to fragile overfitting strategies, resulting in severely insufficient generalization ability when facing unknown adversarial situations. Third, regarding information flow, if there is only a top-down unidirectional transmission of commands between levels, upper-level command decisions will become blind commands due to the inability to obtain key information such as real-time battle losses, ammunition consumption, and local enemy situations from the lower levels, failing to form a closed-loop command process of "decision-execution-feedback-re-decision."

[0004] Existing technologies face fundamental difficulties in effectively decoupling "combat intent" and "execution strategy" between different levels, making it difficult to establish a two-way situational awareness channel that balances information refinement and real-time performance between different levels.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention discloses a method, device, equipment, and storage medium for the adversarial command of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning. It aims to solve the problems of tight coupling between command intentions and execution strategies between levels in multi-level unmanned combat swarms, difficulty in convergence of multi-agent collaborative training, and incomplete interaction of uplink and downlink situational information.

[0007] The first embodiment of the present invention provides a method for command and control of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning, which is executed based on pre-built company-level agent networks, platoon-level agent networks, and single-equipment-level agent networks, including:

[0008] Global battlefield situation data and uplink summary information of each platoon are input into the company-level agent network. After processing by the first policy network in the company-level agent network, a corresponding tactical target vector is generated for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector.

[0009] The tactical target vector and platoon-level local observation data are input into the platoon-level agent network. The target interpretation network in the platoon-level agent network generates an internal reward signal and a policy bias vector. After processing by the second policy network in the platoon-level agent network, the platoon-level coordinated action command is output.

[0010] The platoon-level coordinated action command and the single-equipment self-observation data are input into the single-equipment-level intelligent agent network. After being processed by the third strategy network in the single-equipment-level intelligent agent network, the single-equipment tactical action is output.

[0011] The state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network to generate the uplink summary information, which is then fed back to the input of the cascade-level agent network to form a command and decision-making closed loop.

[0012] The chain-level agent network, platoon-level agent network, and single-unit-level agent network are obtained by sequentially training using a hierarchical course learning method. The hierarchical course learning method trains from bottom to top layer by layer in the order of single-unit-level, platoon-level, and chain-level. At each stage, the policy parameters of the trained layers are frozen, and finally all layers are unfrozen for joint fine-tuning.

[0013] The second embodiment of the present invention provides an army unmanned combat swarm confrontation command device based on hierarchical multi-agent reinforcement learning, which is executed based on pre-built company-level agent networks, platoon-level agent networks, and single-equipment-level agent networks, including:

[0014] The company-level decision-making module is used to input global battlefield situation data and uplink summary information of each platoon into the company-level intelligent agent network. After processing by the first policy network in the company-level intelligent agent network, a corresponding tactical target vector is generated for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector.

[0015] The platoon-level coordination module is used to input the tactical target vector and platoon-level local observation data into the platoon-level agent network. The target interpretation network in the platoon-level agent network generates an internal reward signal and a policy bias vector, and after processing by the second policy network in the platoon-level agent network, it outputs the platoon-level coordination action command.

[0016] The single-assembly execution module is used to input the platoon-level coordinated action command and the single-assembly's own observation data into the single-assembly-level intelligent agent network, and output the single-assembly tactical action after processing by the third strategy network in the single-assembly-level intelligent agent network.

[0017] The uplink communication module is used to generate the uplink summary information after the state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network, and then fed back to the input of the liaison-level agent network to form a command and decision-making closed loop.

[0018] The chain-level agent network, platoon-level agent network, and single-unit-level agent network are obtained by sequentially training using a hierarchical course learning method. The hierarchical course learning method trains from bottom to top layer by layer in the order of single-unit-level, platoon-level, and chain-level. At each stage, the policy parameters of the trained layers are frozen, and finally all layers are unfrozen for joint fine-tuning.

[0019] The third embodiment of the present invention provides an army unmanned combat swarm confrontation command device based on hierarchical multi-agent reinforcement learning, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize any of the above-mentioned army unmanned combat swarm confrontation command methods based on hierarchical multi-agent reinforcement learning.

[0020] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the army unmanned combat swarm confrontation command method based on hierarchical multi-agent reinforcement learning as described in any of the above claims.

[0021] Based on the hierarchical multi-agent reinforcement learning-based army unmanned combat swarm confrontation command method, device, equipment, and storage medium provided by this invention, the intention and execution are decoupled by generating tactical target vectors describing combat intentions from global situational data through a company-level agent network, replacing traditional hard commands. The target interpretation network in the platoon-level agent network transforms this vector into internal reward signals and policy bias vectors to guide autonomous collaborative decision-making, and the single-equipment-level agent network outputs specific tactical actions accordingly. At the same time, the status of each single equipment is compressed into summary information by an uplink attention network and fed back to the company level to form a decision-making closed loop. Each level network is trained using a bottom-up, layer-by-layer frozen hierarchical course learning method to avoid the convergence difficulties of joint training, and finally, cross-level collaborative effectiveness is optimized through global fine-tuning. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for commanding and controlling unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning, provided in the first embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a module of an army unmanned combat cluster confrontation command device based on hierarchical multi-agent reinforcement learning, provided in the second embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] This invention discloses a method, device, equipment, and storage medium for the adversarial command of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning. It aims to solve the problems of tight coupling between command intentions and execution strategies between levels in multi-level unmanned combat swarms, difficulty in convergence of multi-agent collaborative training, and incomplete interaction of uplink and downlink situational information.

[0027] The first embodiment of the present invention provides a method for command and control of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning, which is executed based on pre-built company-level agent networks, platoon-level agent networks, and single-equipment-level agent networks, including:

[0028] S101, input the global battlefield situation data and the uplink summary information of each platoon into the company-level agent network. After processing by the first policy network in the company-level agent network, generate a corresponding tactical target vector for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector.

[0029] In a real-world combat simulation scenario, the Red Team is organized into a tank company with three platoons, each containing n tanks. The Blue Team consists of a tank platoon. The two sides engage in combat within a pre-defined battlefield environment that includes terrain, buildings, and obstacles. At each decision step, the company-level agent network receives global battlefield situation data and uplink summary information from each platoon as input. The global battlefield situation data includes the position and status information of all friendly units, the distribution information of known enemy units, and the control status information of key areas. The uplink summary information from each platoon is a fixed-dimensional vector generated by the lower-level uplink attention network. This input data is first fused through a shared multilayer perceptron backbone network within the company-level agent network to extract a feature representation that comprehensively reflects the current global situation. This feature representation is then fed into the first policy network for processing. The first policy network has multiple independent classification output heads corresponding to the number of subordinate platoons. Each output head outputs a probability distribution on a predefined set of tactical task types and independently selects a tactical task type for each subordinate platoon, thereby generating a corresponding tactical target vector.

[0030] In this embodiment, the tactical target sub-vector of each platoon is a fixed-length dense vector with a total dimension D equal to D_type plus D_target plus D_aux. The platoon-level agent network generates tactical target sub-vectors for each of the three subordinate platoons, and the three sub-vectors together constitute the platoon-level tactical target vector matrix. Specifically, the tactical target sub-vector includes three encoding parts: the first part is the task type encoding, with a dimension D_type of 8 dimensions, using one-hot encoding to represent 7 predefined tactical task types and reserving 1 extension bit. The 7 task types are advance, dispersal, deployment, assault, firepower application, occupation of position, and pursuit. For example, the encoding for "assault" is an 8-dimensional vector with the fourth position set to 1 and the remaining positions set to 0; the second part is the target area encoding, with a dimension D_target of 6 dimensions, using a parameterized continuous vector to represent the specific location and attributes of the target, including normalization to 0 to 1. The first part consists of three parts: the target center point X and Y coordinates, the normalized target area radius, the target type identifier to distinguish between area targets, enemy unit targets, and key point targets, the target object identifier to specify a specific enemy unit, and the task priority value between 0 and 1; the second part is the auxiliary parameter encoding, with a dimension D_aux of 4, which provides additional constraints and guidance information for the task, including speed requirements representing a shift from cautious maneuvering to full-speed advance, concealment preferences representing speed priority or concealment priority, suggested formations that can be selected from automatic, line, triangle, and echelon, and the normalized expected duration. After selecting a tactical task type for each platoon at each decision step, the first strategy network concatenates the one-hot encoding corresponding to the task type with the target area encoding and auxiliary parameter encoding output by the network parameterization to form a complete tactical target sub-vector, which is then distributed to the corresponding platoon-level agent network.

[0031] S102, The tactical target vector and platoon-level local observation data are input into the platoon-level intelligent agent network. The target interpretation network in the platoon-level intelligent agent network generates an internal reward signal and a strategy bias vector. After processing by the second strategy network in the platoon-level intelligent agent network, the platoon-level coordinated action command is output.

[0032] The platoon-level agent network receives tactical target sub-vectors T_i and platoon-level local observation data O_platoon_i as input from the company-level agent network. The platoon-level local observation data includes the status information of all subordinate units within the platoon, enemy situation information, terrain information, and the relative positions of neighboring platoons. The platoon-level agent network first concatenates the tactical target sub-vectors T_i and O_platoon_i, then performs feature fusion through a shared multilayer perceptron to obtain a fused feature representation. This fused feature representation is simultaneously fed into two core components within the platoon-level agent network: the target interpretation network and the second policy network.

[0033] The target interpretation network is a lightweight multilayer perceptron. Essentially, it learns a mapping function from the current state, agent action, and upper-level intention triple to the degree of intention achievement. This mapping function is not a fixed, manually specified rule, but rather derived inductively from environmental feedback through a data-driven approach. The target interpretation network takes the aforementioned fused feature representation as input and outputs two results: first, an internal reward signal R_internal, a scalar value used to measure the contribution of the platoon-level agent's current behavior to achieving the tactical intention issued by the company level. It can be represented as R_internal = G_φ(s,a,T), where phi is the learnable parameter of the target interpretation network, s is the current state, a is the current action, and T is the received tactical target sub-vector; second, a policy bias vector, whose dimension is the same as the size of the platoon-level coordinated action set (4 dimensions in this embodiment), corresponding to four types of coordinated actions: formation change, advance, firepower application, and capturing the assault target.

[0034] The modulation process of the policy bias vector on the output of the second policy network is as follows: The second policy network outputs the original action score (logits) based on the fused feature representation. This original action score is a vector with the same dimension as the platoon-level coordinated action set, representing the initial preference of each action without considering the upper-level intent. Subsequently, the policy bias vector output by the target interpretation network is added element-wise to this original action score to obtain the modulated action score. Finally, the modulated action score is converted into a probability distribution of platoon-level coordinated action instructions through a softmax normalization function. The platoon-level agent network samples the platoon-level coordinated action instructions for the current decision step based on this probability distribution. Through this additive modulation mechanism, the target interpretation network can apply a positive bias to actions in the action space that conform to the intent based on the upper-level tactical intent and the current situation, giving them a higher probability of being selected, without changing the basic action preference structure already learned by the second policy network itself.

[0035] The target interpretation network is trained using a total loss function consisting of a weighted combination of three loss terms: L_total = αL_value + βL_cf + L_reg, where α is the main loss weight coefficient, used to control the contribution ratio of the main loss term in the total loss, with a recommended initial value of 1.0 and an adjustment range of 0.5 to 2.0; β is the counterfactual baseline loss weight coefficient, used to control the contribution ratio of the counterfactual baseline loss term, with a recommended initial value of 0.3 and an adjustment range of 0.1 to 0.5. The specific design of the three loss terms is as follows: The main loss function L_value is calculated based on the prediction error of the target value in the next state using the internal reward signal, and its expression is L_value = E[(G_φ(s,a,T) - γV_target(s')) 2 In this context, γ is the discount factor, G_φ is the output function of the target interpretation network, s is the current environmental state, a is the action actually performed by the agent at the current time, T is the tactical target sub-vector received from the cascade agent network, s' is the next environmental state transitioned to after performing action a, φ is the learnable parameter of the target interpretation network, and V_target(s') is the target value estimate of the next state provided by the centralized ranking commentator network. This loss forces the internal reward signal output by the target interpretation network to accurately predict the cumulative value that can be obtained in the future after performing the current action, thereby enabling behaviors that truly help achieve the upper-level intentions to obtain higher internal rewards; the counterfactual baseline loss L_cf is expressed as L_cf = E[(G_φ(s,a,T) - E_{a'~π}[G_φ(s,a',T)]) 2], where E_{a'~π}[·] is the conditional expectation of sampling action a' under the current policy π, and a' is a randomly sampled action under the current policy π. This loss, by comparing the difference between the actual action and the average action under the current policy, enables the target interpretation network to more accurately evaluate the marginal contribution of a single action relative to the average level, rather than giving a general high or low reward; regularization loss L_reg = λ1‖φ‖2 2 + λ2· D_KL( p_{G_φ}(s,a,T) || p_U ), where p_{G_φ}(s,a,T) represents the distribution of the internal reward signal output by the target interpretation network, p_U represents a uniform distribution, and D_KL(·||·) explicitly represents the KL divergence operation. λ1 is generally set to 0.001 to prevent overfitting caused by excessively large network parameters. λ2 is selected according to the task complexity: 0.001 to 0.01 for simple discrimination tasks, 0.01 to 0.05 for medium-complexity tasks, 0.05 to 0.1 for complex intent understanding tasks, and 0.1 to 0.2 for highly ambiguous tasks. The role of this KL divergence term is to prevent the output of the internal reward signal from collapsing to a constant value, thereby ensuring that the target interpretation network maintains effective discrimination ability for different actions.

[0036] The target interpretation network and the second policy network are jointly trained using an alternating optimization approach. The specific process is as follows: In each training iteration, the parameter theta of the second policy network is first fixed and not updated. A batch of empirical trajectory data is sampled from the empirical replay buffer, and the total loss function L_total is calculated. The parameter phi of the target interpretation network is then updated using gradient descent. Subsequently, the parameter phi of the target interpretation network is also fixed and not updated. The internal reward signal R_internal output by the target interpretation network is weighted and combined with the external reward r_external returned by the environment to obtain the total reward R_total = r_external + λ·R_internal, where λ is the coefficient balancing the contributions of internal and external rewards. Based on this total reward signal, the parameter theta of the second policy network is updated using a proximal policy optimization algorithm. During the backpropagation process of the alternating optimization, gradient truncation ensures that the gradient of the target interpretation network does not propagate to the second policy network via the internal reward signal and policy bias vector, and similarly, the gradient of the second policy network does not flow back into the target interpretation network, thus avoiding the formation of a cyclic gradient dependency between the two networks, which could lead to training instability.

[0037] Furthermore, a centralized platoon-level commentator network is deployed within the platoon-level agent network. This commentator network takes the joint state of the platoon as input, which includes the observation information O_units of all subordinate units in the platoon, the platoon-level local observation data O_platoon_i, and the concatenation of the company-level tactical target sub-vectors T_i. It outputs a scalar value estimate V_platoon_i for the current joint state of the platoon. The key design of the centralized platoon-level commentator network is that it simultaneously serves the training of the second policy network within the same platoon and the third policy network of all subordinate unit-level agents. That is, all agents within the platoon share the same value assessment benchmark, while the unit-level agent network does not have a separate commentator network. This design ensures that within the platoon—a tactical coordination unit—the policy updates of each agent are based on the same global value baseline for advantage estimation, thereby achieving efficient and consistent credit allocation among agents within the platoon and ensuring that platoon-level coordination instructions and unit-level tactical actions remain coordinated in the optimization direction.

[0038] S103, The platoon-level coordinated action command and the single-equipment self-observation data are input into the single-equipment-level intelligent agent network, and after being processed by the third strategy network in the single-equipment-level intelligent agent network, the single-equipment tactical action is output.

[0039] The single-unit-level agent network receives platoon-level coordinated action commands A_platoon, output from the platoon-level agent network, and single-unit observation data O_unit_j as input. The single-unit observation data includes the vehicle's current position coordinates, remaining health, ammunition reserves, enemy target information within the sensor's range, and friendly unit information. The platoon-level coordinated action commands are discrete coordinated action types and their corresponding parameters output from the second strategy network; for example, formation change commands carry target formation type parameters, and advance commands carry advance direction and distance parameters. The single-unit-level agent network first performs one-hot encoding of the platoon-level coordinated action commands A_platoon, concatenates them with the single-unit observation data O_unit_j, and then performs feature fusion through a multilayer perceptron to obtain the encoded joint feature representation. The joint feature representation is fed into the third policy network for processing. The third policy network outputs a probability distribution on a predefined set of single-unit tactical actions and samples the single-unit tactical action A_unit for the current decision step based on this distribution. This action set includes five basic tactical action types: utilizing terrain, battlefield maneuvering, movement under enemy fire, target elimination, and assault. The third policy network also outputs specific execution parameters corresponding to the selected action type. For example, when selecting battlefield maneuvering, it outputs the coordinates of the maneuvering target point and the speed control quantity; when selecting target elimination, it outputs the target identifier. The single-unit level agent network does not have an independent critic network. The state value evaluation required during its training process is uniformly provided by the centralized platoon-level critic network of its respective platoon. That is, when the third policy network performs advantage estimation calculations for the proximal policy optimization algorithm, it directly calls the joint state value V_platoon_i output by the centralized platoon-level critic network of that platoon as the baseline value. Through this design, the strategy updates of each individual unit within the same platoon are all based on the same value benchmark. This allows each unit to implicitly consider the impact of the behavior and state of other units within the platoon on the overall platoon-level mission completion when executing its own tactical actions, thus achieving close coordination among units within the platoon without explicit communication. In the actual decision-making chain, platoon-level coordinated action commands provide meso-level constraints and guidance for the behavior of individual units. For example, when the platoon issues an advance command, each unit will autonomously select specific battlefield maneuver paths and movement methods under enemy fire under these constraints. When the platoon issues a firepower application command, each unit will autonomously allocate targets and select actions to eliminate them, thereby realizing the autonomous refinement and implementation of individual unit behaviors from the platoon-level coordinated mode.

[0040] S104, the state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network to generate the uplink summary information, and then fed back to the input of the cascade-level agent network to form a command and decision-making closed loop.

[0041] Within each decision cycle, the uplink attention network in the platoon-level agent network is responsible for compressing and refining the state information of each subordinate unit-level agent, generating a fixed-dimensional uplink summary information U_platoon, and reporting it to the company-level agent network. This constructs a bottom-up situational awareness channel, which, together with the downlink channel of the company-level agent network sending tactical target vectors, forms a complete command and decision-making closed loop. Specifically, the platoon-level agent network first collects the state information S_units of all unit-level agents within the platoon. The state information of each unit includes its current position coordinates, remaining health, ammunition level, and local situational features within the sensor's perception range. The uplink attention network maps the state information of each unit to a corresponding key vector K_j and value vector V_j through a shared parameter encoding network. The key vector is used to represent the semantic features of the unit's state information for subsequent relevance matching, while the value vector carries the substantive content to be reported in the unit's state information. Simultaneously, the uplink attention network encodes the task context information executed by the current platoon-level agent into a query vector Q. This task context information originates from the fusion features of the tactical target vector T_i received by the platoon-level agent network and the current platoon-level local observation data O_platoon_i. Subsequently, the uplink attention network calculates the dot product similarity between the query vector Q and the key vectors K_j of each unit, and performs softmax normalization to obtain the importance weight w_j of each unit's state information relative to the current platoon-level task. This attention weight calculation enables the uplink attention network to automatically identify which unit's state information is more critical in the current task context. For example, when the platoon is performing an assault mission, units at the forefront with sufficient ammunition will receive higher attention weights, while units at the rear with intact state will have relatively lower weights. Based on the calculated importance weight w_j, the uplink attention network performs a weighted summation of the value vectors V_j of all units to generate a fixed-dimensional uplink summary information U_platoon_i. The uplink summary information, represented as a dense vector, encodes the overall situation of the entire platoon, implicitly encoding key information such as the platoon's average health, ammunition sufficiency, and proximity to the mission objective. Since the parameters of the uplink attention network are trained along with the platoon-level agent network in the second stage of hierarchical course learning, its attention weight allocation strategy can automatically learn the most valuable information filtering rules under different mission types and situational conditions through data-driven methods, eliminating the need for manually designing fixed information filtering templates. After the uplink summary information U_platoon_i for each platoon is generated, the uplink summary information for each platoon is concatenated and fed into the input of the company-level agent network, serving as the observation input for the first policy network along with the global battlefield situational data.Through this mechanism, when generating the tactical target vector for the next decision step, the platoon-level agent network can perceive the key situational changes of each subordinate platoon in real time. For example, if a platoon encounters strong resistance and its health feature in the uplink summary information decreases significantly, the platoon-level agent network can adjust the task allocation for that platoon or transfer the task to a platoon in a better state. This enables dynamic command and adjustment based on complete situational information, avoiding blind decision-making due to missing information.

[0042] In one possible implementation of this invention, the chain-level agent network, platoon-level agent network, and single-assembly-level agent network are sequentially trained using a hierarchical course learning method. This method trains layer by layer from bottom to top in the order of single-assembly level, platoon level, and chain level, freezing the policy parameters of the trained levels at each stage, and finally unfreezing all levels for joint fine-tuning. This method decomposes the high-dimensional joint optimization problem that requires simultaneous optimization of policy parameters at all levels into multiple independently manageable sub-problems. The trained levels, as stable environmental components, provide deterministic behavioral responses for the current level, thereby reducing non-stationarity and the dimensionality of the exploration space during training. The hierarchical course learning method specifically includes four sequentially executed training stages.

[0043] The first phase is the basic capability solidification training phase for individual-level agents. The policy parameters of the company-level and platoon-level agent networks remain fixed. Both networks use preset simple rule policies to generate tactical target vectors and platoon-level coordinated action instructions, providing basic task inputs and a framework for coordinated instructions for individual-level agents. Under this setup, the third policy network is trained intensively, enabling each individual-level agent to gradually master individual tactical skills such as utilizing terrain, battlefield maneuvering, movement under enemy fire, target elimination, and assault. Training employs a proximal policy optimization algorithm. The state value assessment required for advantage estimation is provided by the centralized platoon-level commentator network, which is simultaneously trained and updated during this phase. Within each platoon, an internal adversarial training mechanism is implemented. Individual agents within the same platoon are randomly divided into two adversarial groups with conflicting task objectives, such as one group capturing an area while the other defends the same area. This forces individual-level agents to learn more robust individual tactical skills through active adversarial training. The adversarial groups are randomly reassigned in different training rounds to avoid over-adaptation of strategies to specific adversarial groups. Training continues until the third policy network reaches a preset convergence criterion on individual tactical skill evaluation metrics.

[0044] The second phase is the platoon-level coordination capability training phase. All parameters of the third policy network trained in the first phase are frozen, and each individual equipment-level agent uses a fixed policy as a stable execution unit. The company-level agent network policy parameters remain fixed, and diverse tactical target vectors are generated using rule-based or random policies to provide rich task input distributions for the platoon-level agents. Under this setting, the second policy network and the target interpretation network are unlocked for joint training, employing a multi-agent proximal policy optimization algorithm combined with the internal reward signal provided by the target interpretation network for policy updates. In this phase, the platoon-level agents learn to correctly understand the semantic meaning of tactical target vectors through the target interpretation network, selecting appropriate coordinated actions based on different task types and battlefield situations. The parameters of the uplink attention network are trained synchronously in this phase, learning to extract key situational summaries from the status information of subordinate individual equipment units. Since the parameters of the third policy network are frozen, the response behavior of individual equipment-level agents to platoon-level commands is deterministic, constructing a stable environmental dynamics model for platoon-level policy learning. This phase involves organizing inter-platoon adversarial exercises within the company, assigning conflicting or competitive tactical objectives to different platoons, such as two platoons vying for control of the same key area. This forces platoon-level agents to optimize their coordination strategies and resource allocation schemes in dynamic adversarial situations. Training continues until the second policy network and the objective interpretation network reach the preset convergence criteria on the platoon-level coordination effectiveness evaluation indicators.

[0045] The third phase is the company-level command capability training phase. All parameters of the second strategy network, target interpretation network, and third strategy network are frozen, and platoon-level and single-equipment-level agents use fixed strategies as a stable lower-level execution system. Under this setting, the first strategy network is unlocked for training, employing a proximal strategy optimization algorithm. Value assessment is provided by the value network within the company-level agent network, which uses global battlefield situation data and platoon-level uplink summary information as input and output scalar state value estimates. The company-level agent learns to generate a tactical target vector sequence that maximizes the adversarial win rate based on the global situation and platoon-level uplink summary information. This includes selecting task types based on the enemy / friendly situation, setting target area parameters and priorities, and dynamically adjusting task allocation based on platoon casualties and ammunition status. In this phase, the company-level agent conducts adversarial training against a Blue Force tank platoon, continuing until the first strategy network reaches a preset convergence criterion in the global adversarial win rate indicator.

[0046] The fourth stage is the global collaborative fine-tuning stage. All freezing constraints imposed on the network parameters of each layer in the previous three stages are removed, ensuring that all parameters of the first policy network, second policy network, target interpretation network, third policy network, uplink attention network, and centralized ranking commentator network are in an updatable state, allowing for end-to-end joint fine-tuning training. The training learning rate used in this stage is strictly lower than that used in the previous three stages, ensuring that each level of policy is only locally optimized based on convergence rather than experiencing drastic policy shifts. Joint fine-tuning aims to optimize the collaborative details across layers and address the policy mismatch between layers that may occur during independent hierarchical training. Since each level of policy has a good foundation upon entering this stage, the environmental dynamics caused by parameter updates under a small learning rate are smooth and bounded. The semantic space of the tactical target vectors transmitted between layers remains stable. Simultaneously, the uplink attention communication mechanism plays a closed-loop regulatory role during fine-tuning; when policy fine-tuning leads to a deterioration of the situation at a lower layer, situational information is fed back to the upper layer in real time, thus suppressing erroneous update directions. Training continues until the global adversarial effectiveness index stabilizes and converges, completing the entire training process.

[0047] The second embodiment of the present invention provides an army unmanned combat swarm confrontation command device based on hierarchical multi-agent reinforcement learning, which is executed based on pre-built company-level agent networks, platoon-level agent networks, and single-equipment-level agent networks, including:

[0048] The company-level decision-making module is used to input global battlefield situation data and uplink summary information of each platoon into the company-level intelligent agent network. After processing by the first policy network in the company-level intelligent agent network, a corresponding tactical target vector is generated for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector.

[0049] The platoon-level coordination module is used to input the tactical target vector and platoon-level local observation data into the platoon-level agent network. The target interpretation network in the platoon-level agent network generates an internal reward signal and a policy bias vector, and after processing by the second policy network in the platoon-level agent network, it outputs the platoon-level coordination action command.

[0050] The single-assembly execution module is used to input the platoon-level coordinated action command and the single-assembly's own observation data into the single-assembly-level intelligent agent network, and output the single-assembly tactical action after processing by the third strategy network in the single-assembly-level intelligent agent network.

[0051] The uplink communication module is used to generate the uplink summary information after the state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network, and then fed back to the input of the liaison-level agent network to form a command and decision-making closed loop.

[0052] The chain-level agent network, platoon-level agent network, and single-unit-level agent network are obtained by sequentially training using a hierarchical course learning method. The hierarchical course learning method trains from bottom to top layer by layer in the order of single-unit-level, platoon-level, and chain-level. At each stage, the policy parameters of the trained layers are frozen, and finally all layers are unfrozen for joint fine-tuning.

[0053] The third embodiment of the present invention provides an army unmanned combat swarm confrontation command device based on hierarchical multi-agent reinforcement learning, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize any of the above-mentioned army unmanned combat swarm confrontation command methods based on hierarchical multi-agent reinforcement learning.

[0054] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the army unmanned combat swarm confrontation command method based on hierarchical multi-agent reinforcement learning as described in any of the above claims.

[0055] Based on the hierarchical multi-agent reinforcement learning-based army unmanned combat swarm confrontation command method, device, equipment, and storage medium provided by this invention, the intention and execution are decoupled by generating tactical target vectors describing combat intentions from global situational data through a company-level agent network, replacing traditional hard commands. The target interpretation network in the platoon-level agent network transforms this vector into internal reward signals and policy bias vectors to guide autonomous collaborative decision-making, and the single-equipment-level agent network outputs specific tactical actions accordingly. At the same time, the status of each single equipment is compressed into summary information by an uplink attention network and fed back to the company level to form a decision-making closed loop. Each level network is trained using a bottom-up, layer-by-layer frozen hierarchical course learning method to avoid the convergence difficulties of joint training, and finally, cross-level collaborative effectiveness is optimized through global fine-tuning.

[0056] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in an army unmanned combat swarm confrontation command device that implements hierarchical multi-agent reinforcement learning. For example, the apparatus described in the second embodiment of the present invention.

[0057] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of a hierarchical multi-agent reinforcement learning-based army unmanned combat swarm confrontation command method, connecting various parts of the method through various interfaces and lines.

[0058] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, implements various functions of an army unmanned combat swarm confrontation command method based on hierarchical multi-agent reinforcement learning. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0059] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0060] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0061] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for command and control of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning, characterized in that, Execution is based on pre-built cascade-level agent networks, platoon-level agent networks, and single-unit-level agent networks, including: Global battlefield situation data and uplink summary information of each platoon are input into the company-level agent network. After processing by the first policy network in the company-level agent network, a corresponding tactical target vector is generated for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector. The tactical target vector and platoon-level local observation data are input into a platoon-level agent network. An internal reward signal and a policy bias vector are generated by the target interpretation network within the platoon-level agent network. After processing by the second policy network within the platoon-level agent network, a platoon-level coordinated action command is output. The policy bias vector is element-wise added to the original logits output by the second policy network, and then processed by a softmax function to obtain the probability distribution of the platoon-level coordinated action command. During backpropagation, the target interpretation network and the second policy network mutually block gradients; the gradient of the target interpretation network does not propagate to the second policy network via the internal reward signal and the policy bias vector. The platoon-level coordinated action command and the single-equipment self-observation data are input into the single-equipment-level intelligent agent network. After being processed by the third strategy network in the single-equipment-level intelligent agent network, the single-equipment tactical action is output. The state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network to generate the uplink summary information, which is then fed back to the input of the company-level agent network to form a command and decision-making closed loop. The uplink attention network uses the fusion features of the tactical target vector and the platoon-level local observation data as a query vector, maps the state information of each unit-level agent to a key vector and a value vector, and generates the uplink summary information by weighted summation of the value vectors based on the similarity between the query vector and each key vector. The chain-level agent network, platoon-level agent network, and single-unit-level agent network are obtained by sequentially training using a hierarchical course learning method. The hierarchical course learning method trains from bottom to top layer by layer in the order of single-unit-level, platoon-level, and chain-level. At each stage, the policy parameters of the trained layers are frozen, and finally all layers are unfrozen for joint fine-tuning.

2. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The tactical target vector is a fixed-length dense vector, employing a hybrid encoding method combining one-hot encoding of the task type and parameter vectors, specifically including: In the mission type coding section, one-hot coding is used to represent predefined tactical mission types including advance, dispersal, deployment, assault, firepower application, occupation of positions and pursuit. The target region encoding section includes normalized target center point coordinates, target region radius, target type identifier, target object identifier, and task priority; The auxiliary parameters include speed requirements, concealment preferences, suggested formations, and expected duration.

3. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The target interpretation network takes the fusion features of the tactical target vector and platoon-level local observation data as input, and outputs the internal reward signal and the policy bias vector. The policy bias vector is added element-wise to the original logits output by the second policy network, and then the probability distribution of the platoon-level coordinated action command is obtained by the softmax function.

4. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 3, characterized in that, The target interpretation network is trained using the following total loss function: L_total = α* L_value + β * L_cf + L_reg Wherein, L_value is the principal loss function, which is calculated based on the prediction error of the target value of the next state based on the internal reward signal; L_cf is the counterfactual baseline loss, which evaluates the marginal contribution by comparing the difference in internal rewards between the actual action and the average action; L_reg is the regularization loss, which includes the parameter L2 regularization term and the KL divergence term to prevent the output of the internal reward signal from collapsing; α is the principal loss weight coefficient and β is the counterfactual baseline loss weight coefficient. The target interpretation network and the second policy network are trained using an alternating optimization method. In each training iteration, the parameters of the second policy network are first fixed and the parameters of the target interpretation network are optimized. Then, the parameters of the target interpretation network are fixed and the second policy network is updated with a weighted combination of the internal reward signal and the external environment reward as the total reward. During backpropagation, the propagation of the gradient of the target interpretation network to the second policy network is blocked.

5. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The processing steps of the uplink attention network include: The state information of each unit is mapped into a key vector and a value vector through an encoding network; the current ranking task context is encoded into a query vector; the importance weight of each unit's state information is calculated through an attention mechanism; and the value vectors are weighted and summed based on the importance weights to generate the uplink summary information with fixed dimensions.

6. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The tiered curriculum learning method includes: In the first stage, the strategy parameters of the fixed company-level agent network and the platoon-level agent network are fixed, and the third strategy network is trained to enable the single-unit-level agent to master individual tactical skills, and internal confrontation training is carried out by dividing the platoon into adversarial groups. In the second stage, the parameters of the third policy network are frozen, the policy parameters of the platoon-level agent network are fixed, the second policy network and the target interpretation network are unlocked to conduct platoon-level collaborative capability training, and inter-platoon confrontation exercises are organized within the platoon. In the third stage, the parameters of the second strategy network, the target interpretation network, and the third strategy network are frozen, and the training company-level global tactical scheduling capability of the first strategy network is unlocked. In the fourth stage, all network parameters are unfrozen, and global joint fine-tuning is performed with a learning rate lower than that of the previous training stages.

7. The method for army unmanned combat swarm confrontation command based on hierarchical multi-agent reinforcement learning according to claim 1, characterized in that, The rank-level agent network also includes a centralized rank-level commentator network, which takes the joint state of the rank as input and provides state value evaluation for the second policy network and the third policy network. The single-assembly-level agent network does not have an independent commentator network.

8. A command and control device for army unmanned combat swarm based on hierarchical multi-agent reinforcement learning, characterized in that, Execution is based on pre-built cascade-level agent networks, platoon-level agent networks, and single-unit-level agent networks, including: The company-level decision-making module is used to input global battlefield situation data and uplink summary information of each platoon into the company-level intelligent agent network. After processing by the first policy network in the company-level intelligent agent network, a corresponding tactical target vector is generated for each subordinate platoon. The tactical target vector includes task type encoding and target parameter vector. The platoon-level coordination module is used to input the tactical target vector and platoon-level local observation data into the platoon-level agent network. The target interpretation network within the platoon-level agent network generates an internal reward signal and a policy bias vector. After processing by the second policy network within the platoon-level agent network, a platoon-level coordinated action command is output. The policy bias vector is element-wise added to the original logits output by the second policy network and then processed by a softmax function to obtain the probability distribution of the platoon-level coordinated action command. During backpropagation, the target interpretation network and the second policy network mutually block gradients; the gradient of the target interpretation network does not propagate to the second policy network via the internal reward signal and the policy bias vector. The single-assembly execution module is used to input the platoon-level coordinated action command and the single-assembly's own observation data into the single-assembly-level intelligent agent network, and output the single-assembly tactical action after processing by the third strategy network in the single-assembly-level intelligent agent network. The uplink communication module is used to generate the uplink summary information after the state information of each unit-level agent is processed by the uplink attention network in the platoon-level agent network, and then fed back to the input of the company-level agent network to form a command and decision-making closed loop. The uplink attention network uses the fusion feature of the tactical target vector and the platoon-level local observation data as the query vector, maps the state information of each unit-level agent to a key vector and a value vector, and generates the uplink summary information by weighted summation of the value vectors based on the similarity between the query vector and each key vector. The chain-level agent network, platoon-level agent network, and single-unit-level agent network are obtained by sequentially training using a hierarchical course learning method. The hierarchical course learning method trains from bottom to top layer by layer in the order of single-unit-level, platoon-level, and chain-level. At each stage, the policy parameters of the trained layers are frozen, and finally all layers are unfrozen for joint fine-tuning.

9. A command and control system for unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement the army unmanned combat swarm confrontation command method based on hierarchical multi-agent reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement a method for command and control of unmanned combat swarms in the army based on hierarchical multi-agent reinforcement learning as described in any one of claims 1 to 7.

Images