A real-time command control strategy optimization method, device and equipment based on multi-agent reinforcement learning and a storage medium

Abstract

The application provides a real-time command control strategy optimization method and device based on multi-agent reinforcement learning, an equipment and a storage medium. A pre-trained cooperative graph network is deployed on the edge equipment of each combat unit, local observation data and its historical sequence are encoded into a low-dimensional local cooperative graph vector, and the original high-dimensional data is replaced as the communication content between units. Combined with an adaptive intention broadcast mechanism, whether to broadcast is dynamically determined according to observation uncertainty and channel state, and on-demand allocation of communication resources is realized. After each unit receives the adjacent cooperative graph vector, the adjacent intention is spatially weighted by a lightweight space-time attention module in the pre-trained local strategy execution network, and the time characteristics of the historical intention of the unit are fused, so that a cooperative and consistent real-time command control instruction is generated without global information convergence.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and command and control, and in particular to a method, apparatus, device and storage medium for real-time command and control strategy optimization based on multi-agent reinforcement learning. Background Technology

[0002] In modern battlefield environments, multiple heterogeneous combat units (drones, unmanned combat vehicles, individual soldier terminals, etc.) need to achieve rapid coordinated decision-making in tactical-level combat scenarios. A typical practical problem is that in multi-unmanned platform cooperative operations under strong electromagnetic warfare conditions, each platform can only obtain local battlefield information through its own sensors. Communication links are constantly facing interference, eavesdropping, or forced silence, while tactical-level decisions need to be completed within a time window of seconds or even milliseconds. This means that the system cannot rely on a central node to gather global information for centralized optimization and decision-making, nor can it allow each unit to act completely independently and lose its cooperative effectiveness—a method is needed that enables each combat unit to autonomously generate coordinated and consistent decision-making behavior under conditions of local perception and limited communication.

[0003] In existing technologies, the first type of approach employs a hierarchical hybrid intelligent architecture, applying multi-agent reinforcement learning algorithms only at the top-level scheduling layer to allocate tasks among formations, while the middle and bottom layers rely on predefined rules for execution. This approach presupposes that global situational information can be stably aggregated to the central node, and the bottom-level decision-making logic is embedded in the rules. Once communication degrades, the upper-level intelligent decision-making loses information input, the bottom-level rules cannot adaptively adjust, and the overall collaborative capability drops sharply. The second type of approach uses a rule-based hierarchical task planning method, constructing a two-layer planning framework based on the OODA loop model, relying entirely on a pre-set rule base for situational assessment and action decisions. The decision-making capability of this approach is limited by the coverage of the rule base; it cannot learn and adjust online to face unforeseen adversarial situations during the training phase, and the large-scale rule matching process suffers significant computational latency when multiple units are concurrent.

[0004] In summary, the existing technologies mainly suffer from the following unresolved technical bottlenecks:

[0005] 1) Rigid coupling between communication load and coordination performance: Existing solutions either broadcast all the time, which has a high exposure risk, or reduce the frequency based solely on rules, which leads to uncontrollable coordination quality and lacks the ability to autonomously determine "when communication is necessary";

[0006] 2) Semantic uncontrollability of intent representation: Conventional dimensionality reduction compression only pursues information retention rate and does not guarantee that the compressed low-dimensional vector carries "alignment semantics required for collaboration", making it difficult for the receiver to extract effective collaborative intent from it;

[0007] 3) High computational threshold for spatiotemporal fusion: Most existing attention mechanisms are designed as global fully connected mechanisms, which have latency bottlenecks when running in real time on edge computing devices.

[0008] This invention utilizes an adaptive broadcasting mechanism and intention... Figure 1 Consistent reward constraints and lightweight, separable spatiotemporal attention systematically solve the above three problems. Therefore, this application is submitted. Summary of the Invention

[0009] This invention discloses a real-time command and control strategy optimization method, device, equipment, and storage medium based on multi-agent reinforcement learning. It aims to solve the problem of how to replace global information aggregation with low-dimensional intention vector interaction to achieve distributed real-time collaborative command and control decision-making in adversarial environments where multiple combat units are locally observable and communication is limited.

[0010] The present invention provides a first embodiment of a real-time command and control strategy optimization method based on multi-agent reinforcement learning, applied to a distributed collaborative decision-making scenario involving multiple combat units. Each combat unit's edge computing device is equipped with a pre-trained collaborative intent network, a pre-trained local policy execution network, and an adaptive intent broadcasting mechanism. Each combat unit performs the following steps:

[0011] Acquire local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector;

[0012] The local observation data and the current communication channel state are input into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step; based on the broadcast decision, the local cooperative intent vector is selectively sent to neighboring combat units in the communication neighborhood, and the cooperative intent vector sent by neighboring combat units in the communication neighborhood is received, denoted as the neighboring cooperative intent vector;

[0013] The local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector are input into the pre-trained local policy execution network. The lightweight spatiotemporal attention module in the pre-trained local policy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector. Then, the command and control command for the current time step is generated and executed.

[0014] Preferably, the pre-trained cooperative intent network and the pre-trained local policy enforcement network are trained in the following manner:

[0015] In the simulation environment, a centralized training and distributed execution framework is adopted. Multiple agents collect their own local observation data, generate their own local collaborative intent vectors through a collaborative intent network, and then fuse their local collaborative intent vectors with their corresponding neighboring collaborative intent vectors through a local policy execution network to generate actions. This is based on factors including team rewards and neighborhood intent. Figure 1 The total loss function, including consistency rewards and communication cost negative rewards, is used to iteratively update the parameters of the collaborative intent network and the local policy execution network through backpropagation until the policy converges; after training, the network is subjected to fixed-point quantization to adapt to edge computing devices.

[0016] Preferably, the neighborhood meaning Figure 1 The calculation method for consistency rewards is as follows:

[0017]

[0018] in, This is the local collaborative intent vector generated by agent i at time step t. Let be the neighboring cooperative intent vector received by agent i from neighboring agent j. For agent i, the communication neighborhood at time step t. Cosine similarity is used to measure the consistency of the directions of two collaborative intent vectors. The task relevance weight is dynamically calculated by the task allocation module based on the overlap between roles and targets among agents.

[0019] Preferably, the communication cost negative reward model is as follows: ,in The communication cost coefficient, An indication function for whether to perform a broadcast at the current time step, used to enable the cooperative intent network and the local policy enforcement network to learn a policy of broadcasting communication only when necessary during training.

[0020] Preferably, the pre-trained collaborative intent network is based on the historical observation sequence of agent i. Generate local collaborative intent vectors The local collaborative intent vector is an abstract representation of the agent's future action goals and required collaboration, and its dimension d is much smaller than the original dimension of the local observation data.

[0021] Preferably, the specific process by which the adaptive intent broadcasting mechanism generates the broadcast decision includes:

[0022] The local observation data Observed features are obtained through the encoder Predicted action distribution based on the observed features Calculate the local observation uncertainty entropy:

[0023]

[0024] Obtain the channel quality index from the current communication channel state and map it to an available communication resource score. ;

[0025] The broadcast probability is calculated based on the local observation uncertainty entropy, the available communication resource score, and the current state value deviation:

[0026]

[0027] in For the sigmoid function, For learnable weights, The degree to which the current state value deviates from the mean;

[0028] If random sampling If the broadcast decision is generated, the local cooperative intent vector is sent to the communication neighborhood with the minimum necessary power; otherwise, the silent broadcast decision is generated, and only the neighboring cooperative intent vector is received.

[0029] Preferably, the lightweight spatiotemporal attention module includes a spatial attention submodule and a temporal attention submodule;

[0030] The spatial attention submodule is used for the set of neighboring collaborative intent vectors. Attention-weighted calculation is performed, with the weights calculated as follows:

[0031]

[0032] in This is the local collaborative intent vector. The neighboring cooperative intent vector, , For linear projection, the dimension of the collaborative intent vector is compressed from d to d / 4 to obtain spatially weighted intent features;

[0033] The time attention submodule analyzes the historical sequence of the local collaborative intent vector. Temporal features are extracted using gated recurrent units or lightweight causal convolutions, and key historical moments are focused on through one-dimensional attention.

[0034]

[0035] in, This is the query vector output by the time encoder at the current time step. The key vectors output by the time encoder at each historical time step are used to obtain the time attention context features;

[0036] The spatially weighted intent features are concatenated with the temporal attention context features, and then input together with the local observation data and the local collaborative intent vector into the pre-trained local policy execution network to generate the command and control instructions.

[0037] The present invention provides a second embodiment of a real-time command and control strategy optimization device based on multi-agent reinforcement learning, applicable to a distributed collaborative decision-making scenario involving multiple combat units, including multiple combat units, each combat unit having the following deployed on its edge computing device:

[0038] The collaborative intent generation module is used to acquire local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector.

[0039] An adaptive communication module is used to input the local observation data and the current communication channel state into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step; based on the broadcast decision, it selectively sends the local cooperative intent vector to neighboring combat units in the communication neighborhood, and receives cooperative intent vectors sent by neighboring combat units in the communication neighborhood, denoted as the neighboring cooperative intent vector;

[0040] The strategy execution module is used to input the local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector into the pre-trained local strategy execution network. The lightweight spatiotemporal attention module in the pre-trained local strategy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector, and then generates and executes the command and control command for the current time step.

[0041] The present invention provides a third embodiment of a real-time command and control strategy optimization device based on multi-agent reinforcement learning, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any of the above embodiments.

[0042] The present invention provides a fourth embodiment of a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device in which the computer-readable storage medium is located, to implement a real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any of the above claims.

[0043] Based on the real-time command and control strategy optimization method, apparatus, device, and storage medium based on multi-agent reinforcement learning provided by this invention, a pre-trained cooperative intent network is deployed on the edge devices of each combat unit to encode local observation data and its historical sequence into low-dimensional local cooperative intent vectors, replacing the original high-dimensional data as the content of inter-unit communication. Combined with an adaptive intent broadcasting mechanism, the broadcasting is dynamically determined according to the observation uncertainty and channel state to realize the on-demand allocation of communication resources. After each unit receives the neighboring cooperative intent vectors, the lightweight spatiotemporal attention module in the pre-trained local policy execution network spatially weights the neighboring intents and integrates the temporal features of its own historical intents to generate coordinated real-time command and control instructions without the need for global information aggregation. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a real-time command and control strategy optimization method based on multi-agent reinforcement learning provided in the first embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of a module of a real-time command and control strategy optimization device based on multi-agent reinforcement learning provided in the second embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] This invention discloses a real-time command and control strategy optimization method, device, equipment, and storage medium based on multi-agent reinforcement learning. It aims to solve the problem of how to replace global information aggregation with low-dimensional intention vector interaction to achieve distributed real-time collaborative command and control decision-making in adversarial environments where multiple combat units are locally observable and communication is limited.

[0049] Please see Figure 1This invention provides a first embodiment of a real-time command and control strategy optimization method based on multi-agent reinforcement learning, applicable to distributed collaborative decision-making scenarios involving multiple combat units with no central node, limited communication bandwidth, and localized observation. To achieve decentralized real-time collaborative decision-making without requiring global information aggregation and simultaneously reduce communication overhead, each combat unit's edge computing device is equipped with a pre-trained collaborative intent network, a pre-trained local policy execution network, and an adaptive intent broadcasting mechanism. Each combat unit executes the following steps:

[0050] In this embodiment, the training process of the pre-trained cooperative intent network and the pre-trained local policy execution network is described in detail. The training phase adopts a centralized training and distributed execution framework, which is executed in a high-fidelity battlefield simulation environment. In each training round, the simulation environment generates a randomized initial battlefield situation. Each agent collects local observation data from its own perception range, inputs the local observation data and its historical observation sequence into the cooperative intent network to generate its own local cooperative intent vector, and then sends the local cooperative intent vector to neighboring agents through the communication neighborhood and receives neighboring cooperative intent vectors sent by neighboring agents. Subsequently, the local observation data, local cooperative intent vector, and neighboring cooperative intent vector are jointly input into the local policy execution network. After being fused by a lightweight spatiotemporal attention module, the action of the current time step is output. After all agents execute the action synchronously, the simulation environment advances to the next time step.

[0051] Parameter updates during training are driven by a total loss function, which is composed of team reward and neighborhood significance. Figure 1 The reward consists of three parts: a consensus reward, a communication cost negative reward, and a team reward. The team reward is determined by a global evaluation from the simulation environment based on the combined actions of all agents, reflecting the overall task completion quality. (Neighborhood intention) Figure 1 Coherent reward, as an intrinsic reward function for task awareness, is used to guide agents to generate semantically consistent cooperative intention vectors when they cannot directly observe each other's states. Specifically, for agent i at time step t, its neighborhood intention... Figure 1 Consistent reward The calculation method is as follows:

[0052]

[0053] in, This is the local collaborative intent vector generated by agent i at time step t. Let be the neighboring cooperative intent vector received by agent i from neighboring agent j. For agent i, the communication neighborhood at time step t. Cosine similarity is calculated as follows:

[0054] This metric measures the degree of alignment between two cooperating intent vectors in direction, rather than the proximity of their absolute values, thus focusing the reward signal on the cooperating alignment of intent directions. The task relevance weight is dynamically calculated by the task allocation module based on the role attributes and target overlap between agents i and j. When two agents are assigned to the same subtask, this weight approaches one to encourage high alignment of intents. When two agents belong to different and unrelated subtasks, this weight approaches zero or decays with distance to avoid forcibly aligning the intents of unrelated agents. The above formula calculates the average weighted cosine similarity of all neighboring agents within the communication neighborhood to obtain the neighborhood intent of agent i at the current time step. Figure 1 The consistency reward value mechanism does not rely on external global supervision signals, but implicitly promotes policy cooperation through local intent similarity measurement, thus alleviating the non-stationarity problem of multi-agent training in some observable environments.

[0055] Negative rewards for communication costs are used to constrain the communication behavior of agents, avoiding unnecessary resource consumption and tactical exposure risks caused by broadcasting. The modeling method is as follows:

[0056] in The communication cost coefficient, This is an indicator function for whether to perform a broadcast at the current time step. It takes a value of one when the agent performs an intention broadcast at this time step, and zero otherwise. This negative reward term is incorporated into the total reward function, enabling the cooperative intention network and the local policy enforcement network to learn a communication restraint strategy during training. That is, they choose to broadcast the local cooperative intention vector only when the local observation uncertainty is high or the cooperative need is urgent, and remain silent when the situation is clear or the cooperative need is low.

[0057] After each training round, the team reward and neighborhood intention are calculated using the global information available in the simulation environment during the training phase. Figure 1 A weighted combination of consistency rewards and communication cost negative rewards yields a total loss function value, including policy gradient loss, intent alignment loss, and value function loss. This value is then used to synchronously update all parameters of the cooperative intent network and the local policy execution network via backpropagation. After numerous training iterations, the policy converges to a stable cooperative decision-making policy. Following training, the converged network undergoes fixed-point quantization, converting floating-point parameters into low-bit-width fixed-point representations. This significantly reduces storage and computational overhead while maintaining decision accuracy, enabling deployment on edge computing devices in various combat units.

[0058] S101, Obtain local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector;

[0059] In this embodiment, during each decision cycle of the battlefield execution phase, each combat unit collects local observation data at the current time step through its own onboard sensor system (including but not limited to radar, photoelectric / infrared detectors, electronic reconnaissance equipment, etc.). This local observation data reflects the battlefield situation information within the combat unit's own perception range, including the position and movement of surrounding enemy targets, the relative orientation of friendly units, terrain features, and electromagnetic environment status. However, due to the physical limitations of sensor detection range and field of view, each combat unit can only acquire local battlefield information within a limited area centered on itself, and cannot perceive the global situation.

[0060] After acquiring the local observation data at the current time step, each combat unit inputs this data, along with the historical observation sequence accumulated over multiple consecutive time steps, into a pre-trained collaborative intent network deployed on a local edge computing device. Let the local observation history of agent i at time step t be... The collaborative intent network encodes the time-series observation information, extracts the situational evolution trend and action requirement features contained therein, and compresses and maps the high-dimensional original observation information into a low-dimensional collaborative intent vector. This is denoted as the local collaborative intent vector. The local collaborative intent vector is an abstract representation of the agent's future action goals and required collaboration. Its semantic connotation encompasses the action patterns the agent tends to adopt under the current situation, such as the attack focus direction, defense focus area, and maneuver intent patterns—tactical-level intent information. The dimension 'd' of the local collaborative intent vector is a small value, such as 8 or 16, much smaller than the dimension of the original local observation data. This design ensures that subsequent communication between agents consists only of this low-dimensional vector, rather than high-dimensional original observation data or specific action commands. This significantly reduces the communication load while preserving key collaborative semantic information, enhancing communication anti-jamming capability and stealth in strong electromagnetic warfare environments. Furthermore, since the local collaborative intent vector is an abstract expression of tactical intent rather than a plaintext description of specific action plans, even if intercepted by the adversary during transmission, it is difficult to directly infer specific operational deployments and action details, thus providing additional protection at the information security level. After the collaborative intent network is encoded, each combat unit obtains a local collaborative intent vector that can be used for subsequent communication interactions and strategy execution.

[0061] S102, input the local observation data and the current communication channel state into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step; according to the broadcast decision, selectively send the local cooperative intent vector to neighboring combat units in the communication neighborhood, and receive the cooperative intent vector sent by the neighboring combat units in the communication neighborhood, denoted as the neighboring cooperative intent vector;

[0062] In this embodiment, after each combat unit obtains its local cooperative intent vector, it needs to decide whether to broadcast the vector to neighboring combat units within the communication neighborhood. In a real battlefield environment, unrestrained communication broadcasting can lead to two negative effects. On the one hand, frequent electromagnetic radiation increases the risk of being detected and located by enemy electronic reconnaissance systems. On the other hand, under conditions of limited communication bandwidth and multiple units sharing a channel, excessive broadcasting can cause channel congestion, resulting in delays in the transmission of critical information. Therefore, an adaptive mechanism that can dynamically adjust communication behavior according to actual needs is required.

[0063] Specifically, the adaptive intent broadcasting mechanism first performs an uncertainty assessment on the local observation data at the current time step. Each combat unit then broadcasts the local observation data... The observed features are obtained after processing by the encoder network. Based on this observation feature, the policy network outputs the predicted probability distribution of each candidate action in the current state. And based on this, calculate the local observation uncertainty entropy:

[0064]

[0065] The uncertainty entropy reflects the degree of confidence of the combat unit in the optimal course of action under the current situation. When the uncertainty entropy is high, it indicates that the combat unit is facing a complex and ambiguous situation and it is difficult to independently judge the optimal action. At this time, it is more necessary to exchange intention information with neighboring units to assist in decision-making. When the uncertainty entropy is low, it indicates that the current situation is clear and the combat unit has a high degree of confidence in making the correct decision independently. At this time, the marginal benefit of communication is low and it is more advantageous to remain silent.

[0066] While calculating the local observation uncertainty entropy, each combat unit obtains the current communication channel quality indicators in real time through the underlying communication module, including parameters such as signal-to-noise ratio and packet loss rate, and maps these indicators together into an available communication resource score with a value between zero and one. The closer the value is to one, the better the current channel quality and the more abundant the available communication resources; the closer the value is to zero, the more severe the channel interference or congestion. In addition, each combat unit also calculates the value deviation of the current state. Value deviation refers to the absolute deviation between the estimated value of the value function under the current state and the historical average. This deviation reflects the degree of abnormality of the current situation relative to the normal state. When the value deviation is large, it means that the combat unit is at a critical situation node that has a significant impact on the success or failure of the overall mission. At this time, the urgency of coordinated communication is even higher.

[0067] Based on the evaluation results of the above three dimensions, the adaptive intent broadcasting mechanism calculates the broadcast probability at the current time step using the following formula:

[0068]

[0069] in, The sigmoid function maps the result of a linear combination to a probability value between zero and one. All are learnable weight parameters greater than zero, obtained through backpropagation optimization during the training phase. The design logic of the above formula lies in the fact that when the local observation uncertainty entropy... The higher the value, the more abundant the communication resources, and the lower the current state value deviation. The larger the value, the greater the result of the weighted sum of the three terms. After sigmoid mapping, the broadcast probability approaches one, meaning that the combat unit is more inclined to broadcast its own intentions. Conversely, when the situation is clear, the channel is poor, and the situation is normal, the broadcast probability approaches zero, and the combat unit tends to remain silent.

[0070] After obtaining the broadcast probability, each combat unit generates a specific broadcast decision through random sampling, that is, sampling a random number from a uniform distribution of zero to one. ,like The broadcast decision then generates a broadcast, and the combat unit transmits the local cooperative intent vector with the minimum necessary power. The purpose of transmitting to nearby combat units within the communication range using the minimum necessary power is to minimize electromagnetic radiation characteristics and reduce the probability of enemy detection while meeting the communication range requirements of the neighborhood; if This generates a silent broadcast decision, whereby the combat unit does not send any information at the current time step, but passively receives the cooperative intent vector broadcast by neighboring combat units only when channel conditions permit. Furthermore, the broadcast frequency can be further constrained, for example, by setting a maximum of one broadcast allowed per T consecutive time steps, to control the cumulative risk of communication exposure over a longer timescale.

[0071] Through the aforementioned adaptive intent broadcasting mechanism, after completing the broadcast decision at the current time step, each combat unit selectively sends its local collaborative intent vector to its communication neighborhood. Simultaneously, it receives collaborative intent vectors from other neighboring combat units that have also executed broadcast decisions within the communication neighborhood; these are denoted as neighboring collaborative intent vectors. It should be noted that since each neighboring combat unit independently executes the same adaptive intent broadcasting mechanism, not all neighboring combat units will send collaborative intent vectors at the same time step. The actual number of neighboring collaborative intent vectors received by each combat unit dynamically changes with the battlefield situation and communication conditions. The lightweight spatiotemporal attention module in the local policy execution network is designed to adapt to this dynamic change in the number of inputs, ensuring that it can effectively fuse and generate reliable command and control instructions when receiving different numbers of neighboring collaborative intent vectors.

[0072] S103, the local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector are input into the pre-trained local policy execution network. The lightweight spatiotemporal attention module in the pre-trained local policy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector, and then generates and executes the command and control command for the current time step.

[0073] In this embodiment, after each combat unit obtains its local collaborative intent vector and completes communication interaction to obtain neighboring collaborative intent vectors, it needs to effectively fuse the aforementioned multi-source information to generate the final command and control instructions. The core challenge of this fusion process is that the collaborative intent vectors sent by different neighboring combat units contribute differently to the current combat unit's decision-making, and the reference value of the intent information accumulated by the combat unit at different historical moments for the current decision-making also varies. Therefore, a fusion mechanism that can adaptively assess and filter the importance of multi-party information is needed.

[0074] In this embodiment, the pre-trained local policy execution network embeds a lightweight spatiotemporal attention module, which differs from the standard Transformer architecture's O(n) complexity. 2 The fully connected multi-head self-attention mechanism employs separate spatial attention sub-modules and temporal attention sub-modules to handle information fusion tasks of different dimensions, which significantly reduces computational overhead while maintaining adaptive information filtering capabilities, thus meeting the real-time inference requirements of edge computing devices.

[0075] In the spatial dimension, the spatial attention submodule performs importance assessment and weighted fusion of all neighboring cooperative intent vectors received at the current time step. Specifically, suppose combat unit i receives communication neighborhood vectors at the current time step. Set of neighboring cooperative intent vectors sent by neighboring combat units The spatial attention submodule uses the local collaborative intent vector of combat unit i. As a query benchmark, for each neighboring collaborative intent vector Calculate attention weights:

[0076]

[0077] in and Two small linear projection matrices are used to project the local collaborative intent vector and the neighboring collaborative intent vector from the original dimension d to a low-dimensional space of d / 4, respectively, before performing an inner product operation. After softmax normalization, the attention weights of each neighboring collaborative intent vector are obtained. The key to this design is that the attention calculation is performed only within the communication neighborhood rather than for all agents globally, and the dimensionality is compressed through linear projection before the inner product operation, which significantly reduces the computational and parameter count compared to the fully connected computation method of the standard Transformer attention mechanism. By weighting and summing the neighboring collaborative intent vectors using the above spatial attention weights, a spatially weighted intent feature is obtained. This feature adaptively highlights the neighboring intent information most relevant to the current combat unit's decision, while suppressing interference from less relevant intents. This allows the combat unit to always extract the most valuable collaborative information when faced with a dynamically changing number of neighboring intent inputs.

[0078] In the time dimension, the time attention submodule analyzes the historical sequence of local collaborative intent vectors generated by the combat unit itself within multiple recent consecutive time steps. Temporal modeling is performed to extract key features from the trends of situational evolution and the patterns of changing intentions. This submodule first encodes the historical intention sequence with temporal features using either gated recurrent units (GRUs) or lightweight causal convolutions. GRUs selectively retain and forget historical information through their internal update and reset gate mechanisms, while lightweight causal convolutions ensure temporal causality by performing convolution operations only on the current and past time steps. Both methods can extract short-term temporal dependencies with low computational cost. Subsequently, the temporal attention submodule applies a one-dimensional attention mechanism to the encoded temporal feature sequence, focusing on the key historical moments most valuable for current decision-making.

[0079]

[0080] in This is the query vector output by the time encoder at the current time step. The key vectors output by the time encoder at each historical time step are used to obtain the temporal attention weights for each historical moment after inner product operation and softmax normalization. The temporal encoding features of each historical moment are weighted and summed using this weight to obtain the temporal attention context features. This mechanism enables combat units to adaptively extract the most relevant reference information from historical experience based on the current situation. For example, when the situation is changing rapidly, more attention can be paid to the evolution of intentions in the most recent moment, while when the situation is relatively stable, trend information over a longer time window can be referenced.

[0081] After completing the attention calculations for both spatial and temporal dimensions, the lightweight spatiotemporal attention module concatenates the spatially weighted intent features with the temporal attention context features. This fusion method differs from the residual superposition method used in the standard Transformer architecture. By concatenating the features, the two dimensions are represented independently before being jointly processed by subsequent network layers, avoiding information confusion that may be caused by the direct superposition of features from different dimensions in residual connections. The concatenated fused features, along with the local observation data and local collaborative intent vector of the combat unit at the current time step, are input into the subsequent fully connected layer of the local policy execution network. After nonlinear transformation, the command and control instructions for the current time step are output. These instructions cover the specific tactical actions that the combat unit needs to perform within the current decision cycle, including but not limited to movement direction and speed, target selection and fire allocation, reconnaissance path planning, and defensive position adjustment. After generating the instructions, the combat unit immediately executes them through its own execution mechanisms (such as flight control systems, weapon control systems, and maneuver drive systems), completing the decision loop for the current time step. In the next time step, the execution loop restarts from step S1, enabling continuous perception, collaborative reasoning, and real-time response to the dynamic battlefield situation.

[0082] Please see Figure 2 The present invention provides a second embodiment of a real-time command and control strategy optimization device based on multi-agent reinforcement learning, applicable to a distributed collaborative decision-making scenario involving multiple combat units, including multiple combat units, each combat unit having the following deployed on its edge computing device:

[0083] The collaborative intent generation module 201 is used to acquire local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector.

[0084] The adaptive communication module 202 is used to input the local observation data and the current communication channel state into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step; according to the broadcast decision, it selectively sends the local cooperative intent vector to neighboring combat units in the communication neighborhood, and receives the cooperative intent vector sent by neighboring combat units in the communication neighborhood, denoted as the neighboring cooperative intent vector.

[0085] The strategy execution module 203 is used to input the local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector into the pre-trained local strategy execution network. The lightweight spatiotemporal attention module in the pre-trained local strategy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector, and then generates and executes the command and control command for the current time step.

[0086] The present invention provides a third embodiment of a real-time command and control strategy optimization device based on multi-agent reinforcement learning, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any of the above embodiments.

[0087] The present invention provides a fourth embodiment of a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device in which the computer-readable storage medium is located, to implement a real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any of the above claims.

[0088] Based on the real-time command and control strategy optimization method, apparatus, device, and storage medium based on multi-agent reinforcement learning provided by this invention, a pre-trained cooperative intent network is deployed on the edge devices of each combat unit to encode local observation data and its historical sequence into low-dimensional local cooperative intent vectors, replacing the original high-dimensional data as the content of inter-unit communication. Combined with an adaptive intent broadcasting mechanism, the broadcasting is dynamically determined according to the observation uncertainty and channel state to realize the on-demand allocation of communication resources. After each unit receives the neighboring cooperative intent vectors, the lightweight spatiotemporal attention module in the pre-trained local policy execution network spatially weights the neighboring intents and integrates the temporal features of its own historical intents to generate coordinated real-time command and control instructions without the need for global information aggregation.

[0089] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a real-time command and control strategy optimization device based on multi-agent reinforcement learning. For example, the apparatus described in the second embodiment of the present invention.

[0090] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the aforementioned real-time command and control strategy optimization method based on multi-agent reinforcement learning, connecting various parts of the method through various interfaces and lines.

[0091] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a real-time command and control strategy optimization method based on multi-agent reinforcement learning. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0092] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0093] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0094] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time command and control strategy optimization method based on multi-agent reinforcement learning, applied to a distributed collaborative decision-making scenario involving multiple combat units, wherein each combat unit's edge computing device is equipped with a pre-trained collaborative intent network, a pre-trained local policy execution network, and an adaptive intent broadcasting mechanism, characterized in that... The pre-trained collaborative intent network and the pre-trained local policy execution network are trained in the following manner: In the simulation environment, a centralized training and distributed execution framework is adopted. Multiple agents collect their own local observation data, generate their own local collaborative intent vectors through a collaborative intent network, and then fuse their local collaborative intent vectors with their corresponding neighboring collaborative intent vectors through a local policy execution network to generate actions. Based on a total loss function that includes team rewards, neighborhood intent consistency rewards, and negative communication cost rewards, the parameters of the collaborative intent network and the local policy execution network are iteratively updated through backpropagation until the policy converges. After training, the network is subjected to fixed-point quantization to adapt to edge computing devices. Each combat unit executes the following steps: Acquire local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector; The local observation data and the current communication channel state are input into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step. Specifically, the local observation data... Observed features are obtained through the encoder Predicted action distribution based on the observed features Calculate the local observation uncertainty entropy: Obtain the channel quality index from the current communication channel state and map it to an available communication resource score. ; The broadcast probability is calculated based on the local observation uncertainty entropy, the available communication resource score, and the current state value deviation: in For the sigmoid function, For learnable weights, The degree to which the current state value deviates from the mean. Let be any action in the candidate action space. The value network is based on the local observation data. The output is the current state value estimate. The historical mean of state values ​​obtained during training; If random sampling If the broadcast decision is generated, the local cooperative intent vector is sent to the communication neighborhood with the minimum necessary power; otherwise, the silent broadcast decision is generated, and only the neighboring cooperative intent vector is received. According to the broadcast decision, the local cooperative intent vector is selectively sent to the neighboring combat units in the communication neighborhood, and the cooperative intent vector sent by the neighboring combat units in the communication neighborhood is received, denoted as the neighboring cooperative intent vector. The local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector are input into the pre-trained local policy execution network. The lightweight spatiotemporal attention module in the pre-trained local policy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector. Then, the command and control command for the current time step is generated and executed.

2. The real-time command and control strategy optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The calculation method for the neighborhood intent consistency reward is as follows: in, This is the local collaborative intent vector generated by agent i at time step t. Let be the neighboring cooperative intent vector received by agent i from neighboring agent j. For agent i, the communication neighborhood at time step t. Cosine similarity is used to measure the consistency of the directions of two collaborative intent vectors. The task relevance weight is dynamically calculated by the task allocation module based on the overlap between roles and targets among agents.

3. The real-time command and control strategy optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The communication cost negative reward model is as follows: ,in The communication cost coefficient, The function indicates whether to perform a broadcast at the current time step. It takes a value of one when the agent performs a broadcast at that time step, and a value of zero otherwise. The negative reward for communication costs is included as a negative reward in the total loss function, so that the collaborative intent network and the local policy execution network will weigh the communication cost generated by each broadcast behavior against the collaborative decision-making benefits brought by the broadcast during the training process.

4. The real-time command and control strategy optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The pre-trained collaborative intent network is based on the historical observation sequence of agent i. Generate local collaborative intent vectors The local collaborative intent vector is an abstract representation of the agent's future action goals and required collaboration, and its dimension d is much smaller than the original dimension of the local observation data.

5. The real-time command and control strategy optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that, The lightweight spatiotemporal attention module includes a spatial attention submodule and a temporal attention submodule; The spatial attention submodule is used for the set of neighboring collaborative intent vectors. Attention-weighted calculation is performed, with the weights calculated as follows: in This is the local collaborative intent vector generated by agent i at time step t. Let be the neighboring cooperative intent vector received by agent i from neighboring agent j. , For linear projection, the dimension of the collaborative intent vector is compressed from d to d / 4 to obtain spatially weighted intent features; The time attention submodule analyzes the historical sequence of the local collaborative intent vector. Temporal features are extracted using gated recurrent units or lightweight causal convolutions, and key historical moments are focused on through one-dimensional attention. in, This is the query vector output by the time encoder at the current time step. The key vectors output by the time encoder at each historical time step are used to obtain the time attention context features; The spatially weighted intent features are concatenated with the temporal attention context features, and then input together with the local observation data and the local collaborative intent vector into the pre-trained local policy execution network to generate the command and control instructions.

6. A real-time command and control strategy optimization device based on multi-agent reinforcement learning, applied to a distributed collaborative decision-making scenario involving multiple combat units, wherein each combat unit's edge computing device is equipped with a pre-trained collaborative intent network, a pre-trained local policy execution network, and an adaptive intent broadcasting mechanism, characterized in that... The pre-trained collaborative intent network and the pre-trained local policy execution network are trained in the following manner: In the simulation environment, a centralized training and distributed execution framework is adopted. Multiple agents collect their own local observation data, generate their own local collaborative intent vectors through a collaborative intent network, and generate actions by fusing their local collaborative intent vectors with their corresponding neighboring collaborative intent vectors through a local policy execution network. Based on the total loss function, which includes team reward, neighborhood intent consistency reward and communication cost negative reward, the parameters of the collaborative intent network and the local policy execution network are iteratively updated through backpropagation until the policy converges. After training, the network undergoes point-to-point quantization to adapt it to edge computing devices. Each combat unit includes: The collaborative intent generation module is used to acquire local observation data at the current time step, input the local observation data and its corresponding historical observation sequence into the pre-trained collaborative intent network, and generate a low-dimensional collaborative intent vector at the current time step, denoted as the local collaborative intent vector. The adaptive communication module is used to input the local observation data and the current communication channel state into the adaptive intent broadcasting mechanism to generate a broadcast decision for the current time step. Specifically, it inputs the local observation data... Observed features are obtained through the encoder Predicted action distribution based on the observed features Calculate the local observation uncertainty entropy: Obtain the channel quality index from the current communication channel state and map it to an available communication resource score. ; The broadcast probability is calculated based on the local observation uncertainty entropy, the available communication resource score, and the current state value deviation: in For the sigmoid function, For learnable weights, The degree to which the current state value deviates from the mean. Let be any action in the candidate action space. The value network is based on the local observation data. The output is the current state value estimate. The historical mean of state values ​​obtained during training; If random sampling If the broadcast decision is generated, the local cooperative intent vector is sent to the communication neighborhood with the minimum necessary power; otherwise, the silent broadcast decision is generated, and only the neighboring cooperative intent vector is received. According to the broadcast decision, the local cooperative intent vector is selectively sent to the neighboring combat units in the communication neighborhood, and the cooperative intent vector sent by the neighboring combat units in the communication neighborhood is received, denoted as the neighboring cooperative intent vector. The strategy execution module is used to input the local observation data, the local collaborative intent vector, and the neighboring collaborative intent vector into the pre-trained local strategy execution network. The lightweight spatiotemporal attention module in the pre-trained local strategy execution network performs spatial attention weighting on the neighboring collaborative intent vector and extracts temporal attention features by combining the historical sequence of the local collaborative intent vector, and then generates and executes the command and control command for the current time step.

7. A real-time command and control strategy optimization device based on multi-agent reinforcement learning, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement a real-time command and control strategy optimization method based on multi-agent reinforcement learning as described in any one of claims 1 to 5.

Images